Posted by
Rich Ulrich on
URL: http://spssx-discussion.165.s1.nabble.com/Revision-control-for-SPSS-source-code-tp4267835p4268738.html
> Date: Mon, 28 Mar 2011 13:34:29 -0700
> From:
[hidden email]
> Subject: exact versus asymptotic p-value under normal circumstances
> To:
[hidden email]
>
> One more question...
>
> I have the SPSS Exact Tests module, which allows one to compute an exact
> p-value for a chi-square test on any size table (not just 2x2). I was
> taught to use exact tests under certain circumstances, e.g. when the
> expected value in any cell falls below 5. My dataset is about 250, so
> usually that condition doesn't apply. However, I am still finding that the
> exact p-value differs significantly from the asymptotic p
> value...sometimes to the extent that they lead to different conclusions.
> I'm not sure whether I should rely on the exact p-value anytime it differs
> from the asymptotic one, or if I should stick with the asymptotic when the
> usual assumptions for chi-square are met.
The difference in results is *not* due to the failure of the Pearson
approximation -- That one "fails" when there are tiny Expectations, and
the symptom is that the resulting test is too big. You have the opposite
result - your Pearson test is less significant.
I expect that if you look at the other "approximate" test, the Likelihood
test, you will also see a difference from the Pearson. For a table larger
than 2x2, the two tests effectively "weight" differently the observed
deviations from Expected. The Pearson will give a tinier p-value when
there tends to have a single extreme value, and it will give a larger p-
value when deviations are shown in several cells.
Agresti (Categorical Data Analysis, 1990) mentions the family of "power
divergence" statistics introduced by Cressie and Read (1984). These two
tests are special cases of the family.
- The so-called "exact" tests offer more than one choice for weighting,
if I am not recalling a different package. Several choices certainly do
exist. Anyway, the difference in weighting is almost surely the source
of your observation. That's a little detail about "exact" which is too
often overlooked.
>
> For example: On the crosstabs I just ran, X2=4.49, 3 degrees of freedom,
> asymptotic sig is showing up as .212, and exact sig as .028. All expected
> cell counts were greater than 5.
>
> If I didn't have the exact tests module, I would have just gone with the
> .212 and retained the null hypothesis. Now that I have exact, should I use
> it or not?
The most important thing is that you don't cherry-pick among results. That is,
you should use the same version of testing across all your results.
I prefer to see a chi-squared tests, because the X^2's give me another basis
for comparing various results within a study. (And a bit of redundancy,
to help avoid typographical errors, etc.)
The next important thing might be -- Use whatever is standard in the journals
where you may publish.
--
Rich Ulrich
=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD