Re: exact versus asymptotic p-value under normal circumstances

Posted by Bruce Weaver on
URL: http://spssx-discussion.165.s1.nabble.com/Revision-control-for-SPSS-source-code-tp4267835p4268413.html

Hi Tony.  I disagree about exact tests always trumping asymptotic tests.  Take the 2x2 table, for example.  Fisher's exact test was designed for the situation where both the row and column totals are fixed in advance (Model I, as Barnard 1947 called it).  If that is not the case, FET is known to be too conservative.  When the marginal totals are not fixed in advance, the N-1 chi-square test (an asymptotic test) is a much better choice, IMO.  

For more details about this particular example, take a look at Ian Campbell's website, and his article in Statistics in Medicine.

   http://www.iancampbell.co.uk/twobytwo/background.htm

The same thing applies to certain confidence intervals.  E.g., exact (Clopper-Pearson) confidence intervals for binomial proportions are considered by many to be inferior to other methods.  Here's a note from the GraphPad site that makes this argument:

   http://www.graphpad.com/articles/CIofProportion.htm

Cheers,
Bruce

Anthony Babinec wrote

The exact test always "trumps" the asymptotic test; you should use the exact
test
result. The rule of thumb about the expected value in any cell falling below
5 is just
that - a rule of thumb - and no more. Exact tests are useful in situations
of sparseness,
skewness, extreme imbalance, and small sample sizes. It is difficult to
state before
the fact when you would get a different result.

Tony Babinec
[e-mail]

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[e-mail]] On Behalf Of
Bridgette Portman
Sent: Monday, March 28, 2011 3:34 PM
To: [e-mail]
Subject: exact versus asymptotic p-value under normal circumstances

One more question...

I have the SPSS Exact Tests module, which allows one to compute an exact
p-value for a chi-square test on any size table (not just 2x2). I was taught
to use exact tests under certain circumstances, e.g. when the expected value
in any cell falls below 5. My dataset is about 250, so usually that
condition doesn't apply. However, I am still finding that the exact p-value
differs significantly from the asymptotic p value...sometimes to the extent
that they lead to different conclusions.
I'm not sure whether I should rely on the exact p-value anytime it differs
from the asymptotic one, or if I should stick with the asymptotic when the
usual assumptions for chi-square are met.

For example:  On the crosstabs I just ran, X2=4.49, 3 degrees of freedom,
asymptotic sig is showing up as .212, and exact sig as .028. All expected
cell counts were greater than 5.

If I didn't have the exact tests module, I would have just gone with the
.212 and retained the null hypothesis. Now that I have exact, should I use
it or not?

Thanks for any advice,
Bridgette

=====================
To manage your subscription to SPSSX-L, send a message to
[e-mail] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command SIGNOFF SPSSX-L For a list of
commands to manage subscriptions, send the command INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[e-mail] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD