It would be interesting to know list members' experiences wrt how often and
under what conditions exact tests led to different substantive statements.
I haven't had the need to use exact tests in recent years (decades?) but
in Chapter 2 of the SPSS Exact Tests ver 21 manual (p31) the authors
identify the conditions under which the "asymptotic test/prob" (i.e., usual
test statistic) can be different from the "Exact test/prob". I copy the
relevant text below:
When to Use Asymptotic P Values
Although the exact p value can be shown to converge mathematically to the corresponding asymptotic
p value as the sample size becomes infinitely large, this property is not of much practical value
in guaranteeing the accuracy of the asymptotic p value for any specific data set. There are many
different data configurations where the asymptotic methods perform poorly. These include small data
sets, data sets containing ties, large but unbalanced data sets, and sparse data sets. A numerical
example follows for each of these situations.
*********************************************************
There are probably other conditions where the obtained/asymptotic test/probability
will differ from the exact test/prob but I think the major distinction might be between
experimental designs and observational studies, with the latter providing more
opportunities for differences between asymptotic vs exact value. With good experimental
design one should be able to avoid small datasets, unbalanced groupings, sparse
data set, etc. But this may also depend upon whether the independent variables
are actually under control of the research, the populations that are sampled have
adequate numbers of available participants, and other oddities that can affect data
collection.
-Mike Palij
New York University
-----
Art Kendall
Social Research Consultants