t-test with small n

[aschoff]

Hello,

we are dealing with survey data for two independent groups we would
like to compare. Due to missing values we have very small and unevenly
distributed data for some parts of the survey (e.g. 5 responses for
one group, 8 responses for the control group, answers on likert
scales). We have a discussion whether a t-test can be used to compare
these two groups.

We assume that is is not appropriate to use the t-test but we would
like to understand the exact nature of the problem. We are aware of
the requirments for the t-test like normal distribution, equal
variances etc. We are also aware that this test has a low test-power,
however when we try it out the test turns out to be significant.

What would be the correct argument to refuse using the t-test for this
situation (if it is to be refused?). Is it just because we are having
difficulties to prove that we meet the requirements? What would you
consider a lower boundary that justifies to use a t-test? Would it be
more appropriate to use a non parametric test or is it just impossible
to show a systematic difference with a small N like this?

Thanks a lot for your input.
Robinson

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Bruce Weaver]

aschoff wrote

Hello,

we are dealing with survey data for two independent groups we would
like to compare. Due to missing values we have very small and unevenly
distributed data for some parts of the survey (e.g. 5 responses for
one group, 8 responses for the control group, answers on likert
scales). We have a discussion whether a t-test can be used to compare
these two groups.

We assume that is is not appropriate to use the t-test but we would
like to understand the exact nature of the problem. We are aware of
the requirments for the t-test like normal distribution, equal
variances etc. We are also aware that this test has a low test-power,
however when we try it out the test turns out to be significant.

What would be the correct argument to refuse using the t-test for this
situation (if it is to be refused?). Is it just because we are having
difficulties to prove that we meet the requirements? What would you
consider a lower boundary that justifies to use a t-test? Would it be
more appropriate to use a non parametric test or is it just impossible
to show a systematic difference with a small N like this?

Thanks a lot for your input.
Robinson

You might find this BMJ note useful.

  http://www.bmj.com/cgi/content/full/338/apr06_1/a3166

[David Hitchin]

On Tue, 26 Jan 2010 10:56:23 +0100, Robinson Aschoff <[hidden email]>
wrote:
 Due to missing values we have very small and unevenly
> distributed data for some parts of the survey (e.g. 5 responses for
> one group, 8 responses for the control group, answers on likert
> scales). We have a discussion whether a t-test can be used to compare
> these two groups.

The whole point of the t-test is that it is designed for SMALL samples. For
larger samples, perhaps beginning at 30 or so, tests based directly on the
normal distribution are adequate, and as samples get larger, normal-based
tests become even more appropriate. As the t distribution approximates the
normal distribution when samples get very large, it is quite OK to continue
using the t distribution.

One of the apparent paradoxes in this situation is that you need much
larger samples to check that the test is appropriate than to actually do
the test. Often there may be reasons to assume that the populations are
normal - or reasonably close to it - without having to check this on the
particular samples. The tests might have been used many times before in
similar situations, and therefore there might be good reason to assume that
the distributions are the same in the new samples, with the exception of
possibly different means and variances.

As tests of normality are not very powerful with small samples, you might
be tempted to eye-ball the data. BEWARE! You need quite a lot of experience
before you can judge whether your samples deviate much from normality. To
check my assertion, you can use SPSS to generate many samples from normal
distributions, and draw the histograms. See how many YOU think are normal,
or even better, ask a friend to judge. Of course this experiment needs to
be tried several times with different sample sizes. Most people who haven't
worked systematically through a lot of teaching data reject far too many
samples as "non-normal".

The t-test is pretty robust, but there are exceptions. One case is where
the distributions are highly skewed. Another is where the variances are
very different - in which case SPSS provides a modified version of the test
which takes this into account.

If you are doing the test to see if there is significance, say at the 5%
level, and you get results better than 1% or worse than 10% your
conclusions should be quite clear - even if the p-value isn't right to four
decimal places. If the results are so obviously significant or not
significant, there is little point in seeking more refined tests to get
more accurate p-values.

David Hitchin

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD