Sample size

[Humphrey Paulie]

Dear folks,

I have a very small sample of 30 subjects. I have divided the sample into 8 groups. In each group there are approximately 3-4 subjects. I want to run one-eay ANOVA but with 4 subjects in each group the results cannot be very dependeble, right?  Is there any way around the problem (except testing more people)?

How about simulation on the basis of existing data? Does it work?

Id be thankful for your comments.

Regards

Humphrey

 

[SR Millis-3]

I think that we need to know more about your design, eg,

--hypotheses

--dependent variable

--groupings

Scott Millis

~~~~~~~~~~~
"Kunst ist schön, macht aber viel Arbeit."

Scott R Millis, PhD, ABPP (CN,CL,RP), CStat, CSci
Professor & Director of Research
Dept of Physical Medicine & Rehabilitation
Dept of Emergency Medicine
Wayne State University School of Medicine
261 Mack Blvd
Detroit, MI 48201
Email:  [hidden email]
Tel: [phone]
Fax: [phone]


--- On Mon, 10/19/09, Humphrey Paulie <[hidden email]> wrote:

> From: Humphrey Paulie <[hidden email]>
> Subject: Sample size
> To: [hidden email]
> Date: Monday, October 19, 2009, 11:50 AM
>
> Dear folks,
> I
> have a very small sample of 30 subjects. I have divided the
> sample into 8 groups. In each group there are approximately
> 3-4 subjects. I want to run one-eay ANOVA but with 4
> subjects in each group the results cannot be very
> dependeble, right?  Is there any way around the problem
> (except testing more people)?
> How
> about simulation on the basis of existing data? Does it
> work?
> Id
> be thankful for your comments.
> Regards
> Humphrey
>
>
>
>
>
>
>
>

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Khaleel Hussaini]

Mr. Humphrey,
       I would be concerned with running any parametric statistics with such a small sample. Is it reasonable to assume the variables you are examining are normally distributed? I would recommend using nonparametric statistics especially Kruskal-Wallis test. K-W is similar to One-way ANOVA except that it does not make any assumptions about gaussian distribution. Best,

NPAR TESTS
/K-W=variables of interest
/Missing Analysis.

On Mon, Oct 19, 2009 at 8:50 AM, Humphrey Paulie <[hidden email]> wrote:

Dear folks,

I have a very small sample of 30 subjects. I have divided the sample into 8 groups. In each group there are approximately 3-4 subjects. I want to run one-eay ANOVA but with 4 subjects in each group the results cannot be very dependeble, right?  Is there any way around the problem (except testing more people)?

How about simulation on the basis of existing data? Does it work?

Id be thankful for your comments.

Regards

Humphrey

 

[Steve Simon, P.Mean Consulting]

Humphrey Paulie wrote:

> I have a very small sample of 30 subjects. I have divided the sample
> into 8 groups. In each group there are approximately 3-4 subjects. I
> want to run one-eay ANOVA but with 4 subjects in each group the results
> cannot be very dependeble, right?  Is there any way around the problem
> (except testing more people)?
 >
> How about simulation on the basis of existing data? Does it work?

I get this sort of question all the time. What you really want is the
"Blood From A Turnip" test.

Actually, the word "dependable" can mean several different things.

1. Can you insure that the Type I error rate is actually 0.05?
2. Can you guarantee that this test will reject the null
hypothesis?(Actually, I can do this for you, as long as you don't mind
an alpha level of 1.0).
3. A more reasonable approach than question #2 is: Can you insure that
this test will declare statistical significance when there is a
clinically important difference among the groups? (Alternately: Is the
Type II error rate small/is the power large?)
4. Can you insure that a peer reviewer won't reject my work because of
the small sample size?

A closely related question is:

5. Does my small sample size make my test more sensitive to assumptions.

As someone else noted, there is no easy answer to this question without
knowing more about your problem. I would suggest, however, that
simulations are not a good choice for you. Simulations are messy and
error prone and are best handled by someone who is more experienced than
you are. They may help with Type I error rate and sensitivity to
assumptions, but don't help as much as with other issues, especially
Type II error rates.

Here's what I would suggest.

1. Did you conduct a power calculation prior to the start of data
collection? I already know that the answer is "no" (otherwise you would
have mentioned the power calculation in your email). I would strongly
encourage you to consider conducting a power calculation before you
start your next research study.  Failure to conduct a power calculation
prior to data collection is a serious breach of scientific integrity and
research ethics. The fact that it is done by huge numbers of researchers
besides yourself doesn't make it any less problematic. The biggest
crisis in research today is that we have too many researchers chasing
too few research subjects. We need to be doing less research and doing
it better. This often means taking the time to conduct a multi-center
trail rather than research at a single site. It also means less fame for
the average researcher, because as part of a larger research team, you
are less likely to get that glorious indication of status, a first
author publication.

2. If you don't have a strong a priori reason to consider an alternative
to ANOVA and if your data does not seem to have too unusual a
distribution, stick with what you know: ANOVA. If you have used
alternatives before, like randomization tests, bootstrapping, or
nonparametric procedures, go ahead and run them if you like. But don't
run something that you've never run before unless you get some expert
guidance. I'm also a big fan of transformations if you have trouble
meeting your underlying assumptions, and transformations (especially the
log transformation) are pretty easy to apply.

3. Always, always, always, report confidence intervals with your
results. The width of the confidence intervals is a measure of the
adequacy of the sample size. It might be that you have very narrow
confidence intervals in spite of your small sample size because you have
a carefully controlled setting where noise is kept to a very low level.
The confidence interval is then your best refutation to a peer reviewer
who might suggest that the small sample size makes your work
unpublishable. On the other hand, if your confidence intervals are wide
enough to drive a truck through, then that is the best way to honestly
own up to the limitations of your small sample size.

I hope this helps. I apologize for appearing a bit strident above, but
the problem of too many researchers and too few subjects is an issue
that needs to be emphasized at every opportunity.

P.S. It may be an awful thing to do, and I can't tell without more
context. But have you considered combining some of your groups? An
analysis with 3 groups of roughly 10-11 people is going to be less
sensitive to assumptions and may have superior power. But it might be
like mixing apples and oranges. If the groups are extremely dissimilar
then lumping some of them together will greatly increase your noise
level, which might outweigh any other gains you might accrue.
--
Steve Simon, Standard Disclaimer
Sign up for The Monthly Mean at www.pmean.com/news

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[E. Bernardo]

Hi Steve,

 

Can you suggest a free/downloadable power analysis software?

 

Eins

--- On Mon, 10/19/09, Steve Simon, P.Mean Consulting <[hidden email]> wrote:

From: Steve Simon, P.Mean Consulting <[hidden email]>
Subject: Re: Sample size
To: [hidden email]
Date: Monday, 19 October, 2009, 8:02 PM

Humphrey Paulie wrote:

> I have a very small sample of 30 subjects. I have divided the sample
> into 8 groups. In each group there are approximately 3-4 subjects. I
> want to run one-eay ANOVA but with 4 subjects in each group the results
> cannot be very dependeble, right?  Is there any way around the problem
> (except testing more people)?
>
> How about simulation on the basis of existing data? Does it work?

I get this sort of question all the time. What you really want is the
"Blood From A Turnip" test.

Actually, the word "dependable" can mean several different things.

1. Can you insure that the Type I error rate is actually 0.05?
2. Can you guarantee that this test will reject the null
hypothesis?(Actually, I can do this for you, as long as you don't mind
an alpha level of 1.0).
3. A more reasonable approach than question #2 is: Can you insure that
this test will declare statistical significance when there is a
clinically important difference among the groups? (Alternately: Is the
Type II error rate small/is the power large?)
4. Can you insure that a peer reviewer won't reject my work because of
the small sample size?

A closely related question is:

5. Does my small sample size make my test more sensitive to assumptions.

As someone else noted, there is no easy answer to this question without
knowing more about your problem. I would suggest, however, that
simulations are not a good choice for you. Simulations are messy and
error prone and are best handled by someone who is more experienced than
you are. They may help with Type I error rate and sensitivity to
assumptions, but don't help as much as with other issues, especially
Type II error rates.

Here's what I would suggest.

1. Did you conduct a power calculation prior to the start of data
collection? I already know that the answer is "no" (otherwise you would
have mentioned the power calculation in your email). I would strongly
encourage you to consider conducting a power calculation before you
start your next research study.  Failure to conduct a power calculation
prior to data collection is a serious breach of scientific integrity and
research ethics. The fact that it is done by huge numbers of researchers
besides yourself doesn't make it any less problematic. The biggest
crisis in research today is that we have too many researchers chasing
too few research subjects. We need to be doing less research and doing
it better. This often means taking the time to conduct a multi-center
trail rather than research at a single site. It also means less fame for
the average researcher, because as part of a larger research team, you
are less likely to get that glorious indication of status, a first
author publication.

2. If you don't have a strong a priori reason to consider an alternative
to ANOVA and if your data does not seem to have too unusual a
distribution, stick with what you know: ANOVA.. If you have used
alternatives before, like randomization tests, bootstrapping, or
nonparametric procedures, go ahead and run them if you like. But don't
run something that you've never run before unless you get some expert
guidance. I'm also a big fan of transformations if you have trouble
meeting your underlying assumptions, and transformations (especially the
log transformation) are pretty easy to apply.

3. Always, always, always, report confidence intervals with your
results. The width of the confidence intervals is a measure of the
adequacy of the sample size. It might be that you have very narrow
confidence intervals in spite of your small sample size because you have
a carefully controlled setting where noise is kept to a very low level.
The confidence interval is then your best refutation to a peer reviewer
who might suggest that the small sample size makes your work
unpublishable. On the other hand, if your confidence intervals are wide
enough to drive a truck through, then that is the best way to honestly
own up to the limitations of your small sample size.

I hope this helps. I apologize for appearing a bit strident above, but
the problem of too many researchers and too few subjects is an issue
that needs to be emphasized at every opportunity.

P.S. It may be an awful thing to do, and I can't tell without more
context. But have you considered combining some of your groups? An
analysis with 3 groups of roughly 10-11 people is going to be less
sensitive to assumptions and may have superior power. But it might be
like mixing apples and oranges. If the groups are extremely dissimilar
then lumping some of them together will greatly increase your noise
level, which might outweigh any other gains you might accrue.
--
Steve Simon, Standard Disclaimer
Sign up for The Monthly Mean at www.pmean.com/news

=====================
To manage your subscription to SPSSX-L, send a message to
LISTSERV@... (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

---

Connect instantly with more friends on your blog and personal website?
Create your latest Pingbox today!

[Sarah Carroll]

See http://www.psycho.uni-duesseldorf.de/aap/projects/gpower/

 

 

Sarah Carroll, PhD

Research Director & Associate Professor, PsyD Program

JFKU Graduate School of Professional Psychology

100 Ellinwood Way, Pleasant Hill, CA  94523

ofc: 925.969.3496     email: [hidden email]

 

From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Eins Bernardo
Sent: Monday, October 19, 2009 6:34 PM
To: [hidden email]
Subject: Re: [SPSSX-L] Sample size

 

Hi Steve,

 

Can you suggest a free/downloadable power analysis software?

 

Eins

--- On Mon, 10/19/09, Steve Simon, P.Mean Consulting <[hidden email]> wrote:

From: Steve Simon, P.Mean Consulting <[hidden email]>
Subject: Re: Sample size
To: [hidden email]
Date: Monday, 19 October, 2009, 8:02 PM

Humphrey Paulie wrote:

> I have a very small sample of 30 subjects. I have divided the sample
> into 8 groups. In each group there are approximately 3-4 subjects. I
> want to run one-eay ANOVA but with 4 subjects in each group the results
> cannot be very dependeble, right?  Is there any way around the problem
> (except testing more people)?
>
> How about simulation on the basis of existing data? Does it work?

I get this sort of question all the time. What you really want is the
"Blood From A Turnip" test.

Actually, the word "dependable" can mean several different things.

1. Can you insure that the Type I error rate is actually 0.05?
2. Can you guarantee that this test will reject the null
hypothesis?(Actually, I can do this for you, as long as you don't mind
an alpha level of 1.0).
3. A more reasonable approach than question #2 is: Can you insure that
this test will declare statistical significance when there is a
clinically important difference among the groups? (Alternately: Is the
Type II error rate small/is the power large?)
4. Can you insure that a peer reviewer won't reject my work because of
the small sample size?

A closely related question is:

5. Does my small sample size make my test more sensitive to assumptions.

As someone else noted, there is no easy answer to this question without
knowing more about your problem. I would suggest, however, that
simulations are not a good choice for you. Simulations are messy and
error prone and are best handled by someone who is more experienced than
you are. They may help with Type I error rate and sensitivity to
assumptions, but don't help as much as with other issues, especially
Type II error rates.

Here's what I would suggest.

1. Did you conduct a power calculation prior to the start of data
collection? I already know that the answer is "no" (otherwise you would
have mentioned the power calculation in your email). I would strongly
encourage you to consider conducting a power calculation before you
start your next research study.  Failure to conduct a power calculation
prior to data collection is a serious breach of scientific integrity and
research ethics. The fact that it is done by huge numbers of researchers
besides yourself doesn't make it any less problematic. The biggest
crisis in research today is that we have too many researchers chasing
too few research subjects. We need to be doing less research and doing
it better. This often means taking the time to conduct a multi-center
trail rather than research at a single site. It also means less fame for
the average researcher, because as part of a larger research team, you
are less likely to get that glorious indication of status, a first
author publication.

2. If you don't have a strong a priori reason to consider an alternative
to ANOVA and if your data does not seem to have too unusual a
distribution, stick with what you know: ANOVA.. If you have used
alternatives before, like randomization tests, bootstrapping, or
nonparametric procedures, go ahead and run them if you like. But don't
run something that you've never run before unless you get some expert
guidance. I'm also a big fan of transformations if you have trouble
meeting your underlying assumptions, and transformations (especially the
log transformation) are pretty easy to apply.

3. Always, always, always, report confidence intervals with your
results. The width of the confidence intervals is a measure of the
adequacy of the sample size. It might be that you have very narrow
confidence intervals in spite of your small sample size because you have
a carefully controlled setting where noise is kept to a very low level.
The confidence interval is then your best refutation to a peer reviewer
who might suggest that the small sample size makes your work
unpublishable. On the other hand, if your confidence intervals are wide
enough to drive a truck through, then that is the best way to honestly
own up to the limitations of your small sample size.

I hope this helps. I apologize for appearing a bit strident above, but
the problem of too many researchers and too few subjects is an issue
that needs to be emphasized at every opportunity.

P.S. It may be an awful thing to do, and I can't tell without more
context. But have you considered combining some of your groups? An
analysis with 3 groups of roughly 10-11 people is going to be less
sensitive to assumptions and may have superior power. But it might be
like mixing apples and oranges. If the groups are extremely dissimilar
then lumping some of them together will greatly increase your noise
level, which might outweigh any other gains you might accrue.
--
Steve Simon, Standard Disclaimer
Sign up for The Monthly Mean at www.pmean.com/news

=====================
To manage your subscription to SPSSX-L, send a message to
LISTSERV@... (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

 

---

Connect instantly with more friends on your blog and personal website?
Create your latest Pingbox today!

[Marta Garcia-Granero]

Hi

Khaleel Hussaini wrote:

> I would be concerned with running any parametric statistics with such
> a small sample. Is it reasonable to assume the variables you are
> examining are normally distributed? I would recommend using
> nonparametric statistics especially Kruskal-Wallis test. K-W is
> similar to One-way ANOVA except that it does not make any assumptions
> about gaussian distribution.
>
> NPAR TESTS
> /K-W=variables of interest
> /Missing Analysis.

I must disagree. This is a very common error to be avoided. With very
small sample sizes, like in this case, non parametric tests should be
avoided because they can NEVER render significant results. Even if KW
gave a significant result, post-hoc comparisons based on Mann-Whitney U
tests would never be significant with sample sizes below 5. You can read
more on the topic (from a more solid source than me) here:

1) Bland JM, Altman DG. (2009) Analysis of continuous data from small
samples. 338, a3166. http://www.bmj.com/cgi/content/full/338/apr06_1/a3166.
2) An excerpt of Martin Bland's book An Introduction to Medical
Statistics: Parametric or non-parametric methods?
"There is a common misconception that when the number of observations is
very small, […], Normal distribution methods such as t tests and
regression must not be used and that rank methods should be used
instead. I have never seen any argument put forward in support of this,
but inspection of the tables of the test statistics for rank methods
will show that it is nonsense. For such small samples rank tests cannot
produce any significance at the usual 5% level. Should one need
statistical analysis of such small samples, Normal methods are required."


With an overall sample size of 30 subjects, normality can (and must) be
checked on the residuals, and if data are reasonably normal (or, at
least, not very deviated from normality) then oneway ANOVA should be
used instead of Kruskal-Wallis.

Best regards,
Marta GG

>
> On Mon, Oct 19, 2009 at 8:50 AM, Humphrey Paulie
> <[hidden email] <mailto:[hidden email]>> wrote:
>
>     Dear folks,
>     I have a very small sample of 30 subjects. I have divided the
>     sample into 8 groups. In each group there are approximately 3-4
>     subjects. I want to run one-eay ANOVA but with 4 subjects in each
>     group the results cannot be very dependeble, right? Is there any
>     way around the problem (except testing more people)?
>     How about simulation on the basis of existing data? Does it work?
>     Id be thankful for your comments.
>     Regards
>     Humphrey
>
>
>
>

--
For miscellaneous SPSS related statistical stuff, visit:
http://gjyp.nl/marta/

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Bruce Weaver]

García-Granero wrote

Hi

Khaleel Hussaini wrote:
> I would be concerned with running any parametric statistics with such
> a small sample. Is it reasonable to assume the variables you are
> examining are normally distributed? I would recommend using
> nonparametric statistics especially Kruskal-Wallis test. K-W is
> similar to One-way ANOVA except that it does not make any assumptions
> about gaussian distribution.
>
> NPAR TESTS
> /K-W=variables of interest
> /Missing Analysis.

I must disagree. This is a very common error to be avoided. With very
small sample sizes, like in this case, non parametric tests should be
avoided because they can NEVER render significant results. Even if KW
gave a significant result, post-hoc comparisons based on Mann-Whitney U
tests would never be significant with sample sizes below 5. You can read
more on the topic (from a more solid source than me) here:

1) Bland JM, Altman DG. (2009) Analysis of continuous data from small
samples. 338, a3166. http://www.bmj.com/cgi/content/full/338/apr06_1/a3166.
2) An excerpt of Martin Bland's book An Introduction to Medical
Statistics: Parametric or non-parametric methods?
"There is a common misconception that when the number of observations is
very small, […], Normal distribution methods such as t tests and
regression must not be used and that rank methods should be used
instead. I have never seen any argument put forward in support of this,
but inspection of the tables of the test statistics for rank methods
will show that it is nonsense. For such small samples rank tests cannot
produce any significance at the usual 5% level. Should one need
statistical analysis of such small samples, Normal methods are required."


With an overall sample size of 30 subjects, normality can (and must) be
checked on the residuals, and if data are reasonably normal (or, at
least, not very deviated from normality) then oneway ANOVA should be
used instead of Kruskal-Wallis.

Best regards,
Marta GG
>

Thank you for posting this, Marta.  I particularly like the last couple paragraphs of the BMJ article.


--- start excerpt from Bland & Altman (2009) ---

We have often come across the idea that we should not use t distribution methods for small samples but should instead use rank based methods. The statement is sometimes that we should not use t methods at all for samples of fewer than six observations.[4] But, as we noted, rank based methods cannot produce anything useful for such small samples.

The aversion to parametric methods for small samples may arise from the inability to assess the distribution shape when there are so few observations. How can we tell whether data follow a normal distribution if we have only a few observations? The answer is that we have not only the data to be analysed, but usually also experience of other sets of measurements of the same thing. In addition, general experience tells us that body size measurements are usually approximately normal, as are the logarithms of many blood concentrations and the square roots of counts.

--- end excerpt from Bland & Altman (2009) ---