Inconsistent post-hoc test results

[Tanja Gabriele Baudson]

Hi,

I am currently running an analysis of a questionnaire which was administered
to teachers at different schools. I performed an ANOVA across all items with
school as the between-subjects factor. For three items, I obtained
significant between-group differences (p=.029, .048 and .038, respectively);
however, when checking the post-hoc comparisons (I tried Bonferroni, Tukey,
and Games-Howell), there are hardly any significant differences between
individual schools (when using Bonferroni or Tukey, for instance, a
significantly different pair can be found for only for one of the three
items. Considering the type I error correction (the correction that is done
when using multiple t-tests; I'm not sure if the term is correct), this
cannot be the case, actually. Do you have any ideas?

Thanks in advance
Tanya

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Granaas, Michael]

Hello Tanya,

Based on your message it appears that you ran one ANOVA for each of several questions--is that correct?

If so you need a great deal more error correction than is being provided by post-hoc testing.  Any one, or all,  of your three significant findings could be simple false positives.  Given the pattern of results you are reporting I think this is the most likely explanation.

Michael

****************************************************
Michael Granaas             [hidden email]
Assoc. Prof.                Phone: [phone]
Dept. of Psychology         FAX:  [phone]
University of South Dakota
414 E. Clark St.
Vermillion, SD 57069
*****************************************************
________________________________________
From: SPSSX(r) Discussion [[hidden email]] On Behalf Of Tanya [[hidden email]]
Sent: Friday, August 20, 2010 10:54 AM
To: [hidden email]
Subject: Inconsistent post-hoc test results

Hi,

I am currently running an analysis of a questionnaire which was administered
to teachers at different schools. I performed an ANOVA across all items with
school as the between-subjects factor. For three items, I obtained
significant between-group differences (p=.029, .048 and .038, respectively);
however, when checking the post-hoc comparisons (I tried Bonferroni, Tukey,
and Games-Howell), there are hardly any significant differences between
individual schools (when using Bonferroni or Tukey, for instance, a
significantly different pair can be found for only for one of the three
items. Considering the type I error correction (the correction that is done
when using multiple t-tests; I'm not sure if the term is correct), this
cannot be the case, actually. Do you have any ideas?

Thanks in advance
Tanya

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD