One-Way Anova Repeated Measures: Main effect is significant, but post-hoc tests are not significant

[JohnDoe]

Hi,

I ran a one-way within-subjects ANOVA, with equal group sizes, and my data met the assumption of sphericity. I found a significant Main effect, but Bonferroni's tests did not reveal any significant post-hoc effects.

Does anyone know why this might have happened? And how would I report this result?

Thanks very much

[Mike]

You may have gotten nonsignificant Bonferroni results because:

(1) you lack statistical power to detect differences.  Consider
specifying LSD post hoc comparisons to see if any of the
differences are significant. The problem with this approach is
that you no longer have overall alpha = .05.  But this could
indicate where the differences are if you increase the number
of subjects.

(2) the post hoc tests compare the means on a pairwise basis
(i.e., one mean vs another mean) but the significant ANOVA
only indicates that *some* combination of means are different.
You don't specify how many levels you have but if you have
four means, it is possible that [(mean 1 + mean2)/2] - [(mean3 -
mean4)/2] is the combination that is statistically significant.

Kirk's textbook of Experimental Design has a good chapter
on planned comparisons/post hoc tests that you might consider
looking at.

-Mike Palij
New York University
[hidden email]

P.S.  The Dunn-Sidak version of the Bonferroni test is supposed
to be somewhat more powerful than ordinary Bonferroni test, so
you might also consider this if you focus is only on pairwise
comparisons.

----- Original Message -----
From: "JohnDoe" <[hidden email]>
To: <[hidden email]>
Sent: Thursday, January 02, 2014 11:35 AM
Subject: One-Way Anova Repeated Measures: Main effect is significant,
but post-hoc tests are not significant

> Hi,
>
> I ran a one-way within-subjects ANOVA, with equal group sizes, and my
> data
> met the assumption of sphericity. I found a significant Main effect,
> but
> Bonferroni's tests did not reveal any significant post-hoc effects.
>
> Does anyone know why this might have happened? And how would I report
> this
> result?
>
> Thanks very much
>
>
>
> --
> View this message in context:
> http://spssx-discussion.1045642.n5.nabble.com/One-Way-Anova-Repeated-Measures-Main-effect-is-significant-but-post-hoc-tests-are-not-significant-tp5723746.html
> Sent from the SPSSX Discussion mailing list archive at Nabble.com.
>
> =====================
> To manage your subscription to SPSSX-L, send a message to
> [hidden email] (not to SPSSX-L), with no body text except
> the
> command. To leave the list, send the command
> SIGNOFF SPSSX-L
> For a list of commands to manage subscriptions, send the command
> INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[JohnDoe]

Thanks,

As it was a repeated measures test, I was unable to select any post hocs except for LSD, Bonferroni, and Sidak. I have four levels in my design

[Bruce Weaver]

-----
--
Bruce Weaver
[hidden email]
http://sites.google.com/a/lakeheadu.ca/bweaver/

"When all else fails, RTFM."

NOTE: My Hotmail account is not monitored regularly.
To send me an e-mail, please use the address shown above.

--
View this message in context: http://spssx-discussion.1045642.n5.nabble.com/One-Way-Anova-Repeated-Measures-Main-effect-is-significant-but-post-hoc-tests-are-not-significant-tp5723746p5723749.html
Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Bruce Weaver]

What is the nature of the repeated measures factor?  If there is an underlying quantitative variable, polynomial trend analysis may be useful.  

JohnDoe wrote

Thanks,

As it was a repeated measures test, I was unable to select any post hocs except for LSD, Bonferroni, and Sidak. I have four levels in my design

[Rich Ulrich]

Bonferroni is an *alternate* way to test the whole
set of effects, so it is not unreasonable that it
might give different results. "There is an overall
difference; but none of the separate comparisons
meet a Bonferroni criterion, which is stiffer in
certain respects, though it might be more likely to
detect a single, extreme mean."

By the way, the "assumption of sphericity" is *not*
demonstrated by failing to reject at a 5% level, since
it happens to be a remarkably weak test in small samples.
If the p-value for that test is larger than 0.50  (not
0.05), then the sphericity is probably not a problem.
(Also, the Wikip article on Mauchly's suggests that the
test over-detects when Ns are large.)

- Yes, non-sphericity could contribute to non-significant
pair-wise testing, if the extreme groups have the larger
variances. How similar are the variances?

--
Rich Ulrich

----------------------------------------

> Date: Thu, 2 Jan 2014 08:35:45 -0800
> From: [hidden email]
> Subject: One-Way Anova Repeated Measures: Main effect is significant, but post-hoc tests are not significant
> To: [hidden email]
>
> Hi,
>
> I ran a one-way within-subjects ANOVA, with equal group sizes, and my data
> met the assumption of sphericity. I found a significant Main effect, but
> Bonferroni's tests did not reveal any significant post-hoc effects.
>
> Does anyone know why this might have happened? And how would I report this
> result?
>
> Thanks very much
>
>

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[David Marso]

Then there is always the whole going back to WHY did you do the experiment?  
Wonderful thing PLANNED COMPARISONS!

Bruce Weaver wrote

What is the nature of the repeated measures factor?  If there is an underlying quantitative variable, polynomial trend analysis may be useful.  

JohnDoe wrote

Thanks,

As it was a repeated measures test, I was unable to select any post hocs except for LSD, Bonferroni, and Sidak. I have four levels in my design

[Ryan]

JohnDoe,

Under certain conditions, it's perfectly acceptable to employ the Tukey HSD procedure as a follow-up procedure in testing all-possible pairwise comparisons in a repeated measures ANOVA. While it might take a little bit of effort, this procedure can be employed in SPSS for your design.

But let's step back for a moment. What is your understanding of the assumption of sphericity? How could a departure in sphericity affect the the sampling distribution of the ratio of mean squares? Are you familiar with the index, espilon? What is your estimate of epsilon?  What is your understanding of the rule(s) of thumb regarding epsilon? What are the corrections offered by SPSS? 

I really appreciated Bruce's question/comment about the "...nature of the repeated measures factor." It could have an impact not only on the post-hoc tests performed but the model as well (e.g., general linear model versus linear mixed model). For example, if your repeated measures variable is a measure of time, it's possible to observe a decay in correlations between residuals from observations more distant in time. This type of residual correlation structure could be modeled in the MIXED procedure. Sorry, but I can't resist plugging the MIXED procedure. :-)

Ryan

On Thu, Jan 2, 2014 at 12:13 PM, JohnDoe <[hidden email]> wrote:

Thanks,

As it was a repeated measures test, I was unable to select any post hocs
except for LSD, Bonferroni, and Sidak. I have four levels in my design



--
View this message in context: http://spssx-discussion.1045642.n5.nabble.com/One-Way-Anova-Repeated-Measures-Main-effect-is-significant-but-post-hoc-tests-are-not-significant-tp5723746p5723748.html

Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Kornbrot, Diana]

It is possible for a factor to be significant overalland no pair wise comparisons to be significant even with perfect sphericity.
Contrasts do not have to be pairwise. E.g. It amy be that mean of groups 1 and 2 combines is greater than mean groups 3,3,45 combined – but neither 1 or 2 is signficantly different form any of 3,4,5. There are many more possibilities than df.

If the study has a priori hypotheses [no mre than k-1] then these can be tested without corrections.
If the study is exploratory, aka fishing expedition, then corrections for multiples should always be made.

It is very important to look at effect sizes. If a post hoc omparison has large but ns effect then more power is needed. No conclusions can be drawn, but it is worth replicating with more observations.

In any event a priori power should ALWAYS be reported for ns tests, low power with ns means that study s not adding much to knowledge.

Note that current guidelines,e.g. Equator, and ‘respectable’ journals recommend that the data analysis plan is produced BEFORE the data are collected.
Conclusions based on post hoc unplanned tests, are dubious unless effect sizes are very large, and even then they should be the basis of further studies with panned tests of replication

Best

Diana


On 03/01/2014 05:43, "Ryan Black" <ryan.andrew.black@...> wrote:

JohnDoe,

Under certain conditions, it's perfectly acceptable to employ the Tukey HSD procedure as a follow-up procedure in testing all-possible pairwise comparisons in a repeated measures ANOVA. While it might take a little bit of effort, this procedure can be employed in SPSS for your design.

But let's step back for a moment. What is your understanding of the assumption of sphericity? How could a departure in sphericity affect the the sampling distribution of the ratio of mean squares? Are you familiar with the index, espilon? What is your estimate of epsilon?  What is your understanding of the rule(s) of thumb regarding epsilon? What are the corrections offered by SPSS? 

I really appreciated Bruce's question/comment about the "...nature of the repeated measures factor." It could have an impact not only on the post-hoc tests performed but the model as well (e.g., general linear model versus linear mixed model). For example, if your repeated measures variable is a measure of time, it's possible to observe a decay in correlations between residuals from observations more distant in time. This type of residual correlation structure could be modeled in the MIXED procedure. Sorry, but I can't resist plugging the MIXED procedure. :-)

Ryan


On Thu, Jan 2, 2014 at 12:13 PM, JohnDoe <roryboyle18@...> wrote:

Thanks,

As it was a repeated measures test, I was unable to select any post hocs
except for LSD, Bonferroni, and Sidak. I have four levels in my design



--
View this message in context: http://spssx-discussion.1045642.n5.nabble.com/One-Way-Anova-Repeated-Measures-Main-effect-is-significant-but-post-hoc-tests-are-not-significant-tp5723746p5723748.html
Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to
LISTSERV@... (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

---

Professor Diana Kornbrot
email: : d.e.kornbrot@...
web:    http://dianakornbrot.wordpress.com/
            http://go.herts.ac.uk/diana_kornbrot
Work
Department of Psychology
School of Life and Medical Sciences
University of Hertfordshire
College Lane, Hatfield, Hertfordshire AL10 9AB, UK
voice:   [phone]
Home
19 Elmhurst Avenue
London N2 0LT, UK
voice:   [phone]
mobile: [phone]

[Bruce Weaver]

Ryan Black wrote

Sorry, but I can't resist plugging the MIXED procedure. :-)

I'm shocked!  What next?  Jon promoting the use of Python, perhaps?  :-O

Ryan Black wrote

JohnDoe,

Under certain conditions, it's perfectly acceptable to employ the Tukey HSD
procedure as a follow-up procedure in testing all-possible pairwise
comparisons in a repeated measures ANOVA. While it might take a little bit
of effort, this procedure can be employed in SPSS for your design.

But let's step back for a moment. What is your understanding of the
assumption of sphericity? How could a departure in sphericity affect the
the sampling distribution of the ratio of mean squares? Are you familiar
with the index, espilon? What is your estimate of epsilon?  What is your
understanding of the rule(s) of thumb regarding epsilon? What are the
corrections offered by SPSS?

I really appreciated Bruce's question/comment about the "...nature of the
repeated measures factor." It could have an impact not only on the post-hoc
tests performed but the model as well (e.g., general linear model versus
linear mixed model). For example, if your repeated measures variable is a
measure of time, it's possible to observe a decay in correlations between
residuals from observations more distant in time. This type of residual
correlation structure could be modeled in the MIXED procedure. Sorry, but I
can't resist plugging the MIXED procedure. :-)

Ryan


On Thu, Jan 2, 2014 at 12:13 PM, JohnDoe <[hidden email]> wrote:

> Thanks,
>
> As it was a repeated measures test, I was unable to select any post hocs
> except for LSD, Bonferroni, and Sidak. I have four levels in my design
>
>
>
> --
> View this message in context:
> http://spssx-discussion.1045642.n5.nabble.com/One-Way-Anova-Repeated-Measures-Main-effect-is-significant-but-post-hoc-tests-are-not-significant-tp5723746p5723748.html
> Sent from the SPSSX Discussion mailing list archive at Nabble.com.
>
> =====================
> To manage your subscription to SPSSX-L, send a message to
> [hidden email] (not to SPSSX-L), with no body text except the
> command. To leave the list, send the command
> SIGNOFF SPSSX-L
> For a list of commands to manage subscriptions, send the command
> INFO REFCARD
>

[David Marso]

I have decided as a New Years resolution to swear off any further use of SPSS Macros.
In fact I am no longer going to use syntax at all.  
Henceforth, I'm going to do everything with the cute little pointy cliquey menu thingies!
I will also turn off my journaling to save disc space.
I will use OMS to save everything I could do with AGGREVATE.
My new TABLELOOK is going to be a reincarnated HotDog.

Bruce Weaver wrote

Ryan Black wrote

Sorry, but I can't resist plugging the MIXED procedure. :-)

I'm shocked!  What next?  Jon promoting the use of Python, perhaps?  :-O

Ryan Black wrote

JohnDoe,

Under certain conditions, it's perfectly acceptable to employ the Tukey HSD
procedure as a follow-up procedure in testing all-possible pairwise
comparisons in a repeated measures ANOVA. While it might take a little bit
of effort, this procedure can be employed in SPSS for your design.

But let's step back for a moment. What is your understanding of the
assumption of sphericity? How could a departure in sphericity affect the
the sampling distribution of the ratio of mean squares? Are you familiar
with the index, espilon? What is your estimate of epsilon?  What is your
understanding of the rule(s) of thumb regarding epsilon? What are the
corrections offered by SPSS?

I really appreciated Bruce's question/comment about the "...nature of the
repeated measures factor." It could have an impact not only on the post-hoc
tests performed but the model as well (e.g., general linear model versus
linear mixed model). For example, if your repeated measures variable is a
measure of time, it's possible to observe a decay in correlations between
residuals from observations more distant in time. This type of residual
correlation structure could be modeled in the MIXED procedure. Sorry, but I
can't resist plugging the MIXED procedure. :-)

Ryan


On Thu, Jan 2, 2014 at 12:13 PM, JohnDoe <[hidden email]> wrote:

> Thanks,
>
> As it was a repeated measures test, I was unable to select any post hocs
> except for LSD, Bonferroni, and Sidak. I have four levels in my design
>
>
>
> --
> View this message in context:
> http://spssx-discussion.1045642.n5.nabble.com/One-Way-Anova-Repeated-Measures-Main-effect-is-significant-but-post-hoc-tests-are-not-significant-tp5723746p5723748.html
> Sent from the SPSSX Discussion mailing list archive at Nabble.com.
>
> =====================
> To manage your subscription to SPSSX-L, send a message to
> [hidden email] (not to SPSSX-L), with no body text except the
> command. To leave the list, send the command
> SIGNOFF SPSSX-L
> For a list of commands to manage subscriptions, send the command
> INFO REFCARD
>