Stratified anova vs interaction effect

[parisec]

Hi all,

Study design: 2 groups (T1, T2), fixed factor: sex (M, F)

We hypothesized that T1 would have a stronger effect in Men versus women. I plotted means and they  clearly indicated an interaction. i stratified the analysis by sex and ran separate ANOVAs, it was clear that there was an effect in Men (p <0.02) but not in Women. I should have quit there!

But i didn't,  i ran a factorial anova that included group, sex, and the group*sex interaction term. I added the simple effects of group within sex.

  /EMMEANS=TABLES(sex*group) COMPARE (group) ADJ(LSD)

The main effect was just above significance (0.056) and interaction term was not significant, yet the simple effects term showed strong statistical significance, which i know should not be interpreted without a significant interaction.

Now i'm at a loss. The stratified analysis clearly indicates an interaction yet the formal test for interaction didn't pan out.

Is it wrong to just report the results of a stratified analysis?

Thanks for your thoughts.
Carol

[Bruce Weaver]

You said, "We hypothesized that T1 would have a stronger effect in Men versus women."  I'm not sure what you mean.  Given the description of the design, I would have expected you to say that the difference between T1 and T2 is expected to be larger in men than in women.  

If you really are talking about the T2-T1 difference, then your stratified analysis shows that the T2-T1 difference is significantly different from 0 for men, but not for women.  That is not the same thing as showing that the two T2-T1 differences (for M and F) are not the same.  If your hypothesis is about the interaction, you must use the F-test for the interaction, not the results of the stratified analysis.

HTH.

parisec wrote

Hi all,

Study design: 2 groups (T1, T2), fixed factor: sex (M, F)

We hypothesized that T1 would have a stronger effect in Men versus women. I plotted means and they  clearly indicated an interaction. i stratified the analysis by sex and ran separate ANOVAs, it was clear that there was an effect in Men (p <0.02) but not in Women. I should have quit there!

But i didn't,  i ran a factorial anova that included group, sex, and the group*sex interaction term. I added the simple effects of group within sex.

  /EMMEANS=TABLES(sex*group) COMPARE (group) ADJ(LSD)

The main effect was just above significance (0.056) and interaction term was not significant, yet the simple effects term showed strong statistical significance, which i know should not be interpreted without a significant interaction.

Now i'm at a loss. The stratified analysis clearly indicates an interaction yet the formal test for interaction didn't pan out.

Is it wrong to just report the results of a stratified analysis?

Thanks for your thoughts.
Carol

[Rich Ulrich]

I will add to what Bruce has said - saying it stronger, and with
one illustration of what is happening.

This is an error that fools some experienced, non-statistician
investigators; and it reveals their lack of good statistical background.
It is one of the things that might get your statistical co-author (if
there is one) to threaten to have his name removed from the piece,
because there is no doubt about the nature of the error. 

If you want to show an interaction "statistically", you have to test
the interaction.  Just as Bruce says.  What you have before that test
is "the suggestion of an interaction" a hint that a larger sample may
show the difference *if*  it is robust.  Here is what happens, by the
numbers.

The "significant test" you see for males is the test that T1-T2 "is different
from zero" -- where that is a *fixed* zero.  Even if the difference for females
were to be exactly zero (say), then the test of the interaction might
*not*  be significant.  It would definitely have a p-value that is not as
significant, because the zero-for-females has an error attached; it is not
a fixed value of zero.  To repeat:  Male(T1-T2)  is significant when tested
against its own variance, and that test shows (T1-T2) is different from zero. 

Interaction(T1-T2)  has to test Male(T1-T2) against Female(T1-T2), and that
test uses the sum of the two variances -- so it has to be a less powerful
test, in general.  In particular, it will never show as small a p-value when
comparing Males to Females as it would when comparing Males to a fixed
value that represents the means for Females.

--
Rich Ulrich

> Date: Fri, 5 Dec 2014 16:19:56 -0700

> From: [hidden email]
> Subject: Re: Stratified anova vs interaction effect
> To: [hidden email]
>
> You said, "We hypothesized that T1 would have a stronger effect in Men versus
> women." I'm not sure what you mean. Given the description of the design, I
> would have expected you to say that the difference between T1 and T2 is
> expected to be larger in men than in women.
>
> If you really are talking about the T2-T1 difference, then your stratified
> analysis shows that the T2-T1 difference is significantly different from 0
> for men, but not for women. That is not the same thing as showing that the
> two T2-T1 differences (for M and F) are not the same. If your hypothesis is
> about the interaction, you must use the F-test for the interaction, not the
> results of the stratified analysis.
>
> HTH.
>
>
>
> parisec wrote
> > Hi all,
> >
> > Study design: 2 groups (T1, T2), fixed factor: sex (M, F)
> >
> > We hypothesized that T1 would have a stronger effect in Men versus women.
> > I plotted means and they clearly indicated an interaction. i stratified
> > the analysis by sex and ran separate ANOVAs, it was clear that there was
> > an effect in Men (p <0.02) but not in Women. I should have quit there!
> >
> > But i didn't, i ran a factorial anova that included group, sex, and the
> > group*sex interaction term. I added the simple effects of group within
> > sex.
> >
> > /EMMEANS=TABLES(sex*group) COMPARE (group) ADJ(LSD)
> >
> > The main effect was just above significance (0.056) and interaction term
> > was not significant, yet the simple effects term showed strong statistical
> > significance, which i know should not be interpreted without a significant
> > interaction.
> >
> > Now i'm at a loss. The stratified analysis clearly indicates an
> > interaction yet the formal test for interaction didn't pan out.
> >
> > Is it wrong to just report the results of a stratified analysis?
> >
> > Thanks for your thoughts.
> > Carol
>
>
>
>
>
> -----
> --
> Bruce Weaver
> [hidden email]
> http://sites.google.com/a/lakeheadu.ca/bweaver/
>
> "When all else fails, RTFM."
>
> NOTE: My Hotmail account is not monitored regularly.
> To send me an e-mail, please use the address shown above.
>
> --
> View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Stratified-anova-vs-interaction-effect-tp5728125p5728126.html
> Sent from the SPSSX Discussion mailing list archive at Nabble.com.
>
> =====================
> To manage your subscription to SPSSX-L, send a message to
> [hidden email] (not to SPSSX-L), with no body text except the
> command. To leave the list, send the command
> SIGNOFF SPSSX-L
> For a list of commands to manage subscriptions, send the command
> INFO REFCARD

===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD

[parisec]

Bruce and Rich,

Thank you for your great insights and information. You explained just why the results are so different and clearly shows that I cannot use the results of the stratified analysis.

Carol