Re: contradictory results

[Lemmens, Meike]

Hi all,

 

We are dealing with contradictory results and want to understand how to
interpret these. Based on a pre - post assessment of teacher's content
knowledge we expected two groups to start out with the same knowledge
and the experimental group to gain significantly more knowledge over
time than the control group. Indeed a repeated measures ANOVA showed is
a significant (p=.002) interaction effect between the groups over time.
But a simple ANOVA on the post-test score show no significant difference
between the 2 groups.

GROUP

Mean pre score

Mean post score

control

18.5

24.5

experimental

17.5

27

 

 

 

We also ran a two-block regression model were we predicted the post test
score by a first set of predictors which included group membership and
pre test score. In our second set we also entered the interaction as a
third predictor. We expected this interaction still to be significant
based on the repeated measures ANOVA results. However, according to the
regression analysis the interaction was no longer significant. What does
this mean and how can we make sense of these contradictory results?  

 

Meike Lemmens


***********************************************************************
This email may contain confidential material.
If you were not an intended recipient,
please notify the sender and delete all copies.
We may monitor email to and from our network.

***********************************************************************

[Maguin, Eugene]

Mike,

So it sounds like you ran two different analyses. One was a repeated
measures ANOVA. The means for that analysis were as follows.

GROUP    pre   post  delta
Control  18.5  24.5   6.0
Tx       17.5  27     9.5

You had a significant interaction effect. If you look at the pre-post delta,
the Tx group has a bigger increment. But the control-tx difference at post
is only 2.5. However, if the control-tx difference at pretest had been 0.0
instead of 1.0, the post group difference would have been 3.5. So, I think
that one explanation is that the possibly nonsignificant difference at
pretest may be the cause of the nonsiginificant group difference at post.
Other things also matter, like within group SD at post. What I think you
should do is to use the emmeans subcommand and compare pre-post for control
and pre-post for tx. You'll have one of two outcomes: both significant or tx
significant but not control. The problem you have is that while both gained,
tx gained more.

It sounds like your second analysis was the regression equivalent of ANCOVA.
While there is no necessary reason for the two analyses to give the same
results, I think you made an interpretation error. The interaction that you
entered sounds like it must have been group by pretest. If so, then you are
testing whether the within group regression slopes are the same between the
two groups, which you should fervently hope that they are. You should be
looking at the group effect.

Gene Maguin