how to test small samples with the same treatment?

[stonecai2]

I want to see whether the 10 high-level students and 10 low-level students have achieved the same progress under the same treatment of teaching.(same sample) what test I should use? independent samples t-test?

[stonecai2]

correcion: I want to see whether the 10 high-level students and 10 low-level students have achieved the same progress under the same treatment of teaching.(small samples) what test I should use? independent samples t-test?

[Bruce Weaver]

stonecai2 wrote

correcion: I want to see whether the 10 high-level students and 10 low-level students have achieved the same progress under the same treatment of teaching.(small samples) what test I should use? independent samples t-test?

Judging by what you've said, you must have both pre- and post-intervention scores for the students.  Assuming the scores have (at least approximate) interval scale properties, I would suggest using ANCOVA, with post-intervention score as the DV, pre-intervention score as the covariate, and Group as the "fixed factor".  The F-test for Group tests the null hypothesis that the groups are random samples from populations with the same mean change.  

Sometimes, people find it hard to believe that the F-test for Group can be testing a null about change, when the DV is the post-intervention score.  For fun, re-run that model, but with the post minus pre change score as the DV.  You will find that the F-test for Group is identical in the two models.

Finally, the ANCOVA model assumes that the slope for the relationship between pre and post scores is the same in the two groups.  You should create a scatter-plot with separate fit lines for the two groups to visually assess whether this assumption is tenable.  If it appears that the slopes are quite different for the two groups, then you should probably include the Group x Pre interaction.  If you do, the model is no longer an ANCOVA model, strictly  speaking.  And if you include the interaction, you can no longer make a blanket statement about the difference between groups, because the difference will depend on what value of the pre-intervention score you select to look at.

HTH.

[Art Kendall]

How did you select the high level and low level students?

Art Kendall
Social Research Consultants

On 8/24/2010 9:16 PM, stonecai2 wrote:

I want to see whether the 10 high-level students and 10 low-level students
have achieved the same progress under the same treatment of teaching.(same
sample) what test I should use? independent samples t-test?
--
View this message in context: http://spssx-discussion.1045642.n5.nabble.com/how-to-test-small-samples-with-the-same-treatment-tp2652064p2652064.html
Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD

[Ryan]

In addition to Art's question, I would want to know how the treatment was provided? In sub-groups, groups, individually?

On Wed, Aug 25, 2010 at 9:42 AM, Art Kendall <[hidden email]> wrote:

How did you select the high level and low level students?

Art Kendall
Social Research Consultants

On 8/24/2010 9:16 PM, stonecai2 wrote:

I want to see whether the 10 high-level students and 10 low-level students
have achieved the same progress under the same treatment of teaching.(same
sample) what test I should use? independent samples t-test?
--
View this message in context: http://spssx-discussion.1045642.n5.nabble.com/how-to-test-small-samples-with-the-same-treatment-tp2652064p2652064.html
Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD

[stonecai2]

thanks for your reply. i have pre-test and post-test to determine their levels and the change after the treatment.
they are two sub-groups in the same group influenced by the same treatment

[Art Kendall]

Was the treatment applied separately to each individual or was the group treated in something like a class room?

How did you select the high vs low groups? Coarsened the pre-test score or some other score?  Different tracks in a school?

Do you have a continuous measure of "level" available?

Art Kendall
Social Research Consultants

On 8/25/2010 11:52 AM, stonecai2 wrote:

thanks for your reply. i have pre-test and post-test to determine their
levels and the change after the treatment.
they are two sub-groups in the same group influenced by the same treatment
--
View this message in context: http://spssx-discussion.1045642.n5.nabble.com/how-to-test-small-samples-with-the-same-treatment-tp2652064p2652994.html
Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD

[Ryan]

Bruce,

 

I would agree that the F-test associated with the "Group" main effect in a one-way ANOVA on change scores is mathematically equivalent to the "F-test" associated with the "Group*Time" interaction effect in a 2X2 mixed ANOVA. At least this is what I've observed empirically whenever I've tested this assumption.

 

However, I do not see how the F-test associated with the "Group" main effect in an ANCOVA is necessarily mathematically equivalent to either of the F-tests I just mentioned. Perhaps I've misunderstood what you stated or perhaps I've misunderstood the design in question. If you have the time, would you mind elaborating.

 

I provide a simple example below demonstrating that the F-test associated with the "Group" main effect in the ANCOVA model is different.

 

Ryan

 

--

 

data list list / group t1 t2.
begin data
1 23 18
1 17 15
1 12 10
2 14 11
2 33 32
2 43 17
end data.

 

COMPUTE change_score=t2-t1.
EXECUTE.

 

*Run One-Way ANOVA on Change Scores.
UNIANOVA change_score BY group
  /METHOD=SSTYPE(3)
  /INTERCEPT=INCLUDE
  /CRITERIA=ALPHA(.05)
  /DESIGN=group.

 

*Run 2X2 mixed ANOVA.
GLM t1 t2 BY group
  /WSFACTOR=time 2 Polynomial
  /METHOD=SSTYPE(3)
  /CRITERIA=ALPHA(.05)
  /WSDESIGN=time
  /DESIGN=group.

 

*Run ANCOVA with t1 as Covariate.
UNIANOVA t2 BY group WITH t1
  /METHOD=SSTYPE(3)
  /INTERCEPT=INCLUDE
  /CRITERIA=ALPHA(0.05)
  /DESIGN=t1 group.

On Wed, Aug 25, 2010 at 7:30 AM, Bruce Weaver <[hidden email]> wrote:

stonecai2 wrote:
>
> correcion: I want to see whether the 10 high-level students and 10
> low-level students have achieved the same progress under the same
> treatment of teaching.(small samples) what test I should use? independent
> samples t-test?
>

Judging by what you've said, you must have both pre- and post-intervention
scores for the students.  Assuming the scores have (at least approximate)
interval scale properties, I would suggest using ANCOVA, with
post-intervention score as the DV, pre-intervention score as the covariate,
and Group as the "fixed factor".  The F-test for Group tests the null
hypothesis that the groups are random samples from populations with the same
mean change.

Sometimes, people find it hard to believe that the F-test for Group can be
testing a null about change, when the DV is the post-intervention score.
For fun, re-run that model, but with the post minus pre change score as the
DV.  You will find that the F-test for Group is identical in the two models.

Finally, the ANCOVA model assumes that the slope for the relationship
between pre and post scores is the same in the two groups.  You should
create a scatter-plot with separate fit lines for the two groups to visually
assess whether this assumption is tenable.  If it appears that the slopes
are quite different for the two groups, then you should probably include the
Group x Pre interaction.  If you do, the model is no longer an ANCOVA model,
strictly  speaking.  And if you include the interaction, you can no longer
make a blanket statement about the difference between groups, because the
difference will depend on what value of the pre-intervention score you
select to look at.

HTH.


-----
--
Bruce Weaver
[hidden email]
http://sites.google.com/a/lakeheadu.ca/bweaver/

"When all else fails, RTFM."

NOTE: My Hotmail account is not monitored regularly.
To send me an e-mail, please use the address shown above.

--
View this message in context: http://spssx-discussion.1045642.n5.nabble.com/how-to-test-small-samples-with-the-same-treatment-tp2652064p2652530.html

Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[stonecai2]

thanks to all of you! As a return, I would elaborate my question (this is one research question in my study): I adopted a new approach in a class. The subjects are 50 students in a class. They were tested before and after the intervention. I want to see whether there is a significant difference in the improvement between the top 10 and the lowest 10. Students were ranked according to the pre-test.

ten students receive the highest scores in pre-test and their scores in post-test

Subject S10  S26 S30  S40  S3  S15  S9  S29 S38  S1 Mean
Pre-test 76    76 73    72  69  68  66  65 64    63 　
Post-test 89   86 74    79  71  67  63  73 67    64 　
Difference13   10   1     7    2  -1  -3    8   3     1   4.1

ten students receive the lowest scores in pre-test and their scores in post-test

Subject S25  S35  S42 S33  S5  S2  S19  S14  S11  S13  Mean
Pre-test 44    44    44   43    42  41   40   39    38    28 　
Post-test 56   54  46   79    46  68   50   54    49    62 　
Difference12   10     2    36     4   27   10   15    11    34    16.1

[Bruce Weaver]

R B wrote

Bruce,

I would agree that the F-test associated with the "Group" main effect in a
one-way ANOVA on change scores is mathematically equivalent to the "F-test"
associated with the "Group*Time" interaction effect in a 2X2 mixed ANOVA. At
least this is what I've observed empirically whenever I've tested this
assumption.

However, I do not see how the F-test associated with the "Group" main
effect in an ANCOVA is necessarily mathematically equivalent to either
of the F-tests I just mentioned. Perhaps I've misunderstood what you stated
or perhaps I've misunderstood the design in question. If you have the time,
would you mind elaborating.

I provide a simple example below demonstrating that the F-test associated
with the "Group" main effect in the ANCOVA model is different.

Ryan

--

data list list / group t1 t2.
begin data
1 23 18
1 17 15
1 12 10
2 14 11
2 33 32
2 43 17
end data.

COMPUTE change_score=t2-t1.
EXECUTE.

*Run One-Way ANOVA on Change Scores.
UNIANOVA change_score BY group
  /METHOD=SSTYPE(3)
  /INTERCEPT=INCLUDE
  /CRITERIA=ALPHA(.05)
  /DESIGN=group.

*Run 2X2 mixed ANOVA.
GLM t1 t2 BY group
  /WSFACTOR=time 2 Polynomial
  /METHOD=SSTYPE(3)
  /CRITERIA=ALPHA(.05)
  /WSDESIGN=time
  /DESIGN=group.

*Run ANCOVA with t1 as Covariate.
UNIANOVA t2 BY group WITH t1
  /METHOD=SSTYPE(3)
  /INTERCEPT=INCLUDE
  /CRITERIA=ALPHA(0.05)
  /DESIGN=t1 group.

Hi Ryan.  Here is what I was talking about, using your sample data.

data list list / group t1 t2.
begin data
1 23 18
1 17 15
1 12 10
2 14 11
2 33 32
2 43 17
end data.
 
COMPUTE change_score=t2-t1.
EXECUTE.

*Run ANCOVA with t1 as Covariate.
UNIANOVA t2 BY group WITH t1
  /METHOD=SSTYPE(3)
  /INTERCEPT=INCLUDE
  /CRITERIA=ALPHA(0.05)
  /DESIGN=t1 group.

*Run ANCOVA with t1 as Covariate and Change as the DV.
UNIANOVA change_score BY group WITH t1
  /METHOD=SSTYPE(3)
  /INTERCEPT=INCLUDE
  /CRITERIA=ALPHA(0.05)
  /DESIGN=t1 group.


The F-test for Group is identical in these two models.  But the first model is the one that would normally be used.

Cheers,
Bruce

[Art Kendall]

It is difficult to eyeball the data when it is not aligned.� But is there a possible ceiling effect? What is the maximum possible score on the test? What is the lowest possible score?

Did you try a scatter plot of all 50 cases? Perhaps with
Did you try a ladder graph? Do the rungs become less steep when the pre-tests are high?
With very small data sets, when you post data like this it enables others to check their recommendations if you present all of it. (be sure to test the sytax before sending it.)
data list list/subject (a3) Pre_test (f3) Post_test (f3).
begin data
S01 76 89
S02 76 86
...
S50 55 65
end data.

Art Kendall
Social Research Consultants


On 8/25/2010 9:41 PM, stonecai2 wrote:

Subject S10  S26         S30  S40  S3  S15        S9  S29        S38  S1         Mean
Pre-test        76    76         73    72         69  68          66  65         64    63       　
Post-test 89   86        74    79         71  67          63  73         67    64       　
Difference13   10          1     7          2  -1         -3    8          3     1         4.1

ten students receive the lowest scores in pre-test and their scores in
post-test

Subject S25  S35  S42 S33         S5  S2  S19  S14  S11  S13  Mean
Pre-test        44    44    44   43    42  41   40   39    38    28     　
Post-test 56   54         46   79    46  68   50   54    49    62       　
Difference12   10     2    36     4   27   10   15    11    34    16.1

===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD

[Bruce Weaver]

stonecai2 wrote

thanks to all of you! As a return, I would elaborate my question (this is one research question in my study): I adopted a new approach in a class. The subjects are 50 students in a class. They were tested before and after the intervention. I want to see whether there is a significant difference in the improvement between the top 10 and the lowest 10. Students were ranked according to the pre-test.

This is a recipe for regression to the mean, of course.  I.e., whenever you select people who have extreme scores on Y at time 1, and then measure Y again at time 2, many/most of the Y-scores will be closer to the mean the second time, even if the intervention has no effect.  You need a control group if you want to test the effectiveness of the intervention.

[Ryan]

Hi, Bruce:

 

Thanks for the clarification. Now I see where you were headed, and I think it's an excellent point!

 

Best,

 

Ryan

On Thu, Aug 26, 2010 at 8:28 AM, Bruce Weaver <[hidden email]> wrote:

R B wrote:
>
> Bruce,
>
> I would agree that the F-test associated with the "Group" main effect in a
> one-way ANOVA on change scores is mathematically equivalent to the
> "F-test"
> associated with the "Group*Time" interaction effect in a 2X2 mixed ANOVA.
> At
> least this is what I've observed empirically whenever I've tested this
> assumption.
>
> However, I do not see how the F-test associated with the "Group" main
> effect in an ANCOVA is necessarily mathematically equivalent to either
> of the F-tests I just mentioned. Perhaps I've misunderstood what you
> stated
> or perhaps I've misunderstood the design in question. If you have the
> time,
> would you mind elaborating.
>
> I provide a simple example below demonstrating that the F-test associated
> with the "Group" main effect in the ANCOVA model is different.

>
> Ryan
>
> --
>
> data list list / group t1 t2.
> begin data
> 1 23 18
> 1 17 15
> 1 12 10

> 2 14 11

> 2 33 32
> 2 43 17
> end data.
>
> COMPUTE change_score=t2-t1.
> EXECUTE.
>
> *Run One-Way ANOVA on Change Scores.
> UNIANOVA change_score BY group
>   /METHOD=SSTYPE(3)

>   /INTERCEPT=INCLUDE
>   /CRITERIA=ALPHA(.05)
>   /DESIGN=group.
>
> *Run 2X2 mixed ANOVA.
> GLM t1 t2 BY group
>   /WSFACTOR=time 2 Polynomial
>   /METHOD=SSTYPE(3)
>   /CRITERIA=ALPHA(.05)
>   /WSDESIGN=time
>   /DESIGN=group.
>
> *Run ANCOVA with t1 as Covariate.
> UNIANOVA t2 BY group WITH t1
>   /METHOD=SSTYPE(3)
>   /INTERCEPT=INCLUDE
>   /CRITERIA=ALPHA(0.05)
>   /DESIGN=t1 group.
>
>

Hi Ryan.  Here is what I was talking about, using your sample data.

data list list / group t1 t2.
begin data
1 23 18
1 17 15
1 12 10
2 14 11
2 33 32
2 43 17
end data.

COMPUTE change_score=t2-t1.
EXECUTE.

*Run ANCOVA with t1 as Covariate.
UNIANOVA t2 BY group WITH t1
 /METHOD=SSTYPE(3)
 /INTERCEPT=INCLUDE
 /CRITERIA=ALPHA(0.05)
 /DESIGN=t1 group.

*Run ANCOVA with t1 as Covariate and Change as the DV.
UNIANOVA change_score BY group WITH t1

 /METHOD=SSTYPE(3)
 /INTERCEPT=INCLUDE
 /CRITERIA=ALPHA(0.05)
 /DESIGN=t1 group.

The F-test for Group is identical in these two models.  But the first model
is the one that would normally be used.

Cheers,
Bruce

-----
--
Bruce Weaver
[hidden email]
http://sites.google.com/a/lakeheadu.ca/bweaver/

"When all else fails, RTFM."

NOTE: My Hotmail account is not monitored regularly.
To send me an e-mail, please use the address shown above.

--

View this message in context: http://spssx-discussion.1045642.n5.nabble.com/how-to-test-small-samples-with-the-same-treatment-tp2652064p2706087.html

Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[stonecai2]

I understand. It's very reasonable to see high-score ss achieve less than the low-score ss, because they are close to the limit. I drop this question. I have control group and experimental group, but I just want to see I can find something in this comparison.

I'm kind of a newbie in SPSS.