Matching proficiency levels shown by statistics
Locked
 
|
I have conducted a language acquisition study for an article with two groups using the same test sentences. First I made them do an independent English test to categorize the students into 3 levels (low-mid-high) of proficiency.
There were 3 types of sentences, 8 test sentences for each type. Then I conducted a factorial ANOVA for both groups, the 3 levels as between subject factor and type of sentences as within subjects factor (3) in order to see if there are significant differences among their correct production of the three types. Now the reviewer tells me that I should match the proficiency level of the two groups [b]to be shown by statistics [/b]in order to be able to talk about their production (make sure that both groups are at comparable levels of proficiency in English). My question is HOW? Could somebody help me? As you see I am a complete beginner in the field of statistics, i would need clear instructions. |
|
|
My first question to you is why didn't you include the two groups
as a second between-subject factor? You would then have a 2 (BS-groups) x 3 (BS-proficiency levels) x 3 (WS-sentence type). What you would be interested in seeing is whether there are significant interactions between groups and proficiency levels. This would reveal whether the effect of proficiency is constant or different across levels of group. This shows whether proficiency is different in the two groups as well as the nature of that effect. As for the reviewer's comment, what you could do is rank order participants on proficiency level within the two groups and then match the rank 1 (lowest proficiency score) in group 1 with the person with rank 1 (lowest proficiency score) in group 2. Continue this for all ranks. Matching on proficiency score should cause a significant positive correlation to exist between the two groups. Perform a correlated groups t-test to determine whether the difference in mean proficiency scores is statistically significant (i.e., the two groups differ on mean proficiency after taking into account systematic variance to subject differences) or not significant (i.e., the two groups are statistically equivalent). To do this you would have to restructure the data such that matched pairs are now the unit of analysis, such as the following: Pair Group1 Group2 1 Rank01 Rank01 2 Rank02 Rank02 .... N RankN RankN Where Group1 contains the ordering of the proficiency score (Rank01 = lowest proficiency score -- do not use the rank value) and Group 2 contains the matched value in Group 2 (i.e., the lowest proficiency score in Group1 is matched with the lowest proficiency score in Group2 and so on). I think that the reviewer's point is that you need to control for proficiency differences between the two groups. I think that this could be done by including group as a between-subjects factor which would make is a 3-way "mixed" design. Another way of controlling for proficiency would be to have a 2-way design consisting of Group (2-levels) x Sentence Type (3 levels) and using proficiency as a covariate (i.e., statistically equating the two groups on proficiency). There probably are other ways of controlling/taking into account proficiency effects and other people on SPSS may chime in on this. -Mike Palij New York University [hidden email] ----- Original Message ----- From: "eva9" <[hidden email]> To: <[hidden email]> Sent: Tuesday, July 19, 2011 4:12 AM Subject: Matching proficiency levels shown by statistics >I have conducted a language acquisition study for an article with two groups > using the same test sentences. First I made them do an independent English > test to categorize the students into 3 levels (low-mid-high) of proficiency. > There were 3 types of sentences, 8 test sentences for each type. Then I > conducted a factorial ANOVA for both groups, the 3 levels as between subject > factor and type of sentences as within subjects factor (3) in order to see > if there are significant differences among their correct production of the > three types. > Now the reviewer tells me that I should match the proficiency level of the > two groups [b]to be shown by statistics [/b]in order to be able to talk > about their production (make sure that both groups are at comparable levels > of proficiency in English). My question is HOW? > Could somebody help me? As you see I am a complete beginner in the field of > statistics, i would need clear instructions. ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD |
|
|
|
|
|
----- Original Message -----
From: eva9 To: [hidden email] Sent: Tuesday, July 19, 2011 10:26 AM Subject: Re: Matching proficiency levels shown by statistics >Thank you Mike! >This was a great help! I did the first option and I even tried the >same adding proficiency as covariate. The groups*levels interaction >appears in the BS table and shows F(2,72)=1.058 p=.352. >(in the second case it is even less significant). You mean proficiency is not a significant covariate? Also, what about the 3-way interaction (groups*levels*sentence type)? >So can I say that >"the analysis revealed that the effect of proficiency is constant >across levels of the two studies". If the 3-way interaction is not significant than I would suggest that you say "the effect of proficiency is not signficiantly different across levels of the two studies and sentence type". >And there is one more small thing regarding this. Yesterday, before I >got desperate trying to overcome my lack of knowledge in statistics, >I ended up exploring with SPSS whether the different levels of each >study (i.e. comparing low levels, mid and high levels) were normally >and homogeneously distributed. Were no significant results and then >I checked variability with the Levene's statistics, homogeneous. Could >you tell me if this procedure was of any use? If all of these measures are nonsignificant, then there are two possible interpretations: (a) the properties of your dependent variable match the assumptions for the test or (b) you lack statistical power to detect differences. Which one of these you should focus on will depend on which statistical church you belong to. ;-) By the way, since you have a mixed design with a one-way 3 level within-subject factor, you should have gotten Mauchley's test if you were analyzing a 3-way mixed repeated measures design. >By the way, thank you for the other suggestions as well, but -if I >understand it right- I have different number of subjects in each study >and if you meant Group (2) as WS, I would have to redo the whole database. In your original design, Group is a 2 level between-subject factor which would ordinarily be tested with some sort of independent groups test (i.e., t-test or independent group ANOVA). However, if you match subjects in one group with subjects in the other group on the basis of ranked proficiency score, your two groups are no longer independent -- matching creates a correlation between the proficiency scores in Group 1 with those in Group 2. And independent groups t-test would be invalid/misleading because the groups are no longer independent. So, you can create a new file where you have: Pair#, Group 1 proficiency value, and matching Group 2 proficiency value. This would be a multi-step process but a straightforward one. >Well, thanks again for having taken the time to answer. >My best, Eva You're very welcome. -Mike Palij New York University [hidden email] --- El mar, 19/7/11, Mike Palij [via SPSSX Discussion] <[hidden email]> escribió: De: Mike Palij [via SPSSX Discussion] <[hidden email]> Asunto: Re: Matching proficiency levels shown by statistics Para: "eva9" <[hidden email]> Fecha: martes, 19 de julio, 2011 15:10 My first question to you is why didn't you include the two groups as a second between-subject factor? You would then have a 2 (BS-groups) x 3 (BS-proficiency levels) x 3 (WS-sentence type). What you would be interested in seeing is whether there are significant interactions between groups and proficiency levels. This would reveal whether the effect of proficiency is constant or different across levels of group. This shows whether proficiency is different in the two groups as well as the nature of that effect. As for the reviewer's comment, what you could do is rank order participants on proficiency level within the two groups and then match the rank 1 (lowest proficiency score) in group 1 with the person with rank 1 (lowest proficiency score) in group 2. Continue this for all ranks. Matching on proficiency score should cause a significant positive correlation to exist between the two groups. Perform a correlated groups t-test to determine whether the difference in mean proficiency scores is statistically significant (i.e., the two groups differ on mean proficiency after taking into account systematic variance to subject differences) or not significant (i.e., the two groups are statistically equivalent). To do this you would have to restructure the data such that matched pairs are now the unit of analysis, such as the following: Pair Group1 Group2 1 Rank01 Rank01 2 Rank02 Rank02 .... N RankN RankN Where Group1 contains the ordering of the proficiency score (Rank01 = lowest proficiency score -- do not use the rank value) and Group 2 contains the matched value in Group 2 (i.e., the lowest proficiency score in Group1 is matched with the lowest proficiency score in Group2 and so on). I think that the reviewer's point is that you need to control for proficiency differences between the two groups. I think that this could be done by including group as a between-subjects factor which would make is a 3-way "mixed" design. Another way of controlling for proficiency would be to have a 2-way design consisting of Group (2-levels) x Sentence Type (3 levels) and using proficiency as a covariate (i.e., statistically equating the two groups on proficiency). There probably are other ways of controlling/taking into account proficiency effects and other people on SPSS may chime in on this. -Mike Palij New York University [hidden email] ----- Original Message ----- From: "eva9" <[hidden email]> To: <[hidden email]> Sent: Tuesday, July 19, 2011 4:12 AM Subject: Matching proficiency levels shown by statistics >I have conducted a language acquisition study for an article with two groups > using the same test sentences. First I made them do an independent English > test to categorize the students into 3 levels (low-mid-high) of proficiency. > There were 3 types of sentences, 8 test sentences for each type. Then I > conducted a factorial ANOVA for both groups, the 3 levels as between subject > factor and type of sentences as within subjects factor (3) in order to see > if there are significant differences among their correct production of the > three types. > Now the reviewer tells me that I should match the proficiency level of the > two groups [b]to be shown by statistics [/b]in order to be able to talk > about their production (make sure that both groups are at comparable levels > of proficiency in English). My question is HOW? > Could somebody help me? As you see I am a complete beginner in the field of > statistics, i would need clear instructions. ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD |
«
Return to SPSSX Discussion
|
1 view|%1 views
| Free forum by Nabble | Edit this page |