Selecting a Statistical Test for unmatched pre post survey
Locked
 
|
I am not sure which statistical test is the most appropriate for my sample.
We implemented a nurtition education program to local elementary schools and we administered surveys (Pretest and Posttest). The problem is the sample do not match(from the same population) between pre and post. Survey score is in interval. |
Re: Selecting a Statistical Test for unmatched pre post survey
Administrator
|
What test would you use if you did have matched pairs of scores? Is there a version of that test for independent groups?
--
Bruce Weaver bweaver@lakeheadu.ca http://sites.google.com/a/lakeheadu.ca/bweaver/ "When all else fails, RTFM." PLEASE NOTE THE FOLLOWING: 1. My Hotmail account is not monitored regularly. To send me an e-mail, please use the address shown above. 2. The SPSSX Discussion forum on Nabble is no longer linked to the SPSSX-L listserv administered by UGA (https://listserv.uga.edu/). |
Re: Selecting a Statistical Test for unmatched pre post survey
|
|
What do you mean by "the samples do
not match"? If they are totally different samples, then it's not really
a pretest-posttest comparison, is it?
From: Bruce Weaver <[hidden email]> To: [hidden email] Date: 09/22/2011 09:22 AM Subject: Re: Selecting a Statistical Test for unmatched pre post survey Sent by: "SPSSX(r) Discussion" <[hidden email]> What test would you use if you /did/ have matched pairs of scores? Is there a version of that test for independent groups? seguchi wrote: > > I am not sure which statistical test is the most appropriate for my > sample. > We implemented a nurtition education program to local elementary schools > and we administered surveys (Pretest and Posttest). The problem is the > sample do not match(from the same population) between pre and post. Survey > score is in interval. > ----- -- Bruce Weaver [hidden email] http://sites.google.com/a/lakeheadu.ca/bweaver/ "When all else fails, RTFM." NOTE: My Hotmail account is not monitored regularly. To send me an e-mail, please use the address shown above. -- View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Selecting-a-Statistical-Test-for-unmatched-pre-post-survey-tp4830008p4830321.html Sent from the SPSSX Discussion mailing list archive at Nabble.com. ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD |
|
|
They are not paired samples. Students (N=366) took the pretest immediately before the 6 week nutrition program and they (N=371) have taken the same test as posttest (repeated measure) upon the completion of the program. SO there is a difference in subject number(students) and pretest and posttest cannot be paired for each subject. I just have a whole bunch of pretest, and posttest from the same population.
|
Re: Selecting a Statistical Test for unmatched pre post survey
|
|
In reply to this post by seguchi
Try to be more specific. Is far as I understand, you have a population, and you took two samples from the same population at diferent times. You use one as pre-test and the other as post-test. Are some of the pretest cases the same as the post test cases or are they completly diferent cases? Andrés Mg. Andrés Burga León Coordinador de Análisis e Informática Unidad de Medición de la Calidad Educativa Ministerio de Educación del Perú Calle El Comercio s/n (espalda del Museo de la Nación) Lima 41 Perú Teléfono 615-5840
I am not sure which statistical test is the most appropriate for my sample. We implemented a nurtition education program to local elementary schools and we administered surveys (Pretest and Posttest). The problem is the sample do not match(from the same population) between pre and post. Survey score is in interval. -- View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Selecting-a-Statistical-Test-for-unmatched-pre-post-survey-tp4830008p4830008.html Sent from the SPSSX Discussion mailing list archive at Nabble.com. ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD |
Re: Selecting a Statistical Test for unmatched pre post survey
|
|
In reply to this post by seguchi
So, for those who are in both the pre-
and post-test groups, you have no way of matching them? If you can match
them, then I would think a paired samples t-test would be appropriate.
If you can't match them, I don't know how much of a violation of assumptions
it would be to use an independent samples t-test.
From: seguchi <[hidden email]> To: [hidden email] Date: 09/22/2011 11:00 AM Subject: Re: Selecting a Statistical Test for unmatched pre post survey Sent by: "SPSSX(r) Discussion" <[hidden email]> They are not paired samples. Students (N=366) took the pretest immediately before the 6 week nutrition program and they (N=371) have taken the same test as posttest (repeated measure) upon the completion of the program. SO there is a difference in subject number(students) and pretest and posttest cannot be paired for each subject. I just have a whole bunch of pretest, and posttest from the same population. -- View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Selecting-a-Statistical-Test-for-unmatched-pre-post-survey-tp4830008p4830620.html Sent from the SPSSX Discussion mailing list archive at Nabble.com. ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD |
Re: Selecting a Statistical Test for unmatched pre post survey
|
|
So your diference is only 5 cases. Would 5 cases afect the geralizability of your results? I mean, if you use a paired sample t-test you could do it with 366 cases. If you found a match for each case you would do it with 371 cases. Are this 5 cases really a big problem for generalizond the results to the population? Andrés Mg. Andrés Burga León Coordinador de Análisis e Informática Unidad de Medición de la Calidad Educativa Ministerio de Educación del Perú Calle El Comercio s/n (espalda del Museo de la Nación) Lima 41 Perú Teléfono 615-5840
So, for those who are in both the pre- and post-test groups, you have no way of matching them? If you can match them, then I would think a paired samples t-test would be appropriate. If you can't match them, I don't know how much of a violation of assumptions it would be to use an independent samples t-test. From: seguchi <[hidden email]> To: [hidden email] Date: 09/22/2011 11:00 AM Subject: Re: Selecting a Statistical Test for unmatched pre post survey Sent by: "SPSSX(r) Discussion" <[hidden email]> They are not paired samples. Students (N=366) took the pretest immediately before the 6 week nutrition program and they (N=371) have taken the same test as posttest (repeated measure) upon the completion of the program. SO there is a difference in subject number(students) and pretest and posttest cannot be paired for each subject. I just have a whole bunch of pretest, and posttest from the same population. -- View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Selecting-a-Statistical-Test-for-unmatched-pre-post-survey-tp4830008p4830620.html Sent from the SPSSX Discussion mailing list archive at Nabble.com. ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD |
Re: Selecting a Statistical Test for unmatched pre post survey
|
In reply to this post by seguchi
How large is the population from which you drew
these samples?
Or is 371 sort of the whole population? Art Kendall Social Research Consultants On 9/22/2011 11:53 AM, seguchi wrote: ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARDThey are not paired samples. Students (N=366) took the pretest immediately before the 6 week nutrition program and they (N=371) have taken the same test as posttest (repeated measure) upon the completion of the program. SO there is a difference in subject number(students) and pretest and posttest cannot be paired for each subject. I just have a whole bunch of pretest, and posttest from the same population. -- View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Selecting-a-Statistical-Test-for-unmatched-pre-post-survey-tp4830008p4830620.html Sent from the SPSSX Discussion mailing list archive at Nabble.com. ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD
Art Kendall
Social Research Consultants |
|
|
In reply to this post by ANDRES ALBERTO BURGA LEON
Most of the pretest cases are the same as the posttest cases. However, some students dropped out of the program, so they took only pretest not posttest. And some students jumped into the program (due to change of school or...), and took only posttest no pretest. There were a lot of errors when administering the the survey/the pre test.
|
Re: Selecting a Statistical Test for unmatched pre post survey
|
|
So you can't match the pretest cases with the posttest cases and must use an independent samples t test because of that. What you lose, assuming a nice correlation between pretest and posttest (which you can't know in this sample since you can't match them up), is power.
Dr. Paul R. Swank, Children's Learning Institute Professor, Department of Pediatrics, Medical School Adjunct Professor, School of Public Health University of Texas Health Science Center-Houston -----Original Message----- From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of seguchi Sent: Thursday, September 22, 2011 1:03 PM To: [hidden email] Subject: Re: Selecting a Statistical Test for unmatched pre post survey Most of the pretest cases are the same as the posttest cases. However, some students dropped out of the program, so they took only pretest not posttest. And some students jumped into the program (due to change of school or...), and took only posttest no pretest. There were a lot of errors when administering the the survey/the pre test. -- View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Selecting-a-Statistical-Test-for-unmatched-pre-post-survey-tp4830008p4831000.html Sent from the SPSSX Discussion mailing list archive at Nabble.com. ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD |
Re: Selecting a Statistical Test for unmatched pre post survey
Administrator
|
In reply to this post by seguchi
So you have something which looks like this.
Pretest Posttest PRESENT PRESENT MATCHED CASES (N) PRESENT missing UNMATCHED CASES (N) missing PRESENT UNMATCHED CASES (N) ---------------------- What are these N's,the means and SD for each 'cell' and the correlation between Pretest and Posttest for the complete (MATCHED CASES). ----- In general (except when the pre/post are negatively correlated) the independent groups t test will be more conservative. How are your data currently structured?
Please reply to the list and not to my personal email.
Those desiring my consulting or training services please feel free to email me. --- "Nolite dare sanctum canibus neque mittatis margaritas vestras ante porcos ne forte conculcent eas pedibus suis." Cum es damnatorum possederunt porcos iens ut salire off sanguinum cliff in abyssum?" |
Re: Selecting a Statistical Test for unmatched pre post survey
|
|
In reply to this post by seguchi
I would start with EXPLORE and box plots.
Then I would take a large dose of salt and do a t-test pretending that I had two separate groups. On the assumption that the correlation between pre-test and post-test would not be negative if it were possible to calculate it, the size of the error term would be unduly large. That means that the t would be smaller than it would be if you could do it correctly. The significance would be understated. You can still apply the "so what" test to the difference in means even if it turns out that your means are not statistically distinct because you could not do a correct test. Be sure to strongly alert your audience to the limitations of your conclusions. In your future research it would be strongly advisable take the care to preserve the info to pair your data points and then sanitize your data if there is a confidentiality consideration. Art Kendall Social Research Consultants On 9/22/2011 2:03 PM, seguchi wrote: ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARDMost of the pretest cases are the same as the posttest cases. However, some students dropped out of the program, so they took only pretest not posttest. And some students jumped into the program (due to change of school or...), and took only posttest no pretest. There were a lot of errors when administering the the survey/the pre test. -- View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Selecting-a-Statistical-Test-for-unmatched-pre-post-survey-tp4830008p4831000.html Sent from the SPSSX Discussion mailing list archive at Nabble.com. ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD
Art Kendall
Social Research Consultants |
Re: Selecting a Statistical Test for unmatched pre post survey
|
Administrator
|
In reply to this post by seguchi
When I responded earlier, I understood you to have two completely different (independent) groups of students pre and post program. Things are clearer now. But it's still not clear to me whether you can match for those students with two data points. If you cannot match, then I would do as others have suggested and use an independent groups t-test.
If you can match for students with two data points, you could use a multilevel model with occasion clustered in ID (using the MIXED procedure). In that kind of model, it's fine for some students to have only 1 of the two data points. Note that this would require a data set with one row per data point rather than one row per student (with a variable indicating pre vs post). HTH.
--
Bruce Weaver bweaver@lakeheadu.ca http://sites.google.com/a/lakeheadu.ca/bweaver/ "When all else fails, RTFM." PLEASE NOTE THE FOLLOWING: 1. My Hotmail account is not monitored regularly. To send me an e-mail, please use the address shown above. 2. The SPSSX Discussion forum on Nabble is no longer linked to the SPSSX-L listserv administered by UGA (https://listserv.uga.edu/). |
|
|
In reply to this post by David Marso
N is the subject numbers who took the pretest/posttest. The number increased in posttest for some reasons (could be change of school or available seating in classroom or...). So far I was able to divide up both pretest and posttest in grade levels. but there is no way I can find matched pairs. The survey only had the pictures of fruits and vegetables, and students were only have to circle the pictures that apply to them.
I have tried independent t-test, but I was not sure how accurate my analysis was. I thought of doing dependent t-test since both dependent and independent varibles are on interval level. My data looks like this in excel Grade level School Testscore Pretest/Posttest 1st PS1 20 Pre 2nd PS2 5 Post : : : : I am cleaning the data and putting them in SPSS and just grew concerned about the reliability of my analysis (since I was not there to administer those tests nor was able to instruct the evaluation team on how to administer the tests). However, I greatly appreciate all of you for these valuable advises. If you have any thoughts or anything, I'd be grateful to hear them. Thanks so much. |
Re: Selecting a Statistical Test for unmatched pre post survey
|
|
How many schools are included?
Dr. Paul R. Swank, Children's Learning Institute Professor, Department of Pediatrics, Medical School Adjunct Professor, School of Public Health University of Texas Health Science Center-Houston -----Original Message----- From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of seguchi Sent: Thursday, September 22, 2011 2:13 PM To: [hidden email] Subject: Re: Selecting a Statistical Test for unmatched pre post survey N is the subject numbers who took the pretest/posttest. The number increased in posttest for some reasons (could be change of school or available seating in classroom or...). So far I was able to divide up both pretest and posttest in grade levels. but there is no way I can find matched pairs. The survey only had the pictures of fruits and vegetables, and students were only have to circle the pictures that apply to them. I have tried independent t-test, but I was not sure how accurate my analysis was. I thought of doing dependent t-test since both dependent and independent varibles are on interval level. My data looks like this in excel Grade level School Testscore Pretest/Posttest 1st PS1 20 Pre 2nd PS2 5 Post : : : : I am cleaning the data and putting them in SPSS and just grew concerned about the reliability of my analysis (since I was not there to administer those tests nor was able to instruct the evaluation team on how to administer the tests). However, I greatly appreciate all of you for these valuable advises. If you have any thoughts or anything, I'd be grateful to hear them. Thanks so much. -- View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Selecting-a-Statistical-Test-for-unmatched-pre-post-survey-tp4830008p4831215.html Sent from the SPSSX Discussion mailing list archive at Nabble.com. ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD |
|
|
23 schools
|
Re: Selecting a Statistical Test for unmatched pre post survey
|
Administrator
|
Paul Swank:
How many schools are included? "seguchi": 23 schools This is sounding more like a multilevel model all the time. Occasions are clustered within students, and students are clustered within schools (or possibly within classes within schools). To the OP: I suggest you look at Jos Twisk's very accessible introduction to mulitlevel models, "Applied Multilevel Analysis". At least some of it is available for preview via Google Books. There are other more comprehensive books (e.g., Snijders & Bosker), but I think you'll find them easier to digest if you start with Twisk. HTH.
--
Bruce Weaver bweaver@lakeheadu.ca http://sites.google.com/a/lakeheadu.ca/bweaver/ "When all else fails, RTFM." PLEASE NOTE THE FOLLOWING: 1. My Hotmail account is not monitored regularly. To send me an e-mail, please use the address shown above. 2. The SPSSX Discussion forum on Nabble is no longer linked to the SPSSX-L listserv administered by UGA (https://listserv.uga.edu/). |
Re: Selecting a Statistical Test for unmatched pre post survey
|
|
The other option is to do the analysis at the school level, averaging the data within schools unless you want to consider grade level as well. The means at the school level would be paired and so a dependent samples test could be done. 23 schools is a little light for multilevel. Most folks recommend at least 30 level 2 units to get stable variance estimates.
Dr. Paul R. Swank, Children's Learning Institute Professor, Department of Pediatrics, Medical School Adjunct Professor, School of Public Health University of Texas Health Science Center-Houston -----Original Message----- From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Bruce Weaver Sent: Thursday, September 22, 2011 3:33 PM To: [hidden email] Subject: Re: Selecting a Statistical Test for unmatched pre post survey Paul Swank: How many schools are included? "seguchi": 23 schools This is sounding more like a multilevel model all the time. Occasions are clustered within students, and students are clustered within schools (or possibly within classes within schools). To the OP: I suggest you look at Jos Twisk's very accessible introduction to mulitlevel models, "Applied Multilevel Analysis". At least some of it is available for preview via Google Books. There are other more comprehensive books (e.g., Snijders & Bosker), but I think you'll find them easier to digest if you start with Twisk. HTH. ----- -- Bruce Weaver [hidden email] http://sites.google.com/a/lakeheadu.ca/bweaver/ "When all else fails, RTFM." NOTE: My Hotmail account is not monitored regularly. To send me an e-mail, please use the address shown above. -- View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Selecting-a-Statistical-Test-for-unmatched-pre-post-survey-tp4830008p4831434.html Sent from the SPSSX Discussion mailing list archive at Nabble.com. ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD |
«
Return to SPSSX Discussion
|
1 view|%1 views
| Free forum by Nabble | Edit this page |