Choosing Accurate Statistical Test for unpaired pre post survey

[Teesquared2014]

We implemented an educational program and we administered surveys (Pretest and Posttest). The problem is the sample do not match (from the same population) between pre and post.  For example, participants(N=97) took the pretest immediately before the 8 week program and only (N=49) took the same test as posttest (repeated measure) upon the completion of the program.  Only 40 of my sample matched for pre and post, the remaining cannot be paired which means some participants dropped out of the program, so they took only pretest not posttest; and some students jumped into the program and took only posttest.

My question pertains to the unpaired sample - do I run an independent t test? If no, what is the most accurate test for the unmpaired sample. Or do I exclude the unpaired participants from my sample, and run a paired t test?

[Rich Ulrich]

[I sent this a few hours ago to the List and to the OP.  The OP replied to me but the list-server never showed me any sign that SPSSX-L received it.  Re-posted now through Nabble.]

What do you want to say about the program?  What do your data let you say?

You have Pre data on 40 of 49 who have Post.  You have 57 cases with Pre
only, and those 9 with Post but not Pre.  Assuming that there is a decent r
between Pre and Post, almost all of your power of analysis will be used when
you compare the paired ratings for the Pre and Post.

A fuller description of the results also compares the Post for the 9 (little power)
with the Post for the 40.  A description of the dropouts compares the Pre for 40
with the Pre for 57.   If any of these tests are strongly suggestive of mean
differences, then it is hard to justify "pooling" the single-test subjects with the
others, except for overall description.  - It probably is possible to run a single,
sophisticated and complicated ANOVA that allows you to extract most of those
tests, but that is a misleading sophistication when it vastly increases the chance
of mis-reading the resulting means and contrasts, or using the wrong error terms.

If you have demographic data, you should compare that information on the
same subgroups.

--
Rich Ulrich

[Ryan]

I agree that you need to explore, to the best of your ability, if there are differences between subjects who have missing data from those who do not. Are "jumping in" and "jumping out" of the program random? Any particular reason you are aware of why people left the program early or started the program midstream that could implicate the findings?

Assuming the data are missing completely at random and you plan on performing a test on change in means, then a linear mixed model should be considered, since it will include all possible data in the parameter estimation (e.g., the pretest score from a subject who randomly dropped out before study completion will be included in the estimation of the pre-score mean).

Conclusions you may make from a single group study design such as this one are limited, but such issues go beyond what I generally discuss on SPSS-L.

Ryan

p.s., Although I have chosen to not ask about the psychometric properties of the test scores, they should be considered.

On Mon, Apr 21, 2014 at 11:39 AM, TeeTee2014 <[hidden email]> wrote:

We implemented an educational program and we administered surveys (Pretest
and Posttest). The problem is the sample do not match (from the same
population) between pre and post.  For example, participants(N=97) took the
pretest immediately before the 8 week program and only (N=49) took the same
test as posttest (repeated measure) upon the completion of the program.
Only 40 of my sample matched for pre and post, the remaining cannot be
paired which means some participants dropped out of the program, so they
took only pretest not posttest; and some students jumped into the program
and took only posttest.

My question pertains to the unpaired sample - do I run an independent t
test? If no, what is the most accurate test for the unmpaired sample. Or do
I exclude the unpaired participants from my sample, and run a paired t test?



--
View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Choosing-Accurate-Statistical-Test-for-unpaired-pre-post-survey-tp5725560.html
Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Teesquared2014]

Thank you all for your comments. It is extremely helpful - I will explore the suggestions proposed.

[Teesquared2014]

I have to follow - up questions:

In the same sample above, I would like to also know if the patients that indicated they had arthristis experienced reduced level of pain in pre/post intervention. What is the most appropiate analysis?
 
3) My other question is similar to initial post – Similar educational program but in this case my N = 14 for pretest and N = 14 for my posttest; however only 3 of my sample are paired pre and post which gives a total sample of N=25 if merged. What is the most accurate test?

Thank you!!