Tests for unmatched pre/post/follow-up data

[Meddyg]

Hi all,

I would like to ask for help with the most appropriate (if there is one) statistical test to apply to some data I have. I have had a look at this forum to see if a similar query has been answered before to no avail.

Simply, I have created a new lesson for students. There are around 100 students in total. I asked them to complete a test before the new lesson, then after the lesson and then finally around 6 months later to see if they still remembered the material from the lessons. Each student attempt was then scored.

The test was not compulsory and collected anonymous responses, so I have no way of matching any of the pre/post/follow up responses. Also because the test was voluntary, I have different numbers of participants in each group ranging from around 110 pre, 95, post and 70 in follow up. All I know is that all participants took part in the new lesson so they have all received the intervention in the post and follow up groups and none of them had attended the session prior to this.

All the statistical tests I have looked at which would be appropriate for three time points, either assume the participants are different or are participants with same numbers in each group whose responses can be matched.

Could anyone advise on the best way of analysing the data? Happy to follow up with more info as required.

Many thanks

[Andy W]

I think generally in this situation doing independent sample t-tests is reasonable.

The point estimate is the same (ignoring missing data). One *typically* gains power by doing paired sample t-tests (when the measures have a positive correlation, the standard error of the difference is smaller). So given that assumption (people who test high early are also more likely to test high later), that standard error of the difference for independent samples will be larger than in the paired sample scenario (if you want to be really conservative choose the sample size N to be the smallest of the three samples).

Sample attrition is likely a bigger deal than worrying about the correlation between tests in this scenario. E.g. if people are more likely to take the later test if they know they will do well, or people who did poorly were more likely to drop the course. This is something that I believe is unsolvable with the information provided, so is a fundamental limitation of the analysis.

(Maybe you could come up with reasonable potential bounds on the bias though given the sample attrition? I am not sure.)