Help in choosing correct tests and in interpretation (parametric vs. non-parametric)

[lost1]

Hello, 

I need to analyse a dataset with pre- and post-test scores. 

I'm not sure which tests to use and if I interpret them correctly.

Can I use an ANOVA  or should I conduct individual non-parametric tests? 

Depending on the independent variables in the ANOVA the cells can have 2 samples or even be empty.

Below I have posted my results divided by the most important questions I'd like to answer:

I'd really appreciate if I could get some help here. 

I've uploaded the dataset here:

https://app.box.com/s/jzn6rhlplekd6n1s24le

Is there overall an improvement in the scores pre- to post-test?

Judging from the histograms they look quite normally distributed for me. But Shapiro yields a significant result (p < .001) for the difference scores for ScoreA and TotalScore. So a t-test probably is not appropriate. I'm not sure what to believe. So I try both parametric and non-parametric tests.

A paired t-test claims no significant differences for each scores.

T-TEST PAIRS=TotalScore_I ScoreA_I ScoreB_I WITH TotalScore_II ScoreA_II ScoreB_II (PAIRED)

  /CRITERIA=CI(.9500)

  /MISSING=ANALYSIS.

But Wilcoxon test claims a significant improvement in ScoreA, U = -2.19, p < .05, r = -0.27. 

NPAR TESTS

  /WILCOXON=TotalScore_I ScoreA_I ScoreB_I WITH TotalScore_II ScoreA_II ScoreB_II (PAIRED)

  /MISSING ANALYSIS

  /METHOD=EXACT TIMER(5).

Which test should I choose? 

Does location or year have a significant effect on the scores?

Repeated measures ANOVA with ScoreA_I-II and ScoreB_I-II as within factors and location and year as a between subject factor.

GLM ScoreA_I ScoreA_II ScoreB_I ScoreB_II BY location year

  /WSFACTOR=ScoreA 2 Polynomial ScoreB 2 Polynomial

  /METHOD=SSTYPE(3)

  /PLOT=PROFILE(ScoreA)

  /EMMEANS=TABLES(ScoreA) COMPARE ADJ(BONFERRONI)

  /PRINT=DESCRIPTIVE

  /CRITERIA=ALPHA(.05)

  /WSDESIGN=ScoreA ScoreB ScoreA*ScoreB

  /DESIGN=location year location*year.

Whether year and location nor their combination have an significant effect on any of the scores. 

The mean scores for the within factor ScoreA are significantly different, F(1, 58 ) = 5.19, p < .05.

Judging from the arithmetic means and the pairwise comparison the post-test ScoreA was significantly higher. 

The profile plot shows the exact opposite direction though, indicating a significant decrease. 

Which direction is correct?

Why does the ANOVA yield a significant result for ScoreA, whereas the paired t-test did not?

Non-parametric Wilcoxon method

SORT CASES  BY year.

SPLIT FILE LAYERED BY year.

NPAR TESTS

  /WILCOXON=TotalScore_I ScoreA_I ScoreB_I WITH TotalScore_II ScoreA_II ScoreB_II (PAIRED)

  /MISSING ANALYSIS

  /METHOD=EXACT TIMER(5).

This test claims first year subjects improved significantly in the total score and ScoreA, whereas the second year did not. 

ANOVA did not yield significance here.

SORT CASES  BY location.

SPLIT FILE LAYERED BY location.

NPAR TESTS

  /WILCOXON=TotalScore_I ScoreA_I ScoreB_I WITH TotalScore_II ScoreA_II ScoreB_II (PAIRED)

  /MISSING ANALYSIS

  /METHOD=EXACT TIMER(5).

First year subjects in location A improved significantly in the total score (U = -2.21, p < .05, r = -0.28) and ScoreA (U = -2.87, p < .01, r = -0.36). 

SORT CASES  BY year location.

SPLIT FILE LAYERED BY year location.

NPAR TESTS

  /WILCOXON=TotalScore_I ScoreA_I ScoreB_I WITH TotalScore_II ScoreA_II ScoreB_II (PAIRED)

  /MISSING ANALYSIS

  /METHOD=EXACT TIMER(5).

First year subjects in location A improved significantly in the total score (U = -2.09, p < .05, r = -0.26) and ScoreA (U = -2.04, p < .05, r = -0.26). 

Do I need to apply a Bonferroni correction? If yes, with what factor do I have to multiply the p-values?

Thank you very much & best regards!

[Rich Ulrich]

Your questions are not about using SPSS, and I'm not re-running the
analyses.  But here are some comments

Non-parametric vs parametric.  One useful *answer* for an appearance
of non-normality is when you can say, "It comes out the same either way."
It is nice to be able to point to robustness.  Absence of robustness is a
problem, and, possibly, part of the conclusion.

In your case, it seemingly does not come out the same way.  But you
provide p-levels in the form that journals should not like, these days --
"p < 0.05"  rather than, for example, "p= 0.02".  A result of p=0.049 is not
really very different from "p=0.051".  How big is your difference?
 
" ... histograms ... look ... normally distributed to me."  I have heard that
before, from folks who don't appreciate the impact of a single high/low outlier.
"Variance" is the sum of squared deviations from the mean.  Consider a single
score accounting for 9 percent of the variance among 100 scores:  That is the
contribution of a z-score of 3, which is not unusual among 100 standard normal
scores.  A z-score of 5, however, accounts for 25% -- You do not want one case
to be worth that much; and that is one way that even one outlier can mess up
an ANOVA.  Increasing the overall variance is another possible distortion.  How
extreme is your most extreme?

"... the cells can have 2 samples or even be empty."  Your design does not look
that complicated.  Does this comment indicate a lot of missing data?  Or does it
imply that "location" as a variable is not readily tested with any power?

Multiple tests.  One of the best solutions to "too many tests", when it can be
applied, is to set up a clear hierarchy of tests.  For instance:  If TOTAL is
supposed to subsume testA and testB, then your overall test should be on TOTAL;
a single result for one of the others is thus relegated to being a "suggestive result"
rather than a finding.   Similarly:  If your interest is that there should be a result that
is consistent across years, then the overall test (lumping) is the important one.
A difference between years (or, maybe, relevant interaction) is a *disconfirming*
outcome -- In that construction of the testing, it is not a finding, nor is it a desirable
outcome, or a main thing to be tested.   Similarly: for locations.  An interaction between
locations and outcome is what shows that locations differ; not, "It is p< 0.05 at one
location but not (quite) at another." 

Question about Profile plot and inconsistency with means:  The preceding ANOVA
(GLM) describes polynomial contrasts between the two two-leveled within-factors. 
Do you know for sure which direction the contrasts run?
Also: Do scores A and B have the same standard deviations, so that it is proper to
pool them in the analysis?  (I think your analysis does that.)

--
Rich Ulrich

---

Date: Fri, 21 Mar 2014 14:23:38 +0100
From: [hidden email]
Subject: Help in choosing correct tests and in interpretation (parametric vs. non-parametric)
To: [hidden email]

Hello, 

I need to analyse a dataset with pre- and post-test scores. 

I'm not sure which tests to use and if I interpret them correctly.

Can I use an ANOVA  or should I conduct individual non-parametric tests? 

Depending on the independent variables in the ANOVA the cells can have 2 samples or even be empty.

Below I have posted my results divided by the most important questions I'd like to answer:

I'd really appreciate if I could get some help here. 

I've uploaded the dataset here:

https://app.box.com/s/jzn6rhlplekd6n1s24le

Is there overall an improvement in the scores pre- to post-test?

Judging from the histograms they look quite normally distributed for me. But Shapiro yields a significant result (p < .001) for the difference scores for ScoreA and TotalScore. So a t-test probably is not appropriate. I'm not sure what to believe. So I try both parametric and non-parametric tests.

A paired t-test claims no significant differences for each scores.

T-TEST PAIRS=TotalScore_I ScoreA_I ScoreB_I WITH TotalScore_II ScoreA_II ScoreB_II (PAIRED)

  /CRITERIA=CI(.9500)

  /MISSING=ANALYSIS.

But Wilcoxon test claims a significant improvement in ScoreA, U = -2.19, p < .05, r = -0.27. 

NPAR TESTS

  /WILCOXON=TotalScore_I ScoreA_I ScoreB_I WITH TotalScore_II ScoreA_II ScoreB_II (PAIRED)

  /MISSING ANALYSIS

  /METHOD=EXACT TIMER(5).

Which test should I choose? 

Does location or year have a significant effect on the scores?

Repeated measures ANOVA with ScoreA_I-II and ScoreB_I-II as within factors and location and year as a between subject factor.

GLM ScoreA_I ScoreA_II ScoreB_I ScoreB_II BY location year

  /WSFACTOR=ScoreA 2 Polynomial ScoreB 2 Polynomial

  /METHOD=SSTYPE(3)

  /PLOT=PROFILE(ScoreA)

  /EMMEANS=TABLES(ScoreA) COMPARE ADJ(BONFERRONI)

  /PRINT=DESCRIPTIVE

  /CRITERIA=ALPHA(.05)

  /WSDESIGN=ScoreA ScoreB ScoreA*ScoreB

  /DESIGN=location year location*year.

Whether year and location nor their combination have an significant effect on any of the scores. 

The mean scores for the within factor ScoreA are significantly different, F(1, 58 ) = 5.19, p < .05.

Judging from the arithmetic means and the pairwise comparison the post-test ScoreA was significantly higher. 

The profile plot shows the exact opposite direction though, indicating a significant decrease. 

Which direction is correct?

Why does the ANOVA yield a significant result for ScoreA, whereas the paired t-test did not?

Non-parametric Wilcoxon method

SORT CASES  BY year.

SPLIT FILE LAYERED BY year.

NPAR TESTS

  /WILCOXON=TotalScore_I ScoreA_I ScoreB_I WITH TotalScore_II ScoreA_II ScoreB_II (PAIRED)

  /MISSING ANALYSIS

  /METHOD=EXACT TIMER(5).

This test claims first year subjects improved significantly in the total score and ScoreA, whereas the second year did not. 

ANOVA did not yield significance here.

SORT CASES  BY location.

SPLIT FILE LAYERED BY location.

NPAR TESTS

  /WILCOXON=TotalScore_I ScoreA_I ScoreB_I WITH TotalScore_II ScoreA_II ScoreB_II (PAIRED)

  /MISSING ANALYSIS

  /METHOD=EXACT TIMER(5).

First year subjects in location A improved significantly in the total score (U = -2.21, p < .05, r = -0.28) and ScoreA (U = -2.87, p < .01, r = -0.36). 

SORT CASES  BY year location.

SPLIT FILE LAYERED BY year location.

NPAR TESTS

  /WILCOXON=TotalScore_I ScoreA_I ScoreB_I WITH TotalScore_II ScoreA_II ScoreB_II (PAIRED)

  /MISSING ANALYSIS

  /METHOD=EXACT TIMER(5).

First year subjects in location A improved significantly in the total score (U = -2.09, p < .05, r = -0.26) and ScoreA (U = -2.04, p < .05, r = -0.26). 

Do I need to apply a Bonferroni correction? If yes, with what factor do I have to multiply the p-values?

Thank you very much & best regards!