comparing groups before and after treatment

[Angshu Bhowmik]

Hello everyone,

I wonder if anyone could advise me (a clinical doctor with no statistical
training) about the correct way to test whether people have improved in 8
variables (e.g. how far they can walk, blood oxygen levels, spirometry, etc)
after an exercise program. I think they have, and the patients love it, but
I need to formally confirm it in order to keep the service running.

If I understand correctly, simply doing paired t-tests or Wilcoxon's tests
on each variable is not correct because multiple comparisons make it more
likely that something will show up. Is doing multiple t-tests and changing
the desired p value to 0.05 / 8 i.e. 0.00625 the only way to do this?

Many thanks for your kind help

Angshu

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[David Hitchin]

Quoting Angshu Bhowmik <[hidden email]>:

> Hello everyone,
>
> I wonder if anyone could advise me (a clinical doctor with no
> statistical training) about the correct way to test whether people
> have improved in 8 variables (e.g. how far they can walk, blood oxygen
levels, spirometry, etc)...

My first thought as a statistician is that you get over the problem of
multiple testing by performing a multivariate test, i.e. Hotelling's t
test or multivariate analysis of variance (HTT is the 2-sample case of
manova), but in fact this just gives the significance of change rather
than improvement. If the subjects have got much better on some measures
and worse on others, then you have significant CHANGE, but that isn't
very informative.

I think that you would be quite correct to do the tests one at a time,
provided that you publish all of them and not just the best of the
bunch, as each test tells you something different.

However, significance tests are NOT the best way of analysing and
presenting the data, even if the statistics courses that you did once
mostly taught signficance testing. You would do much better to think of
how you present the results and use both numeric results and graphical
displays (providing that you have enough data, graphics look rather
scrappy when you have only a few cases).

I suggest that you use Descriptives > Explore in SPSS and from the table
of statistics quote the means with their 95% lower and upper confidence
intervals. You might also, from Explore, consider using box-plots,
although you will need to provide a little annotation to explain to the
reader what they show.

David Hitchin

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Swank, Paul R]

Two points. First, a multivariate test would be appropriate if there is
a strong relation among the measured variables but you need to be
prepared to interpret a composite. Secondly, while this test will tell
you whether or not there was significant change over time, it cannot
tell you it was due to what you did. Without random assignment to
treatment or control conditions, you cannot infer causality.

Paul R. Swank, Ph.D.
Professor and Director of Research
Children's Learning Institute
University of Texas Health Science Center - Houston


-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of
Angshu Bhowmik
Sent: Tuesday, May 20, 2008 6:28 PM
To: [hidden email]
Subject: comparing groups before and after treatment

Hello everyone,

I wonder if anyone could advise me (a clinical doctor with no
statistical
training) about the correct way to test whether people have improved in
8
variables (e.g. how far they can walk, blood oxygen levels, spirometry,
etc)
after an exercise program. I think they have, and the patients love it,
but
I need to formally confirm it in order to keep the service running.

If I understand correctly, simply doing paired t-tests or Wilcoxon's
tests
on each variable is not correct because multiple comparisons make it
more
likely that something will show up. Is doing multiple t-tests and
changing
the desired p value to 0.05 / 8 i.e. 0.00625 the only way to do this?

Many thanks for your kind help

Angshu

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[David Hitchin]

Quoting Angshu Bhowmik <[hidden email]>:

> Hello everyone,
>
> I wonder if anyone could advise me (a clinical doctor with no
> statistical training) about the correct way to test whether people
> have improved in 8 variables (e.g. how far they can walk, blood
> oxygen levels,spirometry, etc) after an exercise program. I think
> they have, ...

Further to my previous suggestion, on the assumption that you have
"before" and "after" measures for each variable, compute "change" =
"after" - "before". A positive change is an improvement, a negative one
a deterioration. You might then use Explore to produce a box-and-whisker
plot, or simply report the improvements by percentages of patients, e.g.
something like "75% of patients could walk another mile", the top 50%
could walk another 2 miles and the top 25% could walk an extra 3 miles."
Statisticians nearly always prefer nice clear presentations close to the
original data, ideally in plot form, rather than significance tests.

David Hitchin

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Johnny Amora]

Angshu wrote:

>I wonder if anyone could advise me (a clinical doctor with no statistical
training) about the correct way to test whether people have improved in 8
variables (e.g. how far they can walk, blood oxygen levels, spirometry, etc)
after an exercise program. I think they have, and the patients love it, but
I need to formally confirm it in order to keep the service running.
If I understand correctly, simply doing paired t-tests or Wilcoxon's tests
on each variable is not correct because multiple comparisons make it more
likely that something will show up. Is doing multiple t-tests and changing
the desired p value to 0.05 / 8 i.e. 0.00625 the only way to do this?

Reply:

If your patients were measured (on the 8 variables) in several occasions(say, four or five) within the duration of the exercise program, then go with latent growth modeling.  Using latent growth modeling approach, the growth (change) can be best uncover if the respondents are measured in several time points, rather than two points (before and after).


Johnny T. Amora
Center for Learning and Performance Assessment
De La Salle-College of Saint Benilde
Manila, Philippines

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Richard Ristow]

At 05:51 PM 5/21/2008, David Hitchin wrote:

>Further to my previous suggestion, on the assumption that you have
>"before" and "after" measures for each variable, compute "change" =
>"after" - "before". A positive change is an improvement, a negative
>one a deterioration.  You might then use Explore to produce a
>box-and-whisker plot, ...

Yes. I'd also present side-by-side box-and-whisker plots of 'before'
and 'after'. That will be illuminating, though for 'before' vs.
'after' designs, it may be too pessimistic.  For example, suppose the
standard deviation of 'before' is 50, all patients improved 10
points, and a 10-point improvement is clinically meaningful. Then the
side-by-side plots will look like little has happened, when actually
something useful has.

>Statisticians nearly always prefer nice clear presentations close to
>the original data, ideally in plot form, rather than significance tests.

Also true, though the significance tests also MUST be performed, and
presented. As has been repeatedly said, there are many circumstances
in which a significant result on a statistical test is meaningless.
However, if the relevant statistical test fails to show significance,
it is conclusive that no effect has been observed.

(An effect may still *exist*, and may be observable otherwise, but
that's another story.)

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[ViAnn Beadle]

Depending upon the number of cases, a parallel coordinate plot might be more
illuminating since it directly shows the direction and magnitude of change
for each case. Once you get more than 50 cases, the results look like
somebody dropped the box of spaghetti onto the floor. This chart is
available using GPL and specifying a parallel coordinate.

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of
Richard Ristow
Sent: Friday, May 23, 2008 10:16 AM
To: [hidden email]
Subject: Re: comparing groups before and after treatment

At 05:51 PM 5/21/2008, David Hitchin wrote:

>Further to my previous suggestion, on the assumption that you have
>"before" and "after" measures for each variable, compute "change" =
>"after" - "before". A positive change is an improvement, a negative
>one a deterioration.  You might then use Explore to produce a
>box-and-whisker plot, ...

Yes. I'd also present side-by-side box-and-whisker plots of 'before'
and 'after'. That will be illuminating, though for 'before' vs.
'after' designs, it may be too pessimistic.  For example, suppose the
standard deviation of 'before' is 50, all patients improved 10
points, and a 10-point improvement is clinically meaningful. Then the
side-by-side plots will look like little has happened, when actually
something useful has.
<snip/>

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Art Kendall]

Great! I had recommended that SPSS implement a parallel coordinate plot.

Art

ViAnn Beadle wrote:

> Depending upon the number of cases, a parallel coordinate plot might be more
> illuminating since it directly shows the direction and magnitude of change
> for each case. Once you get more than 50 cases, the results look like
> somebody dropped the box of spaghetti onto the floor. This chart is
> available using GPL and specifying a parallel coordinate.
>
> -----Original Message-----
> From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of
> Richard Ristow
> Sent: Friday, May 23, 2008 10:16 AM
> To: [hidden email]
> Subject: Re: comparing groups before and after treatment
>
> At 05:51 PM 5/21/2008, David Hitchin wrote:
>
>
>> Further to my previous suggestion, on the assumption that you have
>> "before" and "after" measures for each variable, compute "change" =
>> "after" - "before". A positive change is an improvement, a negative
>> one a deterioration.  You might then use Explore to produce a
>> box-and-whisker plot, ...
>>
>
> Yes. I'd also present side-by-side box-and-whisker plots of 'before'
> and 'after'. That will be illuminating, though for 'before' vs.
> 'after' designs, it may be too pessimistic.  For example, suppose the
> standard deviation of 'before' is 50, all patients improved 10
> points, and a 10-point improvement is clinically meaningful. Then the
> side-by-side plots will look like little has happened, when actually
> something useful has.
> <snip/>
>
> =====================
> To manage your subscription to SPSSX-L, send a message to
> [hidden email] (not to SPSSX-L), with no body text except the
> command. To leave the list, send the command
> SIGNOFF SPSSX-L
> For a list of commands to manage subscriptions, send the command
> INFO REFCARD
>
>
>

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Art Kendall]

Parallel coordinate plots are useful not only for individual cases when
there is a relatively small number but they are traditional for showing
the profiles of clusters.
They are also useful for showing compositional data, Independent
variables for Discriminant Function types of analysis, and Dependent
variables in MANOVA etc.

If the the data are not already on the same response scale, e.g., Likert
items, z-scores, T-scores, percentiles, growth profiles, etc. You would
want to transform them.

WRT plotting cases:
When there are only two coordinates representing the same construct
measured twice, this can be called a "pre-post ladder" chart .
When two variables are in percentiles or standardized scores, this can
also be called a ladder chart.
Either of these can be used as additional ways to visualize a
correlation.  The rungs give an impression of relative moves up or down.

Art Kendall
Social Research Consultants
*Celebrating the 60th Anniversary of the UN's Universal Declaration of
Human Rights*


Art Kendall wrote:

> Great! I had recommended that SPSS implement a parallel coordinate plot.
>
> Art
>
> ViAnn Beadle wrote:
>> Depending upon the number of cases, a parallel coordinate plot might
>> be more
>> illuminating since it directly shows the direction and magnitude of
>> change
>> for each case. Once you get more than 50 cases, the results look like
>> somebody dropped the box of spaghetti onto the floor. This chart is
>> available using GPL and specifying a parallel coordinate.
>>
>> -----Original Message-----
>> From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of
>> Richard Ristow
>> Sent: Friday, May 23, 2008 10:16 AM
>> To: [hidden email]
>> Subject: Re: comparing groups before and after treatment
>>
>> At 05:51 PM 5/21/2008, David Hitchin wrote:
>>
>>
>>> Further to my previous suggestion, on the assumption that you have
>>> "before" and "after" measures for each variable, compute "change" =
>>> "after" - "before". A positive change is an improvement, a negative
>>> one a deterioration.  You might then use Explore to produce a
>>> box-and-whisker plot, ...
>>>
>>
>> Yes. I'd also present side-by-side box-and-whisker plots of 'before'
>> and 'after'. That will be illuminating, though for 'before' vs.
>> 'after' designs, it may be too pessimistic.  For example, suppose the
>> standard deviation of 'before' is 50, all patients improved 10
>> points, and a 10-point improvement is clinically meaningful. Then the
>> side-by-side plots will look like little has happened, when actually
>> something useful has.
>> <snip/>
>>
>> =====================
>> To manage your subscription to SPSSX-L, send a message to
>> [hidden email] (not to SPSSX-L), with no body text except the
>> command. To leave the list, send the command
>> SIGNOFF SPSSX-L
>> For a list of commands to manage subscriptions, send the command
>> INFO REFCARD
>>
>>
>>
>
> =====================
> To manage your subscription to SPSSX-L, send a message to
> [hidden email] (not to SPSSX-L), with no body text except the
> command. To leave the list, send the command
> SIGNOFF SPSSX-L
> For a list of commands to manage subscriptions, send the command
> INFO REFCARD
>
>

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Richard Ristow]

I just noticed I'd missed this one. At 06:47 PM
5/23/2008, Angshu Bhowmik wrote, off-list:

>Thank you very much for your (as always) useful
>observations. I shall do as you say.

And good luck with it.

>But is it appropriate to just do separate
>t-tests for each of the observations and use the
>usual level of significance (p = 0.05) or do I
>need to use a cut-off of p = 0.00625 (for 8
>different parameters being measured)?

This gets into areas where I'm not the most
useful person on the list; that's one reason I'm
responding on-list. Take this as a beginning, only.

First, using "a cut-off of p = 0.00625" is the
Bonferroni correction. It's been argued on good
authority that it can be needlessly conservative.
Search archives for posts by Marta García-Granero
dealing with multiple comparisons.

Second (and this is as far as I'll take this), in
choosing your analysis, you need to think what
your questions are. For example (and perhaps
you've posted answers to some of these),

. Are these 8 quantities measures of health or
quality, which you hope the treatment will
increase; and you're testing whether the
treatment can be shown to improve health? If so,
that's a multivariate analysis; if the analysis
shows an effect of treatment, analyze for which
factors seem to show the effect. See the GLM
section on "Multivariate Analysis" (and consult others than I).

. If you're simply interested in those 8
quantities individually, then you have a
multiple-comparison situation, as you're aware.
You also have an experiment that is very
difficult to discuss and interpret: why THOSE
eight? what does the pattern of significant vs.
non-significant mean? It's important that, rather
than significant tests, you estimate confidence
intervals, i.e. estimates of effect sizes, and
then be prepared to discuss the relative
importance of the effect sizes you see. In
general, such a design (sometimes called a
'scatter-gun' design) must be used with caution,
for precisely these reasons of interpretability.

-Good luck to you,
  Richard

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD