independent t-test with dependent data

[Arlin Cuncic-2]

Hi list,

I'm not sure if this is a simple problem or not.  I'm analyzing the
results of a pre-and post survey in which no id numbers were  used to
match participants. As a result I have a file with pre data and a file
with post data, but no way to match participants.  I want to compare the
mean responses on several survey questions and the only way I can see to
do it is using independent t-tests.  Is this going to cause problems in
terms of interpretation?

Thanks,
Arlin

Arlin Cuncic, M.A.
Research and Assessment Associate
Research and Assessment Department
Thames Valley District School Board
1250 Dundas St.
London, ON N5W 5P2
519-452-2000 ext. 20115
[hidden email]

[Jerabek Jindrich]

Hi,

You might find useful some threads on propensity scores matching:
http://listserv.uga.edu/cgi-bin/wa?S2=spssx-l&q=propensity+score&0=S&s=&f=&a=&b=
or search the Raynald's site.

As for the unpaired test IMHO it can be used and the "only" problem is that it has less power than paired test.

regards
Jindra

> ------------ Původní zpráva ------------
> Od: Arlin Cuncic <[hidden email]>
> Předmět: independent t-test with dependent data
> Datum: 25.6.2007 17:49:58
> ----------------------------------------
> Hi list,
>
> I'm not sure if this is a simple problem or not.  I'm analyzing the
> results of a pre-and post survey in which no id numbers were  used to
> match participants. As a result I have a file with pre data and a file
> with post data, but no way to match participants.  I want to compare the
> mean responses on several survey questions and the only way I can see to
> do it is using independent t-tests.  Is this going to cause problems in
> terms of interpretation?
>
> Thanks,
> Arlin
>
> Arlin Cuncic, M.A.
> Research and Assessment Associate
> Research and Assessment Department
> Thames Valley District School Board
> 1250 Dundas St.
> London, ON N5W 5P2
> 519-452-2000 ext. 20115
> [hidden email]
>
>
>

[Swank, Paul R]

Propensity score matching might be a viable alternative if you have enough data in order to match them on things other than the pre and posttests. I might add, however, that while the independent samples  procedure is usually less powerful that the dependent samples t-test, this is based on the assumption of a positive correlation between the measures. If the correlation is 0 then the two procedures are the same and if, by chance, there happens to be a negative correlation between them, the independent samples t-test will be more powerful, but incorrectly so.

Paul R. Swank, Ph.D. Professor
Director of Reseach
Children's Learning Institute
University of Texas Health Science Center-Houston


-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Jerabek Jindrich
Sent: Monday, June 25, 2007 2:01 PM
To: [hidden email]
Subject: [SPAM+] Re:independent t-test with dependent data

Hi,

You might find useful some threads on propensity scores matching:
http://listserv.uga.edu/cgi-bin/wa?S2=spssx-l&q=propensity+score&0=S&s=&f=&a=&b=
or search the Raynald's site.

As for the unpaired test IMHO it can be used and the "only" problem is that it has less power than paired test.

regards
Jindra

> ------------ Původní zpráva ------------
> Od: Arlin Cuncic <[hidden email]>
> Předmět: independent t-test with dependent data
> Datum: 25.6.2007 17:49:58
> ----------------------------------------
> Hi list,
>
> I'm not sure if this is a simple problem or not.  I'm analyzing the
> results of a pre-and post survey in which no id numbers were  used to
> match participants. As a result I have a file with pre data and a file
> with post data, but no way to match participants.  I want to compare
> the mean responses on several survey questions and the only way I can
> see to do it is using independent t-tests.  Is this going to cause
> problems in terms of interpretation?
>
> Thanks,
> Arlin
>
> Arlin Cuncic, M.A.
> Research and Assessment Associate
> Research and Assessment Department
> Thames Valley District School Board
> 1250 Dundas St.
> London, ON N5W 5P2
> 519-452-2000 ext. 20115
> [hidden email]
>
>
>

[Marta Garcia-Granero]

Hi Arlin

The extent to which the results will be affected depends on the
intensity of the association between pre and post data. If it is weak,
then the results obtained by an unpaired or a paired test will be
roughly the same. But if the association is strong, then using an
independent samples test will be shift your results towards
non-significance (loss of power). Can you estimate the correlation (R)
between the pre and post data? If so, you could use SPSS to calculate
the means and Sd of pre and post data and then compute by hand the
following t-statistic:

                       mean1-mean2
T=--------------------------------------
    QRT(SD1^2+SD2^2-2·SD1·SD2.R)

Regards,
Marta Garcia-Granero

> I'm not sure if this is a simple problem or not.  I'm analyzing the
> results of a pre-and post survey in which no id numbers were  used to
> match participants. As a result I have a file with pre data and a file
> with post data, but no way to match participants.  I want to compare the
> mean responses on several survey questions and the only way I can see to
> do it is using independent t-tests.  Is this going to cause problems in
> terms of interpretation?
>
> Thanks,
> Arlin
>
> Arlin Cuncic, M.A.
> Research and Assessment Associate
> Research and Assessment Department
> Thames Valley District School Board
> 1250 Dundas St.
> London, ON N5W 5P2
> 519-452-2000 ext. 20115
> [hidden email]
>
>

[Marta Garcia-Granero]

I hit Send to fast. Just a small correction, the formula for the t test
should read:

                            mean1-mean2
T=-------------------------------------------
   SQRT[(SD1^2+SD2^2-2·SD1·SD2.R)/n]

(I forgot an element in the denominator of the equation).

> The extent to which the results will be affected depends on the
> intensity of the association between pre and post data. If it is weak,
> then the results obtained by an unpaired or a paired test will be
> roughly the same. But if the association is strong, then using an
> independent samples test will be shift your results towards
> non-significance (loss of power). Can you estimate the correlation (R)
> between the pre and post data? If so, you could use SPSS to calculate
> the means and Sd of pre and post data and then compute by hand the
> following t-statistic:
>
>                       mean1-mean2
> T=---------------------------------------
>     SQRT(SD1^2+SD2^2-2·SD1·SD2.R)
>
> Regards,
> Marta Garcia-Granero
>> I'm not sure if this is a simple problem or not.  I'm analyzing the
>> results of a pre-and post survey in which no id numbers were  used to
>> match participants. As a result I have a file with pre data and a file
>> with post data, but no way to match participants.  I want to compare the
>> mean responses on several survey questions and the only way I can see to
>> do it is using independent t-tests.  Is this going to cause problems in
>> terms of interpretation?
>>

[Marta Garcia-Granero]

Arlin Cuncic escribió:

> I'm not sure if this is a simple problem or not.  I'm analyzing the
> results of a pre-and post survey in which no id numbers were  used to
> match participants. As a result I have a file with pre data and a file
> with post data, but no way to match participants.  I want to compare the
> mean responses on several survey questions and the only way I can see to
> do it is using independent t-tests.  Is this going to cause problems in
> terms of interpretation?
>
>
The main problem will be power loss. If  the association between pre and
post values is strong, then you loose quite a lot o power, if the
association is weak/absent, then the independent samples t-test will
yield quite correct results.

Can you estimate the correlation between pre and post measures? Then you
can compute the paired samples t-test by hand:

t=(Mean1-Mean2)/[SQRT(Variance1+Variance2-R*SD1*SD2)]/n

Where:

n: number of pairs
SD: standard deviation (square root of the variance)
R: estimated correlation between pre&post measures.

As you can see, a strong correlation means that the standard error will
be lower, leading to more power.

HTH,
Marta

[Alexander J. Shackman-2]

if other investigators have published the test-retest stability of the
instrument, you might be able to use that as a best-guess of R

hth, alex shackman

On 8/8/07, Marta Garcia-Granero <[hidden email]> wrote:

>
> Arlin Cuncic escribió:
>
> > I'm not sure if this is a simple problem or not.  I'm analyzing the
> > results of a pre-and post survey in which no id numbers were  used to
> > match participants. As a result I have a file with pre data and a file
> > with post data, but no way to match participants.  I want to compare the
> > mean responses on several survey questions and the only way I can see to
> > do it is using independent t-tests.  Is this going to cause problems in
> > terms of interpretation?
> >
> >
> The main problem will be power loss. If  the association between pre and
> post values is strong, then you loose quite a lot o power, if the
> association is weak/absent, then the independent samples t-test will
> yield quite correct results.
>
> Can you estimate the correlation between pre and post measures? Then you
> can compute the paired samples t-test by hand:
>
> t=(Mean1-Mean2)/[SQRT(Variance1+Variance2-R*SD1*SD2)]/n
>
> Where:
>
> n: number of pairs
> SD: standard deviation (square root of the variance)
> R: estimated correlation between pre&post measures.
>
> As you can see, a strong correlation means that the standard error will
> be lower, leading to more power.
>
> HTH,
> Marta
>

--
Alexander J. Shackman
Laboratory for Affective Neuroscience
Waisman Laboratory for Brain Imaging & Behavior
University of Wisconsin-Madison
1202 West Johnson Street
Madison, Wisconsin 53706

Telephone: [phone]
FAX: [phone]
EMAIL: [hidden email]
http://psyphz.psych.wisc.edu/~shackman