Post-stratification and sig tests
Locked
Post-stratification and sig tests
 
|
Hi. Suppose that survey responses have been assigned a
column of weights so that the data’s demographic proportions (e.g.,
gender, age) will match proportions of population strata, without the weighting
increasing total N (the average weight is 1). And suppose this data set
is then brought into SPSS PASW. Then, as I understand it (based on posts to this ListServ in
2003 and 2006), I would need to use the “Complex Sampling” module
to correctly estimate the sig tests performed on such data, because the base
module assumes only simple random sampling (without stratified weighting) and
so would underestimate variance. Is this still the case with the latest version of SPSS PASW just
released? The only correct sig tests for such weighted data are in the
Complex Sampling module? Regards, Chris Stetson |
Re: Post-stratification and sig tests
|
|
The problem does not reside in the fact
that you are applying weights (as long as they do not increase total N), but in
the sampling design. If the sample is a simple or stratified random sample, not
involving clustering, you would not need complex samples: the weighted sample
(with non expansionary weights) would behave (approximately) as a simple random
sample of the same size, and SPSS significance tests would be valid. When the
sampling design involves clustering (e.g. if you first select cities out of a
list of cities, and then select city blocks out of the list of city blocks in
each city, and finally do a random selection of households within selected city
blocks), the selection of clusters (cities and city blocks) increases the
margin of error, and therefore the ordinary SPSS sig. tests would underestimate
the actual sampling error. In such cases, you’d need the complex samples
module. However, if you do have the module, you can use it anyway even for a
non-clustered stratified sample. Hector From: SPSSX(r)
Discussion Hi.
Suppose that survey responses have been assigned a column of weights so that
the data’s demographic proportions (e.g., gender, age) will match
proportions of population strata, without the weighting increasing total N (the
average weight is 1). And suppose this data set is then brought into SPSS
PASW. Then,
as I understand it (based on posts to this ListServ in 2003 and 2006), I would
need to use the “Complex Sampling” module to correctly estimate the
sig tests performed on such data, because the base module assumes only simple
random sampling (without stratified weighting) and so would underestimate
variance. Is
this still the case with the latest version of SPSS PASW just released?
The only correct sig tests for such weighted data are in the Complex Sampling
module? Regards, Chris
Stetson No virus found in this incoming message. |
|
Hector, by this statement
"If the sample is a simple or
stratified random sample, not involving clustering, you would not need complex
samples: the weighted sample (with non expansionary weights) would behave
(approximately) as a simple random sample of the same size, and SPSS
significance tests would be valid."
Are you saying that data
non-cluster weighting does not affect tests of significance?
W From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Hector Maletta Sent: Thursday, August 27, 2009 6:40 PM To: [hidden email] Subject: Re: Post-stratification and sig tests The problem does not
reside in the fact that you are applying weights (as long as they do not
increase total N), but in the sampling design. If the sample is a simple or
stratified random sample, not involving clustering, you would not need complex
samples: the weighted sample (with non expansionary weights) would behave
(approximately) as a simple random sample of the same size, and SPSS
significance tests would be valid. When the sampling design involves clustering
(e.g. if you first select cities out of a list of cities, and then select city
blocks out of the list of city blocks in each city, and finally do a random
selection of households within selected city blocks), the selection of clusters
(cities and city blocks) increases the margin of error, and therefore the
ordinary SPSS sig. tests would underestimate the actual sampling error. In such
cases, you’d need the complex samples module. However, if you do have the
module, you can use it anyway even for a non-clustered stratified sample.
Hector From: SPSSX(r)
Discussion Hi. Suppose that survey responses have been
assigned a column of weights so that the data’s demographic proportions (e.g.,
gender, age) will match proportions of population strata, without the weighting
increasing total N (the average weight is 1). And suppose this data set is
then brought into SPSS PASW. Then,
as I understand it (based on posts to this ListServ in 2003 and 2006), I would
need to use the “Complex Sampling” module to correctly estimate the sig tests
performed on such data, because the base module assumes only simple random
sampling (without stratified weighting) and so would underestimate
variance. Is
this still the case with the latest version of SPSS PASW just released?
The only correct sig tests for such weighted data are in the Complex Sampling
module? Regards, Chris
Stetson No virus found in this incoming message.
Will
Statistical Services ============ info.statman@earthlink.net http://home.earthlink.net/~z_statman/ ============ |
Re: Post-stratification and sig tests
|
|
I probably was a bit careless in my
previous message. First, note that all this refers to
non-expansionary weights, i.e. preserving sample size (weighted number of cases
= unweighted number of cases). It refers also to random sampling. SPSS significance tests assume the
(weighted) data set is a SIMPLE RANDOM SAMPLE drawn from a population which is
vastly larger than the sample. Relative to a simple random sample, clustering
increases sampling error, while stratification decreases sampling error.
Treating a stratified (non clustered) random sample as a simple random sample,
one obtains a conservative, probably underestimated significance level: taking
stratification into account would make the results more significant. On the
other hand, ignoring clustering may exaggerate the significance of results: you
may obtain a result which appears to be “significant at the 95% level”
when in fact (once the effect of clustering is considered) the results were not
significant. Therefore, in response to your question:
strictly speaking, SPSS significance tests (without using Complex Samples) are
only good for simple random samples. They ignore the gains (in precision) from
stratification, and the losses from clustering. If a sample is stratified and
not clustered, SPSS significance tests are an upper bound for the standard
error and a conservative estimate of significance. Hector From: Statmanz Hector, by this statement "If the sample is a simple or stratified random sample, not involving
clustering, you would not need complex samples: the weighted sample (with non
expansionary weights) would behave (approximately) as a simple random sample of
the same size, and SPSS significance tests would be valid." Are you
saying that data non-cluster weighting does not affect tests of significance? W From: SPSSX(r)
Discussion The problem does not reside in the fact
that you are applying weights (as long as they do not increase total N), but in
the sampling design. If the sample is a simple or stratified random sample, not
involving clustering, you would not need complex samples: the weighted sample
(with non expansionary weights) would behave (approximately) as a simple random
sample of the same size, and SPSS significance tests would be valid. When the
sampling design involves clustering (e.g. if you first select cities out of a
list of cities, and then select city blocks out of the list of city blocks in
each city, and finally do a random selection of households within selected city
blocks), the selection of clusters (cities and city blocks) increases the
margin of error, and therefore the ordinary SPSS sig. tests would underestimate
the actual sampling error. In such cases, you’d need the complex samples
module. However, if you do have the module, you can use it anyway even for a
non-clustered stratified sample. Hector From: SPSSX(r)
Discussion Hi.
Suppose that survey responses have been assigned a column of weights so that
the data’s demographic proportions (e.g., gender, age) will match
proportions of population strata, without the weighting increasing total N (the
average weight is 1). And suppose this data set is then brought into SPSS
PASW. Then,
as I understand it (based on posts to this ListServ in 2003 and 2006), I would
need to use the “Complex Sampling” module to correctly estimate the
sig tests performed on such data, because the base module assumes only simple
random sampling (without stratified weighting) and so would underestimate
variance. Is
this still the case with the latest version of SPSS PASW just released?
The only correct sig tests for such weighted data are in the Complex Sampling
module? Regards, Chris
Stetson No
virus found in this incoming message. No virus found in this incoming message. |
|
|
Yes, Hector, thanks for clarifying; Would not
won't someone to read too much into
that statement.
Will From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Hector Maletta Sent: Thursday, August 27, 2009 9:16 PM To: [hidden email] Subject: Re: Post-stratification and sig tests I probably was a bit
careless in my previous message. First, note that all
this refers to non-expansionary weights, i.e. preserving sample size (weighted
number of cases = unweighted number of cases). It refers also to random
sampling. SPSS significance tests
assume the (weighted) data set is a SIMPLE RANDOM SAMPLE drawn from a population
which is vastly larger than the sample. Relative to a simple
random sample, clustering increases sampling error, while stratification
decreases sampling error. Treating a stratified (non clustered) random sample as
a simple random sample, one obtains a conservative, probably underestimated
significance level: taking stratification into account would make the results
more significant. On the other hand, ignoring clustering may exaggerate the
significance of results: you may obtain a result which appears to be
“significant at the 95% level” when in fact (once the effect of clustering is
considered) the results were not significant. Therefore, in response
to your question: strictly speaking, SPSS significance tests (without using
Complex Samples) are only good for simple random samples. They ignore the gains
(in precision) from stratification, and the losses from clustering. If a sample
is stratified and not clustered, SPSS significance tests are an upper bound for
the standard error and a conservative estimate of significance.
Hector From: Statmanz
Hector, by this
statement "If the sample is a simple
or stratified random sample, not involving clustering, you would not need
complex samples: the weighted sample (with non expansionary weights) would
behave (approximately) as a simple random sample of the same size, and SPSS
significance tests would be valid." Are you saying that data
non-cluster weighting does not affect tests of significance? W From: SPSSX(r)
Discussion The problem does not
reside in the fact that you are applying weights (as long as they do not
increase total N), but in the sampling design. If the sample is a simple or
stratified random sample, not involving clustering, you would not need complex
samples: the weighted sample (with non expansionary weights) would behave
(approximately) as a simple random sample of the same size, and SPSS
significance tests would be valid. When the sampling design involves clustering
(e.g. if you first select cities out of a list of cities, and then select city
blocks out of the list of city blocks in each city, and finally do a random
selection of households within selected city blocks), the selection of clusters
(cities and city blocks) increases the margin of error, and therefore the
ordinary SPSS sig. tests would underestimate the actual sampling error. In such
cases, you’d need the complex samples module. However, if you do have the
module, you can use it anyway even for a non-clustered stratified sample.
Hector From: SPSSX(r)
Discussion Hi. Suppose that survey responses have been
assigned a column of weights so that the data’s demographic proportions (e.g.,
gender, age) will match proportions of population strata, without the weighting
increasing total N (the average weight is 1). And suppose this data set is
then brought into SPSS PASW. Then,
as I understand it (based on posts to this ListServ in 2003 and 2006), I would
need to use the “Complex Sampling” module to correctly estimate the sig tests
performed on such data, because the base module assumes only simple random
sampling (without stratified weighting) and so would underestimate
variance. Is
this still the case with the latest version of SPSS PASW just released?
The only correct sig tests for such weighted data are in the Complex Sampling
module? Regards, Chris
Stetson No
virus found in this incoming message. No virus found in this incoming message.
Will
Statistical Services ============ info.statman@earthlink.net http://home.earthlink.net/~z_statman/ ============ |
Re: Post-stratification and sig tests
|
|
Thanks, Hector (and Will).
So if I stick to non-expansionary weights (i.e., averaging to 1), applied after
fielding is done when balancing to a population, then I should generally be
able to come out okay in SPSS, without using Complex Samples (in the absence of
cluster and stratification sampling applied during fielding). Chris From: SPSSX(r) Discussion
[mailto:[hidden email]] On Behalf Of Statmanz Yes, Hector, thanks for clarifying; Would not won't someone
to read too much into that statement. Will From: SPSSX(r) Discussion
[mailto:[hidden email]] On Behalf Of Hector Maletta I probably was a bit careless in my previous message. First, note that all this refers to non-expansionary weights, i.e.
preserving sample size (weighted number of cases = unweighted number of cases).
It refers also to random sampling. SPSS significance tests assume the (weighted) data set is a SIMPLE
RANDOM SAMPLE drawn from a population which is vastly larger than the sample. Relative to a simple random sample, clustering increases sampling
error, while stratification decreases sampling error. Treating a stratified
(non clustered) random sample as a simple random sample, one obtains a
conservative, probably underestimated significance level: taking stratification
into account would make the results more significant. On the other hand,
ignoring clustering may exaggerate the significance of results: you may obtain
a result which appears to be “significant at the 95% level” when in
fact (once the effect of clustering is considered) the results were not
significant. Therefore, in response to your question: strictly speaking, SPSS
significance tests (without using Complex Samples) are only good for simple
random samples. They ignore the gains (in precision) from stratification, and
the losses from clustering. If a sample is stratified and not clustered, SPSS
significance tests are an upper bound for the standard error and a conservative
estimate of significance. Hector From: Statmanz
[mailto:[hidden email]] Hector, by this statement "If the sample is a simple or stratified random sample, not involving
clustering, you would not need complex samples: the weighted sample (with non
expansionary weights) would behave (approximately) as a simple random sample of
the same size, and SPSS significance tests would be valid." Are you saying that data non-cluster weighting does not affect tests
of significance? W From: SPSSX(r) Discussion
[mailto:[hidden email]] On Behalf Of Hector Maletta The problem does not reside in the fact that you are applying
weights (as long as they do not increase total N), but in the sampling design.
If the sample is a simple or stratified random sample, not involving
clustering, you would not need complex samples: the weighted sample (with non
expansionary weights) would behave (approximately) as a simple random sample of
the same size, and SPSS significance tests would be valid. When the sampling
design involves clustering (e.g. if you first select cities out of a list of
cities, and then select city blocks out of the list of city blocks in each
city, and finally do a random selection of households within selected city
blocks), the selection of clusters (cities and city blocks) increases the
margin of error, and therefore the ordinary SPSS sig. tests would underestimate
the actual sampling error. In such cases, you’d need the complex samples
module. However, if you do have the module, you can use it anyway even for a
non-clustered stratified sample. Hector From: SPSSX(r) Discussion
[mailto:[hidden email]] On Behalf Of Chris Stetson Hi. Suppose that survey responses have been assigned a
column of weights so that the data’s demographic proportions (e.g.,
gender, age) will match proportions of population strata, without the weighting
increasing total N (the average weight is 1). And suppose this data set
is then brought into SPSS PASW. Then, as I understand it (based on posts to this ListServ in
2003 and 2006), I would need to use the “Complex Sampling” module
to correctly estimate the sig tests performed on such data, because the base
module assumes only simple random sampling (without stratified weighting) and
so would underestimate variance. Is this still the case with the latest version of SPSS PASW
just released? The only correct sig tests for such weighted data are in
the Complex Sampling module? Regards, Chris Stetson No virus
found in this incoming message. No virus
found in this incoming message. |
| Free forum by Nabble | Edit this page |