comparing regression coefficients over time using cross-sectional data

[Claudiu Tufis]

Dear list members,



I know this is not strictly SPSS related but I think I could benefit from
your expertise.





I am working with survey data (six cross-sectional datasets over a period of
10 years). I want to test if the effect of certain independent variables
changes over time.

The solution that I have right now is to put all the datasets into a single
dataset, create dummy variables for "year" and run the regression models
with the variables of interest, the dummies for year, and interaction terms
between the variables and these dummies. The significance and the sign of
the interaction terms will show then if the effects of the variables do
change over time.



Can you think of any other approaches that I could use for this problem?



Thank you very much,



Claudiu Tufis

[Hector Maletta]

         Pooling all the datasets together is a delicate operation which may
not be advisable if the data come from not comparable samples, or if these
samples are of different size, or not adequately weighted, or if the various
surveys are otherwise not strictly comparable to the degree required for
regarding them as a single data set.
         An alternative approach is running the analysis separately in the
various datasets, obtain measures of the impact of the IV on DV at the
various dates, and then compare the various results to evaluate whether
their differences may be due to mere chance or to the contrary they reveal
some trend or some change over time. In this alternative you would be
comparing several measures of regression, correlation or association coming
from different samples, testing the null hypothesis that they all come from
a population with the same measure.
         A still better approach is to use a multilevel model, with
individual cases at a certain cross section as the elementary level, and
dates as a higher level. Anyway, just remember to consider the question of
comparability, sample size and weights when you pool or compare the
different samples.

         Hector


         -----Mensaje original-----
De: SPSSX(r) Discussion [mailto:[hidden email]] En nombre de
Claudiu Tufis
Enviado el: 06 January 2007 16:53
Para: [hidden email]
Asunto: comparing regression coefficients over time using cross-sectional
data

         Dear list members,



         I know this is not strictly SPSS related but I think I could
benefit from
         your expertise.





         I am working with survey data (six cross-sectional datasets over a
period of
         10 years). I want to test if the effect of certain independent
variables
         changes over time.

         The solution that I have right now is to put all the datasets into
a single
         dataset, create dummy variables for "year" and run the regression
models
         with the variables of interest, the dummies for year, and
interaction terms
         between the variables and these dummies. The significance and the
sign of
         the interaction terms will show then if the effects of the
variables do
         change over time.



         Can you think of any other approaches that I could use for this
problem?



         Thank you very much,



         Claudiu Tufis

[statisticsdoc]

Cladiu,

Hector has already given some sound advice here, so let me just add a couple
of smaller points.

When judging whether an Independent Variable has the same effect across
samples (years), it is important to consider plausible alternative
hypotheses that could explain why the IV has a weaker effect in some samples
compared with others.  These rival hypotheses are alternatives to the view
that the IV has substantively different effects at different times.

One reason why the relationship between the Independent Variable and the
Dependent Variable might differ across samples arises from differences in
the variability of these measures.  If the IV and/or the DV has restricted
range in some samples, then the relationship between them would be
attenuated due to restriction of range.  There are may ways in which this
situation can arise:  differences between years in how the samples were
drawn, ceiling and floor effects, etc.  If there are marked differences
between samples in the range of the IV and/or the DV, your dataset does not
provide reliable information about differential effects of the IV across
years.

Another issue to be concerned about is differential levels of reliability in
the Independent Variable and/or the Dependent variable in different samples.
To the extent that the variables are measured with less reliability in
certain samples, the relationship between the IV and DV will be attentuated
due to measurement error.  For example, the way in which the survey was
administered in some samples might result in lower reliability (imagine a
high school health survey given en masse to hundreds of students crammed in
a gym under immense time pressure versus one given in better circumstances).
Check on this issue by computing Cronbach's alpha for each measure
separately at each time point.  You might want to consider a structural
equation modeling (SEM) approach to disentangle measurement error from the
strength of relationships between latent constructs.  Treat each sample as a
group.  Develop a measurement model for the IV and the DV.  Fit a
multi-group SEM model in which measurement errors are allowed to vary across
groups.  Compare a model in which the path from the latent IV to the latent
DV is constrained to be equal versus one in which this path is allowed to
vary across samples.

Assuming that the ranges and reliabilities are good and stable across years,
you can also consider a Hierarchical Linear Modeling approach.  The
individual data would be the level-1 variable, cohort membership would be
the level-2 variable, and the effect of interest would be the cross level
interaction between cohort membership and the effect of the IV at level 1 -
this effect would address the question of whether the effect of the IV
differs between samples.

HTH,

Stephen Brand

For personalized and professional consultation in statistics and research
design, visit
www.statisticsdoc.com


-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]]On Behalf Of
Claudiu Tufis
Sent: Saturday, January 06, 2007 2:53 PM
To: [hidden email]
Subject: comparing regression coefficients over time using
cross-sectional data


Dear list members,



I know this is not strictly SPSS related but I think I could benefit from
your expertise.





I am working with survey data (six cross-sectional datasets over a period of
10 years). I want to test if the effect of certain independent variables
changes over time.

The solution that I have right now is to put all the datasets into a single
dataset, create dummy variables for "year" and run the regression models
with the variables of interest, the dummies for year, and interaction terms
between the variables and these dummies. The significance and the sign of
the interaction terms will show then if the effects of the variables do
change over time.



Can you think of any other approaches that I could use for this problem?



Thank you very much,



Claudiu Tufis