Type I error adjustment in Regression Analysis

[Tanja Gabriele Baudson]

Hi all,

to predict a person's score, I am using the person's score from one
year before plus a second variable (here: teacher expectations at t1).
As I am examining several outcomes (e.g., on GPA, self-concept, the
child's social integration, etc.), I am performing several regression
analyses on the same dataset. Is it necessary to adjust the
significance level in order not to misinterpret the importance of the
standardized regression weights and to account for the fact that I am
examining several hypotheses? I'd say an adjustment is not necessary,
as the hypotheses are independent of each other, but I'd be glad for
some information that is more reliable than my gut feeling ;)

Thanks very much in advance!
Tanya


--
Tanja Gabriele Baudson
Universität Trier
FB I Psychologie
Hochbegabtenforschung und -förderung
54286 Trier
Fon 0651/201-4558
Fax [phone]
Email [hidden email]
Web http://www.uni-trier.de/index.php?id=9492

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Swank, Paul R]

I would not use standardized regression weights to judge the importance of predictors. They depend on the model being correct, the data behaving nicely (normality and outliers), and on the reliability and validity with which the predictors are measured. When all the DVs are independent the chances of at least one type I error increases as the number of DVs increases, based on a binomial distribution, ie. P(at least one spurious finding by chance) = 1 - (1-alpha)**k where k = number of tests done. If you want to be more sure that the result is not a chance one, then correcting the error rate is suggested. The Benjamini-Hochberg procedure tends to be less conservative than the Bonferroni. If, however, you are more comfortable with making a type I error than a type II error, then test each at the nominal alpha level.

Dr. Paul R. Swank,
Professor and Director of Research
Children's Learning Institute
University of Texas Health Science Center-Houston


-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Tanja Gabriele Baudson
Sent: Thursday, April 14, 2011 6:50 AM
To: [hidden email]
Subject: Type I error adjustment in Regression Analysis

Hi all,

to predict a person's score, I am using the person's score from one
year before plus a second variable (here: teacher expectations at t1).
As I am examining several outcomes (e.g., on GPA, self-concept, the
child's social integration, etc.), I am performing several regression
analyses on the same dataset. Is it necessary to adjust the
significance level in order not to misinterpret the importance of the
standardized regression weights and to account for the fact that I am
examining several hypotheses? I'd say an adjustment is not necessary,
as the hypotheses are independent of each other, but I'd be glad for
some information that is more reliable than my gut feeling ;)

Thanks very much in advance!
Tanya


--
Tanja Gabriele Baudson
Universität Trier
FB I Psychologie
Hochbegabtenforschung und -förderung
54286 Trier
Fon 0651/201-4558
Fax [phone]
Email [hidden email]
Web http://www.uni-trier.de/index.php?id=9492

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Bruce Weaver]

I share Paul's reservations about the standardized regression coefficients.  Here are some notes I wrote after reading John Fox's comments about this in the book named below.

--- start of notes ---

In his book "Applied Regression Analysis and Generalized Linear Models" (2008, Sage), John Fox is very cautious about the use of standardized regression coefficients.  He gives this interesting example.  When two variables are measured on the same scale (e.g.,years of education, and years of employment), then relative impact of the two can be compared directly.  But suppose those two variables differ substantially in the amount of spread.  In that case, comparison of the standardized regression coefficients would likely yield a very different story than comparison of the raw regression coefficients.  Fox then says:

"If expressing coefficients relative to a measure of spread potentially distorts their comparison when two explanatory variables are commensurable [i.e., measured on the same scale], then why should the procedure magically allow us to compare coefficients [for variables] that are measured in different units?" (p. 95)

Good question!

A page later, Fox adds the following:

"A common misuse of standardized coefficients is to employ them to make comparisons of the effects of the same explanatory variable in two or more samples drawn from different populations.  If the explanatory variable in question has different spreads in these samples, then spurious differences between coefficients may result, even when _unstandardized_ coefficients are similar; on the other hand, differences in unstandardized coefficients can be masked by compensating differences in dispersion." (p. 96)

And finally, this comment on whether or not Y has to be standardized:

"The usual practice standardizes the response variable as well, but this is an inessential element of the computation of standardized coefficients, because the _relative_ size of the slope coefficients does not change when Y is rescaled." (p. 95)

--- end of notes ---

Regarding whether you need to adjust alpha or not, I think it depends on where your study falls on the continuum with "strictly exploratory" at one end and "strictly confirmatory" at the other end.  Bender and Lange (2001) argue that "in exploratory studies, in which data are collected with an objective but not with a prespecified key hypothesis, multiple test adjustments are not strictly required".  But they then immediately acknowledge that not everyone agrees with them!  

Your question also reminded me of one of the articles in the CMAJ series "Basic Statistics for Clinicians".  In the article on hypothesis testing, the authors urge caution when there are multiple outcomes, and suggest a couple of strategies for dealing with the multiple testing problem.  One option is to use a Bonferroni correction.  But another one is to identify ahead of time one or two primary outcomes, and treat the others as secondary.  Results from the primary outcome(s) are treated as hypothesis-confirming, whereas those from the secondary outcomes are treated as hypothesis-generating.

Refs:

Bender & Lange:  Journal of Clinical Epidemiology 54 (2001) 343–349

Basic Statistics for Clinicians - Hypothesis Testing:
  http://collection.nlc-bnc.ca/100/201/300/cdn_medical_association/cmaj/vol-152/0027.htm

Oh yes, and don't forget the example of recording neural activity in a post-mortem Atlantic salmon.  ;-)

  http://prefrontal.org/files/posters/Bennett-Salmon-2009.pdf

Swank, Paul R wrote

I would not use standardized regression weights to judge the importance of predictors. They depend on the model being correct, the data behaving nicely (normality and outliers), and on the reliability and validity with which the predictors are measured. When all the DVs are independent the chances of at least one type I error increases as the number of DVs increases, based on a binomial distribution, ie. P(at least one spurious finding by chance) = 1 - (1-alpha)**k where k = number of tests done. If you want to be more sure that the result is not a chance one, then correcting the error rate is suggested. The Benjamini-Hochberg procedure tends to be less conservative than the Bonferroni. If, however, you are more comfortable with making a type I error than a type II error, then test each at the nominal alpha level.

Dr. Paul R. Swank,
Professor and Director of Research
Children's Learning Institute
University of Texas Health Science Center-Houston


-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Tanja Gabriele Baudson
Sent: Thursday, April 14, 2011 6:50 AM
To: [hidden email]
Subject: Type I error adjustment in Regression Analysis

Hi all,

to predict a person's score, I am using the person's score from one
year before plus a second variable (here: teacher expectations at t1).
As I am examining several outcomes (e.g., on GPA, self-concept, the
child's social integration, etc.), I am performing several regression
analyses on the same dataset. Is it necessary to adjust the
significance level in order not to misinterpret the importance of the
standardized regression weights and to account for the fact that I am
examining several hypotheses? I'd say an adjustment is not necessary,
as the hypotheses are independent of each other, but I'd be glad for
some information that is more reliable than my gut feeling ;)

Thanks very much in advance!
Tanya


--
Tanja Gabriele Baudson
Universität Trier
FB I Psychologie
Hochbegabtenforschung und -förderung
54286 Trier
Fon 0651/201-4558
Fax [phone]
Email [hidden email]
Web http://www.uni-trier.de/index.php?id=9492

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD