Posted by
Rich Ulrich on
URL: http://spssx-discussion.165.s1.nabble.com/significant-F-change-but-nonsignificant-regression-model-overall-tp4269810p4274452.html
Warning. This post is about statistics/ design rather than SPSS.
> Date: Thu, 31 Mar 2011 09:34:16 -0400
> From:
[hidden email]
> Subject: Re: significant F change, but nonsignificant regression model overall
> To:
[hidden email]
>
> On Wednesday, March 30, 2011 11:08 PM, Rich Ulrich wrote:
> >Mike Palij had written in response to Ulrich's earlier post:
> >>Rich Ulrich wrote:
> >> >Mike,
[snip]
I'm going to snip a bunch and respond to just a couple of points
on strategies of experimental design.
> Rich, where do I use the term "stepwise" in my original post or even
> refer to a process that relies on the entry or removal of variables in/out
> of a regression on the basis of some criterion that is typically used
> in stepwise procedures? Here's what I said:
Here is what I was judging by --
***** excerpted from Mike's earlier post
(3) There is a "significant increase in R^2" (F change) when the second
set of predictors was entered. This has me puzzled. It is not clear to
me why or how this could occur. ***** end of excerpt
Was this a rhetorical use of, "not clear to me"?
I took it for being a failure to grasp a simple and appropriate
model of testing.
It now seems that perhaps it was a rhetorical slur on a style of
testing that Mike would not use. I think the error is Mike's.
[snip several paragraphs]
>
> First, if you check for my name in PubMed, you'll that I have been
> involved in psychiatric research, so I'm somewhat familiar with what
> gets done there. Second, you simply do not make sense in what you
> say above. The point behind ANCOVA or the use of covariates in
> multiple regression is to reduce the error variance by identifying variables
> that are systematically related to the dependent variable and/or to
> adjust for differences among groups on the covariates -- see Howell's
> presentation on pages 598-621 in his 7th ed Stat Methods for Psych.
> I agree that if a group of covariates are used, you should report whether
> they are significant or not, but then you go on to use only those that
> are significant in the subsequent stages of analysis.
Concerning the last phrase - I do not consider *that* to be a
generally acceptable style of testing, though many people resort
to it, often justified by the lack of sufficient d.f. to support
larger analyses. It has problems in "model-building" -- which might
be remediated by sufficient cross-validation -- but I don't think it
is, at all, an acceptable strategy for dealing with a set of 4 nuisance
variables, as in this example of "testing".
I find it hard to believe that Howell's book, which has a good
reputation, would broadly endorse the notion, "Use only the significant
covariates for subsequent analyses." That strategy reeks of all
the problems known for stepwise selection.
[snip, Bruce's example]
>[ concerning the example ...]
> On the basis of ordinary R-sq the full model appears to be the better
> model but one has to remember that as the number of predictor/IVs
> increases in a regression equation, so will R-sq which is one reason
> why one focuses on the adjusted-R-sq. So, with just the two variables
> of interest, the adjusted-R-sq= .118. Add in the nonsignificant covariates
> increases the regular R-sq but REDUCES the adjusted-R-sq. This would
> have become apparent if one had left out the nonsignificant covariates
> in Method 1. Their inclusion confuses the matter and may lead one
> to think that the full model is better while in fact the reduced model
> with just 2 predictors/IVs is the best model.
As I understand the problem on hand, it was intended to be a *test*
of two variables, and not an exercise in model-building for any
predictive purposes. - I've mainly been concerned with adjusted
R^2 when considering actual prediction, not testing. - And prediction
needs much larger R^2 than any of these.
>
> I don't doubt that you may be doing things consistent with what
> others may have done in terms of analysis but think about it, what
> is the justification for it? Does this practice enlighten or confuse?
Conservative principles of testing have to assume that there
can be *problems* -- from nuisance variables, or whatever. I
think it is a mistake to confuse testing, where effects are apt
to be marginal, with model-building, where you don't have anything
if you don't have very strong effects, somewhere, at the start.
As to testing --
Mike seems to recommend that one can use multivariate test on several
potential confounding variables, and then omit them all when the overall
test on them (like this one) fails to reject. I hope that there are
not many journals that accept this strategy.
--
Rich Ulrich
=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD