Multinomial regression and weighting problem

[n90]

Hi,

I'm conducting a multinomial logistic regression on some British Crime Survey data. For this analysis I need to weight the dataset in order to account for sampling biases.

The problem I'm having is when I apply the correct weight and run the multinomial regression, my model fitting criteria for the model fitting information and likelihood ratio test come up with a blank (...). As a consequence, everything in my model is coming out as significant, which I know is wrong.

I have run the model before using a different weight, which I now know is wrong for use in my particular analysis, so I believe there is a problem with this new weighting, rather than any of the other inputs into the model.

Has anybody else had experience of this? If it's a problem with the weighting is there anything I can do to fix it?

If it helps the unweighted sample size is 46,484 and weighted is 43,974,190.

Any help would be very much appreciated!

[Bruce Weaver]

Please post your syntax.  At the moment, it's not clear, for example, whether you are using CSLOGISTIC or NOMREG in conjunction with WEIGHT CASES (or something else).  

HTH.

n90 wrote

Hi,

I'm conducting a multinomial logistic regression on some British Crime Survey data. For this analysis I need to weight the dataset in order to account for sampling biases.

The problem I'm having is when I apply the correct weight and run the multinomial regression, my model fitting criteria for the model fitting information and likelihood ratio test come up with a blank (...). As a consequence, everything in my model is coming out as significant, which I know is wrong.

I have run the model before using a different weight, which I now know is wrong for use in my particular analysis, so I believe there is a problem with this new weighting, rather than any of the other inputs into the model.

Has anybody else had experience of this? If it's a problem with the weighting is there anything I can do to fix it?

If it helps the unweighted sample size is 46,484 and weighted is 43,974,190.

Any help would be very much appreciated!

[n90]

Hi Bruce,

I'm not a particularly advanced SPSS user so I'm not typing the syntax commands I'm just using the menus. Here's the output I get though:

DATASET NAME DataSet1 WINDOW=FRONT.
WEIGHT BY IndivWgt.
NOMREG walkdark1 (BASE=FIRST ORDER=ASCENDING) BY Female income_lessthan10k income10k_19k income20k_29k income30k_39k income40k_49k income50kplus Religious not_employed police_confidence victim
  /CRITERIA CIN(95) DELTA(0) MXITER(100) MXSTEP(5) CHKSEP(20) LCONVERGE(0) PCONVERGE(0.000001) SINGULAR(0.00000001)
  /MODEL
  /STEPWISE=PIN(.05) POUT(0.1) MINEFFECT(0) RULE(SINGLE) ENTRYMETHOD(LR) REMOVALMETHOD(LR)
  /INTERCEPT=INCLUDE
  /PRINT=CLASSTABLE FIT PARAMETER SUMMARY LRT CPS STEP MFI IC.

[Rich Ulrich]

[Original post has not appeared in my SPSS mail]

If the testing is apparently being performed on N= 46 million,
and that is what accounts for the results that are not useful,
then the obvious quick-fix  is to divide all the weights by 946
so that the testing reflects the (overall) original N of the analysis. 

Whether that is what you ought to do is trickier.  I have never
liked having tests that are done on weighted data.  (And the
more change in cell N's that you introduce by weighting, the more
questionable the tests are.) 

--
Rich Ulrich

> Date: Sun, 2 Dec 2012 06:42:25 -0800

> From: [hidden email]
> Subject: Re: Multinomial regression and weighting problem
> To: [hidden email]
>
> Please post your syntax. At the moment, it's not clear, for example, whether
> you are using CSLOGISTIC or NOMREG in conjunction with WEIGHT CASES (or
> something else).
>
> HTH.
>
>
> n90 wrote
> > Hi,
> >
> > I'm conducting a multinomial logistic regression on some British Crime
> > Survey data. For this analysis I need to weight the dataset in order to
> > account for sampling biases.
> >
> > The problem I'm having is when I apply the correct weight and run the
> > multinomial regression, my model fitting criteria for the model fitting
> > information and likelihood ratio test come up with a blank (...). As a
> > consequence, everything in my model is coming out as significant, which I
> > know is wrong.
> >
> > I have run the model before using a different weight, which I now know is
> > wrong for use in my particular analysis, so I believe there is a problem
> > with this new weighting, rather than any of the other inputs into the
> > model.
> >
> > Has anybody else had experience of this? If it's a problem with the
> > weighting is there anything I can do to fix it?
> >
> > If it helps the unweighted sample size is 46,484 and weighted is
> > 43,974,190.
> >
> > Any help would be very much appreciated!
>
> ...

[Bruce Weaver]

I wonder if the OP is posting to the Nabble archive without having joined SPSSX-L?  I see that original post in Nabble is still listed as "not yet accepted by the mailing list".  

Meanwhile, the OP did repsond to my post.  You can view that response here:

http://spssx-discussion.1045642.n5.nabble.com/Multinomial-regression-and-weighting-problem-tp5716582p5716584.html

HTH.

Rich Ulrich-2 wrote

[Original post has not appeared in my SPSS mail]

If the testing is apparently being performed on N= 46 million,
and that is what accounts for the results that are not useful,
then the obvious quick-fix  is to divide all the weights by 946
so that the testing reflects the (overall) original N of the analysis.  

Whether that is what you ought to do is trickier.  I have never
liked having tests that are done on weighted data.  (And the
more change in cell N's that you introduce by weighting, the more
questionable the tests are.)  

--
Rich Ulrich

> Date: Sun, 2 Dec 2012 06:42:25 -0800
> From: [hidden email]
> Subject: Re: Multinomial regression and weighting problem
> To: [hidden email]
>
> Please post your syntax.  At the moment, it's not clear, for example, whether
> you are using CSLOGISTIC or NOMREG in conjunction with WEIGHT CASES (or
> something else).
>
> HTH.
>
>
> n90 wrote
> > Hi,
> >
> > I'm conducting a multinomial logistic regression on some British Crime
> > Survey data. For this analysis I need to weight the dataset in order to
> > account for sampling biases.
> >
> > The problem I'm having is when I apply the correct weight and run the
> > multinomial regression, my model fitting criteria for the model fitting
> > information and likelihood ratio test come up with a blank (...). As a
> > consequence, everything in my model is coming out as significant, which I
> > know is wrong.
> >
> > I have run the model before using a different weight, which I now know is
> > wrong for use in my particular analysis, so I believe there is a problem
> > with this new weighting, rather than any of the other inputs into the
> > model.
> >
> > Has anybody else had experience of this? If it's a problem with the
> > weighting is there anything I can do to fix it?
> >
> > If it helps the unweighted sample size is 46,484 and weighted is
> > 43,974,190.
> >
> > Any help would be very much appreciated!
>
> ...

[n90]

Rich- thanks, what you said seems to have been confirmed by something I've found after hours trawling the british crime survey technical report archives. I've rerun using the new weights and it does all appear to be working!

Bruce- I think I have subscribed, am I posting in the wrong place? This forum has been really useful for checking things previously so I'd like to keep up to date.

[Bruce Weaver]

n90, your first two posts in this thread still show (in Nabble) as not yet accepted by the mailing list, whereas this last one was accepted.  That's a bit puzzling.

n90 wrote

Rich- thanks, what you said seems to have been confirmed by something I've found after hours trawling the british crime survey technical report archives. I've rerun using the new weights and it does all appear to be working!

Bruce- I think I have subscribed, am I posting in the wrong place? This forum has been really useful for checking things previously so I'd like to keep up to date.

[Maguin, Eugene]

I'm not denying what you describe or that the analysis might be wrong or intending to be sarcastic or snarky but ... tell me how large would the 'd' effect size have to be to be significant given an N of 44M???

I understand you want to do a weighted analysis. I'm wondering if it might be better to compute the weights so that they are proportional to the sample size so that the analysed N is 46,484 but weighted to be proportional to the population (I hope that is clear; I'm not sure it is and I apologize.) Something to check on is whether Logistic regression or Nomreg or Plum or GenLin will accept fractional weights. You will definitely have them (and you may have them now) and I'm unsure whether those procedures will accept non-integer weights.

Gene Maguin

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of n90
Sent: Sunday, December 02, 2012 8:46 AM
To: [hidden email]
Subject: Multinomial regression and weighting problem

Hi,

I'm conducting a multinomial logistic regression on some British Crime Survey data. For this analysis I need to weight the dataset in order to account for sampling biases.

The problem I'm having is when I apply the correct weight and run the multinomial regression, my model fitting criteria for the model fitting information and likelihood ratio test come up with a blank (...). As a consequence, everything in my model is coming out as significant, which I know is wrong.

I have run the model before using a different weight, which I now know is wrong for use in my particular analysis, so I believe there is a problem with this new weighting, rather than any of the other inputs into the model.

Has anybody else had experience of this? If it's a problem with the weighting is there anything I can do to fix it?

If it helps the unweighted sample size is 46,484 and weighted is 43,974,190.

Any help would be very much appreciated!



--
View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Multinomial-regression-and-weighting-problem-tp5716582.html
Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD