Multilevel model results with multiple imputation

[Bruce Weaver]

I have a couple questions about the output from MIXED when it is used in conjunction with multiple imputation.  

The first question concerns the table of fixed effects estimates.  I have 5 imputed data sets, so the fixed effects estimates table shows estimates for the original data set, for each of the 5 imputations, and pooled estimates (pooling over the 5 imputed data sets).  For the pooled estimates, no degrees of freedom are given, but a t-test result (with p-value) is still reported for each estimate.  Surely the df are needed to work out the p-value, are they not?  And if they are needed, why are they not reported in the output?  

Second, the Information Criteria table does not give pooled estimates.  I am wondering what is recommended for comparing nested models in this case.  I.e., without the multiple imputation, one uses the change in -2LL from one model to the next (chi-square test with df = difference in the number of parameters).  When using pooled estimates, can one simply compute the average deviance (-2LL) to use in the likelihood ratio test, for example?  

As a sidebar, I know that the sum of independent chi-squares is itself a chi-square (with df = the sum of the df), but I don't think the -2LL values from the 5 imputed data sets could be considered independent.

Thanks for any direction you can offer.

[Joost van Ginkel]

Dear Bruce,

The degrees of freedom do exist for the pooled results in Mixed, and I'm
sure SPSS has computed them for determining the p-values. I don't know
why SPSS doesn't report them in the output; for a simple two-sample
t-test SPSS does report the degrees of freedom (and for mixed models,
they are computed in the same way). To my knowledge no rules are
available for combining information criteria. However, Schafer (1997,
pp. 116-118) gives some rules for combining likelihood-ratio statistics.
Perhaps you can find a solution there. The complete reference is:

Schafer, J. L. (1997). Analysis of incomplete multivariate data. London:
Chapman & Hall.

A remaining problem is of course that SPSS still does not report
combined information criteria so you would still have to compute them
manually.

Best regards,

Joost van Ginkel


Joost R. Van Ginkel, PhD
Leiden University
Faculty of Social and Behavioural Sciences
Data Theory Group
PO Box 9555
2300 RB Leiden
The Netherlands
Tel: [phone]
Fax: [phone]


-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of
Bruce Weaver
Sent: 12 March 2010 14:35
To: [hidden email]
Subject: Multilevel model results with multiple imputation

I have a couple questions about the output from MIXED when it is used in
conjunction with multiple imputation.

The first question concerns the table of fixed effects estimates.  I
have 5 imputed data sets, so the fixed effects estimates table shows
estimates for the original data set, for each of the 5 imputations, and
pooled estimates (pooling over the 5 imputed data sets).  For the pooled
estimates, no degrees of freedom are given, but a t-test result (with
p-value) is still reported for each estimate.  Surely the df are needed
to work out the p-value, are they not?  And if they are needed, why are
they not reported in the output?

Second, the Information Criteria table does not give pooled estimates.
I am wondering what is recommended for comparing nested models in this
case.
I.e., without the multiple imputation, one uses the change in -2LL from
one model to the next (chi-square test with df = difference in the
number of parameters).  When using pooled estimates, can one simply
compute the average deviance (-2LL) to use in the likelihood ratio test,
for example?

As a sidebar, I know that the sum of independent chi-squares is itself a
chi-square (with df = the sum of the df), but I don't think the -2LL
values from the 5 imputed data sets could be considered independent.

Thanks for any direction you can offer.



-----
--
Bruce Weaver
[hidden email]
http://sites.google.com/a/lakeheadu.ca/bweaver/
"When all else fails, RTFM."

NOTE:  My Hotmail account is not monitored regularly.
To send me an e-mail, please use the address shown above.
--
View this message in context:
http://old.nabble.com/Multilevel-model-results-with-multiple-imputation-
tp27874740p27874740.html
Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command SIGNOFF SPSSX-L For a list
of commands to manage subscriptions, send the command INFO REFCARD

**********************************************************************
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager.
**********************************************************************

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Bruce Weaver]

Thanks for that, Joost.  I am able to see those pages from Schafer's book at Google Books.  Will look them over and see if I can figure out a way to proceed.

Cheers,
Bruce

Ginkel, Joost van wrote

Dear Bruce,

The degrees of freedom do exist for the pooled results in Mixed, and I'm
sure SPSS has computed them for determining the p-values. I don't know
why SPSS doesn't report them in the output; for a simple two-sample
t-test SPSS does report the degrees of freedom (and for mixed models,
they are computed in the same way). To my knowledge no rules are
available for combining information criteria. However, Schafer (1997,
pp. 116-118) gives some rules for combining likelihood-ratio statistics.
Perhaps you can find a solution there. The complete reference is:

Schafer, J. L. (1997). Analysis of incomplete multivariate data. London:
Chapman & Hall.

A remaining problem is of course that SPSS still does not report
combined information criteria so you would still have to compute them
manually.

Best regards,

Joost van Ginkel


Joost R. Van Ginkel, PhD
Leiden University
Faculty of Social and Behavioural Sciences
Data Theory Group
PO Box 9555
2300 RB Leiden
The Netherlands
Tel: [phone]
Fax: [phone]


-----Original Message-----
From: SPSSX(r) Discussion [mailto:[e-mail]] On Behalf Of
Bruce Weaver
Sent: 12 March 2010 14:35
To: [e-mail]
Subject: Multilevel model results with multiple imputation

I have a couple questions about the output from MIXED when it is used in
conjunction with multiple imputation.

The first question concerns the table of fixed effects estimates.  I
have 5 imputed data sets, so the fixed effects estimates table shows
estimates for the original data set, for each of the 5 imputations, and
pooled estimates (pooling over the 5 imputed data sets).  For the pooled
estimates, no degrees of freedom are given, but a t-test result (with
p-value) is still reported for each estimate.  Surely the df are needed
to work out the p-value, are they not?  And if they are needed, why are
they not reported in the output?

Second, the Information Criteria table does not give pooled estimates.
I am wondering what is recommended for comparing nested models in this
case.
I.e., without the multiple imputation, one uses the change in -2LL from
one model to the next (chi-square test with df = difference in the
number of parameters).  When using pooled estimates, can one simply
compute the average deviance (-2LL) to use in the likelihood ratio test,
for example?

As a sidebar, I know that the sum of independent chi-squares is itself a
chi-square (with df = the sum of the df), but I don't think the -2LL
values from the 5 imputed data sets could be considered independent.

Thanks for any direction you can offer.



-----
--
Bruce Weaver
[e-mail]
http://sites.google.com/a/lakeheadu.ca/bweaver/
"When all else fails, RTFM."

NOTE:  My Hotmail account is not monitored regularly.
To send me an e-mail, please use the address shown above.
--
View this message in context:
http://old.nabble.com/Multilevel-model-results-with-multiple-imputation-
tp27874740p27874740.html
Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to
[e-mail] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command SIGNOFF SPSSX-L For a list
of commands to manage subscriptions, send the command INFO REFCARD

**********************************************************************
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager.
**********************************************************************

=====================
To manage your subscription to SPSSX-L, send a message to
[e-mail] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[peter link]

Bruce -

This doesnt answer your question, but there is a Multilevel Listserv that
should be of better assistance.  You can find more details here.

https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=multilevel

Enjoy.

Peter Link
VA San Diego Healthcare System

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]]On Behalf Of
Bruce Weaver
Sent: Friday, March 12, 2010 8:32 AM
To: [hidden email]
Subject: Re: [SPSSX-L] Multilevel model results with multiple imputation


Thanks for that, Joost.  I am able to see those pages from Schafer's book at
Google Books.  Will look them over and see if I can figure out a way to
proceed.

Cheers,
Bruce



Ginkel, Joost van wrote:

>
> Dear Bruce,
>
> The degrees of freedom do exist for the pooled results in Mixed, and I'm
> sure SPSS has computed them for determining the p-values. I don't know
> why SPSS doesn't report them in the output; for a simple two-sample
> t-test SPSS does report the degrees of freedom (and for mixed models,
> they are computed in the same way). To my knowledge no rules are
> available for combining information criteria. However, Schafer (1997,
> pp. 116-118) gives some rules for combining likelihood-ratio statistics.
> Perhaps you can find a solution there. The complete reference is:
>
> Schafer, J. L. (1997). Analysis of incomplete multivariate data. London:
> Chapman & Hall.
>
> A remaining problem is of course that SPSS still does not report
> combined information criteria so you would still have to compute them
> manually.
>
> Best regards,
>
> Joost van Ginkel
>
>
> Joost R. Van Ginkel, PhD
> Leiden University
> Faculty of Social and Behavioural Sciences
> Data Theory Group
> PO Box 9555
> 2300 RB Leiden
> The Netherlands
> Tel: [phone]
> Fax: [phone]
>
>
> -----Original Message-----
> From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of
> Bruce Weaver
> Sent: 12 March 2010 14:35
> To: [hidden email]
> Subject: Multilevel model results with multiple imputation
>
> I have a couple questions about the output from MIXED when it is used in
> conjunction with multiple imputation.
>
> The first question concerns the table of fixed effects estimates.  I
> have 5 imputed data sets, so the fixed effects estimates table shows
> estimates for the original data set, for each of the 5 imputations, and
> pooled estimates (pooling over the 5 imputed data sets).  For the pooled
> estimates, no degrees of freedom are given, but a t-test result (with
> p-value) is still reported for each estimate.  Surely the df are needed
> to work out the p-value, are they not?  And if they are needed, why are
> they not reported in the output?
>
> Second, the Information Criteria table does not give pooled estimates.
> I am wondering what is recommended for comparing nested models in this
> case.
> I.e., without the multiple imputation, one uses the change in -2LL from
> one model to the next (chi-square test with df = difference in the
> number of parameters).  When using pooled estimates, can one simply
> compute the average deviance (-2LL) to use in the likelihood ratio test,
> for example?
>
> As a sidebar, I know that the sum of independent chi-squares is itself a
> chi-square (with df = the sum of the df), but I don't think the -2LL
> values from the 5 imputed data sets could be considered independent.
>
> Thanks for any direction you can offer.
>
>
>
> -----
> --
> Bruce Weaver
> [hidden email]
> http://sites.google.com/a/lakeheadu.ca/bweaver/
> "When all else fails, RTFM."
>
> NOTE:  My Hotmail account is not monitored regularly.
> To send me an e-mail, please use the address shown above.
> --
> View this message in context:
> http://old.nabble.com/Multilevel-model-results-with-multiple-imputation-
> tp27874740p27874740.html
> Sent from the SPSSX Discussion mailing list archive at Nabble.com.
>
> =====================
> To manage your subscription to SPSSX-L, send a message to
> [hidden email] (not to SPSSX-L), with no body text except the
> command. To leave the list, send the command SIGNOFF SPSSX-L For a list
> of commands to manage subscriptions, send the command INFO REFCARD
>
> **********************************************************************
> This email and any files transmitted with it are confidential and
> intended solely for the use of the individual or entity to whom they
> are addressed. If you have received this email in error please notify
> the system manager.
> **********************************************************************
>
> =====================
> To manage your subscription to SPSSX-L, send a message to
> [hidden email] (not to SPSSX-L), with no body text except the
> command. To leave the list, send the command
> SIGNOFF SPSSX-L
> For a list of commands to manage subscriptions, send the command
> INFO REFCARD
>
>

-----
--
Bruce Weaver
[hidden email]
http://sites.google.com/a/lakeheadu.ca/bweaver/
"When all else fails, RTFM."

NOTE:  My Hotmail account is not monitored regularly.
To send me an e-mail, please use the address shown above.
--
View this message in context:
http://old.nabble.com/Multilevel-model-results-with-multiple-imputation-tp27
874740p27879636.html
Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Bruce Weaver]

Thanks Peter.  I am aware of that list, and was even a member for a while.  I just don't like mailing lists, not to put too fine a point on it.  I prefer the newsgroup format--largely because my Inbox does not get cluttered with junk.  The only reason I participate in this list (SPSSX-L) is that I can read & post via Nabble.com.  But, I may have to overcome my distaste for mailing lists and join again, at least for a while.  ;-)

Bruce

peter link wrote

Bruce -

This doesnt answer your question, but there is a Multilevel Listserv that
should be of better assistance.  You can find more details here.

https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=multilevel

Enjoy.

Peter Link
VA San Diego Healthcare System

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[e-mail]]On Behalf Of
Bruce Weaver
Sent: Friday, March 12, 2010 8:32 AM
To: [e-mail]
Subject: Re: [SPSSX-L] Multilevel model results with multiple imputation


Thanks for that, Joost.  I am able to see those pages from Schafer's book at
Google Books.  Will look them over and see if I can figure out a way to
proceed.

Cheers,
Bruce



Ginkel, Joost van wrote:
>
> Dear Bruce,
>
> The degrees of freedom do exist for the pooled results in Mixed, and I'm
> sure SPSS has computed them for determining the p-values. I don't know
> why SPSS doesn't report them in the output; for a simple two-sample
> t-test SPSS does report the degrees of freedom (and for mixed models,
> they are computed in the same way). To my knowledge no rules are
> available for combining information criteria. However, Schafer (1997,
> pp. 116-118) gives some rules for combining likelihood-ratio statistics.
> Perhaps you can find a solution there. The complete reference is:
>
> Schafer, J. L. (1997). Analysis of incomplete multivariate data. London:
> Chapman & Hall.
>
> A remaining problem is of course that SPSS still does not report
> combined information criteria so you would still have to compute them
> manually.
>
> Best regards,
>
> Joost van Ginkel
>
>
> Joost R. Van Ginkel, PhD
> Leiden University
> Faculty of Social and Behavioural Sciences
> Data Theory Group
> PO Box 9555
> 2300 RB Leiden
> The Netherlands
> Tel: [phone]
> Fax: [phone]
>
>
> -----Original Message-----
> From: SPSSX(r) Discussion [mailto:[e-mail]] On Behalf Of
> Bruce Weaver
> Sent: 12 March 2010 14:35
> To: [e-mail]
> Subject: Multilevel model results with multiple imputation
>
> I have a couple questions about the output from MIXED when it is used in
> conjunction with multiple imputation.
>
> The first question concerns the table of fixed effects estimates.  I
> have 5 imputed data sets, so the fixed effects estimates table shows
> estimates for the original data set, for each of the 5 imputations, and
> pooled estimates (pooling over the 5 imputed data sets).  For the pooled
> estimates, no degrees of freedom are given, but a t-test result (with
> p-value) is still reported for each estimate.  Surely the df are needed
> to work out the p-value, are they not?  And if they are needed, why are
> they not reported in the output?
>
> Second, the Information Criteria table does not give pooled estimates.
> I am wondering what is recommended for comparing nested models in this
> case.
> I.e., without the multiple imputation, one uses the change in -2LL from
> one model to the next (chi-square test with df = difference in the
> number of parameters).  When using pooled estimates, can one simply
> compute the average deviance (-2LL) to use in the likelihood ratio test,
> for example?
>
> As a sidebar, I know that the sum of independent chi-squares is itself a
> chi-square (with df = the sum of the df), but I don't think the -2LL
> values from the 5 imputed data sets could be considered independent.
>
> Thanks for any direction you can offer.
>
>
>
> -----
> --
> Bruce Weaver
> [e-mail]
> http://sites.google.com/a/lakeheadu.ca/bweaver/
> "When all else fails, RTFM."
>
> NOTE:  My Hotmail account is not monitored regularly.
> To send me an e-mail, please use the address shown above.
> --
> View this message in context:
> http://old.nabble.com/Multilevel-model-results-with-multiple-imputation-
> tp27874740p27874740.html
> Sent from the SPSSX Discussion mailing list archive at Nabble.com.
>
> =====================
> To manage your subscription to SPSSX-L, send a message to
> [e-mail] (not to SPSSX-L), with no body text except the
> command. To leave the list, send the command SIGNOFF SPSSX-L For a list
> of commands to manage subscriptions, send the command INFO REFCARD
>
> **********************************************************************
> This email and any files transmitted with it are confidential and
> intended solely for the use of the individual or entity to whom they
> are addressed. If you have received this email in error please notify
> the system manager.
> **********************************************************************
>
> =====================
> To manage your subscription to SPSSX-L, send a message to
> [e-mail] (not to SPSSX-L), with no body text except the
> command. To leave the list, send the command
> SIGNOFF SPSSX-L
> For a list of commands to manage subscriptions, send the command
> INFO REFCARD
>
>


-----
--
Bruce Weaver
[e-mail]
http://sites.google.com/a/lakeheadu.ca/bweaver/
"When all else fails, RTFM."

NOTE:  My Hotmail account is not monitored regularly.
To send me an e-mail, please use the address shown above.
--
View this message in context:
http://old.nabble.com/Multilevel-model-results-with-multiple-imputation-tp27
874740p27879636.html
Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to
[e-mail] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[e-mail] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD