How to compare nested models estimated via GENLINMIXED?
Locked
How to compare nested models estimated via GENLINMIXED?
 
Administrator
|
This is an appeal to Ryan, Alex, Gene, and anyone else who is more knowledgeable about GENLINMIXED than I am!
In an old thread from 2012 (http://spssx-discussion.1045642.n5.nabble.com/GenLinMixed-question-tp5715073p5715236.html), Ryan wrote: "It would be inappropriate to compare nested mixed effects models based on differences in -2 log-likelihood values derived from the GENLINMIXED procedure since, AFAIA, it employs a residual pseudo-likelihood estimation method." I'm finally getting around to a bit of reading up on GENLINMIXED. Heck, Thomas & Tabata (http://www.psypress.com/books/details/9781848729568/) say this on p. 105: --- Start Excerpt --- Categorical models with nested data structures require more complex estimation of model parameters that rely on quasilikelihood approximation or numerical integration. Integration over the random-effects distribution is necessary, which requires approximation. A number of different approaches are described in the literature (e.g., Hedeker, 2005; Hox, 2010; McCullagh & Nelder, 1989; Raudenbush & Byrk, 2002). Because the estimates are only approximate, this makes comparison of successive multilevel models with categorical outcomes more tenuous compared with the single-level case. --- End Excerpt --- First, I assume that Ryan's pseudo-likelihood = Tabata's quasilikelihood. Correct? Second, Tabata et al. describe comparison of nested models (via change in -2LL) as "more tenuous", but do not appear to absolutely prohibit doing so. Any comments? Third, how is one meant to compare nested multilevel models (with categorical outcomes) estimated via GENLINMIXED, if not by means of a likelihood ratio test on the change in -2LL? (Perhaps I'll find an answer to this last one as I keep working through the book.) Any insights you can offer will be gratefully received! ;-) Cheers, Bruce
--
Bruce Weaver bweaver@lakeheadu.ca http://sites.google.com/a/lakeheadu.ca/bweaver/ "When all else fails, RTFM." PLEASE NOTE THE FOLLOWING: 1. My Hotmail account is not monitored regularly. To send me an e-mail, please use the address shown above. 2. The SPSSX Discussion forum on Nabble is no longer linked to the SPSSX-L listserv administered by UGA (https://listserv.uga.edu/). |
Re: How to compare nested models estimated via GENLINMIXED?
|
Administrator
|
This post was updated on .
EDITED 19-Feb-2015: Fixed a typo ("usint" changed to "using").
As I suspected, Heck, Thomas & Tabata (2012) say more about this issue a bit later. This excerpt is from p. 148. --- Start Excerpt --- Readers should also keep in mind that, for multilevel models with categorical outcomes, particular estimation methods (e.g., those featuring quasilikelihood approximations) may make the use of model testing techniques that depend on the real likelihood function suspect. In particular, this affects direct comparisons of models (or single parameters) using the deviance statistic (-2LL) and the likelihood ratio test. In general, methods relying on full information ML and numerical integration will produce results that support direct comparisons of model fit using the difference in model log likelihoods. Because multilevel modeling with categorical outcomes is relatively new, it is likely that continued methodological progress will be made in providing expanded estimation options for these models. This should result in more accurate comparisons of model fit. --- End Excerpt ---
--
Bruce Weaver bweaver@lakeheadu.ca http://sites.google.com/a/lakeheadu.ca/bweaver/ "When all else fails, RTFM." PLEASE NOTE THE FOLLOWING: 1. My Hotmail account is not monitored regularly. To send me an e-mail, please use the address shown above. 2. The SPSSX Discussion forum on Nabble is no longer linked to the SPSSX-L listserv administered by UGA (https://listserv.uga.edu/). |
|
Bruce, I am sorry for not responding earlier but I've been terribly busy. I had reviewed the estimation method employed by GENLINMIXED a while back and it appeared to me at that time to not be suitable for model fit comparison using the -2LL (when the DV is not continuous) because of the absence of a true log-likelihoood. OTOH, some integral approximation methods can be used to compare GLMMs (e.g., Laplace, Adaptive Quadrature). Unfortunately, GENLINMIXED does not currently offer these estimation methods. If you are simply trying to decide whether or not a predictor should be in the model, then you could look at the p-value associated with the effect rather than perform a model fit comparison using the -2LL. This topic can get a bit mathetically heavy but I welcome further discussion, and would be happy to elaborate more to the best of my abilliy. In fact, as I think about this more, I believe a simulation experiment could actually provide a clear demonstration of why one should steer clear of comparing -2LLs derived from GENLINMIXED (assuming my understanding of the estimation method was correct), but I don't have the time right now to do so. Ryan On Thu, Feb 19, 2015 at 10:54 AM, Bruce Weaver <[hidden email]> wrote: As I suspected, Heck, Thomas & Tabata (2012) say more about this issue a bit |
Re: How to compare nested models estimated via GENLINMIXED?
|
Administrator
|
Thanks Ryan. I think a good simulation study would be very helpful to a lot of people. So I hope you find time to work on it sometime.
Bruce
--
Bruce Weaver bweaver@lakeheadu.ca http://sites.google.com/a/lakeheadu.ca/bweaver/ "When all else fails, RTFM." PLEASE NOTE THE FOLLOWING: 1. My Hotmail account is not monitored regularly. To send me an e-mail, please use the address shown above. 2. The SPSSX Discussion forum on Nabble is no longer linked to the SPSSX-L listserv administered by UGA (https://listserv.uga.edu/). |
|
|
Hi Bruce,
I read the explanation of the estimation method for GENLINMIXED again and now I remember exactly why I did not think one should compare likelihood values across nested GLMMs when the response is non-normal. More technically, one cannot compare nested models when the variance-covariance structure changes across models because as stated here: http://www-01.ibm.com/support/knowledgecenter/?lang=de#!/SSLVMB_21.0.0/com.ibm.spss.statistics.help/alg_glmm_estimation.htm ...a pseudo-response is generated using a linearization which is then used to estimate the parameters via in a linear mixed model. Why is this a problem? Well, we know that one cannot compare nested models if we were to use different response ("y") values. While the actual response values may not change across nested models, I would expect the pseudo-response values fed into the linear mixed model to change when the variance covariance structures are not the same across models due to the linearization that is performed. I could be wrong but I don't think I am based on the documentation I reread tonight. Ryan Sent from my iPhone > On Feb 19, 2015, at 2:09 PM, Bruce Weaver <[hidden email]> wrote: > > Thanks Ryan. I think a good simulation study would be very helpful to a lot > of people. So I hope you find time to work on it sometime. > > Bruce > > > > Ryan Black wrote >> Bruce, >> >> I am sorry for not responding earlier but I've been terribly busy. I had >> reviewed the estimation method employed by GENLINMIXED a while back and it >> appeared to me at that time to not be suitable for model fit comparison >> using the -2LL (when the DV is not continuous) because of the absence of a >> true log-likelihoood. >> >> OTOH, some integral approximation methods can be used to compare GLMMs >> (e.g., Laplace, Adaptive Quadrature). Unfortunately, GENLINMIXED does not >> currently offer these estimation methods. >> >> If you are simply trying to decide whether or not a predictor should be in >> the model, then you could look at the p-value associated with the effect >> rather than perform a model fit comparison using the -2LL. >> >> This topic can get a bit mathetically heavy but I welcome further >> discussion, and would be happy to elaborate more to the best of my >> abilliy. >> In fact, as I think about this more, I believe a simulation experiment >> could actually provide a clear demonstration of why one should steer clear >> of comparing -2LLs derived from GENLINMIXED (assuming my understanding of >> the estimation method was correct), but I don't have the time right now to >> do so. >> >> Ryan >> >> On Thu, Feb 19, 2015 at 10:54 AM, Bruce Weaver < > >> bruce.weaver@ > >> > >> wrote: >> >>> As I suspected, Heck, Thomas & Tabata (2012) say more about this issue a >>> bit >>> later. This excerpt is from p. 148. >>> >>> --- Start Excerpt --- >>> Readers should also keep in mind that, for multilevel models with >>> categorical outcomes, particular estimation methods (e.g., those >>> featuring >>> quasilikelihood approximations) may make the use of model testing >>> techniques >>> that depend on the real likelihood function suspect. In particular, this >>> affects direct comparisons of models (or single parameters) using the >>> deviance statistic (-2LL) and the likelihood ratio test. In general, >>> methods relying on full information ML and numerical integration will >>> produce results that support direct comparisons of model fit usint the >>> difference in model log likelihoods. Because multilevel modeling with >>> categorical outcomes is relatively new, it is likely that continued >>> methodological progress will be made in providing expanded estimation >>> options for these models. This should result in more accurate >>> comparisons >>> of model fit. >>> --- End Excerpt --- >>> >>> >>> >>> Bruce Weaver wrote >>>> This is an appeal to Ryan, Alex, Gene, and anyone else who is more >>>> knowledgeable about GENLINMIXED than I am! >>>> >>>> In an old thread from 2012 >>>> ( >>> http://spssx-discussion.1045642.n5.nabble.com/GenLinMixed-question-tp5715073p5715236.html >>> ), >>>> Ryan wrote: >>>> >>>> "It would be inappropriate to compare nested mixed effects models based >>> on >>>> differences in -2 log-likelihood values derived from the GENLINMIXED >>>> procedure since, AFAIA, it employs a residual pseudo-likelihood >>> estimation >>>> method." >>>> >>>> I'm finally getting around to a bit of reading up on GENLINMIXED. >>> Heck, >>>> Thomas & Tabata (http://www.psypress.com/books/details/9781848729568/) >>> say >>>> this on p. 105: >>>> >>>> --- Start Excerpt --- >>>> Categorical models with nested data structures require more complex >>>> estimation of model parameters that rely on quasilikelihood >>> approximation >>>> or numerical integration. Integration over the random-effects >>>> distribution is necessary, which requires approximation. A number of >>>> different approaches are described in the literature (e.g., Hedeker, >>> 2005; >>>> Hox, 2010; McCullagh & Nelder, 1989; Raudenbush & Byrk, 2002). Because >>>> the estimates are only approximate, this makes comparison of successive >>>> multilevel models with categorical outcomes more tenuous compared with >>> the >>>> single-level case. >>>> --- End Excerpt --- >>>> >>>> First, I assume that Ryan's pseudo-likelihood = Tabata's >>> quasilikelihood. >>>> Correct? >>>> >>>> Second, Tabata et al. describe comparison of nested models (via change >>> in >>>> -2LL) as "more tenuous", but do not appear to absolutely prohibit doing >>>> so. Any comments? >>>> >>>> Third, how is one meant to compare nested multilevel models (with >>>> categorical outcomes) estimated via GENLINMIXED, if not by means of a >>>> likelihood ratio test on the change in -2LL? (Perhaps I'll find an >>> answer >>>> to this last one as I keep working through the book.) >>>> >>>> Any insights you can offer will be gratefully received! ;-) >>>> >>>> Cheers, >>>> Bruce >>> >>> >>> >>> >>> >>> ----- >>> -- >>> Bruce Weaver > >> bweaver@ > >>> http://sites.google.com/a/lakeheadu.ca/bweaver/ >>> >>> "When all else fails, RTFM." >>> >>> NOTE: My Hotmail account is not monitored regularly. >>> To send me an e-mail, please use the address shown above. >>> >>> -- >>> View this message in context: >>> http://spssx-discussion.1045642.n5.nabble.com/How-to-compare-nested-models-estimated-via-GENLINMIXED-tp5728713p5728729.html >>> Sent from the SPSSX Discussion mailing list archive at Nabble.com. >>> >>> ===================== >>> To manage your subscription to SPSSX-L, send a message to > >> LISTSERV@.UGA > >> (not to SPSSX-L), with no body text except the >>> command. To leave the list, send the command >>> SIGNOFF SPSSX-L >>> For a list of commands to manage subscriptions, send the command >>> INFO REFCARD >> >> ===================== >> To manage your subscription to SPSSX-L, send a message to > >> LISTSERV@.UGA > >> (not to SPSSX-L), with no body text except the >> command. To leave the list, send the command >> SIGNOFF SPSSX-L >> For a list of commands to manage subscriptions, send the command >> INFO REFCARD > > > > > > ----- > -- > Bruce Weaver > [hidden email] > http://sites.google.com/a/lakeheadu.ca/bweaver/ > > "When all else fails, RTFM." > > NOTE: My Hotmail account is not monitored regularly. > To send me an e-mail, please use the address shown above. > > -- > View this message in context: http://spssx-discussion.1045642.n5.nabble.com/How-to-compare-nested-models-estimated-via-GENLINMIXED-tp5728713p5728735.html > Sent from the SPSSX Discussion mailing list archive at Nabble.com. > > ===================== > To manage your subscription to SPSSX-L, send a message to > [hidden email] (not to SPSSX-L), with no body text except the > command. To leave the list, send the command > SIGNOFF SPSSX-L > For a list of commands to manage subscriptions, send the command > INFO REFCARD ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD |
|
|
This what I actually read that led me the conclusion:
http://www-01.ibm.com/support/knowledgecenter/?lang=de#!/SSLVMB_21.0.0/com.ibm.spss.statistics.help/alg_glmm_estimation_iterations.htm Ryan Sent from my iPhone > On Feb 19, 2015, at 10:12 PM, [hidden email] wrote: > > Hi Bruce, > > I read the explanation of the estimation method for GENLINMIXED again and now I remember exactly why I did not think one should compare likelihood values across nested GLMMs when the response is non-normal. > > More technically, one cannot compare nested models when the variance-covariance structure changes across models because as stated here: > > http://www-01.ibm.com/support/knowledgecenter/?lang=de#!/SSLVMB_21.0.0/com.ibm.spss.statistics.help/alg_glmm_estimation.htm > > ...a pseudo-response is generated using a linearization which is then used to estimate the parameters via in a linear mixed model. > > Why is this a problem? > > Well, we know that one cannot compare nested models if we were to use different response ("y") values. > > While the actual response values may not change across nested models, I would expect the pseudo-response values fed into the linear mixed model to change when the variance covariance structures are not the same across models due to the linearization that is performed. I could be wrong but I don't think I am based on the documentation I reread tonight. > > Ryan > > Sent from my iPhone > >> On Feb 19, 2015, at 2:09 PM, Bruce Weaver <[hidden email]> wrote: >> >> Thanks Ryan. I think a good simulation study would be very helpful to a lot >> of people. So I hope you find time to work on it sometime. >> >> Bruce >> >> >> >> Ryan Black wrote >>> Bruce, >>> >>> I am sorry for not responding earlier but I've been terribly busy. I had >>> reviewed the estimation method employed by GENLINMIXED a while back and it >>> appeared to me at that time to not be suitable for model fit comparison >>> using the -2LL (when the DV is not continuous) because of the absence of a >>> true log-likelihoood. >>> >>> OTOH, some integral approximation methods can be used to compare GLMMs >>> (e.g., Laplace, Adaptive Quadrature). Unfortunately, GENLINMIXED does not >>> currently offer these estimation methods. >>> >>> If you are simply trying to decide whether or not a predictor should be in >>> the model, then you could look at the p-value associated with the effect >>> rather than perform a model fit comparison using the -2LL. >>> >>> This topic can get a bit mathetically heavy but I welcome further >>> discussion, and would be happy to elaborate more to the best of my >>> abilliy. >>> In fact, as I think about this more, I believe a simulation experiment >>> could actually provide a clear demonstration of why one should steer clear >>> of comparing -2LLs derived from GENLINMIXED (assuming my understanding of >>> the estimation method was correct), but I don't have the time right now to >>> do so. >>> >>> Ryan >>> >>> On Thu, Feb 19, 2015 at 10:54 AM, Bruce Weaver < >> >>> bruce.weaver@ >> >>> > >>> wrote: >>> >>>> As I suspected, Heck, Thomas & Tabata (2012) say more about this issue a >>>> bit >>>> later. This excerpt is from p. 148. >>>> >>>> --- Start Excerpt --- >>>> Readers should also keep in mind that, for multilevel models with >>>> categorical outcomes, particular estimation methods (e.g., those >>>> featuring >>>> quasilikelihood approximations) may make the use of model testing >>>> techniques >>>> that depend on the real likelihood function suspect. In particular, this >>>> affects direct comparisons of models (or single parameters) using the >>>> deviance statistic (-2LL) and the likelihood ratio test. In general, >>>> methods relying on full information ML and numerical integration will >>>> produce results that support direct comparisons of model fit usint the >>>> difference in model log likelihoods. Because multilevel modeling with >>>> categorical outcomes is relatively new, it is likely that continued >>>> methodological progress will be made in providing expanded estimation >>>> options for these models. This should result in more accurate >>>> comparisons >>>> of model fit. >>>> --- End Excerpt --- >>>> >>>> >>>> >>>> Bruce Weaver wrote >>>>> This is an appeal to Ryan, Alex, Gene, and anyone else who is more >>>>> knowledgeable about GENLINMIXED than I am! >>>>> >>>>> In an old thread from 2012 >>>>> ( >>>> http://spssx-discussion.1045642.n5.nabble.com/GenLinMixed-question-tp5715073p5715236.html >>>> ), >>>>> Ryan wrote: >>>>> >>>>> "It would be inappropriate to compare nested mixed effects models based >>>> on >>>>> differences in -2 log-likelihood values derived from the GENLINMIXED >>>>> procedure since, AFAIA, it employs a residual pseudo-likelihood >>>> estimation >>>>> method." >>>>> >>>>> I'm finally getting around to a bit of reading up on GENLINMIXED. >>>> Heck, >>>>> Thomas & Tabata (http://www.psypress.com/books/details/9781848729568/) >>>> say >>>>> this on p. 105: >>>>> >>>>> --- Start Excerpt --- >>>>> Categorical models with nested data structures require more complex >>>>> estimation of model parameters that rely on quasilikelihood >>>> approximation >>>>> or numerical integration. Integration over the random-effects >>>>> distribution is necessary, which requires approximation. A number of >>>>> different approaches are described in the literature (e.g., Hedeker, >>>> 2005; >>>>> Hox, 2010; McCullagh & Nelder, 1989; Raudenbush & Byrk, 2002). Because >>>>> the estimates are only approximate, this makes comparison of successive >>>>> multilevel models with categorical outcomes more tenuous compared with >>>> the >>>>> single-level case. >>>>> --- End Excerpt --- >>>>> >>>>> First, I assume that Ryan's pseudo-likelihood = Tabata's >>>> quasilikelihood. >>>>> Correct? >>>>> >>>>> Second, Tabata et al. describe comparison of nested models (via change >>>> in >>>>> -2LL) as "more tenuous", but do not appear to absolutely prohibit doing >>>>> so. Any comments? >>>>> >>>>> Third, how is one meant to compare nested multilevel models (with >>>>> categorical outcomes) estimated via GENLINMIXED, if not by means of a >>>>> likelihood ratio test on the change in -2LL? (Perhaps I'll find an >>>> answer >>>>> to this last one as I keep working through the book.) >>>>> >>>>> Any insights you can offer will be gratefully received! ;-) >>>>> >>>>> Cheers, >>>>> Bruce >>>> >>>> >>>> >>>> >>>> >>>> ----- >>>> -- >>>> Bruce Weaver >> >>> bweaver@ >> >>>> http://sites.google.com/a/lakeheadu.ca/bweaver/ >>>> >>>> "When all else fails, RTFM." >>>> >>>> NOTE: My Hotmail account is not monitored regularly. >>>> To send me an e-mail, please use the address shown above. >>>> >>>> -- >>>> View this message in context: >>>> http://spssx-discussion.1045642.n5.nabble.com/How-to-compare-nested-models-estimated-via-GENLINMIXED-tp5728713p5728729.html >>>> Sent from the SPSSX Discussion mailing list archive at Nabble.com. >>>> >>>> ===================== >>>> To manage your subscription to SPSSX-L, send a message to >> >>> LISTSERV@.UGA >> >>> (not to SPSSX-L), with no body text except the >>>> command. To leave the list, send the command >>>> SIGNOFF SPSSX-L >>>> For a list of commands to manage subscriptions, send the command >>>> INFO REFCARD >>> >>> ===================== >>> To manage your subscription to SPSSX-L, send a message to >> >>> LISTSERV@.UGA >> >>> (not to SPSSX-L), with no body text except the >>> command. To leave the list, send the command >>> SIGNOFF SPSSX-L >>> For a list of commands to manage subscriptions, send the command >>> INFO REFCARD >> >> >> >> >> >> ----- >> -- >> Bruce Weaver >> [hidden email] >> http://sites.google.com/a/lakeheadu.ca/bweaver/ >> >> "When all else fails, RTFM." >> >> NOTE: My Hotmail account is not monitored regularly. >> To send me an e-mail, please use the address shown above. >> >> -- >> View this message in context: http://spssx-discussion.1045642.n5.nabble.com/How-to-compare-nested-models-estimated-via-GENLINMIXED-tp5728713p5728735.html >> Sent from the SPSSX Discussion mailing list archive at Nabble.com. >> >> ===================== >> To manage your subscription to SPSSX-L, send a message to >> [hidden email] (not to SPSSX-L), with no body text except the >> command. To leave the list, send the command >> SIGNOFF SPSSX-L >> For a list of commands to manage subscriptions, send the command >> INFO REFCARD ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD |
«
Return to SPSSX Discussion
|
1 view|%1 views
| Free forum by Nabble | Edit this page |