Are nearly identical QIC scores considered equal?

[titopuente]

I am using QIC (Quasilikelihood under the Independence model Criterion) to evaluate different correlation matrices in a GEE (using identical covariates in all models). I have 2 scores: 474.381 and 474.375.

QIC is in the smaller is better format, but with such minimal differences at what point would it be worthwhile to consider these two model fits to be identical? Is the "smaller is better" so strong that even a 0.006 difference is enough for one correlation structure to be clearly favored over another?

(A further bit of information, the QICc (465.957) for the larger QIC is smaller than the QICc (465.961) for the smaller QIC. As far as I understand it, QICc is only meant for choosing between covariates given a correct correlation matrix, but I thought I would include this info as well in case this might somehow be of use in a tiebreaker situation, if that is indeed what I have here.)

Thanks for looking.

[Poes, Matthew Joseph]

I typically interpret those as being practically identical, and rely on theory and sensible interpretation from that point.  I'd be interested to know what others think or do, but as I understood things, such a small difference would mean very little.  To me, it would still seem that you need to have a way to practically interpret this in the model coefficients themselves.  Maybe some additional context in that regard would help?  I mean, did the model fit slightly improve when you added a single non-significant variable?  My understanding would be that utilizing the QIC during model development/selection, such a small difference would reflect the addition of a very low effect variable, and then you really need to make a theoretical judgment?  For instance, if you find that you can improve productivity by adding widget gear A, and this difference is small, relative to all other components of the widget machine, such that efficient is only increased by 1%, but the cost of wid!
 get gear A is considerably less than the financial benefit of a 1% efficient improvement, then while not a major contributor to the model of widget efficiency, its still important and worth consideration.  Conversely, if the addition of widget gear A was significant and even showed relatively good efficiency improvement (say 10%), but the cost relative to the size is so great, and all other components are showing much greater contributions, you might argue that even considering this variable is negligible in the big picture (there appear to be more important considerations to explore first).

Matthew J Poes
Research Data Specialist
Center for Prevention Research and Development
University of Illinois
510 Devonshire Dr.
Champaign, IL 61820
Phone: [phone]
email: [hidden email]


-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of titopuente
Sent: Sunday, July 15, 2012 10:54 PM
To: [hidden email]
Subject: Are nearly identical QIC scores considered equal?

I am using QIC (Quasilikelihood under the Independence model Criterion) to evaluate different correlation matrices in a GEE (using identical covariates in all models). I have 2 scores: 474.381 and 474.375.

QIC is in the smaller is better format, but with such minimal differences at what point would it be worthwhile to consider these two model fits to be identical? Is the "smaller is better" so strong that even a 0.006 difference is enough for one correlation structure to be clearly favored over another?

(A further bit of information, the QICc (465.957) for the larger QIC is smaller than the QICc (465.961) for the smaller QIC. As far as I understand it, QICc is only meant for choosing between covariates given a correct correlation matrix, but I thought I would include this info as well in case this might somehow be of use in a tiebreaker situation, if that is indeed what I have here.)

Thanks for looking.

--
View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Are-nearly-identical-QIC-scores-considered-equal-tp5714238.html
Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[titopuente]

Thanks for your reply.

My impression was also that such a small difference would be negligible and would signify that the fits are more or less identical, but pretty much I only see model selection using QIC to be given as somewhat of an ultimate "smaller is better."

I can clarify a little on how I arrived at my situation. I fit several models, all identical in every way (the same single covariate) except for my use of different correlation structures in each model. The two QIC scores I listed above are the two smallest (from the independent and AR1 correlation structures). In both cases the covariate is significant in the GEE. However, in the case with the slightly higher QIC, the estimated marginal means contrast (the covaraite is a factor with 2 levels) the contrast is significant (p=0.036) while the model with the slightly lower QIC the contrast is (barely) non-significant (p=0.06). It seems to me that, given nearly identical QIC scores, and a term that is significant in both GEEs, I probably better off going with the model that shows a significant EMM contrast because it seems to more accurately describe the data even if it has a QIC score that is higher by 0.006.