Dependent is some form of deviation form ‘correct’ Want to know best form of that deviation Possibilities are: raw(observed-correct), ratio(observed/correct), ln[ratio(observed/correct)] BIC value, with smaller being better are: BIC raw(observed-correct) = 2196 BIC ratio(observed/correct) = 57.5 BIC ln[ratio(observed/correct)] = 11.4 I already knew the raw was a non-starter Now would like, for theoretical reasons, to claim that ln(ratio) is better than ratio. Obviously 11 is lower than 57, but is it ‘ENOUGH’ lower? How would one know what is a difference in BIC large enough to be useful? Or put another way what IS the distribution of the BIC All help gratefully received Best Diana Emeritus Professor Diana Kornbrot email: d.e.kornbrot@... web: http://dianakornbrot.wordpress.com/ Work Department of Psychology School of Life and Medical Sciences University of Hertfordshire College Lane, Hatfield, Hertfordshire AL10 9AB, UK voice: +44 (0) 170 728 4626 Home 19 Elmhurst Avenue London N2 0LT, UK voice: +44 (0) 208 444 2081 mobile: +44 (0) 740 318 1612 |
I'm not sure I understand what you're trying
to do below. The BIC is a measure for comparing two models, related
to the likelihood. When using likelihoods to compare two models,
you would typically do a likelihood ratio test in order to see whether
the difference in the likelihoods is due to a real difference in the models,
or to chance variation. When using the BIC, the basic theory is that
you simply compare the BIC values of the two models, because the BIC has
already accounted for the differences in model complexity.
Alex From: "Kornbrot, Diana" <[hidden email]> To: [hidden email], Date: 02/17/2013 07:55 AM Subject: Schwartz Bayesian Criterion, BIC in Mixed models Sent by: "SPSSX(r) Discussion" <[hidden email]> Have mixed model with 1 between 2 level factor; 1 repeated 2 level factor; 1 rpeated 5 level factor Dependent is some form of deviation form ‘correct’ Want to know best form of that deviation Possibilities are: raw(observed-correct), ratio(observed/correct), ln[ratio(observed/correct)] BIC value, with smaller being better are: BIC raw(observed-correct) = 2196 BIC ratio(observed/correct) = 57.5 BIC ln[ratio(observed/correct)] = 11.4 I already knew the raw was a non-starter Now would like, for theoretical reasons, to claim that ln(ratio) is better than ratio. Obviously 11 is lower than 57, but is it ‘ENOUGH’ lower? How would one know what is a difference in BIC large enough to be useful? Or put another way what IS the distribution of the BIC All help gratefully received Best Diana |
In reply to this post by Kornbrot, Diana
Here is a note from the Wikip article on BIC.
It is important to keep in mind that the BIC can be used to compare estimated models only when the numerical values of the dependent variable are identical for all estimates being compared. The models being compared need not be nested unlike the case when models are being compared using an F or likelihood ratio test. I'm certainly no expert in AIC/BIC but I'm pretty sure that the Wikip article is making a valid point, and that I'm extrapolating legitimately.... You are in a really iffy situation when you try to compare BICs with different versions of the outcome variable. The residuals of the fit differ in magnitude, and that matters. ("How" it matters is less obvious with likelihood estimation than with OLS, but it does matter for both.) John Tukey prescribed using the various power transformations computed in terms of the technical derivatives, instead of simply taking "log" or "square root", etc. Doing so, he argued, keeps the overall magnitude of the residuals of an analysis (relatively?) constant -- and you have a better shot at a fair comparison. I saw that in one of his co-authored textbooks. One criterion that probably helped you intuitively rule out the "raw" form of the outcome is the distribution of the residuals of prediction: You can see that there are enormous outliers that you *know* do not deserve all that implicit weight for shaping the fit. That is generally one standard that I try to apply. For your particular problem, I would keep in mind that the log of a ratio is often a more "natural" transformation than the ratio itself, for model building in general. -- Rich Ulrich Date: Sun, 17 Feb 2013 13:52:31 +0000 From: [hidden email] Subject: Schwartz Bayesian Criterion, BIC in Mixed models To: [hidden email] Dependent is some form of deviation form ‘correct’ Want to know best form of that deviation Possibilities are: raw(observed-correct), ratio(observed/correct), ln[ratio(observed/correct)] BIC value, with smaller being better are: BIC raw(observed-correct) = 2196 BIC ratio(observed/correct) = 57.5 BIC ln[ratio(observed/correct)] = 11.4 I already knew the raw was a non-starter Now would like, for theoretical reasons, to claim that ln(ratio) is better than ratio. Obviously 11 is lower than 57, but is it ‘ENOUGH’ lower? How would one know what is a difference in BIC large enough to be useful? Or put another way what IS the distribution of the BIC All help gratefully received ... |
By the way, to add to my comment -- The design features repeated
measures, 2x5. When there is much correlation, Repeated Measures are great for looking at which transformation gives a reasonable, equal- interval sort of error. I usually have Pre-Post, and I look to see which transformed scatterplot gives a Normal sort of ellipse. For my data, that's almost always the choice that gives the right model for the error. For a 2x5 repeated design, there are choices of what to plot. Something that includes the widest range of scores might be most useful. -- Rich Ulrich From: [hidden email] To: [hidden email]; [hidden email] Subject: RE: Schwartz Bayesian Criterion, BIC in Mixed models Date: Tue, 19 Feb 2013 13:03:59 -0500 Here is a note from the Wikip article on BIC. It is important to keep in mind that the BIC can be used to compare estimated models only when the numerical values of the dependent variable are identical for all estimates being compared. The models being compared need not be nested unlike the case when models are being compared using an F or likelihood ratio test. I'm certainly no expert in AIC/BIC but I'm pretty sure that the Wikip article is making a valid point, and that I'm extrapolating legitimately.... You are in a really iffy situation when you try to compare BICs with different versions of the outcome variable. The residuals of the fit differ in magnitude, and that matters. ("How" it matters is less obvious with likelihood estimation than with OLS, but it does matter for both.) John Tukey prescribed using the various power transformations computed in terms of the technical derivatives, instead of simply taking "log" or "square root", etc. Doing so, he argued, keeps the overall magnitude of the residuals of an analysis (relatively?) constant -- and you have a better shot at a fair comparison. I saw that in one of his co-authored textbooks. One criterion that probably helped you intuitively rule out the "raw" form of the outcome is the distribution of the residuals of prediction: You can see that there are enormous outliers that you *know* do not deserve all that implicit weight for shaping the fit. That is generally one standard that I try to apply. For your particular problem, I would keep in mind that the log of a ratio is often a more "natural" transformation than the ratio itself, for model building in general. -- Rich Ulrich Date: Sun, 17 Feb 2013 13:52:31 +0000 From: [hidden email] Subject: Schwartz Bayesian Criterion, BIC in Mixed models To: [hidden email] Dependent is some form of deviation form ‘correct’ Want to know best form of that deviation Possibilities are: raw(observed-correct), ratio(observed/correct), ln[ratio(observed/correct)] BIC value, with smaller being better are: BIC raw(observed-correct) = 2196 BIC ratio(observed/correct) = 57.5 BIC ln[ratio(observed/correct)] = 11.4 I already knew the raw was a non-starter Now would like, for theoretical reasons, to claim that ln(ratio) is better than ratio. Obviously 11 is lower than 57, but is it ‘ENOUGH’ lower? How would one know what is a difference in BIC large enough to be useful? Or put another way what IS the distribution of the BIC All help gratefully received ... |
Free forum by Nabble | Edit this page |