Schwartz Bayesian Criterion, BIC in Mixed models

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Schwartz Bayesian Criterion, BIC in Mixed models

Kornbrot, Diana
Schwartz Bayesian Criterion, BIC in Mixed models Have mixed model with 1 between 2 level factor; 1 repeated 2 level factor; 1 rpeated 5 level factor
Dependent is some form of deviation form ‘correct’
Want to know best form of that deviation
Possibilities are: raw(observed-correct), ratio(observed/correct), ln[ratio(observed/correct)]
BIC value, with smaller being better are:
BIC raw(observed-correct)       = 2196
BIC ratio(observed/correct)      =    57.5
BIC ln[ratio(observed/correct)] =    11.4

I already knew the raw was a non-starter
Now would like, for theoretical reasons, to claim that ln(ratio) is better than ratio.
Obviously 11 is lower than 57, but is it ‘ENOUGH’ lower?
How would one know what is a difference in BIC large enough to be useful?
Or put another way what IS the distribution of the BIC

All help gratefully received

Best

Diana

Emeritus Professor Diana Kornbrot
email:  d.e.kornbrot@...    
 web:    http://dianakornbrot.wordpress.com/
Work
Department of Psychology
School of Life and Medical Sciences
University of Hertfordshire
College Lane, Hatfield, Hertfordshire AL10 9AB, UK
voice:   +44 (0) 170 728 4626
Home
19 Elmhurst Avenue
London N2 0LT, UK
voice:   +44 (0) 208  444 2081
mobile: +44 (0) 740 318 1612


Reply | Threaded
Open this post in threaded view
|

Re: Schwartz Bayesian Criterion, BIC in Mixed models

Alex Reutter
I'm not sure I understand what you're trying to do below.  The BIC is a measure for comparing two models, related to the likelihood.  When using likelihoods to compare two models, you would typically do a likelihood ratio test in order to see whether the difference in the likelihoods is due to a real difference in the models, or to chance variation.  When using the BIC, the basic theory is that you simply compare the BIC values of the two models, because the BIC has already accounted for the differences in model complexity.

Alex




From:        "Kornbrot, Diana" <[hidden email]>
To:        [hidden email],
Date:        02/17/2013 07:55 AM
Subject:        Schwartz Bayesian Criterion, BIC in Mixed models
Sent by:        "SPSSX(r) Discussion" <[hidden email]>




Have mixed model with 1 between 2 level factor; 1 repeated 2 level factor; 1 rpeated 5 level factor
Dependent is some form of deviation form ‘correct’
Want to know best form of that deviation
Possibilities are: raw(observed-correct), ratio(observed/correct), ln[ratio(observed/correct)]
BIC value, with smaller being better are:
BIC raw(observed-correct)       = 2196
BIC ratio(observed/correct)      =    57.5
BIC ln[ratio(observed/correct)] =    11.4

I already knew the raw was a non-starter
Now would like, for theoretical reasons, to claim that ln(ratio) is better than ratio.
Obviously 11 is lower than 57, but is it ‘ENOUGH’ lower?
How would one know what is a difference in BIC large enough to be useful?
Or put another way what IS the distribution of the BIC

All help gratefully received

Best

Diana

Reply | Threaded
Open this post in threaded view
|

Re: Schwartz Bayesian Criterion, BIC in Mixed models

Rich Ulrich
In reply to this post by Kornbrot, Diana
Here is a note from the Wikip article on BIC.

   It is important to keep in mind that the BIC can be used to

   compare estimated models only when the numerical values

   of the dependent variable are identical for all estimates being

   compared. The models being compared need not be nested

   unlike the case when models are being compared using an F

   or likelihood ratio test.



I'm certainly no expert in AIC/BIC  but I'm pretty sure that the Wikip
article is making a valid point, and that I'm extrapolating legitimately....


You are in a really iffy situation when you try to compare BICs
with different versions of the outcome variable.  The residuals
of the fit differ in magnitude, and that matters.  ("How" it matters is
less obvious with likelihood estimation than with OLS, but it does
matter for both.) 




John Tukey prescribed using the various power transformations computed

in terms of the technical derivatives, instead of simply taking "log" or
"square root", etc.  Doing so, he argued,  keeps the overall magnitude of the
residuals of an analysis (relatively?) constant -- and you have a better shot at

a fair comparison.  I saw that in one of his co-authored textbooks.



One criterion that probably helped you intuitively rule out the "raw" form of
the outcome is the distribution of the residuals of prediction:  You can see

that there are enormous outliers that you *know*  do not deserve all that

implicit weight for shaping the fit.  That is generally one standard that I try
to apply.  For your particular problem, I would keep in mind that the log of a
ratio is often a more "natural" transformation than the ratio itself, for model
building in general.



--
Rich Ulrich




Date: Sun, 17 Feb 2013 13:52:31 +0000
From: [hidden email]
Subject: Schwartz Bayesian Criterion, BIC in Mixed models
To: [hidden email]

Schwartz Bayesian Criterion, BIC in Mixed models Have mixed model with 1 between 2 level factor; 1 repeated 2 level factor; 1 rpeated 5 level factor
Dependent is some form of deviation form ‘correct’
Want to know best form of that deviation
Possibilities are: raw(observed-correct), ratio(observed/correct), ln[ratio(observed/correct)]
BIC value, with smaller being better are:
BIC raw(observed-correct)       = 2196
BIC ratio(observed/correct)      =    57.5
BIC ln[ratio(observed/correct)] =    11.4

I already knew the raw was a non-starter
Now would like, for theoretical reasons, to claim that ln(ratio) is better than ratio.
Obviously 11 is lower than 57, but is it ‘ENOUGH’ lower?
How would one know what is a difference in BIC large enough to be useful?
Or put another way what IS the distribution of the BIC

All help gratefully received
...
Reply | Threaded
Open this post in threaded view
|

Re: Schwartz Bayesian Criterion, BIC in Mixed models

Rich Ulrich
By the way, to add to my comment -- The design features repeated
measures, 2x5.  When there is much correlation, Repeated Measures
are great for looking at which transformation gives a reasonable, equal-
interval sort of error.  I usually have Pre-Post, and I look to see which
transformed scatterplot gives a Normal sort of ellipse.  For my data, that's
almost always the choice that gives the right model for the error.

For a 2x5 repeated design, there are choices of what to plot.  Something
that includes the widest range of scores might be most useful.

--
Rich Ulrich


From: [hidden email]
To: [hidden email]; [hidden email]
Subject: RE: Schwartz Bayesian Criterion, BIC in Mixed models
Date: Tue, 19 Feb 2013 13:03:59 -0500

Here is a note from the Wikip article on BIC.

   It is important to keep in mind that the BIC can be used to
   compare estimated models only when the numerical values
   of the dependent variable are identical for all estimates being
   compared. The models being compared need not be nested
   unlike the case when models are being compared using an F
   or likelihood ratio test.


I'm certainly no expert in AIC/BIC  but I'm pretty sure that the Wikip
article is making a valid point, and that I'm extrapolating legitimately....

You are in a really iffy situation when you try to compare BICs
with different versions of the outcome variable.  The residuals
of the fit differ in magnitude, and that matters.  ("How" it matters is
less obvious with likelihood estimation than with OLS, but it does
matter for both.) 

John Tukey prescribed using the various power transformations computed
in terms of the technical derivatives, instead of simply taking "log" or
"square root", etc.  Doing so, he argued,  keeps the overall magnitude of the
residuals of an analysis (relatively?) constant -- and you have a better shot at
a fair comparison.  I saw that in one of his co-authored textbooks.

One criterion that probably helped you intuitively rule out the "raw" form of
the outcome is the distribution of the residuals of prediction:  You can see
that there are enormous outliers that you *know*  do not deserve all that
implicit weight for shaping the fit.  That is generally one standard that I try
to apply.  For your particular problem, I would keep in mind that the log of a
ratio is often a more "natural" transformation than the ratio itself, for model
building in general.

--
Rich Ulrich




Date: Sun, 17 Feb 2013 13:52:31 +0000
From: [hidden email]
Subject: Schwartz Bayesian Criterion, BIC in Mixed models
To: [hidden email]

Schwartz Bayesian Criterion, BIC in Mixed models Have mixed model with 1 between 2 level factor; 1 repeated 2 level factor; 1 rpeated 5 level factor
Dependent is some form of deviation form ‘correct’
Want to know best form of that deviation
Possibilities are: raw(observed-correct), ratio(observed/correct), ln[ratio(observed/correct)]
BIC value, with smaller being better are:
BIC raw(observed-correct)       = 2196
BIC ratio(observed/correct)      =    57.5
BIC ln[ratio(observed/correct)] =    11.4

I already knew the raw was a non-starter
Now would like, for theoretical reasons, to claim that ln(ratio) is better than ratio.
Obviously 11 is lower than 57, but is it ‘ENOUGH’ lower?
How would one know what is a difference in BIC large enough to be useful?
Or put another way what IS the distribution of the BIC

All help gratefully received
...