Re: Bayesian Information Criterion & Checking for Normality

Posted by Ryan on Apr 20, 2013; 1:39am
URL: http://spssx-discussion.165.s1.nabble.com/Bayesian-Information-Criterion-Checking-for-Normality-tp5719504p5719577.html

Needless to say, or perhaps not needless to say, comparisons of non-nested models via the AIC, BIC, and/or the Vuong statistic require that the same data be used; that is, the dependent variable values must be exactly the same, and the same valid cases used to fit one model must also be used to fit the other model.

 

Ryan

On Fri, Apr 19, 2013 at 9:20 PM, R B <[hidden email]> wrote:

I just came across this post. It is true that one cannot compare statistical models using the BIC when the response variable is not the same. Having said that, the BIC, and AIC for that matter, may be used for comparison of non-nested models where the response variable is the same. It is also possible to perform a statistical test, called the "Vuong test," to compare non-nested models (again assuming the response variable is the same), which may or may not include an AIC or BIC correction.  

 

Ryan

On Tue, Apr 16, 2013 at 2:37 PM, Kornbrot, Diana <[hidden email]> wrote:

Dear Colleagues,

Some time back I posted following query
Can I use BIC to compare 3 different transformations of a dependent variable, here accuracy of time estimation?
I am examining ‘accuracy’ as function of 1 between and 2 repeated measures factors using SPSS ‘MIXED’.

True value is A, obtained participant estimate is X. Possible accuracy measures are:
X-A         symmetric about accurate value =0, BIC astronomical ~2000
X/A         NOT symmetric  about accurate  value=1, BIC =57
Ln(X/A)  symmetric about accurate value =0, BIC = 11

Since df are all the same any improvement in BIC show model superiority. Want to strongly claim that ln(X/A) is ‘best’ measure of accuracy in this situation. The DATA should decide best measure

Because X/A is not very different from 1, the effect sizes &  p-values are quite similar for the ln(X/A) and X/A  analyses (cf expanding series for X/A about 1).

Realize now that I am making non-standard use of BIC as a test of normality.
Is this legitimate?

Several people said very firmly NO. BIC is ONLY for comparing nested models
Thanks Ric Ulric, Jo Pemberton and others. YOU ARE RIGHT , sorry for being stupid over this
BIC & other information criteria re useless for this purpose because they are NOT invariant under changes of scale.

The best way to check for normality is to SAVE the residuals & then check residual for normality.
Old ways are BEST!!
IN my example, ln(x/a) residuals have ns Kolmogorov-Smirnov, where x/a and x-a residuals had highly significant K-S. This solves my problem & hopefully is of some use to others. Shapiro-Wilks was significantly non-normal for all models, but with 390 observations, power was high. By eye deviation from normality was small but detectable in detrended Q-Q plot (max deviation 1 sd). For my particular problem, kurtosis was significantly > for normal, there is a reason for this

Thanks for everyone’s help

Best

Diana

---

Emeritus Professor Diana Kornbrot
email:  d.e.kornbrot@...    
 web:    http://dianakornbrot.wordpress.com/
Work
Department of Psychology
School of Life and Medical Sciences
University of Hertfordshire
College Lane, Hatfield, Hertfordshire AL10 9AB, UK
voice:   <a href="tel:%2B44%20%280%29%20170%20728%204626" target="_blank" value="[phone]">[phone]
Home
19 Elmhurst Avenue
London N2 0LT, UK
voice:   <a href="tel:%2B44%20%280%29%20208%20%C2%A0444%202081" target="_blank" value="[phone]">[phone]
mobile: <a href="tel:%2B44%20%280%29%20740%20318%201612" target="_blank" value="[phone]">[phone]