Obtaining 95% Confidence Limits about an Intraclass Correlation Coefficient from the MIXED procedure

[Ryan]

All,

 

A recent question about computing an ICC reminded me of a simulation experiment I conducted a while back to assess whether or not one could directly obtain 95% confidence limits about an ICC for a specific situation from the MIXED procedure. Here is the situation in which one can directly obtain 95% confidence limits:

 

An inter-rater reliability study is carried out under the following conditions: (1) exactly two raters are used [treated as a fixed effect], (2) subjects are sampled randomly from the population of interest, (3) both raters evaluate each subject, and (4) ratings are ordered categories.

 

Below my name is the simulation experiment. Note that the first MIXED code provides the ICC and 95% confidence limits in the "Estimates of Covariance Parameters" Table. The variance components [estimated from the second MIXED code] entered into the equation in the COMPUTE statement provide confirmation that specifying a first-order autoregressive residual covariance structure in the first MIXED code does in fact produce the ICC.

 

Hope this is of interest to others.

 

Ryan

--

 

*Generate data for MIXED model to estimate ICC and 95% Confidence Limits.

set seed 65923454.

new file.
inp pro.

 

 compute subject=-99.
 compute rater = -99.
 compute x1 = -99.
 compute x2 = -99.
 compute e1 = -99.
 compute e2 = -99.
 compute sigma = 1.
 compute rho = 0.70.
 compute a11 = 1.
 compute a21 = rho.
 compute a22 = sqrt(1 - rho**2).

 

 leave subject to a22.

 

  loop subject= 1 to 1000.

  compute x1 = rv.normal(0,1).
  compute x2 = rv.normal(0,1).

  compute e1 = sigma * a11*x1.
  compute e2 = sigma * (a21*x1 + a22*x2).

 

 loop rater = 1 to 2.

 compute y = 1.5 + 2.4*(rater=1) + e1*(rater=1) + e2*(rater=2).

 end case.
 end loop.
end loop.
end file.
end inp pro.
exe.

 

delete variables x1 x2 sigma rho a11 a21 a22 e1 e2.

 

MIXED y BY rater
  /FIXED=rater | SSTYPE(3)
  /METHOD=REML
  /PRINT= TESTCOV
  /REPEATED=rater | SUBJECT(subject) COVTYPE(AR1).

 

MIXED y BY rater
  /FIXED=rater | SSTYPE(3)
  /METHOD=REML
  /PRINT=G
  /RANDOM=intercept | SUBJECT(subject).

 
COMPUTE ICC=0.7288263600198803 / (0.7288263600198803 + 0.3157454299761715).
EXECUTE.

[Bruce Weaver]

Hi Ryan.  You could provide further confirmation as follows.

* To provide further confirmation, restructure the file
* and use RELIABILITY to compute the ICC and 95% CI.

FORMATS subject rater (f5.0).

CASESTOVARS
  /ID=subject
  /INDEX=rater
  /FIXED = ICC
  /GROUPBY=VARIABLE.

RELIABILITY
  /VARIABLES=y.1 y.2
  /SCALE('ALL VARIABLES') ALL
  /MODEL=ALPHA
  /ICC=MODEL(MIXED) TYPE(CONSISTENCY) CIN=95 TESTVAL=0.


Look at the output for the Single-Measures ICC.

R B wrote

All,

A recent question about computing an ICC reminded me of a simulation
experiment I conducted a while back to assess whether or not one could
directly obtain 95% confidence limits about an ICC for a specific
situation from the MIXED procedure. Here is the situation in which one can
directly obtain 95% confidence limits:

An inter-rater reliability study is carried out under the following
conditions: (1) exactly two raters are used [treated as a fixed
effect], (2) subjects are sampled randomly from the population of interest,
(3) both raters evaluate each subject, and (4) ratings are ordered
categories.

Below my name is the simulation experiment. Note that the first MIXED code
provides the ICC and 95% confidence limits in the "Estimates of Covariance
Parameters" Table. The variance components [estimated from the second MIXED
code] entered into the equation in the COMPUTE statement provide
confirmation that specifying a first-order autoregressive residual
covariance structure in the first MIXED code does in fact produce the ICC.

Hope this is of interest to others.

Ryan
--

*Generate data for MIXED model to estimate ICC and 95% Confidence Limits.

set seed 65923454.
new file.
inp pro.

 compute subject=-99.
 compute rater = -99.
 compute x1 = -99.
 compute x2 = -99.
 compute e1 = -99.
 compute e2 = -99.
 compute sigma = 1.
 compute rho = 0.70.
 compute a11 = 1.
 compute a21 = rho.
 compute a22 = sqrt(1 - rho**2).

 leave subject to a22.

  loop subject= 1 to 1000.
  compute x1 = rv.normal(0,1).
  compute x2 = rv.normal(0,1).
  compute e1 = sigma * a11*x1.
  compute e2 = sigma * (a21*x1 + a22*x2).

 loop rater = 1 to 2.
 compute y = 1.5 + 2.4*(rater=1) + e1*(rater=1) + e2*(rater=2).

 end case.
 end loop.
end loop.
end file.
end inp pro.
exe.

delete variables x1 x2 sigma rho a11 a21 a22 e1 e2.

MIXED y BY rater
  /FIXED=rater | SSTYPE(3)
  /METHOD=REML
  /PRINT= TESTCOV
  /REPEATED=rater | SUBJECT(subject) COVTYPE(AR1).

MIXED y BY rater
  /FIXED=rater | SSTYPE(3)
  /METHOD=REML
  /PRINT=G
  /RANDOM=intercept | SUBJECT(subject).

COMPUTE ICC=0.7288263600198803 / (0.7288263600198803 + 0.3157454299761715).
EXECUTE.