Re: inter-rater reliability with multiple raters

Posted by Ryan on Jun 18, 2014; 2:49am
URL: http://spssx-discussion.165.s1.nabble.com/inter-rater-reliability-with-multiple-raters-tp5726465p5726515.html

Okay. Let me start by saying I'm a bit (okay maybe very) under the weather and paid work is catching up with me, so apologies for any typos/mistakes.  Having said that, this topic is quite interesting as it relates to showing some connections between generalizability coefficients, various forms of ICCs, and coefficient alpha. 

Before discussing those connections, however, the short answer to the question about whether there are valid estimators of an ICC in an unbalanced design would be a solid "it depends." I would argue that to obtain a valid ICC we need to appropriately decompose the variance to obtain between subject variance and variance attributable to all other sources. This could prove challenging.

At any rate, if the raters tend to agree in their ratings of each subject, then the between subject variance will tend to be much larger than other sources of variances, and the ICC should approach 1.0. 

With that said, the ICC is defined as:

ICC = var(between Ss) / Total Variance

where,

Total Variance = 

(1) between Ss variance

(2) between Rater Variance

(3) error variance

If one initially planned a crossed design (all subjects were intended to be rated by all raters) but due to random circumstances some raters were unable to rate some subjects and those missing data can be assumed to be missing at random (MAR), then I would suggest that one could theoretically estimate a valid estimate of the ICC using an ML estimator via the MIXED procedure in SPSS from which the estimated variance components would be inserted into the following ICC equation:

ICC = var(between Ss )/ [var(between Ss) + error variance]

I believe there are more sophisticated ways to deal with unbalanced designs that have been published in the past 5 years, but I am not fully versed in such methods. With that said, please see below for a small demonstration using SPSS syntax that might help make connections between generalizability coefficients from a one-facet design, ICC, and coefficient alpha using various procedures in SPSS:

DATA LIST LIST / rater1 rater2 rater3 (3f1.0).

BEGIN DATA

3 4 4

1 2 3

4 4 5

7 6 10

1 2 3

END DATA.

*Calculate ICC via Reliabillity Procedure: MIXED model with CONSISTENCY type. 

RELIABILITY

  /VARIABLES=rater1 rater2 rater3

  /SCALE('ALL VARIABLES') ALL

  /MODEL=ALPHA

  /ICC=MODEL(MIXED) TYPE(CONSISTENCY) CIN=95 TESTVAL=0.

*Calculate ICC via Reliabillity Procedure: RANDOM model with ABSOLUTE type. 

RELIABILITY

  /VARIABLES=rater1 rater2 rater3

  /SCALE('ALL VARIABLES') ALL

  /MODEL=ALPHA

  /ICC=MODEL(RANDOM) TYPE(ABSOLUTE) CIN=95 TESTVAL=0.

*Restructure dataset from wide to long to more easily obtain variance components.

VARSTOCASES

  /ID=id

  /MAKE rating FROM rater1 rater2 rater3

  /INDEX=rater(3) 

  /KEEP=

  /NULL=KEEP.

MIXED rating BY id rater

  /FIXED=| SSTYPE(3)

  /METHOD=REML

  /RANDOM=id rater | COVTYPE(VC).

*ICC version where all subjects are rated by same/consistent random subset of all possible raters. 

*ICC matches the Single Measure ICC via Reliability Procedure: RANDOM model with ABSOLUTE type. 

*Note: ICC_1 = var(between Ss) / [var(between Ss) + var(raters) + error].

COMPUTE ICC_1 = 5.200 / (5.200 + 0.767 + 0.633).

EXECUTE.

MIXED rating BY id rater

  /FIXED=rater| SSTYPE(3)

  /METHOD=REML

  /RANDOM=id | COVTYPE(VC).

*ICC version where subjects are rated by the assumed population of all possible raters. 

*Matches Single Measure ICC from Reliability Procedure.

*Note: ICC_2 = var(between Ss) / [var(between Ss) + error].

COMPUTE ICC_2= 5.200 / (5.200 + 0.633).

EXECUTE.

MIXED rating BY id rater

  /FIXED=| SSTYPE(3)

  /METHOD=REML

  /RANDOM=id | COVTYPE(VC).

*ICC version where subjects are assumed to be randomly assigned to a sample of raters, assuming a balanced design.

*In the MIXED model above, note that rater is removed from the model entirely.

COMPUTE ICC_3 = 4.944 / (4.944 + 1.400).

EXECUTE.

*Next, let's employ a one-facet G theory model using ANOVA.

DATASET DECLARE vc.

VARCOMP rating BY rater id 

  /RANDOM=rater id

  /OUTFILE=VAREST (vc)

  /METHOD=SSTYPE(3)

  /PRINT=SS

  /DESIGN=rater id

  /INTERCEPT=INCLUDE.

DATASET ACTIVATE vc.

*Using variance components estimated from the mean squares of the ANOVA above.

*the relative and absolute G coefficients are estimated. 

*Note that the relative G coefficient = coefficient alpha.

*Note, also, ICC matches the average measure ICC via Reliability Procedure:. 

*RANDOM model with CONSISTENCY type.

*VC1 = var(rater).

*VC2 = var(subject).

*VC3 = error variance.

*Based on the equation below, it becomes apparent that coefficient alpha assumes.

*that between rater variance ("VC1") = zero. 

COMPUTE relative_g_coeff_3raters = VC2 / (VC2 + VC3/3).

*If we assume the number of raters is 1 (one), we obtain the.

*ICC for single measures calculated from Reliability Procedure.

*and the second MIXED procedure.

COMPUTE relative_g_coeff_1rater = VC2 / (VC2 + VC3/1).

*Next, compute the absolute g coefficient assuming 3 raters.

*Note: Also, ICC matches the average measure ICC via Reliability Procedure:. 

*RANDOM model with ABSOLUTE type. 

COMPUTE absolute_g_coeff_3raters = VC2 / (VC1/3 + VC2 + VC3/3).

*Next, compute the absolute g coefficient assuming 1 rater.

*Note: Matches ICC estimated from first MIXED model which assumes.

*that subjects and raters are random.

*Note: Also, ICC matches the single measure ICC via Reliability Procedure:. 

*RANDOM model with ABSOLUTE type. 

COMPUTE absolute_g_coeff_1rater = VC2 / (VC1/1 + VC2 + VC3/1).

EXECUTE.