Re: inter-rater reliability with multiple raters

Posted by Ryan on Jun 22, 2014; 9:42pm
URL: http://spssx-discussion.165.s1.nabble.com/inter-rater-reliability-with-multiple-raters-tp5726465p5726572.html

The linear MIXED modeling (LMM) procedure is a preferred approach to estimating parameters using REML for unbalanced designs where the data are assumed to be missing at random (MAR). Consistent with the underlying theory of LMM, I have generated data below for which the number of ratings per subject varies according to a random uniform variate. I then estimate the between subject variance and residual variance using the default REML estimation method offered by the MIXED procedure, which is all that is needed to calculate an ICC. 

It should be noted that the linear MIXED model I have employed conforms to one way of estimating an ICC for a specific design described by:

Shrout, P.E. and Fleiss, J.L (1979). Intraclass correlations: uses in assessing rater reliability. Psychological Bulletin, 86, 420-428

where,

1. Each subject is rated by multiple raters

2. Raters are assumed to be randomly assigned to subjects

3. All subjects have the same number of raters

Note that this is consistent with one of the examples in my previous post:

https://listserv.uga.edu/cgi-bin/wa?A2=ind1406&L=SPSSX-L&P=R29580&I=-3&d=No+Match%3BMatch%3BMatches

The data generated below conform to 1. and 2., but obviously not 3. However, as I mentioned  before, as long as the data are MAR, the REML estimators should reasonably recover the parameters. Using the example below, the population between subjects VC = .300 and the population ICC=.231. The MIXED model estimates the VC = .306 and the estimated ICC = 0.230.

Note: I generated data that do not specify which raters were assigned to each subject. This post is not intended to compare the proposed method to other methods, but simply to provide a concrete example of how one might estimate an ICC where each subject has been rated by a varying random number of raters.

Ryan

--

/*Generate Data*/.

SET SEED 987879546.

NEW FILE.

INPUT PROGRAM.

COMPUTE subject_ID = -99.

COMPUTE intercept= -99.

COMPUTE nobs = -99.

LEAVE subject_ID to nobs.

LOOP subject_ID = 1 to 100.

COMPUTE intercept = sqrt(.30)*rv.normal(0,1).

COMPUTE nobs = rnd(rv.uniform(12,35)).

LOOP j=1 to nobs.

COMPUTE y = intercept + rv.normal(0,1).

END CASE.

END LOOP.

END LOOP.

END FILE.

END INPUT PROGRAM.

EXECUTE.

*Fit Linear Mixed Model.

MIXED y BY subject_ID

  /FIXED= | SSTYPE(3)

  /METHOD=REML

  /PRINT= G 

  /RANDOM=subject_ID.

*Estimate ICC.

COMPUTE ICC =.30628546692761 / (.30628546692761 + 1.02273871556984 ).

EXECUTE.

On Wed, Jun 18, 2014 at 12:48 PM, Rich Ulrich <[hidden email]> wrote:

ICCs with unequal Ns -

I got a reference in 1995   for computing a simple ICC with unequal Ns from the
Usenet stats group, sci.stat.consult.  I put the formula (below) in my stats-FAQ,
which I maintained from 1997 to about 2006.  No one ever provided newer references.

This is owing to Ernest Haggard, Intraclass Correlation and the Analysis of Variance
(1958) , as posted by Michael Bailey and reformatted and adapted by me.  I hope I have
not screwed it up.

Let, R=intraclass correlation,
    BSMS=between Subject mean square,
    WMS=mean square within,
    c=number of Subjects, and
    ki=number for the ith Subject. Then:

         R = (BSMS - WMS) /  ( BSMS + (k' -1)*WMS )
         where k' =  [ sum(ki) -   (sum(ki**2))/sum(ki) ]   / (c-1)


The value of k'   does need to work out to something in the range of an average ki
number of ratings for the subjects.  I think that I remember using the "reciprocal
mean" of the counts, but I don't remember using this formula for k' for getting it.



--
Rich Ulrich

---

Date: Tue, 17 Jun 2014 22:49:46 -0400

From: [hidden email]
Subject: Re: inter-rater reliability with multiple raters
To: [hidden email]

Okay. Let me start by saying I'm a bit (okay maybe very) under the weather and paid work is catching up with me, so apologies for any typos/mistakes.  Having said that, this topic is quite interesting as it relates to showing some connections between generalizability coefficients, various forms of ICCs, and coefficient alpha. 

Before discussing those connections, however, the short answer to the question about whether there are valid estimators of an ICC in an unbalanced design would be a solid "it depends." I would argue that to obtain a valid ICC we need to appropriately decompose the variance to obtain between subject variance and variance attributable to all other sources. This could prove challenging.

At any rate, if the raters tend to agree in their ratings of each subject, then the between subject variance will tend to be much larger than other sources of variances, and the ICC should approach 1.0. 

With that said, the ICC is defined as:

ICC = var(between Ss) / Total Variance

where,

Total Variance = 

(1) between Ss variance

(2) between Rater Variance

(3) error variance

If one initially planned a crossed design (all subjects were intended to be rated by all raters) but due to random circumstances some raters were unable to rate some subjects and those missing data can be assumed to be missing at random (MAR), then I would suggest that one could theoretically estimate a valid estimate of the ICC using an ML estimator via the MIXED procedure in SPSS from which the estimated variance components would be inserted into the following ICC equation:

ICC = var(between Ss )/ [var(between Ss) + error variance]

I believe there are more sophisticated ways to deal with unbalanced designs that have been published in the past 5 years, but I am not fully versed in such methods. With that said, please see below for a small demonstration using SPSS syntax that might help make connections between generalizability coefficients from a one-facet design, ICC, and coefficient alpha using various procedures in SPSS:

[snip, SPSS code for alpha examples]

===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD