Posted by
bdates on
Jan 20, 2018; 5:04pm
URL: http://spssx-discussion.165.s1.nabble.com/interrater-reliabillity-but-for-any-pair-of-raters-tp5735428p5735429.html
Talma,
First of all, if the numbers 1 - 4 are ordinal, then you should use the intraclass correlation coefficient, ICC. It is equivalent to Fleiss' weighted kappa. This doesn't solve your need to produce 435 pair-wise ICC's though. There are two options in the ICC process, one called Average Measure, which produces the kappa equivalent to Fleiss' weighted; and the other called Single Measure, which produces a kappa indicative of a typical single judge. It's not exactly what you have indicated, but certainly it approaches your request.
If the numbers are simply labels for nominal data, then kappa is probably in order. There is a version of Cohen's kappa (not Fleiss') that produces kappa's for each pair of raters. It is called Light's kappa, and the final value is an average of all pair-wise Cohen's kappa's. I have written syntax for this in matrix format, and in the process a vector of all pairwise kappa's is produced, which could be saved as an outfile and then called. It's in order in terms of the judges, i.e., in your case the first 29 kappa's would be indicative of rater 1 with all other raters, the next 28 indicative of rater 2 with raters 3 through 30, etc. Light's kappa is an acceptable statistic in journals, so the individual kappa's would be good indications of pairwise agreement. Additionally, two-rater kappa's using Cohen's approach are equivalent to Fleiss' kappa as long as there is marginal homogeneity, with is the theoretical underpinning of Fleiss' kappa anyway.
Brian
________________________________________
From: SPSSX(r) Discussion [
[hidden email]] on behalf of Talma [
[hidden email]]
Sent: Saturday, January 20, 2018 7:52 AM
To:
[hidden email]
Subject: interrater reliabillity - but for any pair of raters?
Dear list members,
I suspect I have a somewhat unusual questions - specifically, for a number
of categorical judgements (on a scale with judgements 1, 2, 3 or 4), I have
data from 30 judges (j1, j2, j3 etc.).
Judges are in the columns and judgements in rows.
My task is to calculate an interrater reliability for any pair of the 30
judges. So this makes n*(n-1)/2 = 30*(29)/2 = 435 interrater coefficients,
one for each dyad of raters.
For any dyadic interrater coefficient, I could use e.g. the kappa
coefficient available in the SPSS crosstabs procedure.
But how could I tell SPSS to repeat the procedure for each combination of
the 30 judges? So that ultimately, I receive as many kappa coefficients as
there are pairwise comparisons (=435, in this example)? I would be very
grateful for your suggestions how to implement such an automated procedure?
Many thanks...Talma
--
Sent from:
http://spssx-discussion.1045642.n5.nabble.com/=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD
=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD