SPSSX Discussion

interrater reliabillity - but for any pair of raters?

Classic

List

Threaded

11 messages Options

Talma

interrater reliabillity - but for any pair of raters?

Dear list members,

I suspect I have a somewhat unusual questions - specifically, for a number
of categorical judgements (on a scale with judgements 1, 2, 3 or 4), I have
data from 30 judges (j1, j2, j3 etc.).

Judges are in the columns and judgements in rows.

My task is to calculate an interrater reliability for any pair of the 30
judges. So this makes n*(n-1)/2 = 30*(29)/2 = 435 interrater coefficients,
one for each dyad of raters.

For any dyadic interrater coefficient, I could use e.g. the kappa
coefficient available in the SPSS crosstabs procedure.

But how could I tell SPSS to repeat the procedure for each combination of
the 30 judges? So that ultimately, I receive as many kappa coefficients as
there are pairwise comparisons (=435, in this example)? I would be very
grateful for your suggestions how to implement such an automated procedure?

Many thanks...Talma

--
Sent from: http://spssx-discussion.1045642.n5.nabble.com/

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

bdates

Re: interrater reliabillity - but for any pair of raters?

Talma,

First of all, if the numbers 1 - 4 are ordinal, then you should use the intraclass correlation coefficient, ICC. It is equivalent to Fleiss' weighted kappa. This doesn't solve your need to produce 435 pair-wise ICC's though. There are two options in the ICC process, one called Average Measure, which produces the kappa equivalent to Fleiss' weighted; and the other called Single Measure, which produces a kappa indicative of a typical single judge. It's not exactly what you have indicated, but certainly it approaches your request.

If the numbers are simply labels for nominal data, then kappa is probably in order. There is a version of Cohen's kappa (not Fleiss') that produces kappa's for each pair of raters. It is called Light's kappa, and the final value is an average of all pair-wise Cohen's kappa's. I have written syntax for this in matrix format, and in the process a vector of all pairwise kappa's is produced, which could be saved as an outfile and then called. It's in order in terms of the judges, i.e., in your case the first 29 kappa's would be indicative of rater 1 with all other raters, the next 28 indicative of rater 2 with raters 3 through 30, etc. Light's kappa is an acceptable statistic in journals, so the individual kappa's would be good indications of pairwise agreement. Additionally, two-rater kappa's using Cohen's approach are equivalent to Fleiss' kappa as long as there is marginal homogeneity, with is the theoretical underpinning of Fleiss' kappa anyway.

Brian
________________________________________
From: SPSSX(r) Discussion [[hidden email]] on behalf of Talma [[hidden email]]
Sent: Saturday, January 20, 2018 7:52 AM
To: [hidden email]
Subject: interrater reliabillity - but for any pair of raters?

Dear list members,

I suspect I have a somewhat unusual questions - specifically, for a number
of categorical judgements (on a scale with judgements 1, 2, 3 or 4), I have
data from 30 judges (j1, j2, j3 etc.).

Judges are in the columns and judgements in rows.

My task is to calculate an interrater reliability for any pair of the 30
judges. So this makes n*(n-1)/2 = 30*(29)/2 = 435 interrater coefficients,
one for each dyad of raters.

For any dyadic interrater coefficient, I could use e.g. the kappa
coefficient available in the SPSS crosstabs procedure.

But how could I tell SPSS to repeat the procedure for each combination of
the 30 judges? So that ultimately, I receive as many kappa coefficients as
there are pairwise comparisons (=435, in this example)? I would be very
grateful for your suggestions how to implement such an automated procedure?

Many thanks...Talma

--
Sent from: http://spssx-discussion.1045642.n5.nabble.com/

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

David Marso-2

Re: interrater reliabillity - but for any pair of raters?

In reply to this post by Talma

Macro with nested !DO and CROSSTABS in body. Use OMS to capture output. AGGREGATE to get average. Or maybe Brian will post a link to his MATRIX program.

UNTESTED. I'm on my phone...

DEFINE ! AllPairsKappa( raters !CMDEND)
!LET !CPY = !raters
!DO ! j1 !IN ( !raters)
!LET ! CPY= !TAIL( !CPY)
!DO !j2 !IN (!CPY)
CROSSTABS TABLES !j1 BY !j2 /STATISTICS KAPPA.
!DOEND
!DOEND
!ENDDEFINE.

SET UP OMS..RTFM FOR THAT...

!AllPairsKappa raters= j1 j2 j3 ..... j30.

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

Jon Peck

Re: interrater reliabillity - but for any pair of raters?

Looks like 2 of the ! chars have an illegal following blank.

On Sat, Jan 20, 2018 at 7:04 PM David Marso <[hidden email]> wrote:

Macro with nested !DO and CROSSTABS in body. Use OMS to capture output. AGGREGATE to get average. Or maybe Brian will post a link to his MATRIX program.

UNTESTED. I'm on my phone...

DEFINE ! AllPairsKappa( raters !CMDEND)
!LET !CPY = !raters
!DO ! j1 !IN ( !raters)
!LET ! CPY= !TAIL( !CPY)
!DO !j2 !IN (!CPY)
CROSSTABS TABLES !j1 BY !j2 /STATISTICS KAPPA.
!DOEND
!DOEND
!ENDDEFINE.

SET UP OMS..RTFM FOR THAT...

!AllPairsKappa raters= j1 j2 j3 ..... j30.

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

Jon K Peck
[hidden email]

===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD

David Marso-2

Re: interrater reliabillity - but for any pair of raters?

In reply to this post by Talma

Note to self.
Do not post code to SPSS List from phone.

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

bdates

Re: interrater reliabillity - but for any pair of raters?

In reply to this post by David Marso-2

This looks great. My syntax has all the matrix stuff to calculate Fleiss, so it wouldn't be as elegant as this, in which the OP just needs to figure out how to insert the OMS. I hope the OP from earlier this week is reading this thread, too. It's exactly what he wanted to do.

My one concern is that the ordinal nature of the responses, or at least what looks ordinal, calls for the ICC rather than Fleiss. But that's not really our issue.

Brian
________________________________________
From: SPSSX(r) Discussion [[hidden email]] on behalf of David Marso [[hidden email]]
Sent: Saturday, January 20, 2018 9:04 PM
To: [hidden email]
Subject: Re: interrater reliabillity - but for any pair of raters?

Macro with nested !DO and CROSSTABS in body. Use OMS to capture output. AGGREGATE to get average. Or maybe Brian will post a link to his MATRIX program.

UNTESTED. I'm on my phone...

DEFINE ! AllPairsKappa( raters !CMDEND)
!LET !CPY = !raters
!DO ! j1 !IN ( !raters)
!LET ! CPY= !TAIL( !CPY)
!DO !j2 !IN (!CPY)
CROSSTABS TABLES !j1 BY !j2 /STATISTICS KAPPA.
!DOEND
!DOEND
!ENDDEFINE.

SET UP OMS..RTFM FOR THAT...

!AllPairsKappa raters= j1 j2 j3 ..... j30.

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

Talma

AW: interrater reliabillity - but for any pair of raters?

Dear all,

many thanks for your time + efforts! Because I am a bit slow I adapting the output management system (=SPSS novice, using it for the first time:) ) to David's syntax; I would also be interested in the matrix code Brian referred to?

Best,

Talma

Gesendet von Outlook

Von: SPSSX(r) Discussion <[hidden email]> im Auftrag von Dates, Brian <[hidden email]>
Gesendet: Sonntag, 21. Januar 2018 18:57
An: [hidden email]
Betreff: Re: interrater reliabillity - but for any pair of raters?

bdates

Re: interrater reliabillity - but for any pair of raters?

Talma,

Before you abandon David's code, he inserted the OMS section to provide a file to average the pair-wise kappa's. If you want to calculate the overall kappa in addition to the pairwise kappa's, then pasting and inserting syntax for kappa from the built-in kappa analysis after !End Define will do the trick. This will be a whole lot easier than for me to adapt my syntax.

Brian
________________________________________
From: Talma Claviger [[hidden email]]
Sent: Sunday, January 21, 2018 2:55 PM
To: [hidden email]; Dates, Brian
Subject: AW: interrater reliabillity - but for any pair of raters?

Dear all,

many thanks for your time + efforts! Because I am a bit slow I adapting the output management system (=SPSS novice, using it for the first time:) ) to David's syntax; I would also be interested in the matrix code Brian referred to?

Best,

Talma
Gesendet von Outlook<http://aka.ms/weboutlook>

________________________________
Von: SPSSX(r) Discussion <[hidden email]> im Auftrag von Dates, Brian <[hidden email]>
Gesendet: Sonntag, 21. Januar 2018 18:57
An: [hidden email]
Betreff: Re: interrater reliabillity - but for any pair of raters?

This looks great. My syntax has all the matrix stuff to calculate Fleiss, so it wouldn't be as elegant as this, in which the OP just needs to figure out how to insert the OMS. I hope the OP from earlier this week is reading this thread, too. It's exactly what he wanted to do.

My one concern is that the ordinal nature of the responses, or at least what looks ordinal, calls for the ICC rather than Fleiss. But that's not really our issue.

Brian
________________________________________
From: SPSSX(r) Discussion [[hidden email]] on behalf of David Marso [[hidden email]]
Sent: Saturday, January 20, 2018 9:04 PM
To: [hidden email]
Subject: Re: interrater reliabillity - but for any pair of raters?

Macro with nested !DO and CROSSTABS in body. Use OMS to capture output. AGGREGATE to get average. Or maybe Brian will post a link to his MATRIX program.

UNTESTED. I'm on my phone...

DEFINE ! AllPairsKappa( raters !CMDEND)
!LET !CPY = !raters
!DO ! j1 !IN ( !raters)
!LET ! CPY= !TAIL( !CPY)
!DO !j2 !IN (!CPY)
CROSSTABS TABLES !j1 BY !j2 /STATISTICS KAPPA.
!DOEND
!DOEND
!ENDDEFINE.

SET UP OMS..RTFM FOR THAT...

!AllPairsKappa raters= j1 j2 j3 ..... j30.

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

David Marso

Re: interrater reliabillity - but for any pair of raters?

Administrator

Here is a tested version with some revision for efficiency
-Just passes the data once per main iteration rather than each pair-
along with OMS and AGGREGATE.

---SIMULATED DATA--- EXCLUDE FROM PRODUCTION RUN.
NEW FILE.
DATASET CLOSE ALL.
MATRIX.
SAVE TRUNC(UNIFORM(200,30)*4+1) / OUTFILE= * /VARIABLES j1 TO j30.
END MATRIX.
DATASET NAME raw.

DEFINE !AllPairsKappa( raters !CMDEND)
!LET !CPY = !raters
!DO !j1 !IN ( !raters)
!LET !CPY= !TAIL(!CPY)
!IF ( !CPY !NE !NULL) !THEN
CROSSTABS TABLES !j1 BY !CPY /STATISTICS KAPPA.
!IFEND
!DOEND
!ENDDEFINE.

* OMS.
DATASET DECLARE Kappa.
OMS
/SELECT TABLES /IF COMMANDS=['Crosstabs'] /DESTINATION VIEWER=NO.
OMS /SELECT TABLES
/IF COMMANDS=['Crosstabs'] SUBTYPES=['Symmetric Measures']
/DESTINATION FORMAT=SAV NUMBERED=TableNumber_
OUTFILE='Kappa' VIEWER=NO.
!AllPairsKappa raters = j1 j2 j3 j4 j5 j6 j7 j8 j9 j10 j11 j12 j13 j14 j15
j16 j17 j18 j19 j20 j21 j22 j23 j24 j25 j26 j27 j28
j29 j30.
OMSEND.

DATASET ACTIVATE Kappa.
DELETE VARIABLES TableNumber_ Command_ Subtype_ Label_ Var1.
RENAME VARIABLES Var2 =Kappa.
SELECT IF Kappa EQ 'Kappa'.
DATASET DECLARE agg.
AGGREGATE OUTFILE agg /BREAK Kappa /AvgKappa=MEAN( Value).

bdates wrote

> Talma,
>
> Before you abandon David's code, he inserted the OMS section to provide a
> file to average the pair-wise kappa's. If you want to calculate the
> overall kappa in addition to the pairwise kappa's, then pasting and
> inserting syntax for kappa from the built-in kappa analysis after !End
> Define will do the trick. This will be a whole lot easier than for me to
> adapt my syntax.
>
> Brian
> ________________________________________
> From: Talma Claviger [

> talma.claviger@

> ]
> Sent: Sunday, January 21, 2018 2:55 PM
> To:

> SPSSX-L@.UGA

> ; Dates, Brian
> Subject: AW: interrater reliabillity - but for any pair of raters?
>
> Dear all,
>
>
> many thanks for your time + efforts! Because I am a bit slow I adapting
> the output management system (=SPSS novice, using it for the first time:)
> ) to David's syntax; I would also be interested in the matrix code Brian
> referred to?
>
>
> Best,
>
> Talma
> Gesendet von Outlook<http://aka.ms/weboutlook>
>
>
> ________________________________
> Von: SPSSX(r) Discussion <

> SPSSX-L@.UGA

> > im Auftrag von Dates, Brian <

> BDATES@

> >
> Gesendet: Sonntag, 21. Januar 2018 18:57
> An:

> SPSSX-L@.UGA

> Betreff: Re: interrater reliabillity - but for any pair of raters?
>
> This looks great. My syntax has all the matrix stuff to calculate Fleiss,
> so it wouldn't be as elegant as this, in which the OP just needs to figure
> out how to insert the OMS. I hope the OP from earlier this week is reading
> this thread, too. It's exactly what he wanted to do.
>
> My one concern is that the ordinal nature of the responses, or at least
> what looks ordinal, calls for the ICC rather than Fleiss. But that's not
> really our issue.
>
> Brian
> ________________________________________
> From: SPSSX(r) Discussion [

> SPSSX-L@.UGA

> ] on behalf of David Marso [

> Dmmugaspss@

> ]
> Sent: Saturday, January 20, 2018 9:04 PM
> To:

> SPSSX-L@.UGA

> Subject: Re: interrater reliabillity - but for any pair of raters?
>
> Macro with nested !DO and CROSSTABS in body. Use OMS to capture output.
> AGGREGATE to get average. Or maybe Brian will post a link to his MATRIX
> program.
>
> UNTESTED. I'm on my phone...
>
> DEFINE ! AllPairsKappa( raters !CMDEND)
> !LET !CPY = !raters
> !DO ! j1 !IN ( !raters)
> !LET ! CPY= !TAIL( !CPY)
> !DO !j2 !IN (!CPY)
> CROSSTABS TABLES !j1 BY !j2 /STATISTICS KAPPA.
> !DOEND
> !DOEND
> !ENDDEFINE.
>
> SET UP OMS..RTFM FOR THAT...
>
> !AllPairsKappa raters= j1 j2 j3 ..... j30.
>
> =====================
> To manage your subscription to SPSSX-L, send a message to

> LISTSERV@.UGA

> (not to SPSSX-L), with no body text except the
> command. To leave the list, send the command
> SIGNOFF SPSSX-L
> For a list of commands to manage subscriptions, send the command
> INFO REFCARD
>
> =====================
> To manage your subscription to SPSSX-L, send a message to

> LISTSERV@.UGA

> (not to SPSSX-L), with no body text except the
> command. To leave the list, send the command
> SIGNOFF SPSSX-L
> For a list of commands to manage subscriptions, send the command
> INFO REFCARD
>
> =====================
> To manage your subscription to SPSSX-L, send a message to

> LISTSERV@.UGA

> (not to SPSSX-L), with no body text except the
> command. To leave the list, send the command
> SIGNOFF SPSSX-L
> For a list of commands to manage subscriptions, send the command
> INFO REFCARD

-----
Please reply to the list and not to my personal email.
Those desiring my consulting or training services please feel free to email me.
---
"Nolite dare sanctum canibus neque mittatis margaritas vestras ante porcos ne forte conculcent eas pedibus suis."
Cum es damnatorum possederunt porcos iens ut salire off sanguinum cliff in abyssum?"
--
Sent from: http://spssx-discussion.1045642.n5.nabble.com/

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

Please reply to the list and not to my personal email.
Those desiring my consulting or training services please feel free to email me.
---
"Nolite dare sanctum canibus neque mittatis margaritas vestras ante porcos ne forte conculcent eas pedibus suis."
Cum es damnatorum possederunt porcos iens ut salire off sanguinum cliff in abyssum?"

bdates

Re: interrater reliabillity - but for any pair of raters?

Great syntax, David! Thanks. One caution to Talma...the average of all two-rater kappa's is not equivalent to Fleiss' kappa calculated by Fleiss' formula.

Brian
________________________________________
From: SPSSX(r) Discussion [[hidden email]] on behalf of David Marso [[hidden email]]
Sent: Monday, January 22, 2018 1:15 AM
To: [hidden email]
Subject: Re: interrater reliabillity - but for any pair of raters?

Here is a tested version with some revision for efficiency
-Just passes the data once per main iteration rather than each pair-
along with OMS and AGGREGATE.

---SIMULATED DATA--- EXCLUDE FROM PRODUCTION RUN.
NEW FILE.
DATASET CLOSE ALL.
MATRIX.
SAVE TRUNC(UNIFORM(200,30)*4+1) / OUTFILE= * /VARIABLES j1 TO j30.
END MATRIX.
DATASET NAME raw.

DEFINE !AllPairsKappa( raters !CMDEND)
!LET !CPY = !raters
!DO !j1 !IN ( !raters)
!LET !CPY= !TAIL(!CPY)
!IF ( !CPY !NE !NULL) !THEN
CROSSTABS TABLES !j1 BY !CPY /STATISTICS KAPPA.
!IFEND
!DOEND
!ENDDEFINE.

* OMS.
DATASET DECLARE Kappa.
OMS
/SELECT TABLES /IF COMMANDS=['Crosstabs'] /DESTINATION VIEWER=NO.
OMS /SELECT TABLES
/IF COMMANDS=['Crosstabs'] SUBTYPES=['Symmetric Measures']
/DESTINATION FORMAT=SAV NUMBERED=TableNumber_
OUTFILE='Kappa' VIEWER=NO.
!AllPairsKappa raters = j1 j2 j3 j4 j5 j6 j7 j8 j9 j10 j11 j12 j13 j14 j15
j16 j17 j18 j19 j20 j21 j22 j23 j24 j25 j26 j27 j28
j29 j30.
OMSEND.

DATASET ACTIVATE Kappa.
DELETE VARIABLES TableNumber_ Command_ Subtype_ Label_ Var1.
RENAME VARIABLES Var2 =Kappa.
SELECT IF Kappa EQ 'Kappa'.
DATASET DECLARE agg.
AGGREGATE OUTFILE agg /BREAK Kappa /AvgKappa=MEAN( Value).

bdates wrote

> talma.claviger@

> ]
> Sent: Sunday, January 21, 2018 2:55 PM
> To:

> SPSSX-L@.UGA

> SPSSX-L@.UGA

> > im Auftrag von Dates, Brian <

> BDATES@

> >
> Gesendet: Sonntag, 21. Januar 2018 18:57
> An:

> SPSSX-L@.UGA

> SPSSX-L@.UGA

> ] on behalf of David Marso [

> Dmmugaspss@

> ]
> Sent: Saturday, January 20, 2018 9:04 PM
> To:

> SPSSX-L@.UGA

Talma

Re: interrater reliabillity - but for any pair of raters?

In reply to this post by David Marso

...wow...how cool is that? You were so helpful! Many thanks!!
Best
Talma

--
Sent from: http://spssx-discussion.1045642.n5.nabble.com/

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD