SPSSX Discussion

Cohen's Kappa for multiple raters

Classic

List

Threaded

5 messages Options

Paul Mcgeoghan

Cohen's Kappa for multiple raters

Hi,

I am using the syntax below from Raynald's SPSS Tools website:
http://www.spsstools.net/Syntax/Matrix/CohensKappa.txt

In my case I have 6 raters rating 5 subjects and there are 2 categories so the data is as below:

Subj rater1 rater2 rater3 rater4 rater5 rater6
1 2 2 2 2 2 2
2 2 2 2 2 2 2
3 2 2 2 2 2 2
4 1 2 2 2 2 2
5 2 2 2 2 2 2

This gives a value of -.0345 which indicates no agreement according to the following article:
http://en.wikipedia.org/wiki/Fleiss%27_kappa

Most of the raters agree in the above table so why is Cohen's Kappa negative, indicating no
agreement.

Also in the output, it gives Cohen's Kappa where Kappa is -.0345 and
Cohen's Kappa Fleiss - adjusted standard error as .1155

Which value do I use for Cohen's Kappa among multiple raters, is it -.0345 or .1155?

Paul

==================
Paul McGeoghan,
Application support specialist (Statistics and Databases),
University Infrastructure Group (UIG),
Information Services,
Cardiff University.
Tel. 02920 (875035).

bdates

Re: Cohen's Kappa for multiple raters

Paul,

The negative kappa is an indication that the degree of agreement is less
than would be expected by chance. What you've run into is the paradox that
occurs with kappa and most kappa-like statistics that Feinstein and Cichetti
first mentioned. As marginal homogeneity decreases (trait prevalence
becomes more skewed), the value of kappa decreases in spite of the fact that
rater agreement might be very high. Scott's pi, Cohen's kappa, and Conger's
kappa were all developed based on the assumption of marginal homogeneity.
The greater the deviation, the more kappa is diminished. This is one of the
criticisms of kappa type statistics. Regarding your second question, use the
kappa value of -0.0345. The 0.1155 is the asymptotic standard error for use
in computing confidence intervals.

Brian

Brian G. Dates, Director of Quality Assurance
Southwest Counseling and Development Services
1700 Waterman
Detroit, Michigan 48209
Telephone: 313.841.7442
FAX: 313.841.4470
email: [hidden email]

> -----Original Message-----
> From: Paul Mcgeoghan [SMTP:[hidden email]]
> Sent: Thursday, November 09, 2006 11:33 AM
> To: [hidden email]
> Subject: Cohen's Kappa for multiple raters
>
> Hi,
>
> I am using the syntax below from Raynald's SPSS Tools website:
> http://www.spsstools.net/Syntax/Matrix/CohensKappa.txt
>
> In my case I have 6 raters rating 5 subjects and there are 2 categories so
> the data is as below:
>
> Subj rater1 rater2 rater3 rater4 rater5
> rater6
> 1 2 2 2 2
> 2 2
> 2 2 2 2 2
> 2 2
> 3 2 2 2 2
> 2 2
> 4 1 2 2 2
> 2 2
> 5 2 2 2 2
> 2 2
>
> This gives a value of -.0345 which indicates no agreement according to the
> following article:
> http://en.wikipedia.org/wiki/Fleiss%27_kappa
>
> Most of the raters agree in the above table so why is Cohen's Kappa
> negative, indicating no
> agreement.
>
> Also in the output, it gives Cohen's Kappa where Kappa is -.0345 and
> Cohen's Kappa Fleiss - adjusted standard error as .1155
>
> Which value do I use for Cohen's Kappa among multiple raters, is it -.0345
> or .1155?
>
> Paul
>
>
> ==================
> Paul McGeoghan,
> Application support specialist (Statistics and Databases),
> University Infrastructure Group (UIG),
> Information Services,
> Cardiff University.
> Tel. 02920 (875035).
>
>

Confidentiality Notice for Email Transmissions: The information in this
message is confidential and may be legally privileged. It is intended solely
for the addressee. Access to this message by anyone else is unauthorised.
If you are not the intended recipient, any disclosure, copying, or
distribution of the message, or any action or omission taken by you in
reliance on it, is prohibited and may be unlawful. Please immediately
contact the sender if you have received this message in error. Thank you.

Meyer, Gregory J

Re: Cohen's Kappa for multiple raters

In reply to this post by Paul Mcgeoghan

Paul, the coefficient is so low because there is almost no measurable
individual differences in your subjects. They all receive values of 2 by
five of the six raters. Only one subject receives a value of 1 by just
one of the raters. Kappa, or any coefficient of agreement (e.g.,
correlation), would be impossible to compute if you just looked at data
from Raters 2 through 6 because there would be no variability at all
(i.e., all scores are a constant). Unless the rating scale needs to be
applied in such a homogeneous sample, the way to address this is to get
a larger and more diverse sample of subjects included in the analyses.

Greg

| -----Original Message-----
| From: SPSSX(r) Discussion [mailto:[hidden email]]
| On Behalf Of Paul Mcgeoghan
| Sent: Thursday, November 09, 2006 11:33 AM
| To: [hidden email]
| Subject: Cohen's Kappa for multiple raters
|
| Hi,
|
| I am using the syntax below from Raynald's SPSS Tools website:
| http://www.spsstools.net/Syntax/Matrix/CohensKappa.txt
|
| In my case I have 6 raters rating 5 subjects and there are 2
| categories so the data is as below:
|
| Subj rater1 rater2 rater3 rater4
| rater5 rater6
| 1 2 2 2
| 2 2 2
| 2 2 2 2
| 2 2 2
| 3 2 2 2
| 2 2 2
| 4 1 2 2
| 2 2 2
| 5 2 2 2
| 2 2 2
|
| This gives a value of -.0345 which indicates no agreement
| according to the following article:
| http://en.wikipedia.org/wiki/Fleiss%27_kappa
|
| Most of the raters agree in the above table so why is Cohen's
| Kappa negative, indicating no
| agreement.
|
| Also in the output, it gives Cohen's Kappa where Kappa is -.0345 and
| Cohen's Kappa Fleiss - adjusted standard error as .1155
|
| Which value do I use for Cohen's Kappa among multiple raters,
| is it -.0345 or .1155?
|
| Paul
|
|
| ==================
| Paul McGeoghan,
| Application support specialist (Statistics and Databases),
| University Infrastructure Group (UIG),
| Information Services,
| Cardiff University.
| Tel. 02920 (875035).
|

Meel

Re: Cohen's Kappa for multiple raters

In reply to this post by bdates

Brian,

You wrote: "As marginal homogeneity decreases (trait prevalence becomes more
skewed), the value of kappa decreases". Warrens (2010) says: "A paradox
associated with Cohen’s kappa is that, for a fixed value of the proportion
of observed agreement, tables with marginal asymmetry produce
higher values of kappa than tables with homogeneous
marginals"(https://openaccess.leidenuniv.nl/bitstream/handle/1887/16310/Warrens_2010_JoC_27_322_332.pdf?sequence=2).
What am I missing? Can you explain a bit more?

--
Sent from: http://spssx-discussion.1045642.n5.nabble.com/

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

Rich Ulrich

Re: Cohen's Kappa for multiple raters

I think you want to generate a few tables, and pay close

attention to what the author means by "symmetrical"

marginals (both raters have the same skew) and asymmetrical

(different skew). And the notion of "agreement".

A 2x2 table (81, 9; 9,1) - has 82% agreement, with k=0.

Whereas, 82% agreement is FINE for a table with "uniform"

marginals.

I used my own programs for generating 2x2 kappas so that

I would always see the McNemar's test for "difference",

i.e., how much the two raters differed in the off-diagonal.

The table (81,18; 0,1) has the same agreement, YY+NN, as

above, but GREAT disagreement in comparison of YN vs NY.

I was disappointed by a paper where the authors showed a

list of cases (for some purpose) and never noted in the article

that the 7 of the 8 cases of "disagreement", which mattered

in the article, were accounted for by fewer diagnoses assigned

by one rater.

Unless there was a set of highly parallel comparisons, I

avoided using kappas for tables larger than 2x2. (Collapse

categories to get several 2x2 tables, if you want assessments.)

Rich Ulrich

From: SPSSX(r) Discussion <[hidden email]> on behalf of Meel <[hidden email]>
Sent: Friday, November 29, 2019 8:42 AM
To: [hidden email] <[hidden email]>
Subject: Re: Cohen's Kappa for multiple raters

===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD