Cohen's kappa coefficient (often get low coefficients)
Locked
Cohen's kappa coefficient (often get low coefficients)
 
|
Hi, A common question I get asked is why does Cohen's Kappa give low coefficient when it can range between 0 and 1. I use the Kappa syntax obtained from Raynald's site: As an example, I have data with 13 raters and 4 categories and 264 items. The overall value of Cohen's Kappa is .0422 and a p-value of .0079. This is quite a low value of Kappa, given that it can range between 0 and 1? The Percent of Rater agreement is .5283 The value of k for individual categories is Category k 1 -.0035 2 .1410 3 .0059 4 .0489 Categories 2 and 4 are significant. Very few raters indicate category 1 but is it possible to have a negative Kappa when it ranges between 0 and 1. Can anyone offer advice on this? Thanks, Paul Paul McGeoghan, Application Support Specialist (Statistics and Databases), University Infrastructure Group (UIG), Information Services Cardiff University Tel. 02920 875035 |
|
|
Paul,
Negative kappa values are unusual, but do occur. Negative values indicate that raters agreed less than would have been expected by chance. Brian ________________________________ From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Paul McGeoghan Sent: Wednesday, April 02, 2008 7:17 AM To: [hidden email] Subject: Cohen's kappa coefficient (often get low coefficients) Hi, A common question I get asked is why does Cohen's Kappa give low coefficient when it can range between 0 and 1. I use the Kappa syntax obtained from Raynald's site: http://www.spsstools.net/Syntax/Matrix/CohensKappa.txt As an example, I have data with 13 raters and 4 categories and 264 items. The overall value of Cohen's Kappa is .0422 and a p-value of .0079. This is quite a low value of Kappa, given that it can range between 0 and 1? The Percent of Rater agreement is .5283 The value of k for individual categories is Category k 1 -.0035 2 .1410 3 .0059 4 .0489 Categories 2 and 4 are significant. Very few raters indicate category 1 but is it possible to have a negative Kappa when it ranges between 0 and 1. Can anyone offer advice on this? Thanks, Paul Paul McGeoghan, Application Support Specialist (Statistics and Databases), University Infrastructure Group (UIG), Information Services Cardiff University Tel. 02920 875035 ======= To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD ====================To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD |
|
David Nichols of SPSS wrote an article in 1999 for Keywords titled "HOW
NEGATIVE RELIABILITY COEFFICIENTS CAN OCCUR: My Coefficient a is Negative!" that could help. WMB Statistical Services ========================================= mailto:[hidden email] http://home.earthlink.net/~info.statman ========================================= Virus Scan Notice: This email is certified to be virus free. -----Original Message----- From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Dates, Brian Sent: Wednesday, April 02, 2008 8:17 AM To: [hidden email] Subject: Re: Cohen's kappa coefficient (often get low coefficients) Paul, Negative kappa values are unusual, but do occur. Negative values indicate that raters agreed less than would have been expected by chance. Brian ________________________________ From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Paul McGeoghan Sent: Wednesday, April 02, 2008 7:17 AM To: [hidden email] Subject: Cohen's kappa coefficient (often get low coefficients) Hi, A common question I get asked is why does Cohen's Kappa give low coefficient when it can range between 0 and 1. I use the Kappa syntax obtained from Raynald's site: http://www.spsstools.net/Syntax/Matrix/CohensKappa.txt As an example, I have data with 13 raters and 4 categories and 264 items. The overall value of Cohen's Kappa is .0422 and a p-value of .0079. This is quite a low value of Kappa, given that it can range between 0 and 1? The Percent of Rater agreement is .5283 The value of k for individual categories is Category k 1 -.0035 2 .1410 3 .0059 4 .0489 Categories 2 and 4 are significant. Very few raters indicate category 1 but is it possible to have a negative Kappa when it ranges between 0 and 1. Can anyone offer advice on this? Thanks, Paul Paul McGeoghan, Application Support Specialist (Statistics and Databases), University Infrastructure Group (UIG), Information Services Cardiff University Tel. 02920 875035 ======= To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD ======= To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD
Will
Statistical Services ============ info.statman@earthlink.net http://home.earthlink.net/~z_statman/ ============ |
Re: Cohen's kappa coefficient (often get low coefficients)
|
|
Kappa is very dependent on the distribution of the responses. Because it
corrects for chance agreement, the more likely a chance agreement, the lower Kappa tends to be. If you have a dichotomous item where 90% of the respondents sat=y "A", it won't take much disagreement to cause kappa to be low. Paul R. Swank, Ph.D. Professor and Director of Research Children's Learning Institute University of Texas Health Science Center - Houston -----Original Message----- From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Statman (WMB) Sent: Wednesday, April 02, 2008 7:56 AM To: [hidden email] Subject: Re: Cohen's kappa coefficient (often get low coefficients) David Nichols of SPSS wrote an article in 1999 for Keywords titled "HOW NEGATIVE RELIABILITY COEFFICIENTS CAN OCCUR: My Coefficient a is Negative!" that could help. WMB Statistical Services ========================================= mailto:[hidden email] http://home.earthlink.net/~info.statman ========================================= Virus Scan Notice: This email is certified to be virus free. -----Original Message----- From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Dates, Brian Sent: Wednesday, April 02, 2008 8:17 AM To: [hidden email] Subject: Re: Cohen's kappa coefficient (often get low coefficients) Paul, Negative kappa values are unusual, but do occur. Negative values indicate that raters agreed less than would have been expected by chance. Brian ________________________________ From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Paul McGeoghan Sent: Wednesday, April 02, 2008 7:17 AM To: [hidden email] Subject: Cohen's kappa coefficient (often get low coefficients) Hi, A common question I get asked is why does Cohen's Kappa give low coefficient when it can range between 0 and 1. I use the Kappa syntax obtained from Raynald's site: http://www.spsstools.net/Syntax/Matrix/CohensKappa.txt As an example, I have data with 13 raters and 4 categories and 264 items. The overall value of Cohen's Kappa is .0422 and a p-value of .0079. This is quite a low value of Kappa, given that it can range between 0 and 1? The Percent of Rater agreement is .5283 The value of k for individual categories is Category k 1 -.0035 2 .1410 3 .0059 4 .0489 Categories 2 and 4 are significant. Very few raters indicate category 1 but is it possible to have a negative Kappa when it ranges between 0 and 1. Can anyone offer advice on this? Thanks, Paul Paul McGeoghan, Application Support Specialist (Statistics and Databases), University Infrastructure Group (UIG), Information Services Cardiff University Tel. 02920 875035 ======= To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD ======= To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD |
«
Return to SPSSX Discussion
|
1 view|%1 views
| Free forum by Nabble | Edit this page |