SPSS Python Extension for Fleiss Kappa

[daniel klein]

Brian's comment concerning the advantages of Krippendorff's alpha might not
be quite accurate. Although the original author implies that his coefficient
is somehow superior to others in the literature, Gwet (2011, 2014) shows
that (i) Krippendorff's alpha can be computed in a very similar fashion to
other coefficients and (ii) other coefficients can be modified to handle
ordinal, interval, ratio (, circular, bipoloar, ...) ratings, just like
Krippendorff's alpha can. The method for handling missing data can also be
extended to other coefficients. Details are given in Gwet (2014) and are
implemented in -kappaetc- for Stata (code is completely open source).

I would be interested in details about the "Krippendorf macros" that Art
Kendall mentions, especially with respect to complex designs.

Best
Daniel

Gwet, K. L. (2014). Handbook of Inter-Rater Reliability. Gaithersburg, MD:
Advanced Analytics, LLC.

Gwet, K. L. (2011). On The Krippendorff’s Alpha Coefficient. Manuscript
submitted for publication in Communication Methods and Measures (August
2011).
https://www.researchgate.net/publication/267823285_On_Krippendorff's_Alpha_Coefficient



--
Sent from: http://spssx-discussion.1045642.n5.nabble.com/

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[bdates]

Daniel,

Thanks for chiming in. Actually I've never been a Krippendorff devotee. He really hasn't added much to Fleiss, and as he says, with greater numbers of raters and items, alpha and kappa are the same. I almost never use ordinal, interval, and certainly not ratio data for agreement. I just rely on Fleiss and Cohen (1973) on the equivalence of weighted kappa and the ICC. I've become more intrigued with approaches like Aikin or Ubersax (who advocates a latent class analysis approach.) I've also developed the approach that the choice of agreement stats depends on the nature of the data, and what we know. For example, are all categories equally probable, is there an underlying categorical structure that's not uniform, should raters be taken into consideration? I think each agreement study is different, and one size doesn't necessarily fit all.

Brian

________________________________________
From: SPSSX(r) Discussion [[hidden email]] on behalf of daniel klein [[hidden email]]
Sent: Friday, October 20, 2017 5:25 AM
To: [hidden email]
Subject: Re: SPSS Python Extension for Fleiss Kappa

Brian's comment concerning the advantages of Krippendorff's alpha might not
be quite accurate. Although the original author implies that his coefficient
is somehow superior to others in the literature, Gwet (2011, 2014) shows
that (i) Krippendorff's alpha can be computed in a very similar fashion to
other coefficients and (ii) other coefficients can be modified to handle
ordinal, interval, ratio (, circular, bipoloar, ...) ratings, just like
Krippendorff's alpha can. The method for handling missing data can also be
extended to other coefficients. Details are given in Gwet (2014) and are
implemented in -kappaetc- for Stata (code is completely open source).

I would be interested in details about the "Krippendorf macros" that Art
Kendall mentions, especially with respect to complex designs.

Best
Daniel

Gwet, K. L. (2014). Handbook of Inter-Rater Reliability. Gaithersburg, MD:
Advanced Analytics, LLC.

Gwet, K. L. (2011). On The Krippendorff’s Alpha Coefficient. Manuscript
submitted for publication in Communication Methods and Measures (August
2011).
https://www.researchgate.net/publication/267823285_On_Krippendorff's_Alpha_Coefficient



--
Sent from: http://spssx-discussion.1045642.n5.nabble.com/

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD