At the suggestion of Bruce Weaver, I'm including a summary of last week's discussion of the SPSS Python extension for Fleiss' kappa. In 1971, Fleiss (Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76 (5), 378-382.)published the first work on his kappa, the first inter-rater agreement statistic for nominal data which would accommodate more than 2 raters. In it, he presented formulas for the standard error of kappa for both the overall kappa and the individual kappas for each rating category. In 1979, he published again on kappa (Fleiss, J. L., Nee, J. C. M., & Landis, J. R. (1979). Large sample variance of kappa in the case of different sets of raters. Psychological Bulletin, 86, 974-977.) In the second article, he altered the formulas for both overall and category standard errors. The formula for category standard errors was changed from one in which the proportions of responses in each category was a factor to a much simpler approach which was dependent only on the number of raters and the number of subjects/items. This resulted in the same standard error for all categories. Many syntaxes available online, including mine, may have changed the standard error for the overall kappa but not for the category kappas. The SPSS Python extension did, indeed, change the formulas for each and therefore uses the most recent approach.
As a result, I am altering my syntax to reflect the most recent approach to calculation of standard errors. The full thread of responses can be examined by using the link below. In the meantime, my thanks to both Jon Peck and Bruce Weaver for their input and suggestions, including communications with authors of syntax in other programs - SAS and STATA.
http://spssx-discussion.1045642.n5.nabble.com/SPSS-Python-Extension-for-Fleiss-Kappa-tc5734963.html=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD
Locked
