http://spssx-discussion.165.s1.nabble.com/SPSS-Python-Extension-for-Fleiss-Kappa-tp5734963p5734998.html
Brian, when you've finished sorting it all out, I hope you'll post a summary.
data & output to Nabble again too.
my Amazon.ca wish list.
> Thanks, Jon. I have all those works. I'm in the process of making changes
> >
> Subject: Re: SPSS Python Extension for Fleiss Kappa
>
> Hi Brian. I also sent Daniel Klein an e-mail alerting him to this thread.
> He does not use SPSS, and is probably not going to join this list himself.
> But he did ask me to post the following comments he sent me.
>
> Cheers,
> Bruce
> -------------------------------------
>
> First, note that Gwet (2008, 2014) uses an approach to statistical
> inference
> that is quite different from anything that has been proposed so far in the
> area of inter-rater agreement. He calls what is usually done the
> "model-based" approach. Here, a hypothetical distribution of the kappa
> coefficient under H0 is used to derive test statistics for testing against
> 0. Such approaches are not necessarily valid for confidence interval
> construction (also see Reichenheim 2004). So I suppose there might be a
> reason why StataCorp decided not to return a standard error and confidence
> interval in their -kap- coammand. What
> Gwet (2008, 2014) suggest instead is a "design-based" approach that is
> based
> on finite sample theory and that is implemented in -kappaetc-. This
> explains
> the difference between the standard error that Brian obtains for combined
> kappa (0.05770; which is what you confirm using the -kap- command) and the
> SE reported by Gwet's SAS macro (0.08132) and -kappaetc- (0.08223). To say
> more about the (small) difference between the latter two, I would need to
> know which macro was used to produce the results. Brian states that he
> used
> INTER_RATER.MAC but does neither state the software version nor where it
> is
> from.
>
> [BW: I believe the loop Daniel mentions in the next paragraph is
> referring
> to the Stata output I uploaded here:
>
http://spssx-discussion.1045642.n5.nabble.com/file/t7186/Fleiss_kappa_problem.txt.]
>
> When you program the loop to get kappa for each category, note that
> -kappaetc- does not(!) report the same standard error every time. It
> differs
> for each category. It is the SE you obtain from -kap- that is constant
> across categories. As I have said, I do not know whether this SE is
> trustworthy for confidence interval construction but it seems the Python
> extension uses the very same approach. Note that it is documented in [R]
> kappa. However, the SEs reported by -kappaetc- do not match Brian's or
> those
> from the SAS macro. I have no explanation for that. All I can tell you is
> that -kappaetc- does nothing special here, i.e., it retains Gwet's
> "design-based" approach. It seems the SAS macro and Brian have implemented
> another formula to get the SE for
> each category, separately (and perhaps there is a reason for this). It
> seems
> a bit odd to me that the SEs are so much higher than the combined one,
> given
> that they are all based on the same number of subjects. Also, I find it
> somehow counterintuitive to clearly rejecting the null in the combined
> coefficient when only one of the category-wise coefficients is
> statistically
> significant different from zero.
>
> I notice that the help file for -kappaetc- might be misleading or
> incomplete
> concerning differences to Stata's -kap- and -kappa-. Here is the already
> revised paragraph that will be included in an updated version of
> -kappaetc-
> (hopefuly towards the end of the year).
>
> --- Start of excerpt from help for kappaetc ----
>
> Relation to official Stata's kap and kappa commands
>
> The percent agreement that is reported by kappaetc is the same as the
> observed agreement returned by Stata's kap command. The latter, however,
> does not provide a standard error or confidence intervals for the
> coefficient.
>
> For two unique raters and no missing ratings both, Stata's kap and the
> kappaetc command, estimate the same Cohen's kappa coefficient. The
> standard
> error and p-value will differ between the two results, though. Stata's kap
> command implements a model-based (theoretical) approximate formula for the
> standard error and reports a one-sided test using the standard normal
> distribution. kappaetc implements a design-based formula to obtain the
> standard error and reports a two-sided t test with n-1 degrees of freedom.
> kappaetc will, additionally, provide a confidence interval.
>
> For more than two (nonunique) raters and no missing ratings (i.e. a
> constant
> number of raters) both, Stata's kap (or kappa) command and kappaetc
> (possibly with the frequency option) estimate the same Fleiss' kappa
> coefficient. kappaetc will, however, not report a kappa value for each
> rating category. Standard errors and p-values will differ between the
> commands for the same reasons explained in the two unique raters case
> above.
>
> --- End of excerpt from help for kappaetc ----
>
> A last minor comment on category 4 not being used by either rater. In such
> a
> case it is useful to specify all possible rating categories with the
> -categories()- option of -kappaetc-. Failing to do so results in wrong
> estimates for Gwet's AC and the Brennan and Prediger coefficient, both of
> which depend on the number of rating categories. Note that weighted kappa
> might also be off.
>
> Best
> Daniel
>
>
> Gwet, K. L. (2014). Handbook of Inter-Rater Reliability. Gaithersburg,
> MD: Advanced Analytics, LLC.
>
> Gwet, K. L. (2008). Computing inter-rater reliability and its variance
> in the presence of high agreement. British Journal of Mathematical and
> Statistical Psycholgy, 61, 29-48.
>
> Reichenheim, M. E. (2004). Confidence intervals for the kappa
> statistic. The Stata Journal, 4, 421-428.
>
>
>
> bdates wrote
>> Thanks, Bruce. I've sent an email to him. He mentions Gwet in his
>> acknowledgments, so it'll be interesting what Gwet says about all of
>> this.
>>
>> Brian
>>
>> ________________________________________
>> From: SPSSX(r) Discussion [
>
>> SPSSX-L@.UGA
>
>> ] on behalf of Bruce Weaver [
>
>> bruce.weaver@
>
>> ]
>> Sent: Monday, October 09, 2017 4:45 PM
>> To:
>
>> SPSSX-L@.UGA
>
>> Subject: Re: SPSS Python Extension for Fleiss Kappa
>>
>> Hi Brian. The author of kappaetc can be reached via the e-mail address
>> at
>> the bottom of that text file I uploaded. You could always ask him
>> directly
>> what method(s) he used.
>>
>> Cheers,
>> Bruce
>>
>>
>>
>> bdates wrote
>>> Bruce,
>>>
>>> Thanks for this. I'm wondering what formulae they're using for category
>>> SE's. I'm going to check with other sources and get back to you. This is
>>> certainly a conundrum. We have multiple results from multiple sources. I
>>> didn't send this along, but I also used the application from
>>> RealStatistics in Excel and got the same results as my syntax and that
>>> of
>>> Gwet's SAS solution. I'm using the updated 1979 computation for the
>>> kappa's and standard errors. Maybe they're using the 1971 version.
>>>
>>> Brian
>>>
>>>
>>> ________________________________________
>>> From: SPSSX(r) Discussion [
>>
>>> SPSSX-L@.UGA
>>
>>> ] on behalf of Bruce Weaver [
>>
>>> bruce.weaver@
>>
>>> ]
>>> Sent: Monday, October 09, 2017 2:23 PM
>>> To:
>>
>>> SPSSX-L@.UGA
>>
>>> Subject: Re: SPSS Python Extension for Fleiss Kappa
>>>
>>> Thanks Brian. I don't know if this will helpful to you or not, but I've
>>> uploaded (in Nabble) a text file containing results from some analyses
>>> carried out using kappaetc, a user-written program for Stata.
>>> Unfortunately, kappaetc does not report a kappa for each category
>>> separately. But with a little programming, I was able to obtain those.
>>> But
>>> the way I did it yielded the same SE for every category, and it matched
>>> the
>>> value shown by the Python extension command for SPSS.
>>>
>>> Below my output, I appended the help file info for kappaetc (including a
>>> reference list), in case it is helpful to you.
>>>
>>> Cheers,
>>> Bruce
>>>
>>> Fleiss_kappa_problem.txt
>>> <
http://spssx-discussion.1045642.n5.nabble.com/file/t7186/Fleiss_kappa_problem.txt>>>>
>>>
>>> bdates wrote
>>>> I'm uploading a Word document with the results of my syntax, Kilem
>>>> Gwet's
>>>> SAS
>>>> macro, and the SPSS Python extension, in that order. I'm also uploading
>>>> an
>>>> SPSS data file with the raw data. Notice the Standard Error column in
>>>> the
>>>> Python extension solution. All values are the same.
>>>>
>>>> Brian Fleiss_kappa_Discrepancies.docx
>>>> <
http://spssx-discussion.1045642.n5.nabble.com/file/t47633/Fleiss_kappa_Discrepancies.docx>>>>> irr_test_data.sav
>>>> <
http://spssx-discussion.1045642.n5.nabble.com/file/t47633/irr_test_data.sav>>>>>
>>>>
>>>>
>>>> --
>>>> Sent from:
http://spssx-discussion.1045642.n5.nabble.com/>>>>
>>>> =====================
>>>> To manage your subscription to SPSSX-L, send a message to
>>>
>>>> LISTSERV@.UGA
>>>
>>>> (not to SPSSX-L), with no body text except the
>>>> command. To leave the list, send the command
>>>> SIGNOFF SPSSX-L
>>>> For a list of commands to manage subscriptions, send the command
>>>> INFO REFCARD
>>>
>>>
>>>
>>>
>>>
>>> -----
>>> --
>>> Bruce Weaver
>>
>>> bweaver@
>>
>>>
http://sites.google.com/a/lakeheadu.ca/bweaver/>>>
>>> "When all else fails, RTFM."
>>>
>>> NOTE: My Hotmail account is not monitored regularly.
>>> To send me an e-mail, please use the address shown above.
>>>
>>> --
>>> Sent from:
http://spssx-discussion.1045642.n5.nabble.com/>>>
>>> =====================
>>> To manage your subscription to SPSSX-L, send a message to
>>
>>> LISTSERV@.UGA
>>
>>> (not to SPSSX-L), with no body text except the
>>> command. To leave the list, send the command
>>> SIGNOFF SPSSX-L
>>> For a list of commands to manage subscriptions, send the command
>>> INFO REFCARD
>>>
>>> =====================
>>> To manage your subscription to SPSSX-L, send a message to
>>
>>> LISTSERV@.UGA
>>
>>> (not to SPSSX-L), with no body text except the
>>> command. To leave the list, send the command
>>> SIGNOFF SPSSX-L
>>> For a list of commands to manage subscriptions, send the command
>>> INFO REFCARD
>>
>>
>>
>>
>>
>> -----
>> --
>> Bruce Weaver
>
>> bweaver@
>
>>
http://sites.google.com/a/lakeheadu.ca/bweaver/>>
>> "When all else fails, RTFM."
>>
>> NOTE: My Hotmail account is not monitored regularly.
>> To send me an e-mail, please use the address shown above.
>>
>> --
>> Sent from:
http://spssx-discussion.1045642.n5.nabble.com/>>
>> =====================
>> To manage your subscription to SPSSX-L, send a message to
>
>> LISTSERV@.UGA
>
>> (not to SPSSX-L), with no body text except the
>> command. To leave the list, send the command
>> SIGNOFF SPSSX-L
>> For a list of commands to manage subscriptions, send the command
>> INFO REFCARD
>>
>> =====================
>> To manage your subscription to SPSSX-L, send a message to
>
>> LISTSERV@.UGA
>
>> (not to SPSSX-L), with no body text except the
>> command. To leave the list, send the command
>> SIGNOFF SPSSX-L
>> For a list of commands to manage subscriptions, send the command
>> INFO REFCARD
>
>
>
>
>
> -----
> --
> Bruce Weaver
> command. To leave the list, send the command
> command. To leave the list, send the command
> command. To leave the list, send the command
command. To leave the list, send the command