Posted by
Rich Ulrich on
Nov 30, 2019; 12:00am
URL: http://spssx-discussion.165.s1.nabble.com/Cohen-s-Kappa-for-multiple-raters-tp1072037p5738639.html
I think you want to generate a few tables, and pay close
attention to what the author means by "symmetrical"
marginals (both raters have the same skew) and asymmetrical
(different skew). And the notion of "agreement".
A 2x2 table (81, 9; 9,1) - has 82% agreement, with k=0.
Whereas, 82% agreement is FINE for a table with "uniform"
marginals.
I used my own programs for generating 2x2 kappas so that
I would always see the McNemar's test for "difference",
i.e., how much the two raters differed in the off-diagonal.
The table (81,18; 0,1) has the same agreement, YY+NN, as
above, but GREAT disagreement in comparison of YN vs NY.
I was disappointed by a paper where the authors showed a
list of cases (for some purpose) and never noted in the article
that the 7 of the 8 cases of "disagreement", which mattered
in the article, were accounted for by fewer diagnoses assigned
by one rater.
Unless there was a set of highly parallel comparisons, I
avoided using kappas for tables larger than 2x2. (Collapse
categories to get several 2x2 tables, if you want assessments.)
--
Rich Ulrich
Brian,
You wrote: "As marginal homogeneity decreases (trait prevalence becomes more
skewed), the value of kappa decreases". Warrens (2010) says: "A paradox
associated with Cohen’s kappa is that, for a fixed value of the proportion
of observed agreement, tables with marginal asymmetry produce
higher values of kappa than tables with homogeneous
marginals"(
https://openaccess.leidenuniv.nl/bitstream/handle/1887/16310/Warrens_2010_JoC_27_322_332.pdf?sequence=2).
What am I missing? Can you explain a bit more?
--
Sent from:
http://spssx-discussion.1045642.n5.nabble.com/
=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD
=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD