Login  Register

Re: Fleiss Kappa (Inter-rater reliability questions)

Posted by henryilian on Jan 09, 2013; 9:45pm
URL: http://spssx-discussion.165.s1.nabble.com/Fleiss-Kappa-Inter-rater-reliability-questions-tp5717299p5717303.html

Brian,

I have a related kappa question. I have a case reading instrument with 120 items. Most of the items have three answer choices, but some have two and some have four. There is also one with five. We did an inter-rater reliability study with five readers reading the same case. I haven't looked at the data yet, but I'll need to soon. I was going to use Cohen's Kappa, which SPSS gives you. I was going to set up the data file so that similar items--related topic areas with the same number of answer choices--were treated as separate cases, analogous to a set of MRIs rated by different clinicians, as I've seen in a number of articles. I'm not happy about that because the content for the item may not be similar enough. Also, this approach doesn't allow me to identify items that are troublesome.

Can you see a better way of approaching the task?

Thanks,

Henry

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Dates, Brian
Sent: Wednesday, January 09, 2013 4:07 PM
To: [hidden email]
Subject: Re: Fleiss Kappa (Inter-rater reliability questions)

I can send you a macro for Fleiss' kappa, which will give you the overall as well as the category kappa's, based on Fleiss' 1971 Psych. Bulletin article.  It uses, however, the updated 1979 formula for variance. How many items will the raters categorize?  You have other options, e.g., Krippendorf's alpha and Gwet's AC1 statistic.  Although Fleiss is the most used, there's some use of these as well in the literature.  If you're interested in Cohen's kappa, there are some multiple rater models, most notably Light (averages all the two-rater kappas) and Hubert (averages the two-rater Pe's).  Let me know which you want, and I'll get them off to you.

Brian

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of BAM86
Sent: Wednesday, January 09, 2013 3:25 PM
To: [hidden email]
Subject: Fleiss Kappa (Inter-rater reliability questions)

Hi all,

I am conducting research and need some help with doing the Fleiss Kappa
analysis.

First and foremost let me give a bit of a layout of the study. We have a
sample of 75 students in the social sciences who were asked to categorize
potential items for a new measure based on descriptions of 2 theoretical
constructs. This was meant to be a preliminary check to help with construct
validity and take it a step beyond just the research team validating items
initially.

The items are supposed to characterize coping behaviours following romantic
loss. The constructs are loss-oriented coping and restoration-oriented
coping, however, we also gave the option for participants to say potential
items were both types of coping behaviour or something else altogether
(basically an 'other' option).

Therefore we have 75 participants categorizing items into one of 4
categories. We want to have some measure of the inter-rater reliability of
all these participants though. My understanding is that Fleiss Kappa would
be an appropriate tool for this purpose, however, if anyone is aware of
other statistics which may be a better fit for our model then please let me
know.

I also need to the syntax for the Fleiss Kappa statistic if anyone has it
available. Thanks in advance for any help people can provide.



--
View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Fleiss-Kappa-Inter-rater-reliability-questions-tp5717299.html
Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD


Confidentiality Notice: This e-mail communication, and any attachments, contains confidential and privileged information for the exclusive use of the recipient(s) named above. If you are not an intended recipient, or the employee or agent responsible to deliver it to an intended recipient, you are hereby notified that you have received this communication in error and that any review, disclosure, dissemination, distribution or copying of it or its contents is prohibited. If you have received this communication in error, please notify me immediately by replying to this message and delete this communication from your computer. Thank you.

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD