Posted by
bdates on
Jan 09, 2013; 10:44pm
URL: http://spssx-discussion.165.s1.nabble.com/Fleiss-Kappa-Inter-rater-reliability-questions-tp5717299p5717310.html
I used Fleiss' 1979 variance, and I don't know which one ('71 or '79) David used. I added syntax for category kappa's, which Fleiss developed, and an upper and lower 95% confidence interval on all of them. I also report on the number of responses each rater has in each category, the number of agreements each rater has with the other rater(s) in each category, the number of subjects, number of categories, number of raters, and raw proportion of agreement. So there's more information, but the overall kappa's still the same.
B
-----Original Message-----
From: SPSSX(r) Discussion [mailto:
[hidden email]] On Behalf Of Bruce Weaver
Sent: Wednesday, January 09, 2013 4:42 PM
To:
[hidden email]
Subject: Re: Fleiss Kappa (Inter-rater reliability questions)
Brian, how does your macro compare to this one written by David Nichols?
http://imaging.mrc-cbu.cam.ac.uk/statswiki/FAQ/kappa/multipleThanks,
Bruce
Dates, Brian wrote
> I can send you a macro for Fleiss' kappa, which will give you the overall
> as well as the category kappa's, based on Fleiss' 1971 Psych. Bulletin
> article. It uses, however, the updated 1979 formula for variance. How
> many items will the raters categorize? You have other options, e.g.,
> Krippendorf's alpha and Gwet's AC1 statistic. Although Fleiss is the most
> used, there's some use of these as well in the literature. If you're
> interested in Cohen's kappa, there are some multiple rater models, most
> notably Light (averages all the two-rater kappas) and Hubert (averages the
> two-rater Pe's). Let me know which you want, and I'll get them off to
> you.
>
> Brian
>
> -----Original Message-----
> From: SPSSX(r) Discussion [mailto:
> SPSSX-L@.UGA
> ] On Behalf Of BAM86
> Sent: Wednesday, January 09, 2013 3:25 PM
> To:
> SPSSX-L@.UGA
> Subject: Fleiss Kappa (Inter-rater reliability questions)
>
> Hi all,
>
> I am conducting research and need some help with doing the Fleiss Kappa
> analysis.
>
> First and foremost let me give a bit of a layout of the study. We have a
> sample of 75 students in the social sciences who were asked to categorize
> potential items for a new measure based on descriptions of 2 theoretical
> constructs. This was meant to be a preliminary check to help with
> construct
> validity and take it a step beyond just the research team validating items
> initially.
>
> The items are supposed to characterize coping behaviours following
> romantic
> loss. The constructs are loss-oriented coping and restoration-oriented
> coping, however, we also gave the option for participants to say potential
> items were both types of coping behaviour or something else altogether
> (basically an 'other' option).
>
> Therefore we have 75 participants categorizing items into one of 4
> categories. We want to have some measure of the inter-rater reliability of
> all these participants though. My understanding is that Fleiss Kappa would
> be an appropriate tool for this purpose, however, if anyone is aware of
> other statistics which may be a better fit for our model then please let
> me
> know.
>
> I also need to the syntax for the Fleiss Kappa statistic if anyone has it
> available. Thanks in advance for any help people can provide.
>
>
>
> --
> View this message in context:
>
http://spssx-discussion.1045642.n5.nabble.com/Fleiss-Kappa-Inter-rater-reliability-questions-tp5717299.html> Sent from the SPSSX Discussion mailing list archive at Nabble.com.
>
> =====================
> To manage your subscription to SPSSX-L, send a message to
> LISTSERV@.UGA
> (not to SPSSX-L), with no body text except the
> command. To leave the list, send the command
> SIGNOFF SPSSX-L
> For a list of commands to manage subscriptions, send the command
> INFO REFCARD
>
> =====================
> To manage your subscription to SPSSX-L, send a message to
> LISTSERV@.UGA
> (not to SPSSX-L), with no body text except the
> command. To leave the list, send the command
> SIGNOFF SPSSX-L
> For a list of commands to manage subscriptions, send the command
> INFO REFCARD
-----
--
Bruce Weaver
[hidden email]
http://sites.google.com/a/lakeheadu.ca/bweaver/"When all else fails, RTFM."
NOTE: My Hotmail account is not monitored regularly.
To send me an e-mail, please use the address shown above.
--
View this message in context:
http://spssx-discussion.1045642.n5.nabble.com/Fleiss-Kappa-Inter-rater-reliability-questions-tp5717299p5717302.htmlSent from the SPSSX Discussion mailing list archive at Nabble.com.
=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD
=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD