Fleiss Kappa (Inter-rater reliability questions)

[BAM86]

Hi all,

I am conducting research and need some help with doing the Fleiss Kappa analysis.

First and foremost let me give a bit of a layout of the study. We have a sample of 75 students in the social sciences who were asked to categorize potential items for a new measure based on descriptions of 2 theoretical constructs. This was meant to be a preliminary check to help with construct validity and take it a step beyond just the research team validating items initially.

The items are supposed to characterize coping behaviours following romantic loss. The constructs are loss-oriented coping and restoration-oriented coping, however, we also gave the option for participants to say potential items were both types of coping behaviour or something else altogether (basically an 'other' option).

Therefore we have 75 participants categorizing items into one of 4 categories. We want to have some measure of the inter-rater reliability of all these participants though. My understanding is that Fleiss Kappa would be an appropriate tool for this purpose, however, if anyone is aware of other statistics which may be a better fit for our model then please let me know.

I also need to the syntax for the Fleiss Kappa statistic if anyone has it available. Thanks in advance for any help people can provide.  

I am currently out of the office and have limited access to email. I will get back to you as soon as possible. 

Thank you,
McKayla

[bdates]

I can send you a macro for Fleiss' kappa, which will give you the overall as well as the category kappa's, based on Fleiss' 1971 Psych. Bulletin article.  It uses, however, the updated 1979 formula for variance. How many items will the raters categorize?  You have other options, e.g., Krippendorf's alpha and Gwet's AC1 statistic.  Although Fleiss is the most used, there's some use of these as well in the literature.  If you're interested in Cohen's kappa, there are some multiple rater models, most notably Light (averages all the two-rater kappas) and Hubert (averages the two-rater Pe's).  Let me know which you want, and I'll get them off to you.

Brian

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of BAM86
Sent: Wednesday, January 09, 2013 3:25 PM
To: [hidden email]
Subject: Fleiss Kappa (Inter-rater reliability questions)

Hi all,

I am conducting research and need some help with doing the Fleiss Kappa
analysis.

First and foremost let me give a bit of a layout of the study. We have a
sample of 75 students in the social sciences who were asked to categorize
potential items for a new measure based on descriptions of 2 theoretical
constructs. This was meant to be a preliminary check to help with construct
validity and take it a step beyond just the research team validating items
initially.

The items are supposed to characterize coping behaviours following romantic
loss. The constructs are loss-oriented coping and restoration-oriented
coping, however, we also gave the option for participants to say potential
items were both types of coping behaviour or something else altogether
(basically an 'other' option).

Therefore we have 75 participants categorizing items into one of 4
categories. We want to have some measure of the inter-rater reliability of
all these participants though. My understanding is that Fleiss Kappa would
be an appropriate tool for this purpose, however, if anyone is aware of
other statistics which may be a better fit for our model then please let me
know.

I also need to the syntax for the Fleiss Kappa statistic if anyone has it
available. Thanks in advance for any help people can provide.



--
View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Fleiss-Kappa-Inter-rater-reliability-questions-tp5717299.html
Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Bruce Weaver]

Brian, how does your macro compare to this one written by David Nichols?

  http://imaging.mrc-cbu.cam.ac.uk/statswiki/FAQ/kappa/multiple

Thanks,
Bruce

Dates, Brian wrote

I can send you a macro for Fleiss' kappa, which will give you the overall as well as the category kappa's, based on Fleiss' 1971 Psych. Bulletin article.  It uses, however, the updated 1979 formula for variance. How many items will the raters categorize?  You have other options, e.g., Krippendorf's alpha and Gwet's AC1 statistic.  Although Fleiss is the most used, there's some use of these as well in the literature.  If you're interested in Cohen's kappa, there are some multiple rater models, most notably Light (averages all the two-rater kappas) and Hubert (averages the two-rater Pe's).  Let me know which you want, and I'll get them off to you.

Brian

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of BAM86
Sent: Wednesday, January 09, 2013 3:25 PM
To: [hidden email]
Subject: Fleiss Kappa (Inter-rater reliability questions)

Hi all,

I am conducting research and need some help with doing the Fleiss Kappa
analysis.

First and foremost let me give a bit of a layout of the study. We have a
sample of 75 students in the social sciences who were asked to categorize
potential items for a new measure based on descriptions of 2 theoretical
constructs. This was meant to be a preliminary check to help with construct
validity and take it a step beyond just the research team validating items
initially.

The items are supposed to characterize coping behaviours following romantic
loss. The constructs are loss-oriented coping and restoration-oriented
coping, however, we also gave the option for participants to say potential
items were both types of coping behaviour or something else altogether
(basically an 'other' option).

Therefore we have 75 participants categorizing items into one of 4
categories. We want to have some measure of the inter-rater reliability of
all these participants though. My understanding is that Fleiss Kappa would
be an appropriate tool for this purpose, however, if anyone is aware of
other statistics which may be a better fit for our model then please let me
know.

I also need to the syntax for the Fleiss Kappa statistic if anyone has it
available. Thanks in advance for any help people can provide.



--
View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Fleiss-Kappa-Inter-rater-reliability-questions-tp5717299.html
Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[henryilian]

Brian,

I have a related kappa question. I have a case reading instrument with 120 items. Most of the items have three answer choices, but some have two and some have four. There is also one with five. We did an inter-rater reliability study with five readers reading the same case. I haven't looked at the data yet, but I'll need to soon. I was going to use Cohen's Kappa, which SPSS gives you. I was going to set up the data file so that similar items--related topic areas with the same number of answer choices--were treated as separate cases, analogous to a set of MRIs rated by different clinicians, as I've seen in a number of articles. I'm not happy about that because the content for the item may not be similar enough. Also, this approach doesn't allow me to identify items that are troublesome.

Can you see a better way of approaching the task?

Thanks,

Henry

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Dates, Brian
Sent: Wednesday, January 09, 2013 4:07 PM
To: [hidden email]
Subject: Re: Fleiss Kappa (Inter-rater reliability questions)

I can send you a macro for Fleiss' kappa, which will give you the overall as well as the category kappa's, based on Fleiss' 1971 Psych. Bulletin article.  It uses, however, the updated 1979 formula for variance. How many items will the raters categorize?  You have other options, e.g., Krippendorf's alpha and Gwet's AC1 statistic.  Although Fleiss is the most used, there's some use of these as well in the literature.  If you're interested in Cohen's kappa, there are some multiple rater models, most notably Light (averages all the two-rater kappas) and Hubert (averages the two-rater Pe's).  Let me know which you want, and I'll get them off to you.

Brian

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of BAM86
Sent: Wednesday, January 09, 2013 3:25 PM
To: [hidden email]
Subject: Fleiss Kappa (Inter-rater reliability questions)

Hi all,

I am conducting research and need some help with doing the Fleiss Kappa
analysis.

First and foremost let me give a bit of a layout of the study. We have a
sample of 75 students in the social sciences who were asked to categorize
potential items for a new measure based on descriptions of 2 theoretical
constructs. This was meant to be a preliminary check to help with construct
validity and take it a step beyond just the research team validating items
initially.

The items are supposed to characterize coping behaviours following romantic
loss. The constructs are loss-oriented coping and restoration-oriented
coping, however, we also gave the option for participants to say potential
items were both types of coping behaviour or something else altogether
(basically an 'other' option).

Therefore we have 75 participants categorizing items into one of 4
categories. We want to have some measure of the inter-rater reliability of
all these participants though. My understanding is that Fleiss Kappa would
be an appropriate tool for this purpose, however, if anyone is aware of
other statistics which may be a better fit for our model then please let me
know.

I also need to the syntax for the Fleiss Kappa statistic if anyone has it
available. Thanks in advance for any help people can provide.



--
View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Fleiss-Kappa-Inter-rater-reliability-questions-tp5717299.html
Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD


Confidentiality Notice: This e-mail communication, and any attachments, contains confidential and privileged information for the exclusive use of the recipient(s) named above. If you are not an intended recipient, or the employee or agent responsible to deliver it to an intended recipient, you are hereby notified that you have received this communication in error and that any review, disclosure, dissemination, distribution or copying of it or its contents is prohibited. If you have received this communication in error, please notify me immediately by replying to this message and delete this communication from your computer. Thank you.

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[David Marso]

Hi Brian,
  Would it be possible to post those here in the NG as an attachment?
David
--

Dates, Brian wrote

I can send you a macro for Fleiss' kappa, which will give you the overall as well as the category kappa's, based on Fleiss' 1971 Psych. Bulletin article.  It uses, however, the updated 1979 formula for variance. How many items will the raters categorize?  You have other options, e.g., Krippendorf's alpha and Gwet's AC1 statistic.  Although Fleiss is the most used, there's some use of these as well in the literature.  If you're interested in Cohen's kappa, there are some multiple rater models, most notably Light (averages all the two-rater kappas) and Hubert (averages the two-rater Pe's).  Let me know which you want, and I'll get them off to you.

Brian

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of BAM86
Sent: Wednesday, January 09, 2013 3:25 PM
To: [hidden email]
Subject: Fleiss Kappa (Inter-rater reliability questions)

Hi all,

I am conducting research and need some help with doing the Fleiss Kappa
analysis.

First and foremost let me give a bit of a layout of the study. We have a
sample of 75 students in the social sciences who were asked to categorize
potential items for a new measure based on descriptions of 2 theoretical
constructs. This was meant to be a preliminary check to help with construct
validity and take it a step beyond just the research team validating items
initially.

The items are supposed to characterize coping behaviours following romantic
loss. The constructs are loss-oriented coping and restoration-oriented
coping, however, we also gave the option for participants to say potential
items were both types of coping behaviour or something else altogether
(basically an 'other' option).

Therefore we have 75 participants categorizing items into one of 4
categories. We want to have some measure of the inter-rater reliability of
all these participants though. My understanding is that Fleiss Kappa would
be an appropriate tool for this purpose, however, if anyone is aware of
other statistics which may be a better fit for our model then please let me
know.

I also need to the syntax for the Fleiss Kappa statistic if anyone has it
available. Thanks in advance for any help people can provide.



--
View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Fleiss-Kappa-Inter-rater-reliability-questions-tp5717299.html
Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Ryan]

You might consider adding Krippendorff's alpha as a custom dialog to SPSS:

 

http://www.afhayes.com/spss-sas-and-mplus-macros-and-code.html

 

Search for KALPHA on the website linked above and download the kalpha.spd file.

 

The documentation for KALPHA is here:

 

http://www.afhayes.com/public/cmm2007.pdf

 

Ryan

On Wed, Jan 9, 2013 at 3:24 PM, BAM86 <[hidden email]> wrote:

Hi all,

I am conducting research and need some help with doing the Fleiss Kappa
analysis.

First and foremost let me give a bit of a layout of the study. We have a
sample of 75 students in the social sciences who were asked to categorize
potential items for a new measure based on descriptions of 2 theoretical
constructs. This was meant to be a preliminary check to help with construct
validity and take it a step beyond just the research team validating items
initially.

The items are supposed to characterize coping behaviours following romantic
loss. The constructs are loss-oriented coping and restoration-oriented
coping, however, we also gave the option for participants to say potential
items were both types of coping behaviour or something else altogether
(basically an 'other' option).

Therefore we have 75 participants categorizing items into one of 4
categories. We want to have some measure of the inter-rater reliability of
all these participants though. My understanding is that Fleiss Kappa would
be an appropriate tool for this purpose, however, if anyone is aware of
other statistics which may be a better fit for our model then please let me
know.

I also need to the syntax for the Fleiss Kappa statistic if anyone has it
available. Thanks in advance for any help people can provide.



--
View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Fleiss-Kappa-Inter-rater-reliability-questions-tp5717299.html
Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[bdates]

Right.  I know I’ve been in touch with Andy, and all he asks is that he’s cited appropriately.  I think the correct citation may even be in his syntax.

 

B

 

---

From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of R B
Sent: Wednesday, January 09, 2013 5:32 PM
To: [hidden email]
Subject: Re: Fleiss Kappa (Inter-rater reliability questions)

 

You might consider adding Krippendorff's alpha as a custom dialog to SPSS:

 

http://www.afhayes.com/spss-sas-and-mplus-macros-and-code.html

 

Search for KALPHA on the website linked above and download the kalpha.spd file.

 

The documentation for KALPHA is here:

 

http://www.afhayes.com/public/cmm2007.pdf

 

Ryan

On Wed, Jan 9, 2013 at 3:24 PM, BAM86 <[hidden email]> wrote:

Hi all,

I am conducting research and need some help with doing the Fleiss Kappa
analysis.

First and foremost let me give a bit of a layout of the study. We have a
sample of 75 students in the social sciences who were asked to categorize
potential items for a new measure based on descriptions of 2 theoretical
constructs. This was meant to be a preliminary check to help with construct
validity and take it a step beyond just the research team validating items
initially.

The items are supposed to characterize coping behaviours following romantic
loss. The constructs are loss-oriented coping and restoration-oriented
coping, however, we also gave the option for participants to say potential
items were both types of coping behaviour or something else altogether
(basically an 'other' option).

Therefore we have 75 participants categorizing items into one of 4
categories. We want to have some measure of the inter-rater reliability of
all these participants though. My understanding is that Fleiss Kappa would
be an appropriate tool for this purpose, however, if anyone is aware of
other statistics which may be a better fit for our model then please let me
know.

I also need to the syntax for the Fleiss Kappa statistic if anyone has it
available. Thanks in advance for any help people can provide.



--
View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Fleiss-Kappa-Inter-rater-reliability-questions-tp5717299.html
Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

 

[bdates]

I used Fleiss' 1979 variance, and I don't know which one ('71 or '79) David used.  I added syntax for category kappa's, which Fleiss developed, and an upper and lower 95% confidence interval on all of them.  I also report on the number of responses each rater has in each category, the number of agreements each rater has with the other rater(s) in each category, the number of subjects, number of categories, number of raters, and raw proportion of agreement.  So there's more information, but the overall kappa's still the same.

B

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Bruce Weaver
Sent: Wednesday, January 09, 2013 4:42 PM
To: [hidden email]
Subject: Re: Fleiss Kappa (Inter-rater reliability questions)

Brian, how does your macro compare to this one written by David Nichols?

  http://imaging.mrc-cbu.cam.ac.uk/statswiki/FAQ/kappa/multiple

Thanks,
Bruce



Dates, Brian wrote

> I can send you a macro for Fleiss' kappa, which will give you the overall
> as well as the category kappa's, based on Fleiss' 1971 Psych. Bulletin
> article.  It uses, however, the updated 1979 formula for variance. How
> many items will the raters categorize?  You have other options, e.g.,
> Krippendorf's alpha and Gwet's AC1 statistic.  Although Fleiss is the most
> used, there's some use of these as well in the literature.  If you're
> interested in Cohen's kappa, there are some multiple rater models, most
> notably Light (averages all the two-rater kappas) and Hubert (averages the
> two-rater Pe's).  Let me know which you want, and I'll get them off to
> you.
>
> Brian
>
> -----Original Message-----
> From: SPSSX(r) Discussion [mailto:

> SPSSX-L@.UGA

> ] On Behalf Of BAM86
> Sent: Wednesday, January 09, 2013 3:25 PM
> To:

> SPSSX-L@.UGA

> Subject: Fleiss Kappa (Inter-rater reliability questions)
>
> Hi all,
>
> I am conducting research and need some help with doing the Fleiss Kappa
> analysis.
>
> First and foremost let me give a bit of a layout of the study. We have a
> sample of 75 students in the social sciences who were asked to categorize
> potential items for a new measure based on descriptions of 2 theoretical
> constructs. This was meant to be a preliminary check to help with
> construct
> validity and take it a step beyond just the research team validating items
> initially.
>
> The items are supposed to characterize coping behaviours following
> romantic
> loss. The constructs are loss-oriented coping and restoration-oriented
> coping, however, we also gave the option for participants to say potential
> items were both types of coping behaviour or something else altogether
> (basically an 'other' option).
>
> Therefore we have 75 participants categorizing items into one of 4
> categories. We want to have some measure of the inter-rater reliability of
> all these participants though. My understanding is that Fleiss Kappa would
> be an appropriate tool for this purpose, however, if anyone is aware of
> other statistics which may be a better fit for our model then please let
> me
> know.
>
> I also need to the syntax for the Fleiss Kappa statistic if anyone has it
> available. Thanks in advance for any help people can provide.
>
>
>
> --
> View this message in context:
> http://spssx-discussion.1045642.n5.nabble.com/Fleiss-Kappa-Inter-rater-reliability-questions-tp5717299.html
> Sent from the SPSSX Discussion mailing list archive at Nabble.com.
>
> =====================
> To manage your subscription to SPSSX-L, send a message to

> LISTSERV@.UGA

>  (not to SPSSX-L), with no body text except the
> command. To leave the list, send the command
> SIGNOFF SPSSX-L
> For a list of commands to manage subscriptions, send the command
> INFO REFCARD
>
> =====================
> To manage your subscription to SPSSX-L, send a message to

> LISTSERV@.UGA

>  (not to SPSSX-L), with no body text except the
> command. To leave the list, send the command
> SIGNOFF SPSSX-L
> For a list of commands to manage subscriptions, send the command
> INFO REFCARD





-----
--
Bruce Weaver
[hidden email]
http://sites.google.com/a/lakeheadu.ca/bweaver/

"When all else fails, RTFM."

NOTE: My Hotmail account is not monitored regularly.
To send me an e-mail, please use the address shown above.

--
View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Fleiss-Kappa-Inter-rater-reliability-questions-tp5717299p5717302.html
Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Ryan]

In response to how to how to cite his macro, see the top of this document:

 

http://www.afhayes.com/public/kalpha.pdf

 

Ryan

On Wed, Jan 9, 2013 at 5:39 PM, Dates, Brian <[hidden email]> wrote:

Right.  I know I’ve been in touch with Andy, and all he asks is that he’s cited appropriately.  I think the correct citation may even be in his syntax.

 

B

 

---

From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of R B
Sent: Wednesday, January 09, 2013 5:32 PM
To: [hidden email]
Subject: Re: Fleiss Kappa (Inter-rater reliability questions)

 

You might consider adding Krippendorff's alpha as a custom dialog to SPSS:

 

http://www.afhayes.com/spss-sas-and-mplus-macros-and-code.html

 

Search for KALPHA on the website linked above and download the kalpha.spd file.

 

The documentation for KALPHA is here:

 

http://www.afhayes.com/public/cmm2007.pdf

 

Ryan

On Wed, Jan 9, 2013 at 3:24 PM, BAM86 <[hidden email]> wrote:

Hi all,

I am conducting research and need some help with doing the Fleiss Kappa
analysis.

First and foremost let me give a bit of a layout of the study. We have a
sample of 75 students in the social sciences who were asked to categorize
potential items for a new measure based on descriptions of 2 theoretical
constructs. This was meant to be a preliminary check to help with construct
validity and take it a step beyond just the research team validating items
initially.

The items are supposed to characterize coping behaviours following romantic
loss. The constructs are loss-oriented coping and restoration-oriented
coping, however, we also gave the option for participants to say potential
items were both types of coping behaviour or something else altogether
(basically an 'other' option).

Therefore we have 75 participants categorizing items into one of 4
categories. We want to have some measure of the inter-rater reliability of
all these participants though. My understanding is that Fleiss Kappa would
be an appropriate tool for this purpose, however, if anyone is aware of
other statistics which may be a better fit for our model then please let me
know.

I also need to the syntax for the Fleiss Kappa statistic if anyone has it
available. Thanks in advance for any help people can provide.



--
View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Fleiss-Kappa-Inter-rater-reliability-questions-tp5717299.html
Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

 

[Maximilian Koenig]

Hello there!

First time poster and very glad I found this mailing list. I'm currently working on my graduation thesis for which I want to use Fleiss Kappa and maybe Krippendorph's alpha to calculate inter-rater-agreement. Would you send me those macros too, if you could be so kind?

Thank you very much,

Max

[bdates]

Max,

I sent them out yesterday to the entire listserv, so Fleiss should be in the archives.  If you have trouble, contact me offline, and I'll send it directly to you rather than load up the listserv.  Andrew Hayes at the Ohio State University has a macro for Krippendorf's alpha.  You can download it from http://afhayes.com/spss-sas-and-mplus-macros-and-code.html.  The name of the macro is KALPHA.

Brian

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Maximilian Koenig
Sent: Thursday, January 10, 2013 11:35 AM
To: [hidden email]
Subject: Re: Fleiss Kappa (Inter-rater reliability questions)

Hello there!

First time poster and very glad I found this mailing list. I'm currently
working on my graduation thesis for which I want to use Fleiss Kappa and
maybe Krippendorph's alpha to calculate inter-rater-agreement. Would you
send me those macros too, if you could be so kind?

Thank you very much,

Max



--
View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Fleiss-Kappa-Inter-rater-reliability-questions-tp5717299p5717320.html
Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[David Marso]

Hi Brian,
I did not see the post.  Do you have a link to the posting?  I didn't
see anything with an attachment.
David

On Thu, Jan 10, 2013 at 12:20 PM, Dates, Brian [via SPSSX Discussion]
<[hidden email]> wrote:

> Max,
>
> I sent them out yesterday to the entire listserv, so Fleiss should be in the
> archives.  If you have trouble, contact me offline, and I'll send it
> directly to you rather than load up the listserv.  Andrew Hayes at the Ohio
> State University has a macro for Krippendorf's alpha.  You can download it
> from http://afhayes.com/spss-sas-and-mplus-macros-and-code.html.  The name
> of the macro is KALPHA.
>
> Brian
>
> -----Original Message-----
> From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Maximilian
> Koenig
> Sent: Thursday, January 10, 2013 11:35 AM
> To: [hidden email]
> Subject: Re: Fleiss Kappa (Inter-rater reliability questions)
>
> Hello there!
>
> First time poster and very glad I found this mailing list. I'm currently
> working on my graduation thesis for which I want to use Fleiss Kappa and
> maybe Krippendorph's alpha to calculate inter-rater-agreement. Would you
> send me those macros too, if you could be so kind?
>
> Thank you very much,
>
> Max
>
>
>
> --
> View this message in context:
> http://spssx-discussion.1045642.n5.nabble.com/Fleiss-Kappa-Inter-rater-reliability-questions-tp5717299p5717320.html
>
> Sent from the SPSSX Discussion mailing list archive at Nabble.com.
>
> =====================
> To manage your subscription to SPSSX-L, send a message to
> [hidden email] (not to SPSSX-L), with no body text except the
> command. To leave the list, send the command
> SIGNOFF SPSSX-L
> For a list of commands to manage subscriptions, send the command
> INFO REFCARD
>
> =====================
> To manage your subscription to SPSSX-L, send a message to
> [hidden email] (not to SPSSX-L), with no body text except the
> command. To leave the list, send the command
> SIGNOFF SPSSX-L
> For a list of commands to manage subscriptions, send the command
> INFO REFCARD
>
>
> ________________________________
> If you reply to this email, your message will be added to the discussion
> below:
> http://spssx-discussion.1045642.n5.nabble.com/Fleiss-Kappa-Inter-rater-reliability-questions-tp5717299p5717321.html
> To unsubscribe from Fleiss Kappa (Inter-rater reliability questions), click
> here.
> NAML

[Rich Ulrich]

You have had a lot of response about computing kappas, but I have seen
nothing about, How appropriate is this?  - I have some doubts about the
appropriateness of what you described.

First, I will mention that kappas are often problematic when they are not
the simple ones on 2x2 tables.  My experience is with Cohen's kappa, but
it looks like the same problems should exist with Fleiss's. 

Second, I will say that your "four categories" seem to reduce to two
dichotomies -- No/Yes for Loss-oriented, and No/Yes for Restoration-
oriented.  (The category Other seems to denote the No-No cell, if you
draw up the two categories as another 2x2 table.)

Therefore -- I think I would restructure this as two analyses done separately
for Loss and Restoration.  You can look for reliability of high versus low counts
on one dimension at a time.  In a minimal sense, this would support the
reliability - and face validity? - of the "constructs" taken separately. 

This approach does not yet tackle the "discriminative" ability of the items,
if you want to separate the two kinds of coping.  It also does not look at the
chance that there are additional latent factors in the items that exist for
all 75 subjects or for subsets of them.

--
Rich Ulrich

> Date: Wed, 9 Jan 2013 12:24:32 -0800

> From: [hidden email]
> Subject: Fleiss Kappa (Inter-rater reliability questions)
> To: [hidden email]
>
> Hi all,
>
> I am conducting research and need some help with doing the Fleiss Kappa
> analysis.
>
> First and foremost let me give a bit of a layout of the study. We have a
> sample of 75 students in the social sciences who were asked to categorize
> potential items for a new measure based on descriptions of 2 theoretical
> constructs. This was meant to be a preliminary check to help with construct
> validity and take it a step beyond just the research team validating items
> initially.
>
> The items are supposed to characterize coping behaviours following romantic
> loss. The constructs are loss-oriented coping and restoration-oriented
> coping, however, we also gave the option for participants to say potential
> items were both types of coping behaviour or something else altogether
> (basically an 'other' option).
>
> Therefore we have 75 participants categorizing items into one of 4
> categories. We want to have some measure of the inter-rater reliability of
> all these participants though. My understanding is that Fleiss Kappa would
> be an appropriate tool for this purpose, however, if anyone is aware of
> other statistics which may be a better fit for our model then please let me
> know.
>
> I also need to the syntax for the Fleiss Kappa statistic if anyone has it
> available. Thanks in advance for any help people can provide.
> ...