Kappa help

[Nicx]

Hi there, I was hoping someone might be able to help? i couldn't see anything similar posted recently.I'm using SPSS v.18 and trying to get a kappa score but my data set is asymmetrical (one observer hasn't used all of the categories.) I have read numerous suggestions and have tried entering dummy observations which are negatively weighted but still no joy. I did this by using a previously used syntax but I am no SPSS expert so this may have not been for the correct SPSS version etc. I'd be really grateful if anyone has any ideas.

[Bruce Weaver]

I assume you are using CROSSTABS.  Have you tried "integer mode"?  See the FM for examples.

Nicx wrote

Hi there, I was hoping someone might be able to help? i couldn't see anything similar posted recently.I'm using SPSS v.18 and trying to get a kappa score but my data set is asymmetrical (one observer hasn't used all of the categories.) I have read numerous suggestions and have tried entering dummy observations which are negatively weighted but still no joy. I did this by using a previously used syntax but I am no SPSS expert so this may have not been for the correct SPSS version etc. I'd be really grateful if anyone has any ideas.

[Rich Ulrich]

IMHO, the multi-category kappa is a pretty useless statistics in
*most* of its proposed applications.  Its size depends so thoroughly
on the margin-totals that it is mainly useful for a within-study set
of comparisons, say, using multiple raters and looking at them two
at a time.  (The multi-rater kappa is also pretty useless.)

What is usually interesting, instead, is the set of 2x2 comparisons
of A/not-A,  B/not-B, and so on.  And you toss in a comment about
the non-used (by one rater) category.

--
Rich Ulrich

> Date: Wed, 5 Jun 2013 00:38:56 -0700

> From: [hidden email]
> Subject: Kappa help
> To: [hidden email]
>
> Hi there, I was hoping someone might be able to help? i couldn't see anything
> similar posted recently.I'm using SPSS v.18 and trying to get a kappa score
> but my data set is asymmetrical (one observer hasn't used all of the
> categories.) I have read numerous suggestions and have tried entering dummy
> observations which are negatively weighted but still no joy. I did this by
> using a previously used syntax but I am no SPSS expert so this may have not
> been for the correct SPSS version etc. I'd be really grateful if anyone has
> any ideas.
>
> ...

[Nicx]

Thanks both, I have tried getting it into integer mode but being completely new to syntax I'm not sure I'm doing it correctly. This may sound stupid but can I just copy the integer instruction from the manual or do I have to add extra commands relating to my results?

[bdates]

I have to echo a lot of what Rich said.  I’ve been studying and developing macros for a lot of nominal level ‘kappas’ for about five years, and after another month or so for edit, I’m going to find some way of posting the monograph and syntax for the listserv.  With that said, the real issue in any inter-rater reliability study is the confidence interval, not the value or its significance.  In the case of kappa’s, with perhaps the exception of Gwet’s AC1, which is impervious to marginal totals, a significant kappa can have a value of .20, which means it’s only 20% above what would be expected by chance.  In the case of ICC, which is appropriate for any ordered categories, the result is a correlation, so agreement isn’t guaranteed, just relationship.  Regardless of what you intend to do, look at the actual agreement and the confidence interval.  If it’s anywhere near the neutral value (no different from chance), then it’s questionable in its application.

 

Brian

 

---

From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Rich Ulrich
Sent: Wednesday, June 05, 2013 12:44 PM
To: [hidden email]
Subject: Re: Kappa help

 

IMHO, the multi-category kappa is a pretty useless statistics in
*most* of its proposed applications.  Its size depends so thoroughly
on the margin-totals that it is mainly useful for a within-study set
of comparisons, say, using multiple raters and looking at them two
at a time.  (The multi-rater kappa is also pretty useless.)

What is usually interesting, instead, is the set of 2x2 comparisons
of A/not-A,  B/not-B, and so on.  And you toss in a comment about
the non-used (by one rater) category.

--
Rich Ulrich

> Date: Wed, 5 Jun 2013 00:38:56 -0700
> From: [hidden email]
> Subject: Kappa help
> To: [hidden email]
>
> Hi there, I was hoping someone might be able to help? i couldn't see anything
> similar posted recently.I'm using SPSS v.18 and trying to get a kappa score
> but my data set is asymmetrical (one observer hasn't used all of the
> categories.) I have read numerous suggestions and have tried entering dummy
> observations which are negatively weighted but still no joy. I did this by
> using a previously used syntax but I am no SPSS expert so this may have not
> been for the correct SPSS version etc. I'd be really grateful if anyone has
> any ideas.
>
> ...