Re: inter rater reliability multiple attributes

Posted by David Marso on Aug 07, 2014; 10:40pm
URL: http://spssx-discussion.165.s1.nabble.com/inter-rater-reliability-multiple-attributes-tp5726913p5726914.html

Maybe you need to be more forthcoming about your data!
From what it sounds like you simply have 28 values (2 raters x 14 attributes)?
I don't see how a Kappa is calculated for 2 raters for even a single attribute!
Please share with the class!

susan.richardson wrote

Hello All,

I have 2 raters (S & J) who rate schools on 14 attributes. Each attribute is rated on a dichotomous scale.

I know that I can look at the kappa for each attribute (S1 and J1, S2 and J2, and so forth) but is there any way to account for the fact that I am doing this 14 times? Or a way to summarize overall agreement?

I found the Kupper and Hafner (1989) paper describing what to do with more than one nominal variable being rated, and I am hoping there is an SPSS solution that may be similar(?)

If there isn't a answer, what would you suggest for reporting the range of kappas across attributes?

Thanks,
Susan