For multiple raters with an ordinal or interval scale, the ICC is considered equivalent to Fleiss' kappa (Fleiss and Cohen, 1973); so use the ICC. The single measure ICC is the equivalent of Fleiss' kappa; the average measure is the mean of each of the raters' scores.
There is syntax for calculating the rwg at:
LeBreton, J. M. & Senter,J. L. (2008). Answers to 20 Questions About Interrater Reliability and Interrater Agreement. Organizational Research Methods,11, 815-852.
I'm not sure that it calculates the significance level, however. Maybe other listers know of other syntax.
Brian
Hi
I need to find out the level of agreement between 9 raters that rated 28
items on a 7 point Lickert scale. What index should I use? Is the ICC
calculated by SPSS good for this? If it is, does anyone have a table for
interpreting the coefficients - for example it gave me a single measure ICC
of 0.029 and an average measure of 0.21. Also, I'm not to familiar to this
but I asume that my raters should be in spss on columns and the items on
rows to calculate the index?
I've also read that Fleiss's kappa or the rwg would be appropriate. How
could I calculate the rwg for this dataset, is there a syntax for it?
many thanks,
--
View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Agreement-indexes-tp5714002.html
Sent from the SPSSX Discussion mailing list archive at Nabble.com.
=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD
Locked
