http://spssx-discussion.165.s1.nabble.com/Fleiss-Kappa-Macro-for-SPSS-tp4994433p4998441.html
"Ordered" -- If there were three categories labelled [Never, Sometimes, Always],
those would be "ordered" and it would be convenient and usually wise to treat
them as continuous, and then to compute some version of an ICC (intra-class
correlation).
It may be less likely that your 9 categories are ordered, than if you had only
3 or 4.
However - without knowing your data, which might make a difference - I
think you should consider abandoning any kappa based on 9 categories.
Kappas with multiple categories are *so* heavily dependent on the
relative sizes of the marginal totals that the absolute size of kappa is
nearly meaningless. That is to say, kappas can give you a test of
agreement; but the kappa as a report of agreement is seldom useful
except for a narrow comparison of coefficients within your data on hand,
whenever they have more than 2 categories.
And if you are contrasting kappas for pairs chosen from three raters (say),
where the actual values already have almost no meaning, there is little
purpose in considering what the 3-person "overall" kappa might be.
What I have found useful, instead, is to consider the 2x2 kappas based
on "category1 vs. all other", and so on. (You might see that there is
good success for raters on the rationally-important categories or the
most-used ones, with indifferent success on several others.) - If the
kappas are similar, presenting the average kappa should be satisfactory.
If the kappas are not similar, you should probably not try to represent
them with one number, no matter how you compute it.
--
Rich Ulrich
> Date: Wed, 16 Nov 2011 08:03:24 -0800
> From:
[hidden email]> Subject: Re: Fleiss Kappa Macro for SPSS
> To:
[hidden email]>
> There are 9 categories which 3 raters(judges) can choose from, and each rater
> has rated over 200 different objects. I am not sure what you mean by the
> categories being 'ordered'.
> ------------------------------------------
[snip, previous]