Re: inter rater reliability multiple attributes

Posted by susan.richardson on Aug 08, 2014; 12:35pm
URL: http://spssx-discussion.165.s1.nabble.com/inter-rater-reliability-multiple-attributes-tp5726913p5726916.html

Sorry, David, let me try again.

There are two raters (S & J).

There are 12 schools being rated (1-12).

Each school is being rated dichotomously on 14 attributes (did/did not use Strategy 1, did/did not use Strategy 2, and so forth).

As you might expect (as would be typical for Kappa), I have the schools set  as the cases (rows), and I have the raters set as the variables, also by  strategy (columns). So variable 1 (V1) is S1 (rater S strategy 1) and V2 is J1 (rater J strategy 1), V3 is S2, and V4 is J2, etc.

Each rater rates each school on each attribute, so I have no missing data.

If I only had one strategy, this would be simple. But I have 14, and while I can look at the IRR for each strategy, it doesn't take into account the multivariate nature of the data. I was hoping the forum would have some insight.

If my description above is still lacking, I can post a sample table. But I  hope this will be enough to get the conversation started.

Thanks for any help!