Re: inter-rater reliability with multiple raters

Posted by Bruce Weaver on Jun 14, 2014; 6:27pm
URL: http://spssx-discussion.165.s1.nabble.com/inter-rater-reliability-with-multiple-raters-tp5726465p5726474.html

tzoblotsky wrote

There was a discussion of this topic in an earlier thread: http://spssx-discussion.1045642.n5.nabble.com/Fleiss-Kappa-Inter-rater-reliability-questions-td5717299.html

You may also want to look into G Theory as an option, as it allows you to look at different combinations of sources of variance in you outcomes (e.g., raters, occasions).  For example, see http://journals.cambridge.org/action/displayAbstract?fromPage=online&aid=85843 (sorry, but could not find a link to a free version of the article).

Here's a free version:  http://www.researchgate.net/publication/11770441_G_theory_and_the_reliability_of_psychophysiological_measures_a_tutorial/file/9fcfd50dd7cbc6d6bf.pdf