As Stephen points out, 32 is a small N for reliability.
I will further point out that having a survey instrument of all-nominal
items is unusual; and that kappa is not a very informative measure
when there are more than 2 categories unless it simply confirms
that the number of discrepancies is remarkably small.
- For a much larger N, I have used a kappa for each of the 2x2 tables
that are be derived for each variable -- taking each category as Yes/No
for the four responses, in your example. If I'm really interested in the
reliability, I'm concerned about each of the categories. I wrote a
program that used to table that up for me, but it doesn't run on my
64-bit PC. It showed me both the kappa (for similarity) and the McNemar
chi-squared (for *systematic* change) for each category. This is
parallel to my favorite way of documenting paired continuous items,
which uses the correlation r and the paired t-test.
What you have as raw data are your "discrepancies." Are there many?
Count them. Report them. If there were 8 responses in a category at
Pre, a change by "one" is more than 10%, and already worth noting.
A change by "two" must show a pretty labile measurement. Reporting
the counts will be much more informative than using a derived statistic.
--
Rich Ulrich
Date: Thu, 8 Dec 2011 09:34:58 +0800
From:
[hidden email]Subject: Reliability for survey questionnaire with nominal response options
To:
[hidden email]
Dear all,
The items of a survey instrument are designed with four nominal response options. It was administered to the same sample (n=32) in two occasions (Test-re-test). What coefficient shall I use in order to test for the reliability of each item and the instrument as a whole?
Locked
