Kappa for unsymmetrical table

[Jean Hanson]

Dear All,
We are trying to calcuate a Kappa and the table is unsymmetrical. Is there a legitimate way around this?

Thank you,
Jean Hanson

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[bdates]

Jean,

How is the table unsymmetrical?

Brian

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of
Jean Hanson
Sent: Friday, October 31, 2008 4:45 PM
To: [hidden email]
Subject: Kappa for unsymmetrical table

Dear All,
We are trying to calcuate a Kappa and the table is unsymmetrical. Is
there a legitimate way around this?

Thank you,
Jean Hanson

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Jean Hanson]

I am trying to assess inter-rater reliability with a six category nominal measure. One rater did not have any cases in one of the categories. Consequently SPSS will not produce a Kappa statistic.

-------------- Original message --------------
From: "Dates, Brian" <[hidden email]>

> Jean,
>
> How is the table unsymmetrical?
>
> Brian
>
> -----Original Message-----
> From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of
> Jean Hanson
> Sent: Friday, October 31, 2008 4:45 PM
> To: [hidden email]
> Subject: Kappa for unsymmetrical table
>
> Dear All,
> We are trying to calcuate a Kappa and the table is unsymmetrical. Is
> there a legitimate way around this?
>
> Thank you,
> Jean Hanson
>
> =====================
> To manage your subscription to SPSSX-L, send a message to
> [hidden email] (not to SPSSX-L), with no body text except the
> command. To leave the list, send the command
> SIGNOFF SPSSX-L
> For a list of commands to manage subscriptions, send the command
> INFO REFCARD
>
> =====================
> To manage your subscription to SPSSX-L, send a message to
> [hidden email] (not to SPSSX-L), with no body text except the
> command. To leave the list, send the command
> SIGNOFF SPSSX-L
> For a list of commands to manage subscriptions, send the command
> INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Marta Garcia-Granero]

Jean Hanson wrote:
> I am trying to assess inter-rater reliability with a six category nominal measure. One rater did not have any cases in one of the categories. Consequently SPSS will not produce a Kappa statistic.
>
>
Jean, the following code assumes that the nominal rating is coded 1, 2,
3, 4, 5 and 6 for both raters, that the rater with missing cases in one
category is rater 2, and missing category is the last (#6). It can be
easily modified for your particular case (if you give me more details -
variable names, missing category...,I will adapt it for you). It uses
SPSS 14/newer.

* Aggregate dataset on raters to get the frequencies in variable N *.
DATASET DECLARE Aggregated.
AGGREGATE
  /OUTFILE='Aggregated'
  /BREAK=rater1 rater2
  /N=N.
DATASET ACTIVATE Aggregated.
* Restructure dataset to get variables with frequencies *.
CASESTOVARS
 /ID = rater1
 /index= rater2
 /GROUPBY = VARIABLE .
* Create extra variable for missing category (#6) in rater 2, giving it
a close to 0 frequency *.
COMPUTE N.6=1E-6.

* Restructure dataset back *
VARSTOCASES  /MAKE N FROM N.1 N.2 N.3 N.4 N.5 N.6
 /INDEX = rater2(6)
 /KEEP =  rater1
 /NULL = KEEP.

* Weight by N *.
WEIGHT BY N.

* Compute kappa *.
CROSSTABS
  /TABLES=rater1  BY rater2
  /FORMAT= AVALUE TABLES
  /STATISTIC=KAPPA
  /CELLS= COUNT
  /COUNT ROUND CELL .

HTH,
Marta García-Granero


--
For miscellaneous statistical stuff, visit:
http://gjyp.nl/marta/

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Marta Garcia-Granero]

Jean Hanson wrote:
> The rater with the missing data is Rater 1. The missing category is #4. Yes, the coding is from 1 to 6.
>
This modified code should work.

Let me know if it fails (post the error codes, they are very helpful in
error debugging).

Marta

* Aggregate dataset on raters to get the frequencies in variable N *.
DATASET DECLARE Aggregated.
AGGREGATE
 /OUTFILE='Aggregated'
 /BREAK=rater2 rater1
 /N=N.
DATASET ACTIVATE Aggregated.
* Restructure dataset to get variables with frequencies *.
CASESTOVARS
 /ID = rater2
 /index= rater1
 /GROUPBY = VARIABLE .
* Create extra variable for missing category (#4) in rater 1, giving it
  a close to 0 frequency *.
COMPUTE N.4=1E-6.

* Restructure dataset back *
VARSTOCASES /MAKE N FROM N.1 N.2 N.3 N.4 N.5 N.6
 /INDEX = rater1(6)
 /KEEP = rater2
 /NULL = KEEP.

* Weight by N *.
 WEIGHT BY N.

* Compute kappa *.
CROSSTABS
 /TABLES=rater1 BY rater2
 /FORMAT= AVALUE TABLES
 /STATISTIC=KAPPA
 /CELLS= COUNT
 /COUNT ROUND CELL .



--
For miscellaneous statistical stuff, visit:
http://gjyp.nl/marta/

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Marta Garcia-Granero]

Jean Hanson wrote:

Which SPSS version are you using? it must SPSS 14/newer or it will not
work as it is right now (it should be modified). I tested the code with
fake data and it worked OK.

Marta GG

> Marta,
> It did fail. here are the error codes.
> Aggregate dataset on raters to get the frequencies in variable N *.
>
>> Error # 10907 in column 11. Text: dataset
>> The OUTFILE subcommand on the AGGREGATE command must be followed by the
>> optional MISSING or PRESORTED or DOCUMENT subcommands or the required BREAK
>> subcommand. None of these was found.
>> This command not executed.
>>
>
>
>> Error # 10917 in column 256. Text: (End of Command)
>> SPSS expected the definition of a new variable on the AGGREGATE command but
>> did not find a new variable name.
>>
> DATASET DECLARE Aggregated.
>
> -------------- Original message --------------
> From: Marta García-Granero <[hidden email]>
>
>
>> Jean Hanson wrote:
>>
>>> The rater with the missing data is Rater 1. The missing category is #4. Yes,
>>>
>> the coding is from 1 to 6.
>>
>> This modified code should work.
>>
>> Let me know if it fails (post the error codes, they are very helpful in
>> error debugging).
>>
>> Marta
>>
>> * Aggregate dataset on raters to get the frequencies in variable N *.
>> DATASET DECLARE Aggregated.
>> AGGREGATE
>> /OUTFILE='Aggregated'
>> /BREAK=rater2 rater1
>> /N=N.
>> DATASET ACTIVATE Aggregated.
>> * Restructure dataset to get variables with frequencies *.
>> CASESTOVARS
>> /ID = rater2
>> /index= rater1
>> /GROUPBY = VARIABLE .
>> * Create extra variable for missing category (#4) in rater 1, giving it
>> a close to 0 frequency *.
>> COMPUTE N.4=1E-6.
>>
>> * Restructure dataset back *
>> VARSTOCASES /MAKE N FROM N.1 N.2 N.3 N.4 N.5 N.6
>> /INDEX = rater1(6)
>> /KEEP = rater2
>> /NULL = KEEP.
>>
>> * Weight by N *.
>> WEIGHT BY N.
>>
>> * Compute kappa *.
>> CROSSTABS
>> /TABLES=rater1 BY rater2
>> /FORMAT= AVALUE TABLES
>> /STATISTIC=KAPPA
>> /CELLS= COUNT
>> /COUNT ROUND CELL .
>>
>>

--
For miscellaneous statistical stuff, visit:
http://gjyp.nl/marta/

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Marta Garcia-Granero]

I have SPSS 15 and still get an error message.

 

DATASET DECLARE Aggregated.

 

Dataset Declare

 

                                                                          Warnings

 

Dataset name Aggregated already defined.

 

This one can be ignored, it is just a warning.

AGGREGATE

 /OUTFILE='Aggregated'

 /BREAK=rater2 rater1

 /N=N.

 DATASET ACTIVATE Aggregated.

Stop after the above command, and take a look at the 'Aggregated' dataset. It should look like this (concerning rater1 and rater2, the N are not important here)

rater2 rater1 N
1 1 .
1 2 .
1 3
1 4
1 5
1 6
2 1
2 2
2 3
2 4
2 5
2 6
.
more lines
.
5 1
5 2
5 3
5 4
5 5
5 6

If some blank lines exist, then that's the problem. Eliminate them (all), and go on with the rest of the code.

CASESTOVARS

 /ID = rater2

 /index= rater1

 /GROUPBY = VARIABLE .

 

Cases to Variables

 

[Aggregated]

 

                                                                          Warnings

 

An invalid index value was detected at case 1. String index values may not be blank. Numeric index values must be non-negative integers.

This command is not executed.

 

This one is problematic. Do you have blank values in rater1? (see above)

The rest of the errors arise from the fact that the cases to variables is not executed.

COMPUTE N.4=1E-6.

VARSTOCASES /MAKE N FROM N.1 N.2 N.3 N.4 N.5 N.6

 /INDEX = rater1(6)

 /KEEP = rater2

 /NULL = KEEP.

 

Variables to Cases

 

[Aggregated]

 

 

 

                                                                          Warnings

 

Text: N.1

An undefined variable name, or a scratch or system variable was specified in a variable list which accepts only standard variables.  Check spelling and verify the existence of this variable.

This command not executed.

 

Missing parameter in MAKE subcommand.

 

Thanks for your persistence.

Jean

-------------- Original message --------------
From: Marta García-Granero [hidden email]

> Jean Hanson wrote:
> > It did fail. here are the error codes.
> > Aggregate dataset on raters to get the frequencies in variable N *.
> >
> >> Error # 10907 in column 11. Text: dataset
> >> The OUTFILE subcommand on the AGGREGATE command must be followed by the
> >> optional MISSING or PRESORTED or DOCUMENT subcommands or the required BREAK
> >> subcommand. None of these was found.
> >> This command not executed.
> >>
> I see the error now! When you copy-pasted everything, you missed the "*"
> at the beginning of the comment, and SPSS tried to execute the comment
> as if it were true command.
>
> Run this simplified version of the co de, without any comment line. If
> you are using SPSS 14/newer, it will work OK.
>
> DATASET DECLARE Aggregated.
> AGGREGATE
> /OUTFILE='Aggregated'
> /BREAK=rater2 rater1
> /N=N.
> DATASET ACTIVATE Aggregated.
> CASESTOVARS
> /ID = rater2
> /index= rater1
> /GROUPBY = VARIABLE .
> COMPUTE N.4=1E-6.
> VARSTOCASES /MAKE N FROM N.1 N.2 N.3 N.4 N.5 N.6
> /INDEX = rater1(6)
> /KEEP = rater2
> /NULL = KEEP.
> WEIGHT BY N.
> CROSSTABS
> /TABLES=rater1 BY rater2
> /FORMAT= AVALUE TABLES
> /STATISTIC=KAPPA
> /CELLS= COUNT
> /COUNT ROUND CELL .
>
> Marta

===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD