SPSSX Discussion

SPSS weighted kappa error message and syntax

Classic

List

Threaded

7 messages Options

paola

SPSS weighted kappa error message and syntax

Hallo,

I am working with ordinal scales for cerebral atrophy (ratings 0=no atrophy
to 4=severe atrophy) and vascular burden (ratings 0=no burden to 3=severe
burden) assessed by two raters and I want to calculate intra- and
inter-rater reliability. I am having a few problems:

1) Since I am dealing with an ordinal scale I have decided to use weighted
kappa. I have installed the extension bundle for weighted kappa from SPSS
(version 25) but I keep getting the error message "some ratings are less
than one this command is not executed". I have looked through the threads in
stats forums and in the Internet but as far as I can see nobody has come up
with this issue. I know from the literature that intra- and inter-rater
reliability are commonly calculated for these scales using weighted kappa.
Also, weighted kappa seems to be applicable to ratings of zero
inSPSS(e.g.http://www.spssusers.co.uk/Events/2015/ALDRIDGE2015.pdf). The
only solution I see is recoding the ratings to avoid the value of zero but,
since I would have to keep in my dataset (which is very large!) the original
ratings (which are the ones that are clinically meaningful), I would end up
having so many variables that it would all get very confusing and it seems a
pity, especially if the error message is due to me making some mistake I am
unaware of.

2) I then thought I would see if one can use ratings of zero with the IBM
SPSS syntax for weighted kappa
(http://www-01.ibm.com/support/docview.wss?uid=swg21477357). I first tried
the syntax out on a simplified "dummy dataset" in which ratings go from 1 to
4 instead of 0 to 4, with rater 1 and rater 2 rating atrophy on the right
and left side of the brain and 4 variables (atrophyright1, atrophyright2,
atrophyleft1, atrophyleft2). However, I never got to the next step of trying
with ratings of zero because I keep getting error messages. I have trouble
understanding the syntax and I must be putting variables in the wrong place.
I have looked at other examples of syntax for weighted kappa
(http://www.spssusers.co.uk/Events/2015/ALDRIDGE2015.pdf, Valiquette et al
1994 Computing kappa coefficients using SPSS matrix) but it hasn't helped me
get it right.

I have used the following syntax for inter-rater reproducibility for atrophy
on the right side of the brain and my ratings matrix is underneath here. I
am showing in italics the bits I put in the syntax.

Rater 1

1 2 3 4
1 3 2 1 1
2 2 4 2 0 Rater 2
3 1 2 3 2
4 1 2 2 3

GET x /var=/atrophyright1 atrophyright 2/
DATA LIST LIST / /x1 to x4/
BEGIN DATA
/3211
2420
1232
1223/
END DATA.
matrix.
GET x /var=/x1 to x4/.
compute wt1=make(nrow(x),ncol(x),0).
compute wt2=make(nrow(x),ncol(x),0).
compute prop=x/msum(x).
loop i=1 to nrow(x).
loop j=1 to ncol(x).
compute wt1(i,j)=1-(abs(i-j)/(nrow(x)-1)).
compute wt2(i,j)=1-((i-j)/(nrow(x)-1))**2.
end loop.
end loop.
compute
wk1num=msum(wt1&*prop)-msum(mdiag(rsum(prop))*wt1*mdiag(csum(prop))).
compute wk1den=1-msum(mdiag(rsum(prop))*wt1*mdiag(csum(prop))).
compute wk1=wk1num/wk1den.
print wk1.
compute
wk2num=msum(wt2&*prop)-msum(mdiag(rsum(prop))*wt2*mdiag(csum(prop))).
compute wk2den=1-msum(mdiag(rsum(prop))*wt2*mdiag(csum(prop))).
compute wk2=wk2num/wk2den.
print wk2.
end matrix.

Thanks in advance for any help you can give me!
Paola

--
Sent from: http://spssx-discussion.1045642.n5.nabble.com/

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

bdates

Re: SPSS weighted kappa error message and syntax

Paola,

You need to review the Data List List command. You have slashes, "/", where none belong. Rather than point these out, I corrected your syntax to be the same as Vicki Aldridge's. Compare this to what you sent.

DATA LIST LIST /x1 to x4 .

BEGIN DATA

3 2 1 1

2 4 2 0

1 2 3 2

1 2 2 3

END DATA.

matrix.

GET x /var=x1 to x4.

Etc......

When I ran this it worked just fine. Also, there are several seminal articles that demonstrate the equivalence of weighted kappa to the intraclass correlation coefficient, so you may prefer that instead.

Brian

From: SPSSX(r) Discussion <[hidden email]> on behalf of paola <[hidden email]>
Sent: Thursday, January 3, 2019 6:00:31 AM
To: [hidden email]
Subject: SPSS weighted kappa error message and syntax

Hallo,

I am working with ordinal scales for cerebral atrophy (ratings 0=no atrophy
to 4=severe atrophy) and vascular burden (ratings 0=no burden to 3=severe
burden) assessed by two raters and I want to calculate intra- and
inter-rater reliability. I am having a few problems:

1) Since I am dealing with an ordinal scale I have decided to use weighted
kappa. I have installed the extension bundle for weighted kappa from SPSS
(version 25) but I keep getting the error message "some ratings are less
than one this command is not executed". I have looked through the threads in
stats forums and in the Internet but as far as I can see nobody has come up
with this issue. I know from the literature that intra- and inter-rater
reliability are commonly calculated for these scales using weighted kappa.
Also, weighted kappa seems to be applicable to ratings of zero
inSPSS(e.g.http://www.spssusers.co.uk/Events/2015/ALDRIDGE2015.pdf). The
only solution I see is recoding the ratings to avoid the value of zero but,
since I would have to keep in my dataset (which is very large!) the original
ratings (which are the ones that are clinically meaningful), I would end up
having so many variables that it would all get very confusing and it seems a
pity, especially if the error message is due to me making some mistake I am
unaware of.

2) I then thought I would see if one can use ratings of zero with the IBM
SPSS syntax for weighted kappa
(http://www-01.ibm.com/support/docview.wss?uid=swg21477357). I first tried
the syntax out on a simplified "dummy dataset" in which ratings go from 1 to
4 instead of 0 to 4, with rater 1 and rater 2 rating atrophy on the right
and left side of the brain and 4 variables (atrophyright1, atrophyright2,
atrophyleft1, atrophyleft2). However, I never got to the next step of trying
with ratings of zero because I keep getting error messages. I have trouble
understanding the syntax and I must be putting variables in the wrong place.
I have looked at other examples of syntax for weighted kappa
(http://www.spssusers.co.uk/Events/2015/ALDRIDGE2015.pdf, Valiquette et al
1994 Computing kappa coefficients using SPSS matrix) but it hasn't helped me
get it right.

I have used the following syntax for inter-rater reproducibility for atrophy
on the right side of the brain and my ratings matrix is underneath here. I
am showing in italics the bits I put in the syntax.


        Rater 1

        1      2        3       4
1       3       2       1       1
2       2       4       2       0        Rater 2
3       1       2       3       2
4       1       2       2       3

GET x /var=/atrophyright1 atrophyright 2/
DATA LIST LIST / /x1 to x4/
BEGIN DATA
/3211
2420
1232
1223/
END DATA.
matrix.
GET x /var=/x1 to x4/.
compute wt1=make(nrow(x),ncol(x),0).
compute wt2=make(nrow(x),ncol(x),0).
compute prop=x/msum(x).
loop i=1 to nrow(x).
loop j=1 to ncol(x).
compute wt1(i,j)=1-(abs(i-j)/(nrow(x)-1)).
compute wt2(i,j)=1-((i-j)/(nrow(x)-1))**2.
end loop.
end loop.
compute
wk1num=msum(wt1&*prop)-msum(mdiag(rsum(prop))*wt1*mdiag(csum(prop))).
compute wk1den=1-msum(mdiag(rsum(prop))*wt1*mdiag(csum(prop))).
compute wk1=wk1num/wk1den.
print wk1.
compute
wk2num=msum(wt2&*prop)-msum(mdiag(rsum(prop))*wt2*mdiag(csum(prop))).
compute wk2den=1-msum(mdiag(rsum(prop))*wt2*mdiag(csum(prop))).
compute wk2=wk2num/wk2den.
print wk2.
end matrix.

Thanks in advance for any help you can give me!
Paola

--
Sent from: http://spssx-discussion.1045642.n5.nabble.com/

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD

paola

Re: SPSS weighted kappa error message and syntax

In reply to this post by paola

Thank you so much! I apologize for the double slashes in the syntax: I was
clumsily trying to put some of the actual syntax in italics... I saw from
your syntax that the big mistake I was making was starting with the GET part
(plus I had missed a full stop after x 1 to x 4!). As you showed me,
starting with DATA LIST LIST all works well. From the IBM site syntax I was
under the impression that with the GET command SPSS would also compute the
matrix for me, which I don't know if it does. At the moment I am manually
inputting the matrix from the SPSS contingency table, it's a bit
time-consuming (some of my real dataset matrices are 9 x 9) but by copying
and pasting the format changes in a way that it is not recognized by the
syntax.
I also highly appreciate your suggestion of using the ICC. In fact I had
another question on ordinal variables which I was not sure from the forum
guidelines I could post since it is not technical. I am writing it now in my
reply here but if you or anyone feels it is not appropriate I will be happy
to re-edit the post and/or delete thus part. Clinically it is common to show
and discuss the average value of atrophy for the brain (i.e the average of
the right and left sides) although the scale is ordinal. I know that taking
an average of ordinal variables is a complex issue but I have seen papers
give kappa (unweighted) for the average of ordinal variables, possibly
because it is considered appropriate to give some reliability value for the
measures in the form in which they are subsequently analyzed (e.g. analyzing
correlations between, say, a neuropsychological test and the atrophy of
different structures of the brain one would use the average value of atrophy
for each structure and not the separate right and left values, even to
minimize multiple comparisons). Yet, if I run (unweighted) kappa for the
average of atrophy ((left+right)/2) I get kappa values which are very much
smaller than the original kappa values for the left and right sides
considered separately. I don't understand why this happens. I would have
expected a larger kappa since there might be perfect agreement in the
average value without there being perfect agreement in the left and right
ratings (e.g. 3 left and 2 right gives the same average as 2 left and 3
right). Or maybe it's just because of this : more agreements being
expected to occur by chance? I guess the ICC might solve this problem. If
you have any reference to give me I would greatly appreciate it (all I seem
to find is literature citing weighted kappa for ordinal variables and the
ICC for continuous ones).

--
Sent from: http://spssx-discussion.1045642.n5.nabble.com/

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

Bruce Weaver

Re: SPSS weighted kappa error message and syntax

Administrator

In reply to this post by bdates

To Brian's point about the equivalence of weighted kappa (with quadratic
weights) and the ICC, here is a demo I cobbled together:

https://sites.google.com/a/lakeheadu.ca/bweaver/Home/statistics/files/Weighted_kappa_%26_ICC.sps

Notice that the ICC needed to show this equivalence requires these settings
on the RELIABILITY command:

/ICC=MODEL(MIXED) TYPE(CONSISTENCY)

Notice too that when the marginal distributions for the contingency table
are the same, weighted kappa = ICC = Pearson r.

But finally, notice too that the CI for weighted kappa (obtained from the
Vassar Stats online calculator) is wider than the CI for the ICC.

HTH.

bdates wrote

> --- snip ---
>
> Also, there are several seminal articles that demonstrate the equivalence
> of weighted kappa to the intraclass correlation coefficient, so you may
> prefer that instead.
>
>
> Brian
>
> --- snip ---

-----
--
Bruce Weaver
[hidden email]
http://sites.google.com/a/lakeheadu.ca/bweaver/

"When all else fails, RTFM."

NOTE: My Hotmail account is not monitored regularly.
To send me an e-mail, please use the address shown above.

--
Sent from: http://spssx-discussion.1045642.n5.nabble.com/

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

--
Bruce Weaver
bweaver@lakeheadu.ca
http://sites.google.com/a/lakeheadu.ca/bweaver/

"When all else fails, RTFM."

PLEASE NOTE THE FOLLOWING:
1. My Hotmail account is not monitored regularly. To send me an e-mail, please use the address shown above.
2. The SPSSX Discussion forum on Nabble is no longer linked to the SPSSX-L listserv administered by UGA (https://listserv.uga.edu/).

Rich Ulrich

Re: SPSS weighted kappa error message and syntax

In reply to this post by paola

2x2 kappa is fine. Unweighted kappa on any larger table is problematic.

It is highly dependent on marginal counts, and it counts as "correct" only

the table's diagonal, the matches which are exact - so, splitting categories

won't increase kappa (and - I think - it can't).

The only time I would ever consider showing unweighted kappa for a table

larger than 2x2 is when someone wants to see a set of results on a

several comparisons which are highly parallel. Like, "Here are the kappas

on the likert-type items which were used to construct the overall scale."

But I would try to talk the PI out of it unless there was some amazing

result to be illustrated.

Rich Ulrich

From: SPSSX(r) Discussion <[hidden email]> on behalf of paola <[hidden email]>
Sent: Friday, January 4, 2019 1:14 PM
To: [hidden email]
Subject: Re: SPSS weighted kappa error message and syntax

paola

Re: SPSS weighted kappa error message and syntax

Thank you very much Ulrich, now I understand it may be a problem of marginal
counts. I had read about the "kappa paradox" in stats forum threads and
papers but your explanation brought it home to me . The reason why I was
trying to use unweighted kappa is that we want to show our ratings are "very
reliable", so I think I need a measure of perfect agreement. I understand
weighted kappa is less conservative because it also takes into account the
discrepancy between raters. I perfectly understand the example and the point
you make. I think that if we were validating a brain atrophy scale weighted
kappa would be fine because the difference in, say, a rating of 0 and 1 is
less than between 0 and 2 and in the first case it may not be clinically
relevant (scores below 2 are considered non pathological). However, we are
not interested in whether the score is pathological or not but in how much
the scores from the two raters (or from one rater) are exactly the same. Do
you think I could maybe use the ICC (model: two-way random, single measures,
agreement)?

--
Sent from: http://spssx-discussion.1045642.n5.nabble.com/

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

Rich Ulrich

Re: SPSS weighted kappa error message and syntax

A few days ago, you seemed to direct this question to me.

I must admit that I don't know which procedure you are

referring to, if there is an ICC particularly on a dichotomy,

for Agreement.

I want to add some comments about the doctors who read

brain scans. I heard, long ago, that my own, short experience

did generalize - Namely, these guys are extremely proud about

the lack of ambiguity of their own "readings". They did NOT

like to think about "reliability" and, worse, "validity". However,

my brush with the field was 20 years ago. Maybe they are

more sensible today.

You probably can't do useful test-retest, even after 6 months,

because an excellent visual memory (assumed) will tell them

that they've seen this X-ray or scan before, and what they

concluded. I suspect that an alternate-view would also be

recognized. (I would like to see a discussion between two

experts who disagreed about a single chart.)

If you point to a fatal outcome, one might rationalize that he

READ the picture right; he did not attempt to predict. Or,

in the other direction, one might say that he was "giving the

best prediction" rather than the simplest diagnosis. What I

am saying is that the /judgments/ that you ask for should

be plural, not singular. (a) Is it "bad" (Y/N)? (b) What chance

(percentage) is there that the same brain will look bad in a

few months? (c) Did it look bad from every view (if there

were several) ?

Rich Ulrich

From: SPSSX(r) Discussion <[hidden email]> on behalf of paola <[hidden email]>
Sent: Monday, January 7, 2019 12:53 PM
To: [hidden email]
Subject: Re: SPSS weighted kappa error message and syntax