SPSSX Discussion

Agreement of Raters in one item

Classic

List

Threaded

14 messages Options

E. Bernardo

Agreement of Raters in one item

Dear All,

A question of somebody was sent to me last day. It's a basic question but I dont know the answer, thus, I forward it to this list.

Here is the question: An item (3-point response options such as not appropriate, appropriate, not appropriate) is rated by 26 raters. Can you suggest a coefficient that measures the agreement of the responses among raters?

Best,
Eins

Gosse, Michelle

Automatic reply: Agreement of Raters in one item

I am away from the office until 10 December, and will respond to your email on my return.

cheers
Michelle

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

David Marso

Re: Agreement of Raters in one item

Administrator

In reply to this post by E. Bernardo

I think you need to put a little more thought into presentation of your question.
FWICT you simply have a 26 x 3 XTAB. Not a lot you can do with that .

------------

Eins Bernardo wrote

Dear All,

A question of somebody was sent to me last day. It's a basic question but I dont know the answer, thus, I forward it to this list.

Here is the question: An
item (3-point response options such as not appropriate, appropriate, not appropriate) is rated by 26 raters. Can you
suggest a coefficient that measures the agreement of the responses
among raters?

Best,
Eins

Please reply to the list and not to my personal email.
Those desiring my consulting or training services please feel free to email me.
---
"Nolite dare sanctum canibus neque mittatis margaritas vestras ante porcos ne forte conculcent eas pedibus suis."
Cum es damnatorum possederunt porcos iens ut salire off sanguinum cliff in abyssum?"

David Marso

Re: Agreement of Raters in one item

Administrator

In reply to this post by E. Bernardo

*AND* You might want to consult a dictionary for the term reciprocity!!!
If for some reason this tap of the clue stick goes over your head I would be happy to elaborate!
-------------------

Eins Bernardo wrote

Dear All,

A question of somebody was sent to me last day. It's a basic question but I dont know the answer, thus, I forward it to this list.

Here is the question: An
item (3-point response options such as not appropriate, appropriate, not appropriate) is rated by 26 raters. Can you
suggest a coefficient that measures the agreement of the responses
among raters?

Best,
Eins

E. Bernardo

Re: Agreement of Raters in one item

In reply to this post by David Marso

Marso,

In other words, you claimed that we cannot compute the level of agreement of raters if there is only one subject/item being rated?

I have no idea about it that is why I forwarded the question to the list.

Eins

From: David Marso <[hidden email]>
To: [hidden email]
Sent: Wednesday, December 5, 2012 9:42 PM
Subject: Re: Agreement of Raters in one item

I think you need to put a little more thought into presentation of your
question.
FWICT you simply have a 26 x 3 XTAB. Not a lot you can do with that .

------------

Eins Bernardo wrote

> Dear All,
>
> A question of somebody was sent to me last day.Â It's a basic question but
> I dont know the answer, thus, I forward it to this list.
>
> Here is the question: An
> item (3-point response options such as not appropriate, appropriate, not
> appropriate) is rated by 26 raters.Â Can you
> suggest a coefficient that measures the agreement of the responses
> among raters?
>
> Best,
> Eins

-----
Please reply to the list and not to my personal email.
Those desiring my consulting or training services please feel free to email me.
--
View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Agreement-of-Raters-in-one-item-tp5716702p5716706.html
Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

Rich Ulrich

Re: Agreement of Raters in one item

In reply to this post by E. Bernardo

When you have one score from each of 26 raters/subjects,
then the "agreement" is like the "agreement" of (say) age or sex --

For a categorical variable, you can report that "umpteen percent were X"
or give the fractions for each of several categories. (For questions like
"political party membership", you might consider an "index of diversity"
if you were going to compare several countries eventually.)

For a measure that you consider continuous, the standard deviation
shows the "agreement" or similarity. A confidence interval is also
sometimes appropriate.

--
Rich Ulrich

Date: Thu, 6 Dec 2012 11:52:49 +0800
From: [hidden email]
Subject: Agreement of Raters in one item
To: [hidden email]

Dear All,

ViAnn Beadle

Re: Agreement of Raters in one item

How about a 1-sample chi-square?

From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Rich Ulrich
Sent: Thursday, December 06, 2012 12:10 PM
To: [hidden email]
Subject: Re: Agreement of Raters in one item

Date: Thu, 6 Dec 2012 11:52:49 +0800
From: [hidden email]
Subject: Agreement of Raters in one item
To: [hidden email]

Dear All,

Bruce Weaver

Re: Agreement of Raters in one item

Administrator

I was thinking of a chi-square goodness of fit too. Complete agreement would have all of the observations in one category, whereas the opposite extreme would have equal numbers of observations in each of the categories (and chi-square = 0). For a given sample size and number of categories, you could work out the maximum possible value of chi-square (with all observations in a single category), and possibly use that as the denominator of a ratio -- observed chi-square / max possible chi-square. But I tried a little demo (see below), and it looks like the scaling is not right -- i.e., the increase in that ratio is not a nice linear function of "agreement".

When that didn't work out so well, I tried Rich's idea of using the standard deviation. That worked quite a bit better. Here is the syntax for my little demo.

data list list / O1 to O3 (3f2.0).
begin data
26 0 0
25 1 0
24 1 1
22 2 2
20 3 3
18 4 4
16 5 5
14 6 6
12 7 7
10 8 8
9 9 8
end data.

compute N = sum(O1 to O3).
compute E = N / nvalid(O1 to O3).
vector O = O1 to O3.
compute chisqr = 0.
loop # = 1 to 3.
. compute chisqr = chisqr + (O(#)-E)**2/E.
end loop.
compute StD = SD(O1 to O3).
execute.

AGGREGATE
/OUTFILE=* MODE=ADDVARIABLES
/BREAK=
/MaxChisqr=MAX(chisqr)
/MaxSD = MAX(StD)
.

compute CSratio = chisqr / MaxChisqr.
compute SDratio = StD/MaxSD.
formats chisqr CSratio SDratio(f8.3) / N (F5.0).
list.

GRAPH /SCATTERPLOT(BIVAR)=O1 WITH chisqr .
GRAPH /SCATTERPLOT(BIVAR)=O1 WITH CSratio .

GRAPH /SCATTERPLOT(BIVAR)=O1 WITH StD .
GRAPH /SCATTERPLOT(BIVAR)=O1 WITH SDratio .

For those who don't have access to SPSS, here is the output from the LIST command -- you might wish to generate your own scatter-plots using some other software. The bottom line is that the last two scatter-plots show nice linear relationships, whereas the first two show curvilinear relationships that rise slowly for lower values of O1, and more quickly for higher values of O1.

OUTPUT from LIST:

O1 O2 O3 N E chisqr StD MaxChisqr MaxSD CSratio SDratio

26 0 0 26 8.67 52.000 15.01 52.00 15.01 1.000 1.000
25 1 0 26 8.67 46.231 14.15 52.00 15.01 .889 .943
24 1 1 26 8.67 40.692 13.28 52.00 15.01 .783 .885
22 2 2 26 8.67 30.769 11.55 52.00 15.01 .592 .769
20 3 3 26 8.67 22.231 9.81 52.00 15.01 .428 .654
18 4 4 26 8.67 15.077 8.08 52.00 15.01 .290 .538
16 5 5 26 8.67 9.308 6.35 52.00 15.01 .179 .423
14 6 6 26 8.67 4.923 4.62 52.00 15.01 .095 .308
12 7 7 26 8.67 1.923 2.89 52.00 15.01 .037 .192
10 8 8 26 8.67 .308 1.15 52.00 15.01 .006 .077
9 9 8 26 8.67 .077 .58 52.00 15.01 .001 .038

Number of cases read: 11 Number of cases listed: 11

HTH.

ViAnn Beadle wrote

How about a 1-sample chi-square?

From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of
Rich Ulrich
Sent: Thursday, December 06, 2012 12:10 PM
To: [hidden email]
Subject: Re: Agreement of Raters in one item

When you have one score from each of 26 raters/subjects,
then the "agreement" is like the "agreement" of (say) age or sex --

For a categorical variable, you can report that "umpteen percent were X"
or give the fractions for each of several categories. (For questions like
"political party membership", you might consider an "index of diversity"
if you were going to compare several countries eventually.)

For a measure that you consider continuous, the standard deviation
shows the "agreement" or similarity. A confidence interval is also
sometimes appropriate.

--
Rich Ulrich

_____

Date: Thu, 6 Dec 2012 11:52:49 +0800
From: [hidden email]
Subject: Agreement of Raters in one item
To: [hidden email]

Dear All,

A question of somebody was sent to me last day. It's a basic question but I
dont know the answer, thus, I forward it to this list.

Here is the question: An item (3-point response options such as not
appropriate, appropriate, not appropriate) is rated by 26 raters. Can you
suggest a coefficient that measures the agreement of the responses among
raters?

Best,
Eins

--
Bruce Weaver
bweaver@lakeheadu.ca
http://sites.google.com/a/lakeheadu.ca/bweaver/

"When all else fails, RTFM."

PLEASE NOTE THE FOLLOWING:
1. My Hotmail account is not monitored regularly. To send me an e-mail, please use the address shown above.
2. The SPSSX Discussion forum on Nabble is no longer linked to the SPSSX-L listserv administered by UGA (https://listserv.uga.edu/).

E. Bernardo

Re: Agreement of Raters in one item

In reply to this post by ViAnn Beadle

Thank you to those who responded and gave their suggestions regarding the agreement of raters on one item/subject.

The researcher (undergrad student) finally decided to use 1-sample chi-square to address his question.

Eins

From: ViAnn Beadle <[hidden email]>
To: [hidden email]
Sent: Friday, December 7, 2012 3:19 AM
Subject: Re: Agreement of Raters in one item

How about a 1-sample chi-square?

Date: Thu, 6 Dec 2012 11:52:49 +0800
From: [hidden email]
Subject: Agreement of Raters in one item
To: [hidden email]

Dear All,

Rich Ulrich

Re: Agreement of Raters in one item

In reply to this post by Bruce Weaver

[2nd attempt to post this. Though I see the OP has decided.]
What might be Bruce's best suggestion here is his demonstration --
Cobble up your own representative data, compute some indexes,
and see what works best for your own circumstances.

There are a half dozen other possibilities that you can find detailed
in the Wikip page on Diversity Index - assuming that Diversity is an
appropriate reciprocal to concordance. That page mentions ecology
as the area of study that uses Diversity. I had previously mentioned
the political science concern with the "number of political parties."

The problem that I have seen in applying these indexes mainly arise
when you want a number that can compare different numbers of
categories. - If you know that you have, say, exactly 3 categories, then
I think that all of the measures on the Wikip might give the same ordering,
though not the same intervals. (Bruce mentions curvilinearity.)

But - Do consider that you have "more diversity" when you have [1 big count
plus a dozen tiny ones] or [several moderate counts]? And - Is the "total
count" useful in some way? -- as it might be, for counting species found within
a fixed area. Different applications require different answers.

--
Rich Ulrich

> Date: Thu, 6 Dec 2012 13:47:12 -0800

> From: [hidden email]
> Subject: Re: Agreement of Raters in one item
> To: [hidden email]
>
> I was thinking of a chi-square goodness of fit too. Complete agreement would
> have all of the observations in one category, whereas the opposite extreme
> would have equal numbers of observations in each of the categories (and
> chi-square = 0). For a given sample size and number of categories, you
> could work out the maximum possible value of chi-square (with all
> observations in a single category), and possibly use that as the denominator
> of a ratio -- observed chi-square / max possible chi-square. But I tried a
> little demo (see below), and it looks like the scaling is not right -- i.e.,
> the increase in that ratio is not a nice linear function of "agreement".
>
> When that didn't work out so well, I tried Rich's idea of using the standard
> deviation. That worked quite a bit better. Here is the syntax for my
> little demo.
>

... snip, details

Ryan

Re: Agreement of Raters in one item

In reply to this post by Bruce Weaver

It is possible to obtain a p-value under a scenario where cell(s) include zero frequencies. Furthermore, if the expected values are less than 5, it is possible to obtain exact p-values under scenarios with and without zero frequency cells.

Ryan

On Dec 6, 2012, at 4:47 PM, Bruce Weaver <[hidden email]> wrote:

> I was thinking of a chi-square goodness of fit too. Complete agreement would
> have all of the observations in one category, whereas the opposite extreme
> would have equal numbers of observations in each of the categories (and
> chi-square = 0). For a given sample size and number of categories, you
> could work out the maximum possible value of chi-square (with all
> observations in a single category), and possibly use that as the denominator
> of a ratio -- observed chi-square / max possible chi-square. But I tried a
> little demo (see below), and it looks like the scaling is not right -- i.e.,
> the increase in that ratio is not a nice linear function of "agreement".
>
> When that didn't work out so well, I tried Rich's idea of using the standard
> deviation. That worked quite a bit better. Here is the syntax for my
> little demo.
>
>
> data list list / O1 to O3 (3f2.0).
> begin data
> 26 0 0
> 25 1 0
> 24 1 1
> 22 2 2
> 20 3 3
> 18 4 4
> 16 5 5
> 14 6 6
> 12 7 7
> 10 8 8
> 9 9 8
> end data.
>
> compute N = sum(O1 to O3).
> compute E = N / nvalid(O1 to O3).
> vector O = O1 to O3.
> compute chisqr = 0.
> loop # = 1 to 3.
> . compute chisqr = chisqr + (O(#)-E)**2/E.
> end loop.
> compute StD = SD(O1 to O3).
> execute.
>
> AGGREGATE
> /OUTFILE=* MODE=ADDVARIABLES
> /BREAK=
> /MaxChisqr=MAX(chisqr)
> /MaxSD = MAX(StD)
> .
>
> compute CSratio = chisqr / MaxChisqr.
> compute SDratio = StD/MaxSD.
> formats chisqr CSratio SDratio(f8.3) / N (F5.0).
> list.
>
> GRAPH /SCATTERPLOT(BIVAR)=O1 WITH chisqr .
> GRAPH /SCATTERPLOT(BIVAR)=O1 WITH CSratio .
>
> GRAPH /SCATTERPLOT(BIVAR)=O1 WITH StD .
> GRAPH /SCATTERPLOT(BIVAR)=O1 WITH SDratio .
>
>
> For those who don't have access to SPSS, here is the output from the LIST
> command -- you might wish to generate your own scatter-plots using some
> other software. The bottom line is that the last two scatter-plots show
> nice linear relationships, whereas the first two show curvilinear
> relationships that rise slowly for lower values of O1, and more quickly for
> higher values of O1.
>
> OUTPUT from LIST:
>
> O1 O2 O3 N E chisqr StD MaxChisqr MaxSD CSratio
> SDratio
>
> 26 0 0 26 8.67 52.000 15.01 52.00 15.01 1.000
> 1.000
> 25 1 0 26 8.67 46.231 14.15 52.00 15.01 .889
> .943
> 24 1 1 26 8.67 40.692 13.28 52.00 15.01 .783
> .885
> 22 2 2 26 8.67 30.769 11.55 52.00 15.01 .592
> .769
> 20 3 3 26 8.67 22.231 9.81 52.00 15.01 .428
> .654
> 18 4 4 26 8.67 15.077 8.08 52.00 15.01 .290
> .538
> 16 5 5 26 8.67 9.308 6.35 52.00 15.01 .179
> .423
> 14 6 6 26 8.67 4.923 4.62 52.00 15.01 .095
> .308
> 12 7 7 26 8.67 1.923 2.89 52.00 15.01 .037
> .192
> 10 8 8 26 8.67 .308 1.15 52.00 15.01 .006
> .077
> 9 9 8 26 8.67 .077 .58 52.00 15.01 .001
> .038
>
> Number of cases read: 11 Number of cases listed: 11
>
> HTH.
>
>
>
> ViAnn Beadle wrote
>> How about a 1-sample chi-square?
>>
>>
>>
>> From: SPSSX(r) Discussion [mailto:
>
>> SPSSX-L@.UGA
>
>> ] On Behalf Of
>> Rich Ulrich
>> Sent: Thursday, December 06, 2012 12:10 PM
>> To:
>
>> SPSSX-L@.UGA
>
>> Subject: Re: Agreement of Raters in one item
>>
>>
>>
>> When you have one score from each of 26 raters/subjects,
>> then the "agreement" is like the "agreement" of (say) age or sex --
>>
>> For a categorical variable, you can report that "umpteen percent were X"
>> or give the fractions for each of several categories. (For questions like
>> "political party membership", you might consider an "index of diversity"
>> if you were going to compare several countries eventually.)
>>
>> For a measure that you consider continuous, the standard deviation
>> shows the "agreement" or similarity. A confidence interval is also
>> sometimes appropriate.
>>
>> --
>> Rich Ulrich
>>
>>
>>
>> _____
>>
>> Date: Thu, 6 Dec 2012 11:52:49 +0800
>> From:
>
>> einsbernardo@.com
>
>> Subject: Agreement of Raters in one item
>> To:
>
>> SPSSX-L@.UGA
>
>>
>> Dear All,
>>
>>
>> A question of somebody was sent to me last day. It's a basic question but
>> I
>> dont know the answer, thus, I forward it to this list.
>>
>> Here is the question: An item (3-point response options such as not
>> appropriate, appropriate, not appropriate) is rated by 26 raters. Can you
>> suggest a coefficient that measures the agreement of the responses among
>> raters?
>>
>> Best,
>> Eins
>
>
>
>
>
> -----
> --
> Bruce Weaver
> [hidden email]
> http://sites.google.com/a/lakeheadu.ca/bweaver/
>
> "When all else fails, RTFM."
>
> NOTE: My Hotmail account is not monitored regularly.
> To send me an e-mail, please use the address shown above.
>
> --
> View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Agreement-of-Raters-in-one-item-tp5716702p5716738.html
> Sent from the SPSSX Discussion mailing list archive at Nabble.com.
>
> =====================
> To manage your subscription to SPSSX-L, send a message to
> [hidden email] (not to SPSSX-L), with no body text except the
> command. To leave the list, send the command
> SIGNOFF SPSSX-L
> For a list of commands to manage subscriptions, send the command
> INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

Bruce Weaver

Re: Agreement of Raters in one item

Administrator

I don't disagree with what you're saying Ryan, but I don't understand why you're saying it. Are you suggesting using the p-value from a chi-square goodness of fit test as the index of agreement the OP was looking for? Thanks for clarifying.

Bruce

R B wrote

It is possible to obtain a p-value under a scenario where cell(s) include zero frequencies. Furthermore, if the expected values are less than 5, it is possible to obtain exact p-values under scenarios with and without zero frequency cells.

Ryan

On Dec 6, 2012, at 4:47 PM, Bruce Weaver <[hidden email]> wrote:

> I was thinking of a chi-square goodness of fit too. Complete agreement would
> have all of the observations in one category, whereas the opposite extreme
> would have equal numbers of observations in each of the categories (and
> chi-square = 0). For a given sample size and number of categories, you
> could work out the maximum possible value of chi-square (with all
> observations in a single category), and possibly use that as the denominator
> of a ratio -- observed chi-square / max possible chi-square. But I tried a
> little demo (see below), and it looks like the scaling is not right -- i.e.,
> the increase in that ratio is not a nice linear function of "agreement".
>
> When that didn't work out so well, I tried Rich's idea of using the standard
> deviation. That worked quite a bit better. Here is the syntax for my
> little demo.
>
>
> data list list / O1 to O3 (3f2.0).
> begin data
> 26 0 0
> 25 1 0
> 24 1 1
> 22 2 2
> 20 3 3
> 18 4 4
> 16 5 5
> 14 6 6
> 12 7 7
> 10 8 8
> 9 9 8
> end data.
>
> compute N = sum(O1 to O3).
> compute E = N / nvalid(O1 to O3).
> vector O = O1 to O3.
> compute chisqr = 0.
> loop # = 1 to 3.
> . compute chisqr = chisqr + (O(#)-E)**2/E.
> end loop.
> compute StD = SD(O1 to O3).
> execute.
>
> AGGREGATE
> /OUTFILE=* MODE=ADDVARIABLES
> /BREAK=
> /MaxChisqr=MAX(chisqr)
> /MaxSD = MAX(StD)
> .
>
> compute CSratio = chisqr / MaxChisqr.
> compute SDratio = StD/MaxSD.
> formats chisqr CSratio SDratio(f8.3) / N (F5.0).
> list.
>
> GRAPH /SCATTERPLOT(BIVAR)=O1 WITH chisqr .
> GRAPH /SCATTERPLOT(BIVAR)=O1 WITH CSratio .
>
> GRAPH /SCATTERPLOT(BIVAR)=O1 WITH StD .
> GRAPH /SCATTERPLOT(BIVAR)=O1 WITH SDratio .
>
>
> For those who don't have access to SPSS, here is the output from the LIST
> command -- you might wish to generate your own scatter-plots using some
> other software. The bottom line is that the last two scatter-plots show
> nice linear relationships, whereas the first two show curvilinear
> relationships that rise slowly for lower values of O1, and more quickly for
> higher values of O1.
>
> OUTPUT from LIST:
>
> O1 O2 O3 N E chisqr StD MaxChisqr MaxSD CSratio
> SDratio
>
> 26 0 0 26 8.67 52.000 15.01 52.00 15.01 1.000
> 1.000
> 25 1 0 26 8.67 46.231 14.15 52.00 15.01 .889
> .943
> 24 1 1 26 8.67 40.692 13.28 52.00 15.01 .783
> .885
> 22 2 2 26 8.67 30.769 11.55 52.00 15.01 .592
> .769
> 20 3 3 26 8.67 22.231 9.81 52.00 15.01 .428
> .654
> 18 4 4 26 8.67 15.077 8.08 52.00 15.01 .290
> .538
> 16 5 5 26 8.67 9.308 6.35 52.00 15.01 .179
> .423
> 14 6 6 26 8.67 4.923 4.62 52.00 15.01 .095
> .308
> 12 7 7 26 8.67 1.923 2.89 52.00 15.01 .037
> .192
> 10 8 8 26 8.67 .308 1.15 52.00 15.01 .006
> .077
> 9 9 8 26 8.67 .077 .58 52.00 15.01 .001
> .038
>
> Number of cases read: 11 Number of cases listed: 11
>
> HTH.
>
>
>
> ViAnn Beadle wrote
>> How about a 1-sample chi-square?
>>
>>
>>
>> From: SPSSX(r) Discussion [mailto:
>
>> SPSSX-L@.UGA
>
>> ] On Behalf Of
>> Rich Ulrich
>> Sent: Thursday, December 06, 2012 12:10 PM
>> To:
>
>> SPSSX-L@.UGA
>
>> Subject: Re: Agreement of Raters in one item
>>
>>
>>
>> When you have one score from each of 26 raters/subjects,
>> then the "agreement" is like the "agreement" of (say) age or sex --
>>
>> For a categorical variable, you can report that "umpteen percent were X"
>> or give the fractions for each of several categories. (For questions like
>> "political party membership", you might consider an "index of diversity"
>> if you were going to compare several countries eventually.)
>>
>> For a measure that you consider continuous, the standard deviation
>> shows the "agreement" or similarity. A confidence interval is also
>> sometimes appropriate.
>>
>> --
>> Rich Ulrich
>>
>>
>>
>> _____
>>
>> Date: Thu, 6 Dec 2012 11:52:49 +0800
>> From:
>
>> einsbernardo@.com
>
>> Subject: Agreement of Raters in one item
>> To:
>
>> SPSSX-L@.UGA
>
>>
>> Dear All,
>>
>>
>> A question of somebody was sent to me last day. It's a basic question but
>> I
>> dont know the answer, thus, I forward it to this list.
>>
>> Here is the question: An item (3-point response options such as not
>> appropriate, appropriate, not appropriate) is rated by 26 raters. Can you
>> suggest a coefficient that measures the agreement of the responses among
>> raters?
>>
>> Best,
>> Eins
>
>
>
>
>
> -----
> --
> Bruce Weaver
> [hidden email]
> http://sites.google.com/a/lakeheadu.ca/bweaver/
>
> "When all else fails, RTFM."
>
> NOTE: My Hotmail account is not monitored regularly.
> To send me an e-mail, please use the address shown above.
>
> --
> View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Agreement-of-Raters-in-one-item-tp5716702p5716738.html
> Sent from the SPSSX Discussion mailing list archive at Nabble.com.
>
> =====================
> To manage your subscription to SPSSX-L, send a message to
> [hidden email] (not to SPSSX-L), with no body text except the
> command. To leave the list, send the command
> SIGNOFF SPSSX-L
> For a list of commands to manage subscriptions, send the command
> INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

Ryan

Re: Agreement of Raters in one item

No. It was tangential. Just pointing out that, despite SPSS refusing
to include zero-frequency cells when calculating p-values, it is in
fact possible to derive a valid p-value.

Ryan

On Fri, Dec 7, 2012 at 9:13 AM, Bruce Weaver <[hidden email]> wrote:

> I don't disagree with what you're saying Ryan, but I don't understand why
> you're saying it. Are you suggesting using the p-value from a chi-square
> goodness of fit test as the index of agreement the OP was looking for?
> Thanks for clarifying.
>
> Bruce
>
>
>
> R B wrote
>> It is possible to obtain a p-value under a scenario where cell(s) include
>> zero frequencies. Furthermore, if the expected values are less than 5, it
>> is possible to obtain exact p-values under scenarios with and without zero
>> frequency cells.
>>
>> Ryan
>>
>> On Dec 6, 2012, at 4:47 PM, Bruce Weaver <
>
>> bruce.weaver@
>
>> > wrote:
>>
>>> I was thinking of a chi-square goodness of fit too. Complete agreement
>>> would
>>> have all of the observations in one category, whereas the opposite
>>> extreme
>>> would have equal numbers of observations in each of the categories (and
>>> chi-square = 0). For a given sample size and number of categories, you
>>> could work out the maximum possible value of chi-square (with all
>>> observations in a single category), and possibly use that as the
>>> denominator
>>> of a ratio -- observed chi-square / max possible chi-square. But I tried
>>> a
>>> little demo (see below), and it looks like the scaling is not right --
>>> i.e.,
>>> the increase in that ratio is not a nice linear function of "agreement".
>>>
>>> When that didn't work out so well, I tried Rich's idea of using the
>>> standard
>>> deviation. That worked quite a bit better. Here is the syntax for my
>>> little demo.
>>>
>>>
>>> data list list / O1 to O3 (3f2.0).
>>> begin data
>>> 26 0 0
>>> 25 1 0
>>> 24 1 1
>>> 22 2 2
>>> 20 3 3
>>> 18 4 4
>>> 16 5 5
>>> 14 6 6
>>> 12 7 7
>>> 10 8 8
>>> 9 9 8
>>> end data.
>>>
>>> compute N = sum(O1 to O3).
>>> compute E = N / nvalid(O1 to O3).
>>> vector O = O1 to O3.
>>> compute chisqr = 0.
>>> loop # = 1 to 3.
>>> . compute chisqr = chisqr + (O(#)-E)**2/E.
>>> end loop.
>>> compute StD = SD(O1 to O3).
>>> execute.
>>>
>>> AGGREGATE
>>> /OUTFILE=* MODE=ADDVARIABLES
>>> /BREAK=
>>> /MaxChisqr=MAX(chisqr)
>>> /MaxSD = MAX(StD)
>>> .
>>>
>>> compute CSratio = chisqr / MaxChisqr.
>>> compute SDratio = StD/MaxSD.
>>> formats chisqr CSratio SDratio(f8.3) / N (F5.0).
>>> list.
>>>
>>> GRAPH /SCATTERPLOT(BIVAR)=O1 WITH chisqr .
>>> GRAPH /SCATTERPLOT(BIVAR)=O1 WITH CSratio .
>>>
>>> GRAPH /SCATTERPLOT(BIVAR)=O1 WITH StD .
>>> GRAPH /SCATTERPLOT(BIVAR)=O1 WITH SDratio .
>>>
>>>
>>> For those who don't have access to SPSS, here is the output from the LIST
>>> command -- you might wish to generate your own scatter-plots using some
>>> other software. The bottom line is that the last two scatter-plots show
>>> nice linear relationships, whereas the first two show curvilinear
>>> relationships that rise slowly for lower values of O1, and more quickly
>>> for
>>> higher values of O1.
>>>
>>> OUTPUT from LIST:
>>>
>>> O1 O2 O3 N E chisqr StD MaxChisqr MaxSD CSratio
>>> SDratio
>>>
>>> 26 0 0 26 8.67 52.000 15.01 52.00 15.01 1.000
>>> 1.000
>>> 25 1 0 26 8.67 46.231 14.15 52.00 15.01 .889
>>> .943
>>> 24 1 1 26 8.67 40.692 13.28 52.00 15.01 .783
>>> .885
>>> 22 2 2 26 8.67 30.769 11.55 52.00 15.01 .592
>>> .769
>>> 20 3 3 26 8.67 22.231 9.81 52.00 15.01 .428
>>> .654
>>> 18 4 4 26 8.67 15.077 8.08 52.00 15.01 .290
>>> .538
>>> 16 5 5 26 8.67 9.308 6.35 52.00 15.01 .179
>>> .423
>>> 14 6 6 26 8.67 4.923 4.62 52.00 15.01 .095
>>> .308
>>> 12 7 7 26 8.67 1.923 2.89 52.00 15.01 .037
>>> .192
>>> 10 8 8 26 8.67 .308 1.15 52.00 15.01 .006
>>> .077
>>> 9 9 8 26 8.67 .077 .58 52.00 15.01 .001
>>> .038
>>>
>>> Number of cases read: 11 Number of cases listed: 11
>>>
>>> HTH.
>>>
>>>
>>>
>>> ViAnn Beadle wrote
>>>> How about a 1-sample chi-square?
>>>>
>>>>
>>>>
>>>> From: SPSSX(r) Discussion [mailto:
>>>
>>>> SPSSX-L@.UGA
>>>
>>>> ] On Behalf Of
>>>> Rich Ulrich
>>>> Sent: Thursday, December 06, 2012 12:10 PM
>>>> To:
>>>
>>>> SPSSX-L@.UGA
>>>
>>>> Subject: Re: Agreement of Raters in one item
>>>>
>>>>
>>>>
>>>> When you have one score from each of 26 raters/subjects,
>>>> then the "agreement" is like the "agreement" of (say) age or sex --
>>>>
>>>> For a categorical variable, you can report that "umpteen percent were X"
>>>> or give the fractions for each of several categories. (For questions
>>>> like
>>>> "political party membership", you might consider an "index of diversity"
>>>> if you were going to compare several countries eventually.)
>>>>
>>>> For a measure that you consider continuous, the standard deviation
>>>> shows the "agreement" or similarity. A confidence interval is also
>>>> sometimes appropriate.
>>>>
>>>> --
>>>> Rich Ulrich
>>>>
>>>>
>>>>
>>>> _____
>>>>
>>>> Date: Thu, 6 Dec 2012 11:52:49 +0800
>>>> From:
>>>
>>>> einsbernardo@.com
>>>
>>>> Subject: Agreement of Raters in one item
>>>> To:
>>>
>>>> SPSSX-L@.UGA
>>>
>>>>
>>>> Dear All,
>>>>
>>>>
>>>> A question of somebody was sent to me last day. It's a basic question
>>>> but
>>>> I
>>>> dont know the answer, thus, I forward it to this list.
>>>>
>>>> Here is the question: An item (3-point response options such as not
>>>> appropriate, appropriate, not appropriate) is rated by 26 raters. Can
>>>> you
>>>> suggest a coefficient that measures the agreement of the responses among
>>>> raters?
>>>>
>>>> Best,
>>>> Eins
>>>
>>>
>>>
>>>
>>>
>>> -----
>>> --
>>> Bruce Weaver
>>>
>
>> bweaver@
>
>>> http://sites.google.com/a/lakeheadu.ca/bweaver/
>>>
>>> "When all else fails, RTFM."
>>>
>>> NOTE: My Hotmail account is not monitored regularly.
>>> To send me an e-mail, please use the address shown above.
>>>
>>> --
>>> View this message in context:
>>> http://spssx-discussion.1045642.n5.nabble.com/Agreement-of-Raters-in-one-item-tp5716702p5716738.html
>>> Sent from the SPSSX Discussion mailing list archive at Nabble.com.
>>>
>>> =====================
>>> To manage your subscription to SPSSX-L, send a message to
>>>
>
>> LISTSERV@.UGA
>
>> (not to SPSSX-L), with no body text except the
>>> command. To leave the list, send the command
>>> SIGNOFF SPSSX-L
>>> For a list of commands to manage subscriptions, send the command
>>> INFO REFCARD
>>
>> =====================
>> To manage your subscription to SPSSX-L, send a message to
>
>> LISTSERV@.UGA
>
>> (not to SPSSX-L), with no body text except the
>> command. To leave the list, send the command
>> SIGNOFF SPSSX-L
>> For a list of commands to manage subscriptions, send the command
>> INFO REFCARD
>
>
>
>
>
> -----
> --
> Bruce Weaver
> [hidden email]
> http://sites.google.com/a/lakeheadu.ca/bweaver/
>
> "When all else fails, RTFM."
>
> NOTE: My Hotmail account is not monitored regularly.
> To send me an e-mail, please use the address shown above.
>
> --
> View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Agreement-of-Raters-in-one-item-tp5716702p5716763.html
> Sent from the SPSSX Discussion mailing list archive at Nabble.com.
>
> =====================
> To manage your subscription to SPSSX-L, send a message to
> [hidden email] (not to SPSSX-L), with no body text except the
> command. To leave the list, send the command
> SIGNOFF SPSSX-L
> For a list of commands to manage subscriptions, send the command
> INFO REFCARD

Bruce Weaver

Re: Agreement of Raters in one item

Administrator

Ah, okay! Thanks for clarifying, Ryan.

R B wrote

No. It was tangential. Just pointing out that, despite SPSS refusing
to include zero-frequency cells when calculating p-values, it is in
fact possible to derive a valid p-value.

Ryan

On Fri, Dec 7, 2012 at 9:13 AM, Bruce Weaver <[hidden email]> wrote:
> I don't disagree with what you're saying Ryan, but I don't understand why
> you're saying it. Are you suggesting using the p-value from a chi-square
> goodness of fit test as the index of agreement the OP was looking for?
> Thanks for clarifying.
>
> Bruce
>
>
>
> R B wrote
>> It is possible to obtain a p-value under a scenario where cell(s) include
>> zero frequencies. Furthermore, if the expected values are less than 5, it
>> is possible to obtain exact p-values under scenarios with and without zero
>> frequency cells.
>>
>> Ryan
>>
>> On Dec 6, 2012, at 4:47 PM, Bruce Weaver <
>
>> bruce.weaver@
>
>> > wrote:
>>
>>> I was thinking of a chi-square goodness of fit too. Complete agreement
>>> would
>>> have all of the observations in one category, whereas the opposite
>>> extreme
>>> would have equal numbers of observations in each of the categories (and
>>> chi-square = 0). For a given sample size and number of categories, you
>>> could work out the maximum possible value of chi-square (with all
>>> observations in a single category), and possibly use that as the
>>> denominator
>>> of a ratio -- observed chi-square / max possible chi-square. But I tried
>>> a
>>> little demo (see below), and it looks like the scaling is not right --
>>> i.e.,
>>> the increase in that ratio is not a nice linear function of "agreement".
>>>
>>> When that didn't work out so well, I tried Rich's idea of using the
>>> standard
>>> deviation. That worked quite a bit better. Here is the syntax for my
>>> little demo.
>>>
>>>
>>> data list list / O1 to O3 (3f2.0).
>>> begin data
>>> 26 0 0
>>> 25 1 0
>>> 24 1 1
>>> 22 2 2
>>> 20 3 3
>>> 18 4 4
>>> 16 5 5
>>> 14 6 6
>>> 12 7 7
>>> 10 8 8
>>> 9 9 8
>>> end data.
>>>
>>> compute N = sum(O1 to O3).
>>> compute E = N / nvalid(O1 to O3).
>>> vector O = O1 to O3.
>>> compute chisqr = 0.
>>> loop # = 1 to 3.
>>> . compute chisqr = chisqr + (O(#)-E)**2/E.
>>> end loop.
>>> compute StD = SD(O1 to O3).
>>> execute.
>>>
>>> AGGREGATE
>>> /OUTFILE=* MODE=ADDVARIABLES
>>> /BREAK=
>>> /MaxChisqr=MAX(chisqr)
>>> /MaxSD = MAX(StD)
>>> .
>>>
>>> compute CSratio = chisqr / MaxChisqr.
>>> compute SDratio = StD/MaxSD.
>>> formats chisqr CSratio SDratio(f8.3) / N (F5.0).
>>> list.
>>>
>>> GRAPH /SCATTERPLOT(BIVAR)=O1 WITH chisqr .
>>> GRAPH /SCATTERPLOT(BIVAR)=O1 WITH CSratio .
>>>
>>> GRAPH /SCATTERPLOT(BIVAR)=O1 WITH StD .
>>> GRAPH /SCATTERPLOT(BIVAR)=O1 WITH SDratio .
>>>
>>>
>>> For those who don't have access to SPSS, here is the output from the LIST
>>> command -- you might wish to generate your own scatter-plots using some
>>> other software. The bottom line is that the last two scatter-plots show
>>> nice linear relationships, whereas the first two show curvilinear
>>> relationships that rise slowly for lower values of O1, and more quickly
>>> for
>>> higher values of O1.
>>>
>>> OUTPUT from LIST:
>>>
>>> O1 O2 O3 N E chisqr StD MaxChisqr MaxSD CSratio
>>> SDratio
>>>
>>> 26 0 0 26 8.67 52.000 15.01 52.00 15.01 1.000
>>> 1.000
>>> 25 1 0 26 8.67 46.231 14.15 52.00 15.01 .889
>>> .943
>>> 24 1 1 26 8.67 40.692 13.28 52.00 15.01 .783
>>> .885
>>> 22 2 2 26 8.67 30.769 11.55 52.00 15.01 .592
>>> .769
>>> 20 3 3 26 8.67 22.231 9.81 52.00 15.01 .428
>>> .654
>>> 18 4 4 26 8.67 15.077 8.08 52.00 15.01 .290
>>> .538
>>> 16 5 5 26 8.67 9.308 6.35 52.00 15.01 .179
>>> .423
>>> 14 6 6 26 8.67 4.923 4.62 52.00 15.01 .095
>>> .308
>>> 12 7 7 26 8.67 1.923 2.89 52.00 15.01 .037
>>> .192
>>> 10 8 8 26 8.67 .308 1.15 52.00 15.01 .006
>>> .077
>>> 9 9 8 26 8.67 .077 .58 52.00 15.01 .001
>>> .038
>>>
>>> Number of cases read: 11 Number of cases listed: 11
>>>
>>> HTH.
>>>
>>>
>>>
>>> ViAnn Beadle wrote
>>>> How about a 1-sample chi-square?
>>>>
>>>>
>>>>
>>>> From: SPSSX(r) Discussion [mailto:
>>>
>>>> SPSSX-L@.UGA
>>>
>>>> ] On Behalf Of
>>>> Rich Ulrich
>>>> Sent: Thursday, December 06, 2012 12:10 PM
>>>> To:
>>>
>>>> SPSSX-L@.UGA
>>>
>>>> Subject: Re: Agreement of Raters in one item
>>>>
>>>>
>>>>
>>>> When you have one score from each of 26 raters/subjects,
>>>> then the "agreement" is like the "agreement" of (say) age or sex --
>>>>
>>>> For a categorical variable, you can report that "umpteen percent were X"
>>>> or give the fractions for each of several categories. (For questions
>>>> like
>>>> "political party membership", you might consider an "index of diversity"
>>>> if you were going to compare several countries eventually.)
>>>>
>>>> For a measure that you consider continuous, the standard deviation
>>>> shows the "agreement" or similarity. A confidence interval is also
>>>> sometimes appropriate.
>>>>
>>>> --
>>>> Rich Ulrich
>>>>
>>>>
>>>>
>>>> _____
>>>>
>>>> Date: Thu, 6 Dec 2012 11:52:49 +0800
>>>> From:
>>>
>>>> einsbernardo@.com
>>>
>>>> Subject: Agreement of Raters in one item
>>>> To:
>>>
>>>> SPSSX-L@.UGA
>>>
>>>>
>>>> Dear All,
>>>>
>>>>
>>>> A question of somebody was sent to me last day. It's a basic question
>>>> but
>>>> I
>>>> dont know the answer, thus, I forward it to this list.
>>>>
>>>> Here is the question: An item (3-point response options such as not
>>>> appropriate, appropriate, not appropriate) is rated by 26 raters. Can
>>>> you
>>>> suggest a coefficient that measures the agreement of the responses among
>>>> raters?
>>>>
>>>> Best,
>>>> Eins
>>>
>>>
>>>
>>>
>>>
>>> -----
>>> --
>>> Bruce Weaver
>>>
>
>> bweaver@
>
>>> http://sites.google.com/a/lakeheadu.ca/bweaver/
>>>
>>> "When all else fails, RTFM."
>>>
>>> NOTE: My Hotmail account is not monitored regularly.
>>> To send me an e-mail, please use the address shown above.
>>>
>>> --
>>> View this message in context:
>>> http://spssx-discussion.1045642.n5.nabble.com/Agreement-of-Raters-in-one-item-tp5716702p5716738.html
>>> Sent from the SPSSX Discussion mailing list archive at Nabble.com.
>>>
>>> =====================
>>> To manage your subscription to SPSSX-L, send a message to
>>>
>
>> LISTSERV@.UGA
>
>> (not to SPSSX-L), with no body text except the
>>> command. To leave the list, send the command
>>> SIGNOFF SPSSX-L
>>> For a list of commands to manage subscriptions, send the command
>>> INFO REFCARD
>>
>> =====================
>> To manage your subscription to SPSSX-L, send a message to
>
>> LISTSERV@.UGA
>
>> (not to SPSSX-L), with no body text except the
>> command. To leave the list, send the command
>> SIGNOFF SPSSX-L
>> For a list of commands to manage subscriptions, send the command
>> INFO REFCARD
>
>
>
>
>
> -----
> --
> Bruce Weaver
> [hidden email]
> http://sites.google.com/a/lakeheadu.ca/bweaver/
>
> "When all else fails, RTFM."
>
> NOTE: My Hotmail account is not monitored regularly.
> To send me an e-mail, please use the address shown above.
>
> --
> View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Agreement-of-Raters-in-one-item-tp5716702p5716763.html
> Sent from the SPSSX Discussion mailing list archive at Nabble.com.
>
> =====================
> To manage your subscription to SPSSX-L, send a message to
> [hidden email] (not to SPSSX-L), with no body text except the
> command. To leave the list, send the command
> SIGNOFF SPSSX-L
> For a list of commands to manage subscriptions, send the command
> INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD