SPSS syntax for r*WG(J) and r'WG(J)

[Max Freund]

Hello all,

Does anyone happen to have SPSS syntax for calculating inter-rater  
agreement using r*WG(J) or r'WG(J)?

I'm not totally clear if there's a significance test for these stats  
(some form of chi-square?) or if it's just a judgment call, but if  
there is a significance testing syntax for them (and for rWG(J)), I'd  
love to get that too!

Here are references:

r*WG:
Lindell, M. K., Brandt, C. J., & Whitney, D. J. (1999). A Revised  
Index of Interrater Agreement for Multi-Item Ratings of a Single  
Target. Applied Psychological Measurement, 23(2), 127-135.

r'WG:
Lindell, M. K. (2001). Assessing and Testing Interrater Agreement on a  
Single Target Using Multi-Item Rating Scales. Applied Psychological  
Measurement, 25(1), 89-99.

rWG:
James, L. R., Demaree, R. G., & Wolf, G. (1984). Estimating within-
group interrater reliability with and without response bias. Journal  
of Applied Psychology, 69(1), 85-98.
James, L. R., Demaree, R. G., & Wolf, G. (1993). r-sub(wg): An  
assessment of within-group interrater agreement. Journal of Applied  
Psychology, 78(2), 306-309.

Thanks,
Max Freund


--
Max Freund, M.I.I.M. • [hidden email] • [phone]
Doctoral Student in Organizational Behavior, Claremont Graduate  
University (www.cgu.edu/sbos)
Partner, LF Leadership (www.lfleadership.com)

====================To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Garry Gelade]

Max,

You can find SPSS syntax for rwG(J) here, which may help (but I don't know
where you can find syntax for the r* and r' variants.)

LeBreton, J. M. & Senter,J. L. (2008). Answers to 20 Questions About
Interrater Reliability and Interrater Agreement. Organizational Research
Methods,11, p 815-852.

Regards
Garry Gelade
Business Analytic Ltd.


-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Max
Freund
Sent: 23 November 2008 11:59
To: [hidden email]
Subject: SPSS syntax for r*WG(J) and r'WG(J)

Hello all,

Does anyone happen to have SPSS syntax for calculating inter-rater agreement
using r*WG(J) or r'WG(J)?

I'm not totally clear if there's a significance test for these stats (some
form of chi-square?) or if it's just a judgment call, but if there is a
significance testing syntax for them (and for rWG(J)), I'd love to get that
too!

Here are references:

r*WG:
Lindell, M. K., Brandt, C. J., & Whitney, D. J. (1999). A Revised Index of
Interrater Agreement for Multi-Item Ratings of a Single Target. Applied
Psychological Measurement, 23(2), 127-135.

r'WG:
Lindell, M. K. (2001). Assessing and Testing Interrater Agreement on a
Single Target Using Multi-Item Rating Scales. Applied Psychological
Measurement, 25(1), 89-99.

rWG:
James, L. R., Demaree, R. G., & Wolf, G. (1984). Estimating within- group
interrater reliability with and without response bias. Journal of Applied
Psychology, 69(1), 85-98.
James, L. R., Demaree, R. G., & Wolf, G. (1993). r-sub(wg): An assessment of
within-group interrater agreement. Journal of Applied Psychology, 78(2),
306-309.

Thanks,
Max Freund


--
Max Freund, M.I.I.M. . [hidden email] . [phone] Doctoral Student
in Organizational Behavior, Claremont Graduate University (www.cgu.edu/sbos)
Partner, LF Leadership (www.lfleadership.com)

=======
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command SIGNOFF SPSSX-L For a list of
commands to manage subscriptions, send the command INFO REFCARD

__________ NOD32 3632 (20081121) Information __________

This message was checked by NOD32 antivirus system.
http://www.eset.com

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Max Freund]

Thanks Gary.

Actually, the LeBreton & Senter article is where I got the syntax I've  
been using for rWG(J), but I don't think it includes a significance  
test.  They basically just give some thumbnail cutoffs.

My problem is that a number of the results have come up as negative  
numbers (indicating bimodal or other odd distributions of ratings).  
I've been counting those as 0, but I wanted to try the r* or r'  
variants to see if they deal better with my data.

(BTW, if anyone else is looking to do multi-level analysis, the  
LeBreton & Senter article is a great overview of inter-rater  
reliability and agreement stats.)

Anyone else have any suggestions?

Best,
Max

--
Max Freund, M.I.I.M. • [hidden email] • [phone]
Doctoral Student in Organizational Behavior, Claremont Graduate  
University (www.cgu.edu/sbos)
Partner, LF Leadership (www.lfleadership.com)

On Nov 23, 2008, at 6:13 AM, Garry Gelade wrote:

> Max,
>
> You can find SPSS syntax for rwG(J) here, which may help (but I  
> don't know
> where you can find syntax for the r* and r' variants.)
>
> LeBreton, J. M. & Senter,J. L. (2008). Answers to 20 Questions About
> Interrater Reliability and Interrater Agreement. Organizational  
> Research
> Methods,11, p 815-852.
>
> Regards
> Garry Gelade
> Business Analytic Ltd.
>
>
> -----Original Message-----
> From: SPSSX(r) Discussion [mailto:[hidden email]] On  
> Behalf Of Max
> Freund
> Sent: 23 November 2008 11:59
> To: [hidden email]
> Subject: SPSS syntax for r*WG(J) and r'WG(J)
>
> Hello all,
>
> Does anyone happen to have SPSS syntax for calculating inter-rater  
> agreement
> using r*WG(J) or r'WG(J)?
>
> I'm not totally clear if there's a significance test for these stats  
> (some
> form of chi-square?) or if it's just a judgment call, but if there  
> is a
> significance testing syntax for them (and for rWG(J)), I'd love to  
> get that
> too!
>
> Here are references:
>
> r*WG:
> Lindell, M. K., Brandt, C. J., & Whitney, D. J. (1999). A Revised  
> Index of
> Interrater Agreement for Multi-Item Ratings of a Single Target.  
> Applied
> Psychological Measurement, 23(2), 127-135.
>
> r'WG:
> Lindell, M. K. (2001). Assessing and Testing Interrater Agreement on a
> Single Target Using Multi-Item Rating Scales. Applied Psychological
> Measurement, 25(1), 89-99.
>
> rWG:
> James, L. R., Demaree, R. G., & Wolf, G. (1984). Estimating within-  
> group
> interrater reliability with and without response bias. Journal of  
> Applied
> Psychology, 69(1), 85-98.
> James, L. R., Demaree, R. G., & Wolf, G. (1993). r-sub(wg): An  
> assessment of
> within-group interrater agreement. Journal of Applied Psychology,  
> 78(2),
> 306-309.
>
> Thanks,
> Max Freund
>
>
> --
> Max Freund, M.I.I.M. . [hidden email] . [phone] Doctoral  
> Student
> in Organizational Behavior, Claremont Graduate University (www.cgu.edu/sbos)
> Partner, LF Leadership (www.lfleadership.com)
>
> =======
> To manage your subscription to SPSSX-L, send a message to
> [hidden email] (not to SPSSX-L), with no body text except  
> the
> command. To leave the list, send the command SIGNOFF SPSSX-L For a  
> list of
> commands to manage subscriptions, send the command INFO REFCARD
>
> __________ NOD32 3632 (20081121) Information __________
>
> This message was checked by NOD32 antivirus system.
> http://www.eset.com
>
> =====================
> To manage your subscription to SPSSX-L, send a message to
> [hidden email] (not to SPSSX-L), with no body text except  
> the
> command. To leave the list, send the command
> SIGNOFF SPSSX-L
> For a list of commands to manage subscriptions, send the command
> INFO REFCARD

====================To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Jims More]

My apology for cross posting.May I know if there is a listserv for MS Access?Thanks.Jims


      Start your day with Yahoo!7 and win a Sony Bravia TV. Enter now http://au.docs.yahoo.com/homepageset/?p1=other&p2=au&p3=tagline

====================To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Brian Cooper]

In a study I am involved with, (social disadvantage), a problem has arisen
dealing with the issue of causal and coincidental correlations. Most of the
variables do correlate, but the question thus arises how many of these
correlations casual and how many are coincidental? The only obvious
relationship between these variables is spatial and depending on scale
change. What statistical approach should be adopted here to attend to this
problem?

Yours truly,
Brian Cooper

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[David Hitchin]

Quoting Brian Cooper <[hidden email]>:

> In a study I am involved with, (social disadvantage), a problem has
> arisen dealing with the issue of causal and coincidental
> correlations. ... What statistical approach should be adopted here
> to attend to this problem?

I can think of two approaches.

1. Work with large samples and strict significance tests. If you test at
the 0.001 level then only 1 in 1000 tests on average will wrongly show
significance when there is no difference between the populations.

2.Replicate the experiment. The causal correlations will tend to be
found in the second experiment, while the coincidental ones will tend to
pop up between different pairs of variables.

This assumes that the two replications of the experiment are
independent. As an example, over many years there was a correlation
between teachers' salaries and drunkeness convictions. This was
(probably) not cause and effect between the two variables, but the
consequence of better incomes for most people.

Therefore you need to consider and eliminate background variables which
could result in false assumptions of cause and effect between two
variables when in fact they are driven by a third variable.

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Garry Gelade]

Dear Max

One way to get a significance test would be bootstrapping. I'm not sure
where I came across the idea, but it's straightforward in principle.  On
each of say 500 bootstrap replications, you randomize the group membership
eg by randomly reordering the group membership variable, but keeping the
original order of the rest of the data. You calculate rwg on each
replication, and store the rwg value. This gives you a distribution for rwg
against which you can test the rwg value calculated with the proper group
membership. If it is within the top or bottom 2.5% of the bootstrap
distribution, you have significance.

This method was proposed in a paper, but I can't remember where I saw it.
It would certainly be possible to implement it with SPSS syntax.

Regards
Garry

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Max
Freund
Sent: 24 November 2008 00:29
To: [hidden email]
Subject: Re: SPSS syntax for r*WG(J) and r'WG(J)

Thanks Gary.

Actually, the LeBreton & Senter article is where I got the syntax I've been
using for rWG(J), but I don't think it includes a significance test.  They
basically just give some thumbnail cutoffs.

My problem is that a number of the results have come up as negative
numbers (indicating bimodal or other odd distributions of ratings).
I've been counting those as 0, but I wanted to try the r* or r'
variants to see if they deal better with my data.

(BTW, if anyone else is looking to do multi-level analysis, the LeBreton &
Senter article is a great overview of inter-rater reliability and agreement
stats.)

Anyone else have any suggestions?

Best,
Max

--
Max Freund, M.I.I.M. . [hidden email] . [phone] Doctoral Student
in Organizational Behavior, Claremont Graduate University (www.cgu.edu/sbos)
Partner, LF Leadership (www.lfleadership.com)

On Nov 23, 2008, at 6:13 AM, Garry Gelade wrote:

> Max,
>
> You can find SPSS syntax for rwG(J) here, which may help (but I don't
> know where you can find syntax for the r* and r' variants.)
>
> LeBreton, J. M. & Senter,J. L. (2008). Answers to 20 Questions About
> Interrater Reliability and Interrater Agreement. Organizational
> Research Methods,11, p 815-852.
>
> Regards
> Garry Gelade
> Business Analytic Ltd.
>
>
> -----Original Message-----
> From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf
> Of Max Freund
> Sent: 23 November 2008 11:59
> To: [hidden email]
> Subject: SPSS syntax for r*WG(J) and r'WG(J)
>
> Hello all,
>
> Does anyone happen to have SPSS syntax for calculating inter-rater
> agreement using r*WG(J) or r'WG(J)?
>
> I'm not totally clear if there's a significance test for these stats
> (some form of chi-square?) or if it's just a judgment call, but if
> there is a significance testing syntax for them (and for rWG(J)), I'd
> love to get that too!
>
> Here are references:
>
> r*WG:
> Lindell, M. K., Brandt, C. J., & Whitney, D. J. (1999). A Revised
> Index of Interrater Agreement for Multi-Item Ratings of a Single
> Target.
> Applied
> Psychological Measurement, 23(2), 127-135.
>
> r'WG:
> Lindell, M. K. (2001). Assessing and Testing Interrater Agreement on a
> Single Target Using Multi-Item Rating Scales. Applied Psychological
> Measurement, 25(1), 89-99.
>
> rWG:
> James, L. R., Demaree, R. G., & Wolf, G. (1984). Estimating within-
> group interrater reliability with and without response bias. Journal
> of Applied Psychology, 69(1), 85-98.
> James, L. R., Demaree, R. G., & Wolf, G. (1993). r-sub(wg): An
> assessment of within-group interrater agreement. Journal of Applied
> Psychology, 78(2), 306-309.
>
> Thanks,
> Max Freund
>
>
> --
> Max Freund, M.I.I.M. . [hidden email] . [phone] Doctoral
> Student in Organizational Behavior, Claremont Graduate University
> (www.cgu.edu/sbos) Partner, LF Leadership (www.lfleadership.com)
>
> =======
> To manage your subscription to SPSSX-L, send a message to
> [hidden email] (not to SPSSX-L), with no body text except
> the command. To leave the list, send the command SIGNOFF SPSSX-L For a
> list of commands to manage subscriptions, send the command INFO
> REFCARD
>
> __________ NOD32 3632 (20081121) Information __________
>
> This message was checked by NOD32 antivirus system.
> http://www.eset.com
>
> =====================
> To manage your subscription to SPSSX-L, send a message to
> [hidden email] (not to SPSSX-L), with no body text except
> the command. To leave the list, send the command SIGNOFF SPSSX-L For a
> list of commands to manage subscriptions, send the command INFO
> REFCARD

=======
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command SIGNOFF SPSSX-L For a list of
commands to manage subscriptions, send the command INFO REFCARD

__________ NOD32 3632 (20081121) Information __________

This message was checked by NOD32 antivirus system.
http://www.eset.com

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Nancy Darling-2]

If by 'spatial' you mean physical location (i.e. census tracks with high
percent of female headed households also tend to be poor, tend to have a
high percentage of minority families, tend to have low per capita
income), I assume you are already using multi-level models to nest your
data?  For example, you look within schools, zip codes, census tracks,
or whatever your sampling unit is?  If so, looking within units and
individual differences will help you to untease some of those
relationships.   Then there's a whole literature on causal modeling to
draw upon.

Brian Cooper wrote:

> In a study I am involved with, (social disadvantage), a problem has arisen
> dealing with the issue of causal and coincidental correlations. Most of the
> variables do correlate, but the question thus arises how many of these
> correlations casual and how many are coincidental? The only obvious
> relationship between these variables is spatial and depending on scale
> change. What statistical approach should be adopted here to attend to this
> problem?
>
> Yours truly,
> Brian Cooper
>
> =====================
> To manage your subscription to SPSSX-L, send a message to
> [hidden email] (not to SPSSX-L), with no body text except the
> command. To leave the list, send the command
> SIGNOFF SPSSX-L
> For a list of commands to manage subscriptions, send the command
> INFO REFCARD
>

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Bob Schacht-3]

At 11:59 PM 11/23/2008, Brian Cooper wrote:

>Bob,
>
>I am involved with developing a more effective measure of social
>disadvantage. Current approaches assume there is a direct relationship
>between the variables but no researcher in Australia has actually
>established that there is a relationship between the variables especially
>if it is a spatial approach.  My concern is to ensure that any correlation
>that may arise is not a cum hoc ergo propter hoc logical fallacy artifact,
>but is relevant to the analysis. The other factor is the effect of
>location on the variables, but this is an issue of geo-stats.
>
>Brian Cooper

Brian,
You don't address the issues that I raised in my off-list response below,
and the cum hoc ergo propter hoc logical fallacy doesn't address those
issues, either, except to warn against this problem without providing a
solution.

Statisticians worth their salt have long known that correlation is not the
same as causation, and you are aware of that issue. However, it sounds like
you think that you can get beyond it using statistical calculations alone.

Blalock et al. (see previous e-mail quoted below) have addressed the
question that you are asking, although maybe not in those exact words.
Besides Blalock et al., see also James A. Davis, The Logic of Causal Order
(Sage Publications, 1985). It is a myth to suppose that causal problems can
be solved by statistical calculations alone, as John L. Sullivan wrote in
the Introduction to Davis's book.

Bob Schacht

>
>
>----------
>From: Bob Schacht [mailto:[hidden email]]
>Sent: Monday, 24 November 2008 7:21 PM
>To: Brian Cooper
>Subject: Re: Causal and Coincidental Correlations
>
>At 09:31 PM 11/23/2008, Brian Cooper wrote:
>
>In a study I am involved with, (social disadvantage), a problem has arisen
>dealing with the issue of causal and coincidental correlations. Most of the
>variables do correlate, but the question thus arises how many of these
>correlations casual and how many are coincidental? The only obvious
>relationship between these variables is spatial and depending on scale
>change. What statistical approach should be adopted here to attend to this
>problem?
>
>Yours truly,
>Brian Cooper
>
>The problem you are asking is not fundamentally a statistical one. "Cause"
>implies a theoretical framework to explain why one variable is "caused" by
>another. You can frame causal explanations as hypotheses and test them
>(the hypothetico-deductive approach).
>
>But your language suggests other problems, as well. In one sentence you
>use "causal," and in another you use "casual." Which do you mean? The two
>are very different. Causal suggests a theoretical connection, whereas
>casual suggests mere random association.  In fact, I'm not sure how you
>might tell the difference between a "casual" (sic.)  relationship and a
>"coincidental" one.
>
>I recommend that you read
>
>
>Causal Models in the Social Sciences (Paperback),
>
>
>
>by
><http://www.amazon.com/exec/obidos/search-handle-url/ref=ntt_athr_dp_sr_1?%5Fencoding=UTF8&search-type=ss&index=books&field-author=H.%20Blalock>H.
>Blalock (Editor) as a classic introduction to the concepts you are
>referring to.
>
>If I have misunderstood what you are asking, please clarify your terms and
>the scholarship on which they are based.
>
>Good luck,
>Bob Schacht
>
>Robert M. Schacht, Ph.D., Research Director
>Pacific Basin Research and Training Center
>1268 Young Street, Suite #204
>Research Center, University of Hawaii
>Honolulu, HI 96814
>E-mail <[hidden email]>
>Phone [phone], FAX [phone]

Robert M. Schacht, Ph.D. <[hidden email]>
Pacific Basin Rehabilitation Research & Training Center
1268 Young Street, Suite #204
Research Center, University of Hawaii
Honolulu, HI 96814

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[David Hitchin]

Quoting Brian Cooper <[hidden email]>:

> The only obvious relationship between these variables is spatial
> and depending on scale change. What statistical approach should
> be adopted here to attend to this problem?

I'm sorry that I didn't pick up on the spatial issue in my earlier
response. There was a lot of work on this several decades ago, under the
heading of "spatial autocorrelation" and the article "spatial analysis"
in Wikipedia provides some useful references. I suspect that the
statistics will diagnose your problem, in the sense of estimating the
amount of spatial autocorrelation, but the only solution will be to get
enough sufficiently independent samples.

David Hitchin

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD