Interrater reliability (Dates & King, 2008)

[Albert-Jan Roskam]

Hi Martha and others
I am trying to use the macro for the calculation of various measures for interrater reliability (Dates & King, 2008), as was recently posted by Marta Garcia-Granero [By the way: thank you for that!]


I modified the file and path names and of course also the macro calls.
I use a separate sps file in which I define two file handles, one for the working dir (mydir), and one for the source sav file (myfile). In the macro, I replaced all occurrences of file=' with file='mydir/ and outfile=' with outfile='mydir/ . For testing purposes, I use a file with 100 cases and just 100 bootstrap replications.

I receive a hail of error messages, and sometimes Spss (errrr, PASW) crashes altogether. The first error I get is a zerodivision error. Any idea what's happening? Is it (not) possible to use file handles in save/get within MATRIX - END MATRIX?

Thanks in advance!

Albert-Jan

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Marta Garcia-Granero]

Albert-Jan Roskam wrote:
> Hi Martha and others
> I am trying to use the macro for the calculation of various measures for interrater reliability (Dates & King, 2008), as was recently posted by Marta Garcia-Granero [By the way: thank you for that!]
>
Always welcome :-)
>
> I modified the file and path names and of course also the macro calls.
> I use a separate sps file in which I define two file handles, one for the working dir (mydir), and one for the source sav file (myfile). In the macro, I replaced all occurrences of file=' with file='mydir/ and outfile=' with outfile='mydir/ . For testing purposes, I use a file with 100 cases and just 100 bootstrap replications.
>
> I receive a hail of error messages, and sometimes Spss (errrr, PASW) crashes altogether. The first error I get is a zerodivision error. Any idea what's happening? Is it (not) possible to use file handles in save/get within MATRIX - END MATRIX?
>

I'd like to see the full load of errors. Can you send me the macro (off
line)? I have my own datasets for testing purposes, no need to send the
sav file. I can test your macro both in SPSS 15 & PASW 17.0.2.

Marta

--
For miscellaneous SPSS related statistical stuff, visit:
http://gjyp.nl/marta/
Updated today!

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[bdates]

I'd like to see the errors as well.  I'm in the process of streamlining the syntax, so if you send me the errors and perhaps the altered syntax you used, I can help diagnose the problem and get you on your way.

Brian

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Marta García-Granero
Sent: Monday, May 04, 2009 5:36 AM
To: [hidden email]
Subject: Re: Interrater reliability (Dates & King, 2008)

Albert-Jan Roskam wrote:
> Hi Martha and others
> I am trying to use the macro for the calculation of various measures for interrater reliability (Dates & King, 2008), as was recently posted by Marta Garcia-Granero [By the way: thank you for that!]
>
Always welcome :-)
>
> I modified the file and path names and of course also the macro calls.
> I use a separate sps file in which I define two file handles, one for the working dir (mydir), and one for the source sav file (myfile). In the macro, I replaced all occurrences of file=' with file='mydir/ and outfile=' with outfile='mydir/ . For testing purposes, I use a file with 100 cases and just 100 bootstrap replications.
>
> I receive a hail of error messages, and sometimes Spss (errrr, PASW) crashes altogether. The first error I get is a zerodivision error. Any idea what's happening? Is it (not) possible to use file handles in save/get within MATRIX - END MATRIX?
>

I'd like to see the full load of errors. Can you send me the macro (off
line)? I have my own datasets for testing purposes, no need to send the
sav file. I can test your macro both in SPSS 15 & PASW 17.0.2.

Marta

--
For miscellaneous SPSS related statistical stuff, visit:
http://gjyp.nl/marta/
Updated today!

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Marta Garcia-Granero]

Brian, I had been modifying one of your inter rater macros too, to make
use of DATASET DEFINE instead of files saved to disk. I am also
simplifying some redundant loops (use only one loop to compute several
variables, instead of several loops to compute them one by one) to make
your code faster.When I'm done with it, I will send it to you. I plan to
use it in my advanced stats course (with ll credit being given to you,
of course, I'd never claim your work as mine).

Now, Albert-Jan:

Albert-Jan Roskam wrote:
> I am trying to use the macro for the calculation of various measures for interrater reliability (Dates & King, 2008), as was recently posted by Marta Garcia-Granero [By the way: thank you for that!]
>
>
> I modified the file and path names and of course also the macro calls.
> I use a separate sps file in which I define two file handles, one for the working dir (mydir), and one for the source sav file (myfile). In the macro, I replaced all occurrences of file=' with file='mydir/ and outfile=' with outfile='mydir/ . For testing purposes, I use a file with 100 cases and just 100 bootstrap replications.
>
> I receive a hail of error messages, and sometimes Spss (errrr, PASW) crashes altogether. The first error I get is a zerodivision error. Any idea what's happening? Is it (not) possible to use file handles in save/get within MATRIX - END MATRIX?
>
>
1) In "call_macros.sps" file, I used backslashes when defining the file
handles (I suppose it is not important, but since I copy-pasted the path
from My PC, mine were all backslashes, not forward-slashes as in your
2nd file handle).

My file handles:

file handle mydir /name = 'C:\Documents and
Settings\MartaGG\Escritorio\Nueva carpeta'.
file handle myfile /name ='C:\Documents and
Settings\MartaGG\Escritorio\TEXTOS\Avanzado CUN\KappaPaper.sav'.

Your file handles:

file handle mydir /name = '\\srsv1f\ssb3\GSB_Docum\Temp\arsm\temp'.
file handle myfile /name ='mydir/28042009
Herhaald_coderen_analyse_bestand_test.sav'. (<--- forward slash)

Perhaps the problem has to do with your network drive?

2) After modifying the file "bootstrap4.sps" (to make it run with my
variables, test1&test2, instead of yours), I had it "almost" working (OK
for 4 out of 5 macros). One of the macros has been overmodified, I'm
afraid, and something got lost. I get the following error message right
after the title "Step 2" is printed (steps 1, 3, 4 & 5 are OK).

I haven't been able to spot it.

Run MATRIX procedure:
 >Error encountered in source line #  7622

 >Error # 12354
 >Subscript is out of range.
 >This command not executed.

 >Error encountered in source line #  7623

 >Error # 12492
 >An attempt has been made to use previously undefined matrix (or scalar).
 >This command not executed.

Matrix - 'CKBCAUL' is undefined
 >Error encountered in source line #  7623

 >Error # 12363
 >Undefined operand in the expression inside brackets.

 >Error encountered in source line #  7624

 >Error # 12492
 >An attempt has been made to use previously undefined matrix (or scalar).
 >This command not executed.

Matrix - 'CKBCASUM' is undefined
 >Error encountered in source line #  7624

 >Error # 12459
 >The matrix to save is undefined.


Summary Table - Number of Subjects by Rater and Category
      Category
Rater       20       27       29       24
            24       28       24       24

Summary Table - Number of Agreements by Rater and Category
      Category
Rater       15       18       16       17
            15       18       16       17

------ END MATRIX -----

--
For miscellaneous SPSS related statistical stuff, visit:
http://gjyp.nl/marta/

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[zstatman]

I hope the results (macro set) can be shared with the community - Great Work
to All!

WMB
Statistical Services

============
mailto: [hidden email]
http:\\home.earthlink.net\~info.statman
============


On 5/4/2009 9:55:40 AM, Marta García-Granero ([hidden email])
wrote:

> Brian, I had been modifying one of your inter rater macros too, to make
> use of DATASET DEFINE instead of files saved to disk. I am also
> simplifying some redundant loops (use only one loop to compute several
> variables, instead of several loops to compute them one by one) to make
> your code faster.When
> I'm done with it, I will send it to you. I plan to
> use it in my advanced stats course (with ll credit being given to you,
> of course, I'd
> never claim your work as mine).
>
> Now, Albert-Jan:
>
> Albert-Jan Roskam wrote:
> > I am trying to use the macro for the calculation of various measures for
> interrater reliability (Dates & King, 2008), as was recently posted by
> Marta Garcia-Granero [By the way: thank you for that!]
> >
> >
> > I modified the file and path names and of course also the macro calls.
> > I use a separate sps file in which I define two file handles, one for
> the working dir (mydir), and one for the source sav file (myfile). In the
> macro, I replaced all occurrences of file=' with file='mydir/ and
> outfile=' with outfile='mydir/ . For testing purposes, I use a file with
> 100 cases and just 100 bootstrap replications.
> >

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[bdates]

Marta,

I'm looking forward to your modifications.  I have five other statistics - Bennett's S, Conger's kappa, Light's kappa, Gwet's expansion of Cohen's kappa, and Gwet's AC1 - all of which can benefit from whatever streamlining you make.  While we're at it, I want to make clear to the listserv that the jackknife portion of the syntax that creates the jackknifing process is about 90% yours from Raynald's website, and the bootstrap process part is abstracted from Andrew Hayes' work with Krippendorff's alpha.  On Jason's website, we have a document which steps people through the syntax.  I'm revising that along with each of the syntaxes.  The first section reviews the statistics.  The second section provides directions for using the syntax.  In that section your contribution and that of Andrew are cited with the full citation provided in the reference section.  So please, when you use the syntax, give yourself as much credit as you'd like.  My approach is that we're all working together to help out each other.  Thanks again.

Brian

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Marta García-Granero
Sent: Monday, May 04, 2009 9:56 AM
To: [hidden email]
Subject: Re: Interrater reliability (Dates & King, 2008)

Brian, I had been modifying one of your inter rater macros too, to make
use of DATASET DEFINE instead of files saved to disk. I am also
simplifying some redundant loops (use only one loop to compute several
variables, instead of several loops to compute them one by one) to make
your code faster.When I'm done with it, I will send it to you. I plan to
use it in my advanced stats course (with ll credit being given to you,
of course, I'd never claim your work as mine).

Now, Albert-Jan:

Albert-Jan Roskam wrote:
> I am trying to use the macro for the calculation of various measures for interrater reliability (Dates & King, 2008), as was recently posted by Marta Garcia-Granero [By the way: thank you for that!]
>
>
> I modified the file and path names and of course also the macro calls.
> I use a separate sps file in which I define two file handles, one for the working dir (mydir), and one for the source sav file (myfile). In the macro, I replaced all occurrences of file=' with file='mydir/ and outfile=' with outfile='mydir/ . For testing purposes, I use a file with 100 cases and just 100 bootstrap replications.
>
> I receive a hail of error messages, and sometimes Spss (errrr, PASW) crashes altogether. The first error I get is a zerodivision error. Any idea what's happening? Is it (not) possible to use file handles in save/get within MATRIX - END MATRIX?
>
>
1) In "call_macros.sps" file, I used backslashes when defining the file
handles (I suppose it is not important, but since I copy-pasted the path
from My PC, mine were all backslashes, not forward-slashes as in your
2nd file handle).

My file handles:

file handle mydir /name = 'C:\Documents and
Settings\MartaGG\Escritorio\Nueva carpeta'.
file handle myfile /name ='C:\Documents and
Settings\MartaGG\Escritorio\TEXTOS\Avanzado CUN\KappaPaper.sav'.

Your file handles:

file handle mydir /name = '\\srsv1f\ssb3\GSB_Docum\Temp\arsm\temp'.
file handle myfile /name ='mydir/28042009
Herhaald_coderen_analyse_bestand_test.sav'. (<--- forward slash)

Perhaps the problem has to do with your network drive?

2) After modifying the file "bootstrap4.sps" (to make it run with my
variables, test1&test2, instead of yours), I had it "almost" working (OK
for 4 out of 5 macros). One of the macros has been overmodified, I'm
afraid, and something got lost. I get the following error message right
after the title "Step 2" is printed (steps 1, 3, 4 & 5 are OK).

I haven't been able to spot it.

Run MATRIX procedure:
 >Error encountered in source line #  7622

 >Error # 12354
 >Subscript is out of range.
 >This command not executed.

 >Error encountered in source line #  7623

 >Error # 12492
 >An attempt has been made to use previously undefined matrix (or scalar).
 >This command not executed.

Matrix - 'CKBCAUL' is undefined
 >Error encountered in source line #  7623

 >Error # 12363
 >Undefined operand in the expression inside brackets.

 >Error encountered in source line #  7624

 >Error # 12492
 >An attempt has been made to use previously undefined matrix (or scalar).
 >This command not executed.

Matrix - 'CKBCASUM' is undefined
 >Error encountered in source line #  7624

 >Error # 12459
 >The matrix to save is undefined.


Summary Table - Number of Subjects by Rater and Category
      Category
Rater       20       27       29       24
            24       28       24       24

Summary Table - Number of Agreements by Rater and Category
      Category
Rater       15       18       16       17
            15       18       16       17

------ END MATRIX -----

--
For miscellaneous SPSS related statistical stuff, visit:
http://gjyp.nl/marta/

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Albert-Jan Roskam]

Thank you!
--I thought it didn't matter whether forward or backward slashes are used (except for the beginning of the UNC path, where baackslashes must be used, of course).
--Maybe using a UNC path per se causes problems? I can replace it with the drive letter. I generally don't like this, as it mmakes code less portable (as in: colleague-to-colleague portable).
--Perhaps it's also problematic that the file name contains a space. I didn't consider this before, but it causes problems soometimes. I'll try this tomorrow when I'm back in the office.
--No idea what went wrong with the last macro. I'll try again tomorrow.

Best wishes,
Albert-Jan

--- On Mon, 5/4/09, Marta García-Granero <[hidden email]> wrote:

> From: Marta García-Granero <[hidden email]>
> Subject: Re: Interrater reliability (Dates & King, 2008)
> To: [hidden email]
> Date: Monday, May 4, 2009, 3:55 PM
> Brian, I had been modifying one of
> your inter rater macros too, to make
> use of DATASET DEFINE instead of files saved to disk. I am
> also
> simplifying some redundant loops (use only one loop to
> compute several
> variables, instead of several loops to compute them one by
> one) to make
> your code faster.When I'm done with it, I will send it to
> you. I plan to
> use it in my advanced stats course (with ll credit being
> given to you,
> of course, I'd never claim your work as mine).
>
> Now, Albert-Jan:
>
> Albert-Jan Roskam wrote:
> > I am trying to use the macro for the calculation of
> various measures for interrater reliability (Dates &
> King, 2008), as was recently posted by Marta Garcia-Granero
> [By the way: thank you for that!]
> >
> >
> > I modified the file and path names and of course also
> the macro calls.
> > I use a separate sps file in which I define two file
> handles, one for the working dir (mydir), and one for the
> source sav file (myfile). In the macro, I replaced all
> occurrences of file=' with file='mydir/ and outfile=' with
> outfile='mydir/ . For testing purposes, I use a file with
> 100 cases and just 100 bootstrap replications.
> >
> > I receive a hail of error messages, and sometimes Spss
> (errrr, PASW) crashes altogether. The first error I get is a
> zerodivision error. Any idea what's happening? Is it (not)
> possible to use file handles in save/get within MATRIX - END
> MATRIX?
> >
> >
> 1) In "call_macros.sps" file, I used backslashes when
> defining the file
> handles (I suppose it is not important, but since I
> copy-pasted the path
> from My PC, mine were all backslashes, not forward-slashes
> as in your
> 2nd file handle).
>
> My file handles:
>
> file handle mydir /name = 'C:\Documents and
> Settings\MartaGG\Escritorio\Nueva carpeta'.
> file handle myfile /name ='C:\Documents and
> Settings\MartaGG\Escritorio\TEXTOS\Avanzado
> CUN\KappaPaper.sav'.
>
> Your file handles:
>
> file handle mydir /name =
> '\\srsv1f\ssb3\GSB_Docum\Temp\arsm\temp'.
> file handle myfile /name ='mydir/28042009
> Herhaald_coderen_analyse_bestand_test.sav'. (<---
> forward slash)
>
> Perhaps the problem has to do with your network drive?
>
> 2) After modifying the file "bootstrap4.sps" (to make it
> run with my
> variables, test1&test2, instead of yours), I had it
> "almost" working (OK
> for 4 out of 5 macros). One of the macros has been
> overmodified, I'm
> afraid, and something got lost. I get the following error
> message right
> after the title "Step 2" is printed (steps 1, 3, 4 & 5
> are OK).
>
> I haven't been able to spot it.
>
> Run MATRIX procedure:
> >Error encountered in source line #  7622
>
> >Error # 12354
> >Subscript is out of range.
> >This command not executed.
>
> >Error encountered in source line #  7623
>
> >Error # 12492
> >An attempt has been made to use previously undefined
> matrix (or scalar).
> >This command not executed.
>
> Matrix - 'CKBCAUL' is undefined
> >Error encountered in source line #  7623
>
> >Error # 12363
> >Undefined operand in the expression inside brackets.
>
> >Error encountered in source line #  7624
>
> >Error # 12492
> >An attempt has been made to use previously undefined
> matrix (or scalar).
> >This command not executed.
>
> Matrix - 'CKBCASUM' is undefined
> >Error encountered in source line #  7624
>
> >Error # 12459
> >The matrix to save is undefined.
>
>
> Summary Table - Number of Subjects by Rater and Category
>      Category
> Rater       20
>    27       29
>      24
>            24
>      28
>    24       24
>
> Summary Table - Number of Agreements by Rater and Category
>      Category
> Rater       15
>    18       16
>      17
>            15
>      18
>    16       17
>
> ------ END MATRIX -----
>
> --
> For miscellaneous SPSS related statistical stuff, visit:
> http://gjyp.nl/marta/
>
> =====================
> To manage your subscription to SPSSX-L, send a message to
> [hidden email]
> (not to SPSSX-L), with no body text except the
> command. To leave the list, send the command
> SIGNOFF SPSSX-L
> For a list of commands to manage subscriptions, send the
> command
> INFO REFCARD
>

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD