Multi-rater kappa that allows marginal totals to be free
Locked
Multi-rater kappa that allows marginal totals to be free
 
Administrator
|
I just came across this write-up of a symposium presentation that may be of interest to some list members. (This means you, Brian Dates!)
http://www.eric.ed.gov/PDFS/ED490661.pdf The link that promises SPSS syntax to compute this thing no longer works. I've written to Dr. Randolph to ask for a copy of it. If he does, and if he gives his blessing, I'll post it here in a follow-up message. Meanwhile, there is an online calculator here: http://justusrandolph.net/kappa/ Cheers, Bruce
--
Bruce Weaver bweaver@lakeheadu.ca http://sites.google.com/a/lakeheadu.ca/bweaver/ "When all else fails, RTFM." PLEASE NOTE THE FOLLOWING: 1. My Hotmail account is not monitored regularly. To send me an e-mail, please use the address shown above. 2. The SPSSX Discussion forum on Nabble is no longer linked to the SPSSX-L listserv administered by UGA (https://listserv.uga.edu/). |
|
All,
Justus Randolph's syntax is a multi-rater version of Bennett's S. The approach presumes that all categories are equal in probability of use. This is different from Fleiss' kappa, which presumes an underlying categorical use structure in the data. I have written syntax for Justus' approach. If he does not respond, I'll post it tomorrow for anyone who's interested. B ________________________________________ From: SPSSX(r) Discussion [[hidden email]] on behalf of Bruce Weaver [[hidden email]] Sent: Monday, May 27, 2013 4:46 PM To: [hidden email] Subject: Multi-rater kappa that allows marginal totals to be free I just came across this write-up of a symposium presentation that may be of interest to some list members. (This means you, Brian Dates!) http://www.eric.ed.gov/PDFS/ED490661.pdf The link that promises SPSS syntax to compute this thing no longer works. I've written to Dr. Randolph to ask for a copy of it. If he does, and if he gives his blessing, I'll post it here in a follow-up message. Meanwhile, there is an online calculator here: http://justusrandolph.net/kappa/ Cheers, Bruce ----- -- Bruce Weaver [hidden email] http://sites.google.com/a/lakeheadu.ca/bweaver/ "When all else fails, RTFM." NOTE: My Hotmail account is not monitored regularly. To send me an e-mail, please use the address shown above. -- View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Multi-rater-kappa-that-allows-marginal-totals-to-be-free-tp5720429.html Sent from the SPSSX Discussion mailing list archive at Nabble.com. ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD |
Re: Multi-rater kappa that allows marginal totals to be free
|
Administrator
|
Randolph's syntax can be found here:
http://justusrandolph.net/kappa/mrakmacro.htm Cheers, Bruce
--
Bruce Weaver bweaver@lakeheadu.ca http://sites.google.com/a/lakeheadu.ca/bweaver/ "When all else fails, RTFM." PLEASE NOTE THE FOLLOWING: 1. My Hotmail account is not monitored regularly. To send me an e-mail, please use the address shown above. 2. The SPSSX Discussion forum on Nabble is no longer linked to the SPSSX-L listserv administered by UGA (https://listserv.uga.edu/). |
|
|
Bruce and all,
I'm getting messages for the links that they're no longer working. In addition, each time the macro is run, users have to find the line with the number of categories, etc. The macro I wrote has categories and raters as part of the 'define' statement. Additionally, the ASE, z, and p values are specific to Randolph. Users don't have to ignore them. I'm probably coming across as critical and perhaps arrogant, and I certainly don't want to be; but my macro for Randolph's kappa is more user friendly because folks don't need to comb through the syntax each time they want to use it, and the output fits his specs. Brian ________________________________________ From: SPSSX(r) Discussion [[hidden email]] on behalf of Bruce Weaver [[hidden email]] Sent: Tuesday, May 28, 2013 10:24 AM To: [hidden email] Subject: Re: Multi-rater kappa that allows marginal totals to be free Randolph's syntax can be found here: http://justusrandolph.net/kappa/mrakmacro.htm Cheers, Bruce bdates wrote > All, > > Justus Randolph's syntax is a multi-rater version of Bennett's S. The > approach presumes that all categories are equal in probability of use. > This is different from Fleiss' kappa, which presumes an underlying > categorical use structure in the data. I have written syntax for Justus' > approach. If he does not respond, I'll post it tomorrow for anyone who's > interested. > > B > ________________________________________ > From: SPSSX(r) Discussion [ > SPSSX-L@.UGA > ] on behalf of Bruce Weaver [ > bruce.weaver@ > ] > Sent: Monday, May 27, 2013 4:46 PM > To: > SPSSX-L@.UGA > Subject: Multi-rater kappa that allows marginal totals to be free > > I just came across this write-up of a symposium presentation that may be > of > interest to some list members. (This means you, Brian Dates!) > > http://www.eric.ed.gov/PDFS/ED490661.pdf > > The link that promises SPSS syntax to compute this thing no longer works. > I've written to Dr. Randolph to ask for a copy of it. If he does, and if > he > gives his blessing, I'll post it here in a follow-up message. Meanwhile, > there is an online calculator here: > > http://justusrandolph.net/kappa/ > > Cheers, > Bruce > > > > > ----- > -- > Bruce Weaver > bweaver@ > http://sites.google.com/a/lakeheadu.ca/bweaver/ > > "When all else fails, RTFM." > > NOTE: My Hotmail account is not monitored regularly. > To send me an e-mail, please use the address shown above. > > -- > View this message in context: > http://spssx-discussion.1045642.n5.nabble.com/Multi-rater-kappa-that-allows-marginal-totals-to-be-free-tp5720429.html > Sent from the SPSSX Discussion mailing list archive at Nabble.com. > > ===================== > To manage your subscription to SPSSX-L, send a message to > LISTSERV@.UGA > (not to SPSSX-L), with no body text except the > command. To leave the list, send the command > SIGNOFF SPSSX-L > For a list of commands to manage subscriptions, send the command > INFO REFCARD > > ===================== > To manage your subscription to SPSSX-L, send a message to > LISTSERV@.UGA > (not to SPSSX-L), with no body text except the > command. To leave the list, send the command > SIGNOFF SPSSX-L > For a list of commands to manage subscriptions, send the command > INFO REFCARD ----- -- Bruce Weaver [hidden email] http://sites.google.com/a/lakeheadu.ca/bweaver/ "When all else fails, RTFM." NOTE: My Hotmail account is not monitored regularly. To send me an e-mail, please use the address shown above. -- View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Multi-rater-kappa-that-allows-marginal-totals-to-be-free-tp5720429p5720435.html Sent from the SPSSX Discussion mailing list archive at Nabble.com. ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD |
Re: Multi-rater kappa that allows marginal totals to be free
|
Administrator
|
Sorry, I should have tried downloading the files before posting that -- the links do indeed appear to be dead.
Brian, can you post your macro?
--
Bruce Weaver bweaver@lakeheadu.ca http://sites.google.com/a/lakeheadu.ca/bweaver/ "When all else fails, RTFM." PLEASE NOTE THE FOLLOWING: 1. My Hotmail account is not monitored regularly. To send me an e-mail, please use the address shown above. 2. The SPSSX Discussion forum on Nabble is no longer linked to the SPSSX-L listserv administered by UGA (https://listserv.uga.edu/). |
|
|
Bruce and all,
I'm working from home today on a state grant. I'll post the macro tomorrow when I'm back in the office. B ________________________________________ From: SPSSX(r) Discussion [[hidden email]] on behalf of Bruce Weaver [[hidden email]] Sent: Tuesday, May 28, 2013 11:41 AM To: [hidden email] Subject: Re: Multi-rater kappa that allows marginal totals to be free Sorry, I should have tried downloading the files before posting that -- the links do indeed appear to be dead. Brian, can you post your macro? bdates wrote > Bruce and all, > > I'm getting messages for the links that they're no longer working. In > addition, each time the macro is run, users have to find the line with the > number of categories, etc. The macro I wrote has categories and raters as > part of the 'define' statement. Additionally, the ASE, z, and p values > are specific to Randolph. Users don't have to ignore them. I'm probably > coming across as critical and perhaps arrogant, and I certainly don't want > to be; but my macro for Randolph's kappa is more user friendly because > folks don't need to comb through the syntax each time they want to use it, > and the output fits his specs. > > Brian > ________________________________________ > From: SPSSX(r) Discussion [ > SPSSX-L@.UGA > ] on behalf of Bruce Weaver [ > bruce.weaver@ > ] > Sent: Tuesday, May 28, 2013 10:24 AM > To: > SPSSX-L@.UGA > Subject: Re: Multi-rater kappa that allows marginal totals to be free > > Randolph's syntax can be found here: > > http://justusrandolph.net/kappa/mrakmacro.htm > > Cheers, > Bruce > > > > bdates wrote >> All, >> >> Justus Randolph's syntax is a multi-rater version of Bennett's S. The >> approach presumes that all categories are equal in probability of use. >> This is different from Fleiss' kappa, which presumes an underlying >> categorical use structure in the data. I have written syntax for Justus' >> approach. If he does not respond, I'll post it tomorrow for anyone who's >> interested. >> >> B >> ________________________________________ >> From: SPSSX(r) Discussion [ > >> SPSSX-L@.UGA > >> ] on behalf of Bruce Weaver [ > >> bruce.weaver@ > >> ] >> Sent: Monday, May 27, 2013 4:46 PM >> To: > >> SPSSX-L@.UGA > >> Subject: Multi-rater kappa that allows marginal totals to be free >> >> I just came across this write-up of a symposium presentation that may be >> of >> interest to some list members. (This means you, Brian Dates!) >> >> http://www.eric.ed.gov/PDFS/ED490661.pdf >> >> The link that promises SPSS syntax to compute this thing no longer works. >> I've written to Dr. Randolph to ask for a copy of it. If he does, and if >> he >> gives his blessing, I'll post it here in a follow-up message. Meanwhile, >> there is an online calculator here: >> >> http://justusrandolph.net/kappa/ >> >> Cheers, >> Bruce >> >> >> >> >> ----- >> -- >> Bruce Weaver > >> bweaver@ > >> http://sites.google.com/a/lakeheadu.ca/bweaver/ >> >> "When all else fails, RTFM." >> >> NOTE: My Hotmail account is not monitored regularly. >> To send me an e-mail, please use the address shown above. >> >> -- >> View this message in context: >> http://spssx-discussion.1045642.n5.nabble.com/Multi-rater-kappa-that-allows-marginal-totals-to-be-free-tp5720429.html >> Sent from the SPSSX Discussion mailing list archive at Nabble.com. >> >> ===================== >> To manage your subscription to SPSSX-L, send a message to > >> LISTSERV@.UGA > >> (not to SPSSX-L), with no body text except the >> command. To leave the list, send the command >> SIGNOFF SPSSX-L >> For a list of commands to manage subscriptions, send the command >> INFO REFCARD >> >> ===================== >> To manage your subscription to SPSSX-L, send a message to > >> LISTSERV@.UGA > >> (not to SPSSX-L), with no body text except the >> command. To leave the list, send the command >> SIGNOFF SPSSX-L >> For a list of commands to manage subscriptions, send the command >> INFO REFCARD > > > > > > ----- > -- > Bruce Weaver > bweaver@ > http://sites.google.com/a/lakeheadu.ca/bweaver/ > > "When all else fails, RTFM." > > NOTE: My Hotmail account is not monitored regularly. > To send me an e-mail, please use the address shown above. > > -- > View this message in context: > http://spssx-discussion.1045642.n5.nabble.com/Multi-rater-kappa-that-allows-marginal-totals-to-be-free-tp5720429p5720435.html > Sent from the SPSSX Discussion mailing list archive at Nabble.com. > > ===================== > To manage your subscription to SPSSX-L, send a message to > LISTSERV@.UGA > (not to SPSSX-L), with no body text except the > command. To leave the list, send the command > SIGNOFF SPSSX-L > For a list of commands to manage subscriptions, send the command > INFO REFCARD > > ===================== > To manage your subscription to SPSSX-L, send a message to > LISTSERV@.UGA > (not to SPSSX-L), with no body text except the > command. To leave the list, send the command > SIGNOFF SPSSX-L > For a list of commands to manage subscriptions, send the command > INFO REFCARD ----- -- Bruce Weaver [hidden email] http://sites.google.com/a/lakeheadu.ca/bweaver/ "When all else fails, RTFM." NOTE: My Hotmail account is not monitored regularly. To send me an e-mail, please use the address shown above. -- View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Multi-rater-kappa-that-allows-marginal-totals-to-be-free-tp5720429p5720437.html Sent from the SPSSX Discussion mailing list archive at Nabble.com. ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD |
|
|
In reply to this post by Bruce Weaver
The following is the macro for Justus Randolph's free marginal kappa. It's been tested with his data, as well as other data, and it reproduces his results. Sorry, but you may have to adjust the asterisks and backslashes on the comment lines as the formatting gets lost. The syntax presumes that the active dataset contains the data for analysis.
/* Introduction: */ /* This syntax analyzes and presents the results of Bennett's S for any number of raters and categories based on */ /* the work of Justus Randolph. Only an overall kappa is available. No */ /* solution for category analysis was provided. */ /* */ /* Preparation of the Data: */ /* Data are arranged with subects as rows and raters as columns. Cell contents represent the category */ /* that each rater assigned to each subject. Columns should be headed rater1, rater2, etc. to ratern, */ /* where n is the highest number rater. Enter the category of rating for each subject in the column */ /* representing each rater. The minimum category label allowed is "1". Therefore, if any category was coded 0, */ /* make sure to recode the categories prior to running the syntax. Save the file as an SPSS .sav file. In the first */ /* line of syntax below, beginning "GET FILE...", enter the path and filename of the saved file. */ /* */ /* Preparing the Syntax: */ /* The syntax runs on a single macro, the call line for which can be found near the end of the syntax, noted */ /* Macro Call Line. The Macro Call Line contains information related to the raters and the number of categories. */ On the Macro Call Line, enter the range */ /* of raters, e.g., rater1 to rater3, for three raters, or rater1 to rater6 for six raters. Do this after the prompt */ /* "raters=" and prior to the first "/". After the slash and "category=", enter the number of categories offered to */ /* the raters. If not all categories were used, this does not alter the number offered. For example, if six categories */ /* were offered to the raters, and only five were employed, enter "6" as the number of categories in the macro */ /* call line. */ /* */ /* Running the Syntax: */ /* On the toolbar at the top of the syntax editor, indicate "Run", "All". */ preserve. set printback=off mprint=off. define irr (raters=!charend('/')/category=!charend('/')). set mxloops=10000000. count ms__=!raters (missing). select if ms__=0. matrix. get x /var=!raters. compute n=nrow(x) . compute raters=ncol(x) . compute cat=!category . /* SECTION 1: Compute the requisite starting matrices. */ /* Compute an asymmetric matrix with the cases as rows, categories as */ /*columns, and the number of raters per case */ /* and category as cell entries. */ compute ratersum=make(nrow(x),cat,0). loop i=1 to nrow(x). loop j=1 to ncol(x). loop k=1 to cat. do if x(i,j)=k. compute ratersum(i,k)=ratersum(i,k)+1. end if. end loop. end loop. end loop. /* Compute an asymmetric matrix with categories as rows, raters as */ /* columns, /*and the number of cases per category */ /* and rater as the cell entry. */ compute casesum=make(cat,ncol(x),0) . loop i=1 to nrow(x). loop j=1 to ncol(x) . loop k=1 to cat. do if x(i,j)=k . compute casesum(k,j)=casesum(k,j)+1. end if . end loop. end loop. end loop. /* Compute an asymmetric matrix with categories as rows, raters as columns, and the number of agreements each */ /* rater has with the the other r-1 raters in each category as the cell entry. */ compute y=x . compute rtrcatag=make(cat,ncol(x),0) . loop i=1 to nrow(x) . loop j=1 to ncol(x) . loop k=1 to nrow(y) . loop l=1 to ncol(y) . do if i=k . do if j<>l . do if x(i,j)=y(k,l) . loop q=1 to cat. do if x(i,j)=q. compute rtrcatag(q,j)=rtrcatag(q,j)+1. end if . end loop . end if . end if . end if . end loop . end loop . end loop. end loop. /* SECTION 2: Compute the statistic of interest. */ /* Compute overall kappa empirically. */ compute fpa=mssq(ratersum)/(nrow(ratersum)*raters*(raters-1))-(1/(raters-1)) . compute fpe=1/cat . compute fk=(fpa-fpe)/(1-fpe) . do if fk<-1.00 . compute fk=-1.00 . end if . do if fk>1.00 . compute fk=1.00 . end if . compute bsicstat={n,raters,cat,fpa} . save bsicstat/outfile='bsicstat.sav' /variables=n,raters,cat,fpa . compute rtrprop1=ratersum/raters . compute rtrprop2=(ratersum-1)/(raters-1) . compute ratrprop=rtrprop1&*rtrprop2 . compute itempa=rsum(ratrprop) . compute itemfk=(itempa-fpe)/(1-fpe) . compute fkvar=(itemfk-fk)&**2 . compute fkvar=(1/(n*(n-1)))*csum(fkvar) . compute fkse=sqrt(fkvar)+.000001 . compute fkz=fk/fkse . compute fkp=1-cdfnorm(fkz) . compute fkll=fk-1.96*fkse . compute fkul=fk+1.96*fkse . do if fkll<-1.00 . compute fkll=-1.00 . end if . do if fkul>1.00 . compute fkul=1.00 . end if . compute fkstat={fk,fkse,fkz,fkp,fkll,fkul} . save fkstat/outfile='fkstat.sav' /variables=fk,fkse,fkz,fkp,fkll,fkul . /* Compute supplementary tables. */ compute casetbl=t(casesum) . compute ratertbl=t(rtrcatag) . /* Print supplementary tables while in matrix format. */ print {(casetbl)} /format="f5.0" /rlabels='Rater' /clabels='Category' /title='Summary Table - Number of Subjects by Rater and Category' . print {(ratertbl)} /format="f5.0" /rlabels='Rater' /clabels='Category' /title='Summary Table - Number of Agreements by Rater and Category' . end matrix. /* Prepare and print all information. */ get file='bsicstat.sav' . formats n(f8.0)/raters(f8.0)/cat(f8.0)/fpa(f8.5) . variable labels n 'Number of Items'/cat 'Number of Categories' /raters 'Number of Raters' /fpa 'Proportion of Rater Agreement' . report format=list automatic align(center) /variables=n,cat,raters,fpa /title "Basic Information" . get file='fkstat.sav' . formats fk(f8.5)/fkse(f8.5)/fkz(f8.5)/fkp(f8.5)/fkll(f8.5)/fkul(f8.5) . variable labels fk 'kappa'/fkse 'Standard Error'/fkz 'z'/fkp 'p'/fkll 'Lower 95% Confidence Limit'/ fkul 'Upper 95% Confidence Limit' . report format=list automatic align(center) /variables=fk,fkse,fkz,fkp,fkll,fkul /title "Empirical Confidence Limits - Overall kappa" . !enddefine. restore. /* Macro Call Line. */ irr raters=rater1 to rater3/category=2 . ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD |
Re: Multi-rater kappa that allows marginal totals to be free
|
Administrator
|
In reply to this post by Bruce Weaver
Here's a link for Randolph's macro that works:
http://justusrandolph.net/kappa/mrakmacro.htm The 3 links at the top of that page are dead, but notice that Dave Nichols’ MKAPPASC macro and the accompanying Read Me file are shown at the bottom of the page. As Brian noted, it would be better to modify Nichols' macro to have the number of categories as a macro argument than to edit the macro definition internally. Cheers, Bruce
--
Bruce Weaver bweaver@lakeheadu.ca http://sites.google.com/a/lakeheadu.ca/bweaver/ "When all else fails, RTFM." PLEASE NOTE THE FOLLOWING: 1. My Hotmail account is not monitored regularly. To send me an e-mail, please use the address shown above. 2. The SPSSX Discussion forum on Nabble is no longer linked to the SPSSX-L listserv administered by UGA (https://listserv.uga.edu/). |
Automatic reply: Multi-rater kappa that allows marginal totals to be free
|
|
I am currently out of the office until Friday, May 31. If you need immediate assistance, please call 812-856-5824.
Shimon Sarraf
|
«
Return to SPSSX Discussion
|
1 view|%1 views
| Free forum by Nabble | Edit this page |