Fleiss Kappa Macro for SPSS
Locked
Fleiss Kappa Macro for SPSS
 
|
I am needing to use Fleiss' Kappa analysis in SPSS so that I can calculate the inter-rater reliability where there are more than 2 judges.
I believe that I will need a macro file to be able to perform this analysis in SPSS is this correct? IF so does anyone have a file. Any help would be appreciated. Thanks Kristina Georgiou Research Student
Kristina Georgiou
Research Student |
|
|
Kristina,
I have a macro for Fleiss' kappa. I also have macros for multi-rater adaptations of Cohen's kappa if you're interested. Light's kappa averages the two-rater Cohen's kappa. Hubert's kappa averages the two-rater error terms but uses the overall proportion of agreement. Hubert's kappa has been reinvented by several others - Conger (1980), Davies and Fleiss (1982), Berry & Mielke (1988), Komagata (2002), and Artstein and Poesio (2008). Let me know, and I can send the Fleiss macro with directions. Brian -----Original Message----- From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Kristina Georgiou Sent: Tuesday, November 15, 2011 9:37 AM To: [hidden email] Subject: [NEWSENDER] - Fleiss Kappa Macro for SPSS - Message is from an unknown sender I am needing to use Fleiss' Kappa analysis in SPSS so that I can calculate the inter-rater reliability where there are more than 2 judges. I believe that I will need a macro file to be able to perform this analysis in SPSS is this correct? IF so does anyone have a file. Any help would be appreciated. Thanks Kristina Georgiou Research Student -- View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Fleiss-Kappa-Macro-for-SPS S-tp4994433p4994433.html Sent from the SPSSX Discussion mailing list archive at Nabble.com. ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD |
Re: Fleiss Kappa Macro for SPSS
|
|
In reply to this post by Kristina Georgiou
Brian,
I am not sure if the adaptations of the Cohen's Kappa you mention below for multi-raters would be more suitable than Fleiss' Kappa. Is there some reason why I should research these other analysis options for inter-rater reliability instead of Fleiss Kappa? If not then could you please send through the macro for Fleiss Kappa and any instructions that will assist me in getting the macro to run in the latest version of SPSS. Thanks Kristina __________________________________ Kristina, I have a macro for Fleiss' kappa. I also have macros for multi-rater adaptations of Cohen's kappa if you're interested. Light's kappa averages the two-rater Cohen's kappa. Hubert's kappa averages the two-rater error terms but uses the overall proportion of agreement. Hubert's kappa has been reinvented by several others - Conger (1980), Davies and Fleiss (1982), Berry & Mielke (1988), Komagata (2002), and Artstein and Poesio (2008). Let me know, and I can send the Fleiss macro with directions. Brian -----Original Message----- From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Kristina Georgiou Sent: Tuesday, November 15, 2011 9:37 AM To: [hidden email] Subject: [NEWSENDER] - Fleiss Kappa Macro for SPSS - Message is from an unknown sender I am needing to use Fleiss' Kappa analysis in SPSS so that I can calculate the inter-rater reliability where there are more than 2 judges. I believe that I will need a macro file to be able to perform this analysis in SPSS is this correct? IF so does anyone have a file. Any help would be appreciated. Thanks Kristina Georgiou Research Student -- View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Fleiss-Kappa-Macro-for-SPS S-tp4994433p4994433.html Sent from the SPSSX Discussion mailing list archive at Nabble.com. ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD
Kristina Georgiou
Research Student |
Administrator
|
Kristina, I don't think you've said how many categories are available for raters. If the categories are ordered, or if there are only two categories, you could compute the the most common form of intraclass correlation, which is equivalent to weighted kappa (with quadratic weights). The following is from a newsgroup post I made (to comp.soft-sys.stat.spss) back in 2002.
--- start of old post --- The following is from Norman & Streiner (2000), Biostatistics: The Bare Essentials: ------------------------------------------------------ RELATION BETWEEN KAPPA AND THE INTRACLASS CORRELATION One reason why Cicchetti was fighting a losing battle is that the weighted kappa using quadratic weights has a very general property--it is mathematically (i.e., exactly) equal to the ICC. (p. 221) ------------------------------------------------------ There are several varieties of ICC. The one Norman & Streiner are describing is this one (p. 100 in their book, 2nd Ed): MS_subj - MS_residual ICC = ---------------------------- MS(subj) + (k-1)*MS_residual --- end of old post --- HTH.
--
Bruce Weaver bweaver@lakeheadu.ca http://sites.google.com/a/lakeheadu.ca/bweaver/ "When all else fails, RTFM." PLEASE NOTE THE FOLLOWING: 1. My Hotmail account is not monitored regularly. To send me an e-mail, please use the address shown above. 2. The SPSSX Discussion forum on Nabble is no longer linked to the SPSSX-L listserv administered by UGA (https://listserv.uga.edu/). |
|
|
Good point, Bruce. I assumed that the categories were not ordered and
>2, so sent the syntax. But there's ample evidence that once categories are ordered the ICC provides the best solution. Brian -----Original Message----- From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Bruce Weaver Sent: Wednesday, November 16, 2011 9:43 AM To: [hidden email] Subject: Re: Fleiss Kappa Macro for SPSS Kristina, I don't think you've said how many categories are available for raters. If the categories are ordered, or if there are only two categories, you could compute the the most common form of intraclass correlation, which is equivalent to weighted kappa (with quadratic weights). The following is from a newsgroup post I made (to comp.soft-sys.stat.spss) back in 2002. --- start of old post --- The following is from Norman & Streiner (2000), Biostatistics: The Bare Essentials: ------------------------------------------------------ RELATION BETWEEN KAPPA AND THE INTRACLASS CORRELATION One reason why Cicchetti was fighting a losing battle is that the weighted kappa using quadratic weights has a very general property--it is mathematically (i.e., exactly) equal to the ICC. (p. 221) ------------------------------------------------------ There are several varieties of ICC. The one Norman & Streiner are describing is this one (p. 100 in their book, 2nd Ed): MS_subj - MS_residual ICC = ---------------------------- MS(subj) + (k-1)*MS_residual --- end of old post --- HTH. Kristina Georgiou wrote: > > Brian, > > I am not sure if the adaptations of the Cohen's Kappa you mention below > for multi-raters would be more suitable than Fleiss' Kappa. Is there some > reason why I should research these other analysis options for inter-rater > reliability instead of Fleiss Kappa? If not then could you please send > through the macro for Fleiss Kappa and any instructions that will assist > me in getting the macro to run in the latest version of SPSS. > > Thanks > > Kristina > > > > __________________________________ > Kristina, > > I have a macro for Fleiss' kappa. I also have macros for multi-rater > adaptations of Cohen's kappa if you're interested. Light's kappa > averages the two-rater Cohen's kappa. Hubert's kappa averages the > two-rater error terms but uses the overall proportion of agreement. > Hubert's kappa has been reinvented by several others - Conger (1980), > Davies and Fleiss (1982), Berry & Mielke (1988), Komagata (2002), and > Artstein and Poesio (2008). Let me know, and I can send the Fleiss > macro with directions. > > Brian > > -----Original Message----- > From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of > Kristina Georgiou > Sent: Tuesday, November 15, 2011 9:37 AM > To: [hidden email] > Subject: [NEWSENDER] - Fleiss Kappa Macro for SPSS - Message is from > unknown sender > > I am needing to use Fleiss' Kappa analysis in SPSS so that I can > calculate > the inter-rater reliability where there are more than 2 judges. > > I believe that I will need a macro file to be able to perform this > analysis > in SPSS is this correct? IF so does anyone have a file. > > Any help would be appreciated. > > Thanks > > Kristina Georgiou > Research Student > > -- > View this message in context: > > S-tp4994433p4994433.html > Sent from the SPSSX Discussion mailing list archive at Nabble.com. > > ===================== > To manage your subscription to SPSSX-L, send a message to > [hidden email] (not to SPSSX-L), with no body text except the > command. To leave the list, send the command > SIGNOFF SPSSX-L > For a list of commands to manage subscriptions, send the command > INFO REFCARD > > ===================== > To manage your subscription to SPSSX-L, send a message to > [hidden email] (not to SPSSX-L), with no body text except the > command. To leave the list, send the command > SIGNOFF SPSSX-L > For a list of commands to manage subscriptions, send the command > INFO REFCARD > ----- -- Bruce Weaver [hidden email] http://sites.google.com/a/lakeheadu.ca/bweaver/ "When all else fails, RTFM." NOTE: My Hotmail account is not monitored regularly. To send me an e-mail, please use the address shown above. -- View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Fleiss-Kappa-Macro-for-SPS S-tp4994433p4997918.html Sent from the SPSSX Discussion mailing list archive at Nabble.com. ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD |
Re: Fleiss Kappa Macro for SPSS
|
|
In reply to this post by Bruce Weaver
There are 9 categories which 3 raters(judges) can choose from, and each rater has rated over 200 different objects. I am not sure what you mean by the categories being 'ordered'.
------------------------------------------ Kristina, I don't think you've said how many categories are available for raters. If the categories are ordered, or if there are only two categories, you could compute the the most common form of intraclass correlation, which is equivalent to weighted kappa (with quadratic weights). The following is from a newsgroup post I made (to comp.soft-sys.stat.spss) back in 2002. --- start of old post --- The following is from Norman & Streiner (2000), Biostatistics: The Bare Essentials: ------------------------------------------------------ RELATION BETWEEN KAPPA AND THE INTRACLASS CORRELATION One reason why Cicchetti was fighting a losing battle is that the weighted kappa using quadratic weights has a very general property--it is mathematically (i.e., exactly) equal to the ICC. (p. 221) ------------------------------------------------------ There are several varieties of ICC. The one Norman & Streiner are describing is this one (p. 100 in their book, 2nd Ed): MS_subj - MS_residual ICC = ---------------------------- MS(subj) + (k-1)*MS_residual --- end of old post --- HTH.
Kristina Georgiou
Research Student |
|
|
Kristina,
Ordered categories contain ordinal data rather than nominal data, so there is a natural rank among the categories. Brian -----Original Message----- From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Kristina Georgiou Sent: Wednesday, November 16, 2011 11:03 AM To: [hidden email] Subject: Re: Fleiss Kappa Macro for SPSS There are 9 categories which 3 raters(judges) can choose from, and each rater has rated over 200 different objects. I am not sure what you mean by the categories being 'ordered'. ------------------------------------------ Kristina, I don't think you've said how many categories are available for raters. If the categories are ordered, or if there are only two categories, you could compute the the most common form of intraclass correlation, which is equivalent to weighted kappa (with quadratic weights). The following is from a newsgroup post I made (to comp.soft-sys.stat.spss) back in 2002. --- start of old post --- The following is from Norman & Streiner (2000), Biostatistics: The Bare Essentials: ------------------------------------------------------ RELATION BETWEEN KAPPA AND THE INTRACLASS CORRELATION One reason why Cicchetti was fighting a losing battle is that the weighted kappa using quadratic weights has a very general property--it is mathematically (i.e., exactly) equal to the ICC. (p. 221) ------------------------------------------------------ There are several varieties of ICC. The one Norman & Streiner are describing is this one (p. 100 in their book, 2nd Ed): MS_subj - MS_residual ICC = ---------------------------- MS(subj) + (k-1)*MS_residual --- end of old post --- HTH. ----- Kristina Georgiou Research Student -- View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Fleiss-Kappa-Macro-for-SPS S-tp4994433p4998187.html Sent from the SPSSX Discussion mailing list archive at Nabble.com. ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD |
|
|
In reply to this post by Kristina Georgiou
"Ordered" -- If there were three categories labelled [Never, Sometimes, Always],
those would be "ordered" and it would be convenient and usually wise to treat them as continuous, and then to compute some version of an ICC (intra-class correlation). It may be less likely that your 9 categories are ordered, than if you had only 3 or 4. However - without knowing your data, which might make a difference - I think you should consider abandoning any kappa based on 9 categories. Kappas with multiple categories are *so* heavily dependent on the relative sizes of the marginal totals that the absolute size of kappa is nearly meaningless. That is to say, kappas can give you a test of agreement; but the kappa as a report of agreement is seldom useful except for a narrow comparison of coefficients within your data on hand, whenever they have more than 2 categories. And if you are contrasting kappas for pairs chosen from three raters (say), where the actual values already have almost no meaning, there is little purpose in considering what the 3-person "overall" kappa might be. What I have found useful, instead, is to consider the 2x2 kappas based on "category1 vs. all other", and so on. (You might see that there is good success for raters on the rationally-important categories or the most-used ones, with indifferent success on several others.) - If the kappas are similar, presenting the average kappa should be satisfactory. If the kappas are not similar, you should probably not try to represent them with one number, no matter how you compute it. -- Rich Ulrich > Date: Wed, 16 Nov 2011 08:03:24 -0800 > From: [hidden email] > Subject: Re: Fleiss Kappa Macro for SPSS > To: [hidden email] > > There are 9 categories which 3 raters(judges) can choose from, and each rater > has rated over 200 different objects. I am not sure what you mean by the > categories being 'ordered'. > ------------------------------------------ [snip, previous] |
Re: Fleiss Kappa Macro for SPSS
|
|
In reply to this post by Kristina Georgiou
Kappa in general is not a measure of reliability but of agreement, although there are some forms of kappa that are practically equivalent to the intraclass correlation.
Dr. Paul R. Swank, Children's Learning Institute Professor, Department of Pediatrics, Medical School Adjunct Professor, School of Public Health University of Texas Health Science Center-Houston -----Original Message----- From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Kristina Georgiou Sent: Wednesday, November 16, 2011 8:09 AM To: [hidden email] Subject: Re: Fleiss Kappa Macro for SPSS Brian, I am not sure if the adaptations of the Cohen's Kappa you mention below for multi-raters would be more suitable than Fleiss' Kappa. Is there some reason why I should research these other analysis options for inter-rater reliability instead of Fleiss Kappa? If not then could you please send through the macro for Fleiss Kappa and any instructions that will assist me in getting the macro to run in the latest version of SPSS. Thanks Kristina __________________________________ Kristina, I have a macro for Fleiss' kappa. I also have macros for multi-rater adaptations of Cohen's kappa if you're interested. Light's kappa averages the two-rater Cohen's kappa. Hubert's kappa averages the two-rater error terms but uses the overall proportion of agreement. Hubert's kappa has been reinvented by several others - Conger (1980), Davies and Fleiss (1982), Berry & Mielke (1988), Komagata (2002), and Artstein and Poesio (2008). Let me know, and I can send the Fleiss macro with directions. Brian -----Original Message----- From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Kristina Georgiou Sent: Tuesday, November 15, 2011 9:37 AM To: [hidden email] Subject: [NEWSENDER] - Fleiss Kappa Macro for SPSS - Message is from an unknown sender I am needing to use Fleiss' Kappa analysis in SPSS so that I can calculate the inter-rater reliability where there are more than 2 judges. I believe that I will need a macro file to be able to perform this analysis in SPSS is this correct? IF so does anyone have a file. Any help would be appreciated. Thanks Kristina Georgiou Research Student -- View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Fleiss-Kappa-Macro-for-SPS S-tp4994433p4994433.html Sent from the SPSSX Discussion mailing list archive at Nabble.com. ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD ----- Kristina Georgiou Research Student -- View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Fleiss-Kappa-Macro-for-SPSS-tp4994433p4997817.html Sent from the SPSSX Discussion mailing list archive at Nabble.com. ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD |
|
|
In reply to this post by Rich Ulrich
hello
I have a problem about fleiss macro. I use the below macro, but when I run the macro, a lot of error shows . and ı couldnt find a solution. Please help.... data list list /rater1 rater2 rater3 rater4 rater5 rater6 rater7 rater8 rater9 . begin data *veriler buraya giriliyor/ 1 3 1 1 1 3 1 2 1 1 1 3 1 1 1 1 1 1 1 1 3 1 1 3 1 1 3 1 1 3 1 1 3 1 3 3 1 1 3 1 1 1 1 1 1 1 1 3 1 1 1 1 3 3 1 1 1 1 1 1 1 1 1 1 3 3 1 1 1 1 1 1 1 3 3 1 1 3 1 1 1 1 1 2 1 1 3 1 1 1 1 3 3 1 1 3 1 1 1 1 1 3 1 1 3 1 1 1 1 1 3 1 1 3 1 1 1 1 1 3 1 1 3 1 1 3 1 1 3 1 1 3 1 1 1 1 3 3 1 1 1 1 1 1 1 1 3 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 3 1 1 1 1 1 1 1 1 3 1 1 3 1 1 3 1 3 1 1 1 3 1 1 1 end data . preserve. set printback=off mprint=off. save outfile='k__tmp1.sav'. define cohensk (vars=!charend('/')). set mxloops=100000. count ms__=!vars (missing). select if ms__=0. *This section sets up a matrix(x) based on the raw data file, a matrix(y) with rows equal to the number of items and *columns equal to the number of categories, then determines for y the number of responses per category for each *case or item. matrix. get x /var=!vars. compute c=mmax(x). compute y=make(nrow(x),c,0). loop i=1 to nrow(x). loop j=1 to ncol(x). loop k=1 to c. do if x(i,j)=k. compute y(i,k)=y(i,k)+1. end if. end loop. end loop. end loop. *This section computes the basic information and kappa and its related statistics. compute pe=msum((csum(y)/msum(y))&**2). compute k=ncol(x). compute n=nrow(y) . compute r=ncol(y) . compute pa=mssq(y)/(nrow(y)*k*(k-1))-(1/(k-1)). compute ck=(pa-pe)/(1-pe). compute num=2*(pe-(2*k-3)*(pe**2)+2*(k-2)*msum((csum(y)/msum(y))&**3)). compute den=nrow(y)*k*(k-1)*((1-pe)**2). compute ase=sqrt(num/den). compute z=ck/ase. compute sig=1-chicdf(z**2,1) . compute ckul=ck+1.96*ase . compute ckll=ck-1.96*ase . *This section computes the alternate standard error and related statistics based on Fliess' corrected formula. compute nm=sqrt(n*k*(k-1)) . compute vectora=csum(y)/msum(csum(y)) . compute vectorb=1-csum(y)/msum(csum(y)) . compute vectorc=1-2*(csum(y)/msum(csum(y))) . compute vectord=vectora&*vectorb. compute vectore=vectora&*vectorb&*vectorc . compute e=msum(vectord) . compute f =msum(vectore) . compute fse=(sqrt(2)/(e*nm))*(sqrt(e**2-f)) . compute fsez=ck/fse . compute fsesig=1-chicdf(fsez**2,1) . compute fseu=ck+1.96*fse . compute fsel=ck-1.96*fse . * This section computes the kappas for the individual categories. Each statistic, e.g., k or standard error, is computed * as a vector. The vectors are then assembled into a matrix of all six statistics. As part of this process, -99999.0 is * imputed as the missing data value. compute matz=k-y . compute mata=y&*matz . compute vectorf=csum(mata)+(.0001) . compute vectorg=vectord*(n*k*(k-1))+(.0001) . compute vectorh=1-(vectorf&/vectorg) . compute vectori=(1+(2*(k-1)*(csum(y)/msum(csum(y)))))&**2 . compute vectorj=(2*(k-1))*vectord . compute vectork=(n*k*(k-1)**2)*vectord+(.0001) . compute vectorse=sqrt((vectori+vectorj)&/vectork) . compute vectorz=vectorh&/(vectorse+.0001). compute vectorp=1-cdfnorm(vectorz) . compute vectorll=vectorh-(1.96*vectorse) . compute vectorul=vectorh+(1.96*vectorse) . loop i=1 to ncol(vectorh) . do if (vectorh(i)=0.00) . compute vectorh(i)=-99999) . end if . end loop . loop i=1 to ncol(vectorh) . do if (vectorh(i)=-99999) . compute vectorse(i)=-99999 . end if . end loop . loop i=1 to ncol(vectorh) . do if (vectorh(i)=-99999) . compute vectorz(i)=-99999 . end if . end loop . loop i=1 to ncol(vectorh) . do if (vectorh(i)=-99999) . compute vectorp(i)=-99999 . end if . end loop . loop i=1 to ncol(vectorh) . do if (vectorh(i)=-99999) . compute vectorul(i)=-99999. end if . end loop . loop i=1 to ncol(vectorh) . do if (vectorh(i)=-99999) . compute vectorll(i)=-99999 . end if . end loop . compute ikstat={vectorh;vectorse;vectorz;vectorp;vectorll;vectorul} . save ikstat /outfile='ikstat1.sav' . *This section saves the data and prepares the reporting formats. save {k,n,r,pa,ck,ase,z,sig,ckll,ckul,fse,fsez,fsesig,fseu,fsel} /outfile='k__tmp2.sav' /variables=k,n,r,pa,ck,ase,z,sig,ckll,ckul,fse,fsez,fsesig,fseu,fsel . end matrix . get file='k__tmp2.sav'. formats k (f8.0) /n (f8.0) /r (f8.0) /pa (f8.4) /ck (f8.4) /ase (f8.4) /z (f8.4) /sig (f8.4) /ckul (f8.4) /ckll (f8.4) /fse(f8.4) /fsez (f8.4) /fsesig (f8.4) /fseu (f8.4) /fsel (f8.4) . variable labels k 'Number of Raters' /n 'Number of Items' /r 'Number of Categories' /pa 'Percent of Rater Agreement' /ck 'Kappa' /ase 'Standard Error' /z 'z'/sig 'p' /ckul 'Upper 95% Confidence Limit' /ckll 'Lower 95% Confidence Limit' /fse 'Fleiss SE' /fsez 'z' /fsesig 'p' /fseu 'Upper 95% Confidence Limit' /fsel 'Lower 95% Confidence Limit' . report format=list automatic align(center) /variables=k,n,r,pa /title "Basic Information" . report format=list automatic align(center) /variables=ck ase z sig ckll ckul /title "Cohens Kappa". report format=list automatic align(center) /variables=fse fsez fsesig fsel fseu /title "Cohens Kappa -- Fleiss Adjusted Standard Error" . get file='ikstat1.sav' . flip . delete variable case_lbl . compute n1=$casenum . formats n1 (f8.0) /var001 (f8.4) /var002 (f8.4) /var003 (f8.4) /var004 (f8.4) /var005 (f8.4) /var006 (f8.4) . variable labels n1 'Coding Category' /var001 'k' /var002 'Standard Error' /var003 'z' /var004 'p' /var005 'Lower 95% Confidence Limit' /var006 'Upper 95% Confidence Limit' . save outfile='ikstat.sav' . report format=list automatic align(center) /variables=n1 var001 var002 var003 var004 var005 var006 /Title "Individual Category Statistics" . !enddefine. restore. COHENSK VARS = rater1 to rater9 . |
Administrator
|
Get rid of the extraneous crap between BEGIN DATA and END DATA (ie commented text and blank lines!).
Please reply to the list and not to my personal email.
Those desiring my consulting or training services please feel free to email me. --- "Nolite dare sanctum canibus neque mittatis margaritas vestras ante porcos ne forte conculcent eas pedibus suis." Cum es damnatorum possederunt porcos iens ut salire off sanguinum cliff in abyssum?" |
Re: Fleiss Kappa Macro for SPSS
|
|
In reply to this post by cido
Do you really have a blank line and that line of text following the Begin Data statement and then do you have a blank line between the last data line and the End Data statement? If so, please read the documentation for the Data List command.
Gene Maguin -----Original Message----- From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of cido Sent: Tuesday, July 15, 2014 4:55 AM To: [hidden email] Subject: Re: Fleiss Kappa Macro for SPSS hello I have a problem about fleiss macro. I use the below macro, but when I run the macro, a lot of error shows . and ı couldnt find a solution. Please help.... data list list /rater1 rater2 rater3 rater4 rater5 rater6 rater7 rater8 rater9 . begin data *veriler buraya giriliyor/ 1 3 1 1 1 3 1 2 1 1 1 3 1 1 1 1 1 1 1 1 3 1 1 3 1 1 3 1 1 3 1 1 3 1 3 3 1 1 3 1 1 1 1 1 1 1 1 3 1 1 1 1 3 3 1 1 1 1 1 1 1 1 1 1 3 3 1 1 1 1 1 1 1 3 3 1 1 3 1 1 1 1 1 2 1 1 3 1 1 1 1 3 3 1 1 3 1 1 1 1 1 3 1 1 3 1 1 1 1 1 3 1 1 3 1 1 1 1 1 3 1 1 3 1 1 3 1 1 3 1 1 3 1 1 1 1 3 3 1 1 1 1 1 1 1 1 3 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 3 1 1 1 1 1 1 1 1 3 1 1 3 1 1 3 1 3 1 1 1 3 1 1 1 end data . preserve. set printback=off mprint=off. save outfile='k__tmp1.sav'. define cohensk (vars=!charend('/')). set mxloops=100000. count ms__=!vars (missing). select if ms__=0. *This section sets up a matrix(x) based on the raw data file, a matrix(y) with rows equal to the number of items and *columns equal to the number of categories, then determines for y the number of responses per category for each *case or item. matrix. get x /var=!vars. compute c=mmax(x). compute y=make(nrow(x),c,0). loop i=1 to nrow(x). loop j=1 to ncol(x). loop k=1 to c. do if x(i,j)=k. compute y(i,k)=y(i,k)+1. end if. end loop. end loop. end loop. *This section computes the basic information and kappa and its related statistics. compute pe=msum((csum(y)/msum(y))&**2). compute k=ncol(x). compute n=nrow(y) . compute r=ncol(y) . compute pa=mssq(y)/(nrow(y)*k*(k-1))-(1/(k-1)). compute ck=(pa-pe)/(1-pe). compute num=2*(pe-(2*k-3)*(pe**2)+2*(k-2)*msum((csum(y)/msum(y))&**3)). compute den=nrow(y)*k*(k-1)*((1-pe)**2). compute ase=sqrt(num/den). compute z=ck/ase. compute sig=1-chicdf(z**2,1) . compute ckul=ck+1.96*ase . compute ckll=ck-1.96*ase . *This section computes the alternate standard error and related statistics based on Fliess' corrected formula. compute nm=sqrt(n*k*(k-1)) . compute vectora=csum(y)/msum(csum(y)) . compute vectorb=1-csum(y)/msum(csum(y)) . compute vectorc=1-2*(csum(y)/msum(csum(y))) . compute vectord=vectora&*vectorb. compute vectore=vectora&*vectorb&*vectorc . compute e=msum(vectord) . compute f =msum(vectore) . compute fse=(sqrt(2)/(e*nm))*(sqrt(e**2-f)) . compute fsez=ck/fse . compute fsesig=1-chicdf(fsez**2,1) . compute fseu=ck+1.96*fse . compute fsel=ck-1.96*fse . * This section computes the kappas for the individual categories. Each statistic, e.g., k or standard error, is computed * as a vector. The vectors are then assembled into a matrix of all six statistics. As part of this process, -99999.0 is * imputed as the missing data value. compute matz=k-y . compute mata=y&*matz . compute vectorf=csum(mata)+(.0001) . compute vectorg=vectord*(n*k*(k-1))+(.0001) . compute vectorh=1-(vectorf&/vectorg) . compute vectori=(1+(2*(k-1)*(csum(y)/msum(csum(y)))))&**2 . compute vectorj=(2*(k-1))*vectord . compute vectork=(n*k*(k-1)**2)*vectord+(.0001) . compute vectorse=sqrt((vectori+vectorj)&/vectork) . compute vectorz=vectorh&/(vectorse+.0001). compute vectorp=1-cdfnorm(vectorz) . compute vectorll=vectorh-(1.96*vectorse) . compute vectorul=vectorh+(1.96*vectorse) . loop i=1 to ncol(vectorh) . do if (vectorh(i)=0.00) . compute vectorh(i)=-99999) . end if . end loop . loop i=1 to ncol(vectorh) . do if (vectorh(i)=-99999) . compute vectorse(i)=-99999 . end if . end loop . loop i=1 to ncol(vectorh) . do if (vectorh(i)=-99999) . compute vectorz(i)=-99999 . end if . end loop . loop i=1 to ncol(vectorh) . do if (vectorh(i)=-99999) . compute vectorp(i)=-99999 . end if . end loop . loop i=1 to ncol(vectorh) . do if (vectorh(i)=-99999) . compute vectorul(i)=-99999. end if . end loop . loop i=1 to ncol(vectorh) . do if (vectorh(i)=-99999) . compute vectorll(i)=-99999 . end if . end loop . compute ikstat={vectorh;vectorse;vectorz;vectorp;vectorll;vectorul} . save ikstat /outfile='ikstat1.sav' . *This section saves the data and prepares the reporting formats. save {k,n,r,pa,ck,ase,z,sig,ckll,ckul,fse,fsez,fsesig,fseu,fsel} /outfile='k__tmp2.sav' /variables=k,n,r,pa,ck,ase,z,sig,ckll,ckul,fse,fsez,fsesig,fseu,fsel . end matrix . get file='k__tmp2.sav'. formats k (f8.0) /n (f8.0) /r (f8.0) /pa (f8.4) /ck (f8.4) /ase (f8.4) /z (f8.4) /sig (f8.4) /ckul (f8.4) /ckll (f8.4) /fse(f8.4) /fsez (f8.4) /fsesig (f8.4) /fseu (f8.4) /fsel (f8.4) . variable labels k 'Number of Raters' /n 'Number of Items' /r 'Number of Categories' /pa 'Percent of Rater Agreement' /ck 'Kappa' /ase 'Standard Error' /z 'z'/sig 'p' /ckul 'Upper 95% Confidence Limit' /ckll 'Lower 95% Confidence Limit' /fse 'Fleiss SE' /fsez 'z' /fsesig 'p' /fseu 'Upper 95% Confidence Limit' /fsel 'Lower 95% Confidence Limit' . report format=list automatic align(center) /variables=k,n,r,pa /title "Basic Information" . report format=list automatic align(center) /variables=ck ase z sig ckll ckul /title "Cohens Kappa". report format=list automatic align(center) /variables=fse fsez fsesig fsel fseu /title "Cohens Kappa -- Fleiss Adjusted Standard Error" . get file='ikstat1.sav' . flip . delete variable case_lbl . compute n1=$casenum . formats n1 (f8.0) /var001 (f8.4) /var002 (f8.4) /var003 (f8.4) /var004 (f8.4) /var005 (f8.4) /var006 (f8.4) . variable labels n1 'Coding Category' /var001 'k' /var002 'Standard Error' /var003 'z' /var004 'p' /var005 'Lower 95% Confidence Limit' /var006 'Upper 95% Confidence Limit' . save outfile='ikstat.sav' . report format=list automatic align(center) /variables=n1 var001 var002 var003 var004 var005 var006 /Title "Individual Category Statistics" . !enddefine. restore. COHENSK VARS = rater1 to rater9 . -- View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Fleiss-Kappa-Macro-for-SPSS-tp4994433p5726711.html Sent from the SPSSX Discussion mailing list archive at Nabble.com. ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD |
|
|
In reply to this post by cido
You might want to consider the Fleiss Kappa
custom dialog (STATS FLEISS KAPPA). These require the Python Essentials.
With Statistics 22 you can install this from the Utilities menu;
with older versions you would install this from the SPSS Community website
(www.ibm.com/developerworks/spssdevcentral)
via Downloads for SPSS Statistics in the Extensions Command collection.
Jon Peck (no "h") aka Kim Senior Software Engineer, IBM [hidden email] phone: 720-342-5621 From: cido <[hidden email]> To: [hidden email], Date: 07/15/2014 07:35 AM Subject: Re: [SPSSX-L] Fleiss Kappa Macro for SPSS Sent by: "SPSSX(r) Discussion" <[hidden email]> hello I have a problem about fleiss macro. I use the below macro, but when I run the macro, a lot of error shows . and ı couldnt find a solution. Please help.... data list list /rater1 rater2 rater3 rater4 rater5 rater6 rater7 rater8 rater9 . begin data *veriler buraya giriliyor/ 1 3 1 1 1 3 1 2 1 1 1 3 1 1 1 1 1 1 1 1 3 1 1 3 1 1 3 1 1 3 1 1 3 1 3 3 1 1 3 1 1 1 1 1 1 1 1 3 1 1 1 1 3 3 1 1 1 1 1 1 1 1 1 1 3 3 1 1 1 1 1 1 1 3 3 1 1 3 1 1 1 1 1 2 1 1 3 1 1 1 1 3 3 1 1 3 1 1 1 1 1 3 1 1 3 1 1 1 1 1 3 1 1 3 1 1 1 1 1 3 1 1 3 1 1 3 1 1 3 1 1 3 1 1 1 1 3 3 1 1 1 1 1 1 1 1 3 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 3 1 1 1 1 1 1 1 1 3 1 1 3 1 1 3 1 3 1 1 1 3 1 1 1 end data . preserve. set printback=off mprint=off. save outfile='k__tmp1.sav'. define cohensk (vars=!charend('/')). set mxloops=100000. count ms__=!vars (missing). select if ms__=0. *This section sets up a matrix(x) based on the raw data file, a matrix(y) with rows equal to the number of items and *columns equal to the number of categories, then determines for y the number of responses per category for each *case or item. matrix. get x /var=!vars. compute c=mmax(x). compute y=make(nrow(x),c,0). loop i=1 to nrow(x). loop j=1 to ncol(x). loop k=1 to c. do if x(i,j)=k. compute y(i,k)=y(i,k)+1. end if. end loop. end loop. end loop. *This section computes the basic information and kappa and its related statistics. compute pe=msum((csum(y)/msum(y))&**2). compute k=ncol(x). compute n=nrow(y) . compute r=ncol(y) . compute pa=mssq(y)/(nrow(y)*k*(k-1))-(1/(k-1)). compute ck=(pa-pe)/(1-pe). compute num=2*(pe-(2*k-3)*(pe**2)+2*(k-2)*msum((csum(y)/msum(y))&**3)). compute den=nrow(y)*k*(k-1)*((1-pe)**2). compute ase=sqrt(num/den). compute z=ck/ase. compute sig=1-chicdf(z**2,1) . compute ckul=ck+1.96*ase . compute ckll=ck-1.96*ase . *This section computes the alternate standard error and related statistics based on Fliess' corrected formula. compute nm=sqrt(n*k*(k-1)) . compute vectora=csum(y)/msum(csum(y)) . compute vectorb=1-csum(y)/msum(csum(y)) . compute vectorc=1-2*(csum(y)/msum(csum(y))) . compute vectord=vectora&*vectorb. compute vectore=vectora&*vectorb&*vectorc . compute e=msum(vectord) . compute f =msum(vectore) . compute fse=(sqrt(2)/(e*nm))*(sqrt(e**2-f)) . compute fsez=ck/fse . compute fsesig=1-chicdf(fsez**2,1) . compute fseu=ck+1.96*fse . compute fsel=ck-1.96*fse . * This section computes the kappas for the individual categories. Each statistic, e.g., k or standard error, is computed * as a vector. The vectors are then assembled into a matrix of all six statistics. As part of this process, -99999.0 is * imputed as the missing data value. compute matz=k-y . compute mata=y&*matz . compute vectorf=csum(mata)+(.0001) . compute vectorg=vectord*(n*k*(k-1))+(.0001) . compute vectorh=1-(vectorf&/vectorg) . compute vectori=(1+(2*(k-1)*(csum(y)/msum(csum(y)))))&**2 . compute vectorj=(2*(k-1))*vectord . compute vectork=(n*k*(k-1)**2)*vectord+(.0001) . compute vectorse=sqrt((vectori+vectorj)&/vectork) . compute vectorz=vectorh&/(vectorse+.0001). compute vectorp=1-cdfnorm(vectorz) . compute vectorll=vectorh-(1.96*vectorse) . compute vectorul=vectorh+(1.96*vectorse) . loop i=1 to ncol(vectorh) . do if (vectorh(i)=0.00) . compute vectorh(i)=-99999) . end if . end loop . loop i=1 to ncol(vectorh) . do if (vectorh(i)=-99999) . compute vectorse(i)=-99999 . end if . end loop . loop i=1 to ncol(vectorh) . do if (vectorh(i)=-99999) . compute vectorz(i)=-99999 . end if . end loop . loop i=1 to ncol(vectorh) . do if (vectorh(i)=-99999) . compute vectorp(i)=-99999 . end if . end loop . loop i=1 to ncol(vectorh) . do if (vectorh(i)=-99999) . compute vectorul(i)=-99999. end if . end loop . loop i=1 to ncol(vectorh) . do if (vectorh(i)=-99999) . compute vectorll(i)=-99999 . end if . end loop . compute ikstat={vectorh;vectorse;vectorz;vectorp;vectorll;vectorul} . save ikstat /outfile='ikstat1.sav' . *This section saves the data and prepares the reporting formats. save {k,n,r,pa,ck,ase,z,sig,ckll,ckul,fse,fsez,fsesig,fseu,fsel} /outfile='k__tmp2.sav' /variables=k,n,r,pa,ck,ase,z,sig,ckll,ckul,fse,fsez,fsesig,fseu,fsel . end matrix . get file='k__tmp2.sav'. formats k (f8.0) /n (f8.0) /r (f8.0) /pa (f8.4) /ck (f8.4) /ase (f8.4) /z (f8.4) /sig (f8.4) /ckul (f8.4) /ckll (f8.4) /fse(f8.4) /fsez (f8.4) /fsesig (f8.4) /fseu (f8.4) /fsel (f8.4) . variable labels k 'Number of Raters' /n 'Number of Items' /r 'Number of Categories' /pa 'Percent of Rater Agreement' /ck 'Kappa' /ase 'Standard Error' /z 'z'/sig 'p' /ckul 'Upper 95% Confidence Limit' /ckll 'Lower 95% Confidence Limit' /fse 'Fleiss SE' /fsez 'z' /fsesig 'p' /fseu 'Upper 95% Confidence Limit' /fsel 'Lower 95% Confidence Limit' . report format=list automatic align(center) /variables=k,n,r,pa /title "Basic Information" . report format=list automatic align(center) /variables=ck ase z sig ckll ckul /title "Cohens Kappa". report format=list automatic align(center) /variables=fse fsez fsesig fsel fseu /title "Cohens Kappa -- Fleiss Adjusted Standard Error" . get file='ikstat1.sav' . flip . delete variable case_lbl . compute n1=$casenum . formats n1 (f8.0) /var001 (f8.4) /var002 (f8.4) /var003 (f8.4) /var004 (f8.4) /var005 (f8.4) /var006 (f8.4) . variable labels n1 'Coding Category' /var001 'k' /var002 'Standard Error' /var003 'z' /var004 'p' /var005 'Lower 95% Confidence Limit' /var006 'Upper 95% Confidence Limit' . save outfile='ikstat.sav' . report format=list automatic align(center) /variables=n1 var001 var002 var003 var004 var005 var006 /Title "Individual Category Statistics" . !enddefine. restore. COHENSK VARS = rater1 to rater9 . -- View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Fleiss-Kappa-Macro-for-SPSS-tp4994433p5726711.html Sent from the SPSSX Discussion mailing list archive at Nabble.com. ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD |
«
Return to SPSSX Discussion
|
1 view|%1 views
| Free forum by Nabble | Edit this page |