SPSS-FLEISS KAPPA

[Sai]

Hi

Im in urgent need of help for my final year project studying pharmacy. I have 67 raters and need to analyze using fleiss kappa.

please can someone direct me on how to do this? do i need to convert my data file to a syntax file? how would i go about on doing this.

all 67 raters have rated a consultation which has 5 sections, rated from 1-5.

and how do i use 5 x 5 matrix also. my tutor has thrown all these things at me to do but i have no idea where to even begin?

please please any help will be greatly appreciated.

thanks
Sirah x

[bdates]

Sirah,

Can you give us an idea of the form of the data?  Is it 5 items?  Are
the sections distinct from each other? In that case you may have to rate
agreement across the items for each section.  I have syntax for Fleiss'
kappa for any number of raters and categories which uses an SPSS data
file, but the actual analysis will depend on the data format.

Brian

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of
Sai
Sent: Monday, March 14, 2011 5:03 PM
To: [hidden email]
Subject: [NEWSENDER] - SPSS-FLEISS KAPPA - Message is from an unknown
sender

Hi

Im in urgent need of help for my final year project studying pharmacy. I
have 67 raters and need to analyze using fleiss kappa.

please can someone direct me on how to do this? do i need to convert my
data
file to a syntax file? how would i go about on doing this.

all 67 raters have rated a consultation which has 5 sections, rated from
1-5.

and how do i use 5 x 5 matrix also. my tutor has thrown all these things
at
me to do but i have no idea where to even begin?

please please any help will be greatly appreciated.

thanks
Sirah x

--
View this message in context:
http://spssx-discussion.1045642.n5.nabble.com/SPSS-FLEISS-KAPPA-tp361141
2p3611412.html
Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Sai]

Hi Brian,

Firstly thank you so much for your reply, I am really stuck with this Fleiss Kappa calculation. Basically I am trying to calculate the inter-rater reliability of 67 raters who all watched a video of a consultation between a patient and pharmacist and rated each stage of the consultation. i.e. introduction, data collection/problem ID, action/solution, closure and consultation behavior. the data has been collected over 6 years.

My dissertation tutor has no idea how to analyze the inter-rater reliability and expects me to teach her the full steps on Monday, I have no idea how to convert my sav.file data into a syntax file and how to calculate fleiss kappa from crosstabs.

please could you explain this to me in detail, or could you provide me with your telephone number or skype details so I could call you and you could go through this with me. I would greatly appreciate your help.

I have attached my original file and the one I had to create for cohen's kappa analysis. I'm not sure which one is suitable.

also I have attached my project so far for you to get a better understanding of the project title.

kind regards,

Sirah.

On Tue, Mar 15, 2011 at 12:31 PM, Dates, Brian [via SPSSX Discussion] <[hidden email]> wrote:

Sirah,

Can you give us an idea of the form of the data?  Is it 5 items?  Are
the sections distinct from each other? In that case you may have to rate
agreement across the items for each section.  I have syntax for Fleiss'
kappa for any number of raters and categories which uses an SPSS data
file, but the actual analysis will depend on the data format.

Brian

-----Original Message-----

From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of
Sai
Sent: Monday, March 14, 2011 5:03 PM

To: [hidden email]
Subject: [NEWSENDER] - SPSS-FLEISS KAPPA - Message is from an unknown
sender

Hi

Im in urgent need of help for my final year project studying pharmacy. I
have 67 raters and need to analyze using fleiss kappa.

please can someone direct me on how to do this? do i need to convert my
data
file to a syntax file? how would i go about on doing this.

all 67 raters have rated a consultation which has 5 sections, rated from
1-5.

and how do i use 5 x 5 matrix also. my tutor has thrown all these things
at
me to do but i have no idea where to even begin?

please please any help will be greatly appreciated.

thanks
Sirah x

--
View this message in context:
http://spssx-discussion.1045642.n5.nabble.com/SPSS-FLEISS-KAPPA-tp361141
2p3611412.html
Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to

[hidden email] (not to SPSSX-L), with no body text except the

command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

=====================

To manage your subscription to SPSSX-L, send a message to

[hidden email] (not to SPSSX-L), with no body text except the

command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

---

If you reply to this email, your message will be added to the discussion below:

http://spssx-discussion.1045642.n5.nabble.com/SPSS-FLEISS-KAPPA-tp3611412p3702614.html

To unsubscribe from SPSS-FLEISS KAPPA, click here.

[Art Kendall]

Let's take this a step at a time.
With SPSS there are 3 kinds of files.
A syntax file (*.sps) is a set of instructions to SPSS telling it what you want it to do.
A saved system file (*.sav) is the data you want to work on.� It contains the data itself as well as the definitions of what it contains.
A listing file (*.spv) is the file meant to be read by people.� It can be seen on your monitor, printed, cut-and-pasted from, etc.

You do not need to turn Your .sav file into a syntax file.� You need to develop/write/compose a syntax file to use in conjunction with your .sav file

I'll be out of town for a few days starting tomorrow, but this is an open list so others can take you on the other steps, such as how to rearrange your data if necessary and what kind of inter rater/coder/judge reliability coefficients to use.

First, let's make sure the .sav file is ready.
Save the existing file as it is.�
Make a working copy� as working.sav.

Open working.sav. In the variable view:
Make sure that every record/case/line has a unique identifier.
Make sure that every variable has a mnemonic name and a variable label.
Make sure that there are value labels for for all valid and missing values that your variables can take.
Make sure that missing values are defined as such.

hint: Use <help> or get back to the list on how to get labels etc. into the .sav file if there are many repairs to be made.

If and only if the data does not contain confidential information, send me a copy of working.sav.

What version of SPSS do you have?

What country are you located in? If you are in the US/Canada we might be able to talk on the phone. However, for the sake of the list let's stay on the list for now.

How was your data recorded? Paper forms? etc?
How was your data entered?� Was it proofread or double keyed?

Who created the .sav file?
How many codings/ratings did each rater do?
What was the response scale the ratings were done on? (I.e., what are the legitimate value labels?)


Art Kendall
Social Research Consultants




On 3/15/2011 2:59 PM, Sai wrote:

Hi Brian,

Firstly thank you so much for your reply, I am really stuck with this Fleiss Kappa calculation. Basically I am trying to calculate the inter-rater reliability of 67 raters who all watched a video of a consultation between a patient and pharmacist and rated each stage of the consultation. i.e. introduction, data collection/problem ID, action/solution, closure and consultation behavior. the data has been collected over 6 years.

My dissertation tutor has no idea how to analyze the inter-rater reliability and expects me to teach her the full steps on Monday, I have no idea how to convert my sav.file data into a syntax file and how to calculate fleiss kappa from crosstabs.

please could you explain this to me in detail, or could you provide me with your telephone number or skype details so I could call you and you could go through this with me. I would greatly appreciate your help.

I have attached my original file and the one I had to create for cohen's kappa analysis. I'm not sure which one is suitable.

also I have attached my project so far for you to get a better understanding of the project title.

kind regards,

Sirah.

On Tue, Mar 15, 2011 at 12:31 PM, Dates, Brian [via SPSSX Discussion] <[hidden email]> wrote:

Sirah,

Can you give us an idea of the form of the data? � Is it 5 items? � Are
the sections distinct from each other? In that case you may have to rate
agreement across the items for each section. � I have syntax for Fleiss'
kappa for any number of raters and categories which uses an SPSS data
file, but the actual analysis will depend on the data format.

Brian

-----Original Message-----

From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of
Sai
Sent: Monday, March 14, 2011 5:03 PM

To: [hidden email]
Subject: [NEWSENDER] - SPSS-FLEISS KAPPA - Message is from an unknown
sender

Hi

Im in urgent need of help for my final year project studying pharmacy. I
have 67 raters and need to analyze using fleiss kappa.

please can someone direct me on how to do this? do i need to convert my
data
file to a syntax file? how would i go about on doing this.

all 67 raters have rated a consultation which has 5 sections, rated from
1-5.

and how do i use 5 x 5 matrix also. my tutor has thrown all these things
at
me to do but i have no idea where to even begin?

please please any help will be greatly appreciated.

thanks
Sirah x

--
View this message in context:
http://spssx-discussion.1045642.n5.nabble.com/SPSS-FLEISS-KAPPA-tp361141
2p3611412.html
Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to

[hidden email] (not to SPSSX-L), with no body text except the

command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

=====================

To manage your subscription to SPSSX-L, send a message to

[hidden email] (not to SPSSX-L), with no body text except the

command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

---

If you reply to this email, your message will be added to the discussion below:

http://spssx-discussion.1045642.n5.nabble.com/SPSS-FLEISS-KAPPA-tp3611412p3702614.html

PH4208-SPSS transposed.sav (17K) Download Attachment
Cohen's Kappa transposed. sav.sav (8K) Download Attachment
Research Project.doc (166K) Download Attachment

---

View this message in context: Re: [NEWSENDER] - SPSS-FLEISS KAPPA - Message is from an unknown sender
Sent from the SPSSX Discussion mailing list archive at Nabble.com.

===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD

[Art Kendall]

I am posting this to the list so others have the ability to help you if I did not interpret things correctly.

I looked at your data file.  Please check my understanding.
It appears that you have 67 respondents who rated a single video. 
Ratings from 1 to 4 were assigned to 5 sections of the video.

It appears that rather than being interested in inter-rater reliability (where raters are the variables and things rated are the cases) that you are interested how well the ratings can be used in a summative scale, i.e., internal consistency of the set of ratings across people. 

It appears that you ultimately want to relate the overall score assigned to the characteristics of the respondents doing the ratings.

Aside.  In future research, it would be preferable to use the same response scale that you have for the overall scale score.  As a rule of thumb, using more categories enables detecting relations/effects, i.e., use as many values as your respondents can easily deal with. Here I would have suggested 5 to 7 values.

What to do:

Double check about the person who gave a zero rating.  Check the data gathering instrument.  Does it appear (s)he might have intended "I don't Know"?

Clean up your data file.  Make the value labels communicative. Get rid of the blank case.

When that is done and _if my understanding is correct_, run the syntax below.  See if excluding the case where the rater did not follow instructions makes a meaningful difference.

Hope this helps. I'll be back in town in a few days.

Art Kendall
Social Research Consultants


set unicode yes.
GET FILE "C:\project\PH4208-SPSS transposed.sav".
compute original_order = $casenum.
do if any(1,introduction to Consultation_Behaviour).
print / original_order Introduction to Overall.
end if.
execute.
FREQUENCIES VARIABLES=Introduction DataCollection_ProblemID Actions_Solutions Closure
    Consultation_Behaviour Overall
  /ORDER=ANALYSIS.

RELIABILITY
  /VARIABLES=Introduction DataCollection_ProblemID Actions_Solutions Closure Consultation_Behaviour
  /SCALE('competence') ALL
  /MODEL=ALPHA
  /STATISTICS=DESCRIPTIVE SCALE CORR
  /SUMMARY=TOTAL CORR
  /ICC=MODEL(MIXED) TYPE(CONSISTENCY) CIN=95 TESTVAL=0.

RELIABILITY
  /VARIABLES=Introduction DataCollection_ProblemID Actions_Solutions Closure Consultation_Behaviour
  /SCALE('competence') ALL
  /MODEL=ALPHA
  /STATISTICS=CORR
  /ICC=MODEL(MIXED) TYPE(ABSOLUTE) CIN=95 TESTVAL=0.


On 3/15/2011 5:52 PM, Sirah Asif wrote:

Hi,

You do not need to turn Your .sav file into a syntax file.  You need to develop/write/compose a syntax file to use in conjunction with your .sav file

Thank you for that as I thought i needed to convert to a syntax file in order to run and analyze results. 

I'll be out of town for a few days starting tomorrow, but this is an open list so others can take you on the other steps, such as how to rearrange your data if necessary and what kind of inter rater/coder/judge reliability coefficients to use.

I would appreciate any help, as you can see my knowledge of spss is not very good and really desperately need some guidance.  originally my tutor told me to analyse using cohen's kappa but cohen's kappa only does inter-rater reliability for 2 raters and i have results for 67 raters.

I have attached my data file for you to look at.

What version of SPSS do you have? SPSS 18.0

What country are you located in?

 

Cardiff, UK

 

If you are in the US/Canada we might be able to talk on the phone. However, for the sake of the list let's stay on the list for now.

How was your data recorded? Paper forms? etc?
How was your data entered?  Was it proofread or double keyed?

my data was collected by my tutor over a 6 year period, when she teaches a postgrad course and a part of that is where the students watch a consultation video and rate each section (5sections) and rate between 1-4 with a total value out of 20. but as one person rated 1 section a 0 (zero) i had to tarnspose all values by 1 in order to eliminate the 0. 

Who created the .sav file? I did, just trial and error.
How many codings/ratings did each rater do? 5
What was the response scale the ratings were done on? (I.e., what are the legitimate value labels?) 1-4 with 1 being not competent and 4 being competent. however i increased all values by 1 so 1-5 now as one person rated a zero.


Art Kendall
Social Research Consultants

On 3/15/2011 2:59 PM, Sai wrote:

Hi Brian,

Firstly thank you so much for your reply, I am really stuck with this Fleiss Kappa calculation. Basically I am trying to calculate the inter-rater reliability of 67 raters who all watched a video of a consultation between a patient and pharmacist and rated each stage of the consultation. i.e. introduction, data collection/problem ID, action/solution, closure and consultation behavior. the data has been collected over 6 years.

My dissertation tutor has no idea how to analyze the inter-rater reliability and expects me to teach her the full steps on Monday, I have no idea how to convert my sav.file data into a syntax file and how to calculate fleiss kappa from crosstabs.

please could you explain this to me in detail, or could you provide me with your telephone number or skype details so I could call you and you could go through this with me. I would greatly appreciate your help.

I have attached my original file and the one I had to create for cohen's kappa analysis. I'm not sure which one is suitable.

also I have attached my project so far for you to get a better understanding of the project title.

kind regards,

Sirah.

On Tue, Mar 15, 2011 at 12:31 PM, Dates, Brian [via SPSSX Discussion] <[hidden email]> wrote:

Sirah,

Can you give us an idea of the form of the data?  Is it 5 items?  Are
the sections distinct from each other? In that case you may have to rate
agreement across the items for each section.  I have syntax for Fleiss'
kappa for any number of raters and categories which uses an SPSS data
file, but the actual analysis will depend on the data format.

Brian

-----Original Message-----

From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of
Sai
Sent: Monday, March 14, 2011 5:03 PM

To: [hidden email]
Subject: [NEWSENDER] - SPSS-FLEISS KAPPA - Message is from an unknown
sender

Hi

Im in urgent need of help for my final year project studying pharmacy. I
have 67 raters and need to analyze using fleiss kappa.

please can someone direct me on how to do this? do i need to convert my
data
file to a syntax file? how would i go about on doing this.

all 67 raters have rated a consultation which has 5 sections, rated from
1-5.

and how do i use 5 x 5 matrix also. my tutor has thrown all these things
at
me to do but i have no idea where to even begin?

please please any help will be greatly appreciated.

thanks
Sirah x

--
View this message in context:
http://spssx-discussion.1045642.n5.nabble.com/SPSS-FLEISS-KAPPA-tp361141
2p3611412.html
Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to

[hidden email] (not to SPSSX-L), with no body text except the

command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

=====================

To manage your subscription to SPSSX-L, send a message to

[hidden email] (not to SPSSX-L), with no body text except the

command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

---

If you reply to this email, your message will be added to the discussion below:

http://spssx-discussion.1045642.n5.nabble.com/SPSS-FLEISS-KAPPA-tp3611412p3702614.html

PH4208-SPSS transposed.sav (17K) Download Attachment
Cohen's Kappa transposed. sav.sav (8K) Download Attachment
Research Project.doc (166K) Download Attachment

---

View this message in context: Re: [NEWSENDER] - SPSS-FLEISS KAPPA - Message is from an unknown sender

Sent from the SPSSX Discussion mailing list archive at Nabble.com.

===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD

[gastonshack]

Hello,

I have question along similar lines. I have read through this post However, I still am not completely sure how to calculate my results and I am hoping somebody might be able to help me out.

My goal is to show that a rating system for looking at sinuses has good inter rater reliability. Each rater was given a video to watch of four different subsites of the paranasal sinuses (Ie, the frontal, maxillary, ethmoidal and sphenoidal). Each one of these subsites were assessed for 9 different findings (clear mucus, eos mucus, crust, pus, polypoid, polyps, cystic polyp, mucoid and edema). For each finding, the rater would assign a 0 (absent) or 1 (present) to each subsite. So a patient can then receive a score in more than one category. Thus if a patient had mucoid, polypoid and crust findings in the ethmoid cavity, the rater would mark a 1 next to these findings on their score sheet. In summary:

Number of subjects = 20
Number of raters = 4
Number of subsites to look at 4
Number of characteristics: 9 (not necessarily distinct)

My questions are:

1. Is it possible to use the Fleiss Kappa if the rater is allowed to mark a subject in more than one category
2. If not, is there a test that I can use that takes this into account?
3. If this is not possible, do you have a suggestion for how I can best rearrange my data? (The goal is to show that different raters can reliably score sinuses using our method)

Thank you in advance for any helep you can offer,

Mike