Kappa question
Locked
 
|
I have 10 minutes of mother-infant video. Two coders are classifying the mother’s behavior every 5 second interval into 6 mutually exclusive categories. I end up with two series of 120 codes each. I will treat them via Crosstabs/Kappa. I expect to get an agreement coefficient. There is no reason to reduce the categories down to two?
I’ve ran across terms “code drift” and “waves” a couple times. It seemed in the articles that waves were a method to deal with code drift. Can someone explain what this is about or point me to a reference. TIA. Stephen Salbod, Pace University, NYC |
|
|
Here are two reasons to reduce the categories to a set of 2x2 tables:
=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD
1) You want to know about each of the categories; and 2) a kappa from a table that is larger than 2x2 is mainly useful when you have a dozen of them, based on similar variables, so you can compare them to each other. They do not "compare" well when their marginal frequencies differ, so -- if you are going to use them at all -- you want a set of kappas in which you can point to the highest and lowest. I haven't run across the term "code drift", and Google's best offering, in the first page of hits, is a mystic poem about digital biology or some such. -- Rich Ulrich > Date: Sun, 5 Oct 2014 14:25:47 -0700 > From: [hidden email] > Subject: Kappa question > To: [hidden email] > > I have 10 minutes of mother-infant video. Two coders are classifying the > mother’s behavior every 5 second interval into 6 mutually exclusive > categories. I end up with two series of 120 codes each. I will treat them > via Crosstabs/Kappa. I expect to get an agreement coefficient. There is no > reason to reduce the categories down to two? > > I’ve ran across terms “code drift” and “waves” a couple times. It seemed in > the articles that waves were a method to deal with code drift. Can someone > explain what this is about or point me to a reference. > > TIA. > > Stephen Salbod, Pace University, NYC > > > > > -- > View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Kappa-question-tp5727497.html > Sent from the SPSSX Discussion mailing list archive at Nabble.com. > > ===================== > To manage your subscription to SPSSX-L, send a message to > [hidden email] (not to SPSSX-L), with no body text except the > command. To leave the list, send the command > SIGNOFF SPSSX-L > For a list of commands to manage subscriptions, send the command > INFO REFCARD |
|
|
In reply to this post by Salbod
I don't know that I've seen the phrase "code drift" but I think the reference might be to coder/observer drift. The idea would be that when presented with the same behavior, coders/observers categorize the behavior the same way. The absence of drift would be inferred when repeated presentations of the same interaction are coded the same way by a coder. I did a quick search on the phrase "code drift" in PsychInfo and got this cite
Artifact, bias, and complexity of assessment: The ABCs of reliability. Kazdin, Alan E.; Journal of Applied Behavior Analysis, Vol 10(1), Spr 1977. pp.141-150. A search on this string "(coder or observer) drift" turned up a number of cites that might be useful. Gene Maguin -----Original Message----- From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Salbod Sent: Sunday, October 05, 2014 5:26 PM To: [hidden email] Subject: Kappa question I have 10 minutes of mother-infant video. Two coders are classifying the mother’s behavior every 5 second interval into 6 mutually exclusive categories. I end up with two series of 120 codes each. I will treat them via Crosstabs/Kappa. I expect to get an agreement coefficient. There is no reason to reduce the categories down to two? I’ve ran across terms “code drift” and “waves” a couple times. It seemed in the articles that waves were a method to deal with code drift. Can someone explain what this is about or point me to a reference. TIA. Stephen Salbod, Pace University, NYC -- View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Kappa-question-tp5727497.html Sent from the SPSSX Discussion mailing list archive at Nabble.com. ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD |
|
|
I've encountered the term 'rater drift', which is pretty common lingo in inter-rater reliability studies. I'm presuming this is the same. Gene's statement about consistency and/or agreement among raters is the correct. The term, however, refers to what happens down the line. Typically, raters begin to accommodate to a rating system, and are not as discriminant as time goes by, particularly at the ends of the rating spectrum; thus continued inter-rater agreement/reliability checks are necessary to re-establish reliability, generally at least annually.
Brian -----Original Message----- From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Maguin, Eugene Sent: Monday, October 06, 2014 11:33 AM To: [hidden email] Subject: Re: Kappa question I don't know that I've seen the phrase "code drift" but I think the reference might be to coder/observer drift. The idea would be that when presented with the same behavior, coders/observers categorize the behavior the same way. The absence of drift would be inferred when repeated presentations of the same interaction are coded the same way by a coder. I did a quick search on the phrase "code drift" in PsychInfo and got this cite Artifact, bias, and complexity of assessment: The ABCs of reliability. Kazdin, Alan E.; Journal of Applied Behavior Analysis, Vol 10(1), Spr 1977. pp.141-150. A search on this string "(coder or observer) drift" turned up a number of cites that might be useful. Gene Maguin -----Original Message----- From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Salbod Sent: Sunday, October 05, 2014 5:26 PM To: [hidden email] Subject: Kappa question I have 10 minutes of mother-infant video. Two coders are classifying the mother’s behavior every 5 second interval into 6 mutually exclusive categories. I end up with two series of 120 codes each. I will treat them via Crosstabs/Kappa. I expect to get an agreement coefficient. There is no reason to reduce the categories down to two? I’ve ran across terms “code drift” and “waves” a couple times. It seemed in the articles that waves were a method to deal with code drift. Can someone explain what this is about or point me to a reference. TIA. Stephen Salbod, Pace University, NYC -- View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Kappa-question-tp5727497.html Sent from the SPSSX Discussion mailing list archive at Nabble.com. ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD |
|
|
In reply to this post by Maguin, Eugene
Hi Gene: Thanks for the reference. Yea, it is fatigue phenomenon. Breaking up the series into subsets might be a way to approach the problem. --Steve
-----Original Message----- From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Maguin, Eugene Sent: Monday, October 06, 2014 11:33 AM To: [hidden email] Subject: Re: Kappa question I don't know that I've seen the phrase "code drift" but I think the reference might be to coder/observer drift. The idea would be that when presented with the same behavior, coders/observers categorize the behavior the same way. The absence of drift would be inferred when repeated presentations of the same interaction are coded the same way by a coder. I did a quick search on the phrase "code drift" in PsychInfo and got this cite Artifact, bias, and complexity of assessment: The ABCs of reliability. Kazdin, Alan E.; Journal of Applied Behavior Analysis, Vol 10(1), Spr 1977. pp.141-150. A search on this string "(coder or observer) drift" turned up a number of cites that might be useful. Gene Maguin -----Original Message----- From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Salbod Sent: Sunday, October 05, 2014 5:26 PM To: [hidden email] Subject: Kappa question I have 10 minutes of mother-infant video. Two coders are classifying the mother’s behavior every 5 second interval into 6 mutually exclusive categories. I end up with two series of 120 codes each. I will treat them via Crosstabs/Kappa. I expect to get an agreement coefficient. There is no reason to reduce the categories down to two? I’ve ran across terms “code drift” and “waves” a couple times. It seemed in the articles that waves were a method to deal with code drift. Can someone explain what this is about or point me to a reference. TIA. Stephen Salbod, Pace University, NYC -- View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Kappa-question-tp5727497.html Sent from the SPSSX Discussion mailing list archive at Nabble.com. ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD |
|
|
In reply to this post by bdates
Thanks, Brian. I too had suspected it was a fatigue problem. Apparently, waves are a way to get estimates along the way. Kind of tip you off if there is a problem. Exactly, what you were mentioning.
Now, I'm working on the codebook with the coders and PI. I think I'm in for multiple codes. All the best, Steve -----Original Message----- From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Dates, Brian Sent: Tuesday, October 07, 2014 9:21 AM To: [hidden email] Subject: Re: Kappa question I've encountered the term 'rater drift', which is pretty common lingo in inter-rater reliability studies. I'm presuming this is the same. Gene's statement about consistency and/or agreement among raters is the correct. The term, however, refers to what happens down the line. Typically, raters begin to accommodate to a rating system, and are not as discriminant as time goes by, particularly at the ends of the rating spectrum; thus continued inter-rater agreement/reliability checks are necessary to re-establish reliability, generally at least annually. Brian -----Original Message----- From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Maguin, Eugene Sent: Monday, October 06, 2014 11:33 AM To: [hidden email] Subject: Re: Kappa question I don't know that I've seen the phrase "code drift" but I think the reference might be to coder/observer drift. The idea would be that when presented with the same behavior, coders/observers categorize the behavior the same way. The absence of drift would be inferred when repeated presentations of the same interaction are coded the same way by a coder. I did a quick search on the phrase "code drift" in PsychInfo and got this cite Artifact, bias, and complexity of assessment: The ABCs of reliability. Kazdin, Alan E.; Journal of Applied Behavior Analysis, Vol 10(1), Spr 1977. pp.141-150. A search on this string "(coder or observer) drift" turned up a number of cites that might be useful. Gene Maguin -----Original Message----- From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Salbod Sent: Sunday, October 05, 2014 5:26 PM To: [hidden email] Subject: Kappa question I have 10 minutes of mother-infant video. Two coders are classifying the mother’s behavior every 5 second interval into 6 mutually exclusive categories. I end up with two series of 120 codes each. I will treat them via Crosstabs/Kappa. I expect to get an agreement coefficient. There is no reason to reduce the categories down to two? I’ve ran across terms “code drift” and “waves” a couple times. It seemed in the articles that waves were a method to deal with code drift. Can someone explain what this is about or point me to a reference. TIA. Stephen Salbod, Pace University, NYC -- View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Kappa-question-tp5727497.html Sent from the SPSSX Discussion mailing list archive at Nabble.com. ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD |
«
Return to SPSSX Discussion
|
1 view|%1 views
| Free forum by Nabble | Edit this page |