Help for a nonparametric problem

[Xu, Lihua]

Hey all,
 
I have a data set in which I am trying to compare whether the rookie's rating is the same as the expert's rating. According to expert's rating, subjects' oral presentation was classified into three groups in terms of proficiency: intermediate, advanced and superior. After experts' rating, rookies were asked to rate those subjects. For each level, "agree" means rookies' rating is the same as the experts and "disagree" means otherwise.
For the intermediate level, 153 agrees and 27 disagrees. For the advanced level, 112 agrees and 68 disagrees. For the superior level, 129 agrees and 51 disagrees. The contigency table I set up is as follows:
 
                                Intermediate(1)         Advanced (2)           Superior (3)
Agree (1)                         153 (1,1)                   112 (1,2)                 129 (1,3)
Disagree (2)                     27 (2,1)                      68 (2,2)                     51 (2,3)
 
I set up the data file in SPSS with two columns: the first column is Level (1=intermediate, 2=advanced, 3=superior) and the second column is Rating (1=agree and 2=disagree). The process of setting up the data file is tedious since I didn't know how to use syntax to do it. But anyway, I tried to run the Chi-Square test with it. However, the output doesn't seem right. So I wonder where I was wrong, the way I set up the contigency table or the nonparametric statistics I chose to run or else.
 
Any enlightenment would be greatly appreciate.
 
Lihua Xu
Doctoral Student
Research, Evaluation, Method and Statistics
College of Education
218 Willard Hall
Oklahoma State University
[phone]

[Spousta Jan]

Hi Lihua Xu,

It looks like you should start with a thought about what you really want to know from your data (it is better done _before_ you start to collect data, by the way). When you ask "whether the rookie's rating is the same as the expert's rating" then the answer is clear (the ratings are obviously not the same because there are cases where both raters disagree) but it gives you no useful information.

Please try to formulate the question better - without a good question there is no good answer.

Regards

Jan

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Xu, Lihua
Sent: Tuesday, July 24, 2007 3:27 AM
To: [hidden email]
Subject: Help for a nonparametric problem

Hey all,

I have a data set in which I am trying to compare whether the rookie's rating is the same as the expert's rating. According to expert's rating, subjects' oral presentation was classified into three groups in terms of proficiency: intermediate, advanced and superior. After experts' rating, rookies were asked to rate those subjects. For each level, "agree" means rookies' rating is the same as the experts and "disagree" means otherwise.
For the intermediate level, 153 agrees and 27 disagrees. For the advanced level, 112 agrees and 68 disagrees. For the superior level, 129 agrees and 51 disagrees. The contigency table I set up is as follows:

                                Intermediate(1)         Advanced (2)           Superior (3)
Agree (1)                         153 (1,1)                   112 (1,2)                 129 (1,3)
Disagree (2)                     27 (2,1)                      68 (2,2)                     51 (2,3)

I set up the data file in SPSS with two columns: the first column is Level (1=intermediate, 2=advanced, 3=superior) and the second column is Rating (1=agree and 2=disagree). The process of setting up the data file is tedious since I didn't know how to use syntax to do it. But anyway, I tried to run the Chi-Square test with it. However, the output doesn't seem right. So I wonder where I was wrong, the way I set up the contigency table or the nonparametric statistics I chose to run or else.

Any enlightenment would be greatly appreciate.

Lihua Xu
Doctoral Student
Research, Evaluation, Method and Statistics College of Education
218 Willard Hall
Oklahoma State University
[phone]



_____

Tato zpráva a všechny připojené soubory jsou důvěrné a určené výlučně adresátovi(-ům). Jestliže nejste oprávněným adresátem, je zakázáno jakékoliv zveřejňování, zprostředkování nebo jiné použití těchto informací. Jestliže jste tento mail dostali neoprávněně, prosím, uvědomte odesilatele a smažte zprávu i přiložené soubory. Odesilatel nezodpovídá za jakékoliv chyby nebo opomenutí způsobené tímto přenosem.

This message and any attached files are confidential and intended solely for the addressee(s). Any publication, transmission or other use of the information by a person or entity other than the intended addressee is prohibited. If you receive this in error please contact the sender and delete the message as well as all attached documents. The sender does not accept liability for any errors or omissions as a result of the transmission.

-.- --

[Swank, Paul R]

I agree with Jan and suggest you also think about this classification of the ratings as agree or disagree. Certainly the error of classifying a Superior as an Advanced is better than classifying it as an Intermediate. Your classification does not take this into account.

Paul R. Swank, Ph.D. Professor
Director of Reseach
Children's Learning Institute
University of Texas Health Science Center-Houston

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Spousta Jan
Sent: Tuesday, July 24, 2007 3:46 AM
To: [hidden email]
Subject: Re: Help for a nonparametric problem

Hi Lihua Xu,

It looks like you should start with a thought about what you really want to know from your data (it is better done _before_ you start to collect data, by the way). When you ask "whether the rookie's rating is the same as the expert's rating" then the answer is clear (the ratings are obviously not the same because there are cases where both raters disagree) but it gives you no useful information.

Please try to formulate the question better - without a good question there is no good answer.

Regards

Jan

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Xu, Lihua
Sent: Tuesday, July 24, 2007 3:27 AM
To: [hidden email]
Subject: Help for a nonparametric problem

Hey all,

I have a data set in which I am trying to compare whether the rookie's rating is the same as the expert's rating. According to expert's rating, subjects' oral presentation was classified into three groups in terms of proficiency: intermediate, advanced and superior. After experts' rating, rookies were asked to rate those subjects. For each level, "agree" means rookies' rating is the same as the experts and "disagree" means otherwise.
For the intermediate level, 153 agrees and 27 disagrees. For the advanced level, 112 agrees and 68 disagrees. For the superior level, 129 agrees and 51 disagrees. The contigency table I set up is as follows:

                                Intermediate(1)         Advanced (2)           Superior (3)
Agree (1)                         153 (1,1)                   112 (1,2)                 129 (1,3)
Disagree (2)                     27 (2,1)                      68 (2,2)                     51 (2,3)

I set up the data file in SPSS with two columns: the first column is Level (1=intermediate, 2=advanced, 3=superior) and the second column is Rating (1=agree and 2=disagree). The process of setting up the data file is tedious since I didn't know how to use syntax to do it. But anyway, I tried to run the Chi-Square test with it. However, the output doesn't seem right. So I wonder where I was wrong, the way I set up the contigency table or the nonparametric statistics I chose to run or else.

Any enlightenment would be greatly appreciate.

Lihua Xu
Doctoral Student
Research, Evaluation, Method and Statistics College of Education
218 Willard Hall
Oklahoma State University
[phone]



_____

Tato zpráva a všechny připojené soubory jsou důvěrné a určené výlučně adresátovi(-ům). Jestliže nejste oprávněným adresátem, je zakázáno jakékoliv zveřejňování, zprostředkování nebo jiné použití těchto informací. Jestliže jste tento mail dostali neoprávněně, prosím, uvědomte odesilatele a smažte zprávu i přiložené soubory. Odesilatel nezodpovídá za jakékoliv chyby nebo opomenutí způsobené tímto přenosem.

This message and any attached files are confidential and intended solely for the addressee(s). Any publication, transmission or other use of the information by a person or entity other than the intended addressee is prohibited. If you receive this in error please contact the sender and delete the message as well as all attached documents. The sender does not accept liability for any errors or omissions as a result of the transmission.

-.- --

[ViAnn Beadle]

First, a disclaimer: I claim no specific expertise for this domain.

It seems to me that the starting approach here is to compare ratings in the
most direct way by structuring the data as two columns: expert's rating, and
rookie's rating with one row for each subject rated. The beginning analysis
would be a crosstab of expert's rating by rookie's rating. This table is
akin to a standard classification table in which expert's rating is a proxy
for predicted rating and rookie's rating is a proxy for observed rating. In
a standard classification table, the off-diagonal cells are the
misclassifications. Now this is not really a standard classification table
since you are using proxies, but is a quick way to test your hypotheses (you
must have some, right?). Then, I would proceed to a marginal homogeneity
test which is available using the NPAR TESTS MH subcommand to see if the
off-diagonal counts are significant.

I'd also look at the inter-rater consistency literature for more guidance.

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of
Swank, Paul R
Sent: Tuesday, July 24, 2007 8:57 AM
To: [hidden email]
Subject: Re: Help for a nonparametric problem

I agree with Jan and suggest you also think about this classification of the
ratings as agree or disagree. Certainly the error of classifying a Superior
as an Advanced is better than classifying it as an Intermediate. Your
classification does not take this into account.

Paul R. Swank, Ph.D. Professor
Director of Reseach
Children's Learning Institute
University of Texas Health Science Center-Houston

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of
Spousta Jan
Sent: Tuesday, July 24, 2007 3:46 AM
To: [hidden email]
Subject: Re: Help for a nonparametric problem

Hi Lihua Xu,

It looks like you should start with a thought about what you really want to
know from your data (it is better done _before_ you start to collect data,
by the way). When you ask "whether the rookie's rating is the same as the
expert's rating" then the answer is clear (the ratings are obviously not the
same because there are cases where both raters disagree) but it gives you no
useful information.

Please try to formulate the question better - without a good question there
is no good answer.

Regards

Jan

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Xu,
Lihua
Sent: Tuesday, July 24, 2007 3:27 AM
To: [hidden email]
Subject: Help for a nonparametric problem

Hey all,

I have a data set in which I am trying to compare whether the rookie's
rating is the same as the expert's rating. According to expert's rating,
subjects' oral presentation was classified into three groups in terms of
proficiency: intermediate, advanced and superior. After experts' rating,
rookies were asked to rate those subjects. For each level, "agree" means
rookies' rating is the same as the experts and "disagree" means otherwise.
For the intermediate level, 153 agrees and 27 disagrees. For the advanced
level, 112 agrees and 68 disagrees. For the superior level, 129 agrees and
51 disagrees. The contigency table I set up is as follows:

                                Intermediate(1)         Advanced (2)
Superior (3)
Agree (1)                         153 (1,1)                   112 (1,2)
129 (1,3)
Disagree (2)                     27 (2,1)                      68 (2,2)
51 (2,3)

I set up the data file in SPSS with two columns: the first column is Level
(1=intermediate, 2=advanced, 3=superior) and the second column is Rating
(1=agree and 2=disagree). The process of setting up the data file is tedious
since I didn't know how to use syntax to do it. But anyway, I tried to run
the Chi-Square test with it. However, the output doesn't seem right. So I
wonder where I was wrong, the way I set up the contigency table or the
nonparametric statistics I chose to run or else.

Any enlightenment would be greatly appreciate.

Lihua Xu
Doctoral Student
Research, Evaluation, Method and Statistics College of Education
218 Willard Hall
Oklahoma State University
[phone]



_____

Tato zpráva a všechny připojené soubory jsou důvěrné a určené výlučně
adresátovi(-ům). Jestliže nejste oprávněným adresátem, je zakázáno jakékoliv
zveřejňování, zprostředkování nebo jiné použití těchto informací. Jestliže
jste tento mail dostali neoprávněně, prosím, uvědomte odesilatele a smažte
zprávu i přiložené soubory. Odesilatel nezodpovídá za jakékoliv chyby nebo
opomenutí způsobené tímto přenosem.

This message and any attached files are confidential and intended solely for
the addressee(s). Any publication, transmission or other use of the
information by a person or entity other than the intended addressee is
prohibited. If you receive this in error please contact the sender and
delete the message as well as all attached documents. The sender does not
accept liability for any errors or omissions as a result of the
transmission.

-.- --

[Julius Sim]

Yes, it's surely important to take account of the magnitude of
disagreement, and also the extent of agreement that might be expected
purely by chance. This therefore looks like a good candidate for a
weighted kappa. I don't think you can do a weighted kappa directly in
SPSS, but you could weight the data first and then use the kappa statistic
available under crosstabs.

Using linear weights, an agreement cell would be weighted as 1, a cell
representing disagreement by one category would be weighted as .5, and a
cell representing disagreement by two categories would be weighted as 0.
For quadratic weights, these would be 1, .75 and 0 respectively.

This assumes that you only wish to reflect the magnitude of disagreement,
not the direction of agreement. That is to say, you don't want to treat
the case where the rookie's rating is one category below that of the
expert any differently from the case where it is one category above that
of the expert - but you do want to distinguish a disagreement by one
category from a disagreement by two categories.

Regards.

Julius Sim

> I agree with Jan and suggest you also think about this classification of
> the ratings as agree or disagree. Certainly the error of classifying a
> Superior as an Advanced is better than classifying it as an Intermediate.
> Your classification does not take this into account.
>
> Paul R. Swank, Ph.D. Professor
> Director of Reseach
> Children's Learning Institute
> University of Texas Health Science Center-Houston
>
> -----Original Message-----
> From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of
> Spousta Jan
> Sent: Tuesday, July 24, 2007 3:46 AM
> To: [hidden email]
> Subject: Re: Help for a nonparametric problem
>
> Hi Lihua Xu,
>
> It looks like you should start with a thought about what you really want
> to know from your data (it is better done _before_ you start to collect
> data, by the way). When you ask "whether the rookie's rating is the same
> as the expert's rating" then the answer is clear (the ratings are
> obviously not the same because there are cases where both raters disagree)
> but it gives you no useful information.
>
> Please try to formulate the question better - without a good question
> there is no good answer.
>
> Regards
>
> Jan
>
> -----Original Message-----
> From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of
> Xu, Lihua
> Sent: Tuesday, July 24, 2007 3:27 AM
> To: [hidden email]
> Subject: Help for a nonparametric problem
>
> Hey all,
>
> I have a data set in which I am trying to compare whether the rookie's
> rating is the same as the expert's rating. According to expert's rating,
> subjects' oral presentation was classified into three groups in terms of
> proficiency: intermediate, advanced and superior. After experts' rating,
> rookies were asked to rate those subjects. For each level, "agree" means
> rookies' rating is the same as the experts and "disagree" means otherwise.
> For the intermediate level, 153 agrees and 27 disagrees. For the advanced
> level, 112 agrees and 68 disagrees. For the superior level, 129 agrees and
> 51 disagrees. The contigency table I set up is as follows:
>
>                                 Intermediate(1)         Advanced (2)
>     Superior (3)
> Agree (1)                         153 (1,1)                   112 (1,2)
>              129 (1,3)
> Disagree (2)                     27 (2,1)                      68 (2,2)
>                  51 (2,3)
>
> I set up the data file in SPSS with two columns: the first column is Level
> (1=intermediate, 2=advanced, 3=superior) and the second column is Rating
> (1=agree and 2=disagree). The process of setting up the data file is
> tedious since I didn't know how to use syntax to do it. But anyway, I
> tried to run the Chi-Square test with it. However, the output doesn't seem
> right. So I wonder where I was wrong, the way I set up the contigency
> table or the nonparametric statistics I chose to run or else.
>
> Any enlightenment would be greatly appreciate.
>
> Lihua Xu
> Doctoral Student
> Research, Evaluation, Method and Statistics College of Education
> 218 Willard Hall
> Oklahoma State University
> [phone]
>
>
>
> _____
>
> Tato zpráva a v¹echny pøipojené soubory jsou dùvìrné a urèené výluènì
> adresátovi(-ùm). JestliŸe nejste oprávnìným adresátem, je zakázáno
> jakékoliv zveøejòování, zprostøedkování nebo jiné pouŸití tìchto
> informací. JestliŸe jste tento mail dostali neoprávnìnì, prosím, uvìdomte
> odesilatele a smaŸte zprávu i pøiloŸené soubory. Odesilatel nezodpovídá za
> jakékoliv chyby nebo opomenutí zpùsobené tímto pøenosem.
>
> This message and any attached files are confidential and intended solely
> for the addressee(s). Any publication, transmission or other use of the
> information by a person or entity other than the intended addressee is
> prohibited. If you receive this in error please contact the sender and
> delete the message as well as all attached documents. The sender does not
> accept liability for any errors or omissions as a result of the
> transmission.
>
> -.- --
>