Categorizing study participants based on their scores on summated scales (non-SPSS question).
Locked
Categorizing study participants based on their scores on summated scales (non-SPSS question).
 
|
Hi all,
The research textbooks I've read so far, do not
provide a guideline on how to categorize study participants into categories
like low, moderate and high, open to change, somewhat open to change
and not at all open to change, based on the scores they have obtained on the
summated scale.
To give more clarity to my question, I'll describe
the confusion that I'm facing in my own research on conservation behaviors.
Variables like attitudes towards conservation, awareness of environmental
problems and influence of others on conservation behaviors are all part of my
research. I've constructed scales to measure these variables. I'm having
confusion in categorizing participants into different categories based on
their scores. I'll use an example of a five question scale to
further clarify my question. I've constructed a scale of 5 questions,
It provides 4 standard Likert type responses, i.e. strongly disagree to
strongly agree. Since the scale has 5 questions the maximum score possible is 20
and 5 is the lowest possible score. But I'm unable to find any guideline on the
following things:
What criteria should I use other than personal
judgment to decide that on what is the score based on which participants can be
placed in high or low category. Should scores between 16 and 20 imply high,
scores 11 to 15 moderate and scores between 5 to 10 low? Of course the responses
are coded in a way that higher scores means that the conservation behavior and
conservation attitudes are high. I plan to use the scores on the scale to use in
regression analysis, and if assumptions of regression are not met, then I'd want
to use those scores to make categories and use test like chi-square.
Any suggestions and comments are most appreciated.
Thanks and regards,
Faiz.
|
Re: Categorizing study participants based on their scores on summated scales (non-SPSS question).
|
Faiz
First thing to do is check whether the items relate
to each other and more or less measure the same thing. Are all your items
positively worded or do ome need to be reversed? Taking your example and
assuming five vars v1 to v5, 1 = strongly disagree ~ ~ ~ 5 = strongly
agree you can run (much quicker in syntax):
file > new > syntax:
compute score = sum.5 (v1 to v5) - 5 .
*generates a score with a genuine zero point (ratio
scale) with a range of 0 to 15 .
freq score .
corr score v1 to v5 .
Technically you should use non-parametric stats
with ordinal vars, but pragmatically (we all do it) the above is
simpler.
You can tell a lot just by looking at the
correlation matrix: there may be a single underlying factor or possibly more
than one.
You can split your sample into Hi - Lo groups in
any way you like using quartiles, deciles, median etc. or just by looking at the
cumulative % column in the frequencies table.
You could check reliability using Cronbach's
alpha .
analyze > scale > reliability
analysis
There is a set of SPSS tutorials on simple scale
construction in section 3.5 of Block 3 on my website, but nothing (yet) on
reliability. Statisticians on the list will be able to advise you on
other aspects.
Get back to me off-list and I may be able to offer
more detailed help.
|
Re: Categorizing study participants based on their scores on summated scales (non-SPSS question).
Administrator
|
In reply to this post by Faiz Rasool
For the regression analysis, are your scales explanatory (predictor) variables or outcome (dependent) variables? Regarding assumptions for regression, the most important one is that the residuals be independent of (or uncorrelated with) the explanatory variables. If you are concerned about normality, note that it applies to the errors, not the outcome variable itself, and is not nearly as important as the assumption of independence. See this old post from sci.stat.edu, for example. http://groups.google.com/group/sci.stat.edu/msg/745a15f3122398cf?dmode=source HTH.
--
Bruce Weaver bweaver@lakeheadu.ca http://sites.google.com/a/lakeheadu.ca/bweaver/ "When all else fails, RTFM." PLEASE NOTE THE FOLLOWING: 1. My Hotmail account is not monitored regularly. To send me an e-mail, please use the address shown above. 2. The SPSSX Discussion forum on Nabble is no longer linked to the SPSSX-L listserv administered by UGA (https://listserv.uga.edu/). |
|
|
Faiz,
I have seen two approaches to this. 1) form groups in terms of whether they endorse the positive or negative side of the scale (i.e. strongly disagree + disagree and agree + strongly agree).
2. If you have multiple items and want to form what are essentially latent profiles you can try using cluster analysis. http://www.mvsolution.com/wp-content/uploads/SPSS-Tutorial-Cluster-Analysis.pdf
Good luck,
John
From: Bruce Weaver <[hidden email]> To: [hidden email] Sent: Mon, January 10, 2011 9:19:05 AM Subject: Re: Categorizing study participants based on their scores on summated scales (non-SPSS question). Faiz Rasool wrote: > > Hi all, > > The research textbooks I've read so far, do not provide a guideline on how > to categorize study participants into categories like low, moderate and > high, open to change, somewhat open to change and not at all open to > change, based on the scores they have obtained on the summated scale. > > To give more clarity to my question, I'll describe the confusion that I'm > facing in my own research on conservation behaviors. Variables like > attitudes towards conservation, awareness of environmental problems and > influence of others on conservation behaviors are all part of my research. > I've constructed scales to measure these variables. I'm having confusion > in categorizing participants into different categories based on their > scores. I'll use an example of a five question scale to further clarify > my question. I've constructed a scale of 5 questions, It provides 4 > standard Likert type responses, i.e. strongly disagree to strongly agree. > Since the scale has 5 questions the maximum score possible is 20 and 5 is > the lowest possible score. But I'm unable to find any guideline on the > following things: > > What criteria should I use other than personal judgment to decide that on > what is the score based on which participants can be placed in high or low > category. Should scores between 16 and 20 imply high, scores 11 to 15 > moderate and scores between 5 to 10 low? Of course the responses are coded > in a way that higher scores means that the conservation behavior and > conservation attitudes are high. I plan to use the scores on the scale to > use in regression analysis, and if assumptions of regression are not met, > then I'd want to use those scores to make categories and use test like > chi-square. > > Any suggestions and comments are most appreciated. > > Thanks and regards, > Faiz. > > For the regression analysis, are your scales explanatory (predictor) variables or outcome (dependent) variables? Regarding assumptions for regression, the most important one is that the residuals be independent of (or uncorrelated with) the explanatory variables. If you are concerned about normality, note that it applies to the errors, not the outcome variable itself, and is not nearly as important as the assumption of independence. See this old post from sci.stat.edu, for example. http://groups.google.com/group/sci.stat.edu/msg/745a15f3122398cf?dmode=source HTH. ----- -- Bruce Weaver [hidden email] http://sites.google.com/a/lakeheadu.ca/bweaver/ "When all else fails, RTFM." NOTE: My Hotmail account is not monitored regularly. To send me an e-mail, please use the address shown above. -- View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Categorizing-study-participants-based-on-their-scores-on-summated-scales-non-SPSS-question-tp3334460p3334879.html Sent from the SPSSX Discussion mailing list archive at Nabble.com. ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD |
Re: Categorizing study participants based on their scores on summated scales (non-SPSS question).
|
|
In reply to this post by Faiz Rasool
In addition to the comments of other
listmembers, you might consider conducting a quick read of John Tukey’s
view on exploratory data analysis. He published a classic, titled Exploratory
Data Analysis (referred to as EDA) in which he promoted an alternative view to
standard null hypothesis significance testing. His discussions of exploration
of data, and his use of the box plot and other devices, might provide you with
some guidance on where to draw your arbitrary demarcations for “high”
and “low”. Evan R. Harrington, Ph.D. Associate Professor Forensic Thesis Track
Director The Department of Forensic
Psychology Phone: 312 329-6693 Fax: 312 661-1272 From: SPSSX(r)
Discussion [mailto:[hidden email]] On
Behalf Of Faiz Rasool Hi all, The research textbooks I've read so far, do not provide a
guideline on how to categorize study participants into categories
like low, moderate and high, open to change, somewhat open to change
and not at all open to change, based on the scores they have obtained on the
summated scale. To give more clarity to my question, I'll describe the
confusion that I'm facing in my own research on conservation behaviors.
Variables like attitudes towards conservation, awareness of environmental
problems and influence of others on conservation behaviors are all part of my
research. I've constructed scales to measure these variables. I'm having confusion
in categorizing participants into different categories based on their
scores. I'll use an example of a five question scale to further
clarify my question. I've constructed a scale of 5 questions, It
provides 4 standard Likert type responses, i.e. strongly disagree to strongly
agree. Since the scale has 5 questions the maximum score possible is 20 and 5
is the lowest possible score. But I'm unable to find any guideline on the
following things: What criteria should I use other than personal judgment to
decide that on what is the score based on which participants can be placed in
high or low category. Should scores between 16 and 20 imply high, scores 11 to
15 moderate and scores between 5 to 10 low? Of course the responses are coded
in a way that higher scores means that the conservation behavior and
conservation attitudes are high. I plan to use the scores on the scale to use
in regression analysis, and if assumptions of regression are not met, then I'd
want to use those scores to make categories and use test like chi-square. Any suggestions and comments are most
appreciated. Thanks and regards, Faiz. |
|
|
In reply to this post by Faiz Rasool
Faiz,
Determing cut-points can be quite tricky. Before even considering cut-points, however, I think you have a more pressing issue. You have yet to establish that there are any underlying constructs (a.k.a. factors), and if there are, the actual number of underlying constructs. The classic approach to examine dimensionality would be to perform an exploratory factor analysis (EFA), assuming you have no theory or previous work establishing the structure. EFA falls under the rubric of classical test theory. There are other approaches (which I tend to prefer) that fall under item response theory that should be seriously considered. The bottom line is that your concern about developing cut-points may be a bit premature. Having stated that, there are a variety of ways to establish cut-points. One approach typically used after employing models which fall under classical test theory would be to compare a "normative" sample to a "clinical" sample, so to speak. Details on how to develop cut-points from such a comparison are probably not worth discussing since your situation does not appear to lend itself to this approach. Ryan On Mon, Jan 10, 2011 at 1:07 AM, Faiz Rasool <[hidden email]> wrote: > Hi all, > > The research textbooks I've read so far, do not provide a guideline on how > to categorize study participants into categories like low, moderate and > high, open to change, somewhat open to change and not at all open to change, > based on the scores they have obtained on the summated scale. > > To give more clarity to my question, I'll describe the confusion that I'm > facing in my own research on conservation behaviors. Variables like > attitudes towards conservation, awareness of environmental problems and > influence of others on conservation behaviors are all part of my research. > I've constructed scales to measure these variables. I'm having confusion in > categorizing participants into different categories based on their scores. > I'll use an example of a five question scale to further clarify my > question. I've constructed a scale of 5 questions, It provides 4 standard > Likert type responses, i.e. strongly disagree to strongly agree. Since the > scale has 5 questions the maximum score possible is 20 and 5 is the lowest > possible score. But I'm unable to find any guideline on the following > things: > > What criteria should I use other than personal judgment to decide that on > what is the score based on which participants can be placed in high or low > category. Should scores between 16 and 20 imply high, scores 11 to 15 > moderate and scores between 5 to 10 low? Of course the responses are coded > in a way that higher scores means that the conservation behavior and > conservation attitudes are high. I plan to use the scores on the scale to > use in regression analysis, and if assumptions of regression are not met, > then I'd want to use those scores to make categories and use test like > chi-square. > > Any suggestions and comments are most appreciated. > > Thanks and regards, > Faiz. > ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD |
Re: Categorizing study participants based on their scores on summated scales (non-SPSS question).
|
Administrator
|
In reply to this post by Bruce Weaver
One thing I was driving at here is that I don't (necessarily) see a need for carving them into categories. Why can you not use them as they are? There are lots of articles that describe the consequences of carving variables into categories prior to analysis. See Dave Streiner's "Breaking Up is Hard to Do" article, for example.
--
Bruce Weaver bweaver@lakeheadu.ca http://sites.google.com/a/lakeheadu.ca/bweaver/ "When all else fails, RTFM." PLEASE NOTE THE FOLLOWING: 1. My Hotmail account is not monitored regularly. To send me an e-mail, please use the address shown above. 2. The SPSSX Discussion forum on Nabble is no longer linked to the SPSSX-L listserv administered by UGA (https://listserv.uga.edu/). |
Re: Categorizing study participants based on their scores on summated scales (non-SPSS question).
|
|
In reply to this post by Faiz Rasool
Hi Faiz I am not sure whether am having a strong hand on this topic but still i like to share what i know. I've constructed a scale of 5 questions, It provides 4 standard Likert type responses, i.e. strongly disagree to strongly agree. Since the scale has 5 questions the maximum score possible is 20 and 5 is the lowest possible score. But I'm unable to find any guideline on the following things: from this i could understand tat u created 5 questions ok and each question consist of 5 point scale ie very stongly agree :5 pint strongly agree:4 moderately agree:3 disagree:2 strongly dis agree:1 so u got total of 5 question each with 5point scale so 5x5 =25 will be maximum score 3x5 = 15 is moderately satisfied 2x5 = 10 will be disagree here wen u cal calculate and u get a score if the score is above 15.. u can plot them as stongly agree suppose the score is 21 u can say majorty very stongly agree... if its below10 u can say majorty stongly disagree tanx faiz |
Re: Categorizing study participants based on their scores on summated scales (non-SPSS question).
|
|
The useful way to report Likert summative scales,
in my experience, is to give the "average item score." This lets you give your audience the original anchor-labels as complete explanation of what the score indicates. Since dividing by N-of-items is a linear transformation, it makes absolutely no difference to subsequent tests. -- Rich Ulrich > Date: Thu, 17 May 2012 11:58:31 -0700 > From: [hidden email] > Subject: Re: Categorizing study participants based on their scores on summated scales (non-SPSS question). > To: [hidden email] > > Hi Faiz > > I am not sure whether am having a strong hand on this topic but > still i like to share what i know. > > I've constructed a scale of 5 questions, It provides 4 standard Likert > type responses, i.e. strongly disagree to strongly agree. Since the scale > has 5 questions the maximum score possible is 20 and 5 is the lowest > possible score. But I'm unable to find any guideline on the following > things: > > from this i could understand tat u created 5 questions ok > and each question consist of 5 point scale ie > very stongly agree :5 pint > strongly agree:4 > moderately agree:3 > disagree:2 > strongly dis agree:1 > so u got total of 5 question each with 5point scale > so 5x5 =25 will be maximum score > 3x5 = 15 is moderately satisfied > 2x5 = 10 will be disagree > > here wen u cal calculate and u get a score if the score is above 15.. u can > plot them as stongly agree > suppose the score is 21 u can say majorty very stongly agree... if its > below10 u can say majorty stongly disagree > > |
Re: Categorizing study participants based on their scores on summated scales (non-SPSS question).
|
|
Hey Rich, While what you have recommended is far and away the most common practice, there are some who would suggest that is bad practice. It makes an
assumption that the summative scale retained the properties of the original scale, even though that is not true. It causes people to interpret the sum scale as if it were the individual component scales, but that really isn’t what it represents. Even when
all items are on the same scale, people’s response is going to create local variation for each item, and this local variation creates differences in the meaning of the variation of the final scale. You also have the problem that the final scale will have
much less variability than it’s component scales, which could be argued to reflect a change in the distance between points on the scale as well. I personally believe that the best approach in the somewhat casual use of scales, as we are discussing here, that refined factor scores or object
scores be used instead to reflect the final scale. It maximizes homogeneity, minimizes heterogeneity, and forces a more accurate interpretation of the end scale. I’ve actually presented a white paper on this topic not too long ago, and hope to have a journal
article out in the near future that makes the case for increased use of Refined factor scores and optimal scaling.
A good recent paper that explores this topic is “Understanding and Using Factor Scores: Considerations for the Applied Researcher” by DiStefano,
Zhu, and Mindrila (2009). As I began to do my own lit review, this helped me find some of the good standard papers discussing the various methods of creating factor scores, and there advantages/disadvantages. Another body of literature that discusses this
issue would be the older classical test theory papers in ability measures. I found the best work and descriptions was right when Rasche scaling took over in its place, after that it appears that factor scores from measures became a topic relegated to people
like us. I guess my main point here is that I believe it is potentially/arguably dangerous to interpret a factor as the simple sum of its parts. Rather
I believe it should be looked at for what it is, a latent construct represented by the triangulation of findings across a set of related measures/questions. Its own scale is related but not equal to that of its component measures/variables. The major drawback
to the use of refined factor scores is easy interpretation, which I believe is actually wrong on its face value. If we consider what I’ve said, then factors shouldn’t be interpreted based on their component scaling, and thus the loss of scale or metric from
the original variables is unimportant. Matthew J Poes Research Data Specialist Center for Prevention Research and Development University of Illinois 510 Devonshire Dr. Champaign, IL 61820 Phone: 217-265-4576 email:
[hidden email] From: SPSSX(r) Discussion [mailto:[hidden email]]
On Behalf Of Rich Ulrich The useful way to report Likert summative scales,
> Date: Thu, 17 May 2012 11:58:31 -0700 |
Re: Categorizing study participants based on their scores on summated scales (non-SPSS question).
|
|
Matthew,
I agree that what I do is a "casual use" of scales, though I think of it more as doing what is appropriate for "one-off" applications. With relatively small samples, and samples whose characteristics are unique, it is overkill -- with the risk of screwing it up, either in computation or interpretation -- to be fancy with one-time applications. Moreover, it was established in the 1930s (Likert and others) that, generally, little or nothing is gained by changing the spacing from 1, 2, 3, .... And short scales can become less reliable when heavy weight is given to one or two items. What I do recommend for instances where I want to average unlike items (or sub-scales) is to standardize each scale to SD=1; average the results; express the final scale as a T-score: with mean=50, SD=10. Similar T-scoring is also useful whenever there is a control group or a baseline that is important for a lot of comparisons. I am not sure what you are referring to as Refined factor scores and optimal scaling but I expect that it is less simple than my T scores. Finally, you make a statement that is hard to parse, about face- value interpretation. I think you are admitting that your Refined scales are hard to interpret. In my experience, one of the nastiest errors in data interpretation is that the researcher becomes enchanted by the label that is assigned to a factor. But being "high on bizarreness" gains a proper anchor if you can see that "high" is only 1.3 on a scale from 1 to 4. Then you may check and see that the only subscale item with many responses is the mildest of the 5 items in Bizarreness, "talks about dreams" (occasionally). [real example] You lose that ease with a Total in place of the Average, or with "optimal scaling" ala Correspondence analysis. It may be "dangerous to interpret a factor as the simple sum of its parts." I suppose it is more dangerous to interpret it that way... when it is *not* the simple sum of its parts, but uses obscure weights. -- Rich Ulrich Date: Fri, 18 May 2012 13:29:29 +0000 From: [hidden email] Subject: Re: Categorizing study participants based on their scores on summated scales (non-SPSS question). To: [hidden email] Hey Rich, While what you have recommended is far and away the most common practice, there are some who would suggest that is bad practice. It makes an assumption that the summative scale retained the properties of the original scale, even though that is not true. It causes people to interpret the sum scale as if it were the individual component scales, but that really isn’t what it represents. Even when all items are on the same scale, people’s response is going to create local variation for each item, and this local variation creates differences in the meaning of the variation of the final scale. You also have the problem that the final scale will have much less variability than it’s component scales, which could be argued to reflect a change in the distance between points on the scale as well.
I personally believe that the best approach in the somewhat casual use of scales, as we are discussing here, that refined factor scores or object scores be used instead to reflect the final scale. It maximizes homogeneity, minimizes heterogeneity, and forces a more accurate interpretation of the end scale. I’ve actually presented a white paper on this topic not too long ago, and hope to have a journal article out in the near future that makes the case for increased use of Refined factor scores and optimal scaling.
A good recent paper that explores this topic is “Understanding and Using Factor Scores: Considerations for the Applied Researcher” by DiStefano, Zhu, and Mindrila (2009). As I began to do my own lit review, this helped me find some of the good standard papers discussing the various methods of creating factor scores, and there advantages/disadvantages. Another body of literature that discusses this issue would be the older classical test theory papers in ability measures. I found the best work and descriptions was right when Rasche scaling took over in its place, after that it appears that factor scores from measures became a topic relegated to people like us.
I guess my main point here is that I believe it is potentially/arguably dangerous to interpret a factor as the simple sum of its parts. Rather I believe it should be looked at for what it is, a latent construct represented by the triangulation of findings across a set of related measures/questions. Its own scale is related but not equal to that of its component measures/variables. The major drawback to the use of refined factor scores is easy interpretation, which I believe is actually wrong on its face value. If we consider what I’ve said, then factors shouldn’t be interpreted based on their component scaling, and thus the loss of scale or metric from the original variables is unimportant.
Matthew J Poes Research Data Specialist Center for Prevention Research and Development University of Illinois 510 Devonshire Dr. Champaign, IL 61820 Phone: 217-265-4576 email: [hidden email]
[snip, previous] |
| Free forum by Nabble | Edit this page |