SPSSX Discussion

Question Regarding Analysis

Classic

List

Threaded

8 messages Options

bdates

Question Regarding Analysis

I have been asked to analyze data from a large (n=402) study of an intervention for adolescents. One of the measures, that is quite highly used, is the CAFAS, which assesses functioning in children and youth, age 6 to 18. There are eight items, scored from 0 to 30 in increments of 10 (i.e., 0, 10, 20, 30). The eight items produce a total score ranging from 0 to 240. A review of the literature does not reveal really good reliability, either alpha, or ICC (the measure is clinician-completed). Item-total correlations range from .20 to .57. The distribution of scores on all items is skewed, either positively or negatively. I'm proposing to analyze each of the scales as if they were ordinal and the total score as interval, using a repeated measures approach since there are multiple measures per youth. I neither designed the study nor chose the measures. I've simply been commissioned to analyze the data. Any thoughts on treating the items as ordinal? I know it's conservative, but I have difficulty with a four-point discontinuous scale as really interval in nature.

Brian

===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD

Rich Ulrich

Re: Question Regarding Analysis

Google shows me that this is a scale with a long history of use,

sampling in various populations; and that the 4-point scoring

stands for Severe/ Moderate/ Mild/ Little or no impairment.

On reliability:

It is the wrong idea, to use "internal consistency" as /the/ criterion

for reliability for a composite that covers a variety of areas. You

want rater-rater comparisons for assessing consistency, and some

external comparisons for validity.

On "equal intervals":

If I wanted evidence of the non-interval nature of the scaling, I

would look at the cross-time tabulations, even better than the

cross-rater tabulations. What I would look for is the frequency

of changes between categories. If, for instance, no one "ever"

moves from 0 to 10 (compared to other changes), then that

suggests that the interval is largest between 0 and 10. But is that

perhaps a function of the high-risk population you are sampling?

Do you have a subjective feeling that the distances are unequal?

It is very, very common for 4-point scales to be analyzed by ANOVA

as if they equal-interval to a suitable degree - For a much-used

scale, apparently others have been satisfied. The question is not

"Are these unequal intervals" but, rather, "Are these intervals

improved enough by some transformation to justify the complication

of computation, and the confusion in presenting results?"

On transformation to rank:

The simple rule of thumb is that, if the means do not provide a

good comparison between groups, then you probably should

transform. The usual "non-parametric" test gives an analysis,

essentially by ANOVA, of the rank-transformed scores.

So, for a few skewed variables, compute the rank-transformed

scores. (If 400 scores are /equally/ divided into four groups, your

result effectively matches the "intervals" you started out with,

since the 100 ties in each group give you average ranks of 50.5,

150.5, 250.5, and 350.5 -- Transforming "10" to 50.5, etc., gives

you the same ANOVA F, since you have simply performed the

same linear transformation on each of the scores.)

Does this version of "interval", when applied to the skewed

variables (where it makes a difference) make more sense? In my

experience, I usually haven't preferred the simple, rank scoring.

Advanced scale development goes a step further, and uses a

logistic transformation on the average-rank. This DOES give

a spacing for which there is some theoretical justification,

and better "normality". I won't say any more about that, except

that I want a "norming" sample if I'm going to set norms for that

version of scaling -- and , personally, I never did play with scale

development that intensively.

Rich Ulrich

From: SPSSX(r) Discussion <[hidden email]> on behalf of Dates, Brian <[hidden email]>
Sent: Tuesday, May 7, 2019 10:33 AM
To: [hidden email]
Subject: Question Regarding Analysis

Brian

bdates

Re: Question Regarding Analysis

Rich,

Thanks for taking the quite impressive amount of time to answer this. On the reliability side, I've always favored the ICC rather than Chronbach, although most of the author's work has been with Chronbach. I'll take your advice on looking at the interval/ordinal nature of each item/subscale. Yes, my subjective feeling is that the distances are not equal. I may be confounding the difficulty, but each level of severity, 0 to 30, has a number of anchors, which are quite specific. The score on any item is the most severe category indicated by an anchor. However, one can be assigned a 30 with one anchor checked or with three checked. Lower anchors, e.g., at 20 or 10 are not even counted in assigning a score. So in many ways the score is really categorical, but later assigned a numerical score. I really think a youth with three anchors in a category is probably more disturbed than if only one were present. Just my thoughts.

Thanks again for your time and expertise.

Brian

From: Rich Ulrich <[hidden email]>
Sent: Tuesday, May 7, 2019 12:36:27 PM
To: [hidden email]; Dates, Brian
Subject: Re: Question Regarding Analysis