SPSSX Discussion

ceiling effect problem

Classic

List

Threaded

8 messages Options

Roy Money

ceiling effect problem

Though this is not a specifically SPSS question I am hoping that some
of you may have experience or useful suggestions regarding a data
problem I have encountered. I have done some library research on this
and not found much beyond the acknowledgment that ceiling effects can be
a recurring problem

I have longitudinal survey data on investigator ratings of achievement
in various goals such as
'commitment to interdisciplinary collaboration'
'exposure to divergent points of view'
'unexpected findings have served as source of new ideas',
as well as various ratings of satisfaction .

The rating scale is 1=Not at all, 4= Moderate and 7=Very Much.
As you might imagine we have a ceiling effect on some items and are
trying to work out a meaningful way to handle this
(the ceiling effect, % of responses that are coded at the maximum value,
ranges as high as 20-40% on some items)
We are considering using new anchors for the scale such as 5 = Very
Much and 7=Maximum Possible
but of course we still have the existing data which would need to be
transformed.
Any suggestions or references on an appropriate way to rescale the
maximum values down or otherwise deal with this situation?

Thanks

--
Roy Money M.S.
Programmer Analyst
Department of Psychiatry
Yale University School of Medicine
The Consultation Center
389 Whitney Avenue
New Haven, CT 06511
ph:(203)789-7645 x 126
fax: (203)562-6355

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

Guerrero, Rodrigo

Re: ceiling effect problem

Have you tried to centralize and standardize your data? I find it
useful in this situations to centralize based on each respondent's mean
score. Just subtract the mean from each response.

RG

Rodrigo A. Guerrero | Director Of Marketing Research and Analysis | The
Scooter Store | 830.627.4317

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of
Roy Money
Sent: Monday, March 16, 2009 1:54 PM
To: [hidden email]
Subject: ceiling effect problem

Though this is not a specifically SPSS question I am hoping that some
of you may have experience or useful suggestions regarding a data
problem I have encountered. I have done some library research on this
and not found much beyond the acknowledgment that ceiling effects can be
a recurring problem

I have longitudinal survey data on investigator ratings of achievement
in various goals such as
'commitment to interdisciplinary collaboration'
'exposure to divergent points of view'
'unexpected findings have served as source of new ideas',
as well as various ratings of satisfaction .

The rating scale is 1=Not at all, 4= Moderate and 7=Very Much.
As you might imagine we have a ceiling effect on some items and are
trying to work out a meaningful way to handle this
(the ceiling effect, % of responses that are coded at the maximum value,
ranges as high as 20-40% on some items)
We are considering using new anchors for the scale such as 5 = Very
Much and 7=Maximum Possible
but of course we still have the existing data which would need to be
transformed.
Any suggestions or references on an appropriate way to rescale the
maximum values down or otherwise deal with this situation?

Thanks

--
Roy Money M.S.
Programmer Analyst
Department of Psychiatry
Yale University School of Medicine
The Consultation Center
389 Whitney Avenue
New Haven, CT 06511
ph:(203)789-7645 x 126
fax: (203)562-6355

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

The information transmitted is intended only for the addressee(s) and may contain confidential or privileged material, or both. Any review, receipt, dissemination or other use of this information by non-addressees is prohibited. If you received this in error or are a non-addressee, please contact the sender and delete the transmitted information.

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

statisticsdoc

Re: ceiling effect problem

In reply to this post by Roy Money

Roy,

It is important to work out whether the ceiling effect is an artefact of the measurement scale, or whether it is reasonable to say that many subjects simply have the highest level possible of the attribute that you are measuring. If the latter is the case, it would be worthwhile to consider how to analyze the longitudinal data in ways that take the ceiling effect into consideration. There are a number of ways to go here, including: a.) using cluster analysis to create a typology of trajectories over time; and b.) focusing the analysis on subjects with low initial scores, so you are considering such phenomena as onset in this group. There are of course other options - the main point is that sometimes ceiling effects are real and have to be handled in the analysis.

Can you provide a little more concrete detail on the focus of your study?

HTH,

Steve Brand

---- Roy Money <[hidden email]> wrote:

> Though this is not a specifically SPSS question I am hoping that some
> of you may have experience or useful suggestions regarding a data
> problem I have encountered. I have done some library research on this
> and not found much beyond the acknowledgment that ceiling effects can be
> a recurring problem
>
> I have longitudinal survey data on investigator ratings of achievement
> in various goals such as
> 'commitment to interdisciplinary collaboration'
> 'exposure to divergent points of view'
> 'unexpected findings have served as source of new ideas',
> as well as various ratings of satisfaction .
>
> The rating scale is 1=Not at all, 4= Moderate and 7=Very Much.
> As you might imagine we have a ceiling effect on some items and are
> trying to work out a meaningful way to handle this
> (the ceiling effect, % of responses that are coded at the maximum value,
> ranges as high as 20-40% on some items)
> We are considering using new anchors for the scale such as 5 = Very
> Much and 7=Maximum Possible
> but of course we still have the existing data which would need to be
> transformed.
> Any suggestions or references on an appropriate way to rescale the
> maximum values down or otherwise deal with this situation?
>
> Thanks
>
>
>
> --
> Roy Money M.S.
> Programmer Analyst
> Department of Psychiatry
> Yale University School of Medicine
> The Consultation Center
> 389 Whitney Avenue
> New Haven, CT 06511
> ph:(203)789-7645 x 126
> fax: (203)562-6355
>
> =====================
> To manage your subscription to SPSSX-L, send a message to
> [hidden email] (not to SPSSX-L), with no body text except the
> command. To leave the list, send the command
> SIGNOFF SPSSX-L
> For a list of commands to manage subscriptions, send the command
> INFO REFCARD

--
For personalized and experienced consulting in statistics and research design, visit www.statisticsdoc.com

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

Guerrero, Rodrigo

Re: ceiling effect problem

In reply to this post by Guerrero, Rodrigo

I have tried to change the anchor descriptions changing the scale in
surveys before, but I have not been able to get people respond
differently. I was only able to move the mean by just slightly, but
still VERY top loaded. People just use scales differently. Maybe others
have had other experiences with this.

If you are going back into data collection, you may try to change the
question type from scale to rank order or have them distribute points
across the questions. Admittedly this is more cumbersome for
respondents, and some of them will not understand what they are doing
and quit the survey, but it might get you closer to what you are looking
for.

RG

Rodrigo A. Guerrero | Director Of Marketing Research and Analysis | The
Scooter Store | 830.627.4317

-----Original Message-----
From: Roy Money [mailto:[hidden email]]
Sent: Monday, March 16, 2009 2:22 PM
To: Guerrero, Rodrigo
Subject: Re: ceiling effect problem

Right.
I have considered that and other transformations of the existing data,
but this would not address the fact that I have so many folks who are
already maxed out and have no room for improvement. What we are hoping
to find is a meaningful way to shift the 7's down so an individual's
previous data is commensurate with new data for them based on revised
anchors that would safeguard against so many 7's.

Roy

Guerrero, Rodrigo wrote:
> Have you tried to centralize and standardize your data? I find it
> useful in this situations to centralize based on each respondent's
mean
> score. Just subtract the mean from each response.
>
>
> RG
>
> Rodrigo A. Guerrero | Director Of Marketing Research and Analysis |
The
> Scooter Store | 830.627.4317
>
> -----Original Message-----
> From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf
Of
> Roy Money
> Sent: Monday, March 16, 2009 1:54 PM
> To: [hidden email]
> Subject: ceiling effect problem
>
> Though this is not a specifically SPSS question I am hoping that some
> of you may have experience or useful suggestions regarding a data
> problem I have encountered. I have done some library research on this
> and not found much beyond the acknowledgment that ceiling effects can
be

> a recurring problem
>
> I have longitudinal survey data on investigator ratings of achievement
> in various goals such as
> 'commitment to interdisciplinary collaboration'
> 'exposure to divergent points of view'
> 'unexpected findings have served as source of new ideas',
> as well as various ratings of satisfaction .
>
> The rating scale is 1=Not at all, 4= Moderate and 7=Very Much.
> As you might imagine we have a ceiling effect on some items and are
> trying to work out a meaningful way to handle this
> (the ceiling effect, % of responses that are coded at the maximum

value,

> ranges as high as 20-40% on some items)
> We are considering using new anchors for the scale such as 5 = Very
> Much and 7=Maximum Possible
> but of course we still have the existing data which would need to be
> transformed.
> Any suggestions or references on an appropriate way to rescale the
> maximum values down or otherwise deal with this situation?
>
> Thanks
>
>
>
> --
> Roy Money M.S.
> Programmer Analyst
> Department of Psychiatry
> Yale University School of Medicine
> The Consultation Center
> 389 Whitney Avenue
> New Haven, CT 06511
> ph:(203)789-7645 x 126
> fax: (203)562-6355
>
> =====================
> To manage your subscription to SPSSX-L, send a message to
> [hidden email] (not to SPSSX-L), with no body text except

the
> command. To leave the list, send the command
> SIGNOFF SPSSX-L
> For a list of commands to manage subscriptions, send the command
> INFO REFCARD
>
>
> The information transmitted is intended only for the addressee(s) and
may contain confidential or privileged material, or both. Any review,
receipt, dissemination or other use of this information by
non-addressees is prohibited. If you received this in error or are a
non-addressee, please contact the sender and delete the transmitted
information.
>
> =====================
> To manage your subscription to SPSSX-L, send a message to
> [hidden email] (not to SPSSX-L), with no body text except
the
> command. To leave the list, send the command
> SIGNOFF SPSSX-L
> For a list of commands to manage subscriptions, send the command
> INFO REFCARD
>
>

--
Roy Money M.S.
Programmer Analyst
Department of Psychiatry
Yale University School of Medicine
The Consultation Center
389 Whitney Avenue
New Haven, CT 06511
ph:(203)789-7645 x 126
fax: (203)562-6355

The information transmitted is intended only for the addressee(s) and may contain confidential or privileged material, or both. Any review, receipt, dissemination or other use of this information by non-addressees is prohibited. If you received this in error or are a non-addressee, please contact the sender and delete the transmitted information.

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

Bob Schacht-3

Re: ceiling effect problem

In reply to this post by Roy Money

At 08:54 AM 3/16/2009, Roy Money wrote:

...I have longitudinal survey data on investigator ratings of achievement
in various goals such as
'commitment to interdisciplinary collaboration'
'exposure to divergent points of view'
'unexpected findings have served as source of new ideas',
as well as various ratings of satisfaction .

The rating scale is 1=Not at all, 4= Moderate and 7=Very Much.
As you might imagine we have a ceiling effect on some items and are
trying to work out a meaningful way to handle this
(the ceiling effect, % of responses that are coded at the maximum value,
ranges as high as 20-40% on some items)
We are considering using new anchors for the scale such as 5 = Very
Much and 7=Maximum Possible
but of course we still have the existing data which would need to be transformed.
Any suggestions or references on an appropriate way to rescale the
maximum values down or otherwise deal with this situation?

What you call "ceiling effects" is very common in satisfaction surveys. I've been working on a satisfaction survey redesign for more than a year, so this is an issue of some interest to us as well. Unfortunately, you are in a longitudinal survey situation, so you cannot so easily change the wording of the questions-- or even the wording of the answers. Any changes at all would affect your longitudinal analyses, wouldn't it?

In many satisfaction surveys, it seems that people are either happy campers, or unhappy campers. The happy campers tend to look for the highest satisfaction category on every question, and the unhappy campers look for the highest dissatisfaction category on every question. Sampling bias usually results in a higher probability of participation by happy campers unless there is some mechanism for forcing participation in a representative sample. The result is usually a J-shaped bimodal distribution, with the short leg at the "very dissatisfied" end of the scale, and the long leg at the "very satisfied" end of the scale, rather than a normal distribution.

If all of your questions result in distributions that are skewed positively (your "ceiling effect"), you might want to examine your sampling procedures, and what you can do to improve them.

Bob Schacht

Robert M. Schacht, Ph.D., Research Director

Pacific Basin Research and Training Center

1268 Young Street, Suite #204

Research Center, University of Hawaii

Honolulu, HI 96814

E-mail <[hidden email]>

Phone 808-592-5904, FAX 808-592-5909

Ruben Geert van den Berg

Re: ceiling effect problem

Dear Ron and others,

This problem is familiar to me as well. What seems to work reasonably for us, is to use (what we call) 'skewed' scales. If a sample tends to be overly positive, we use a scale with more positive answers than negative ones, like:

1 (Very) bad

2 Neutral

3 Good

4 Very good

5 Excellent

We typically -though not always- observe that this yields univariate distributions that don't deviate too much from being normal or at least symmetrical.

However, when we conduct CSSs relating to governmental institutions, we have to reverse the skewed scale (an extra negative category) in order to achieve desirable distributions. It seems there's a lot of 'unhappy campers' when it comes to our government!

Well, it's possibly too late for you to adapt your rating scales anyway but I just wanted to point this out as an option.

Kind regards,

Ruben v.d. Berg

Date: Mon, 16 Mar 2009 10:29:20 -1000
From: [hidden email]
Subject: Re: ceiling effect problem
To: [hidden email]

At 08:54 AM 3/16/2009, Roy Money wrote:

...I have longitudinal survey data on investigator ratings of achievement
in various goals such as
'commitment to interdisciplinary collaboration'
'exposure to divergent points of view'
'unexpected findings have served as source of new ideas',
as well as various ratings of satisfaction .

The rating scale is 1=Not at all, 4= Moderate and 7=Very Much.
As you might imagine we have a ceiling effect on some items and are
trying to work out a meaningful way to handle this
(the ceiling effect, % of responses that are coded at the maximum value,
ranges as high as 20-40% on some items)
We are considering using new anchors for the scale such as 5 = Very
Much and 7=Maximum Possible
but of course we still have the existing data which would need to be transformed.
Any suggestions or references on an appropriate way to rescale the
maximum values down or otherwise deal with this situation?

Robert M. Schacht, Ph.D., Research Director

Pacific Basin Research and Training Center

1268 Young Street, Suite #204

Research Center, University of Hawaii

Honolulu, HI 96814

E-mail <[hidden email]>

Phone 808-592-5904, FAX 808-592-5909

See all the ways you can stay connected to friends and family

Roy Money

Re: ceiling effect problem

Thanks Ruben and Bob (before).
We do have a 'situation'.
The idea of a skewed scale will have some appeal but we still have the
existing data which we would like to have a way of equating/comparing to
the new data.
One suggestion that has been made is to provide in future surveys an
additional generalized comparison item with a broader range that could
be used to estimate a conversion ratio for the previously collected
data. One investigator thought there might be some precedent for
something along these lines but I have not been able to find anything in
my library queries.

Roy

Ruben van den Berg wrote:

> .hmmessage P { margin:0px; padding:0px } body.hmmessage { font-size:
> 10pt; font-family:Verdana }
>
> Dear Ron and others,
>
> This problem is familiar to me as well. What seems to work reasonably
> for us, is to use (what we call) 'skewed' scales. If a sample tends to
> be overly positive, we use a scale with more positive answers than
> negative ones, like:
>
> 1 (Very) bad
>
> 2 Neutral
>
> 3 Good
>
> 4 Very good
>
> 5 Excellent
>
> We typically -though not always- observe that this yields univariate
> distributions that don't deviate too much from being normal or at
> least symmetrical.
>
> However, when we conduct CSSs relating to governmental institutions,
> we have to reverse the skewed scale (an extra negative category) in
> order to achieve desirable distributions. It seems there's a lot of
> 'unhappy campers' when it comes to our government!
>
> Well, it's possibly too late for you to adapt your rating scales
> anyway but I just wanted to point this out as an option.
>
>
> Kind regards,
>
> Ruben v.d. Berg
>
>
>
>
>
>
> ------------------------------------------------------------------------
> Date: Mon, 16 Mar 2009 10:29:20 -1000
> From: [hidden email]
> Subject: Re: ceiling effect problem
> To: [hidden email]
>
> At 08:54 AM 3/16/2009, Roy Money wrote:
>
> ...I have longitudinal survey data on investigator ratings of
> achievement
> in various goals such as
> 'commitment to interdisciplinary collaboration'
> 'exposure to divergent points of view'
> 'unexpected findings have served as source of new ideas',
> as well as various ratings of satisfaction .
>
> The rating scale is 1=Not at all, 4= Moderate and 7=Very Much.
> As you might imagine we have a ceiling effect on some items and are
> trying to work out a meaningful way to handle this
> (the ceiling effect, % of responses that are coded at the maximum
> value,
> ranges as high as 20-40% on some items)
> We are considering using new anchors for the scale such as 5 = Very
> Much and 7=Maximum Possible
> but of course we still have the existing data which would need to
> be transformed.
> Any suggestions or references on an appropriate way to rescale the
> maximum values down or otherwise deal with this situation?
>
>
> What you call "ceiling effects" is very common in satisfaction
> surveys. I've been working on a satisfaction survey redesign for more
> than a year, so this is an issue of some interest to us as well.
> Unfortunately, you are in a longitudinal survey situation, so you
> cannot so easily change the wording of the questions-- or even the
> wording of the answers. Any changes at all would affect your
> longitudinal analyses, wouldn't it?
>
> In many satisfaction surveys, it seems that people are either happy
> campers, or unhappy campers. The happy campers tend to look for the
> highest satisfaction category on every question, and the unhappy
> campers look for the highest dissatisfaction category on every
> question. Sampling bias usually results in a higher probability of
> participation by happy campers unless there is some mechanism for
> forcing participation in a representative sample. The result is
> usually a J-shaped bimodal distribution, with the short leg at the
> "very dissatisfied" end of the scale, and the long leg at the "very
> satisfied" end of the scale, rather than a normal distribution.
>
> If all of your questions result in distributions that are skewed
> positively (your "ceiling effect"), you might want to examine your
> sampling procedures, and what you can do to improve them.
>
> Bob Schacht
>
> Robert M. Schacht, Ph.D., Research Director
> Pacific Basin Research and Training Center
> 1268 Young Street, Suite #204
> Research Center, University of Hawaii
> Honolulu, HI 96814
> E-mail <[hidden email]>
> Phone 808-592-5904, FAX 808-592-5909
>
> ------------------------------------------------------------------------
> See all the ways you can stay connected to friends and family
> <http://www.microsoft.com/windows/windowslive/default.aspx>

--
Roy Money M.S.
Programmer Analyst
Department of Psychiatry
Yale University School of Medicine
The Consultation Center
389 Whitney Avenue
New Haven, CT 06511
ph:(203)789-7645 x 126
fax: (203)562-6355

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

Maguin, Eugene

Re: ceiling effect problem

With respect to analyzing data with ceiling or floor effects, I've two
different strategies discussed, in addition to ignoring the problem and
transforming. One strategy is to treat the data as ordinal rather than
continuous.

The other method is to use a two part model. However, I've only seen two
part models applied to behavioral rating scales where there is a 'never or
almost never category', which is different from an opinion or sentiment type
scale with a positive and negative end. Anyway, the idea, as I recall it, is
that the data can be represented as having two parts. One part is a
dichotomous rating of 'problem-no problem' and the other is a ordinal
severity scale. If a respondent says 'never' then dichotomous part is scored
as 0 and the severity part is missng. If the respondent says anything other
than 'never', then the dichotomous part is scored as 1 and the severity part
has a non-missing value.

Also, I've only seen two part models used in a growth model structure, which
might work here since the data are longitudinal.

One reference for two part models is
Olsen, M.K. & Schafer, J.L. (2001). A two-part random-effects model for
semicontinuous
longitudinal data. Journal of the American Statistical Association, 96,
730-745.

But also look in the usual places.

Gene Maguin

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD