Combining Results of Surveys with Slightly Different Questions

[Heidi Green]

Hello from Sunny Southern California ~

 

I have searched the archives and have checked Google, but I have not seen anything that directly relates to this question, hence I am seeking your opinions.

 

I have survey results from a number of years, and I have no control over the wording/choices printed on the survey. The surveys are “tweaked” a little each year, although the majority of the questions remain the same.

 

In the set that I am interested in, I have 39 surveys from 2006 – 2009 with the following question:

Overall Satisfaction with Product X?

5 = Very Satisfied

4 = Somewhat Satisfied

3 = Neither Satisfied Nor Dissatisfied

2 = Somewhat Dissatisfied

1 = Very Dissatisfied

 

and 22 surveys from 2001 – 2005 with the following question:

Overall Satisfaction with Product X?

5 = Extremely Satisfied

4 = Very Satisfied

3 = Somewhat Satisfied

2 = Somewhat Dissatisfied

1 = Very Dissatisfied

 

These questions are given to me by our clients, and I cannot ask them to change the way the survey is written. Each survey has anywhere from 500 – 2,000 responses.

 

Is it in any way statistically valid to create broader range categories that could combine results across all years?

 

I was considering recoding into something like this:

5 = Extremely/Very Satisfied  (this would be choice 5 from the first set + choice 5 & 4 from the second set)

4 = Somewhat Satisfied

3 = Neither Satisfied Nor Dissatisfied

2 = Somewhat Dissatisfied

1 = Very Dissatisfied

 

It troubles me, however, that one set doesn’t offer the “Neither” option. Should I remove those responders? Or keep them and heavily footnote my results? Based on previous experience with similar data, I am probably going to end up with the majority of all responders being in “Extremely/Very Satisfied”. I know that combining groups will exacerbate this issue. To complicate matters, I do not know how the combined results will be presented, or what statistical procedures will be run on them. I have only been asked to prepare the data and pass it on to a colleague.  (FYI: We may have an opening for a statistician/SPSS analyst in our Irvine CA office soon!)

 

Am I stuck with keeping the results in 2 separate groups?

 

Thanks for sharing your thoughts,

 

-Heidi

 

[John F Hall]

Heidi

 

They're all on 1 - 5, so you could forget statistics and treat them all the same.  I had the same problem comparing UK  1 - 7 scales with USA 0 - 10 on the Quality of Life surveys.  The mid-point label is a problem, but you could just ignore it.  Is it worth doing transformations to get the distributions into the same shape?  I doubt it.

 

What's the product?  Shampoo or something more important? 

----- Original Message -----

From: [hidden email]

To: [hidden email]

Sent: Thursday, July 29, 2010 11:41 PM

Subject: Combining Results of Surveys with Slightly Different Questions

Hello from Sunny Southern California ~

 

I have searched the archives and have checked Google, but I have not seen anything that directly relates to this question, hence I am seeking your opinions.

 

I have survey results from a number of years, and I have no control over the wording/choices printed on the survey. The surveys are “tweaked” a little each year, although the majority of the questions remain the same.

 

In the set that I am interested in, I have 39 surveys from 2006 – 2009 with the following question:

Overall Satisfaction with Product X?

5 = Very Satisfied

4 = Somewhat Satisfied

3 = Neither Satisfied Nor Dissatisfied

2 = Somewhat Dissatisfied

1 = Very Dissatisfied

 

and 22 surveys from 2001 – 2005 with the following question:

Overall Satisfaction with Product X?

5 = Extremely Satisfied

4 = Very Satisfied

3 = Somewhat Satisfied

2 = Somewhat Dissatisfied

1 = Very Dissatisfied

 

These questions are given to me by our clients, and I cannot ask them to change the way the survey is written. Each survey has anywhere from 500 – 2,000 responses.

 

Is it in any way statistically valid to create broader range categories that could combine results across all years?

 

I was considering recoding into something like this:

5 = Extremely/Very Satisfied  (this would be choice 5 from the first set + choice 5 & 4 from the second set)

4 = Somewhat Satisfied

3 = Neither Satisfied Nor Dissatisfied

2 = Somewhat Dissatisfied

1 = Very Dissatisfied

 

It troubles me, however, that one set doesn’t offer the “Neither” option. Should I remove those responders? Or keep them and heavily footnote my results? Based on previous experience with similar data, I am probably going to end up with the majority of all responders being in “Extremely/Very Satisfied”. I know that combining groups will exacerbate this issue. To complicate matters, I do not know how the combined results will be presented, or what statistical procedures will be run on them. I have only been asked to prepare the data and pass it on to a colleague.  (FYI: We may have an opening for a statistician/SPSS analyst in our Irvine CA office soon!)

 

Am I stuck with keeping the results in 2 separate groups?

 

Thanks for sharing your thoughts,

 

-Heidi

 

[Mircea Comsa]

Hi!

In my opinion the scales are quite different (no of positive answers, the
presence / absence of a middle point). I expect that the distributions
from the two types of questions are different (due to scales, not to time
/ change).

Because you need a solution I suggest checking for measurement invariance.
Probably the two questions were used for more products (some of them the
same) and the answers are correlated (probably the first factor is a
general one - people that are satisfied with all products - configural
invariance). If the slopes between the two sets of questions (metric
invariance), and the intercepts (scalar invariance) are the same, you
could compare the results. One more thing: you can do measurement
invariance test (separately) for both types of scales (original ones and
transformed as you like) and chose the better solution.

All the best!
Mircea

> Hello from Sunny Southern California ~
>
>
>
> I have searched the archives and have checked Google, but I have not
> seen anything that directly relates to this question, hence I am seeking
> your opinions.
>
>
>
> I have survey results from a number of years, and I have no control over
> the wording/choices printed on the survey. The surveys are "tweaked" a
> little each year, although the majority of the questions remain the
> same.
>
>
>
> In the set that I am interested in, I have 39 surveys from 2006 - 2009
> with the following question:
>
> Overall Satisfaction with Product X?
>
> 5 = Very Satisfied
>
> 4 = Somewhat Satisfied
>
> 3 = Neither Satisfied Nor Dissatisfied
>
> 2 = Somewhat Dissatisfied
>
> 1 = Very Dissatisfied
>
>
>
> and 22 surveys from 2001 - 2005 with the following question:
>
> Overall Satisfaction with Product X?
>
> 5 = Extremely Satisfied
>
> 4 = Very Satisfied
>
> 3 = Somewhat Satisfied
>
> 2 = Somewhat Dissatisfied
>
> 1 = Very Dissatisfied
>
>
>
> These questions are given to me by our clients, and I cannot ask them to
> change the way the survey is written. Each survey has anywhere from 500
> - 2,000 responses.
>
>
>
> Is it in any way statistically valid to create broader range categories
> that could combine results across all years?
>
>
>
> I was considering recoding into something like this:
>
> 5 = Extremely/Very Satisfied  (this would be choice 5 from the first set
> + choice 5 & 4 from the second set)
>
> 4 = Somewhat Satisfied
>
> 3 = Neither Satisfied Nor Dissatisfied
>
> 2 = Somewhat Dissatisfied
>
> 1 = Very Dissatisfied
>
>
>
> It troubles me, however, that one set doesn't offer the "Neither"
> option. Should I remove those responders? Or keep them and heavily
> footnote my results? Based on previous experience with similar data, I
> am probably going to end up with the majority of all responders being in
> "Extremely/Very Satisfied". I know that combining groups will exacerbate
> this issue. To complicate matters, I do not know how the combined
> results will be presented, or what statistical procedures will be run on
> them. I have only been asked to prepare the data and pass it on to a
> colleague.  (FYI: We may have an opening for a statistician/SPSS analyst
> in our Irvine CA office soon!)
>
>
>
> Am I stuck with keeping the results in 2 separate groups?
>
>
>
> Thanks for sharing your thoughts,
>
>
>
> -Heidi
>
>
>
>

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[<Joanne Tse>]

Hi,

 

UCLA Extension will be closed from July 30, 2010 through August 1, 2010.  This closure is due to the salary reduction/furlough plan adopted by the University of California Regents in response to the State of California’s budget crisis.  I am sorry I cannot help you at this time, but I will respond to your message when I return to the office on August 2, 2010.   If you would like to enroll in an Extension course, please visit our website at http://www.uclaextension.edu.

 

Thank you.

 

Joanne Han

Marketing Research Manager

UCLA Extension

www.uclaextension.edu