Order of Questions

[Bob Schacht-3]

I'm on a team that is developing a new questionnaire for consumer
satisfaction. We are trying to cover six domains of satisfaction, and have
developed a list of 90 - 100 items on these six domains. Our plan is to
pilot test these questions on a set of participants who have previously
responded to the current survey (which is only about 10 questions). Based
on the results, we want to then eliminate redundant questions in order to
reduce the entire list of questions to about 24 questions, or about 4
questions from each of the six domains.

Being aware of respondent fatigue (RF), it has occurred to us to rotate the
order of the questions, to avoid saddling the last questions in the survey
with the largest amount of RF. Rather than randomize the order of the
individual questions, we thought to merely rotate the order in which each
of the six sets of questions are presented. So, for example, the questions
about the respondent's counselor would not always come last.

On the other hand, we are also aware that the order in which questions are
asked can itself be a significant factor in how questions are answered.

What is the best way in such circumstances to control for the order of the
questions, respondent fatigue, etc.?

Bob Schacht



Robert M. Schacht, Ph.D. <[hidden email]>
Pacific Basin Rehabilitation Research & Training Center
1268 Young Street, Suite #204
Research Center, University of Hawaii
Honolulu, HI 96814

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Matthew Pirritano]

I have had the pleasure of administering similar sets of questionnaires to groups of up to 40 9 year old elementary school children. Maybe other folks will differ in their opinions on this but my experience has been that if you read the items aloud, to help keep everyone together, and keep it at a good pace, a little faster that you think people can go, 100 questions can be quite painless. You can finish in as little as 15 minutes. The key to reading the items aloud is that people can process the information quicker and easier than if they have to read each item themselves. Some people are quick readers and will move ahead at their own pace. If you actually include item numbers to the left and right of the bubbles (assuming you're using a typical likert scale) people can just move on down through the items, knowing that they're on the correct question, without having to refer to the written items.

That's the long version of my 2 cents. Short version, I don't think response fatigue for 100 items is necessarily a factor. We tested thousands of elementary school students with, believe it or not, really no complaints from them. I think this is a good acid test. If little kids can do it...

Good luck and good night,
Matt

Matthew Pirritano, Ph.D.
Email: [hidden email]
Assistant Professor of Psychology
Smith Hall 116C
Chapman University
Department of Psychology
One University Drive
Orange, CA 92866
Telephone [phone]
FAX [phone]



----- Original Message ----
From: Bob Schacht <[hidden email]>
To: [hidden email]
Sent: Tuesday, April 1, 2008 2:02:51 PM
Subject: Order of Questions

I'm on a team that is developing a new questionnaire for consumer
satisfaction. We are trying to cover six domains of satisfaction, and have
developed a list of 90 - 100 items on these six domains. Our plan is to
pilot test these questions on a set of participants who have previously
responded to the current survey (which is only about 10 questions). Based
on the results, we want to then eliminate redundant questions in order to
reduce the entire list of questions to about 24 questions, or about 4
questions from each of the six domains.

Being aware of respondent fatigue (RF), it has occurred to us to rotate the
order of the questions, to avoid saddling the last questions in the survey
with the largest amount of RF. Rather than randomize the order of the
individual questions, we thought to merely rotate the order in which each
of the six sets of questions are presented. So, for example, the questions
about the respondent's counselor would not always come last.

On the other hand, we are also aware that the order in which questions are
asked can itself be a significant factor in how questions are answered.

What is the best way in such circumstances to control for the order of the
questions, respondent fatigue, etc.?

Bob Schacht



Robert M. Schacht, Ph.D. <[hidden email]>
Pacific Basin Rehabilitation Research & Training Center
1268 Young Street, Suite #204
Research Center, University of Hawaii
Honolulu, HI 96814

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Gary Oliver]

Hi Matt

I have done, and found, the same with adults for 170 questions taking
<30 mins. However I have never found any article/authority supporting
method

Warm regards/gary

Dr Gary R Oliver, CPA
B A Anthrop. & Philosophy (Macq); M Com Acctg & Info Sys (UNSW); Grad
Cert Higher Ed (Usyd), M Ed Higher Ed (USyd); PhD Information Sharing
(USyd)
Discipline of Accounting
School of Business
The University of Sydney
------------------------
      ,-_|\       Building H69, Office 437
     /        \      Corner Codrington Street
     \_,-._*      & Rose Street
                  Darlington 2006
           @      Australia
--------------------------------------
E-mail: [hidden email]
------------------------
Location details:
Travelling from Broadway, turn south off City Road
Navigate toward the Aquatic Centre
------------------------
University Map: http://db.auth.usyd.edu.au/directories/map/index.stm
University Website:
www.usyd.edu.au
Faculty Website
www.econ.usyd.edu.au
------------------------
Faculty Student Information Office
(Timetables, Special Consideration)
Merewether Building
Enter from City Road side
e-mail: [hidden email]
Phone: [phone]
----------------------------------
Executive Officer for the
Discipline of Accounting:
Kate Doyle
Room 302, Building H69
E-mail: : [hidden email]
Phone: [phone]
Fax: [phone]
---------------------------------

>>-----Original Message-----
>>From: SPSSX(r) Discussion [mailto:[hidden email]]
>>On Behalf Of Matthew Pirritano
>>Sent: Wednesday, April 02, 2008 10:36 AM
>>To: [hidden email]
>>Subject: Re: Order of Questions
>>
>>I have had the pleasure of administering similar sets of
>>questionnaires to groups of up to 40 9 year old elementary
>>school children. Maybe other folks will differ in their
>>opinions on this but my experience has been that if you read
>>the items aloud, to help keep everyone together, and keep it
>>at a good pace, a little faster that you think people can go,
>>100 questions can be quite painless. You can finish in as
>>little as 15 minutes. The key to reading the items aloud is
>>that people can process the information quicker and easier
>>than if they have to read each item themselves. Some people
>>are quick readers and will move ahead at their own pace. If
>>you actually include item numbers to the left and right of
>>the bubbles (assuming you're using a typical likert scale)
>>people can just move on down through the items, knowing that
>>they're on the correct question, without having to refer to
>>the written items.
>>
>>That's the long version of my 2 cents. Short version, I don't
>>think response fatigue for 100 items is necessarily a factor.
>>We tested thousands of elementary school students with,
>>believe it or not, really no complaints from them. I think
>>this is a good acid test. If little kids can do it...
>>
>>Good luck and good night,
>>Matt
>>
>>Matthew Pirritano, Ph.D.
>>Email: [hidden email]
>>Assistant Professor of Psychology
>>Smith Hall 116C
>>Chapman University
>>Department of Psychology
>>One University Drive
>>Orange, CA 92866
>>Telephone [phone]
>>FAX [phone]
>>
>>
>>
>>----- Original Message ----
>>From: Bob Schacht <[hidden email]>
>>To: [hidden email]
>>Sent: Tuesday, April 1, 2008 2:02:51 PM
>>Subject: Order of Questions
>>
>>I'm on a team that is developing a new questionnaire for
>>consumer satisfaction. We are trying to cover six domains of
>>satisfaction, and have developed a list of 90 - 100 items on
>>these six domains. Our plan is to pilot test these questions
>>on a set of participants who have previously responded to the
>>current survey (which is only about 10 questions). Based on
>>the results, we want to then eliminate redundant questions in
>>order to reduce the entire list of questions to about 24
>>questions, or about 4 questions from each of the six domains.
>>
>>Being aware of respondent fatigue (RF), it has occurred to us
>>to rotate the order of the questions, to avoid saddling the
>>last questions in the survey with the largest amount of RF.
>>Rather than randomize the order of the individual questions,
>>we thought to merely rotate the order in which each of the
>>six sets of questions are presented. So, for example, the
>>questions about the respondent's counselor would not always come last.
>>
>>On the other hand, we are also aware that the order in which
>>questions are asked can itself be a significant factor in how
>>questions are answered.
>>
>>What is the best way in such circumstances to control for the
>>order of the questions, respondent fatigue, etc.?
>>
>>Bob Schacht
>>
>>
>>
>>Robert M. Schacht, Ph.D. <[hidden email]> Pacific Basin
>>Rehabilitation Research & Training Center
>>1268 Young Street, Suite #204
>>Research Center, University of Hawaii
>>Honolulu, HI 96814
>>
>>=====================
>>To manage your subscription to SPSSX-L, send a message to
>>[hidden email] (not to SPSSX-L), with no body text
>>except the command. To leave the list, send the command
>>SIGNOFF SPSSX-L For a list of commands to manage
>>subscriptions, send the command INFO REFCARD
>>
>>=====================
>>To manage your subscription to SPSSX-L, send a message to
>>[hidden email] (not to SPSSX-L), with no body text
>>except the command. To leave the list, send the command
>>SIGNOFF SPSSX-L For a list of commands to manage
>>subscriptions, send the command INFO REFCARD
>>

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Arthur Kramer]

In regard to changing the order to avoid fatigue phenomena affecting the
questions at the end, the act of reordering may also affect the cognitive
set of the responder, as you stated. If you decide to not reorder, as
discussed by one commenter, ok; but if you do reorder consider creating
blocks of respondents who receive the different orders thereby balancing the
effects of the orderings and having the order nested within the blocks.

Arthur Kramer, Ph.D.
Director of Institutional Research
New Jersey City University

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Bob
Schacht
Sent: Tuesday, April 01, 2008 5:03 PM
To: [hidden email]
Subject: Order of Questions

I'm on a team that is developing a new questionnaire for consumer
satisfaction. We are trying to cover six domains of satisfaction, and have
developed a list of 90 - 100 items on these six domains. Our plan is to
pilot test these questions on a set of participants who have previously
responded to the current survey (which is only about 10 questions). Based
on the results, we want to then eliminate redundant questions in order to
reduce the entire list of questions to about 24 questions, or about 4
questions from each of the six domains.

Being aware of respondent fatigue (RF), it has occurred to us to rotate the
order of the questions, to avoid saddling the last questions in the survey
with the largest amount of RF. Rather than randomize the order of the
individual questions, we thought to merely rotate the order in which each
of the six sets of questions are presented. So, for example, the questions
about the respondent's counselor would not always come last.

On the other hand, we are also aware that the order in which questions are
asked can itself be a significant factor in how questions are answered.

What is the best way in such circumstances to control for the order of the
questions, respondent fatigue, etc.?

Bob Schacht



Robert M. Schacht, Ph.D. <[hidden email]>
Pacific Basin Rehabilitation Research & Training Center
1268 Young Street, Suite #204
Research Center, University of Hawaii
Honolulu, HI 96814

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[zstatman]

As a long time advocate of random presentation to overcome order bias I
recommend that approach here: Random ordering within domain. That said, and
clearly obvious, is the method of presenting the questionnaire. The method
should be applied for phone and Web, paper not so easy unless you create
several questionnaires and send out "randomly" but this could get both
expensive & complicated.

 WMB
Statistical Services

=========================================
mailto:[hidden email]
http://home.earthlink.net/~info.statman
=========================================

Virus Scan Notice: This email is certified to be virus free.

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of
Arthur Kramer
Sent: Wednesday, April 02, 2008 10:20 AM
To: [hidden email]
Subject: Re: Order of Questions

In regard to changing the order to avoid fatigue phenomena affecting the
questions at the end, the act of reordering may also affect the cognitive
set of the responder, as you stated. If you decide to not reorder, as
discussed by one commenter, ok; but if you do reorder consider creating
blocks of respondents who receive the different orders thereby balancing the
effects of the orderings and having the order nested within the blocks.

Arthur Kramer, Ph.D.
Director of Institutional Research
New Jersey City University

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Bob
Schacht
Sent: Tuesday, April 01, 2008 5:03 PM
To: [hidden email]
Subject: Order of Questions

I'm on a team that is developing a new questionnaire for consumer
satisfaction. We are trying to cover six domains of satisfaction, and have
developed a list of 90 - 100 items on these six domains. Our plan is to
pilot test these questions on a set of participants who have previously
responded to the current survey (which is only about 10 questions). Based on
the results, we want to then eliminate redundant questions in order to
reduce the entire list of questions to about 24 questions, or about 4
questions from each of the six domains.

Being aware of respondent fatigue (RF), it has occurred to us to rotate the
order of the questions, to avoid saddling the last questions in the survey
with the largest amount of RF. Rather than randomize the order of the
individual questions, we thought to merely rotate the order in which each of
the six sets of questions are presented. So, for example, the questions
about the respondent's counselor would not always come last.

On the other hand, we are also aware that the order in which questions are
asked can itself be a significant factor in how questions are answered.

What is the best way in such circumstances to control for the order of the
questions, respondent fatigue, etc.?

Bob Schacht



Robert M. Schacht, Ph.D. <[hidden email]> Pacific Basin Rehabilitation
Research & Training Center
1268 Young Street, Suite #204
Research Center, University of Hawaii
Honolulu, HI 96814

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command SIGNOFF SPSSX-L For a list of
commands to manage subscriptions, send the command INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command SIGNOFF SPSSX-L For a list of
commands to manage subscriptions, send the command INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD