On survey development issue

[Haijie Ding]

Dear Listers,
We are using an accademic survey (with 20 years of research) to do
consulting for our clients. The problem for us is we need to collect lots of
infomation from only one survey. So, we want to have a short version of the
original survey. We have collected the data by using the complete version
survey which contained more than 3k cases. When we did correlation analysis
, i found that the correlation between a subset of questions (i call it
short version survey) and the total survey is above 0.9. My question is: (1)
can i use the short version survey in stead of the complete survey in our
future consulting practice? (2) Is there anything besides correlation
analysis to do if i want to make sure the reliability and validity of the
short version survey? (3) we want to get these data published in the
peer-review journals, is it ok for us to do it by using the data from the
short version survey?

Thanks a lot.

Best regards,
Haijie

[Bob Walker-2]

Dear Haijie,Not sure what you mean by the "correlation between a subset of questions and the total survey" -- do you mean the correlation between certain questions and a dependent measure? And are the short version questions simply a reduced set using unchanged wording, or have you qualitatively changed the questions or attributes via data reduction, such as factor? Also, if you are trying to predict group membership, the hit-miss table in discriminant may be the more appropriate indicator...Bob WalkerSurveys & Forecasts, LLCwww.safllc.com----- Original Message -----From: Haijie Ding Date: Tuesday, June 12, 2007 3:16 amSubject: On survey development issueTo: [hidden email]> Dear Listers,> We are using an accademic survey (with 20 years of research) to do> consulting for our clients. The problem for us is we need to > collect lots of> infomation from only one survey. So, we want to have a short > version of the> original survey. We have collected the data by using the > complete version> survey which contained more than 3k cases. When we did > correlation analysis> , i found that the correlation between a subset of questions (i > call it> short version survey) and the total survey is above 0.9. My > question is: (1)> can i use the short version survey in stead of the complete > survey in our> future consulting practice? (2) Is there anything besides correlation> analysis to do if i want to make sure the reliability and > validity of the> short version survey? (3) we want to get these data published in the> peer-review journals, is it ok for us to do it by using the data > from the> short version survey?> > Thanks a lot.> > Best regards,> Haijie>

 Bob Walker
Surveys & Forecasts, LLC
www.safllc.com

[Haijie Ding]

Dear Bob,
Sorry for not explaining clearly. Correlation analysis mean the correlation
between part of the survey and the total survey. We didn't include any
dependent variable in the correlation analysis. Originally, we have 24
questions in this survey. Then we found that the correlation between part of
the survey (8 questions) and the total survey(24 questions) was above 0.9.
We only reduced the number of the the of questions and kept the questions as
original. So, what's more do we need to do to make sure the reliability and
validity of the short-versioned survey.

thanks a lot.

Haijie
On 6/13/07, [hidden email] <[hidden email]> wrote:

>
> Dear Haijie,
>
> Not sure what you mean by the "correlation between a subset of questions
> and the total survey" -- do you mean the correlation between certain
> questions and a dependent measure? And are the short version questions
> simply a reduced set using unchanged wording, or have you qualitatively
> changed the questions or attributes via data reduction, such as factor?
> Also, if you are trying to predict group membership, the hit-miss table in
> discriminant may be the more appropriate indicator...
>
> Bob Walker
> Surveys & Forecasts, LLC
> www.safllc.com
>
>
> ----- Original Message -----
> From: Haijie Ding
> Date: Tuesday, June 12, 2007 3:16 am
> Subject: On survey development issue
> To: [hidden email]
>
> > Dear Listers,
> > We are using an accademic survey (with 20 years of research) to do
> > consulting for our clients. The problem for us is we need to
> > collect lots of
> > infomation from only one survey. So, we want to have a short
> > version of the
> > original survey. We have collected the data by using the
> > complete version
> > survey which contained more than 3k cases. When we did
> > correlation analysis
> > , i found that the correlation between a subset of questions (i
> > call it
> > short version survey) and the total survey is above 0.9. My
> > question is: (1)
> > can i use the short version survey in stead of the complete
> > survey in our
> > future consulting practice? (2) Is there anything besides correlation
> > analysis to do if i want to make sure the reliability and
> > validity of the
> > short version survey? (3) we want to get these data published in the
> > peer-review journals, is it ok for us to do it by using the data
> > from the
> > short version survey?
> >
> > Thanks a lot.
> >
> > Best regards,
> > Haijie
> >
>
> Bob Walker
> Surveys & Forecasts, LLC
> www.safllc.com
>

[Haijie Ding]

Dear Paul,
Thank you for your reply. I will get more information on "scale reduction"
topic, can you share anything you have on this topic?
Thanks a lot.

Bests,
Haijie


On 6/12/07, Paul Thoresen <[hidden email]> wrote:

>
> Hello,
>
> You will get replies to this question I am sure.
> I would suggest you look up "scale reduction" or
> similar terms in a short lit search. It will likely
> be suggested that you do 1. reliability analysis of your scale
> to ensure the Chronbach alpha is similar,
> 2. as well as factor analysis and
> 3. potentially IRT.
>
> I can provide you at lweast one reference of an article of what we did
> in grad school if you are interested.
>
> I would not use straight correlations only.  Best of luck,
>
> Paul
>
> Paul Thoresen, M.A.
> Surveys & Research Manager
> The American Academy of Neurology
> 1080 Montreal Ave., St. Paul, MN 55116
> [hidden email] 651.695.2803
>
>
>
>
> NOTICE:  This message may contain confidential or legally privileged
> information intended only for the use of the addressee(s) named above.
> Unauthorized use, disclosure, distribution, or copying is prohibited.  If
> you have received this message in error, please reply to the sender and
> delete the original message.  Thank you.
> --------------------------------------------------------
>
>
>
> From: Haijie Ding [mailto:[hidden email]]
> Sent: Tuesday, June 12, 2007 2:14 AM
> To: [hidden email]
> Subject: On survey development issue
>
> Dear Listers,
> We are using an accademic survey (with 20 years of research) to do
> consulting for our clients. The problem for us is we need to collect
> lots of
> infomation from only one survey. So, we want to have a short version of
> the
> original survey. We have collected the data by using the complete
> version
> survey which contained more than 3k cases. When we did correlation
> analysis
> , i found that the correlation between a subset of questions (i call it
> short version survey) and the total survey is above 0.9. My question is:
> (1)
> can i use the short version survey in stead of the complete survey in
> our
> future consulting practice? (2) Is there anything besides correlation
> analysis to do if i want to make sure the reliability and validity of
> the
> short version survey? (3) we want to get these data published in the
> peer-review journals, is it ok for us to do it by using the data from
> the
> short version survey?
>
> Thanks a lot.
>
> Best regards,
> Haijie
>

[Paul Ginns-2]

Hi,

That most excellent developer of SPSS macros, Andrew Hayes, has
developed a macro for scale reduction - from
http://www.comm.ohio-state.edu/ahayes/ (under SPSS and SAS macros) -

ALPHAMAX.PDF

This paper describes an SPSS and SAS macro that generates all possible
subscales of at least two items from an additive scale containing k
items.  For each possible subscale, it generates Cronbach's alpha and
the subscale-full scale correlation and displays information about each
subscale in a data spread sheet.  It also generates summary statistics
making it easy to find the most psychometrically appealing subscale in
the set as well as some item analysis statistics useful for scale
construction.  To download the SPSS macro, click here.  For the SAS
version, click here.  See the paper for instructions on the use of the
macro.

Cheers,

Paul

------------------------------

Date:    Wed, 13 Jun 2007 10:02:15 +0800
From:    Haijie Ding <[hidden email]>
Subject: Re: On survey development issue

Dear Paul,
Thank you for your reply. I will get more information on "scale
reduction"
topic, can you share anything you have on this topic?
Thanks a lot.

Bests,
Haijie

[Paul Thoresen]

Look up this article. This was of great use to us:

 


Personnel Psychology


Volume 55 Issue 1 Page 167 Issue 1 - 194 - March 2002

JEFFREY M. STANTON, EVAN F. SINAR, WILLIAM K. BALZER, PATRICIA C. SMITH
(2002)
ISSUES AND STRATEGIES FOR REDUCING THE LENGTH OF SELF-REPORT SCALES
Personnel Psychology 55 (1), 167-194.

 

Good luck,

 

Paul


 
NOTICE:  This message may contain confidential or legally privileged information intended only for the use of the addressee(s) named above. Unauthorized use, disclosure, distribution, or copying is prohibited.  If you have received this message in error, please reply to the sender and delete the original message.  Thank you.
--------------------------------------------------------




From: Haijie Ding [mailto:[hidden email]]
Sent: Tuesday, June 12, 2007 9:02 PM
To: Paul Thoresen
Cc: [hidden email]
Subject: Re: On survey development issue

 

Dear Paul,
Thank you for your reply. I will get more information on "scale
reduction" topic, can you share anything you have on this topic?
Thanks a lot.

Bests,
Haijie



On 6/12/07, Paul Thoresen <[hidden email]> wrote:

Hello,

You will get replies to this question I am sure.
I would suggest you look up "scale reduction" or
similar terms in a short lit search. It will likely
be suggested that you do 1. reliability analysis of your scale
to ensure the Chronbach alpha is similar,
2. as well as factor analysis and
3. potentially IRT.

I can provide you at lweast one reference of an article of what we did
in grad school if you are interested.

I would not use straight correlations only.  Best of luck,

Paul

Paul Thoresen, M.A.
Surveys & Research Manager
The American Academy of Neurology
1080 Montreal Ave., St. Paul, MN 55116
[hidden email] 651.695.2803




NOTICE:  This message may contain confidential or legally privileged
information intended only for the use of the addressee(s) named above.
Unauthorized use, disclosure, distribution, or copying is prohibited.
If you have received this message in error, please reply to the sender
and delete the original message.  Thank you.
--------------------------------------------------------



From: Haijie Ding [mailto:[hidden email]]
Sent: Tuesday, June 12, 2007 2:14 AM
To: [hidden email]
Subject: On survey development issue

Dear Listers,
We are using an accademic survey (with 20 years of research) to do
consulting for our clients. The problem for us is we need to collect
lots of
infomation from only one survey. So, we want to have a short version of
the
original survey. We have collected the data by using the complete
version
survey which contained more than 3k cases. When we did correlation
analysis
, i found that the correlation between a subset of questions (i call it
short version survey) and the total survey is above 0.9. My question is:
(1)
can i use the short version survey in stead of the complete survey in
our
future consulting practice? (2) Is there anything besides correlation
analysis to do if i want to make sure the reliability and validity of
the
short version survey? (3) we want to get these data published in the
peer-review journals, is it ok for us to do it by using the data from
the
short version survey?

Thanks a lot.

Best regards,
Haijie

[Haijie Ding]

Dear Paul and Paul,

Thank you very much on this valuable info. I got both Prof. Hayes' and Prof
Stanton's papers and prgogram. They are quite useful and insightful. I just
fount i have much more to do to make sure the reliability and validity of
the reduced scale version of the survey.
Thanks a lot for both Prof. hayes and Stanton to make researchers' life much
easier on this topic.


Bests,
Haijie

On 6/13/07, Paul Ginns <[hidden email]> wrote:

>
> Hi,
>
> That most excellent developer of SPSS macros, Andrew Hayes, has
> developed a macro for scale reduction - from
> http://www.comm.ohio-state.edu/ahayes/ (under SPSS and SAS macros) -
>
> ALPHAMAX.PDF
>
> This paper describes an SPSS and SAS macro that generates all possible
> subscales of at least two items from an additive scale containing k
> items.  For each possible subscale, it generates Cronbach's alpha and
> the subscale-full scale correlation and displays information about each
> subscale in a data spread sheet.  It also generates summary statistics
> making it easy to find the most psychometrically appealing subscale in
> the set as well as some item analysis statistics useful for scale
> construction.  To download the SPSS macro, click here.  For the SAS
> version, click here.  See the paper for instructions on the use of the
> macro.
>
> Cheers,
>
> Paul
>
> ------------------------------
>
> Date:    Wed, 13 Jun 2007 10:02:15 +0800
> From:    Haijie Ding <[hidden email]>
> Subject: Re: On survey development issue
>
> Dear Paul,
> Thank you for your reply. I will get more information on "scale
> reduction"
> topic, can you share anything you have on this topic?
> Thanks a lot.
>
> Bests,
> Haijie
>