SPSS reliability stat

[albert murillo-2]

Hello all,

I would like to start by saying I have found the listserv to be friendly and
helpful.  Also, I am new in my applied position as a recent Ph.D. graduate.

My question:

I have created six applied exams trying to measure the KSAO's for six
distinct job positions.  The exams were created using a very informative job
analysis.  The exams consists of 100 items each with choices of correct
answers a, b, c, or d.  When I run reliabilities in SPSS my coefficients are
quit low and I get an error message "The value is negative due to a negative
average covariance among items.  This violates reliability model
assumptions.  You may want to check item coding"  Not sure what this means?
What about the item coding?  I never had dealt with a negative reliability
coefficient.  However, my skewness and kurtosis are within + or - 1 which
would indicate the sample coming from a normal distribution.  My sample
ranges from 104 subjects to 34 subject depending on which particular test.
The SPSS results for all tests are the same.

I appreciate the help and time for those who will respond.

Thank you,

Albert

_________________________________________________________________
Get a preview of Live Earth, the hottest event this summer - only on MSN
http://liveearth.msn.com?source=msntaglineliveearthhm

[Hector Maletta]

         Albert,
         You want that all your questions are coded in the same "direction",
from the "worst" to the "best" answer (or the converse). In many tests, the
answers (a,b,c,d in your case) are presented to the respondent in random
order, but for reliability analysis they should be reordered in a way that
makes sense. Once this is done, the questions should have positive
covariance, i.e. they should correlate positively with each other. People
scoring better in one question would tend to score better in other questions
too.

         Hector

         -----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of
albert murillo
Sent: 27 June 2007 10:28
To: [hidden email]
Subject: SPSS reliability stat

         Hello all,

         I would like to start by saying I have found the listserv to be
friendly and
         helpful.  Also, I am new in my applied position as a recent Ph.D.
graduate.

         My question:

         I have created six applied exams trying to measure the KSAO's for
six
         distinct job positions.  The exams were created using a very
informative job
         analysis.  The exams consists of 100 items each with choices of
correct
         answers a, b, c, or d.  When I run reliabilities in SPSS my
coefficients are
         quit low and I get an error message "The value is negative due to a
negative
         average covariance among items.  This violates reliability model
         assumptions.  You may want to check item coding"  Not sure what
this means?
         What about the item coding?  I never had dealt with a negative
reliability
         coefficient.  However, my skewness and kurtosis are within + or - 1
which
         would indicate the sample coming from a normal distribution.  My
sample
         ranges from 104 subjects to 34 subject depending on which
particular test.
         The SPSS results for all tests are the same.

         I appreciate the help and time for those who will respond.

         Thank you,

         Albert

         _________________________________________________________________
         Get a preview of Live Earth, the hottest event this summer - only
on MSN
         http://liveearth.msn.com?source=msntaglineliveearthhm

[Art Kendall-2]

Item coding:  check that when you transformed the raw scores into a new
set of dichotomous variables that you correctly indicated which was the
correct answer.

double check that you are summing the correct dichotomies.

Interitem correlations should have only trivial negative coefficients,
if any. Inspect the matrix of interitem correlations to see which items
in a scale have negative correlations.

When you get closer to a final analysis, check for negative item-total
correlations.

btw: KSAO's are Knowledge, skills, abiilities and what?

Art Kendall
Social Research Consultants


albert murillo wrote:

> Hello all,
>
> I would like to start by saying I have found the listserv to be
> friendly and
> helpful.  Also, I am new in my applied position as a recent Ph.D.
> graduate.
>
> My question:
>
> I have created six applied exams trying to measure the KSAO's for six
> distinct job positions.  The exams were created using a very
> informative job
> analysis.  The exams consists of 100 items each with choices of correct
> answers a, b, c, or d.  When I run reliabilities in SPSS my
> coefficients are
> quit low and I get an error message "The value is negative due to a
> negative
> average covariance among items.  This violates reliability model
> assumptions.  You may want to check item coding"  Not sure what this
> means?
> What about the item coding?  I never had dealt with a negative
> reliability
> coefficient.  However, my skewness and kurtosis are within + or - 1 which
> would indicate the sample coming from a normal distribution.  My sample
> ranges from 104 subjects to 34 subject depending on which particular
> test.
> The SPSS results for all tests are the same.
>
> I appreciate the help and time for those who will respond.
>
> Thank you,
>
> Albert
>
> _________________________________________________________________
> Get a preview of Live Earth, the hottest event this summer - only on MSN
> http://liveearth.msn.com?source=msntaglineliveearthhm
>
>

[Mark A Davenport MADAVENP]

This sometimes happens.  David Nichols at SPSS put out a white paper about
this some time ago. I am scaring that up for you and will send it directly
to your address, not to the list.  I noticed that Hector and Art already
replied to your note.  I agree that your scale, if more along the lines of
'multiple-choice' rather than Likert scale, could be a problem.  When
getting reliability statistitcs (see the Statistics button on the bottom
of the Reliability pup-up) ask for item, scale, and scale if item deleted
statistics.  This may give you a clue as to the offender.  However, may
also want to look closer at your items by doing something more akin to
item analysis (item difficulty, item discrimination, etc.) especially if
you created these exams yourself.

Mark




***************************************************************************************************************************************************************
Mark A. Davenport Ph.D.
Senior Research Analyst
Office of Institutional Research
The University of North Carolina at Greensboro
336.256.0395
[hidden email]

'An approximate answer to the right question is worth a good deal more
than an exact answer to an approximate question.' --a paraphrase of J. W.
Tukey (1962)






albert murillo <[hidden email]>
Sent by: "SPSSX(r) Discussion" <[hidden email]>
06/27/2007 09:29 AM
Please respond to
albert murillo <[hidden email]>


To
[hidden email]
cc

Subject
SPSS reliability stat






Hello all,

I would like to start by saying I have found the listserv to be friendly
and
helpful.  Also, I am new in my applied position as a recent Ph.D.
graduate.

My question:

I have created six applied exams trying to measure the KSAO's for six
distinct job positions.  The exams were created using a very informative
job
analysis.  The exams consists of 100 items each with choices of correct
answers a, b, c, or d.  When I run reliabilities in SPSS my coefficients
are
quit low and I get an error message "The value is negative due to a
negative
average covariance among items.  This violates reliability model
assumptions.  You may want to check item coding"  Not sure what this
means?
What about the item coding?  I never had dealt with a negative reliability
coefficient.  However, my skewness and kurtosis are within + or - 1 which
would indicate the sample coming from a normal distribution.  My sample
ranges from 104 subjects to 34 subject depending on which particular test.
The SPSS results for all tests are the same.

I appreciate the help and time for those who will respond.

Thank you,

Albert

_________________________________________________________________
Get a preview of Live Earth, the hottest event this summer - only on MSN
http://liveearth.msn.com?source=msntaglineliveearthhm

[Pirritano, Matthew]

Hello list,

How can I best interpret the estimates that I get from running a mixed multilevel model? Are they like unstandardized estimates in regression? Should I standardize my variables (z-score, or center) before running this analysis?

Thanks,
Matt

Matthew Pirritano, Ph.D.
Assistant Professor of Psychology
Smith Hall 116C
Chapman University
Department of Psychology
One University Drive
Orange, CA 92866
Telephone [phone]
FAX [phone]

[peter link]

Matt -

The parameter estimates of the fixed effects are similar to the
unstandardized beta in a linear regression.

Centering can be very useful and help aid in your interpretation of results.
Choosing values to center a variable around is important.

I would not standardize variables.  I actually have not seen too much of
this done in the literature, to be honest.

Matt, I would recommend the following book.  Pick it up, it will help you
out tremendously.

Singer & Willett, Applied Longitudinal Data Analysis, Oxford University
Press, 2003.

There are several other really good books out there on this type of
modelling, as well, each with something unique to offer.  The Singer &
Willett book is not very technical, but very thorough, and with an applied
researcher as the primary audience in mind.

Peter Link
VA San Diego Healthcare System

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]]On Behalf Of
Pirritano, Matthew
Sent: Wednesday, June 27, 2007 8:04 AM
To: [hidden email]
Subject: what is the meaning of estimates in mixed linear models?


Hello list,

How can I best interpret the estimates that I get from running a mixed
multilevel model? Are they like unstandardized estimates in regression?
Should I standardize my variables (z-score, or center) before running this
analysis?

Thanks,
Matt

Matthew Pirritano, Ph.D.
Assistant Professor of Psychology
Smith Hall 116C
Chapman University
Department of Psychology
One University Drive
Orange, CA 92866
Telephone [phone]
FAX [phone]

[Swank, Paul R]

Centering is good but I don't recommend standardization. The fixed
estimates are precisely like regression coefficients.


Paul R. Swank, Ph.D.
Professor, Developmental Pediatrics
Director of Research,


University of Texas Health Science Center at Houston

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of
Pirritano, Matthew
Sent: Wednesday, June 27, 2007 10:04 AM
To: [hidden email]
Subject: what is the meaning of estimates in mixed linear models?

Hello list,

How can I best interpret the estimates that I get from running a mixed
multilevel model? Are they like unstandardized estimates in regression?
Should I standardize my variables (z-score, or center) before running
this analysis?

Thanks,
Matt

Matthew Pirritano, Ph.D.
Assistant Professor of Psychology
Smith Hall 116C
Chapman University
Department of Psychology
One University Drive
Orange, CA 92866
Telephone [phone]
FAX [phone]