What is wrong with treating Likert scale as interval data?

[puryfiz]

Dear all
First off, this is my first post here, so I would like to say hello to everyone.
I would like to ask question - in other words - what are the consequences of including variables measured on Likert scale in, for example, regression model or PCA?
Thank you in advance
Cezary

[Art Kendall]

It all depends. (What a surprise that a statistician/psychologist former philosopher would say that!)

Do you mean a well developed Likert scale as the sum or mean of a set of items with  strongly disagree, disagree, neutral, agree, and strongly agree value labels as the response scale?

Where did the scale come from? Has it be evaluated for reliability and validity?
How many scales are there?
How many items does each have?
What is the response scale as presented to the respondent?

How was the scoring key developed?
A sort-of example of the syntax for a scoring key for a single attitude construct.
COMPUTE AttitudeName1 = mean.5(item1, Item4, Item11, ReversedItem13, ReversedItem21, ReversedItem22).

[puryfiz]

Thanks for answer. I wasn't specific enough. My question is purely theoretical and it concerns using variables measured on Likert scale with 5 items. I know the more items given scale has, the better, but still there are purists who do not use Likert scale variables in techniques which require continuous data. And I would like to know why.
From what I read one of the problems that emerge is skewness (because answers may tend to concentrate on agree/strongly agree values).
What else should I be worried about?

[Bruce Weaver]

If you Google on <parametric analysis Likert items> and similar terms, you should find lots of material.  For example:

Carifio, J. and Perla, R. J. (2007). Ten Common Misunderstandings, Misconceptions, Persistent Myths and Urban Legends about Likert Scales and Likert Response Formats and their Antidotes Journal of Social Sciences 3 (3), 106-116. Accessed at: thescipub.com/pdf/10.3844/jssp.2007.106.116

See especially the section headed, "F Is Not Made of Glass" (p. 110).

puryfiz wrote

Thanks for answer. I wasn't specific enough. My question is purely theoretical and it concerns using variables measured on Likert scale with 5 items. I know the more items given scale has, the better, but still there are purists who do not use Likert scale variables in techniques which require continuous data. And I would like to know why.
From what I read one of the problems that emerge is skewness (because answers may tend to concentrate on agree/strongly agree values).
What else should I be worried about?

[Ryan]

I would employ an adjacent-category logit model to evaluate the extent to which the thresholds* are ordered and uniform across items before devising a scoring system. 

By thresholds, I am referring to the trait level at which someone has an equal probability of endorsing adjacent response options.

Ryan

On Mon, Jun 16, 2014 at 3:43 PM, Art Kendall <[hidden email]> wrote:

It all depends. (What a surprise that a statistician/psychologist former
philosopher would say that!)

Do you mean a well developed Likert scale as the sum or mean of a set of
items with  strongly disagree, disagree, neutral, agree, and strongly agree
value labels as the response scale?

Where did the scale come from? Has it be evaluated for reliability and
validity?
How many scales are there?
How many items does each have?
What is the response scale as presented to the respondent?

How was the scoring key developed?
A sort-of example of the syntax for a scoring key for a single attitude
construct.
COMPUTE AttitudeName1 = mean.5(item1, Item4, Item11, ReversedItem13,
ReversedItem21, ReversedItem22).








-----
Art Kendall
Social Research Consultants
--
View this message in context: http://spssx-discussion.1045642.n5.nabble.com/What-is-wrong-with-treating-Likert-scale-as-interval-data-tp5726488p5726491.html
Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[puryfiz]

Thank you all! I just stumbled upon this article

[Rich Ulrich]

The purists you speak of?  I have run into a few, but I have never run
into any who gave reliable advice on data analysis.  A couple of them
were psychology professors who taught statistics, unfortunately.

On the other hand, a "Likert scale" is supposed to be made up of
decently distributed items... so skewness is not likely to be a problem.
But if it is:  For additive scales which cluster at an extreme, I've had
success a few times when I used the square root transformation, taking 
   SQRT(distance from the extreme). 

What is generally the worst solution is retreating to the so-called
"nonparametric" statistics based on rank-orders, which is what those
same professors were taught to prefer in the 1980s, based on
misunderstood lore from the 1960s.

The modern and more technical approaches to scale development find
and adjust and document the distances between scale points.  See Rasch
models, Item Response Theory.

--
Rich Ulrich

> Date: Mon, 16 Jun 2014 13:23:00 -0700

> From: [hidden email]
> Subject: Re: What is wrong with treating Likert scale as interval data?
> To: [hidden email]
>
> Thanks for answer. I wasn't specific enough. My question is purely
> theoretical and it concerns using /variables/ measured on Likert scale with
> 5 items. I know the more items given scale has, the better, but still there
> are purists who do not use Likert scale variables in techniques which
> require continuous data. And I would like to know why.
> From what I read one of the problems that emerge is skewness (because
> answers may tend to concentrate on agree/strongly agree values).
> What else should I be worried about?
>
>

[Bruce Weaver]

Here's another interesting article on the use of binary and 5-point Likert-type variables in confirmatory factor analysis:

http://www.statmodel.com/download/floracurran.pdf

Abstract:
Confirmatory factor analysis (CFA) is widely used for examining hypothesized relations among ordinal variables (e.g., Likert-type items). A theoretically appropriate method fits the CFA model to polychoric correlations using either weighted least squares (WLS) or robust WLS. Importantly, this approach assumes that a continuous, normal latent process determines each observed variable. The extent to which violations of this assumption undermine CFA estimation is not well-known. In this article, the authors empirically study this issue using a computer simulation study. The results suggest that estimation of polychoric correlations is robust to modest violations of underlying normality. Further, WLS performed adequately only at the largest sample size but led to substantial estimation difficulties with smaller
samples. Finally, robust WLS performed well across all conditions.

Bruce Weaver wrote

If you Google on <parametric analysis Likert items> and similar terms, you should find lots of material.  For example:

Carifio, J. and Perla, R. J. (2007). Ten Common Misunderstandings, Misconceptions, Persistent Myths and Urban Legends about Likert Scales and Likert Response Formats and their Antidotes Journal of Social Sciences 3 (3), 106-116. Accessed at: thescipub.com/pdf/10.3844/jssp.2007.106.116

See especially the section headed, "F Is Not Made of Glass" (p. 110).

puryfiz wrote

Thanks for answer. I wasn't specific enough. My question is purely theoretical and it concerns using variables measured on Likert scale with 5 items. I know the more items given scale has, the better, but still there are purists who do not use Likert scale variables in techniques which require continuous data. And I would like to know why.
From what I read one of the problems that emerge is skewness (because answers may tend to concentrate on agree/strongly agree values).
What else should I be worried about?

[James Whanger]

This is a conceptual and theoretical issue that can have important implications for both logical and statistical inference. There are two issues of concern, first is whether the method applied to assessing the nature of a phenomena under investigation is logically defensible and second is whether the amount and type of error associated with the method is practically important. 

If we are measuring economic value using the U.S. Dollar, then we know that the difference between $10 and $9 is precisely the same as the difference between $5,010 and $5,009. We could have 9 financial analysts randomly chosen with regard to experience and level of expertise who would likely come to the same conclusion 9 out of 9 times. 

In contrast, we could also have 9 carpenters randomly selected with regard to experience and level of expertise and ask them to cut a 24 inch piece from a wooden board.  If we then had an expert judge compare each of the 9 boards against a 24 inch 'standard' from the U.S. Department of Weights and Measures we would likely get some variation around the 24 inch mark with the average very close to 24 inches. 

The impact of this measurement error in each case would depend on the context within which the board were to be used and it's impact on the prediction of performance and durability of the product for which the board is used.  This impact can range from entirely unimportant to utterly critical. The statistical method used would depend on the focus of the question. If one wanted to assess the consistency of a group of carpenters across a series of cuts, one could use an ICC or if there was variation across cuts in terms of difficulty of performance, one might fit a two parameter Rasch model or to compare the variation between two groups on a single cut an ANOVA, etc..

Let's compare these continuous measures with an ordinal assessment. Suppose we created 3 items to assess individual differences in the level of importance carpenters placed on precise measurement using a 5pt. Likert type scale. The major theoretical issues are on the front end in first mapping out the phenomenon of interest from similar, but conceptually distinct 

Instructions:  Please respond to the following questions based on your current work project using this scale.

1-Strongly Disagree, 2-Disagree, 3-Neither Agree nor Disagree, 4-Agree, 5-Strongly Agree

Item 1: I believe in the old saying, 'measure twice, cut once'.

Item 2: Precision of measurement is not that important for my job.

Item 3: I only buy tools that result in a high level of precision.

The randomness of our sample of carpenters should also provide variation across type of application and thus the need. So now instead of concatenating units or fractions of precisely the same thing, we are adding or averaging numbers that represent conceptually disparate information and treating it as if it is precisely the same. Thus, if one enters a summated or averaged score of 3 items into a regression equation, the assumption is that 

y = b*x + e where y is the outcome variable, x the dependent variable, b the coefficient and e the error term. The interpretation is then the logically incomprehensible idea that y = a fraction of degree of agreement with (I believe in the old saying, 'measure twice, cut once') + (Precision of measurement is not that important for my job) + (I only buy tools that result in a high level of precision).

The alternative is to use methods designed for ordinal categorical variables.

On Tue, Jun 17, 2014 at 8:44 PM, Bruce Weaver <[hidden email]> wrote:

Here's another interesting article on the use of binary and 5-point
Likert-type variables in confirmatory factor analysis:

http://www.statmodel.com/download/floracurran.pdf

Abstract:
Confirmatory factor analysis (CFA) is widely used for examining hypothesized
relations among ordinal variables (e.g., Likert-type items). A theoretically
appropriate method fits the CFA model to polychoric correlations using
either weighted least squares (WLS) or robust WLS. Importantly, this
approach assumes that a continuous, normal latent process determines each
observed variable. The extent to which violations of this assumption
undermine CFA estimation is not well-known. In this article, the authors
empirically study this issue using a computer simulation study. The results
suggest that estimation of polychoric correlations is robust to modest
violations of underlying normality. Further, WLS performed adequately only
at the largest sample size but led to substantial estimation difficulties
with smaller
samples. Finally, robust WLS performed well across all conditions.





Bruce Weaver wrote

> If you Google on
> <parametric analysis Likert items>
>  and similar terms, you should find lots of material.  For example:
>
> Carifio, J. and Perla, R. J. (2007). Ten Common Misunderstandings,
> Misconceptions, Persistent Myths and Urban Legends about Likert Scales and
> Likert Response Formats and their Antidotes Journal of Social Sciences 3
> (3), 106-116. Accessed at: thescipub.com/pdf/10.3844/jssp.2007.106.116
>
> See especially the section headed, "F Is Not Made of Glass" (p. 110).
>
> puryfiz wrote
>> Thanks for answer. I wasn't specific enough. My question is purely
>> theoretical and it concerns using
/
>> variables
/
>>  measured on Likert scale with 5 items. I know the more items given scale
>> has, the better, but still there are purists who do not use Likert scale
>> variables in techniques which require continuous data. And I would like
>> to know why.
>> From what I read one of the problems that emerge is skewness (because
>> answers may tend to concentrate on agree/strongly agree values).
>> What else should I be worried about?





-----
--
Bruce Weaver
[hidden email]
http://sites.google.com/a/lakeheadu.ca/bweaver/

"When all else fails, RTFM."

NOTE: My Hotmail account is not monitored regularly.
To send me an e-mail, please use the address shown above.

--

View this message in context: http://spssx-discussion.1045642.n5.nabble.com/What-is-wrong-with-treating-Likert-scale-as-interval-data-tp5726488p5726514.html

Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

--

James C. Whanger

[James Whanger]

Notes of correction and clarification: 

1. The major theoretical issues are on the front end in first mapping out the phenomenon of interest from similar, but conceptually distinct phenomena.

2. x is the independent variable (not the dependent as I mistakenly typed)

3. y = a + b*x + e:  The interpretation of the relationship between the ordered categorical variable based on summated likert items is that a 1 unit increase in  (I believe in the old saying, 'measure twice, cut once') + (Precision of measurement is not that important for my job) + (I only buy tools that result in a high level of precision)  results in an a +  b* (I believe in the old saying, 'measure twice, cut once') + (Precision of measurement is not that important for my job) + (I only buy tools that result in a high level of precision) + e increase in y.

4. Alternatives include Bayesian estimation procedures applied to ordered categorical variables

On Wed, Jun 18, 2014 at 3:33 PM, James C. Whanger <[hidden email]> wrote:

This is a conceptual and theoretical issue that can have important implications for both logical and statistical inference. There are two issues of concern, first is whether the method applied to assessing the nature of a phenomena under investigation is logically defensible and second is whether the amount and type of error associated with the method is practically important. 

If we are measuring economic value using the U.S. Dollar, then we know that the difference between $10 and $9 is precisely the same as the difference between $5,010 and $5,009. We could have 9 financial analysts randomly chosen with regard to experience and level of expertise who would likely come to the same conclusion 9 out of 9 times. 

In contrast, we could also have 9 carpenters randomly selected with regard to experience and level of expertise and ask them to cut a 24 inch piece from a wooden board.  If we then had an expert judge compare each of the 9 boards against a 24 inch 'standard' from the U.S. Department of Weights and Measures we would likely get some variation around the 24 inch mark with the average very close to 24 inches. 

The impact of this measurement error in each case would depend on the context within which the board were to be used and it's impact on the prediction of performance and durability of the product for which the board is used.  This impact can range from entirely unimportant to utterly critical. The statistical method used would depend on the focus of the question. If one wanted to assess the consistency of a group of carpenters across a series of cuts, one could use an ICC or if there was variation across cuts in terms of difficulty of performance, one might fit a two parameter Rasch model or to compare the variation between two groups on a single cut an ANOVA, etc..

Let's compare these continuous measures with an ordinal assessment. Suppose we created 3 items to assess individual differences in the level of importance carpenters placed on precise measurement using a 5pt. Likert type scale. The major theoretical issues are on the front end in first mapping out the phenomenon of interest from similar, but conceptually distinct 

Instructions:  Please respond to the following questions based on your current work project using this scale.

1-Strongly Disagree, 2-Disagree, 3-Neither Agree nor Disagree, 4-Agree, 5-Strongly Agree

Item 1: I believe in the old saying, 'measure twice, cut once'.

Item 2: Precision of measurement is not that important for my job.

Item 3: I only buy tools that result in a high level of precision.

The randomness of our sample of carpenters should also provide variation across type of application and thus the need. So now instead of concatenating units or fractions of precisely the same thing, we are adding or averaging numbers that represent conceptually disparate information and treating it as if it is precisely the same. Thus, if one enters a summated or averaged score of 3 items into a regression equation, the assumption is that 

y = b*x + e where y is the outcome variable, x the dependent variable, b the coefficient and e the error term. The interpretation is then the logically incomprehensible idea that y = a fraction of degree of agreement with (I believe in the old saying, 'measure twice, cut once') + (Precision of measurement is not that important for my job) + (I only buy tools that result in a high level of precision).

The alternative is to use methods designed for ordinal categorical variables.

On Tue, Jun 17, 2014 at 8:44 PM, Bruce Weaver <[hidden email]> wrote:

Here's another interesting article on the use of binary and 5-point
Likert-type variables in confirmatory factor analysis:

http://www.statmodel.com/download/floracurran.pdf

Abstract:
Confirmatory factor analysis (CFA) is widely used for examining hypothesized
relations among ordinal variables (e.g., Likert-type items). A theoretically
appropriate method fits the CFA model to polychoric correlations using
either weighted least squares (WLS) or robust WLS. Importantly, this
approach assumes that a continuous, normal latent process determines each
observed variable. The extent to which violations of this assumption
undermine CFA estimation is not well-known. In this article, the authors
empirically study this issue using a computer simulation study. The results
suggest that estimation of polychoric correlations is robust to modest
violations of underlying normality. Further, WLS performed adequately only
at the largest sample size but led to substantial estimation difficulties
with smaller
samples. Finally, robust WLS performed well across all conditions.





Bruce Weaver wrote

> If you Google on
> <parametric analysis Likert items>
>  and similar terms, you should find lots of material.  For example:
>
> Carifio, J. and Perla, R. J. (2007). Ten Common Misunderstandings,
> Misconceptions, Persistent Myths and Urban Legends about Likert Scales and
> Likert Response Formats and their Antidotes Journal of Social Sciences 3
> (3), 106-116. Accessed at: thescipub.com/pdf/10.3844/jssp.2007.106.116
>
> See especially the section headed, "F Is Not Made of Glass" (p. 110).
>
> puryfiz wrote
>> Thanks for answer. I wasn't specific enough. My question is purely
>> theoretical and it concerns using
/
>> variables
/
>>  measured on Likert scale with 5 items. I know the more items given scale
>> has, the better, but still there are purists who do not use Likert scale
>> variables in techniques which require continuous data. And I would like
>> to know why.
>> From what I read one of the problems that emerge is skewness (because
>> answers may tend to concentrate on agree/strongly agree values).
>> What else should I be worried about?





-----
--
Bruce Weaver
[hidden email]
http://sites.google.com/a/lakeheadu.ca/bweaver/

"When all else fails, RTFM."

NOTE: My Hotmail account is not monitored regularly.
To send me an e-mail, please use the address shown above.

--

View this message in context: http://spssx-discussion.1045642.n5.nabble.com/What-is-wrong-with-treating-Likert-scale-as-interval-data-tp5726488p5726514.html

Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

--

James C. Whanger

--

James C. Whanger

[Rich Ulrich]

I guess I have to respond to this one point, because it is so close to
being thoroughly wrong.

That is, "dollars" are not great starting point for illustrating "interval"
versus "ordinal" variables;  I have used almost the same example to make
the *opposite* point.  I start with Tukey's advice, that if your largest data
value is 10 times the smallest, you *might* consider a transformation;
if it is 20 times the smallest, you *certainly*  consider one. 

$5000 versus $10?  That raises the "20 times" question.  What are these
dollars representing?  Is this a silly equation, the mixes the cost of a used
car with the cost of a used DVD?   In social science, the ratio of changes
is apt to be relevant for numbers like these.

"If we are measuring economic value" is the key: The DVs and IV need
to be equal-interval with respect to each other.  If you are stuck with dollars
for "economic value", then you use dollars.  That raises further problems,
so it is not a fine solution that should be generalized as a "good example".

We *prefer* to match up the scales, in addition to having homogeneity of error
across the range, and hopefully seeing a normal distribution in some suitable
population.  If we are measuring latent constructs with arbitrary scales, we
are not as constricted in our scoring as "economic value in dollars".

--
Rich Ulrich

---

Date: Wed, 18 Jun 2014 15:33:50 -0400
From: [hidden email]
Subject: Re: What is wrong with treating Likert scale as interval data?
To: [hidden email]

...

If we are measuring economic value using the U.S. Dollar, then we know that the difference between $10 and $9 is precisely the same as the difference between $5,010 and $5,009. We could have 9 financial analysts randomly chosen with regard to experience and level of expertise who would likely come to the same conclusion 9 out of 9 times. 

...

[James Whanger]

Rich,

You seem to be misunderstanding my argument.  The dollars example was simply to highlight the precision with which we can describe interval differences when using a truly continuous variable. When we move from this directness to a somewhat less direct measurement method, that of distance using a tape measure, we begin to see variation in that precision that would show up as measurement error in my example. or sampling error if the mean resulted in a value substantially different from 24 inches. When we move from this level of directness to that described by the Likert scale ordered categorical assessment, a method which involves the representation of disparate conceptual ideas as if they are not disparate, we are effectively creating variables that are far more crude than the representative numerals assigned them imply. At this point we typically apply mathematics that do not make sense unless we are concatenating intervals of information that are identical, when in fact there exist mathematics that allow us to describe the response distributions in a way that makes sense. My point is simply that given this option, there is little reason to use methods that are inconsistent with the nature of the phenomenon being assessed and the crudeness of the method with which it is being assessed. 

Best,

James

On Thu, Jun 19, 2014 at 6:58 PM, Rich Ulrich <[hidden email]> wrote:

I guess I have to respond to this one point, because it is so close to
being thoroughly wrong.

That is, "dollars" are not great starting point for illustrating "interval"
versus "ordinal" variables;  I have used almost the same example to make
the *opposite* point.  I start with Tukey's advice, that if your largest data
value is 10 times the smallest, you *might* consider a transformation;
if it is 20 times the smallest, you *certainly*  consider one. 

$5000 versus $10?  That raises the "20 times" question.  What are these
dollars representing?  Is this a silly equation, the mixes the cost of a used
car with the cost of a used DVD?   In social science, the ratio of changes
is apt to be relevant for numbers like these.

"If we are measuring economic value" is the key: The DVs and IV need
to be equal-interval with respect to each other.  If you are stuck with dollars
for "economic value", then you use dollars.  That raises further problems,
so it is not a fine solution that should be generalized as a "good example".

We *prefer* to match up the scales, in addition to having homogeneity of error
across the range, and hopefully seeing a normal distribution in some suitable
population.  If we are measuring latent constructs with arbitrary scales, we
are not as constricted in our scoring as "economic value in dollars".

--
Rich Ulrich

---

Date: Wed, 18 Jun 2014 15:33:50 -0400
From: [hidden email]

Subject: Re: What is wrong with treating Likert scale as interval data?
To: [hidden email]

...

If we are measuring economic value using the U.S. Dollar, then we know that the difference between $10 and $9 is precisely the same as the difference between $5,010 and $5,009. We could have 9 financial analysts randomly chosen with regard to experience and level of expertise who would likely come to the same conclusion 9 out of 9 times. 

...

--

James C. Whanger

[Kornbrot, Diana]

<base href="x-msg://1144/">I love turkey, but the advice as quoted is unhelpful

what mater is the DISTRIBUTION 

best

diana

On 19 Jun 2014, at 23:58, Rich Ulrich <[hidden email]> wrote:

I guess I have to respond to this one point, because it is so close to 
being thoroughly wrong. 

That is, "dollars" are not great starting point for illustrating "interval"
versus "ordinal" variables;  I have used almost the same example to make
the *opposite* point.  I start with Tukey's advice, that if your largest data 
value is 10 times the smallest, you *might* consider a transformation; 
if it is 20 times the smallest, you *certainly*  consider one.  

$5000 versus $10?  That raises the "20 times" question.  What are these
dollars representing?  Is this a silly equation, the mixes the cost of a used
car with the cost of a used DVD?   In social science, the ratio of changes
is apt to be relevant for numbers like these.

"If we are measuring economic value" is the key: The DVs and IV need 
to be equal-interval with respect to each other.  If you are stuck with dollars
for "economic value", then you use dollars.  That raises further problems, 
so it is not a fine solution that should be generalized as a "good example". 

We *prefer* to match up the scales, in addition to having homogeneity of error 
across the range, and hopefully seeing a normal distribution in some suitable 
population.  If we are measuring latent constructs with arbitrary scales, we
are not as constricted in our scoring as "economic value in dollars". 

-- 
Rich Ulrich 

---

Date: Wed, 18 Jun 2014 15:33:50 -0400
From: [hidden email]
Subject: Re: What is wrong with treating Likert scale as interval data?
To: [hidden email]

...

If we are measuring economic value using the U.S. Dollar, then we know that the difference between $10 and $9 is precisely the same as the difference between $5,010 and $5,009. We could have 9 financial analysts randomly chosen with regard to experience and level of expertise who would likely come to the same conclusion 9 out of 9 times. 

...

_______________

Professor Diana Kornbrot

University of Hertfordshire

College Lane, Hatfield, Hertfordshire AL10 9AB, UK

[phone]

[phone]

[phone] 

[hidden email]

http://dianakornbrot.wordpress.com/

http://go.herts.ac.uk/Diana_Kornbrot

skype:  kornbrotme_______________________________