Re: doubt about reliability analysis
Locked
Re: doubt about reliability analysis
 
|
Hi Rodrigo,
I'm not sure I totally understand the question here? Are you saying that you found that the means across variables are statistically equal, and you don't understand why you would still get a high cronbach's alpha? Cronbach's alpha is a test of unidimensionality, i.e. similarness of the variables. It's supposed to reflect the degree to which a set of variables are correlated to the "scale" they collectively represent. As a result, if all variables are on a common scale, then the variables in a scale with high reliability would very likely not have significant differences in means. This would be an unimportant finding though, the means are meaningless in assessing the reliability of the scale. It's also worth noting that the use of Alpha is a debated topic. A lot of people feel that the use of better reliability and validation analysis of a scale or set of scales makes a lot more sense, especially when scales are composed of large sets of items. Factor analysis and optimal scaling are often better approaches to assessing the true reliability of a set of scales (along with other measures of reliability such as test-retest and multi-sample consistency). My point is just that alpha isn't all it's cracked up to be, and you may find problems and inconsistencies with alpha that can be better understood through other approaches. Matthew J Poes Research Data Specialist Center for Prevention Research and Development University of Illinois 510 Devonshire Dr. Champaign, IL 61820 Phone: 217-265-4576 email: [hidden email] -----Original Message----- From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Rodrigo Briceño Sent: Wednesday, March 14, 2012 3:51 PM To: [hidden email] Subject: doubt about realibility analysis Dear SPSS users. Usually when we work on developing scales, the attention goes primarily to Alpha-Cronbach (AC) coefficient. One friend of mine that is working on the subject found an AC of 0.776. In his analysis he also considered the test of T2 Hotelling (p-value 0.573). As far as I understand this is a multivariate test for comparison of means of the variables of the scale. In this case the test implies no rejection of H0, what it means that the variables used for developing the scale have all statistically equal means. He asked me if I know what is the relation between the AC and the T2 Hotelling test. In other words: when you got a 0.7 or higher on AC you should get a lowest p-value (rejection of H0) in T2 Hotelling? If anybody knows more about this topic I will appreciate your comments. Thanks -- Rodrigo Briceño Economist [hidden email] MSN: [hidden email] SKYPE: rbriceno1087 ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD |
Re: doubt about reliability analysis
|
|
Mattew, thanks for your explanation. This is precisely what I wanted to confirm.
2012/3/14 Poes, Matthew Joseph <[hidden email]>: > Hi Rodrigo, > I'm not sure I totally understand the question here? Are you saying that you found that the means across variables are statistically equal, and you don't understand why you would still get a high cronbach's alpha? Cronbach's alpha is a test of unidimensionality, i.e. similarness of the variables. It's supposed to reflect the degree to which a set of variables are correlated to the "scale" they collectively represent. As a result, if all variables are on a common scale, then the variables in a scale with high reliability would very likely not have significant differences in means. This would be an unimportant finding though, the means are meaningless in assessing the reliability of the scale. > > It's also worth noting that the use of Alpha is a debated topic. A lot of people feel that the use of better reliability and validation analysis of a scale or set of scales makes a lot more sense, especially when scales are composed of large sets of items. Factor analysis and optimal scaling are often better approaches to assessing the true reliability of a set of scales (along with other measures of reliability such as test-retest and multi-sample consistency). My point is just that alpha isn't all it's cracked up to be, and you may find problems and inconsistencies with alpha that can be better understood through other approaches. > > Matthew J Poes > Research Data Specialist > Center for Prevention Research and Development > University of Illinois > 510 Devonshire Dr. > Champaign, IL 61820 > Phone: 217-265-4576 > email: [hidden email] > > > > -----Original Message----- > From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Rodrigo Briceño > Sent: Wednesday, March 14, 2012 3:51 PM > To: [hidden email] > Subject: doubt about realibility analysis > > Dear SPSS users. Usually when we work on developing scales, the attention goes primarily to Alpha-Cronbach (AC) coefficient. One friend of mine that is working on the subject found an AC of 0.776. In his analysis he also considered the test of T2 Hotelling (p-value 0.573). As far as I understand this is a multivariate test for comparison of means of the variables of the scale. In this case the test implies no rejection of H0, what it means that the variables used for developing the scale have all statistically equal means. > > He asked me if I know what is the relation between the AC and the T2 Hotelling test. In other words: when you got a 0.7 or higher on AC you should get a lowest p-value (rejection of H0) in T2 Hotelling? > > If anybody knows more about this topic I will appreciate your comments. > > Thanks > > -- > Rodrigo Briceño > Economist > [hidden email] > MSN: [hidden email] > SKYPE: rbriceno1087 > > ===================== > To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD -- Rodrigo Briceño Economist [hidden email] MSN: [hidden email] SKYPE: rbriceno1087 ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD |
Re: doubt about reliability analysis
|
|
In reply to this post by Poes, Matthew Joseph-2
What is the nature of that debate about alpha?
- Alpha measures internal consistency, which is only important when you are intending to create a scale from items that are *supposed* to be related. Alpha does say something about how similar they are, but -- contrary to what Poes says, below -- it makes no statement about the scale being unidimensional, if he intends that as a technical statement instead of as an informal description. To the extent that "validity" may depend on covering a certain, broad scope of a concept, emphasizing narrow consistency is opposed to emphasizing validity. You do not want all of the items to be exactly the same, or so close "as to make no difference". You do want to include relevant aspects (which may be thought of as dimensions) of your construct. That is not controversy. That is a simple observation that Guilford put into his textbook, 50 or 60 years ago. I think the only shade of "controversy" arises when some novice assumes, again, that if you have a good alpha, you have everything that matters. -- Rich Ulrich > Date: Wed, 14 Mar 2012 21:03:59 +0000 > From: [hidden email] > Subject: Re: doubt about reliability analysis > To: [hidden email] > > Hi Rodrigo, > I'm not sure I totally understand the question here? Are you saying that you found that the means across variables are statistically equal, and you don't understand why you would still get a high cronbach's alpha? Cronbach's alpha is a test of unidimensionality, i.e. similarness of the variables. It's supposed to reflect the degree to which a set of variables are correlated to the "scale" they collectively represent. As a result, if all variables are on a common scale, then the variables in a scale with high reliability would very likely not have significant differences in means. This would be an unimportant finding though, the means are meaningless in assessing the reliability of the scale. > > It's also worth noting that the use of Alpha is a debated topic. A lot of people feel that the use of better reliability and validation analysis of a scale or set of scales makes a lot more sense, especially when scales are composed of large sets of items. Factor analysis and optimal scaling are often better approaches to assessing the true reliability of a set of scales (along with other measures of reliability such as test-retest and multi-sample consistency). My point is just that alpha isn't all it's cracked up to be, and you may find problems and inconsistencies with alpha that can be better understood through other approaches. > > Matthew J Poes > Research Data Specialist > Center for Prevention Research and Development > University of Illinois |
Automatic reply: doubt about reliability analysis
|
|
I am currently out of the office and will return Monday, March 19. I will respond to your e-mail upon my return. If you need immediate assistance, please call 812-856-5824. Thank you, Shimon SarrafCenter for Postsecondary Research IU-Bloomington
|
Re: doubt about reliability analysis
|
|
In reply to this post by Rich Ulrich
That would be the controversy I was referring to. I didn’t feel like writing a long response to get into all this. Cronbach claimed Alpha indirectly measured
unidimensiality of scales. Numerous subsequent research proved that erroneous.
While you are quite correct that Alpha does correctly assess the degree to which a set of similar measures are similar, I would argue that a Factor analysis
(or PCA) does the same thing with greater detail, so why include both. Another problem with Alpha, that numerous papers have stated is a reason to stop using it, is that having a high number of items always artificially inflates the alpha. Numerous papers
in Psychometrika have been written comparing the coefficient alpha to rasch item reliability scores, confirmatory factor analysis results, and other alternate systems developed over the years. I can’t find it at the moment, but even recall reading a paper
calling for the end of the required use of coefficient alpha, and then did a sort of meta-analysis on 100’s of studies reporting alpha, in which their own reporting indicated major violations of assumptions with alpha. Yes you can point to the fact that only a novice would fail to check assumptions, and that anyone who understands statistics would not be reporting just alpha,
or not report alpha with data that violates the assumptions of correlations. However, the reality seems to be that there are an awful lot of PhD “novices” out there publishing work incorrectly. For a lot of these researchers, Statistics is not a first language,
and these issues don’t always get addressed. On top of that, many journals have erroneously required the reporting of alpha, forcing it’s use in questionable situations. My own solution to these problems is to use more advanced methods in classical test
theory, or to move to item response theory methods, and work back to a proxy to alpha. This typically satiates the reviewers, but reality is, not very many researchers have that level of technical expertise.
I’ll end with this: How many papers have you seen where Factor Analysis was clearly done incorrectly? How many papers draw incorrect conclusions from their
Factor analysis? My point is this, even when people do take it to the next level, it doesn’t always mean it gets done right. I was recently involved in a discussion with a group of researchers at two very well-known private sector research firms in which
I made the argument that anyone who is interpreting the summative scale based on the original metric of the component items is doing so incorrectly. There was absolutely no disagreement on this (yet I would argue the majority of researchers seem to do just
this). We then continued this discussion to look into the merit vs. consequence of utilizing factor or object scores instead of typical mean or summation scores. The final conclusion was that the evidence points to the use of factor or object scores in every
situation where scales are derived from a set of items (especially important when they are not so highly correlated), but again, from what I see in the literature, this is not done much. When classical test theory dominated, this approach was common, as they
moved toward better IRT approaches, and classical test methods were relegated to other fields, the best practices seemed to fall to the wayside.
Matthew J Poes Research Data Specialist Center for Prevention Research and Development University of Illinois 510 Devonshire Dr. Champaign, IL 61820 Phone: 217-265-4576 email:
[hidden email] From: Rich Ulrich [mailto:[hidden email]]
What is the nature of that debate about alpha? > Date: Wed, 14 Mar 2012 21:03:59 +0000 |
Re: doubt about reliability analysis
|
|
Just my $.02 here. I think there is some controversy, but more misinformation about alpha and Cronbach’s understanding of it as
a measure of unidimensionality. Essentially, others claimed that he claimed it was such a measure. While many cite Cronbach, few seem actually to have read what he wrote. Here is a very brief synopsis: Cronbach (1951) was clear that alpha "estimates the proportion of the test variance due to all
common factors among the items. That is, it reports how much the test score depends upon general and group, rather than item specific, factors,” going on to say that alpha “is the proportion of test variance due to common factors” (p. 320). We find a similar
view in his final published statements about alpha (Cronbach, 2004) where he again states – speaking of alpha - that “I particularly cleared the air by getting rid of the assumption that the items of a test were unidimensional” (p. 397). Cronbach argued that
alpha was neither a direct nor indirect estimate of the unidimensionality of a test/measure. He argued that it was a lower-bound estimate of the ratio of the common factor variance (summed across all common factors) to the total variance of a test. Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests.
Psychometrika, 16, 297-334. doi:10.1007/BF02310555 Cronbach, L. J., & Shavelson, R. (2004). My current thoughts on Coefficient Alpha and successor procedures.
Educational And Psychological Measurement, 64, 391-418. doi:10.1177/0013164404266386
Harley
Dr. Harley Baker
Professor of Psychology
California State University Channel Islands
Sage Hall 2129
One University Drive
Camarillo, CA 93012
805.437.8997 (p)
805.437.8951 (f)
From: "Poes, Matthew Joseph" <[hidden email]>
Reply-To: "Poes, Matthew Joseph" <[hidden email]> Date: Thu, 15 Mar 2012 13:47:53 +0000 To: <[hidden email]> Subject: Re: doubt about reliability analysis That would be the controversy I was referring to. I didn’t feel like writing a long response to get into all this. Cronbach claimed Alpha indirectly
measured unidimensiality of scales. Numerous subsequent research proved that erroneous.
While you are quite correct that Alpha does correctly assess the degree to which a set of similar measures are similar, I would argue that a Factor
analysis (or PCA) does the same thing with greater detail, so why include both. Another problem with Alpha, that numerous papers have stated is a reason to stop using it, is that having a high number of items always artificially inflates the alpha. Numerous
papers in Psychometrika have been written comparing the coefficient alpha to rasch item reliability scores, confirmatory factor analysis results, and other alternate systems developed over the years. I can’t find it at the moment, but even recall reading
a paper calling for the end of the required use of coefficient alpha, and then did a sort of meta-analysis on 100’s of studies reporting alpha, in which their own reporting indicated major violations of assumptions with alpha. Yes you can point to the fact that only a novice would fail to check assumptions, and that anyone who understands statistics would not be reporting
just alpha, or not report alpha with data that violates the assumptions of correlations. However, the reality seems to be that there are an awful lot of PhD “novices” out there publishing work incorrectly. For a lot of these researchers, Statistics is not
a first language, and these issues don’t always get addressed. On top of that, many journals have erroneously required the reporting of alpha, forcing it’s use in questionable situations. My own solution to these problems is to use more advanced methods
in classical test theory, or to move to item response theory methods, and work back to a proxy to alpha. This typically satiates the reviewers, but reality is, not very many researchers have that level of technical expertise.
I’ll end with this: How many papers have you seen where Factor Analysis was clearly done incorrectly? How many papers draw incorrect conclusions
from their Factor analysis? My point is this, even when people do take it to the next level, it doesn’t always mean it gets done right. I was recently involved in a discussion with a group of researchers at two very well-known private sector research firms
in which I made the argument that anyone who is interpreting the summative scale based on the original metric of the component items is doing so incorrectly. There was absolutely no disagreement on this (yet I would argue the majority of researchers seem
to do just this). We then continued this discussion to look into the merit vs. consequence of utilizing factor or object scores instead of typical mean or summation scores. The final conclusion was that the evidence points to the use of factor or object
scores in every situation where scales are derived from a set of items (especially important when they are not so highly correlated), but again, from what I see in the literature, this is not done much. When classical test theory dominated, this approach
was common, as they moved toward better IRT approaches, and classical test methods were relegated to other fields, the best practices seemed to fall to the wayside. Matthew J Poes Research Data Specialist Center for Prevention Research and Development University of Illinois 510 Devonshire Dr. Champaign, IL 61820 Phone: 217-265-4576 email:
[hidden email] From: Rich Ulrich [[hidden email]]
What is the nature of that debate about alpha? > Date: Wed, 14 Mar 2012 21:03:59 +0000 |
Re: doubt about reliability analysis
|
|
I think the specifics of it being unidimensional or not is unimportant (as it was not my point in my own comment), but I’ll say this: Cronbach’s original article
referred to alpha as being equal to the first factor concentration, and later mentions a specific case as potentially being unidimensional, save for error. Cronbach himself clearly corrected any misinformation that these comments may have created, be it his
own erroneous assumption that it indicated unidimensiality, or other’s misunderstanding it.
My original point was simply that if scales are being judged solely by alpha and I’d presume face validity, that is really not sufficient. None the less, I appreciate the more accurate historical representation of the information. I recall being actually told that Cronbach believed this erroneously
and only backtracked later in light of the evidence. I’ve never read the later articles you cite, nor am I old enough to even remember this controversy at all (if there ever really was any). Matthew J Poes Research Data Specialist Center for Prevention Research and Development University of Illinois 510 Devonshire Dr. Champaign, IL 61820 Phone: 217-265-4576 email:
[hidden email] From: SPSSX(r) Discussion [mailto:[hidden email]]
On Behalf Of Baker, Harley Just my $.02 here. I think there is some controversy, but more misinformation about alpha and Cronbach’s understanding of it as a measure of unidimensionality. Essentially,
others claimed that he claimed it was such a measure. While many cite Cronbach, few seem actually to have read what he wrote. Here is a very brief synopsis: Cronbach (1951) was clear that alpha "estimates the proportion of the test variance due to all common factors among the items. That
is, it reports how much the test score depends upon general and group, rather than item specific, factors,” going on to say that alpha “is the proportion of test variance due to common factors” (p. 320). We find a similar view in his final published statements
about alpha (Cronbach, 2004) where he again states – speaking of alpha - that “I particularly cleared the air by getting rid of the assumption that the items of a test were unidimensional” (p. 397). Cronbach argued that alpha was neither a direct nor indirect
estimate of the unidimensionality of a test/measure. He argued that it was a lower-bound estimate of the ratio of the common factor variance (summed across all common factors) to the total variance of a test. Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests.
Psychometrika, 16, 297-334. doi:10.1007/BF02310555 Cronbach, L. J., & Shavelson, R. (2004). My current thoughts on Coefficient Alpha and successor procedures.
Educational And Psychological Measurement, 64, 391-418. doi:10.1177/0013164404266386 Harley Dr. Harley Baker Professor of Psychology California State University Channel Islands Sage Hall 2129 One University Drive Camarillo, CA 93012 805.437.8997 (p) 805.437.8951 (f) From:
"Poes, Matthew Joseph" <[hidden email]> That would be the controversy I was referring to. I didn’t feel like writing a long response to get into all this. Cronbach claimed Alpha indirectly measured
unidimensiality of scales. Numerous subsequent research proved that erroneous.
While you are quite correct that Alpha does correctly assess the degree to which a set of similar measures are similar, I would argue that a Factor analysis
(or PCA) does the same thing with greater detail, so why include both. Another problem with Alpha, that numerous papers have stated is a reason to stop using it, is that having a high number of items always artificially inflates the alpha. Numerous papers
in Psychometrika have been written comparing the coefficient alpha to rasch item reliability scores, confirmatory factor analysis results, and other alternate systems developed over the years. I can’t find it at the moment, but even recall reading a paper
calling for the end of the required use of coefficient alpha, and then did a sort of meta-analysis on 100’s of studies reporting alpha, in which their own reporting indicated major violations of assumptions with alpha. Yes you can point to the fact that only a novice would fail to check assumptions, and that anyone who understands statistics would not be reporting just alpha,
or not report alpha with data that violates the assumptions of correlations. However, the reality seems to be that there are an awful lot of PhD “novices” out there publishing work incorrectly. For a lot of these researchers, Statistics is not a first language,
and these issues don’t always get addressed. On top of that, many journals have erroneously required the reporting of alpha, forcing it’s use in questionable situations. My own solution to these problems is to use more advanced methods in classical test
theory, or to move to item response theory methods, and work back to a proxy to alpha. This typically satiates the reviewers, but reality is, not very many researchers have that level of technical expertise.
I’ll end with this: How many papers have you seen where Factor Analysis was clearly done incorrectly? How many papers draw incorrect conclusions from their
Factor analysis? My point is this, even when people do take it to the next level, it doesn’t always mean it gets done right. I was recently involved in a discussion with a group of researchers at two very well-known private sector research firms in which
I made the argument that anyone who is interpreting the summative scale based on the original metric of the component items is doing so incorrectly. There was absolutely no disagreement on this (yet I would argue the majority of researchers seem to do just
this). We then continued this discussion to look into the merit vs. consequence of utilizing factor or object scores instead of typical mean or summation scores. The final conclusion was that the evidence points to the use of factor or object scores in every
situation where scales are derived from a set of items (especially important when they are not so highly correlated), but again, from what I see in the literature, this is not done much. When classical test theory dominated, this approach was common, as they
moved toward better IRT approaches, and classical test methods were relegated to other fields, the best practices seemed to fall to the wayside. Matthew J Poes Research Data Specialist Center for Prevention Research and Development University of Illinois 510 Devonshire Dr. Champaign, IL 61820 Phone: 217-265-4576 email:
[hidden email] From: Rich Ulrich [[hidden email]]
What is the nature of that debate about alpha? > Date: Wed, 14 Mar 2012 21:03:59 +0000 |
Re: doubt about reliability analysis
|
|
In reply to this post by Poes, Matthew Joseph-2
Hi DorraJ, See Below for my response: Matthew J Poes Research Data Specialist Center for Prevention Research and Development University of Illinois 510 Devonshire Dr. Champaign, IL 61820 Phone: 217-265-4576 email:
[hidden email] From: DorraJ Oet [mailto:[hidden email]]
Hi Mattew,
It implies that you have a unidimensional scale. There aren’t subcomponents to the latent construct of the scale.
If I had a better idea what you were trying to create from this, I might be able to better help. If you are trying to explore the underlying latent factors of a set of variables,
then true factor analysis is probably better. If you are simply trying to reduce it to a set of composite scores, then PCA is fine, though I’d argue that you should be using object scores as the composite variable.
There are potential problems relying solely on the eigenvalues, and as a result numerous papers have suggested using other approaches in conjunction with the eigenvalues (For instance
the scree plot). I would highly recommend this, as I’ve worked with experienced teams of researchers who have even retained factors less than .9 (Not something I’ve ever chosen myself might I add). Retaining a factor that is less than 1 is fine, the general
thought is that it should be greater than .9 (otherwise it’s explaining less variance than a single variable).
Typically we want theory to drive these decisions as much as data. If we look at the factors that are retained at the end of the analysis, and we simply can’t apply them back to
the theoretical measurement with a logical set of latent factors, then we want to consider other reasons for the results we get. Factor analysis still has assumptions of the underlying distribution of data, and this could certainly drive separate factors,
but would be meaningless to the scale development.
Different measurements are fine, as long as they all have a similar underlying distribution, i.e. are all Gaussian. Factor analysis/PCA will both convert to z-scores when they
run the analysis. Some people like to convert the scales to a common metric before combining them into a single scale. It’s important to remember that the purpose of a factor analysis is to reduce a large set of variables into a set of underlying latent
constructs. These constructs are typically some form of summation across the common variables within each construct. If you are not using factor scores or object scores (as is the case in PCA), then you do want to normalize your data to a common metric,
such as Z scores (Standardized scores).
I’ll call this an advanced topic, but only because it seems very uncommonly used. When combining multiple variables with different metrics, it’s also possible, if not likely that
the variables will have very different underlying distributions, and a simple standardized z-score coefficient won’t be sufficient to create a logical latent scale. In this case, other methods exist such as correspondence analysis or categorical principal
component analysis. These are used in “Optimal Scaling”. When you save out the object scores for an optimal scale, they will be based on the mean of the standardized coefficients. The standardization will be based on how you set it (nominal, ordinal, ratio,
spline nominal, spline ordinal) discretized by (ranking, binning/grouping, multiplying/ratio). This results in an “optimal” value relative to the underlying latent construct, which minimizes heterogeneity. Date: Thu, 15 Mar 2012 13:47:53 +0000 That would be the controversy I was referring to. I didn’t feel like writing a long response to get into all this. Cronbach claimed Alpha indirectly measured
unidimensiality of scales. Numerous subsequent research proved that erroneous.
While you are quite correct that Alpha does correctly assess the degree to which a set of similar measures are similar, I would argue that a Factor analysis
(or PCA) does the same thing with greater detail, so why include both. Another problem with Alpha, that numerous papers have stated is a reason to stop using it, is that having a high number of items always artificially inflates the alpha. Numerous papers
in Psychometrika have been written comparing the coefficient alpha to rasch item reliability scores, confirmatory factor analysis results, and other alternate systems developed over the years. I can’t find it at the moment, but even recall reading a paper
calling for the end of the required use of coefficient alpha, and then did a sort of meta-analysis on 100’s of studies reporting alpha, in which their own reporting indicated major violations of assumptions with alpha. Yes you can point to the fact that only a novice would fail to check assumptions, and that anyone who understands statistics would not be reporting just alpha,
or not report alpha with data that violates the assumptions of correlations. However, the reality seems to be that there are an awful lot of PhD “novices” out there publishing work incorrectly. For a lot of these researchers, Statistics is not a first language,
and these issues don’t always get addressed. On top of that, many journals have erroneously required the reporting of alpha, forcing it’s use in questionable situations. My own solution to these problems is to use more advanced methods in classical test
theory, or to move to item response theory methods, and work back to a proxy to alpha. This typically satiates the reviewers, but reality is, not very many researchers have that level of technical expertise.
I’ll end with this: How many papers have you seen where Factor Analysis was clearly done incorrectly? How many papers draw incorrect conclusions from their
Factor analysis? My point is this, even when people do take it to the next level, it doesn’t always mean it gets done right. I was recently involved in a discussion with a group of researchers at two very well-known private sector research firms in which
I made the argument that anyone who is interpreting the summative scale based on the original metric of the component items is doing so incorrectly. There was absolutely no disagreement on this (yet I would argue the majority of researchers seem to do just
this). We then continued this discussion to look into the merit vs. consequence of utilizing factor or object scores instead of typical mean or summation scores. The final conclusion was that the evidence points to the use of factor or object scores in every
situation where scales are derived from a set of items (especially important when they are not so highly correlated), but again, from what I see in the literature, this is not done much. When classical test theory dominated, this approach was common, as they
moved toward better IRT approaches, and classical test methods were relegated to other fields, the best practices seemed to fall to the wayside.
Matthew J Poes Research Data Specialist Center for Prevention Research and Development University of Illinois 510 Devonshire Dr. Champaign, IL 61820 Phone: 217-265-4576 email:
[hidden email] From: Rich Ulrich
[hidden email] What is the nature of that debate about alpha? > Date: Wed, 14 Mar 2012 21:03:59 +0000 |
| Free forum by Nabble | Edit this page |