SPSSX Discussion

Reliability and validity for a rubric type questionnaire

Classic

List

Threaded

9 messages Options

E. Bernardo

Reliability and validity for a rubric type questionnaire

I wonder if we can do validation study via statistics for a rubric type instrument with only nine items and for each item has 5 descriptors (ordinal scale). My sample is only less than 30.

Thank you in advance for your inputs.

Eins

Yahoo! Mail Now Faster and Cleaner. Experience it today!

Art Kendall

Re: Reliability and validity for a rubric type questionnaire

What is a "rubric" scale?
What are the labels on your response scale?

For the reliability:
If 1) you have a conventional summative scale, where you simply sum or average over a set of items, and 2) if your response scale is not very far from interval as is usually the situation with Likert items, then try the RELIABILITY procedure.

If your response scale is very discrepant from interval level, use CATPCA to see if you have one factor and if it makes much difference to treat the response scale as ordinal vs interval.

For validity, do you have another measure of the same construct?

Art Kendall
Social Research Consultants

Eins Bernardo wrote:

I wonder if we can do validation study via statistics for a rubric type instrument with only nine items and for each item has 5 descriptors (ordinal scale). My sample is only less than 30.

Thank you in advance for your inputs.

Eins

Yahoo! Mail Now Faster and Cleaner. Experience it today!

Art Kendall
Social Research Consultants

E. Bernardo

Re: Reliability and validity for a rubric type questionnaire

In reply to this post by E. Bernardo

Thank you for your time, Art.

Actually, the response scale cannot be treated as interval. Thus, your suggestion to use CATPCA is the appropriate one.

I have a couple of questions: (1) For the reliability, considering that the response scale cannot be treated as interval, what coefficient is used in place of cronbach alpha? (2) For the validity, are there other ways considering that I dont have another measure of the same constract? That is, can we test the validity of the scale using only the present data (n<30, 9 items)?

Thank you.

Eins

--- On Mon, 6/15/09, Art Kendall <[hidden email]> wrote:

all
From: Art Kendall <[hidden email]>
Subject: Re: Reliability and validity for a rubric type questionnaire
To: [hidden email]
Date: Monday, 15 June, 2009, 12:29 PM

What is a "rubric" scale?
What are the labels on your response scale?

For the reliability:
If 1) you have a conventional summative scale, where you simply sum or average over a set of items, and 2) if your response scale is not very far from interval as is usually the situation with Likert items, then try the RELIABILITY procedure.

If your response scale is very discrepant from interval level, use CATPCA to see if you have one factor and if it makes much difference to treat the response scale as ordinal vs interval.

For validity, do you have another measure of the same construct?

Art Kendall
Social Research Consultants

Eins Bernardo wrote:

I wonder if we can do validation study via statistics for a rubric type instrument with only nine items and for each item has 5 descriptors (ordinal scale). My sample is only less than 30.

Thank you in advance for your inputs.

Eins

Yahoo! Mail Now Faster and Cleaner. Experience it today!

Connect instantly with more friends on your blog and personal website?
Create your latest Pingbox today!

Art Kendall

Re: Reliability and validity for a rubric type questionnaire

What is a "rubric type instrument"?
What are the labels on your response scale? Why are you sure it is very discrepant from interval level? [for future research it is worth the effort to develop a set of values that are not very discrepant from interval level. E.g., Likert items are usually close enough to interval level. Use of numeric as well as verbal stems helps respondents. More response levels often helps.]

When you run CATPCA, be sure to test you measurement level assumption. Perhaps the people from Leiden who developed the CATPCA can recommend how assess internal consistency reliability for purely ordinal data.

Unless the scale has been used on other research where you can show that the scale represents a construct that correlates/associates appropriately with other measures, the only thing left is to rely on "face validity". I.e., "Do some judges agree that the items appear to measure what they purport to measure?"

Art Kendall
Social Research Consultants

Eins Bernardo wrote:

Thank you for your time, Art.

Actually, the response scale cannot be treated as interval. Thus, your suggestion to use CATPCA is the appropriate one.

I have a couple of questions: (1) For the reliability, considering that the response scale cannot be treated as interval, what coefficient is used in place of cronbach alpha? (2) For the validity, are there other ways considering that I dont have another measure of the same constract? That is, can we test the validity of the scale using only the present data (n<30, 9 items)?

Thank you.

Eins

--- On Mon, 6/15/09, Art Kendall [hidden email] wrote:

all
From: Art Kendall [hidden email]
Subject: Re: Reliability and validity for a rubric type questionnaire
To: [hidden email]
Date: Monday, 15 June, 2009, 12:29 PM

What is a "rubric" scale?
What are the labels on your response scale?

For the reliability:
If 1) you have a conventional summative scale, where you simply sum or average over a set of items, and 2) if your response scale is not very far from interval as is usually the situation with Likert items, then try the RELIABILITY procedure.

If your response scale is very discrepant from interval level, use CATPCA to see if you have one factor and if it makes much difference to treat the response scale as ordinal vs interval.

For validity, do you have another measure of the same construct?

Art Kendall
Social Research Consultants

Eins Bernardo wrote:

I wonder if we can do validation study via statistics for a rubric type instrument with only nine items and for each item has 5 descriptors (ordinal scale). My sample is only less than 30.

Thank you in advance for your inputs.

Eins

Yahoo! Mail Now Faster and Cleaner. Experience it today!

Connect instantly with more friends on your blog and personal website?
Create your latest Pingbox today!

Art Kendall
Social Research Consultants

Kooij, A.J. van der

Re: Reliability and validity for a rubric type questionnaire

In reply to this post by E. Bernardo

CATPCA gives Cronbach's alpha (see the Model Summary table). This is alpha for the transformed variables, thus treating your variable as ordinal is taken into account.

Regards,

Anita van der Kooij

Data Theory Group

Leiden University

From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Eins Bernardo
Sent: 16 June 2009 03:42
To: [hidden email]
Subject: Re: Reliability and validity for a rubric type questionnaire

Thank you for your time, Art.

Actually, the response scale cannot be treated as interval. Thus, your suggestion to use CATPCA is the appropriate one.

Thank you.

Eins

--- On Mon, 6/15/09, Art Kendall <[hidden email]> wrote:

all
From: Art Kendall <[hidden email]>
Subject: Re: Reliability and validity for a rubric type questionnaire
To: [hidden email]
Date: Monday, 15 June, 2009, 12:29 PM

What is a "rubric" scale?
What are the labels on your response scale?

For the reliability:
If 1) you have a conventional summative scale, where you simply sum or average over a set of items, and 2) if your response scale is not very far from interval as is usually the situation with Likert items, then try the RELIABILITY procedure.

If your response scale is very discrepant from interval level, use CATPCA to see if you have one factor and if it makes much difference to treat the response scale as ordinal vs interval.

For validity, do you have another measure of the same construct?

Art Kendall
Social Research Consultants

Eins Bernardo wrote:

I wonder if we can do validation study via statistics for a rubric type instrument with only nine items and for each item has 5 descriptors (ordinal scale). My sample is only less than 30.

Thank you in advance for your inputs.

Eins

Yahoo! Mail Now Faster and Cleaner. Experience it today!

Connect instantly with more friends on your blog and personal website?
Create your latest Pingbox today!

**********************************************************************

This email and any files transmitted with it are confidential and

intended solely for the use of the individual or entity to whom they

are addressed. If you have received this email in error please notify

the system manager.

**********************************************************************

Juanito Talili

Re: Reliability and validity for a rubric type questionnaire

In reply to this post by E. Bernardo

How to perform CATPCA? I am using SPSS 17.

Thank you.

Juanito

--- On Tue, 6/16/09, Kooij, A.J. van der <[hidden email]> wrote:

From: Kooij, A.J. van der <[hidden email]>
Subject: Re: Reliability and validity for a rubric type questionnaire
To: [hidden email]
Date: Tuesday, 16 June, 2009, 2:03 PM

CATPCA gives Cronbach's alpha (see the Model Summary table). This is alpha for the transformed variables, thus treating your variable as ordinal is taken into account.

Regards,

Anita van der Kooij

Data Theory Group

Leiden University

From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Eins Bernardo
Sent: 16 June 2009 03:42
To: [hidden email]
Subject: Re: Reliability and validity for a rubric type questionnaire

Thank you for your time, Art.

Actually, the response scale cannot be treated as interval. Thus, your suggestion to use CATPCA is the appropriate one.

I have a couple of questions: (1) For the reliability, considering that the response scale cannot be treated as interval, what coefficient is used in place of cronbach alpha? (2) For the validity, are there other ways considering that I dont have another measure of the same constract? That is, can we test the validity of the scale using only the present data (n<30, 9 items)?

Thank you.

Eins

--- On Mon, 6/15/09, Art Kendall <[hidden email]> wrote:

all
From: Art Kendall <[hidden email]>
Subject: Re: Reliability and validity for a rubric type questionnaire
To: [hidden email]
Date: Monday, 15 June, 2009, 12:29 PM

What is a "rubric" scale?
What are the labels on your response scale?

For the reliability:
If 1) you have a conventional summative scale, where you simply sum or average over a set of items, and 2) if your response scale is not very far from interval as is usually the situation with Likert items, then try the RELIABILITY procedure.

If your response scale is very discrepant from interval level, use CATPCA to see if you have one factor and if it makes much difference to treat the response scale as ordinal vs interval.

For validity, do you have another measure of the same construct?

Art Kendall
Social Research Consultants

Eins Bernardo wrote:

I wonder if we can do validation study via statistics for a rubric type instrument with only nine items and for each item has 5 descriptors (ordinal scale). My sample is only less than 30.

Thank you in advance for your inputs.

Eins

Yahoo! Mail Now Faster and Cleaner. Experience it today!

Connect instantly with more friends on your blog and personal website?
Create your latest Pingbox today!
**********************************************************************

This email and any files transmitted with it are confidential and

intended solely for the use of the individual or entity to whom they

are addressed. If you have received this email in error please notify

the system manager.

**********************************************************************

Surf faster and smarter with Firefox 3!

Kooij, A.J. van der

Re: Reliability and validity for a rubric type questionnaire

Analyze, Dimension Reduction, Optimal Scaling, Choose "Some variable(s) not multiple nominal", click define.

From: Juanito Talili [mailto:[hidden email]]
Sent: 16 June 2009 16:48
To: [hidden email]; Kooij, A.J. van der
Subject: Re: Reliability and validity for a rubric type questionnaire

How to perform CATPCA? I am using SPSS 17.

Thank you.

Juanito

--- On Tue, 6/16/09, Kooij, A.J. van der <[hidden email]> wrote:

From: Kooij, A.J. van der <[hidden email]>
Subject: Re: Reliability and validity for a rubric type questionnaire
To: [hidden email]
Date: Tuesday, 16 June, 2009, 2:03 PM

CATPCA gives Cronbach's alpha (see the Model Summary table). This is alpha for the transformed variables, thus treating your variable as ordinal is taken into account.

Regards,

Anita van der Kooij

Data Theory Group

Leiden University

From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Eins Bernardo
Sent: 16 June 2009 03:42
To: [hidden email]
Subject: Re: Reliability and validity for a rubric type questionnaire

Thank you for your time, Art.

Actually, the response scale cannot be treated as interval. Thus, your suggestion to use CATPCA is the appropriate one.

I have a couple of questions: (1) For the reliability, considering that the response scale cannot be treated as interval, what coefficient is used in place of cronbach alpha? (2) For the validity, are there other ways considering that I dont have another measure of the same constract? That is, can we test the validity of the scale using only the present data (n<30, 9 items)?

Thank you.

Eins

--- On Mon, 6/15/09, Art Kendall <[hidden email]> wrote:

all
From: Art Kendall <[hidden email]>
Subject: Re: Reliability and validity for a rubric type questionnaire
To: [hidden email]
Date: Monday, 15 June, 2009, 12:29 PM

What is a "rubric" scale?
What are the labels on your response scale?

For the reliability:
If 1) you have a conventional summative scale, where you simply sum or average over a set of items, and 2) if your response scale is not very far from interval as is usually the situation with Likert items, then try the RELIABILITY procedure.

If your response scale is very discrepant from interval level, use CATPCA to see if you have one factor and if it makes much difference to treat the response scale as ordinal vs interval.

For validity, do you have another measure of the same construct?

Art Kendall
Social Research Consultants

Eins Bernardo wrote:

I wonder if we can do validation study via statistics for a rubric type instrument with only nine items and for each item has 5 descriptors (ordinal scale). My sample is only less than 30.

Thank you in advance for your inputs.

Eins

Yahoo! Mail Now Faster and Cleaner. Experience it today!

Connect instantly with more friends on your blog and personal website?
Create your latest Pingbox today!
**********************************************************************

This email and any files transmitted with it are confidential and

intended solely for the use of the individual or entity to whom they

are addressed. If you have received this email in error please notify

the system manager.

**********************************************************************

Surf faster and smarter with Firefox 3!

Juanito Talili

Re: Reliability and validity for a rubric type questionnaire

In reply to this post by E. Bernardo

Do you have references that can be used to understand the SPSS outputs?

Thanks a lot for your help.

Junito

--- On Tue, 6/16/09, Kooij, A.J. van der <[hidden email]> wrote:

From: Kooij, A.J. van der <[hidden email]>
Subject: Re: Reliability and validity for a rubric type questionnaire
To: [hidden email]
Date: Tuesday, 16 June, 2009, 2:50 PM

Analyze, Dimension Reduction, Optimal Scaling, Choose "Some variable(s) not multiple nominal", click define.

From: Juanito Talili [mailto:[hidden email]]
Sent: 16 June 2009 16:48
To: [hidden email]; Kooij, A.J. van der
Subject: Re: Reliability and validity for a rubric type questionnaire

How to perform CATPCA? I am using SPSS 17.

Thank you.

Juanito

--- On Tue, 6/16/09, Kooij, A.J. van der <[hidden email]> wrote:

From: Kooij, A.J. van der <[hidden email]>
Subject: Re: Reliability and validity for a rubric type questionnaire
To: [hidden email]
Date: Tuesday, 16 June, 2009, 2:03 PM

CATPCA gives Cronbach's alpha (see the Model Summary table). This is alpha for the transformed variables, thus treating your variable as ordinal is taken into account.

Regards,

Anita van der Kooij

Data Theory Group

Leiden University

From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Eins Bernardo
Sent: 16 June 2009 03:42
To: [hidden email]
Subject: Re: Reliability and validity for a rubric type questionnaire

Thank you for your time, Art.

Actually, the response scale cannot be treated as interval. Thus, your suggestion to use CATPCA is the appropriate one.

I have a couple of questions: (1) For the reliability, considering that the response scale cannot be treated as interval, what coefficient is used in place of cronbach alpha? (2) For the validity, are there other ways considering that I dont have another measure of the same constract? That is, can we test the validity of the scale using only the present data (n<30, 9 items)?

Thank you.

Eins

--- On Mon, 6/15/09, Art Kendall <[hidden email]> wrote:

all
From: Art Kendall <[hidden email]>
Subject: Re: Reliability and validity for a rubric type questionnaire
To: [hidden email]
Date: Monday, 15 June, 2009, 12:29 PM

What is a "rubric" scale?
What are the labels on your response scale?

For the reliability:
If 1) you have a conventional summative scale, where you simply sum or average over a set of items, and 2) if your response scale is not very far from interval as is usually the situation with Likert items, then try the RELIABILITY procedure.

If your response scale is very discrepant from interval level, use CATPCA to see if you have one factor and if it makes much difference to treat the response scale as ordinal vs interval.

For validity, do you have another measure of the same construct?

Art Kendall
Social Research Consultants

Eins Bernardo wrote:

I wonder if we can do validation study via statistics for a rubric type instrument with only nine items and for each item has 5 descriptors (ordinal scale). My sample is only less than 30.

Thank you in advance for your inputs.

Eins

Yahoo! Mail Now Faster and Cleaner. Experience it today!

Connect instantly with more friends on your blog and personal website?
Create your latest Pingbox today!
**********************************************************************

This email and any files transmitted with it are confidential and

intended solely for the use of the individual or entity to whom they

are addressed. If you have received this email in error please notify

the system manager.

**********************************************************************

Surf faster and smarter with Firefox 3!

Firefox 3: Faster, More Secure,Customizable and FREE.

Kooij, A.J. van der

Re: Reliability and validity for a rubric type questionnaire

CATPCA: 1st chapter at https://openaccess.leidenuniv.nl/dspace/handle/1887/12386

CATREG: 1st chapter at https://openaccess.leidenuniv.nl/dspace/handle/1887/12096

From: Juanito Talili [mailto:[hidden email]]
Sent: 16 June 2009 17:02
To: [hidden email]; Kooij, A.J. van der
Subject: Re: Reliability and validity for a rubric type questionnaire

Do you have references that can be used to understand the SPSS outputs?

Thanks a lot for your help.

Junito

--- On Tue, 6/16/09, Kooij, A.J. van der <[hidden email]> wrote:

From: Kooij, A.J. van der <[hidden email]>
Subject: Re: Reliability and validity for a rubric type questionnaire
To: [hidden email]
Date: Tuesday, 16 June, 2009, 2:50 PM

Analyze, Dimension Reduction, Optimal Scaling, Choose "Some variable(s) not multiple nominal", click define.

From: Juanito Talili [mailto:[hidden email]]
Sent: 16 June 2009 16:48
To: [hidden email]; Kooij, A.J. van der
Subject: Re: Reliability and validity for a rubric type questionnaire

How to perform CATPCA? I am using SPSS 17.

Thank you.

Juanito

--- On Tue, 6/16/09, Kooij, A.J. van der <[hidden email]> wrote:

From: Kooij, A.J. van der <[hidden email]>
Subject: Re: Reliability and validity for a rubric type questionnaire
To: [hidden email]
Date: Tuesday, 16 June, 2009, 2:03 PM

CATPCA gives Cronbach's alpha (see the Model Summary table). This is alpha for the transformed variables, thus treating your variable as ordinal is taken into account.

Regards,

Anita van der Kooij

Data Theory Group

Leiden University

From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Eins Bernardo
Sent: 16 June 2009 03:42
To: [hidden email]
Subject: Re: Reliability and validity for a rubric type questionnaire

Thank you for your time, Art.

Actually, the response scale cannot be treated as interval. Thus, your suggestion to use CATPCA is the appropriate one.

I have a couple of questions: (1) For the reliability, considering that the response scale cannot be treated as interval, what coefficient is used in place of cronbach alpha? (2) For the validity, are there other ways considering that I dont have another measure of the same constract? That is, can we test the validity of the scale using only the present data (n<30, 9 items)?

Thank you.

Eins

--- On Mon, 6/15/09, Art Kendall <[hidden email]> wrote:

all
From: Art Kendall <[hidden email]>
Subject: Re: Reliability and validity for a rubric type questionnaire
To: [hidden email]
Date: Monday, 15 June, 2009, 12:29 PM

What is a "rubric" scale?
What are the labels on your response scale?

For the reliability:
If 1) you have a conventional summative scale, where you simply sum or average over a set of items, and 2) if your response scale is not very far from interval as is usually the situation with Likert items, then try the RELIABILITY procedure.

If your response scale is very discrepant from interval level, use CATPCA to see if you have one factor and if it makes much difference to treat the response scale as ordinal vs interval.

For validity, do you have another measure of the same construct?

Art Kendall
Social Research Consultants

Eins Bernardo wrote:

I wonder if we can do validation study via statistics for a rubric type instrument with only nine items and for each item has 5 descriptors (ordinal scale). My sample is only less than 30.

Thank you in advance for your inputs.

Eins

Yahoo! Mail Now Faster and Cleaner. Experience it today!

Connect instantly with more friends on your blog and personal website?
Create your latest Pingbox today!
**********************************************************************

This email and any files transmitted with it are confidential and

intended solely for the use of the individual or entity to whom they

are addressed. If you have received this email in error please notify

the system manager.

**********************************************************************

Surf faster and smarter with Firefox 3!

Firefox 3: Faster, More Secure,Customizable and FREE.