suggestions for an analysis plan
Locked
 
|
I have the following scenario and was wondering about a possible analysis plan for it.
I am working with a student who is interested in determining the positive effects of interventions designed to help siblings cope with having a brother/sister who has a disability. She is compiling a list of interventions that the siblings might have participated in. These range from support groups, family-centered retreats, online educational resources, workshops offered by hospitals or community centers, participating in family-oriented treatment behavioral plans. the study is retrospective in that she will be asking participants if they have participated in these interventions and what impact they have on outcome variables, such as sibling relationship quality, acceptance of role as a family care-giver, understanding of the disability.
My questions are
1) In thinking about these interventions, using a checklist to see if they have participated in the interventions does not seem to capture the intensity or frequency of the intervention. A person might participate in an ongoing support group just once and the participation can last several months. A person might participate in a family-centered retreat, once or twice, and the participation can last a week at a time. A person might read online educational materials, and that can be sporadic and occur several times over a year or more. We do have some notion that certain interventions, such as a family-centered retreat, would have better positive impacts than others, such as reading online materials only. My question is are there suggestions on how to capture the frequency, duration, intensity of a person's participation in these interventions, other than by simply asking how many times, for how long, and how involved the person was in these interventions?
2) once we have that information, how might we scale or weight these interventions? Is there a certain weight or value we could assign to participating in an ongoing support group that would carry greater weight or value than for someone who only read online materials? Complicating matters is that persons are likely to indicate that they have participated in more than one intervention so how could we account for this? I was thinking multi-dimensional scaling might be one approach to sort these interventions into clusters, but then was not sure how to create some sort of index scale that differentiates two persons who had participated in say three, or just one, intervention(s).
3) After assigning these scores that indicate how much experience a person has with an array of interventions, perhaps a path analysis can help determine the associations between these "intervention scores" are related to better outcomes on sibling relationships, acceptance of role as a family care-giver, understanding of the disability.
I was wondering if this scenario brings to mind similar scenarios and if there were suggested analysis plans for this situation. Please comment if additional information is needed to make my inquiry clearer. Thanks in advance for comments, suggestions.
Peter
Peter Ji, Ph.D.
Core Faculty Adler School of Professional Psychology 17 North Dearborn Street Chicago, IL 60602 Peter Ji, Ph.D.
Core Faculty Adler School of Professional Psychology 17 North Dearborn Street Chicago, IL 60602 |
Re: suggestions for an analysis plan
|
|
First let me mention I think that in general people are reluctant to answer this kinds of questions on the SPSS forum. They are unrelated to SPSS specifically,
they aren’t even really stats related, they are evaluation related. See my responses to your questions below. Matthew J Poes Research Data Specialist Center for Prevention Research and Development University of Illinois 510 Devonshire Dr. Champaign, IL 61820 Phone: 217-265-4576 email:
[hidden email] From: SPSSX(r) Discussion [mailto:[hidden email]]
On Behalf Of Ji, Peter I have the following scenario and was wondering about a possible analysis plan for it.
I am working with a student who is interested in determining the positive effects of interventions designed to help siblings cope with having a brother/sister who has
a disability. She is compiling a list of interventions that the siblings might have participated in. These range from support groups, family-centered retreats, online educational resources, workshops offered by hospitals or community centers, participating
in family-oriented treatment behavioral plans. the study is retrospective in that she will be asking participants if they have participated in these interventions and what impact they have on outcome variables, such as sibling relationship quality, acceptance
of role as a family care-giver, understanding of the disability. MP: As you already know retrospective studies make the arguably untenable claim that the intervention happened before the outcome, and thus was responsible
for that outcome, even though all data was actually collected after both have happened. To make matters worse, you are asking people to self –report this connection. I think there would be a strong argument that anything found in such a study could easily
be spurious. I would much prefer to at least see some solid standardized outcome measures which show great reliability and great validity to be used as outcomes.
My questions are
1)
In thinking about these interventions, using a checklist to see if they have participated in the interventions does not seem to capture the intensity
or frequency of the intervention. A person might participate in an ongoing support group just once and the participation can last several months. A person might participate in a family-centered retreat, once or twice, and the participation can last a week
at a time. A person might read online educational materials, and that can be sporadic and occur several times over a year or more. We do have some notion that certain interventions, such as a family-centered retreat, would have better positive impacts than
others, such as reading online materials only. My question is are there suggestions on how to capture the frequency, duration, intensity of a person's participation in these interventions, other than by simply asking how many times, for how long, and how involved
the person was in these interventions? MP: This makes the assumption that intensity or duration are equal to greater outcome, and this isn’t necessarily true. However, I get what you are getting
at, and I would suggest a few things. First, put a time range on the intervention participation questions. Ask the participant in the last 30 days, how many times have they done internet reading on coping with siblings with disabilities. In the last year?
In terms of years, months, weeks, or days, I think that’s a theory driven question for now. The checklist you see will actually come from these questions after the fact, you can recode the questions into participate vs. not, but within that is also the frequency.
In terms of duration, that comes from the two questions, as you know that someone who said yes to 1 year and 30 days has long term participation, but someone with only the 30 days, does not have the long term participation. This allows you to get 3 pieces
of information from 2 questions, and reduces survey fatigue. You might argue that asking them how long (months, years, decades) of involvement is even better, as it gives you as better continuous metric of involvement, but you do have some issues there, as
memory is often faulty, and this overall design is relatively weak, so I’m not sure it buys you anything useful. Think about the interpretation of those coefficients, is it really more useful to say (for every week longer, there is a .05 change in Y outcome)
vs. to say that people who participated a year or more had .35 improvement in Y as compared with those who only participated in the last year.
2)
once we have that information, how might we scale or weight these interventions? Is there a certain weight or value we could assign to participating
in an ongoing support group that would carry greater weight or value than for someone who only read online materials? Complicating matters is that persons are likely to indicate that they have participated in more than one intervention so how could we account
for this? I was thinking multi-dimensional scaling might be one approach to sort these interventions into clusters, but then was not sure how to create some sort of index scale that differentiates two persons who had participated in say three, or just one,
intervention(s). MP: This is always tough, and it’s a common mistake people make to want to quantify the weighted importance of some predictor even though there is no actual
quantitative reason to do so (Quantitatively weighting a qualitative perception basically). Even if you knew that intervention A was more important and more effective than intervention B, there is no reason to say its twice as important, unless you specifically
have a study that always shows twice the effect. Even then, what does that do for you in terms of the final model interpretation. It’s only useful when creating some composite score of intervention participation, and that may get difficult to interpret.
It doesn’t sound to me like enough is known to even consider that. My argument would be that if this is fairly exploratory in nature, then leave everything equally weighted, and use this study as a step toward developing the intervention participation composite
score weights. You then can use the coefficients to reflect the weights. Typically you want large replicated samples before doing this, but for even a dissertation or thesis, I’d imagine this could be acceptable as a first step toward that. You may want
to consider specific weighted scaling approaches such as CATREG or CATPCA to develop this, and then use the output of these in your next predictive model. 3)
After assigning these scores that indicate how much experience a person has with an array of interventions, perhaps a path analysis can help determine
the associations between these "intervention scores" are related to better outcomes on sibling relationships, acceptance of role as a family care-giver, understanding of the disability.
MP: I would just suggest relying on modern methods of “path” analysis, as this term is fairly generic and old at this point, and frequently people do things
that numerous studies have shown to be outright wrong. Remember that, in effect, path analysis is moderation/mediation analysis, so use the methods developed there. At this point I think the most sophisticated “path” modeling is actually done via SEM, but
that would require a fairly large sample, and I have a suspicion you would be violating various assumptions with this design in trying to develop that model, making such an approach untenable. I rely heavily on the work by Andrew Hayes and his colleagues,
look specifically at his new “PROCESS” add on modules for SPSS, the associated paper, upcoming book, etc. This will help your student be certain to use the most appropriate methods for their project.
In terms of the final model analysis, I would try and break it down to a small number of interventions, rely possibly on factor analysis or CATPCA to develop
the intervention categories, and have this indicate the intervention experience. Then have a second variable which reflects the intervention exposure/duration. This will be the longevity variable. As I said, if you feel that getting at duration in a continuous
sense, then you can certainly do this, I just question it’s merit. I’d wonder if maybe asking it in blocks, participation in the last 30 days (recent participation), more than a year ago (long term participation). This can be used to create long vs. short
and recent vs. distal participation variables. Be creative in the wording of the questions such that you can get at what you want. Once you have your frequency/exposure variable, I would use this as a second class of variables. You may have only the one,
or you may create a bunch (to measure different aspects of exposure/experience). From there (since we are now on a common metric of time), you can interact the intervention variables with the exposure variables, and quantify the effect of the specific intervention
on a specific outcomes, relative to amount of time exposed to it. Your model then compares (if setup right) everything to no intervention exposure at all, and you can talk about it in terms of the effect that each intervention has on the outcome, as compared
to no intervention, and relative to amount of exposure time. Since these are all dummy coded, its just a bunch of 0/1 matrices. If you interact the, say, three intervention categories, then you can see the effect of cumulative intervention exposure has,
and again, using a full factorial design, the effect that these cumulative exposures have relative to time. However, remember the power issue, you are going to want at least 5 to 10 people in each of these groupings to make such interpretations useful. Having
one person who happened to participate in all three intervention groups and for a long time isn’t going to yield stable useful coefficients. Which brings about the issue of a proper power analysis to see what kind of sample is even needed to yield this model,
and if not, how complex can you go before running out of steam. I was wondering if this scenario brings to mind similar scenarios and if there were suggested analysis plans for this situation. Please comment if additional information
is needed to make my inquiry clearer. Thanks in advance for comments, suggestions.
Peter Peter Ji, Ph.D. Office 16-403 Peter Ji, Ph.D. Office 16-403 |
|
|
thanks for your response. I sent it to the listserve and it was approved for distribution. but because I've made an error in posting this to the listserve, next time I will refrain from sending questions like this to the forum.
p
Peter Ji, Ph.D.
Core Faculty Adler School of Professional Psychology 17 North Dearborn Street Chicago, IL 60602 From: Poes, Matthew Joseph [[hidden email]] Sent: Monday, July 16, 2012 9:27 AM To: Ji, Peter; '[hidden email]' Subject: RE: suggestions for an analysis plan First let me mention I think that in general people are reluctant to answer this kinds of questions on the SPSS forum. They are unrelated to SPSS specifically, they aren’t even really stats related, they are evaluation related. See my responses to your questions below. Matthew J Poes Research Data Specialist Center for Prevention Research and Development University of Illinois 510 Devonshire Dr. Champaign, IL 61820 Phone: 217-265-4576 email: [hidden email] From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Ji, Peter I have the following scenario and was wondering about a possible analysis plan for it. I am working with a student who is interested in determining the positive effects of interventions designed to help siblings cope with having a brother/sister who has a disability. She is compiling a list of interventions that the siblings might have participated in. These range from support groups, family-centered retreats, online educational resources, workshops offered by hospitals or community centers, participating in family-oriented treatment behavioral plans. the study is retrospective in that she will be asking participants if they have participated in these interventions and what impact they have on outcome variables, such as sibling relationship quality, acceptance of role as a family care-giver, understanding of the disability. MP: As you already know retrospective studies make the arguably untenable claim that the intervention happened before the outcome, and thus was responsible for that outcome, even though all data was actually collected after both have happened. To make matters worse, you are asking people to self –report this connection. I think there would be a strong argument that anything found in such a study could easily be spurious. I would much prefer to at least see some solid standardized outcome measures which show great reliability and great validity to be used as outcomes. My questions are 1) In thinking about these interventions, using a checklist to see if they have participated in the interventions does not seem to capture the intensity or frequency of the intervention. A person might participate in an ongoing support group just once and the participation can last several months. A person might participate in a family-centered retreat, once or twice, and the participation can last a week at a time. A person might read online educational materials, and that can be sporadic and occur several times over a year or more. We do have some notion that certain interventions, such as a family-centered retreat, would have better positive impacts than others, such as reading online materials only. My question is are there suggestions on how to capture the frequency, duration, intensity of a person's participation in these interventions, other than by simply asking how many times, for how long, and how involved the person was in these interventions? MP: This makes the assumption that intensity or duration are equal to greater outcome, and this isn’t necessarily true. However, I get what you are getting at, and I would suggest a few things. First, put a time range on the intervention participation questions. Ask the participant in the last 30 days, how many times have they done internet reading on coping with siblings with disabilities. In the last year? In terms of years, months, weeks, or days, I think that’s a theory driven question for now. The checklist you see will actually come from these questions after the fact, you can recode the questions into participate vs. not, but within that is also the frequency. In terms of duration, that comes from the two questions, as you know that someone who said yes to 1 year and 30 days has long term participation, but someone with only the 30 days, does not have the long term participation. This allows you to get 3 pieces of information from 2 questions, and reduces survey fatigue. You might argue that asking them how long (months, years, decades) of involvement is even better, as it gives you as better continuous metric of involvement, but you do have some issues there, as memory is often faulty, and this overall design is relatively weak, so I’m not sure it buys you anything useful. Think about the interpretation of those coefficients, is it really more useful to say (for every week longer, there is a .05 change in Y outcome) vs. to say that people who participated a year or more had .35 improvement in Y as compared with those who only participated in the last year. 2) once we have that information, how might we scale or weight these interventions? Is there a certain weight or value we could assign to participating in an ongoing support group that would carry greater weight or value than for someone who only read online materials? Complicating matters is that persons are likely to indicate that they have participated in more than one intervention so how could we account for this? I was thinking multi-dimensional scaling might be one approach to sort these interventions into clusters, but then was not sure how to create some sort of index scale that differentiates two persons who had participated in say three, or just one, intervention(s). MP: This is always tough, and it’s a common mistake people make to want to quantify the weighted importance of some predictor even though there is no actual quantitative reason to do so (Quantitatively weighting a qualitative perception basically). Even if you knew that intervention A was more important and more effective than intervention B, there is no reason to say its twice as important, unless you specifically have a study that always shows twice the effect. Even then, what does that do for you in terms of the final model interpretation. It’s only useful when creating some composite score of intervention participation, and that may get difficult to interpret. It doesn’t sound to me like enough is known to even consider that. My argument would be that if this is fairly exploratory in nature, then leave everything equally weighted, and use this study as a step toward developing the intervention participation composite score weights. You then can use the coefficients to reflect the weights. Typically you want large replicated samples before doing this, but for even a dissertation or thesis, I’d imagine this could be acceptable as a first step toward that. You may want to consider specific weighted scaling approaches such as CATREG or CATPCA to develop this, and then use the output of these in your next predictive model. 3) After assigning these scores that indicate how much experience a person has with an array of interventions, perhaps a path analysis can help determine the associations between these "intervention scores" are related to better outcomes on sibling relationships, acceptance of role as a family care-giver, understanding of the disability. MP: I would just suggest relying on modern methods of “path” analysis, as this term is fairly generic and old at this point, and frequently people do things that numerous studies have shown to be outright wrong. Remember that, in effect, path analysis is moderation/mediation analysis, so use the methods developed there. At this point I think the most sophisticated “path” modeling is actually done via SEM, but that would require a fairly large sample, and I have a suspicion you would be violating various assumptions with this design in trying to develop that model, making such an approach untenable. I rely heavily on the work by Andrew Hayes and his colleagues, look specifically at his new “PROCESS” add on modules for SPSS, the associated paper, upcoming book, etc. This will help your student be certain to use the most appropriate methods for their project. In terms of the final model analysis, I would try and break it down to a small number of interventions, rely possibly on factor analysis or CATPCA to develop the intervention categories, and have this indicate the intervention experience. Then have a second variable which reflects the intervention exposure/duration. This will be the longevity variable. As I said, if you feel that getting at duration in a continuous sense, then you can certainly do this, I just question it’s merit. I’d wonder if maybe asking it in blocks, participation in the last 30 days (recent participation), more than a year ago (long term participation). This can be used to create long vs. short and recent vs. distal participation variables. Be creative in the wording of the questions such that you can get at what you want. Once you have your frequency/exposure variable, I would use this as a second class of variables. You may have only the one, or you may create a bunch (to measure different aspects of exposure/experience). From there (since we are now on a common metric of time), you can interact the intervention variables with the exposure variables, and quantify the effect of the specific intervention on a specific outcomes, relative to amount of time exposed to it. Your model then compares (if setup right) everything to no intervention exposure at all, and you can talk about it in terms of the effect that each intervention has on the outcome, as compared to no intervention, and relative to amount of exposure time. Since these are all dummy coded, its just a bunch of 0/1 matrices. If you interact the, say, three intervention categories, then you can see the effect of cumulative intervention exposure has, and again, using a full factorial design, the effect that these cumulative exposures have relative to time. However, remember the power issue, you are going to want at least 5 to 10 people in each of these groupings to make such interpretations useful. Having one person who happened to participate in all three intervention groups and for a long time isn’t going to yield stable useful coefficients. Which brings about the issue of a proper power analysis to see what kind of sample is even needed to yield this model, and if not, how complex can you go before running out of steam. I was wondering if this scenario brings to mind similar scenarios and if there were suggested analysis plans for this situation. Please comment if additional information is needed to make my inquiry clearer. Thanks in advance for comments, suggestions. Peter Peter Ji, Ph.D. Office 16-403 Peter Ji, Ph.D. Office 16-403 |
Re: suggestions for an analysis plan
|
|
My comment was simply meant to reflect that you may get better responses on other forum’s. Even then, the evaluation/social science specific nature of this
question is tough, who knows what you would get. I still did my best to answer your questions, given that this is related to what I do for a living, and people at least hope I somewhat know what I’m doing.
Matthew J Poes Research Data Specialist Center for Prevention Research and Development University of Illinois 510 Devonshire Dr. Champaign, IL 61820 Phone: 217-265-4576 email:
[hidden email] From: Ji, Peter [mailto:[hidden email]]
thanks for your response. I sent it to the listserve and it was approved for distribution. but because I've made an error in posting this to the listserve, next
time I will refrain from sending questions like this to the forum. p Peter Ji, Ph.D. Office 16-403 From: Poes, Matthew Joseph [[hidden email]] First let me mention I think that in general people are reluctant to answer this kinds of questions on the SPSS forum. They are unrelated to SPSS specifically,
they aren’t even really stats related, they are evaluation related. See my responses to your questions below. Matthew J Poes Research Data Specialist Center for Prevention Research and Development University of Illinois 510 Devonshire Dr. Champaign, IL 61820 Phone: 217-265-4576 email:
[hidden email] From: SPSSX(r) Discussion
[hidden email]
On Behalf Of Ji, Peter I have the following scenario and was wondering about a possible analysis plan for it.
I am working with a student who is interested in determining the positive effects of interventions designed to help siblings cope with having a brother/sister who has
a disability. She is compiling a list of interventions that the siblings might have participated in. These range from support groups, family-centered retreats, online educational resources, workshops offered by hospitals or community centers, participating
in family-oriented treatment behavioral plans. the study is retrospective in that she will be asking participants if they have participated in these interventions and what impact they have on outcome variables, such as sibling relationship quality, acceptance
of role as a family care-giver, understanding of the disability. MP: As you already know retrospective studies make the arguably untenable claim that the intervention happened before the outcome, and thus was responsible
for that outcome, even though all data was actually collected after both have happened. To make matters worse, you are asking people to self –report this connection. I think there would be a strong argument that anything found in such a study could easily
be spurious. I would much prefer to at least see some solid standardized outcome measures which show great reliability and great validity to be used as outcomes.
My questions are
1)
In thinking about these interventions, using a checklist to see if they have participated in the interventions does not seem to capture the intensity or frequency of the intervention.
A person might participate in an ongoing support group just once and the participation can last several months. A person might participate in a family-centered retreat, once or twice, and the participation can last a week at a time. A person might read online
educational materials, and that can be sporadic and occur several times over a year or more. We do have some notion that certain interventions, such as a family-centered retreat, would have better positive impacts than others, such as reading online materials
only. My question is are there suggestions on how to capture the frequency, duration, intensity of a person's participation in these interventions, other than by simply asking how many times, for how long, and how involved the person was in these interventions?
MP: This makes the assumption that intensity or duration are equal to greater outcome, and this isn’t necessarily true. However, I get what you are getting
at, and I would suggest a few things. First, put a time range on the intervention participation questions. Ask the participant in the last 30 days, how many times have they done internet reading on coping with siblings with disabilities. In the last year?
In terms of years, months, weeks, or days, I think that’s a theory driven question for now. The checklist you see will actually come from these questions after the fact, you can recode the questions into participate vs. not, but within that is also the frequency.
In terms of duration, that comes from the two questions, as you know that someone who said yes to 1 year and 30 days has long term participation, but someone with only the 30 days, does not have the long term participation. This allows you to get 3 pieces
of information from 2 questions, and reduces survey fatigue. You might argue that asking them how long (months, years, decades) of involvement is even better, as it gives you as better continuous metric of involvement, but you do have some issues there, as
memory is often faulty, and this overall design is relatively weak, so I’m not sure it buys you anything useful. Think about the interpretation of those coefficients, is it really more useful to say (for every week longer, there is a .05 change in Y outcome)
vs. to say that people who participated a year or more had .35 improvement in Y as compared with those who only participated in the last year.
2)
once we have that information, how might we scale or weight these interventions? Is there a certain weight or value we could assign to participating in an ongoing support group
that would carry greater weight or value than for someone who only read online materials? Complicating matters is that persons are likely to indicate that they have participated in more than one intervention so how could we account for this? I was thinking
multi-dimensional scaling might be one approach to sort these interventions into clusters, but then was not sure how to create some sort of index scale that differentiates two persons who had participated in say three, or just one, intervention(s). MP: This is always tough, and it’s a common mistake people make to want to quantify the weighted importance of some predictor even though there is no actual
quantitative reason to do so (Quantitatively weighting a qualitative perception basically). Even if you knew that intervention A was more important and more effective than intervention B, there is no reason to say its twice as important, unless you specifically
have a study that always shows twice the effect. Even then, what does that do for you in terms of the final model interpretation. It’s only useful when creating some composite score of intervention participation, and that may get difficult to interpret.
It doesn’t sound to me like enough is known to even consider that. My argument would be that if this is fairly exploratory in nature, then leave everything equally weighted, and use this study as a step toward developing the intervention participation composite
score weights. You then can use the coefficients to reflect the weights. Typically you want large replicated samples before doing this, but for even a dissertation or thesis, I’d imagine this could be acceptable as a first step toward that. You may want
to consider specific weighted scaling approaches such as CATREG or CATPCA to develop this, and then use the output of these in your next predictive model. 3)
After assigning these scores that indicate how much experience a person has with an array of interventions, perhaps a path analysis can help determine the associations between
these "intervention scores" are related to better outcomes on sibling relationships, acceptance of role as a family care-giver, understanding of the disability.
MP: I would just suggest relying on modern methods of “path” analysis, as this term is fairly generic and old at this point, and frequently people do things
that numerous studies have shown to be outright wrong. Remember that, in effect, path analysis is moderation/mediation analysis, so use the methods developed there. At this point I think the most sophisticated “path” modeling is actually done via SEM, but
that would require a fairly large sample, and I have a suspicion you would be violating various assumptions with this design in trying to develop that model, making such an approach untenable. I rely heavily on the work by Andrew Hayes and his colleagues,
look specifically at his new “PROCESS” add on modules for SPSS, the associated paper, upcoming book, etc. This will help your student be certain to use the most appropriate methods for their project.
In terms of the final model analysis, I would try and break it down to a small number of interventions, rely possibly on factor analysis or CATPCA to develop
the intervention categories, and have this indicate the intervention experience. Then have a second variable which reflects the intervention exposure/duration. This will be the longevity variable. As I said, if you feel that getting at duration in a continuous
sense, then you can certainly do this, I just question it’s merit. I’d wonder if maybe asking it in blocks, participation in the last 30 days (recent participation), more than a year ago (long term participation). This can be used to create long vs. short
and recent vs. distal participation variables. Be creative in the wording of the questions such that you can get at what you want. Once you have your frequency/exposure variable, I would use this as a second class of variables. You may have only the one,
or you may create a bunch (to measure different aspects of exposure/experience). From there (since we are now on a common metric of time), you can interact the intervention variables with the exposure variables, and quantify the effect of the specific intervention
on a specific outcomes, relative to amount of time exposed to it. Your model then compares (if setup right) everything to no intervention exposure at all, and you can talk about it in terms of the effect that each intervention has on the outcome, as compared
to no intervention, and relative to amount of exposure time. Since these are all dummy coded, its just a bunch of 0/1 matrices. If you interact the, say, three intervention categories, then you can see the effect of cumulative intervention exposure has,
and again, using a full factorial design, the effect that these cumulative exposures have relative to time. However, remember the power issue, you are going to want at least 5 to 10 people in each of these groupings to make such interpretations useful. Having
one person who happened to participate in all three intervention groups and for a long time isn’t going to yield stable useful coefficients. Which brings about the issue of a proper power analysis to see what kind of sample is even needed to yield this model,
and if not, how complex can you go before running out of steam. I was wondering if this scenario brings to mind similar scenarios and if there were suggested analysis plans for this situation. Please comment if additional information
is needed to make my inquiry clearer. Thanks in advance for comments, suggestions.
Peter Peter Ji, Ph.D. Office 16-403 Peter Ji, Ph.D. Office 16-403 |
|
|
thank you again for your very thoughtful response. in addition to my own internet search for an evaluation forum/listserve, can you recommend some? it would be helpful for these types of questions.
peter
Peter Ji, Ph.D.
Core Faculty Adler School of Professional Psychology 17 North Dearborn Street Chicago, IL 60602 From: Poes, Matthew Joseph [[hidden email]] Sent: Monday, July 16, 2012 9:58 AM To: Ji, Peter; '[hidden email]' Subject: RE: suggestions for an analysis plan My comment was simply meant to reflect that you may get better responses on other forum’s. Even then, the evaluation/social science specific nature of this question is tough, who knows what you would get. I still did my best to answer your questions, given that this is related to what I do for a living, and people at least hope I somewhat know what I’m doing. Matthew J Poes Research Data Specialist Center for Prevention Research and Development University of Illinois 510 Devonshire Dr. Champaign, IL 61820 Phone: 217-265-4576 email: [hidden email] From: Ji, Peter [mailto:[hidden email]] thanks for your response. I sent it to the listserve and it was approved for distribution. but because I've made an error in posting this to the listserve, next time I will refrain from sending questions like this to the forum. p Peter Ji, Ph.D. Office 16-403 From: Poes, Matthew Joseph [[hidden email]] First let me mention I think that in general people are reluctant to answer this kinds of questions on the SPSS forum. They are unrelated to SPSS specifically, they aren’t even really stats related, they are evaluation related. See my responses to your questions below. Matthew J Poes Research Data Specialist Center for Prevention Research and Development University of Illinois 510 Devonshire Dr. Champaign, IL 61820 Phone: 217-265-4576 email: [hidden email] From: SPSSX(r) Discussion [hidden email] On Behalf Of Ji, Peter I have the following scenario and was wondering about a possible analysis plan for it. I am working with a student who is interested in determining the positive effects of interventions designed to help siblings cope with having a brother/sister who has a disability. She is compiling a list of interventions that the siblings might have participated in. These range from support groups, family-centered retreats, online educational resources, workshops offered by hospitals or community centers, participating in family-oriented treatment behavioral plans. the study is retrospective in that she will be asking participants if they have participated in these interventions and what impact they have on outcome variables, such as sibling relationship quality, acceptance of role as a family care-giver, understanding of the disability. MP: As you already know retrospective studies make the arguably untenable claim that the intervention happened before the outcome, and thus was responsible for that outcome, even though all data was actually collected after both have happened. To make matters worse, you are asking people to self –report this connection. I think there would be a strong argument that anything found in such a study could easily be spurious. I would much prefer to at least see some solid standardized outcome measures which show great reliability and great validity to be used as outcomes. My questions are 1) In thinking about these interventions, using a checklist to see if they have participated in the interventions does not seem to capture the intensity or frequency of the intervention. A person might participate in an ongoing support group just once and the participation can last several months. A person might participate in a family-centered retreat, once or twice, and the participation can last a week at a time. A person might read online educational materials, and that can be sporadic and occur several times over a year or more. We do have some notion that certain interventions, such as a family-centered retreat, would have better positive impacts than others, such as reading online materials only. My question is are there suggestions on how to capture the frequency, duration, intensity of a person's participation in these interventions, other than by simply asking how many times, for how long, and how involved the person was in these interventions? MP: This makes the assumption that intensity or duration are equal to greater outcome, and this isn’t necessarily true. However, I get what you are getting at, and I would suggest a few things. First, put a time range on the intervention participation questions. Ask the participant in the last 30 days, how many times have they done internet reading on coping with siblings with disabilities. In the last year? In terms of years, months, weeks, or days, I think that’s a theory driven question for now. The checklist you see will actually come from these questions after the fact, you can recode the questions into participate vs. not, but within that is also the frequency. In terms of duration, that comes from the two questions, as you know that someone who said yes to 1 year and 30 days has long term participation, but someone with only the 30 days, does not have the long term participation. This allows you to get 3 pieces of information from 2 questions, and reduces survey fatigue. You might argue that asking them how long (months, years, decades) of involvement is even better, as it gives you as better continuous metric of involvement, but you do have some issues there, as memory is often faulty, and this overall design is relatively weak, so I’m not sure it buys you anything useful. Think about the interpretation of those coefficients, is it really more useful to say (for every week longer, there is a .05 change in Y outcome) vs. to say that people who participated a year or more had .35 improvement in Y as compared with those who only participated in the last year. 2) once we have that information, how might we scale or weight these interventions? Is there a certain weight or value we could assign to participating in an ongoing support group that would carry greater weight or value than for someone who only read online materials? Complicating matters is that persons are likely to indicate that they have participated in more than one intervention so how could we account for this? I was thinking multi-dimensional scaling might be one approach to sort these interventions into clusters, but then was not sure how to create some sort of index scale that differentiates two persons who had participated in say three, or just one, intervention(s). MP: This is always tough, and it’s a common mistake people make to want to quantify the weighted importance of some predictor even though there is no actual quantitative reason to do so (Quantitatively weighting a qualitative perception basically). Even if you knew that intervention A was more important and more effective than intervention B, there is no reason to say its twice as important, unless you specifically have a study that always shows twice the effect. Even then, what does that do for you in terms of the final model interpretation. It’s only useful when creating some composite score of intervention participation, and that may get difficult to interpret. It doesn’t sound to me like enough is known to even consider that. My argument would be that if this is fairly exploratory in nature, then leave everything equally weighted, and use this study as a step toward developing the intervention participation composite score weights. You then can use the coefficients to reflect the weights. Typically you want large replicated samples before doing this, but for even a dissertation or thesis, I’d imagine this could be acceptable as a first step toward that. You may want to consider specific weighted scaling approaches such as CATREG or CATPCA to develop this, and then use the output of these in your next predictive model. 3) After assigning these scores that indicate how much experience a person has with an array of interventions, perhaps a path analysis can help determine the associations between these "intervention scores" are related to better outcomes on sibling relationships, acceptance of role as a family care-giver, understanding of the disability. MP: I would just suggest relying on modern methods of “path” analysis, as this term is fairly generic and old at this point, and frequently people do things that numerous studies have shown to be outright wrong. Remember that, in effect, path analysis is moderation/mediation analysis, so use the methods developed there. At this point I think the most sophisticated “path” modeling is actually done via SEM, but that would require a fairly large sample, and I have a suspicion you would be violating various assumptions with this design in trying to develop that model, making such an approach untenable. I rely heavily on the work by Andrew Hayes and his colleagues, look specifically at his new “PROCESS” add on modules for SPSS, the associated paper, upcoming book, etc. This will help your student be certain to use the most appropriate methods for their project. In terms of the final model analysis, I would try and break it down to a small number of interventions, rely possibly on factor analysis or CATPCA to develop the intervention categories, and have this indicate the intervention experience. Then have a second variable which reflects the intervention exposure/duration. This will be the longevity variable. As I said, if you feel that getting at duration in a continuous sense, then you can certainly do this, I just question it’s merit. I’d wonder if maybe asking it in blocks, participation in the last 30 days (recent participation), more than a year ago (long term participation). This can be used to create long vs. short and recent vs. distal participation variables. Be creative in the wording of the questions such that you can get at what you want. Once you have your frequency/exposure variable, I would use this as a second class of variables. You may have only the one, or you may create a bunch (to measure different aspects of exposure/experience). From there (since we are now on a common metric of time), you can interact the intervention variables with the exposure variables, and quantify the effect of the specific intervention on a specific outcomes, relative to amount of time exposed to it. Your model then compares (if setup right) everything to no intervention exposure at all, and you can talk about it in terms of the effect that each intervention has on the outcome, as compared to no intervention, and relative to amount of exposure time. Since these are all dummy coded, its just a bunch of 0/1 matrices. If you interact the, say, three intervention categories, then you can see the effect of cumulative intervention exposure has, and again, using a full factorial design, the effect that these cumulative exposures have relative to time. However, remember the power issue, you are going to want at least 5 to 10 people in each of these groupings to make such interpretations useful. Having one person who happened to participate in all three intervention groups and for a long time isn’t going to yield stable useful coefficients. Which brings about the issue of a proper power analysis to see what kind of sample is even needed to yield this model, and if not, how complex can you go before running out of steam. I was wondering if this scenario brings to mind similar scenarios and if there were suggested analysis plans for this situation. Please comment if additional information is needed to make my inquiry clearer. Thanks in advance for comments, suggestions. Peter Peter Ji, Ph.D. Office 16-403 Peter Ji, Ph.D. Office 16-403 |
| Free forum by Nabble | Edit this page |