Interaction Effect, Yes or No
Locked
 
|
Hi, I'm interested to hear more thoughts about the following: The research question was to test if X (a continuous variables) associated with an outcome Y and how it differed by A (a 4-level categorical variable) after controlling for several other covariates. So I went ahead and tested the main effect of X, A and the interaction effect of X*A after controlling for the covariates. Even though the overall interaction effect is not significant, I still went a step further to test for individual slope within each level of A. And I found that two slopes were tested to be significantly different from zero and the other 2 were not. Some researchers in the team said that this is not reportable as we didn't have significant overall interaction effect. Some still think that the individual slopes meant something and we should not be discouraged by the insignificant interaction effect. My personal opinion will be, if we have 2-level categorical variable A, a non significant interaction between A* X will be clearer, i.e. the relationship between X and Y are indifferent for both A1 and A2. But since we have 4-level within variable A, the interaction test became an average test of the overall interaction effect which does not conclude all pairwise comparisons are insignificant. Therefore, even without overall significant interaction, I would test to see if individual slopes within variable A are different from zero, and/or compare slopes across variable A. Curious to hear what's your take on this situation. Thanks. Best regards Chiew Kwei Kaw (Okrae) phone: 404.329.7722 fax: 404.929.6832 |
Re: Interaction Effect, Yes or No
|
|
I would argue that since you did not have a priori hypotheses, it would be inappropriate to look at pairwise comparisons following a non-significant omnibus test.
Paul Swank ________________________________ From: SPSSX(r) Discussion [[hidden email]] On Behalf Of Chiewkwei Kaw [[hidden email]] Sent: Tuesday, July 28, 2009 10:32 AM To: [hidden email] Subject: Interaction Effect, Yes or No Hi, I'm interested to hear more thoughts about the following: The research question was to test if X (a continuous variables) associated with an outcome Y and how it differed by A (a 4-level categorical variable) after controlling for several other covariates. So I went ahead and tested the main effect of X, A and the interaction effect of X*A after controlling for the covariates. Even though the overall interaction effect is not significant, I still went a step further to test for individual slope within each level of A. And I found that two slopes were tested to be significantly different from zero and the other 2 were not. Some researchers in the team said that this is not reportable as we didn't have significant overall interaction effect. Some still think that the individual slopes meant something and we should not be discouraged by the insignificant interaction effect. My personal opinion will be, if we have 2-level categorical variable A, a non significant interaction between A* X will be clearer, i.e. the relationship between X and Y are indifferent for both A1 and A2. But since we have 4-level within variable A, the interaction test became an average test of the overall interaction effect which does not conclude all pairwise comparisons are insignificant. Therefore, even without overall significant interaction, I would test to see if individual slopes within variable A are different from zero, and/or compare slopes across variable A. Curious to hear what's your take on this situation. Thanks. Best regards Chiew Kwei Kaw (Okrae) phone: 404.329.7722 fax: 404.929.6832 ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD |
Re: Interaction Effect, Yes or No
|
|
Paul Swank, Thanks so much for your input. The PI of the project does believe that X and Y differ across levels of A, although she doesn't specify formally that this is her a priori hypotheses, but I think it fits into this definition. Aside from this case, in practice, throughout the entire research process, many of us do fishing expedition at some level and patch theory later after finding significance results. Setting up a priori hypotheses to me still seems abstracts as to exactly when it should be stated clearly and formally throughout the analysis process as one can easily argue that relevant significant findings are his/her a priori hypothesis. Curious to hear about your experience and reaction to those issues. Many thanks again for your valuable response! Best regards Chiew Kwei Kaw phone: 404.329.7722 fax: 404.929.6832
I would argue that since you did not have a priori hypotheses, it would be inappropriate to look at pairwise comparisons following a non-significant omnibus test. Paul Swank ________________________________ From: SPSSX(r) Discussion [[hidden email]] On Behalf Of Chiewkwei Kaw [[hidden email]] Sent: Tuesday, July 28, 2009 10:32 AM To: [hidden email] Subject: Interaction Effect, Yes or No Hi, I'm interested to hear more thoughts about the following: The research question was to test if X (a continuous variables) associated with an outcome Y and how it differed by A (a 4-level categorical variable) after controlling for several other covariates. So I went ahead and tested the main effect of X, A and the interaction effect of X*A after controlling for the covariates. Even though the overall interaction effect is not significant, I still went a step further to test for individual slope within each level of A. And I found that two slopes were tested to be significantly different from zero and the other 2 were not. Some researchers in the team said that this is not reportable as we didn't have significant overall interaction effect. Some still think that the individual slopes meant something and we should not be discouraged by the insignificant interaction effect. My personal opinion will be, if we have 2-level categorical variable A, a non significant interaction between A* X will be clearer, i.e. the relationship between X and Y are indifferent for both A1 and A2. But since we have 4-level within variable A, the interaction test became an average test of the overall interaction effect which does not conclude all pairwise comparisons are insignificant. Therefore, even without overall significant interaction, I would test to see if individual slopes within variable A are different from zero, and/or compare slopes across variable A. Curious to hear what's your take on this situation. Thanks. Best regards Chiew Kwei Kaw (Okrae) phone: 404.329.7722 fax: 404.929.6832 |
Re: Interaction Effect, Yes or No
|
|
Chiew Kwei and Paul: This appears to be a more subtle variant of a common misunderstanding: If effect A is significant, and effect B is not, then A and B must be different. But of course, the test in A might be significant at p=.04, and B non-significant at p=.06, meaning the effects are very similar. (This often appears as: "We found a difference for drug vs. placebo in men but not in women; therefore, men and women differ in their response to the drug.") In this case, if I follow it correctly, the issue is not whether one slope is significant and another is not, but rather whether the slopes differ from each other -- which I suspect brings you back to the non-significant interaction. Allan At 12:18 PM 7/28/2009, Chiewkwei Kaw wrote: Paul Swank, Research Consultant [hidden email] Business & Cell (any time): 215 820-8100 Home: Voice and fax (8am - 10pm, 7 days/week): 215 885-5313 Address: 108 Cliff Terrace, Wyncote, PA 19095 Visit my Web site at www.dissertationconsulting.net ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD |
Re: Interaction Effect, Yes or No
|
|
In reply to this post by Chiewkwei Kaw
Thanks again Paul and Allan for your valuable reply. As mentioned, my categorical variable have 4 levels, the interaction between categorical variable A and continuous variable X was not significant, but when I compared the slope of X at A1 and A2, the test is significant, but not for A1 vs A3 and A1 vs A4. Is it appropriate to report the different between A1 and A2 given no significant overall interaction effect? Appreciate your input as always. Best regards, Chiew Kwei Kaw phone: 404.329.7722 fax: 404.929.6832
Chiew Kwei and Paul: This appears to be a more subtle variant of a common misunderstanding: If effect A is significant, and effect B is not, then A and B must be different. But of course, the test in A might be significant at p=.04, and B non-significant at p=.06, meaning the effects are very similar. (This often appears as: "We found a difference for drug vs. placebo in men but not in women; therefore, men and women differ in their response to the drug.") In this case, if I follow it correctly, the issue is not whether one slope is significant and another is not, but rather whether the slopes differ from each other -- which I suspect brings you back to the non-significant interaction. Allan At 12:18 PM 7/28/2009, Chiewkwei Kaw wrote: Paul Swank, Thanks so much for your input. The PI of the project does believe that X and Y differ across levels of A, although she doesn't specify formally that this is her a priori hypotheses, but I think it fits into this definition. Aside from this case, in practice, throughout the entire research process, many of us do fishing expedition at some level and patch theory later after finding significance results. Setting up a priori hypotheses to me still seems abstracts as to exactly when it should be stated clearly and formally throughout the analysis process as one can easily argue that relevant significant findings are his/her a priori hypothesis. Curious to hear about your experience and reaction to those issues. Many thanks again for your valuable response! Best regards Chiew Kwei Kaw phone: 404.329.7722 fax: 404.929.6832 "Swank, Paul R" <[hidden email]> 07/28/2009 11:51 AM To "[hidden email]" <[hidden email]>, "[hidden email]" <[hidden email]> cc Subject RE: Interaction Effect, Yes or No I would argue that since you did not have a priori hypotheses, it would be inappropriate to look at pairwise comparisons following a non-significant omnibus test. Paul Swank ________________________________ From: SPSSX(r) Discussion [[hidden email]] On Behalf Of Chiewkwei Kaw [[hidden email]] Sent: Tuesday, July 28, 2009 10:32 AM To: [hidden email] Subject: Interaction Effect, Yes or No Hi, I'm interested to hear more thoughts about the following: The research question was to test if X (a continuous variables) associated with an outcome Y and how it differed by A (a 4-level categorical variable) after controlling for several other covariates. So I went ahead and tested the main effect of X, A and the interaction effect of X*A after controlling for the covariates. Even though the overall interaction effect is not significant, I still went a step further to test for individual slope within each level of A. And I found that two slopes were tested to be significantly different from zero and the other 2 were not. Some researchers in the team said that this is not reportable as we didn't have significant overall interaction effect. Some still think that the individual slopes meant something and we should not be discouraged by the insignificant interaction effect. My personal opinion will be, if we have 2-level categorical variable A, a non significant interaction between A* X will be clearer, i.e. the relationship between X and Y are indifferent for both A1 and A2. But since we have 4-level within variable A, the interaction test became an average test of the overall interaction effect which does not conclude all pairwise comparisons are insignificant. Therefore, even without overall significant interaction, I would test to see if individual slopes within variable A are different from zero, and/or compare slopes across variable A. Curious to hear what's your take on this situation. Thanks. Best regards Chiew Kwei Kaw (Okrae) phone: 404.329.7722 fax: 404.929.6832 Allan Lundy, PhD
|
«
Return to SPSSX Discussion
|
1 view|%1 views
| Free forum by Nabble | Edit this page |