Bonferroni correction and number of comparisons
Locked
 
|
Hi,
I am doing a post hoc analysis comparing column proportions and adjusting (Bonferroni) for multiple comparisons (CTables, test statistics option). I've been asked to provide the corrected p-value. I'd like to know if I am correct in reporting the corrected value as .008. The row variable has two levels (yes and no) and the column variable has 4 levels, so I am making 6 comparisons for 'no' and 6 for 'yes'. n*(n-1)/2 The output shows the comparison for the 'no' and for the yes level of the row variable. Is the Bonferroni adjustment for 6 comparisons and the corrected p-value (.05/6=.008333) Also, I have 19 row variables for which I am running column proportion comparisons. I think that the comparisons for each of the 19 variables are considered independent from each other so no further adjustment is made by SPSS. Is this the way reviewers look at the question? Thanks for any help, Jan ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD |
|
|
You asked about applying a Bonferroni correction in post hoc tests-->
"Is the Bonferroni adjustment for 6 comparisons and the corrected p-value (.05/6=.008333)" Answer. No. You have presumably adjusted the alpha level, not the p-values. Multiply the each p-value by the number of post hoc tests performed (6 in the example you provided). Then compare the adjusted p-values to the alpha level you set (e.g., .05). Ryan On Wed, Jan 19, 2011 at 1:56 PM, J McClure <[hidden email]> wrote: > Hi, > I am doing a post hoc analysis comparing column proportions and > adjusting (Bonferroni) for multiple comparisons (CTables, test > statistics option). > I've been asked to provide the corrected p-value. > I'd like to know if I am correct in reporting the corrected value as .008. > The row variable has two levels (yes and no) and the column variable has > 4 levels, so I am making 6 comparisons for 'no' and 6 for 'yes'. n*(n-1)/2 > The output shows the comparison for the 'no' and for the yes level of > the row variable. Is the Bonferroni adjustment for 6 comparisons and the > corrected p-value (.05/6=.008333) > Also, I have 19 row variables for which I am running column proportion > comparisons. I think that the comparisons for each of the 19 variables > are considered independent from each other so no further adjustment is > made by SPSS. Is this the way reviewers look at the question? > Thanks for any help, > Jan > > ===================== > To manage your subscription to SPSSX-L, send a message to > [hidden email] (not to SPSSX-L), with no body text except the > command. To leave the list, send the command > SIGNOFF SPSSX-L > For a list of commands to manage subscriptions, send the command > INFO REFCARD > ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD |
Re: Bonferroni correction and number of comparisons
|
|
I agree with what Ryan has said, but I believe that the "straight-up Bonferroni" correction is overly conservative. What about considering a sequential approach such as the Holm modification of the Bonferroni correction?
wbw __________________________________________________________________________ William B. Ware, Professor Educational Psychology, CB# 3500 Measurement, and Evaluation University of North Carolina PHONE (919)-962-2511 Chapel Hill, NC 27599-3500 FAX: (919)-962-1533 Office: 118 Peabody Hall EMAIL: [hidden email] Adjunct Professor School of Social Work Academy of Distinguished Teaching Scholars at UNC-Chapel Hill __________________________________________________________________________ -----Original Message----- From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of R B Sent: Wednesday, January 19, 2011 3:58 PM To: [hidden email] Subject: Re: Bonferroni correction and number of comparisons You asked about applying a Bonferroni correction in post hoc tests--> "Is the Bonferroni adjustment for 6 comparisons and the corrected p-value (.05/6=.008333)" Answer. No. You have presumably adjusted the alpha level, not the p-values. Multiply the each p-value by the number of post hoc tests performed (6 in the example you provided). Then compare the adjusted p-values to the alpha level you set (e.g., .05). Ryan On Wed, Jan 19, 2011 at 1:56 PM, J McClure <[hidden email]> wrote: > Hi, > I am doing a post hoc analysis comparing column proportions and > adjusting (Bonferroni) for multiple comparisons (CTables, test > statistics option). > I've been asked to provide the corrected p-value. > I'd like to know if I am correct in reporting the corrected value as .008. > The row variable has two levels (yes and no) and the column variable has > 4 levels, so I am making 6 comparisons for 'no' and 6 for 'yes'. n*(n-1)/2 > The output shows the comparison for the 'no' and for the yes level of > the row variable. Is the Bonferroni adjustment for 6 comparisons and the > corrected p-value (.05/6=.008333) > Also, I have 19 row variables for which I am running column proportion > comparisons. I think that the comparisons for each of the 19 variables > are considered independent from each other so no further adjustment is > made by SPSS. Is this the way reviewers look at the question? > Thanks for any help, > Jan > > ===================== > To manage your subscription to SPSSX-L, send a message to > [hidden email] (not to SPSSX-L), with no body text except the > command. To leave the list, send the command > SIGNOFF SPSSX-L > For a list of commands to manage subscriptions, send the command > INFO REFCARD > ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD |
Re: Bonferroni correction and number of comparisons
Administrator
|
In reply to this post by Ryan
Ryan, multiplying the observed (uncorrected) p-values by the number of contrasts can give you results that make no sense. E.g., if the uncorrected p-value = .25 and the number of contrasts is 6, you'll get a corrected p-value = 1.5. But p-values are conditional probabilities, and must fall within the range 0-1.
Here's an example using oneway ANOVA. Notice that the Bonferroni-corrected p-values (or Sig. values as SPSS labels them) are not simply 3 x the uncorrected (LSD) p-values. MATRIX DATA VARIABLES=Group ROWTYPE_ Score /FACTORS=Group. BEGIN DATA 1 N 96 2 N 96 3 N 96 1 MEAN 22.98 2 MEAN 25.78 3 MEAN 26.56 1 STDDEV 8.79 2 STDDEV 9.08 3 STDDEV 8.50 END DATA. ONEWAY Score BY group / matrix = in(*) / POSTHOC=LSD BONFERRONI ALPHA(0.05) . Jan, given that figuring out how to compute corrected p-values may be quite time-consuming, I would see if I could persuade the person who asked that the usual approach (i.e., comparing p to a corrected alpha level) is sufficient. HTH.
--
Bruce Weaver bweaver@lakeheadu.ca http://sites.google.com/a/lakeheadu.ca/bweaver/ "When all else fails, RTFM." PLEASE NOTE THE FOLLOWING: 1. My Hotmail account is not monitored regularly. To send me an e-mail, please use the address shown above. 2. The SPSSX Discussion forum on Nabble is no longer linked to the SPSSX-L listserv administered by UGA (https://listserv.uga.edu/). |
|
|
Thanks for the example. SPSS doesn't provide p-values for column
proportion comparisons (at least when using CTABLES) so it was useful to see an example in ANOVA. Your suggestion makes sense to me. > I would see if I could persuade the person who asked that > the usual approach (i.e., comparing p to a corrected alpha level) is > sufficient. Is the corrected alpha level .05/6?? On 1/19/2011 1:47 PM, Bruce Weaver wrote: > Ryan, multiplying the observed (uncorrected) p-values by the number of > contrasts can give you results that make no sense. E.g., if the uncorrected > p-value = .25 and the number of contrasts is 6, you'll get a corrected > p-value = 1.5. But p-values are conditional probabilities, and must fall > within the range 0-1. > > Here's an example using oneway ANOVA. Notice that the Bonferroni-corrected > p-values (or Sig. values as SPSS labels them) are not simply 3 x the > uncorrected (LSD) p-values. > > MATRIX DATA VARIABLES=Group ROWTYPE_ Score /FACTORS=Group. > BEGIN DATA > 1 N 96 > 2 N 96 > 3 N 96 > 1 MEAN 22.98 > 2 MEAN 25.78 > 3 MEAN 26.56 > 1 STDDEV 8.79 > 2 STDDEV 9.08 > 3 STDDEV 8.50 > END DATA. > > ONEWAY Score BY group / > matrix = in(*) / > POSTHOC=LSD BONFERRONI ALPHA(0.05) > . > > Jan, given that figuring out how to compute corrected p-values may be quite > time-consuming, I would see if I could persuade the person who asked that > the usual approach (i.e., comparing p to a corrected alpha level) is > sufficient. > > HTH. > > > > R B wrote: >> You asked about applying a Bonferroni correction in post hoc tests--> >> "Is the Bonferroni adjustment for 6 comparisons and the corrected >> p-value (.05/6=.008333)" >> >> Answer. No. You have presumably adjusted the alpha level, not the >> p-values. Multiply the each p-value by the number of post hoc tests >> performed (6 in the example you provided). Then compare the adjusted >> p-values to the alpha level you set (e.g., .05). >> >> Ryan >> >> On Wed, Jan 19, 2011 at 1:56 PM, J McClure<[hidden email]> wrote: >>> Hi, >>> I am doing a post hoc analysis comparing column proportions and >>> adjusting (Bonferroni) for multiple comparisons (CTables, test >>> statistics option). >>> I've been asked to provide the corrected p-value. >>> I'd like to know if I am correct in reporting the corrected value as >>> .008. >>> The row variable has two levels (yes and no) and the column variable has >>> 4 levels, so I am making 6 comparisons for 'no' and 6 for 'yes'. >>> n*(n-1)/2 >>> The output shows the comparison for the 'no' and for the yes level of >>> the row variable. Is the Bonferroni adjustment for 6 comparisons and the >>> corrected p-value (.05/6=.008333) >>> Also, I have 19 row variables for which I am running column proportion >>> comparisons. I think that the comparisons for each of the 19 variables >>> are considered independent from each other so no further adjustment is >>> made by SPSS. Is this the way reviewers look at the question? >>> Thanks for any help, >>> Jan >>> >>> ===================== >>> To manage your subscription to SPSSX-L, send a message to >>> [hidden email] (not to SPSSX-L), with no body text except the >>> command. To leave the list, send the command >>> SIGNOFF SPSSX-L >>> For a list of commands to manage subscriptions, send the command >>> INFO REFCARD >>> >> ===================== >> To manage your subscription to SPSSX-L, send a message to >> [hidden email] (not to SPSSX-L), with no body text except the >> command. To leave the list, send the command >> SIGNOFF SPSSX-L >> For a list of commands to manage subscriptions, send the command >> INFO REFCARD >> >> > > ----- > -- > Bruce Weaver > [hidden email] > http://sites.google.com/a/lakeheadu.ca/bweaver/ > > "When all else fails, RTFM." > > NOTE: My Hotmail account is not monitored regularly. > To send me an e-mail, please use the address shown above. > > -- > View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Bonferroni-correction-and-number-of-comparisons-tp3348337p3348646.html > Sent from the SPSSX Discussion mailing list archive at Nabble.com. > > ===================== > To manage your subscription to SPSSX-L, send a message to > [hidden email] (not to SPSSX-L), with no body text except the > command. To leave the list, send the command > SIGNOFF SPSSX-L > For a list of commands to manage subscriptions, send the command > INFO REFCARD > ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD |
Re: Bonferroni correction and number of comparisons
|
Administrator
|
Yes, the adjusted alpha = the family-wise alpha you wish to maintain (often .05) divided by the number of contrasts.
--
Bruce Weaver bweaver@lakeheadu.ca http://sites.google.com/a/lakeheadu.ca/bweaver/ "When all else fails, RTFM." PLEASE NOTE THE FOLLOWING: 1. My Hotmail account is not monitored regularly. To send me an e-mail, please use the address shown above. 2. The SPSSX Discussion forum on Nabble is no longer linked to the SPSSX-L listserv administered by UGA (https://listserv.uga.edu/). |
|
|
In reply to this post by Bruce Weaver
Bruce,
See a couple of comments interspersed below. On Wed, Jan 19, 2011 at 4:47 PM, Bruce Weaver <[hidden email]> wrote: > Ryan, multiplying the observed (uncorrected) p-values by the number of > contrasts can give you results that make no sense. E.g., if the uncorrected > p-value = .25 and the number of contrasts is 6, you'll get a corrected > p-value = 1.5. But p-values are conditional probabilities, and must fall > within the range 0-1. This is a valid point, but I've yet to encounter a situation in my work where I would've made a different conclusion by correcting alpha directly. > > Here's an example using oneway ANOVA. Notice that the Bonferroni-corrected > p-values (or Sig. values as SPSS labels them) are not simply 3 x the > uncorrected (LSD) p-values. I disagree. In your example, Bonferroni corrected p values associated with 1v2 and 1v3 are exactly the same uncorrected p-values times 3. Is there something I'm missing here? My guess is that SPSS automatically replaced the Bonferroni corrected p-value associated with 2v3 with 1.0, since as you pointed out the conditional probability cannot be above 1.0. > > MATRIX DATA VARIABLES=Group ROWTYPE_ Score /FACTORS=Group. > BEGIN DATA > 1 N 96 > 2 N 96 > 3 N 96 > 1 MEAN 22.98 > 2 MEAN 25.78 > 3 MEAN 26.56 > 1 STDDEV 8.79 > 2 STDDEV 9.08 > 3 STDDEV 8.50 > END DATA. > > ONEWAY Score BY group / > matrix = in(*) / > POSTHOC=LSD BONFERRONI ALPHA(0.05) > . > > Jan, given that figuring out how to compute corrected p-values may be quite > time-consuming, I would see if I could persuade the person who asked that > the usual approach (i.e., comparing p to a corrected alpha level) is > sufficient. > > HTH. > > > > R B wrote: >> >> You asked about applying a Bonferroni correction in post hoc tests--> >> "Is the Bonferroni adjustment for 6 comparisons and the corrected >> p-value (.05/6=.008333)" >> >> Answer. No. You have presumably adjusted the alpha level, not the >> p-values. Multiply the each p-value by the number of post hoc tests >> performed (6 in the example you provided). Then compare the adjusted >> p-values to the alpha level you set (e.g., .05). >> >> Ryan >> >> On Wed, Jan 19, 2011 at 1:56 PM, J McClure <[hidden email]> wrote: >>> Hi, >>> I am doing a post hoc analysis comparing column proportions and >>> adjusting (Bonferroni) for multiple comparisons (CTables, test >>> statistics option). >>> I've been asked to provide the corrected p-value. >>> I'd like to know if I am correct in reporting the corrected value as >>> .008. >>> The row variable has two levels (yes and no) and the column variable has >>> 4 levels, so I am making 6 comparisons for 'no' and 6 for 'yes'. >>> n*(n-1)/2 >>> The output shows the comparison for the 'no' and for the yes level of >>> the row variable. Is the Bonferroni adjustment for 6 comparisons and the >>> corrected p-value (.05/6=.008333) >>> Also, I have 19 row variables for which I am running column proportion >>> comparisons. I think that the comparisons for each of the 19 variables >>> are considered independent from each other so no further adjustment is >>> made by SPSS. Is this the way reviewers look at the question? >>> Thanks for any help, >>> Jan >>> >>> ===================== >>> To manage your subscription to SPSSX-L, send a message to >>> [hidden email] (not to SPSSX-L), with no body text except the >>> command. To leave the list, send the command >>> SIGNOFF SPSSX-L >>> For a list of commands to manage subscriptions, send the command >>> INFO REFCARD >>> >> >> ===================== >> To manage your subscription to SPSSX-L, send a message to >> [hidden email] (not to SPSSX-L), with no body text except the >> command. To leave the list, send the command >> SIGNOFF SPSSX-L >> For a list of commands to manage subscriptions, send the command >> INFO REFCARD >> >> > > > ----- > -- > Bruce Weaver > [hidden email] > http://sites.google.com/a/lakeheadu.ca/bweaver/ > > "When all else fails, RTFM." > > NOTE: My Hotmail account is not monitored regularly. > To send me an e-mail, please use the address shown above. > > -- > View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Bonferroni-correction-and-number-of-comparisons-tp3348337p3348646.html > Sent from the SPSSX Discussion mailing list archive at Nabble.com. > > ===================== > To manage your subscription to SPSSX-L, send a message to > [hidden email] (not to SPSSX-L), with no body text except the > command. To leave the list, send the command > SIGNOFF SPSSX-L > For a list of commands to manage subscriptions, send the command > INFO REFCARD > ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD |
Re: Bonferroni correction and number of comparisons
|
Administrator
|
Hi Ryan. Exporting to Excel and showing lots of decimals shows that you are correct. I.e., SPSS appears to be computing the Bonferronii-corrected p-value as the uncorrected p * the number of contrasts, and if the result is greater than 1, setting it to 1. I expect the details are given somewhere in the ONEWAY algorithms, if Jan wants to confirm this before proceeding.
--
Bruce Weaver bweaver@lakeheadu.ca http://sites.google.com/a/lakeheadu.ca/bweaver/ "When all else fails, RTFM." PLEASE NOTE THE FOLLOWING: 1. My Hotmail account is not monitored regularly. To send me an e-mail, please use the address shown above. 2. The SPSSX Discussion forum on Nabble is no longer linked to the SPSSX-L listserv administered by UGA (https://listserv.uga.edu/). |
«
Return to SPSSX Discussion
|
1 view|%1 views
| Free forum by Nabble | Edit this page |