Posted by
Maguin, Eugene on
Aug 14, 2011; 6:42pm
URL: http://spssx-discussion.165.s1.nabble.com/How-should-I-account-for-multiple-comparisons-when-looking-at-p-values-tp4698517p4698737.html
Hi Martha,
I don't claim to be an expert about what follows; others know more. If you
step back a minute and think about your results, I think there two things to
consider, in addition to the multiple comparisons per se. One is correlation
among results and the other is independent variables subset comparisons,
i.e., comparions involving differing numbers of groups. My only thought on
that is to ignore the subset element and treat them as full set
coparisons--unless the set of all results includes both subset comparisons
and full set comparions of the same DV. To the extant that the DVs are
correlated, the test statistics will be correlated.
The traditional way of controlling for muliple comparisons has been the
Bonferroni adjustment--if the nomial significance threshold is set at .05
and you do 10 tests, reset the threshold to .05/10=.005. Bonferroni is
criticized because power is (much) reduced. An alternative is the false
discovery rate (FDR). I think the procedures have undergone some development
since first being published. The key names are Benjamini, Yoav and Hochberg,
Yosef. I have also seen a reference to Holm, S. Look at the Wikipedia
article on false discovery rate. It's important to understand that there is
a difference between Bonferroni and FDR in terms of what is being controlled
for. The wiki article shows an adjustement to the FDR computation for
correlated tests that I don't recall seeing in the Benjamini et al.
articles. The FDR procedure for uncorrelated tests is to rank the test
results from most significant to least significant and then test the first
at .05 (the assumed threshold), the second at .05/2=.025, the third at
.05/3=.01667, etc until a result fails to pass the threshold.
I think there was a discussion on the list sometime earlier this year on FDR
and correlated tests and you might find something about it in the archives.
Gene Maguin
-----Original Message-----
From: SPSSX(r) Discussion [mailto:
[hidden email]] On Behalf Of
Martha Hewett
Sent: Sunday, August 14, 2011 12:52 PM
To:
[hidden email]
Subject: How should I account for multiple comparisons when looking at p
values?
I am comparing 7 groups on multiple dimensions (demographics, attitudes,
actions). Some comparisons are across all groups and some are among 2 or 3
or 4 of the groups. They include ANOVAs, chi-squares and t tests. I know
with so many comparisons some will be significant by chance. How should I
adjust for this? Also, should such an adjustment be made within types of
questions (e.g. within demographics, within attitudes, and within actions)
or across all items compared?
Thanks for any help you can provide.
=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD
=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD