Multinomial regression: Bonferroni correction SPSS v.21

[sercerto]

Hi, I have a categorical predictor (two levels) and a categorical dependent
variable (eight levels). I want to analyze proportions. With the omnibus
chi-square approach, I can run a 2x8 crosstabulation and specify columns
differences with the z-test option using Bonferroni correction (significance
shown with letter subindexes). However, when using the alternative
multinomial approach, I get a bunch of Wald tests with associated p-values
and odds ratios (when comparing 7 of the DV categories to a reference). What
is not clear to me is whether this analysis uses Bonferroni correction for
the tests of significance associated with the number of comparisons. This
option is not explicit in Multinomial. Of course, I could divide .005/7 =
.00714 and compare this value with the p-values in the table; yet I need to
know whether the analysis has already done or not a correction. Thanx :)




--
Sent from: http://spssx-discussion.1045642.n5.nabble.com/

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Bruce Weaver]

Hello.  It would be easier for folks to help if you provided some sample
data.  Given that you have a 2x8 table, this can be done fairly easily with
DATA LIST, like this:

NEW FILE.
DATASET CLOSE ALL.
DATA LIST LIST / X Y N (3F5.0).
BEGIN DATA
1 1 23
1 2 33
1 3 54
1 4 21
1 5 22
1 6 44
1 7 72
1 8 34
2 1 43
2 2 23
2 3 64
2 4 21
2 5 12
2 6 64
2 7 32
2 8 54
END DATA.
WEIGHT by N.

I made up the values in the N variable, because I don't know your cell
counts.  

Having created some sample data, I decided to have a go at showing how
NOMREG and CROSSTABS can produce the same results.  Here's what I did (using
the data above).


* Multinomial logit model with Y=8 as reference category.
NOMREG Y (BASE=LAST ORDER=ASCENDING) BY X
  /INTERCEPT=INCLUDE
  /PRINT=PARAMETER SUMMARY LRT CPS STEP MFI IC.

* Now compute OR and LR Chi-sqare for series of 2x2 tables for Y=1 to 7 vs Y
= 8.
* Use a macro to loop through the Y-values 1-7.
DEFINE !xtab_loop ()
!DO !i = 1 !TO 7
  TEMPORARY.
  SELECT IF ANY(Y,!i,8).
  CROSSTABS X by Y /STATISTICS=CHISQ RISK.
!DOEND
!ENDDEFINE.

!xtab_loop.

The results showed that the ORs and 95% CIs from CROSSTABS matched (to 3
decimals) those from NOMREG.  What I expected to also see was that the
likelihood ratio Chi-square tests from CROSSTABS would match the Wald tests
(for X=1) from NOMREG.  But they're all off a bit, and I'm not sure why.
Too late on a summer Friday afternoon, I suppose.  Perhaps someone else will
spot what's going on.

Anyway, back to the main point:  Please provide some DATA LIST code as above
with your cell counts in the N column and the rest of the commands you ran
(presumably NOMREG and CTABLES).  

HTH.




sercerto wrote

> Hi, I have a categorical predictor (two levels) and a categorical
> dependent
> variable (eight levels). I want to analyze proportions. With the omnibus
> chi-square approach, I can run a 2x8 crosstabulation and specify columns
> differences with the z-test option using Bonferroni correction
> (significance
> shown with letter subindexes). However, when using the alternative
> multinomial approach, I get a bunch of Wald tests with associated p-values
> and odds ratios (when comparing 7 of the DV categories to a reference).
> What
> is not clear to me is whether this analysis uses Bonferroni correction for
> the tests of significance associated with the number of comparisons. This
> option is not explicit in Multinomial. Of course, I could divide .005/7 =
> .00714 and compare this value with the p-values in the table; yet I need
> to
> know whether the analysis has already done or not a correction. Thanx :)
>
>
>
>
> --
> Sent from: http://spssx-discussion.1045642.n5.nabble.com/
>
> =====================
> To manage your subscription to SPSSX-L, send a message to

> LISTSERV@.UGA

>  (not to SPSSX-L), with no body text except the
> command. To leave the list, send the command
> SIGNOFF SPSSX-L
> For a list of commands to manage subscriptions, send the command
> INFO REFCARD





-----
--
Bruce Weaver
[hidden email]
http://sites.google.com/a/lakeheadu.ca/bweaver/

"When all else fails, RTFM."

NOTE: My Hotmail account is not monitored regularly.
To send me an e-mail, please use the address shown above.

--
Sent from: http://spssx-discussion.1045642.n5.nabble.com/

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[sercerto]

Hi again. Ok, I will try to be more concrete with my concern while providing
my sample.

NEW FILE.
DATASET CLOSE ALL.
DATA LIST LIST / X Y N (3F5.0).
BEGIN DATA
1 1 193
1 2 79
1 3 38
1 4 33
1 5 22
1 6   6
1 7   8
1 8 23
2 1 41
2 2   8
2 3 14
2 4 37
2 5 36
2 6 40
2 7 90
2 8 136
END DATA.
WEIGHT by N


These are the two main analyses I've run so far:

1) Crosstabulation, including z-test comparison between cell proportions
(using Bonferroni correction)

CROSSTABS
  /TABLES= Independent variable BY dependent variable
  /FORMAT=AVALUE TABLES
  /STATISTICS=CHISQ
  /CELLS=COUNT ROW COLUMN ASRESID BPROP
  /COUNT ROUND CELL.  

According to this test, proportions within cells (1-2 vs 1-3) and (2-2
vs. 2-3) are not statistically different.

2)Now, let's say that I want to run an independent 2x2 ODDs ratio between
those cells. The odds ratio and associated CI are exactly the same as if I
run a Multinomial analysis including the 8 categories and using, for
example, category = 2 as the reference.

NOMREG dependent variable (BASE=1 ORDER=ASCENDING) BY independent variable
  /CRITERIA CIN(95) DELTA(0) MXITER(100) MXSTEP(5) CHKSEP(20) LCONVERGE(0) PCONVERGE(0.000001)
    SINGULAR(0.00000001)
  /MODEL
  /STEPWISE=PIN(.05) POUT(0.1) MINEFFECT(0) RULE(SINGLE) ENTRYMETHOD(LR) REMOVALMETHOD(LR)
  /INTERCEPT=INCLUDE
  /PRINT=PARAMETER SUMMARY LRT CPS STEP MFI.

The p-values with regard to the
chi-square test within these two analyses slightly differ (independent 2x2 p
= .006; with multinomial p = .008). Therefore, in contrast to the z-test,
there are significant differences. My conclusion is that Multinomial run 7
independent 2x2 Odds ratio tests and display them in a table. However, if I
consider a Bonferroni correction on 7 simultaneous tests .05/7, my p-value
of reference would be p=.0071. Consequently, the above mentioned significant
result when running multinomial would not be significant anymore, thus,
being in line with the z-test.
Therefore, the question that I am raising here is whether multinomial uses
correction for multiple simultaneous tests or not.





--
Sent from: http://spssx-discussion.1045642.n5.nabble.com/

=====================
To manage your subscription to SPSSX-L, send a message to
[e-mail] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Bruce Weaver]

Thanks for posting your data.  AFAIK, no regression type commands in SPSS (or
any other stats package) apply a Bonferroni correction for multiple tests in
the table of coefficients.  

To convince yourself that no correction is being applied by NOMREG, try the
following:

* Compute the Wald tests and ORs "by hand".
NEW FILE.
DATASET CLOSE ALL.
DATA LIST LIST / a b c d (4F5.0).
BEGIN DATA
79 193   8 41
38 193  14 41
33 193  37 41
22 193  36 41
 6 193  40 41
 8 193  90 41
23 193 136 41
END DATA.

COMPUTE OR_ = a*d/(b*c).
COMPUTE logOR = ln(OR_).
COMPUTE se = sqrt(1/a + 1/b + 1/c + 1/d).
COMPUTE Lower = exp(logOR-1.96*se).
COMPUTE Upper = exp(logOR+1.96*se).
COMPUTE Wald = (logOR/se)**2.
COMPUTE p = 1- CDF.CHISQ(Wald,1).
FORMATS Wald p OR_ Lower Upper (F9.3).
LIST a  to d Wald p OR_ Lower Upper.

Output from LIST:

    a     b     c     d      Wald         p       OR_     Lower     Upper
 
   79   193     8    41     3.282      .070     2.098      .941     4.676
   38   193    14    41     2.381      .123      .577      .287     1.160
   33   193    37    41    31.844      .000      .189      .106      .338
   22   193    36    41    40.544      .000      .130      .069      .243
    6   193    40    41    53.682      .000      .032      .013      .080
    8   193    90    41    95.102      .000      .019      .009      .042
   23   193   136    41   137.609      .000      .036      .021      .063

These results match those I see in the table of parameter estimates when I
run your NOMREG command.

HTH.



sercerto wrote
> --- snip ---
>
> Therefore, the question that I am raising here is whether multinomial uses
> correction for multiple simultaneous tests or not.





-----
--
Bruce Weaver
[hidden email]
http://sites.google.com/a/lakeheadu.ca/bweaver/

"When all else fails, RTFM."

NOTE: My Hotmail account is not monitored regularly.
To send me an e-mail, please use the address shown above.

--
Sent from: http://spssx-discussion.1045642.n5.nabble.com/

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Jon Peck]

If you want to correct for multiple tests from any set of sig levels you can use the STATS PADJUST extension command, downloadable via Extensions > Extension Hub.  It offer Bonferroni, the less conservative Benjamini-Hochberg FDR, and four others.

On Mon, Jul 8, 2019 at 3:27 PM Bruce Weaver <[hidden email]> wrote:

Thanks for posting your data.  AFAIK, no regression type commands in SPSS (or
any other stats package) apply a Bonferroni correction for multiple tests in
the table of coefficients. 

To convince yourself that no correction is being applied by NOMREG, try the
following:

* Compute the Wald tests and ORs "by hand".
NEW FILE.
DATASET CLOSE ALL.
DATA LIST LIST / a b c d (4F5.0).
BEGIN DATA
79 193   8 41
38 193  14 41
33 193  37 41
22 193  36 41
 6 193  40 41
 8 193  90 41
23 193 136 41
END DATA.

COMPUTE OR_ = a*d/(b*c).
COMPUTE logOR = ln(OR_).
COMPUTE se = sqrt(1/a + 1/b + 1/c + 1/d).
COMPUTE Lower = exp(logOR-1.96*se).
COMPUTE Upper = exp(logOR+1.96*se).
COMPUTE Wald = (logOR/se)**2.
COMPUTE p = 1- CDF.CHISQ(Wald,1).
FORMATS Wald p OR_ Lower Upper (F9.3).
LIST a  to d Wald p OR_ Lower Upper.

Output from LIST:

    a     b     c     d      Wald         p       OR_     Lower     Upper

   79   193     8    41     3.282      .070     2.098      .941     4.676
   38   193    14    41     2.381      .123      .577      .287     1.160
   33   193    37    41    31.844      .000      .189      .106      .338
   22   193    36    41    40.544      .000      .130      .069      .243
    6   193    40    41    53.682      .000      .032      .013      .080
    8   193    90    41    95.102      .000      .019      .009      .042
   23   193   136    41   137.609      .000      .036      .021      .063

These results match those I see in the table of parameter estimates when I
run your NOMREG command.

HTH.



sercerto wrote
> --- snip ---
>
> Therefore, the question that I am raising here is whether multinomial uses
> correction for multiple simultaneous tests or not.





-----
--
Bruce Weaver
[hidden email]
http://sites.google.com/a/lakeheadu.ca/bweaver/

"When all else fails, RTFM."

NOTE: My Hotmail account is not monitored regularly.
To send me an e-mail, please use the address shown above.

--
Sent from: http://spssx-discussion.1045642.n5.nabble.com/

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

--

===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD

[sercerto]

Thanks for your answers.
My question is much clearer now ;)

Just in case somebody is interested in post-hoc tests on Chi-square analyses and different types of multiple comparison alpha corrections, I recommend these two publications:

**Sharpe, D. (2015). Your Chi-square is statistically significant: Now what?. Practical Assessment, Research, & Evaluation, 20(8)

**Chen, S. Y., Feng, Z., & Yi, X. (2017). A general introduction to adjustment for multiple comparisons, 9(6), 1725-1729


--
Sent from: http://spssx-discussion.1045642.n5.nabble.com/

=====================
To manage your subscription to SPSSX-L, send a message to
[e-mail] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Bruce Weaver]

Thanks for the references.  I just took a quick peek at the one by Sharpe
(2015), and was pleased to see that it has a section on partitioning the
overall Chi-square into a series of smaller tables.  I was taught that
approach years ago in a grad stats class, but have not seen it used all that
frequently by other folks.  



sercerto wrote

> Thanks for your answers.
> My question is much clearer now ;)
>
> Just in case somebody is interested in post-hoc tests on Chi-square
> analyses and different types of multiple comparison alpha corrections, I
> recommend these two publications:
>
> **Sharpe, D. (2015). Your Chi-square is statistically significant: Now
> what?. Practical Assessment, Research, & Evaluation, 20(8)
>
> **Chen, S. Y., Feng, Z., & Yi, X. (2017). A general introduction to
> adjustment for multiple comparisons, 9(6), 1725-1729
>
>
> --
> Sent from: http://spssx-discussion.1045642.n5.nabble.com/
>
> =====================
> To manage your subscription to SPSSX-L, send a message to
> [hidden email] (not to SPSSX-L), with no body text except the
> command. To leave the list, send the command
> SIGNOFF SPSSX-L
> For a list of commands to manage subscriptions, send the command
> INFO REFCARD

-----
--
Bruce Weaver
[hidden email]
http://sites.google.com/a/lakeheadu.ca/bweaver/

"When all else fails, RTFM."

NOTE: My Hotmail account is not monitored regularly.
To send me an e-mail, please use the address shown above.

--
Sent from: http://spssx-discussion.1045642.n5.nabble.com/

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD