SPSSX Discussion

how to get overall F-test for GLM ANOVA analysis?

Classic

List

Threaded

11 messages Options

Zdaniuk, Bozena-3

how to get overall F-test for GLM ANOVA analysis?

Hello everyone,

Is there a way to get the overall F-test of the model significance (equivalent to the one produced in Regression) for a Repeated Measures ANOVA performed using GLM in SPSS?

Cheers,

Bozena

===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD

Bruce Weaver

Re: how to get overall F-test for GLM ANOVA analysis?

Administrator

When you say repeated measures ANOVA, do you mean that there are repeated
measures factors only, or do you include so-called mixed designs in which
there is at least one between-Ss factor and at least one within-Ss factor?

Also, can you direct us to any textbooks or articles that show an overall
test of the type you have in mind?

Thanks for clarifying.

Zdaniuk, Bozena-3 wrote
> Hello everyone,
> Is there a way to get the overall F-test of the model significance
> (equivalent to the one produced in Regression) for a Repeated Measures
> ANOVA performed using GLM in SPSS?
> Cheers,
> Bozena
>
> =====================
> To manage your subscription to SPSSX-L, send a message to

> LISTSERV@.UGA

> (not to SPSSX-L), with no body text except the
> command. To leave the list, send the command
> SIGNOFF SPSSX-L
> For a list of commands to manage subscriptions, send the command
> INFO REFCARD

-----
--
Bruce Weaver
[hidden email]
http://sites.google.com/a/lakeheadu.ca/bweaver/

"When all else fails, RTFM."

NOTE: My Hotmail account is not monitored regularly.
To send me an e-mail, please use the address shown above.

--
Sent from: http://spssx-discussion.1045642.n5.nabble.com/

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

--
Bruce Weaver
bweaver@lakeheadu.ca
http://sites.google.com/a/lakeheadu.ca/bweaver/

"When all else fails, RTFM."

PLEASE NOTE THE FOLLOWING:
1. My Hotmail account is not monitored regularly. To send me an e-mail, please use the address shown above.
2. The SPSSX Discussion forum on Nabble is no longer linked to the SPSSX-L listserv administered by UGA (https://listserv.uga.edu/).

Zdaniuk, Bozena-3

Re: how to get overall F-test for GLM ANOVA analysis?

Hello Bruce and Others,
We conducted 5 mixed effect ANOVAS (2 within-sb factors and one btw-sb factor) and 4 RM ANOVAS (2 within-sb factors). The reviewer has asked that we control for multiple ANOVA testing by adjusting the p-values of the 'general F-test' for each ANOVA rather than adjusting the main effect or interaction effect p-values. I have searched the internet for hours and the only thing I found was someone on the stats.stackexchange talking about adjusting for multiple ANOVAS by first adjusting the 'overall F-tests" (https://stats.stackexchange.com/questions/373829/adjusting-p-values-for-multiple-anovas) but not explaining what they meant by that. In response to that message, someone further interpreted such a test as "the whole model variance compared to the null model variance" which sounds to me very much like the F-test for regression model.
This is all I have found so far...
Any thoughts on whether it's possible to get such 'general' F-test for ANOVAS? Or is the reviewer's request misguided?
Thanks so much!
Bozena

-----Original Message-----
From: SPSSX(r) Discussion <[hidden email]> On Behalf Of Bruce Weaver
Sent: March 2, 2021 7:19 PM
To: [hidden email]
Subject: Re: how to get overall F-test for GLM ANOVA analysis?

[CAUTION: Non-UBC Email]

When you say repeated measures ANOVA, do you mean that there are repeated measures factors only, or do you include so-called mixed designs in which there is at least one between-Ss factor and at least one within-Ss factor?

Also, can you direct us to any textbooks or articles that show an overall test of the type you have in mind?

Thanks for clarifying.

Zdaniuk, Bozena-3 wrote
> Hello everyone,
> Is there a way to get the overall F-test of the model significance
> (equivalent to the one produced in Regression) for a Repeated Measures
> ANOVA performed using GLM in SPSS?
> Cheers,
> Bozena
>
> =====================
> To manage your subscription to SPSSX-L, send a message to

> LISTSERV@.UGA

> (not to SPSSX-L), with no body text except the command. To leave the
> list, send the command SIGNOFF SPSSX-L For a list of commands to
> manage subscriptions, send the command INFO REFCARD

-----
--
Bruce Weaver
[hidden email]
http://sites.google.com/a/lakeheadu.ca/bweaver/

"When all else fails, RTFM."

NOTE: My Hotmail account is not monitored regularly.
To send me an e-mail, please use the address shown above.

--
Sent from: http://spssx-discussion.1045642.n5.nabble.com/

=====================
To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

Rich Ulrich

Re: how to get overall F-test for GLM ANOVA analysis?

The first thing to note is that if you are controlling for multiple

tests, you want to be specific about HYPOTHESES involved in

the particular F's. In particular, I think you want to focus on

some MAIN hypothesis, not on the several between- and within-

factors that are there and confuse matters. (Sex or age often

account for large amounts of variance while being of no "real"

interest at all. If that is the case, you don't want those tests.)

Personally, I tried to convince PIs to nail down no more than

about 3 "tests" that should tell them, Yes/No, "We found something."

Other partitions of variance are for "control" or for secondary

hypotheses, and the F's may be interesting but they are not crucial.

My preferred way of combining "similar" hypotheses is to construct

a composite score, and do testing on that. For dissimilar hypotheses,

fairly unrelated, doing Bonferroni correction works about as well as

anything ... if you can't convince people that each of the tests deserve

a full 5%-alpha for consideration.

Does it make sense - to you - to control for "multiple testing" in some

version of what the reviewer suggests? The comment, "adjusting the

p-values," makes me think he is suggesting some version of Bonferroni.

Rich Ulrich

From: SPSSX(r) Discussion <[hidden email]> on behalf of Zdaniuk, Bozena <[hidden email]>
Sent: Wednesday, March 3, 2021 6:25 PM
To: [hidden email] <[hidden email]>
Subject: Re: how to get overall F-test for GLM ANOVA analysis?

Mike

Re: how to get overall F-test for GLM ANOVA analysis?

In reply to this post by Zdaniuk, Bozena-3

A couple of points:

(1) I am really puzzled by the statement "General F-tests" because there seems to

to be different ways to interpret what this means or refers to. Assuming that you

are analyzing factorial designs (below, first, a 3-way mixed design, second a 2-way

within-subject design) It is unclear to me why adjusting the p-value of "general F-test"

would be preferable to adjusting the p-values of the individual tests -- you are still

going to do the individual tests and you want to keep the Type I error rate for each

comparison at some reasonable level.

Assuming a one-way design, one interpretation comes from Fisher's protected

"Least Significant Difference" (LSD) t-test which first requires a significant F value

before one did multiple t-tests (typically at alpha= 0.05 for each test, that is, an

ordinary t-test). If memory serves, obtaining a significant F value implies that at

least one orthogonal contrast will be statistically significant (though perhaps not

the one that one is most interested in). Still, the overall F in this situation is still

evaluated at alpha= 0.05 (If one has a one-way design with a large number of

dependent variables, one could do a MANOVA to determine if there is at least

one difference, following the logic of Fisher's LSD, one could do the individual

ANOVAs at .05 or use a Bonferroni corrected alpha).

(2) You write:

"someone further interpreted such a test as "the whole model variance compared

to the null model variance" which sounds to me very much like the F-test for regression mode."

I thought of this as well but in the context of doing Structural Equation Modeling

analysis comparing the empirical data against a model specifying no relationship.

I believe MPlus and R's lavaan would be able to do such an analysis; see the

Mplus website and/or the following article:

Breitsohl, H. (2019). Beyond ANOVA: An introduction to structural equation models

for experimental designs. Organizational Research Methods, 22(3), 649-677.

Go to:

https://journals.sagepub.com/doi/abs/10.1177/1094428118754988

However, it is not clear to me why this (or something similar with ordinary

regression) is a better control over the Type I error rate.

(3) I have pulled out my copy of Kirk's Experimental Design (4th ed) and could

not find anything on "general F-test" or anything conceptually similar to it.

Nor does Hochberg & Tamhane's "Multiple Comparison Procedures" (1st ed)

list anything like it in its index.

Perhaps you should ask that the reviewer provide a reference for the procedure?

-Mike Palij

New York University

[hidden email]

On Wed, Mar 3, 2021 at 6:25 PM Zdaniuk, Bozena <[hidden email]> wrote:

Hello Bruce and Others,
We conducted 5 mixed effect ANOVAS (2 within-sb factors and one btw-sb factor) and 4 RM ANOVAS (2 within-sb factors). The reviewer has asked that we control for multiple ANOVA testing by adjusting the p-values of the 'general F-test' for each ANOVA rather than adjusting the main effect or interaction effect p-values. I have searched the internet for hours and the only thing I found was someone on the stats.stackexchange talking about adjusting for multiple ANOVAS by first adjusting the 'overall F-tests" (https://urldefense.proofpoint.com/v2/url?u=https-3A__stats.stackexchange.com_questions_373829_adjusting-2Dp-2Dvalues-2Dfor-2Dmultiple-2Danovas&d=DwIFAg&c=slrrB7dE8n7gBJbeO0g-IQ&r=A8kXUln5f-BYIUaapBvbXA&m=QI9iMCKHh6oXgVyqyQNvkpYQPHoyUCpPRHBOk2KhqyM&s=RjSV9iEKg95Y9lD0PHh66CmTeoSqqaxGGZInc30do6M&e= ) but not explaining what they meant by that. In response to that message, someone further interpreted such a test as "the whole model variance compared to the null model variance" which sounds to me very much like the F-test for regression model.
This is all I have found so far...
Any thoughts on whether it's possible to get such 'general' F-test for ANOVAS? Or is the reviewer's request misguided?
Thanks so much!
Bozena

-----Original Message-----
From: SPSSX(r) Discussion <[hidden email]> On Behalf Of Bruce Weaver
Sent: March 2, 2021 7:19 PM
To: [hidden email]
Subject: Re: how to get overall F-test for GLM ANOVA analysis?

[CAUTION: Non-UBC Email]

When you say repeated measures ANOVA, do you mean that there are repeated measures factors only, or do you include so-called mixed designs in which there is at least one between-Ss factor and at least one within-Ss factor?

Also, can you direct us to any textbooks or articles that show an overall test of the type you have in mind?

Thanks for clarifying.

Zdaniuk, Bozena-3 wrote
> Hello everyone,
> Is there a way to get the overall F-test of the model significance
> (equivalent to the one produced in Regression) for a Repeated Measures
> ANOVA performed using GLM in SPSS?
> Cheers,
> Bozena
>
> =====================
> To manage your subscription to SPSSX-L, send a message to

> LISTSERV@.UGA

> (not to SPSSX-L), with no body text except the command. To leave the
> list, send the command SIGNOFF SPSSX-L For a list of commands to
> manage subscriptions, send the command INFO REFCARD

-----
--
Bruce Weaver
[hidden email]
https://urldefense.proofpoint.com/v2/url?u=http-3A__sites.google.com_a_lakeheadu.ca_bweaver_&d=DwIFAg&c=slrrB7dE8n7gBJbeO0g-IQ&r=A8kXUln5f-BYIUaapBvbXA&m=QI9iMCKHh6oXgVyqyQNvkpYQPHoyUCpPRHBOk2KhqyM&s=nSDnh-VRJ2zxmiHuc09kdGHmOTlGHb9wpP4iBXEt-4g&e=

"When all else fails, RTFM."

NOTE: My Hotmail account is not monitored regularly.
To send me an e-mail, please use the address shown above.

--
Sent from: https://urldefense.proofpoint.com/v2/url?u=http-3A__spssx-2Ddiscussion.1045642.n5.nabble.com_&d=DwIFAg&c=slrrB7dE8n7gBJbeO0g-IQ&r=A8kXUln5f-BYIUaapBvbXA&m=QI9iMCKHh6oXgVyqyQNvkpYQPHoyUCpPRHBOk2KhqyM&s=GP2GzgwGLt1a9LNC023gGaQX9aStPgDr6b6gOz6d87I&e=

=====================
To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

Bruce Weaver

Re: how to get overall F-test for GLM ANOVA analysis?

Administrator

In reply to this post by Zdaniuk, Bozena-3

Thank you for providing that additional info, Bozena. The thing I am still
stuck on is the notion of a general F-test for the entire model in the case
of a mixed-design ANOVA.

For a two-factor totally between-Ss ANOVA, the first partition of the SS is
into between-cells and within-cells. And MS_between-cells / MS_within-cells
gives an F-test for the "model". I assume the reviewer is asking for
something analogous to that.

But in the case of a repeated measures or mixed design, the first partition
of the SS is into between-subjects and within-subjects. In a completely
within-Ss design, the between-Ss variability is set aside and does not enter
into any of the F-tests. But not so in the mixed design. In both cases, I
do not see how to achieve something analogous to the overall test for the
"model" one gets for the completely between-Ss design. I wonder if the
reviewer actually knows how to do that!

Zdaniuk, Bozena-3 wrote

> Hello Bruce and Others,
> We conducted 5 mixed effect ANOVAS (2 within-sb factors and one btw-sb
> factor) and 4 RM ANOVAS (2 within-sb factors). The reviewer has asked that
> we control for multiple ANOVA testing by adjusting the p-values of the
> 'general F-test' for each ANOVA rather than adjusting the main effect or
> interaction effect p-values. I have searched the internet for hours and
> the only thing I found was someone on the stats.stackexchange talking
> about adjusting for multiple ANOVAS by first adjusting the 'overall
> F-tests"
> (https://stats.stackexchange.com/questions/373829/adjusting-p-values-for-multiple-anovas)
> but not explaining what they meant by that. In response to that message,
> someone further interpreted such a test as "the whole model variance
> compared to the null model variance" which sounds to me very much like the
> F-test for regression model.
> This is all I have found so far...
> Any thoughts on whether it's possible to get such 'general' F-test for
> ANOVAS? Or is the reviewer's request misguided?
> Thanks so much!
> Bozena

-----
--
Bruce Weaver
[hidden email]
http://sites.google.com/a/lakeheadu.ca/bweaver/

"When all else fails, RTFM."

NOTE: My Hotmail account is not monitored regularly.
To send me an e-mail, please use the address shown above.

--
Sent from: http://spssx-discussion.1045642.n5.nabble.com/

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

Bruce Weaver

Re: how to get overall F-test for GLM ANOVA analysis?

Administrator

In reply to this post by Mike

Hi Mike. The classic article (or at least I consider it a classic!) by
Huberty & Morris (1989) explicitly recommends against the practice of
conducting a MANOVA to buy protection against Type I error (a la Fisher's
LSD) and following up with multiple ANOVAs if the MANOVA is statistically
significant. Here are a couple of links for their article:

https://pdfs.semanticscholar.org/ee6c/77c99c8e4530d0cccaedf85ed525fb22a02d.pdf

https://www.researchgate.net/publication/232494414_Multivariate_Analysis_Versus_Multiple_Univariate_Analyses

And here is the first paragraph of the Discussion section, which summarizes
many of the key points:

--- start of excerpt ---
Even though it is a fairly popular analysis route to take in the behavioral
sciences, conducting a MANOVA as a preliminary step to multiple ANOVAS is
not only unnecessary but irrelevant as well. We consider to be a myth the
idea that one is controlling Type I error probability by following a
significant MANOVA test with multiple ANOVA tests, each conducted using
conventional significance levels. Furthermore, the research questions
addressed by a MANOVA and by multiple ANOVAS are different; the results of
one analysis may have little or no direct substantive bearing on the results
of the other. To require MANOVA as a prerequisite of multiple ANOVAS is
illogical, and the comfort of statistical protection is an illusion. The
view that it is inappropriate to follow a significant MANOVA overall test
with univariate tests is shared by others (e.g., Share, 1984).
--- end of excerpt ---

Cheers,
Bruce

Mike wrote

> A couple of points:
>
> --- snip ---
>
> Assuming a one-way design, one interpretation comes from Fisher's
> protected
> "Least Significant Difference" (LSD) t-test which first requires a
> significant F value
> before one did multiple t-tests (typically at alpha= 0.05 for each test,
> that is, an
> ordinary t-test). If memory serves, obtaining a significant F value
> implies that at
> least one orthogonal contrast will be statistically significant (though
> perhaps not
> the one that one is most interested in). Still, the overall F in this
> situation is still
> evaluated at alpha= 0.05 (If one has a one-way design with a large number
> of
> dependent variables, one could do a MANOVA to determine if there is at
> least
> one difference, following the logic of Fisher's LSD, one could do the
> individual
> ANOVAs at .05 or use a Bonferroni corrected alpha).

Mike

Re: how to get overall F-test for GLM ANOVA analysis?

Hi Bruce,

I vaguely remember this argument but since I don't do MANOVA, I am not

invested in using MANOVA as the first stage to determine whether an individual

ANOVA is significant or not. SEM may be more appropriate depending upon

the questions one is asking, the nature of the variables one is using (i.e.,

error free independent variables in traditional experimental designs or

empirical measures of a latent variable such as depression which might

be used as a dependent variable).

The Huberty & Morris argument against using MANOVA prior to individual

ANOVA is somewhat comparable to Rand Wilcox's often stated arguments

that one does need to do an ANOVA if one is going to use standard multiple

comparison procedures (e.g., Tukey's tests, etc.) since these automatically

Control the overall Type I Error Rate/alpha. Only the Fisher's LSD requires

a significant ANOVA before uncorrected t-tests are conducted. If one uses

Bonferroni corrected t-tests, a significant ANOVA is not required either.

In any event, getting back to the OP point of some sort of "general F value",

such a value, at least to me, seems to make sense only in the context of

a two-stage procedure like MANOVA followed by ANOVA or ANOVA followed

by ordinary t-tests.

-Mike Palij

New York University

On Thu, Mar 4, 2021 at 3:05 PM Bruce Weaver <[hidden email]> wrote:

Hi Mike. The classic article (or at least I consider it a classic!) by
Huberty & Morris (1989) explicitly recommends against the practice of
conducting a MANOVA to buy protection against Type I error (a la Fisher's
LSD) and following up with multiple ANOVAs if the MANOVA is statistically
significant. Here are a couple of links for their article:

https://urldefense.proofpoint.com/v2/url?u=https-3A__pdfs.semanticscholar.org_ee6c_77c99c8e4530d0cccaedf85ed525fb22a02d.pdf&d=DwICAg&c=slrrB7dE8n7gBJbeO0g-IQ&r=A8kXUln5f-BYIUaapBvbXA&m=DZEMqc17FKPjMx1EGJmsDt13feNvR7KlDesKBQ3ckq4&s=Ys0wc2RF_1nGZoolPqFfLLFFsGqnM6nd3lvkoQ-cWrA&e=

https://urldefense.proofpoint.com/v2/url?u=https-3A__www.researchgate.net_publication_232494414-5FMultivariate-5FAnalysis-5FVersus-5FMultiple-5FUnivariate-5FAnalyses&d=DwICAg&c=slrrB7dE8n7gBJbeO0g-IQ&r=A8kXUln5f-BYIUaapBvbXA&m=DZEMqc17FKPjMx1EGJmsDt13feNvR7KlDesKBQ3ckq4&s=ctxa4pMHqzKsNmCYPlUleWKPVXw1suwRxOEFp9N_v7Q&e=

And here is the first paragraph of the Discussion section, which summarizes
many of the key points:

--- start of excerpt ---
Even though it is a fairly popular analysis route to take in the behavioral
sciences, conducting a MANOVA as a preliminary step to multiple ANOVAS is
not only unnecessary but irrelevant as well. We consider to be a myth the
idea that one is controlling Type I error probability by following a
significant MANOVA test with multiple ANOVA tests, each conducted using
conventional significance levels. Furthermore, the research questions
addressed by a MANOVA and by multiple ANOVAS are different; the results of
one analysis may have little or no direct substantive bearing on the results
of the other. To require MANOVA as a prerequisite of multiple ANOVAS is
illogical, and the comfort of statistical protection is an illusion. The
view that it is inappropriate to follow a significant MANOVA overall test
with univariate tests is shared by others (e.g., Share, 1984).
--- end of excerpt ---

Cheers,
Bruce

Mike wrote
> A couple of points:
>
> --- snip ---
>
> Assuming a one-way design, one interpretation comes from Fisher's
> protected
> "Least Significant Difference" (LSD) t-test which first requires a
> significant F value
> before one did multiple t-tests (typically at alpha= 0.05 for each test,
> that is, an
> ordinary t-test). If memory serves, obtaining a significant F value
> implies that at
> least one orthogonal contrast will be statistically significant (though
> perhaps not
> the one that one is most interested in). Still, the overall F in this
> situation is still
> evaluated at alpha= 0.05 (If one has a one-way design with a large number
> of
> dependent variables, one could do a MANOVA to determine if there is at
> least
> one difference, following the logic of Fisher's LSD, one could do the
> individual
> ANOVAs at .05 or use a Bonferroni corrected alpha).

-----
--
Bruce Weaver
[hidden email]
https://urldefense.proofpoint.com/v2/url?u=http-3A__sites.google.com_a_lakeheadu.ca_bweaver_&d=DwICAg&c=slrrB7dE8n7gBJbeO0g-IQ&r=A8kXUln5f-BYIUaapBvbXA&m=DZEMqc17FKPjMx1EGJmsDt13feNvR7KlDesKBQ3ckq4&s=d4BKkn3fFgg1FLvxEOOqo4pDW1NPaWrMtjabWspmRbU&e=

"When all else fails, RTFM."

NOTE: My Hotmail account is not monitored regularly.
To send me an e-mail, please use the address shown above.

--
Sent from: https://urldefense.proofpoint.com/v2/url?u=http-3A__spssx-2Ddiscussion.1045642.n5.nabble.com_&d=DwICAg&c=slrrB7dE8n7gBJbeO0g-IQ&r=A8kXUln5f-BYIUaapBvbXA&m=DZEMqc17FKPjMx1EGJmsDt13feNvR7KlDesKBQ3ckq4&s=r6lpIxyhV3dMuetfkwHO_XJD_gX_el7g-zpmJKEJA9s&e=

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

Maguin, Eugene

Re: how to get overall F-test for GLM ANOVA analysis?

In watching the struggle to understand what was meant by the reviewer and to divine how that might accomplished, it seems to me that the reviewer might be wrong and a communication with the editor might be useful.

Gene Maguin

From: SPSSX(r) Discussion <[hidden email]> On Behalf Of Michael Palij
Sent: Thursday, March 04, 2021 4:09 PM
To: [hidden email]
Subject: Re: how to get overall F-test for GLM ANOVA analysis?