Effect size question

[Salbod]

Dear Friends,

                I compared three independent groups with a one-way ANOVA. The F was significant, and I followed it up with a Tukey.

                I plan to report eta squared for the ANOVA. What effect sizes do I report for the Tukey comparisons--Cohen’s d?

 

                Any comments or  suggestions will be greatly appreciated.

 

Thank you for taking the time to read this.

 

Warm regards, from your friend at Pace,

 

Steve   

[Bruce Weaver]

I'll let someone else tackle the issue of what measure of effect size to report.  I was never a big fan of standardized measures of effect size for ANOVA models when was working in experimental psychology, and analyzing response time data--but I'll save that rant for another occasion.*

Today's rant is about whether or not the omnibus F-test has to be significant before you proceed to pair-wise contrasts.  For Tukey's HSD procedure, the answer is NO, it does not.  AFAIK, the only common pair-wise comparisons procedure that DOES require a significant omnibus F-test as a first step is Fisher's LSD.  Fisher's LSD has an undeserved bad reputation for not controlling the family-wise alpha adequately.  What many people do not know, it seems, is that Fisher's LSD DOES control the family-wise alpha when there are 3 groups.  The problem occurs when there are 4 or more groups.  See the references below.  

The advantage of Fisher's LSD, of course, is that it is more powerful than the other standard methods for pair-wise contrasts (assuming you first get over the hurdle of the omnibus F-test).


References:

Howell, DC. Statistical Methods for Psychology (various editions & years, chapter on multiple comparison procedures).

Meier U. A note on the power of Fisher’s least significant difference procedure. Pharmaceut. Statist. 2006; 5: 253–263.

* Anyone who is interested can see a 2003 version of that rant in message #21 here:

http://groups.google.com/group/sci.stat.edu/browse_frm/thread/c475ee2d5e49df2d/4458743735da4d2f?q=Weaver+effect+size+RT#4458743735da4d2f

Salbod, Mr. Stephen wrote

Dear Friends,
                I compared three independent groups with a one-way ANOVA. The F was significant, and I followed it up with a Tukey.
                I plan to report eta squared for the ANOVA. What effect sizes do I report for the Tukey comparisons--Cohen's d?

                Any comments or  suggestions will be greatly appreciated.

Thank you for taking the time to read this.

Warm regards, from your friend at Pace,

Steve

[Evan Harrington, Ph.D.]

I'm not a fan of using eta squared as an effect size measure for omnibus
F tests. This issue has been discussed at length by Rosenthal and Rosnow
(see, for example, their text Essentials of Behavioral Science
Research).

The problem with eta squared is that it does not inform you where the
specific differences are (there's an effect in there somewhere, but
where?).

Nevertheless, eta squared is the effect size reported by SPSS so it is
easy to get.

With regards to focused test (post-hoc), yes you can compute Cohen's d
or effect size r from the results. Again, see Rosenthal and Rosnow for
discussions of this. The calculations are very easy and can be done on a
calculator.

Cheers!


Evan R. Harrington, Ph.D.
Associate Professor
Forensic Thesis Track Director
The Chicago School of Professional Psychology
Department of Forensic Psychology
325 North Wells Street
Chicago, IL 60654

Phone: [phone]
Fax: [phone]

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of
Bruce Weaver
Sent: Thursday, March 25, 2010 4:21 PM
To: [hidden email]
Subject: Re: Effect size question

I'll let someone else tackle the issue of what measure of effect size to
report.  I was never a big fan of standardized measures of effect size
for
ANOVA models when was working in experimental psychology, and analyzing
response time data--but I'll save that rant for another occasion.*

Today's rant is about whether or not the omnibus F-test has to be
significant before you proceed to pair-wise contrasts.  For Tukey's HSD
procedure, the answer is NO, it does not.  AFAIK, the only common
pair-wise
comparisons procedure that DOES require a significant omnibus F-test as
a
first step is Fisher's LSD.  Fisher's LSD has an undeserved bad
reputation
for not controlling the family-wise alpha adequately.  What many people
do
not know, it seems, is that Fisher's LSD DOES control the family-wise
alpha
when there are 3 groups.  The problem occurs when there are 4 or more
groups.  See the references below.

The advantage of Fisher's LSD, of course, is that it is more powerful
than
the other standard methods for pair-wise contrasts (assuming you first
get
over the hurdle of the omnibus F-test).


References:

Howell, DC. Statistical Methods for Psychology (various editions &
years,
chapter on multiple comparison procedures).

Meier U. A note on the power of Fisher's least significant difference
procedure. Pharmaceut. Statist. 2006; 5: 253-263.

* Anyone who is interested can see a 2003 version of that rant in
message
#21 here:

http://groups.google.com/group/sci.stat.edu/browse_frm/thread/c475ee2d5e
49df2d/4458743735da4d2f?q=Weaver+effect+size+RT#4458743735da4d2f



Salbod, Mr. Stephen wrote:
>
> Dear Friends,
>                 I compared three independent groups with a one-way
ANOVA.
> The F was significant, and I followed it up with a Tukey.
>                 I plan to report eta squared for the ANOVA. What
effect
> sizes do I report for the Tukey comparisons--Cohen's d?
>
>                 Any comments or  suggestions will be greatly
appreciated.
>
> Thank you for taking the time to read this.
>
> Warm regards, from your friend at Pace,
>
> Steve
>
>


-----
--
Bruce Weaver
[hidden email]
http://sites.google.com/a/lakeheadu.ca/bweaver/
"When all else fails, RTFM."

NOTE:  My Hotmail account is not monitored regularly.
To send me an e-mail, please use the address shown above.
--
View this message in context:
http://old.nabble.com/Effect-size-question-tp28034754p28035067.html
Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[statisticsdoc]

Steve
When pairs of means differ significantly according to Tukey HSD, then Cohen's d can be used as an index of the effect size. You may wish to report a confidence interval for the estimate of the effect size.
Best,
Steve Brand

www.StatisticsDoc.com

---

From: "Salbod, Mr. Stephen" <[hidden email]>

Date: Thu, 25 Mar 2010 16:54:53 -0400

To: <[hidden email]>

Subject: Effect size question

Dear Friends,

                I compared three independent groups with a one-way ANOVA. The F was significant, and I followed it up with a Tukey.

                I plan to report eta squared for the ANOVA. What effect sizes do I report for the Tukey comparisons--Cohen’s d?

 

                Any comments or  suggestions will be greatly appreciated.

 

Thank you for taking the time to read this.

 

Warm regards, from your friend at Pace,

 

Steve