p-value / type I error

[drfg2008]

Considering the concepts of Fisher and Neyman Pearson:

What is the difference of the p-value and the type I error ?

[Swank, Paul R]

P is the probability of committing a type 1 error, that is, the probability of getting a result this discrepant or more by chance alone (assuming the null hypothesis is true).

Dr. Paul R. Swank,
Professor and Director of Research
Children's Learning Institute
University of Texas Health Science Center-Houston


-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of drfg2008
Sent: Monday, April 11, 2011 12:03 PM
To: [hidden email]
Subject: p-value / type I error

Considering the concepts of Fisher and Neyman Pearson:

What is the difference of the p-value and the type I error ?

-----
FUB

--
View this message in context: http://spssx-discussion.1045642.n5.nabble.com/p-value-type-I-error-tp4296382p4296382.html
Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Bruce Weaver]

Do you by chance mean what is the difference between the p-value and alpha?  If so, bear in mind that the alpha value can (and should) be set before you look at the data.  The p-value, on the other hand, can only be computed after you have the data.  

Alpha is the maximum probability of Type I error you are willing to accept.  Where you set it is arbitrary; but as I'm sure you know, the overwhelming convention in many fields is to set alpha = .05.  

As Paul said, p = the probability of getting a result at least as extreme (i.e., at least as favorable to the alternative hypothesis) as the observed result IF the null hypothesis is true.  In other words, it is a conditional probability.  

Finally, the usual decision rule is:  If p LE alpha, reject H0; if p > alpha, fail to reject H0.

HTH.

drfg2008 wrote

Considering the concepts of Fisher and Neyman Pearson:

What is the difference of the p-value and the type I error ?

[Rich Ulrich]

----------------------------------------
> Date: Mon, 11 Apr 2011 10:03:00 -0700
> From: [hidden email]
> Subject: p-value / type I error
> To: [hidden email]
>
> Considering the concepts of Fisher and Neyman Pearson:
>
> What is the difference of the p-value and the type I error ?
>

In general discussion, "p-value" is usually a reference to
the nominal distribution of a particular test, whereas
speaking of "Type I error" usually implies a consideration
of multiple tests of hypothesis, and how that affects the
overall error rate.

Fisher and Neyman-Pearson refer to two arguments concerning
testing, which most people do not bother to learn about.
Instead, we follow our journal's guidelines.

I think that for a description of the philosophical
differences between styles of testing, you should use
Google to find references.  Googling on
< Pearson Fisher testing > , I quickly found a
discussion article (first page) at

http://www.jstor.org/pss/2291263
Journal of the American Statistical Asso...   >  Vol. 88, No. 424, Dec., 1993   >  The Fisher, Neyman-P...





















--
Rich Ulrich




=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[drfg2008]

Thanks Paul, thanks Bruce, thanks Rich


Quote:
Fisher and Neyman-Pearson refer to two arguments concerning  testing, which most people do not bother to learn about.  Instead, we follow our journal's guidelines.

Well, the guidelines are suggested to be changed (Eid, Gollwitzer, Schmitt [1]). That’s why I’m asking.


Quote:
P is the probability of committing a type 1 error (…).

Well, to “commit an error” implies a decision. Then p would be exactly the same as a type 1 error.


Quote:
As Paul said, p = the probability of getting a result at least as extreme (i.e., at least as favorable to the alternative hypothesis) as the observed result IF the null hypothesis is true.  

Well, Paul’s definition refers to “committing a type 1 error”. Your definition is like the Wiki definition [1], which DOES NOT include a decision. It’s a result. So, you actually provided a different definition.


But my problem, and my question is, what is the DIFFERENCE of  p-value and type 1 error.


Thank you

[1] http://www.amazon.de/gp/product/362127524X/ref=s9_simh_gw_p14_d2_i1?pf_rd_m=A3JWKAKR8XB7XF&pf_rd_s=center-4&pf_rd_r=0RT36HX7Z27Q750V6PPC&pf_rd_t=101&pf_rd_p=463375133&pf_rd_i=301128

[2] http://en.wikipedia.org/wiki/P-value

[Bruce Weaver]

A Type I error is an error--specifically, it is rejection of H0 when H0 is true.  A p-value is not an error (except when you compute it incorrectly, I suppose).*  So I do not understand your question.  That is why I asked if you really want to know the difference between p and alpha.


* You can also find critics of the whole hypothesis testing approach who say that it is complete lunacy to ever compute a p-value...so they might also say it's an error.  But that's not the same thing.  ;-)

drfg2008 wrote

Thanks Paul, thanks Bruce, thanks Rich


Quote:
Fisher and Neyman-Pearson refer to two arguments concerning  testing, which most people do not bother to learn about.  Instead, we follow our journal's guidelines.

Well, the guidelines are suggested to be changed (Eid, Gollwitzer, Schmitt [1]). That’s why I’m asking.


Quote:
P is the probability of committing a type 1 error (…).

Well, to “commit an error” implies a decision. Then p would be exactly the same as a type 1 error.


Quote:
As Paul said, p = the probability of getting a result at least as extreme (i.e., at least as favorable to the alternative hypothesis) as the observed result IF the null hypothesis is true.  

Well, Paul’s definition refers to “committing a type 1 error”. Your definition is like the Wiki definition [1], which DOES NOT include a decision. It’s a result. So, you actually provided a different definition.


But my problem, and my question is, what is the DIFFERENCE of  p-value and type 1 error.


Thank you

[1] http://www.amazon.de/gp/product/362127524X/ref=s9_simh_gw_p14_d2_i1?pf_rd_m=A3JWKAKR8XB7XF&pf_rd_s=center-4&pf_rd_r=0RT36HX7Z27Q750V6PPC&pf_rd_t=101&pf_rd_p=463375133&pf_rd_i=301128

[2] http://en.wikipedia.org/wiki/P-value

[drfg2008]

Thank you Bruce,

(what I mean by “the difference” is the problem how to combine p and type I error concepts.)

The p-value is the probability of obtaining a test statistic at least as extreme as the one that was actually observed, assuming that the null hypothesis is true. [1]

This definition does not imply a decision (!). It’s a ‘result’ (I don’t have a better word for it). However, in scientific research I have to make decisions. And that’s my problem now, since this implies type I and II errors.

If I for example compute a t-Test and get as a result (let’s say)  p=0,0123 I would reject the H0 since p < 0,05. The result is considered ‘statistically significant’. So far so good. However, what about the type I error. Is the type I error 0,0123 ?



Thanks

[1] http://en.wikipedia.org/wiki/P-value

[Mike]

----- Original Message -----
From: "drfg2008" <[hidden email]>
>
> If I for example compute a t-Test and get as a result (let’s say)  p=0,0123
> I would reject the H0 since p < 0,05. The result is considered
> ‘statistically significant’. So far so good. However, what about the type I
> error. Is the type I error 0,0123 ?

If the null hypotheis is true, any statistical test with p< .05 is a
Type I error because such a result would lead to the false rejection
of a true null hypothesis.

If the null hypothesis is false, it is a correct rejection of the null
hypothesis (if one has set alpha=0.05, that is, how extreme should
a result be for one to use it as evidence against the null hypothesis).
If the null hypothesis is false, there is no Type I error (though one
could screw up in other ways).

The problem is in real life research we don't really know if the
null hypothesis is true or not (though some may hold different
opinions about this).  Through replication of the original result,
the probability that a whole series of statistically significant
results are all Type I errors becomes extremely small.

-Mike Palij
New York University
[hidden email]

> Thanks
>
> [1] http://en.wikipedia.org/wiki/P-value
>
>
> -----
> FUB
>
> --
> View this message in context: http://spssx-discussion.1045642.n5.nabble.com/p-value-type-I-error-tp4296382p4297620.html
> Sent from the SPSSX Discussion mailing list archive at Nabble.com.
>
> =====================
> To manage your subscription to SPSSX-L, send a message to
> [hidden email] (not to SPSSX-L), with no body text except the
> command. To leave the list, send the command
> SIGNOFF SPSSX-L
> For a list of commands to manage subscriptions, send the command
> INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Bruce Weaver]

You asked if 0,0123 (or 0.0123) is the Type I error.  You're omitting some important words.  It is the conditional probability of Type I error--conditional on H0 being true, and conditional on you rejecting H0.  (If H0 is false, or you do not reject H0, you cannot make a Type I error.)

HTH.

drfg2008 wrote

Thank you Bruce,

(what I mean by “the difference” is the problem how to combine p and type I error concepts.)

The p-value is the probability of obtaining a test statistic at least as extreme as the one that was actually observed, assuming that the null hypothesis is true. [1]

This definition does not imply a decision (!). It’s a ‘result’ (I don’t have a better word for it). However, in scientific research I have to make decisions. And that’s my problem now, since this implies type I and II errors.

If I for example compute a t-Test and get as a result (let’s say)  p=0,0123 I would reject the H0 since p < 0,05. The result is considered ‘statistically significant’. So far so good. However, what about the type I error. Is the type I error 0,0123 ?



Thanks

[1] http://en.wikipedia.org/wiki/P-value

[Art Kendall]

In my experience, many people have trouble grasping "fail to to reject H0".  I find that it helps them to elaborate that H0 is the default i.e.,  a priori, PREsumed,   or existing  theoretical position, practice, policy, understanding.

An additional elaboration  emphasizes the subjunctive in the statement of "if Ho were true".  These tests can be thought of as looking at the comparison of obtained results with what would happen _if_ there _were_ zero difference or relation.  The HA is asserted only if there is sufficient evidence.

People who are used to common law like that in the US, are familiar with the concept of a presumed position. The assertion of guilt is proven or not proven at a given standard of evidence.  "Not guilty" means "guilt not proven".  A person is presumed not guilty unless it is found that there is sufficient evidence to assert guilt.

If a posteriori there is sufficient evidence, then the a priori H0 is replaced by HA, otherwise H0 stands.

When p does not reach alpha, there is insufficient evidence to adopt HA. p is the amount of evidence, alpha is the standard.

HTH

Art Kendall
Social Research Consultants


On 4/11/2011 2:03 PM, Bruce Weaver wrote:

Do you by chance mean what is the difference between the p-value and alpha?
If so, bear in mind that the alpha value can (and should) be set before you
look at the data.  The p-value, on the other hand, can only be computed
after you have the data.

Alpha is the maximum probability of Type I error you are willing to accept.
Where you set it is arbitrary; but as I'm sure you know, the overwhelming
convention in many fields is to set alpha = .05.

As Paul said, p = the probability of getting a result at least as extreme
(i.e., at least as favorable to the alternative hypothesis) as the observed
result IF the null hypothesis is true.  In other words, it is a conditional
probability.

Finally, the usual decision rule is:  If p LE alpha, reject H0; if p >
alpha, fail to reject H0.

HTH.



drfg2008 wrote:

Considering the concepts of Fisher and Neyman Pearson:

What is the difference of the p-value and the type I error ?

-----
--
Bruce Weaver
[hidden email]
http://sites.google.com/a/lakeheadu.ca/bweaver/

"When all else fails, RTFM."

NOTE: My Hotmail account is not monitored regularly.
To send me an e-mail, please use the address shown above.

--
View this message in context: http://spssx-discussion.1045642.n5.nabble.com/p-value-type-I-error-tp4296382p4296506.html
Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD

[Art Kendall]

Type I error is a possible� result of the decision after the test (trial).

Think of a two by two table to evaluate the decision (verdict) to switch to HA (found guilty) or stay with HO (found not guilty)
the row labels are test results (verdict)
A1 stayed with H0� rejected HA, found insufficient evidence (not guilty)
A2 switched to HA , accepted HA, found sufficient evidence� (guilty).
The columns labels are "truth"
B1 false (did not do it)
B2 true (did it).
a1b1 � staying� � was the right decision, true negative,� no error.
a1b1 � staying� � was the wrong decision, false negative, type II error.
a2b1� � switching was the wrong decision, false positive, type I error.
a2b2� � switching was the right decision, true positive,� no error.

The decision of a test (trial) is whether there is sufficient evidence to switch, i.e., go with HA.�

HTH

Art Kendall
Social Research Consultants

On 4/12/2011 3:37 AM, drfg2008 wrote:

Thank you Bruce,

(what I mean by “the difference” is the problem how to combine p and type I
error concepts.)

The p-value is the probability of obtaining a test statistic at least as
extreme as the one that was actually observed, assuming that the null
hypothesis is true. [1]

This definition does not imply a decision (!). It’s a ‘result’ (I don’t have
a better word for it). However, in scientific research I have to make
decisions. And that’s my problem now, since this implies type I and II
errors.

If I for example compute a t-Test and get as a result (let’s say)  p=0,0123
I would reject the H0 since p < 0,05. The result is considered
‘statistically significant’. So far so good. However, what about the type I
error. Is the type I error 0,0123 ?



Thanks

[1] http://en.wikipedia.org/wiki/P-value


-----
FUB

--
View this message in context: http://spssx-discussion.1045642.n5.nabble.com/p-value-type-I-error-tp4296382p4297620.html
Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD

[Art Kendall]

I should have added two other descriptors of H0.  It is the prevailing  or status quo p/p/p/u.

Art Kendall
Social Research Consultants

On 4/12/2011 10:31 AM, Art Kendall wrote:

In my experience, many people have trouble grasping "fail to to reject H0".  I find that it helps them to elaborate that H0 is the default i.e.,  a priori, PREsumed,   or existing  theoretical position, practice, policy, understanding.

===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD

[drfg2008]

The problem seems to be that testing hypothesis is based on a hybrid model, consisting of two different models: the Fisher concept and the Neyman-Pearson concept.

Fisher followed a complete different approach than Neyman and Pearson. Fisher invented the term: "significance" and the term "p-value". [1] Thus, according to Fisher, a significance test is defined as a procedure for establishing the probability of an outcome (!), as well as more extreme ones, on a null hypothesis of no effect or relationship. Whereas Neyman and Pearson invented the alpha and beta concept. Raymond Hubbard and M.J. Bayarri call it "The distinction between evidence (p’s) and error (α’s)".

My question is, how do these concepts fit together, if at all. Hubbard and Bayarri suggest: "This is achieved by reporting conditional (on p-values) error probabilities." (3.2 Reconciling Fisher’s and Neyman–Pearson’s Methods of Statistical Testing)

My question: Has anyone an idea what is meant by that (and is capable of explaning it in a way that everyone can understand)?


Frank

[1] http://ftp.isds.duke.edu/WorkingPapers/03-26.pdf

(I thank Generalist for the link)

[Mike]

Gerd Gigerenzer tackled these issues, I believe, first in the book
he co-authored "The Probabilistic Revolution" (in a chapter in volume 2).
He reviews these issues again in an entry to (2004) "The Sage Handbook
of Quantitative Methodology for the Social Sciences" (pp391-408).
You might be able to access the chapter here:
http://library.mpib-berlin.mpg.de/ft/gg/GG_Null_2004.pdf

-Mike Palij
New York University
[hidden email]


----- Original Message -----
From: "drfg2008" <[hidden email]>
To: <[hidden email]>
Sent: Wednesday, April 13, 2011 12:06 PM
Subject: Re: p-value / type I error

> The problem seems to be that testing hypothesis is based on a hybrid model,
> consisting of two different models: the Fisher concept and the
> Neyman-Pearson concept.
>
> Fisher followed a complete different approach than Neyman and Pearson.
> Fisher invented the term: "significance" and the term "p-value". [1] Thus,
> according to Fisher, a significance test is defined as a procedure for
> establishing the probability of an outcome (!), as well as more extreme
> ones, on a null hypothesis of no effect or relationship. Whereas Neyman and
> Pearson invented the alpha and beta concept. Raymond Hubbard and M.J.
> Bayarri call it "The distinction between evidence (p’s) and error (α’s)".
>
> My question is, how do these concepts fit together, if at all. Hubbard and
> Bayarri suggest: "This is achieved by reporting conditional (on p-values)
> error probabilities." (3.2 Reconciling Fisher’s and Neyman–Pearson’s Methods
> of Statistical Testing)
>
> My question: Has anyone an idea what is meant by that (and is capable of
> explaning it in a way that everyone can understand)?
>
>
> Frank
>
> [1] http://ftp.isds.duke.edu/WorkingPapers/03-26.pdf
>
> (I thank Generalist for the link)
>
> -----
> FUB
>
> --
> View this message in context: http://spssx-discussion.1045642.n5.nabble.com/p-value-type-I-error-tp4296382p4300978.html
> Sent from the SPSSX Discussion mailing list archive at Nabble.com.
>
> =====================
> To manage your subscription to SPSSX-L, send a message to
> [hidden email] (not to SPSSX-L), with no body text except the
> command. To leave the list, send the command
> SIGNOFF SPSSX-L
> For a list of commands to manage subscriptions, send the command
> INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[drfg2008]

Thanks Mike!

The only problem is that Gerd Gigerenzer (as always) wrote a very interesting article, however he also does not provide a "solution" to the problem how to match p(D|H) and p(H|D).

Frank