SPSSX Discussion

Tests of "significance"

Classic

List

Threaded

31 messages Options

Bob Schacht-3

Tests of "significance"

Please help me out here. The cautious, humble statistician says, "At the
.05 level, the Null Hypothesis is rejected." To the man on the street, this
is just pedantic mumbo-jumbo. So, say I'm using SPSS to do a Chi-square on
responses to a Likert scale question by case outcome. If the Chi-square
comes out with p< .05, I say, somewhat formally, "At the .05 level, the
Null Hypothesis that case outcome and responses to this question are
independent, is rejected."

How can I translate that into plain English that the proverbial man on the
street can understand, while remaining statistically correct?

I am looking for a generic phrase that can be used for all similar
statistical tests based on a null hypothesis of independence.

Thanks,
Bob

Robert M. Schacht, Ph.D. <[hidden email]>
Pacific Basin Rehabilitation Research & Training Center
1268 Young Street, Suite #204
Research Center, University of Hawaii
Honolulu, HI 96814

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

Hector Maletta

Re: Tests of "significance"

You could say that any difference observed might possibly be due to mere
chance. You do not have sufficient grounds to affirm that a relationship or
difference actually exists in the population, based on the relationship
observed in your sample, because the observed difference or relationship is
too weak for the size of your sample, or the size of your sample is too
small for such a weak relationship. You need a larger sample, or a stronger
relationship/difference, or both.

Notice also that this concerns your ability to infer from your sample to the
population, and has nothing to do with the substantive significance of your
hypothesis. Even a very small (and thus substantively "insignificant")
effect may be found to be statistically significant if the sample is
sufficiently large.

Hector

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Bob
Schacht
Sent: 07 April 2008 23:09
To: [hidden email]
Subject: Tests of "significance"

Please help me out here. The cautious, humble statistician says, "At the
.05 level, the Null Hypothesis is rejected." To the man on the street, this
is just pedantic mumbo-jumbo. So, say I'm using SPSS to do a Chi-square on
responses to a Likert scale question by case outcome. If the Chi-square
comes out with p< .05, I say, somewhat formally, "At the .05 level, the
Null Hypothesis that case outcome and responses to this question are
independent, is rejected."

How can I translate that into plain English that the proverbial man on the
street can understand, while remaining statistically correct?

I am looking for a generic phrase that can be used for all similar
statistical tests based on a null hypothesis of independence.

Thanks,
Bob

Robert M. Schacht, Ph.D. <[hidden email]>
Pacific Basin Rehabilitation Research & Training Center
1268 Young Street, Suite #204
Research Center, University of Hawaii
Honolulu, HI 96814

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

David Wasserman

Re: Tests of "significance"

In reply to this post by Bob Schacht-3

Pollsters and the like typically translate the .05 level of significance for
poll results into street language by saying something like "The results are
accurate within 4% [or whatever] 95 times out of 100 (or 19 times out of
20)." Perhaps you can adjust this type of terminology to suit your
situation.

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Bob
Schacht
Sent: April-07-08 8:09 PM
To: [hidden email]
Subject: Tests of "significance"

Please help me out here. The cautious, humble statistician says, "At the
.05 level, the Null Hypothesis is rejected." To the man on the street, this
is just pedantic mumbo-jumbo. So, say I'm using SPSS to do a Chi-square on
responses to a Likert scale question by case outcome. If the Chi-square
comes out with p< .05, I say, somewhat formally, "At the .05 level, the
Null Hypothesis that case outcome and responses to this question are
independent, is rejected."

How can I translate that into plain English that the proverbial man on the
street can understand, while remaining statistically correct?

I am looking for a generic phrase that can be used for all similar
statistical tests based on a null hypothesis of independence.

Thanks,
Bob

Robert M. Schacht, Ph.D. <[hidden email]>
Pacific Basin Rehabilitation Research & Training Center
1268 Young Street, Suite #204
Research Center, University of Hawaii
Honolulu, HI 96814

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

samir-9

Re: Tests of "significance"

In reply to this post by Hector Maletta

Dear Hector:

Just wondering abt one point - how did u assume that the sample Bob is
working on is too small. He hasnt mention it yet. He has talked abt
neither the universe nor the sample.

I certainly aggree with your statement on small sample against a large
universe. But what if the sample is considerable sufficient and
representative?

Cant we say 'the variables have certain association in as many as 95%
cases' if the chi-square value comes out to be significant at .05 level
from the contigency table on the two variables he is concerned with.
Please put some more insights on it if i am not correct to any extent.

Regards,
Sam

> You could say that any difference observed might possibly be due to mere
> chance. You do not have sufficient grounds to affirm that a relationship
> or
> difference actually exists in the population, based on the relationship
> observed in your sample, because the observed difference or relationship
> is
> too weak for the size of your sample, or the size of your sample is too
> small for such a weak relationship. You need a larger sample, or a
> stronger
> relationship/difference, or both.
>
> Notice also that this concerns your ability to infer from your sample to
> the
> population, and has nothing to do with the substantive significance of
> your
> hypothesis. Even a very small (and thus substantively "insignificant")
> effect may be found to be statistically significant if the sample is
> sufficiently large.
>
> Hector
>
> -----Original Message-----
> From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of
> Bob
> Schacht
> Sent: 07 April 2008 23:09
> To: [hidden email]
> Subject: Tests of "significance"
>
> Please help me out here. The cautious, humble statistician says, "At the
> .05 level, the Null Hypothesis is rejected." To the man on the street,
> this
> is just pedantic mumbo-jumbo. So, say I'm using SPSS to do a Chi-square on
> responses to a Likert scale question by case outcome. If the Chi-square
> comes out with p< .05, I say, somewhat formally, "At the .05 level, the
> Null Hypothesis that case outcome and responses to this question are
> independent, is rejected."
>
> How can I translate that into plain English that the proverbial man on the
> street can understand, while remaining statistically correct?
>
> I am looking for a generic phrase that can be used for all similar
> statistical tests based on a null hypothesis of independence.
>
> Thanks,
> Bob
>
>
> Robert M. Schacht, Ph.D. <[hidden email]>
> Pacific Basin Rehabilitation Research & Training Center
> 1268 Young Street, Suite #204
> Research Center, University of Hawaii
> Honolulu, HI 96814
>
> =====================
> To manage your subscription to SPSSX-L, send a message to
> [hidden email] (not to SPSSX-L), with no body text except the
> command. To leave the list, send the command
> SIGNOFF SPSSX-L
> For a list of commands to manage subscriptions, send the command
> INFO REFCARD
>
> =====================
> To manage your subscription to SPSSX-L, send a message to
> [hidden email] (not to SPSSX-L), with no body text except the
> command. To leave the list, send the command
> SIGNOFF SPSSX-L
> For a list of commands to manage subscriptions, send the command
> INFO REFCARD
>

Hector Maletta

Re: Tests of "significance"

Samir,
The probability resulting from a significance test is an increasing function
of two things: the size of the effect or difference observed in the sample,
and the size of the sample. For a small effect, you can always increase
sample size enough to get a significant result. For a given sample size,
there is (almost) always an effect big enough to be statistically
significant (i.e. so big that you are 95% confident it is different from
zero in the population).

Hector

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of
<Samir Kr Paul>
Sent: 08 April 2008 00:31
To: [hidden email]
Subject: Re: Tests of "significance"

Dear Hector:

Just wondering abt one point - how did u assume that the sample Bob is
working on is too small. He hasnt mention it yet. He has talked abt
neither the universe nor the sample.

I certainly aggree with your statement on small sample against a large
universe. But what if the sample is considerable sufficient and
representative?

Cant we say 'the variables have certain association in as many as 95%
cases' if the chi-square value comes out to be significant at .05 level
from the contigency table on the two variables he is concerned with.
Please put some more insights on it if i am not correct to any extent.

Regards,
Sam

Hector Maletta

Re: Tests of "significance"

In reply to this post by samir-9

Just to be precise: In my previous response to Samir I meant the probability
AGAINST the null hypothesis (more than 95% in the example). Ordinarily the
probability used to report significance is the complement, i.e. the
probability of obtaining the result by mere chance (less than 5% in the
example), as in p<0.05; with this convention, this probability (p), which is
the probability of accepting the null hypothesis when it is false, is a
DECREASING function of sample size and the size of the effect.
Hector

-----Original Message-----
From: Hector Maletta [mailto:[hidden email]]
Sent: 08 April 2008 01:24
To: '[hidden email]'; '[hidden email]'
Subject: RE: Tests of "significance"

Samir,
The probability resulting from a significance test is an increasing function
of two things: the size of the effect or difference observed in the sample,
and the size of the sample. For a small effect, you can always increase
sample size enough to get a significant result. For a given sample size,
there is (almost) always an effect big enough to be statistically
significant (i.e. so big that you are 95% confident it is different from
zero in the population).

Hector

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of
<Samir Kr Paul>
Sent: 08 April 2008 00:31
To: [hidden email]
Subject: Re: Tests of "significance"

Dear Hector:

Just wondering abt one point - how did u assume that the sample Bob is
working on is too small. He hasnt mention it yet. He has talked abt
neither the universe nor the sample.

I certainly aggree with your statement on small sample against a large
universe. But what if the sample is considerable sufficient and
representative?

Cant we say 'the variables have certain association in as many as 95%
cases' if the chi-square value comes out to be significant at .05 level
from the contigency table on the two variables he is concerned with.
Please put some more insights on it if i am not correct to any extent.

Regards,
Sam

Art Kendall

Re: Tests of "significance"

In reply to this post by Bob Schacht-3

something like:
There is only a 5% percent chance that the association we see is due to
chance. Now that we have rejected the idea that the association is due
to chance, we need to decide if the association is strong enough to be
meaningful.

Art Kendall
Social Research Consultants

Bob Schacht wrote:

> Please help me out here. The cautious, humble statistician says, "At the
> .05 level, the Null Hypothesis is rejected." To the man on the street,
> this
> is just pedantic mumbo-jumbo. So, say I'm using SPSS to do a
> Chi-square on
> responses to a Likert scale question by case outcome. If the Chi-square
> comes out with p< .05, I say, somewhat formally, "At the .05 level, the
> Null Hypothesis that case outcome and responses to this question are
> independent, is rejected."
>
> How can I translate that into plain English that the proverbial man on
> the
> street can understand, while remaining statistically correct?
>
> I am looking for a generic phrase that can be used for all similar
> statistical tests based on a null hypothesis of independence.
>
> Thanks,
> Bob
>
>
> Robert M. Schacht, Ph.D. <[hidden email]>
> Pacific Basin Rehabilitation Research & Training Center
> 1268 Young Street, Suite #204
> Research Center, University of Hawaii
> Honolulu, HI 96814
>
> =====================
> To manage your subscription to SPSSX-L, send a message to
> [hidden email] (not to SPSSX-L), with no body text except the
> command. To leave the list, send the command
> SIGNOFF SPSSX-L
> For a list of commands to manage subscriptions, send the command
> INFO REFCARD
>
>

Art Kendall
Social Research Consultants

Nana Nadine

confidence interval for means

I have conducted a session scoring analysis recently and the client asked me to give him the confidence level of the score I was providing (ie. 95% +/-2). The sapmle used was not statistically reliable. The session scoring was base on the rating of selected attributes.

What I did was to generate the reliability "alpha test" scores. The alpha scores were within acceptable range but did not tell me the level of confidance of the score. I was able to easily pull out of this one. I am just wondering if there a way (another test) I could have done/perform this better?

In another word is there a test that can allow to derive the confidence level and interval when we are reportting on means.

Thank you,
Nadine Nana

---------------------------------
You rock. That's why Blockbuster's offering you one month of Blockbuster Total Access, No Cost.

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

Peck, Jon

Re: Tests of "significance"

In reply to this post by Hector Maletta

". For a small effect, you can always increase
sample size enough to get a significant result"

Well, yes, but there is nothing wrong with this. The probabilities are what they are. The question is only what you do then. It has been amply discussed that statistical significance and substantive significance are not the same thing.

Regards,
Jon Peck

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Hector Maletta
Sent: Monday, April 07, 2008 10:24 PM
To: [hidden email]
Subject: Re: [SPSSX-L] Tests of "significance"

Samir,
The probability resulting from a significance test is an increasing function
of two things: the size of the effect or difference observed in the sample,
and the size of the sample. For a small effect, you can always increase
sample size enough to get a significant result. For a given sample size,
there is (almost) always an effect big enough to be statistically
significant (i.e. so big that you are 95% confident it is different from
zero in the population).

Hector

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of
<Samir Kr Paul>
Sent: 08 April 2008 00:31
To: [hidden email]
Subject: Re: Tests of "significance"

Dear Hector:

Just wondering abt one point - how did u assume that the sample Bob is
working on is too small. He hasnt mention it yet. He has talked abt
neither the universe nor the sample.

I certainly aggree with your statement on small sample against a large
universe. But what if the sample is considerable sufficient and
representative?

Cant we say 'the variables have certain association in as many as 95%
cases' if the chi-square value comes out to be significant at .05 level
from the contigency table on the two variables he is concerned with.
Please put some more insights on it if i am not correct to any extent.

Regards,
Sam

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

Hector Maletta

Re: Tests of "significance"

In reply to this post by Bob Schacht-3

Right, Jon. It is never repeated enough. Example: a difference of 0.001% in anything (say, in male-female pay or in black-white IQ) may be regarded as significant or not significant (from a STATISTICAL) point of view depending whether the sample is 1000 cases or 100 million cases. A completely different problem is whether such a small difference is SUBSTANTIVELY different. The sample applies to other fields: suppose that eating a certain foods increase your risk of breast cancer by 0.001%, and this is STATISTICALLY significant at the 99.9% level. Significant it may be, in the sense that is likely to be really above zero, but it is still very small and probably negligible for all practical purposes.

Hector

----- Mensaje original -----
De: "Peck, Jon" <[hidden email]>
Fecha: Martes, Abril 8, 2008 10:23 am
Asunto: Re: Tests of "significance"

> ". For a small effect, you can always increase
> sample size enough to get a significant result"
>
> Well, yes, but there is nothing wrong with this. The
> probabilities are what they are. The question is only what you do
> then. It has been amply discussed that statistical significance
> and substantive significance are not the same thing.
>
> Regards,
> Jon Peck
>
> -----Original Message-----
> From: SPSSX(r) Discussion [[hidden email]] On Behalf Of
> Hector Maletta
> Sent: Monday, April 07, 2008 10:24 PM
> To: [hidden email]
> Subject: Re: [SPSSX-L] Tests of "significance"
>
> Samir,
> The probability resulting from a significance test is an
> increasing function
> of two things: the size of the effect or difference observed in
> the sample,
> and the size of the sample. For a small effect, you can always
> increasesample size enough to get a significant result. For a
> given sample size,
> there is (almost) always an effect big enough to be statistically
> significant (i.e. so big that you are 95% confident it is
> different from
> zero in the population).
>
> Hector
>
> -----Original Message-----
> From: SPSSX(r) Discussion [[hidden email]] On Behalf Of
> <Samir Kr Paul>
> Sent: 08 April 2008 00:31
> To: [hidden email]
> Subject: Re: Tests of "significance"
>
> Dear Hector:
>
> Just wondering abt one point - how did u assume that the sample
> Bob is
> working on is too small. He hasnt mention it yet. He has talked abt
> neither the universe nor the sample.
>
> I certainly aggree with your statement on small sample against a large
> universe. But what if the sample is considerable sufficient and
> representative?
>
> Cant we say 'the variables have certain association in as many as 95%
> cases' if the chi-square value comes out to be significant at .05
> levelfrom the contigency table on the two variables he is
> concerned with.
> Please put some more insights on it if i am not correct to any extent.
>
> Regards,
> Sam
>
> > You could say that any difference observed might possibly be due
> to mere
> > chance. You do not have sufficient grounds to affirm that a
> relationship> or
> > difference actually exists in the population, based on the
> relationship> observed in your sample, because the observed
> difference or relationship
> > is
> > too weak for the size of your sample, or the size of your sample
> is too
> > small for such a weak relationship. You need a larger sample, or a
> > stronger
> > relationship/difference, or both.
> >
> > Notice also that this concerns your ability to infer from your
> sample to
> > the
> > population, and has nothing to do with the substantive
> significance of
> > your
> > hypothesis. Even a very small (and thus substantively
> "insignificant")> effect may be found to be statistically
> significant if the sample is
> > sufficiently large.
> >
> > Hector
> >
> > -----Original Message-----
> > From: SPSSX(r) Discussion [[hidden email]] On Behalf Of
> > Bob
> > Schacht
> > Sent: 07 April 2008 23:09
> > To: [hidden email]
> > Subject: Tests of "significance"
> >
> > Please help me out here. The cautious, humble statistician says,
> "At the
> > .05 level, the Null Hypothesis is rejected." To the man on the
> street,> this
> > is just pedantic mumbo-jumbo. So, say I'm using SPSS to do a Chi-
> square on
> > responses to a Likert scale question by case outcome. If the Chi-
> square> comes out with p< .05, I say, somewhat formally, "At the
> .05 level, the
> > Null Hypothesis that case outcome and responses to this question are
> > independent, is rejected."
> >
> > How can I translate that into plain English that the proverbial
> man on the
> > street can understand, while remaining statistically correct?
> >
> > I am looking for a generic phrase that can be used for all similar
> > statistical tests based on a null hypothesis of independence.
> >
> > Thanks,
> > Bob
> >
> >
> > Robert M. Schacht, Ph.D. <[hidden email]>
> > Pacific Basin Rehabilitation Research & Training Center
> > 1268 Young Street, Suite #204
> > Research Center, University of Hawaii
> > Honolulu, HI 96814
> >
> > =====================
> > To manage your subscription to SPSSX-L, send a message to
> > [hidden email] (not to SPSSX-L), with no body text
> except the
> > command. To leave the list, send the command
> > SIGNOFF SPSSX-L
> > For a list of commands to manage subscriptions, send the command
> > INFO REFCARD
> >
> > =====================
> > To manage your subscription to SPSSX-L, send a message to
> > [hidden email] (not to SPSSX-L), with no body text
> except the
> > command. To leave the list, send the command
> > SIGNOFF SPSSX-L
> > For a list of commands to manage subscriptions, send the command
> > INFO REFCARD
> >
>
> =====================
> To manage your subscription to SPSSX-L, send a message to
> [hidden email] (not to SPSSX-L), with no body text
> except the
> command. To leave the list, send the command
> SIGNOFF SPSSX-L
> For a list of commands to manage subscriptions, send the command
> INFO REFCARD
>
> =====================
> To manage your subscription to SPSSX-L, send a message to
> [hidden email] (not to SPSSX-L), with no body text
> except the
> command. To leave the list, send the command
> SIGNOFF SPSSX-L
> For a list of commands to manage subscriptions, send the command
> INFO REFCARD
>
> =====================
> To manage your subscription to SPSSX-L, send a message to
> [hidden email] (not to SPSSX-L), with no body text
> except the
> command. To leave the list, send the command
> SIGNOFF SPSSX-L
> For a list of commands to manage subscriptions, send the command
> INFO REFCARD
>

Granaas, Michael

Re: Tests of "significance"

In reply to this post by Bob Schacht-3

I apologize if this is a duplicate, but I didn't see my earlier response show up on the list and I received no confirmation.

It seems to me that if the test of independence is being rejected the "plain English" explanation is that responses to items and outcomes are correlated.

It is likely wise to compute a phi-coefficient so that size of the correlation can included in the description. E.g., responses to item 7 were very slightly correlated with outcome while responses to item 12 were strongly correlated with outcome.

Michael

****************************************************
Michael Granaas [hidden email]
Assoc. Prof. Phone: 605 677 5295
Dept. of Psychology FAX: 605 677 3195
University of South Dakota
414 E. Clark St.
Vermillion, SD 57069
*****************************************************

-----Original Message-----
From: SPSSX(r) Discussion on behalf of Bob Schacht
Sent: Mon 4/7/08 9:08 PM
To: [hidden email]
Subject: Tests of "significance"

Please help me out here. The cautious, humble statistician says, "At the
.05 level, the Null Hypothesis is rejected." To the man on the street, this
is just pedantic mumbo-jumbo. So, say I'm using SPSS to do a Chi-square on
responses to a Likert scale question by case outcome. If the Chi-square
comes out with p< .05, I say, somewhat formally, "At the .05 level, the
Null Hypothesis that case outcome and responses to this question are
independent, is rejected."

How can I translate that into plain English that the proverbial man on the
street can understand, while remaining statistically correct?

I am looking for a generic phrase that can be used for all similar
statistical tests based on a null hypothesis of independence.

Thanks,
Bob

Robert M. Schacht, Ph.D. <[hidden email]>
Pacific Basin Rehabilitation Research & Training Center
1268 Young Street, Suite #204
Research Center, University of Hawaii
Honolulu, HI 96814

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

====================To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

Swank, Paul R

Re: Tests of "significance"

In reply to this post by Hector Maletta

I like Hector's start to it. I try to deal with it in terms of making decisions. There is sufficient evidence to justify saying that the two variables are related. While you are at it, check out the number of ways of saying it wrong. I will say that 2 of the responses below are correct.

NHST Quiz
How do we interpret the results of a null hypothesis significance test?

A researcher tested the null hypothesis that two population means are equal (H: μ1 = μ2). A t test produced p = .010. Assuming that all assumptions of the test have been satisfied, which of the following statements are true and which are false? Why?

T F 1. There is a 1% likelihood that the result happened by chance.

T F 2. There is a 1% chance that the null hypothesis is true.

T F 3. There is a 1% chance of getting a result (as extreme or) even more extreme than the observed one when H is true.

T F 4. There is a 1% chance that the decision to reject Ho is wrong.

T F 5. There is a 99% chance that the alternative hypothesis is true, given the observed data.

T F 6. A small p value indicates a large effect.

T F 7. Rejection of H confirms the alternative hypothesis.

T F 8. Failure to reject H means that the two population means are probably equal.

T F 9. Rejecting H confirms the quality of the research design.

T F 10. If H is not rejected, the study is a failure.

T F 11 Assuming H is true and the study is repeated many times, 1% of these results will be (as inconsistent with H or) even more inconsistent with H than the observed result.

T F 12. If H is rejected in Study 1 but not rejected in Study 2, there must be a moderator variable that accounts for the difference between the two studies.

T F 13. There is a 99% chance that a replication study will produce significant results.

Adapted from Dale Berger’s post to the Teaching and Learning Statistics List, 14. February 2005, http://lists.psu.edu/archives/edstat-l.html. Dale adapted it from Kline, R. B. (2004). Beyond significance testing: Reforming data analysis methods in behavioral research. Washington, DC: American Psychological Association (pp. 63-69).

Karl Wuensch includes the quiz and results from two of his classes at http://core.ecu.edu/psyc/wuenschk/StatHelp/NHST-Quiz.doc

Paul R. Swank, Ph.D.
Professor and Director of Research
Children's Learning Institute
University of Texas Health Science Center - Houston

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Hector Maletta
Sent: Monday, April 07, 2008 11:34 PM
To: [hidden email]
Subject: Re: Tests of "significance"

Just to be precise: In my previous response to Samir I meant the probability
AGAINST the null hypothesis (more than 95% in the example). Ordinarily the
probability used to report significance is the complement, i.e. the
probability of obtaining the result by mere chance (less than 5% in the
example), as in p<0.05; with this convention, this probability (p), which is
the probability of accepting the null hypothesis when it is false, is a
DECREASING function of sample size and the size of the effect.
Hector

-----Original Message-----
From: Hector Maletta [mailto:[hidden email]]
Sent: 08 April 2008 01:24
To: '[hidden email]'; '[hidden email]'
Subject: RE: Tests of "significance"

Samir,
The probability resulting from a significance test is an increasing function
of two things: the size of the effect or difference observed in the sample,
and the size of the sample. For a small effect, you can always increase
sample size enough to get a significant result. For a given sample size,
there is (almost) always an effect big enough to be statistically
significant (i.e. so big that you are 95% confident it is different from
zero in the population).

Hector

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of
<Samir Kr Paul>
Sent: 08 April 2008 00:31
To: [hidden email]
Subject: Re: Tests of "significance"

Dear Hector:

Just wondering abt one point - how did u assume that the sample Bob is
working on is too small. He hasnt mention it yet. He has talked abt
neither the universe nor the sample.

I certainly aggree with your statement on small sample against a large
universe. But what if the sample is considerable sufficient and
representative?

Cant we say 'the variables have certain association in as many as 95%
cases' if the chi-square value comes out to be significant at .05 level
from the contigency table on the two variables he is concerned with.
Please put some more insights on it if i am not correct to any extent.

Regards,
Sam

Albert-Jan Roskam

Re: Tests of "significance"

In reply to this post by Peck, Jon

Or in short: 'significant' does not necessarily mean
'relevant'

Albert-Jan

--- "Peck, Jon" <[hidden email]> wrote:

> ". For a small effect, you can always increase
> sample size enough to get a significant result"
>
> Well, yes, but there is nothing wrong with this.
> The probabilities are what they are. The question
> is only what you do then. It has been amply
> discussed that statistical significance and
> substantive significance are not the same thing.
>
> Regards,
> Jon Peck
>
> -----Original Message-----
> From: SPSSX(r) Discussion
> [mailto:[hidden email]] On Behalf Of
> Hector Maletta
> Sent: Monday, April 07, 2008 10:24 PM
> To: [hidden email]
> Subject: Re: [SPSSX-L] Tests of "significance"
>
> Samir,
> The probability resulting from a significance test
> is an increasing function
> of two things: the size of the effect or difference
> observed in the sample,
> and the size of the sample. For a small effect, you
> can always increase
> sample size enough to get a significant result. For
> a given sample size,
> there is (almost) always an effect big enough to be
> statistically
> significant (i.e. so big that you are 95% confident
> it is different from
> zero in the population).
>
> Hector
>
> -----Original Message-----
> From: SPSSX(r) Discussion
> [mailto:[hidden email]] On Behalf Of
> <Samir Kr Paul>
> Sent: 08 April 2008 00:31
> To: [hidden email]
> Subject: Re: Tests of "significance"
>
> Dear Hector:
>
> Just wondering abt one point - how did u assume
> that the sample Bob is
> working on is too small. He hasnt mention it yet. He
> has talked abt
> neither the universe nor the sample.
>
> I certainly aggree with your statement on small
> sample against a large
> universe. But what if the sample is considerable
> sufficient and
> representative?
>
> Cant we say 'the variables have certain association
> in as many as 95%
> cases' if the chi-square value comes out to be
> significant at .05 level
> from the contigency table on the two variables he is
> concerned with.
> Please put some more insights on it if i am not
> correct to any extent.
>
> Regards,
> Sam
>
> > You could say that any difference observed might
> possibly be due to mere
> > chance. You do not have sufficient grounds to
> affirm that a relationship
> > or
> > difference actually exists in the population,
> based on the relationship
> > observed in your sample, because the observed
> difference or relationship
> > is
> > too weak for the size of your sample, or the size
> of your sample is too
> > small for such a weak relationship. You need a
> larger sample, or a
> > stronger
> > relationship/difference, or both.
> >
> > Notice also that this concerns your ability to
> infer from your sample to
> > the
> > population, and has nothing to do with the
> substantive significance of
> > your
> > hypothesis. Even a very small (and thus
> substantively "insignificant")
> > effect may be found to be statistically
> significant if the sample is
> > sufficiently large.
> >
> > Hector
> >
> > -----Original Message-----
> > From: SPSSX(r) Discussion
> [mailto:[hidden email]] On Behalf Of
> > Bob
> > Schacht
> > Sent: 07 April 2008 23:09
> > To: [hidden email]
> > Subject: Tests of "significance"
> >
> > Please help me out here. The cautious, humble
> statistician says, "At the
> > .05 level, the Null Hypothesis is rejected." To
> the man on the street,
> > this
> > is just pedantic mumbo-jumbo. So, say I'm using
> SPSS to do a Chi-square on
> > responses to a Likert scale question by case
> outcome. If the Chi-square
> > comes out with p< .05, I say, somewhat formally,
> "At the .05 level, the
> > Null Hypothesis that case outcome and responses to
> this question are
> > independent, is rejected."
> >
> > How can I translate that into plain English that
> the proverbial man on the
> > street can understand, while remaining
> statistically correct?
> >
> > I am looking for a generic phrase that can be used
> for all similar
> > statistical tests based on a null hypothesis of
> independence.
> >
> > Thanks,
> > Bob
> >
> >
> > Robert M. Schacht, Ph.D. <[hidden email]>
> > Pacific Basin Rehabilitation Research & Training
> Center
> > 1268 Young Street, Suite #204
> > Research Center, University of Hawaii
> > Honolulu, HI 96814
> >
> > =====================
> > To manage your subscription to SPSSX-L, send a
> message to
> > [hidden email] (not to SPSSX-L), with
> no body text except the
> > command. To leave the list, send the command
> > SIGNOFF SPSSX-L
> > For a list of commands to manage subscriptions,
> send the command
> > INFO REFCARD
> >
> > =====================
> > To manage your subscription to SPSSX-L, send a
> message to
> > [hidden email] (not to SPSSX-L), with
> no body text except the
> > command. To leave the list, send the command
> > SIGNOFF SPSSX-L
> > For a list of commands to manage subscriptions,
> send the command
> > INFO REFCARD
> >
>
> =====================
> To manage your subscription to SPSSX-L, send a
> message to
> [hidden email] (not to SPSSX-L), with no
> body text except the
> command. To leave the list, send the command
> SIGNOFF SPSSX-L
> For a list of commands to manage subscriptions, send
> the command
> INFO REFCARD
>
> =====================
> To manage your subscription to SPSSX-L, send a
> message to
> [hidden email] (not to SPSSX-L), with no
> body text except the
> command. To leave the list, send the command
> SIGNOFF SPSSX-L
> For a list of commands to manage subscriptions, send
> the command
> INFO REFCARD
>
> =====================
> To manage your subscription to SPSSX-L, send a
> message to
> [hidden email] (not to SPSSX-L), with no
> body text except the
> command. To leave the list, send the command
> SIGNOFF SPSSX-L
> For a list of commands to manage subscriptions, send
> the command
>

=== message truncated ===

____________________________________________________________________________________
You rock. That's why Blockbuster's offering you one month of Blockbuster Total Access, No Cost.
http://tc.deals.yahoo.com/tc/blockbuster/text5.com

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

Bob Schacht-3

Re: Tests of "significance"

In reply to this post by Granaas, Michael

At 06:00 AM 4/8/2008, Granaas, Michael wrote:

>I apologize if this is a duplicate, but I didn't see my earlier response
>show up on the list and I received no confirmation.
>
>It seems to me that if the test of independence is being rejected the
>"plain English" explanation is that responses to items and outcomes are
>correlated.
>
>It is likely wise to compute a phi-coefficient so that size of the
>correlation can included in the description. E.g., responses to item 7
>were very slightly correlated with outcome while responses to item 12 were
>strongly correlated with outcome.
>
>Michael

Peck, Jon

Re: Tests of "significance"

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Bob Schacht
Sent: Tuesday, April 08, 2008 7:11 PM
To: [hidden email]
Subject: Re: [SPSSX-L] Tests of "significance"
[snip]

>Statistically significant results.

"For this question, there is at least a 95% chance that participant
satisfaction and employment outcome are correlated."
I'd say
there is at least a 95% probability that the observed correlation between satisfaction and employment outcome is not due to chance.

Jon Peck

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

Hector Maletta

Re: Tests of "significance"

In reply to this post by Bob Schacht-3

It is nice that you thank everybody, Bob, but Michael Granaas opinion is not
right, for several reasons:
1. The original question was not about correlation but about chi square,
which concerns the difference between observed frequencies and those
expected in case of randomness or independence.
2. Even in the case of evaluating the significance of a correlation, the
question of significance is not about the existence of correlation, but
whether you (based on the correlation observed in a sample of a certain
size) can infer --with a given degree of confidence-- that some nonzero
correlation exists in the population. To see why this is different imagine
the following situations:
(a) Your sample shows a respectable correlation, say r=0.40, but your sample
is very small and your significance level is pretty high (99%), so you
cannot be 99% confident that the actual population correlation is not zero.
(b) The observed sample correlation is very small (say r=0.02) but your
sample is very large (several million cases), so you can say with 99%
confidence that a nonzero correlation exists in the population. If you lower
your desired significance level, say to 95%, you can be able to say the same
with a much smaller sample, perhaps tens of thousands.
In either case, you can commit two kinds of errors:
(i) False positives: You may conclude a nonzero correlation exists in the
population, when none actually exists.
(ii) False negatives: You may conclude that you are not able to discard the
possibility of a zero population correlation, when the population
correlation is actually zero.
Also in either case, rejecting the null hypothesis is not equivalent to
proving the truth of the research hypothesis (other research hypotheses may
be true instead of the one you are after). It is best to think of your
conclusions in a cautious negative phrasing: "I am not able to discard the
null hypothesis that no correlation exists in the population", or "I am not
able to discard the hypothesis that some nonzero correlation exists in the
population", promptly adding that both these statements have in turn a
certain probability of being in error.
Statistics is a course in humility.

Hector

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Bob
Schacht
Sent: 08 April 2008 22:11
To: [hidden email]
Subject: Re: Tests of "significance"

At 06:00 AM 4/8/2008, Granaas, Michael wrote:

Hector Maletta

Re: Tests of "significance"

In reply to this post by Granaas, Michael

Michael,
Your message reached the list all right. In fact, there were even responses.
I copy below my own and the thread.
Hector

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of
Granaas, Michael
Sent: 08 April 2008 13:01
To: [hidden email]
Subject: Re: Tests of "significance"

I apologize if this is a duplicate, but I didn't see my earlier response
show up on the list and I received no confirmation.
It seems to me that if the test of independence is being rejected the "plain
English" explanation is that responses to items and outcomes are correlated.
It is likely wise to compute a phi-coefficient so that size of the
correlation can included in the description. E.g., responses to item 7 were
very slightly correlated with outcome while responses to item 12 were
strongly correlated with outcome.
Michael
****************************************************
Michael Granaas [hidden email]
Assoc. Prof. Phone: 605 677 5295
Dept. of Psychology FAX: 605 677 3195
University of South Dakota
414 E. Clark St.
Vermillion, SD 57069
*****************************************************
Hector Maletta wrote:
It is nice that you thank everybody, Bob, but Michael Granaas opinion is not
right, for several reasons:
1. The original question was not about correlation but about chi square,
which concerns the difference between observed frequencies and those
expected in case of randomness or independence.
2. Even in the case of evaluating the significance of a correlation, the
question of significance is not about the existence of correlation, but
whether you (based on the correlation observed in a sample of a certain
size) can infer --with a given degree of confidence-- that some nonzero
correlation exists in the population. To see why this is different imagine
the following situations:
(a) Your sample shows a respectable correlation, say r=0.40, but your sample
is very small and your significance level is pretty high (99%), so you
cannot be 99% confident that the actual population correlation is not zero.
(b) The observed sample correlation is very small (say r=0.02) but your
sample is very large (several million cases), so you can say with 99%
confidence that a nonzero correlation exists in the population. If you lower
your desired significance level, say to 95%, you can be able to say the same
with a much smaller sample, perhaps tens of thousands.
In either case, you can commit two kinds of errors:
(i) False positives: You may conclude a nonzero correlation exists in the
population, when none actually exists.
(ii) False negatives: You may conclude that you are not able to discard the
possibility of a zero population correlation, when the population
correlation is actually zero.
Also in either case, rejecting the null hypothesis is not equivalent to
proving the truth of the research hypothesis (other research hypotheses may
be true instead of the one you are after). It is best to think of your
conclusions in a cautious negative phrasing: "I am not able to discard the
null hypothesis that no correlation exists in the population", or "I am not
able to discard the hypothesis that some nonzero correlation exists in the
population", promptly adding that both these statements have in turn a
certain probability of being in error.
Statistics is a course in humility.

Hector

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Bob
Schacht
Sent: 08 April 2008 22:11
To: [hidden email]
Subject: Re: Tests of "significance"

At 06:00 AM 4/8/2008, Granaas, Michael wrote:

>I apologize if this is a duplicate, but I didn't see my earlier
>response show up on the list and I received no confirmation.
>
>It seems to me that if the test of independence is being rejected the
>"plain English" explanation is that responses to items and outcomes are
>correlated.
>
>It is likely wise to compute a phi-coefficient so that size of the
>correlation can included in the description. E.g., responses to item 7
>were very slightly correlated with outcome while responses to item 12
>were strongly correlated with outcome.
>
>Michael

Thanks to MIchael, Hector, Art, Jon, & Paul for your interesting and helpful
replies! I like Michael's suggestion to use the word "correlated,"
which seems to be widely understood, if often confused with causation. I
also note how hard it is for most of us to abandon jargon and confine
ourselves to common English. Will the following pass muster?

>Statistically significant results.

"For this question, there is at least a 95% chance that participant
satisfaction and employment outcome are correlated."

>Almost statistically significant

"For this question, there is a 91% chance that participant satisfaction and
employment outcome are correlated. However, this falls short of the 95%
level usually required for statistical significance."

>Not statistically significant

"For this question, it does not appear that participant satisfaction and
employment outcome are correlated."

Thanks,
Bob

Robert M. Schacht, Ph.D. <[hidden email]> Pacific Basin Rehabilitation
Research & Training Center
1268 Young Street, Suite #204
Research Center, University of Hawaii
Honolulu, HI 96814

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command SIGNOFF SPSSX-L For a list of
commands to manage subscriptions, send the command INFO REFCARD

-----Original Message-----
From: SPSSX(r) Discussion on behalf of Bob Schacht
Sent: Mon 4/7/08 9:08 PM
To: [hidden email]
Subject: Tests of "significance"

Please help me out here. The cautious, humble statistician says, "At the
.05 level, the Null Hypothesis is rejected." To the man on the street, this
is just pedantic mumbo-jumbo. So, say I'm using SPSS to do a Chi-square on
responses to a Likert scale question by case outcome. If the Chi-square
comes out with p< .05, I say, somewhat formally, "At the .05 level, the
Null Hypothesis that case outcome and responses to this question are
independent, is rejected."

How can I translate that into plain English that the proverbial man on the
street can understand, while remaining statistically correct?

I am looking for a generic phrase that can be used for all similar
statistical tests based on a null hypothesis of independence.

Thanks,
Bob

Robert M. Schacht, Ph.D. <[hidden email]>
Pacific Basin Rehabilitation Research & Training Center
1268 Young Street, Suite #204
Research Center, University of Hawaii
Honolulu, HI 96814

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

Swank, Paul R

Re: Tests of "significance"

In reply to this post by Hector Maletta

Listen to Hector! In addition, with a Likert scale (assuming a 5 x 5,
the chi square is not a test of correlation since correlation is linear.
The chi square test is a test of correlation only for the 2 x 2.
Otherwise it is a test of the relation of two variables, that relation
possibly being nonlinear, much like an F test with more than two
categories..

Paul R. Swank, Ph.D.
Professor and Director of Research
Children's Learning Institute
University of Texas Health Science Center - Houston

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of
Hector Maletta
Sent: Tuesday, April 08, 2008 9:54 PM
To: [hidden email]
Subject: Re: Tests of "significance"

It is nice that you thank everybody, Bob, but Michael Granaas opinion is
not
right, for several reasons:
1. The original question was not about correlation but about chi square,
which concerns the difference between observed frequencies and those
expected in case of randomness or independence.
2. Even in the case of evaluating the significance of a correlation, the
question of significance is not about the existence of correlation, but
whether you (based on the correlation observed in a sample of a certain
size) can infer --with a given degree of confidence-- that some nonzero
correlation exists in the population. To see why this is different
imagine
the following situations:
(a) Your sample shows a respectable correlation, say r=0.40, but your
sample
is very small and your significance level is pretty high (99%), so you
cannot be 99% confident that the actual population correlation is not
zero.
(b) The observed sample correlation is very small (say r=0.02) but your
sample is very large (several million cases), so you can say with 99%
confidence that a nonzero correlation exists in the population. If you
lower
your desired significance level, say to 95%, you can be able to say the
same
with a much smaller sample, perhaps tens of thousands.
In either case, you can commit two kinds of errors:
(i) False positives: You may conclude a nonzero correlation exists in
the
population, when none actually exists.
(ii) False negatives: You may conclude that you are not able to discard
the
possibility of a zero population correlation, when the population
correlation is actually zero.
Also in either case, rejecting the null hypothesis is not equivalent to
proving the truth of the research hypothesis (other research hypotheses
may
be true instead of the one you are after). It is best to think of your
conclusions in a cautious negative phrasing: "I am not able to discard
the
null hypothesis that no correlation exists in the population", or "I am
not
able to discard the hypothesis that some nonzero correlation exists in
the
population", promptly adding that both these statements have in turn a
certain probability of being in error.
Statistics is a course in humility.

Hector

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of
Bob
Schacht
Sent: 08 April 2008 22:11
To: [hidden email]
Subject: Re: Tests of "significance"

At 06:00 AM 4/8/2008, Granaas, Michael wrote:
>I apologize if this is a duplicate, but I didn't see my earlier
response
>show up on the list and I received no confirmation.
>
>It seems to me that if the test of independence is being rejected the
>"plain English" explanation is that responses to items and outcomes are
>correlated.
>
>It is likely wise to compute a phi-coefficient so that size of the
>correlation can included in the description. E.g., responses to item 7
>were very slightly correlated with outcome while responses to item 12
were
>strongly correlated with outcome.
>
>Michael

Thanks to MIchael, Hector, Art, Jon, & Paul for your interesting and
helpful replies! I like Michael's suggestion to use the word
"correlated,"
which seems to be widely understood, if often confused with causation. I
also note how hard it is for most of us to abandon jargon and confine
ourselves to common English. Will the following pass muster?

>Statistically significant results.

"For this question, there is at least a 95% chance that participant
satisfaction and employment outcome are correlated."

>Almost statistically significant

"For this question, there is a 91% chance that participant satisfaction
and
employment outcome are correlated. However, this falls short of the 95%
level usually required for statistical significance."

>Not statistically significant

"For this question, it does not appear that participant satisfaction and
employment outcome are correlated."

Thanks,
Bob

Robert M. Schacht, Ph.D. <[hidden email]>
Pacific Basin Rehabilitation Research & Training Center
1268 Young Street, Suite #204
Research Center, University of Hawaii
Honolulu, HI 96814

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

cbautista

Variable format problem in file comparison

Hi list,

I have two data files, and I need to merge them; however, I need to know
if the files have the same variable format. For example, some variables
are numeric for file#1, but are string in file#2, so I need to locate this
difference. My files have the same number of variables (~800). I have SPSS
version 12.0

any idea? thanks in advance,

/Christian

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

Granaas, Michael

Re: Tests of "significance"

In reply to this post by Hector Maletta

??????

See my comments below

Michael
****************************************************
Michael Granaas [hidden email]
Assoc. Prof. Phone: 605 677 5295
Dept. of Psychology FAX: 605 677 3195
University of South Dakota
414 E. Clark St.
Vermillion, SD 57069
*****************************************************

>-----Original Message-----
>From: SPSSX(r) Discussion on behalf of Hector Maletta
>Sent: Tue 4/8/08 9:53 PM
>To: [hidden email]
>Subject: Re: Tests of "significance"

>It is nice that you thank everybody, Bob, but Michael Granaas opinion is not
>right, for several reasons:

I think you are the one who is wrong Hector.

>1. The original question was not about correlation but about chi square,
>which concerns the difference between observed frequencies and those
>expected in case of randomness or independence.

Huh? If the test of independence fails what does that mean? It means that the observed frequencies are correlated. For example let's say that we are looking at gender and political party affiliation in the U.S. A chi-square test of independence is rejected indicating that party affiliation is associated (correlated) with gender.

If you have a strong preference for "association" rather than "correlation" I have no objection. But either way we are talking about a statistical test that helps us determine whether or not an association exists.

>2. Even in the case of evaluating the significance of a correlation, the
>question of significance is not about the existence of correlation, but
>whether you (based on the correlation observed in a sample of a certain
>size) can infer --with a given degree of confidence-- that some nonzero
>correlation exists in the population.

I certainly don't remember saying anything different, except for not explicitly stating that the conclusions are about the population and not talking about a "given degree of confidence" which you expand on below. If you wish to misread my comments as limited to samples I suppose that I will have to be more explicit in the future.

>To see why this is different imagine
>the following situations:
>(a) Your sample shows a respectable correlation, say r=0.40, but your sample
>is very small and your significance level is pretty high (99%), so you
>cannot be 99% confident that the actual population correlation is not zero.
>(b) The observed sample correlation is very small (say r=0.02) but your
>sample is very large (several million cases), so you can say with 99%
>confidence that a nonzero correlation exists in the population. If you lower
>your desired significance level, say to 95%, you can be able to say the same
>with a much smaller sample, perhaps tens of thousands.

Bigger sample sizes increase power and allow you to detect smaller effects. Okay. I don't see how you felt that I said anything contradictory to that conclusion.

On the other hand I am not at all sure what you are talking about when you discuss a significance level of 99%. Does that mean that you have a p-value of ~.01?

If you have a p-value = .01 with a sample of n = 25 and again with a sample of n=25,000,000 the risk of a type I error is identical and the strength of your certainty as to the existence of a correlation is not changed at all.

On the other hand, a p-value of ~.99 is much more impressive with a very large sample than with a very small sample.

>In either case, you can commit two kinds of errors:
>(i) False positives: You may conclude a nonzero correlation exists in the
>population, when none actually exists.
>(ii) False negatives: You may conclude that you are not able to discard the
>possibility of a zero population correlation, when the population
>correlation is actually zero.

And this is relevant to the current question how? The gentleman asked how to explain a significant result in plain English.

>Also in either case, rejecting the null hypothesis is not equivalent to
>proving the truth of the research hypothesis (other research hypotheses may
>be true instead of the one you are after). It is best to think of your
>conclusions in a cautious negative phrasing: "I am not able to discard the
>null hypothesis that no correlation exists in the population", or "I am not
>able to discard the hypothesis that some nonzero correlation exists in the
>population", promptly adding that both these statements have in turn a
>certain probability of being in error.
>Statistics is a course in humility.

Huh?
If you have failed to reject these statements make some sense. But if you have rejected then you can certainly, tentatively, conclude that there is evidence of an association.

Michael

>Hector

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Bob
Schacht
Sent: 08 April 2008 22:11
To: [hidden email]
Subject: Re: Tests of "significance"

At 06:00 AM 4/8/2008, Granaas, Michael wrote: