Paired T-test: significant results which seem very unlikely

[Midna]

CONTENTS DELETED

The author has deleted this message.

[Raimundas Vaitkevičius]

Hi,

what seems strange to you? That people behave differently in social
situations vs personal situations? I think not.

What is your sample size?

As to the difference in means, it should be evaluated together with
standard deviation of the difference.

Raimundas Vaitkevičius

Vytautas Magnus University, Kaunas, Lithuania


On Thu, Feb 11, 2016 at 10:42 AM, Midna <[hidden email]> wrote:

> Hi, I'm having trouble using a T-test.
>
> I asked participants about Big Five characteristics, once when in a social
> situation and again (same characteristics) in a personal situation.
> So for example I want to know if there is a significant difference between a
> person's answer on how extravert they are in a social situation vs their
> answer on how extravert they are in a personal situation.
>
> I have done the paired t-test and it shows .000 sign for all my data. Which
> to me seems very strange. As some of the data (Agreeableness) only has 0,3
> difference in mean if you compare the social against the personal situation
> (if you put it in a graph the means are nearly identical). I'm thinking it's
> maybe because of my data. It's a 7 point Likert scale  because as I
> understand data should be continuous for this test (and mine is not). But my
> professor told me to use a t-test, so not sure what I should do ... Normally
> we get a whole course about this but I was allowed to do a shorter program
> because of a pervious degree, but now I'm having a lot of trouble with SPSS
> since I've never used it before until a few months ago :).
>
> Someone suggested I use a Wilcoxon signed ranks test instead, but it gives
> me the same result.
>
> Any advice would be great ! (hope I used al the right terms since English
> isn't my native language :))
> (I'm using version 23 of SPSS)
>
>
>
> --
> View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Paired-T-test-significant-results-which-seem-very-unlikely-tp5731486.html
> Sent from the SPSSX Discussion mailing list archive at Nabble.com.
>
> =====================
> To manage your subscription to SPSSX-L, send a message to
> [hidden email] (not to SPSSX-L), with no body text except the
> command. To leave the list, send the command
> SIGNOFF SPSSX-L
> For a list of commands to manage subscriptions, send the command
> INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Midna]

My sample size is 220.

And no it's not strange that people behave different in these situations but it seems strange to me that for example Emotional Stability has a 0,3 difference in mean between the two groups (which is hardly any difference when you look at the data) and that this would be a significant difference ... I just don't understand how that is possible. Below the SPSS output for this trait (Emotional Stability) as example

Mean: ,29091
Std. Deviation: 1,03342
Std. Error Mean: ,06967
Lower 95% confidence interval: ,15359
Upper 95% confidence interval: ,42822
t: 4,175
df: 219
Sig. (Two-tailed): ,000
 
I don't really know how I should evaluate the standard in difference... I'm sorry, I'm really very bad at statistics

Thanks for the comment!

[Ryan]

Homework...

Sent from my iPhone

> On Feb 11, 2016, at 4:37 AM, Midna <[hidden email]> wrote:
>
> My sample size is 220.
>
> And no it's not strange that people behave different in these situations but
> it seems strange to me that for example Emotional Stability has a 0,3
> difference in mean between the two groups (which is hardly any difference
> when you look at the data) and that this would be a significant difference
> ... I just don't understand how that is possible. Below the SPSS output for
> this trait (Emotional Stability) as example
>
> Mean: ,29091    
> Std. Deviation: 1,03342
> Std. Error Mean:     ,06967    
> Lower 95% confidence interval: ,15359    
> Upper 95% confidence interval: ,42822    
> t: 4,175    
> df: 219    
> Sig. (Two-tailed): ,000
>
> I don't really know how I should evaluate the standard in difference... I'm
> sorry, I'm really very bad at statistics
>
> Thanks for the comment!
>
>
>
>
> --
> View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Paired-T-test-significant-results-which-seem-very-unlikely-tp5731486p5731488.html
> Sent from the SPSSX Discussion mailing list archive at Nabble.com.
>
> =====================
> To manage your subscription to SPSSX-L, send a message to
> [hidden email] (not to SPSSX-L), with no body text except the
> command. To leave the list, send the command
> SIGNOFF SPSSX-L
> For a list of commands to manage subscriptions, send the command
> INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Art Kendall]

if this is homework then it demonstrates an important concept in statistics.

When you have turned in your assignment, post back and the list can give you some feedback.

[Midna]

This is not homework, this is part of my thesis.
My school offers no help whatsoever in this (as they presume you've had the courses before, but like I said I did not because I'm in a special program so I'm left to fend for myself). I've tried to figure it out by myself and have several textbooks but can't find the answer.

But if this is not the place for such questions I'll understand and stop posting (I just didn't know what else I could do).

[Art Kendall]

Statistical significance looks at whether an apparent difference is readily attributable to random variation. In this instance it is not.

In psychology, if an apparent difference is statistically significant, we then use the "So what?" test.  Does the difference make any meaningful  theory/policy/practice difference?  The So What test goes with the Who Cares test.  

If a difference is readily attributable to random variation, then the answer to the So What test is necessarily "nothing". The answer to the Who Cares test is "nobody".

if a difference is not readily attributable the answers can still be "nothing" and/or "nobody" . That is a matter of judgment in the substantive discipline.



Does that help?

[Bruce Weaver]

Hello Midna.  You need to read up on the distinction between statistical significance and clinical significance (aka., practical importance), and on the relationship between sample size and statistical significance.  
Also consider the following, which uses the information you (helpfully) posted.

DATA LIST list / df(F5.0) SE (F8.5).
BEGIN DATA
219 .06967
END DATA.

* Get the two-tailed critical t-value with df = 219.
COMPUTE tcrit = ABS(IDF.T(.025,219)).
* Compute the smallest mean difference that
* would be statistically significant given
* the observed SE and df.
* t = (ObsMeanDiff/SE); so CritMinDiff = tcrit*SE.
COMPUTE CritMeanDiff = tcrit * SE.
* Transform CritMeanDiff to percentage of 7-point scale.
COMPUTE AsPercent = CritMeanDiff*100/7.
FORMATS tcrit CritMeanDiff (F8.5) AsPercent(F5.2).
LIST.

OUTPUT:

   df        SE    tcrit CritMeanDiff AsPercent
 
  219    .06967  1.97086     .13731      1.96


Given your sample size and estimate of the SE, you would only fail to achieve statistical significance if the mean difference was > -.13731 and < .13731.  As a percentage of your 7-point scale, that is less than a 2% difference.  In other words, to NOT be statistically significant, the mean difference would have to be tiny, much smaller than any practically important difference.

HTH.

Midna wrote

My sample size is 220.

And no it's not strange that people behave different in these situations but it seems strange to me that for example Emotional Stability has a 0,3 difference in mean between the two groups (which is hardly any difference when you look at the data) and that this would be a significant difference ... I just don't understand how that is possible. Below the SPSS output for this trait (Emotional Stability) as example

Mean: ,29091
Std. Deviation: 1,03342
Std. Error Mean: ,06967
Lower 95% confidence interval: ,15359
Upper 95% confidence interval: ,42822
t: 4,175
df: 219
Sig. (Two-tailed): ,000
 
I don't really know how I should evaluate the standard in difference... I'm sorry, I'm really very bad at statistics

Thanks for the comment!

[Mike]

This may or may not be homework but clearly (a) you
have not provided all the information, and (b) the responses
you've gotten haven't been helpful.
The traditional formula for the paired or CORRELATED t-test
is

t = (M1 - M2)/sqrt(VE1 + VE2 - 2*r+SE1+SE2

* = multiply
M1, M2 = Means of group 1 and group 2
VE = Variance Error (VE1 for group 1, VE2 for group 2)
SE = Standard Error (SE1 for group 1, SE2 for group 2)
2 = constant (the number 2)
r = Pearson r between values in group 1 and group 2

In the Independent Groups t-test, r = 0.00, so the denominator
becomes sqrt(VE1 + VE2).
But in the paired/correlated groups t-test, there is systematic
variance that is part of the dependent variable that is due to
differences between subjects and this systematic variance
should be subtracted from (VE1 + VE2) which is a pure
measure of error variance.  Note that you subtract the
systematic variance (-2*r*SE1*SE2) from the error
variance (VE1 + VE2).

The higher the Pearson r is, the greater the reduction in
error variance if the Pearson r is positive (if the Pearson r
is negative, the standard equation confounds error and
systematic variance).

So, to better understand why you got the t-value you did,
you could plug in the values for the Variance Error, Standard
Error, and the Pearson r into the equation provided above.

But what critical statistic do you fail to provide below?

Note: if you requested descriptive statistics in the t-test
procedure, that table which precedes the one you report
the info below has the means, standard deviations,
and standard errors.  The table you report below does
not provide this information and corresponds to the
dreaded "direct difference methods" which, while simple
to calculate by hand, bypasses the calculation of the
Pearson r.  But if you did this in SPSS, it is provided
in the output.

Work it out by hand, you have all of the information you
need (or put the numbers into Excel and let it do the
calculations for you).

-Mike Palij
New York University
[hidden email]

On Thursday, February 11, 2016 4:37 AM, Midna wrote:

> My sample size is 220.
>
> And no it's not strange that people behave different in these
> situations but
> it seems strange to me that for example Emotional Stability has a 0,3
> difference in mean between the two groups (which is hardly any
> difference
> when you look at the data) and that this would be a significant
> difference
> ... I just don't understand how that is possible. Below the SPSS
> output for
> this trait (Emotional Stability) as example
>
> Mean: ,29091
> Std. Deviation: 1,03342
> Std. Error Mean: ,06967
> Lower 95% confidence interval: ,15359
> Upper 95% confidence interval: ,42822
> t: 4,175
> df: 219
> Sig. (Two-tailed): ,000
>
> I don't really know how I should evaluate the standard in
> difference... I'm
> sorry, I'm really very bad at statistics
>
> Thanks for the comment!

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[David Marso]

I don't think this has anything to do with an unfamiliarity with SPSS.
It has to do with a fundamental misstep in your educational process.
You didn't study any statistics and now must use them and defend your study.
May I suggest you acquire an introductory volume and spend a weekend or two immersed in the basic concepts.   I wonder how many public policy decisions are made by people with absolutely no idea of what the hell their data means?  I don't think people should be able to even put their mitts on SPSS until a 2nd or 3rd class in statistics (after they have mastered the formulas and conceptual foundations).
How many people are graduated with a simple point/click mentality and no bloody idea of what they are doing?
--

Midna wrote

Hi, I'm having trouble using a T-test.

I asked participants about Big Five characteristics, once when in a social situation and again (same characteristics) in a personal situation.
So for example I want to know if there is a significant difference between a person's answer on how extravert they are in a social situation vs their answer on how extravert they are in a personal situation.

I have done the paired t-test and it shows .000 sign for all my data. Which to me seems very strange. As some of the data (Agreeableness) only has 0,3 difference in mean if you compare the social against the personal situation (if you put it in a graph the means are nearly identical). I'm thinking it's maybe because of my data. It's a 7 point Likert scale  because as I understand data should be continuous for this test (and mine is not). But my professor told me to use a t-test, so not sure what I should do ... Normally we get a whole course about this but I was allowed to do a shorter program because of a pervious degree, but now I'm having a lot of trouble with SPSS since I've never used it before until a few months ago :).

Someone suggested I use a Wilcoxon signed ranks test instead, but it gives me the same result.

Any advice would be great ! (hope I used al the right terms since English isn't my native language :))
(I'm using version 23 of SPSS)

[Midna]

Ok thank you for all the comments!

2016-02-11 15:43 GMT+01:00 David Marso [via SPSSX Discussion] <[hidden email]>:

I don't think this has anything to do with an unfamiliarity with SPSS.
It has to do with a fundamental misstep in your educational process.
You didn't study any statistics and now must use them and defend your study.
May I suggest you acquire an introductory volume and spend a weekend or two immersed in the basic concepts.   I wonder how many public policy decisions are made by people with absolutely no idea of what the hell their data means?  I don't think people should be able to even put their mitts on SPSS until a 2nd or 3rd class in statistics (after they have mastered the formulas and conceptual foundations).
How many people are graduated with a simple point/click mentality and no bloody idea of what they are doing?
--

Midna wrote

Hi, I'm having trouble using a T-test.

I asked participants about Big Five characteristics, once when in a social situation and again (same characteristics) in a personal situation.
So for example I want to know if there is a significant difference between a person's answer on how extravert they are in a social situation vs their answer on how extravert they are in a personal situation.

I have done the paired t-test and it shows .000 sign for all my data. Which to me seems very strange. As some of the data (Agreeableness) only has 0,3 difference in mean if you compare the social against the personal situation (if you put it in a graph the means are nearly identical). I'm thinking it's maybe because of my data. It's a 7 point Likert scale  because as I understand data should be continuous for this test (and mine is not). But my professor told me to use a t-test, so not sure what I should do ... Normally we get a whole course about this but I was allowed to do a shorter program because of a pervious degree, but now I'm having a lot of trouble with SPSS since I've never used it before until a few months ago :).

Someone suggested I use a Wilcoxon signed ranks test instead, but it gives me the same result.

Any advice would be great ! (hope I used al the right terms since English isn't my native language :))
(I'm using version 23 of SPSS)

Please reply to the list and not to my personal email.
Those desiring my consulting or training services please feel free to email me.
---
"Nolite dare sanctum canibus neque mittatis margaritas vestras ante porcos ne forte conculcent eas pedibus suis."
Cum es damnatorum possederunt porcos iens ut salire off sanguinum cliff in abyssum?"

---

If you reply to this email, your message will be added to the discussion below:

http://spssx-discussion.1045642.n5.nabble.com/Paired-T-test-significant-results-which-seem-very-unlikely-tp5731486p5731499.html

To unsubscribe from Paired T-test: significant results which seem very unlikely, click here.
NAML