Rounding algorithm

[Rose, Fred]

I conducted a 1-way ANOVA on a small data set and SPSS (v20 for Mac) gave me a SSgroups of 413.500*.  However, when I calculate it by hand, and never round any steps using a calculator with 16-digits, I get a SSgroups of 413.495xxx (the x’s are additional digits).  My assumption has always been that SPSS gives all responses to 3 decimal places.  Is it common for it to round to the nearest tenth, dropping the rest?  I know the difference between the two is quite small, but I have never come across this before and was wondering if anyone had an explanation.

*If I double-click on the summary table to look beyond the 3 decimal places displayed, the result is 413.50000).


--
Fredric E. Rose, Ph.D.
Associate Professor of Psychology
Palomar College
760-744-1150 x2344
frose@...
====================To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD

[Bruce Weaver]

Hi Fred.  Two questions:

1. Can you share the small data set & your syntax?
2. Which formulae did you use for your hand calculations?  

Cheers,
Bruce

Rose, Fred wrote

Rounding algorithm


I conducted a 1-way ANOVA on a small data set and SPSS (v20 for Mac) gave me a SSgroups of 413.500*.  However, when I calculate it by hand, and never round any steps using a calculator with 16-digits, I get a SSgroups of 413.495xxx (the x’s are additional digits).  My assumption has always been that SPSS gives all responses to 3 decimal places.  Is it common for it to round to the nearest tenth, dropping the rest?  I know the difference between the two is quite small, but I have never come across this before and was wondering if anyone had an explanation.

*If I double-click on the summary table to look beyond the 3 decimal places displayed, the result is 413.50000).


--
Fredric E. Rose, Ph.D.
Associate Professor of Psychology
Palomar College
760-744-1150 x2344
[hidden email] 




====================To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Rick Oliver-3]

The output displays a value rounded to three decimals but retains a value of greater. If it displays a value of 413.50000 on double-click, then that's the unrounded result obtained by the application.

In the example below, all the Sum of Squares values in the output display six decimal places when you double-click the cells, and they are not all 0.

set rng mc seed 123456789.
input program.
loop #i=1 to 1000.
compute groupvar=trunc(rv.normal(1, 5)).
compute scalevar=rv.norma(50,10).
end case.
end loop.
end file.
end input program.
oneway scalevar BY groupvar.

Rick Oliver
Senior Information Developer
IBM Business Analytics (SPSS)
E-mail: [hidden email]




From:        "Rose, Fred" <[hidden email]>
To:        [hidden email],
Date:        11/07/2012 03:31 PM
Subject:        Rounding algorithm
Sent by:        "SPSSX(r) Discussion" <[hidden email]>

---

I conducted a 1-way ANOVA on a small data set and SPSS (v20 for Mac) gave me a SSgroups of 413.500*.  However, when I calculate it by hand, and never round any steps using a calculator with 16-digits, I get a SSgroups of 413.495xxx (the x’s are additional digits).  My assumption has always been that SPSS gives all responses to 3 decimal places.  Is it common for it to round to the nearest tenth, dropping the rest?  I know the difference between the two is quite small, but I have never come across this before and was wondering if anyone had an explanation.

*If I double-click on the summary table to look beyond the 3 decimal places displayed, the result is 413.50000).


--
Fredric E. Rose, Ph.D.
Associate Professor of Psychology
Palomar College
760-744-1150 x2344
frose@...
====================To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD

[Rich Ulrich]

Saying that you computed with a calculator with 16 digits of
accuracy is not really good assurance that you achieved more
than 5 digits of accuracy.  It depends not only on the N but on
the size of the numbers.  - I don't know what the natural
permutation of round-offs give you.

If I recall correctly, it was 30-35 years ago that computational
accuracy was an issue for major stat-packs, and then everyone
made sure that they could do well on the "Longley data."  Now
it is mainly a problem for home-brew analyses (Excel?).

I expect that SPSS is more accurate than what you computed.

--
Rich Ulrich

---

Date: Wed, 7 Nov 2012 13:20:29 -0800
From: [hidden email]
Subject: Rounding algorithm
To: [hidden email]

I conducted a 1-way ANOVA on a small data set and SPSS (v20 for Mac) gave me a SSgroups of 413.500*.  However, when I calculate it by hand, and never round any steps using a calculator with 16-digits, I get a SSgroups of 413.495xxx (the x’s are additional digits).  My assumption has always been that SPSS gives all responses to 3 decimal places.  Is it common for it to round to the nearest tenth, dropping the rest?  I know the difference between the two is quite small, but I have never come across this before and was wondering if anyone had an explanation.

*If I double-click on the summary table to look beyond the 3 decimal places displayed, the result is 413.50000).

...