no. of variables

[Manoj Soni]

Dear Listers

 

 

I have a question regarding no. of variables that what is the maximum no. of variables in a data file of spss , i am using version spss 15.0.

 

thanks in advance

 

 

 

 

Manoj Soni
Assistant Data Analyst
INSTITUTE OF HEALTH MANAGEMENT RESEARCH
1, P.D. Marg, Near Sanganer Airport,
Jaipur. 302001. Rajasthan. INDIA.
Phone : [phone]
Fax : [phone]
[hidden email]
 

[Richard Ristow]

At 02:00 AM 4/3/2009, Manoj Soni wrote:

>what is the maximum no. of variables in a data file of spss


FAQ: How many variables and cases are allowed in SPSS?

Below is a discussion by Jon Peck of SPSS, Inc., which applies to all
SPSS versions that are likely to be in use - back to release 9, at least.

I add to what Jon wrote,

. For most operations, increasing the number of cases will increase
the running time about in proportion. Usually, SPSS can handle a
great many cases gracefully. Jon, below, notes some operations for
which many cases may slow SPSS badly, but often work-around can be
found even for these.

. Increasing the number of variables will generally increase the
running time about in proportion, even if you're not using them all,
because the running time is dominated by the time to read the file
from disk, i.e. the total file size

. After some point hard to estimate (though larger if the machine has
more RAM), increasing the number of variables will increase the
running time out of all proportion, because putting the whole
dictionary and data for one case in RAM may require paging.

. I emphasize Jon's point that "modern database practice would be to
break up your variables into cohesive subsets", i.e. to restructure
with more cases and fewer variables. A typical example is changing
from one record per entity with data for many years, to one record
per entity per year. I've posted a number of solutions in which data
is given such a 'long' representation with many cases, instead of a
'wide' representation with many variables.

At 10:25 AM 6/5/2003, Peck, Jon [of SPSS, Inc.] wrote:

There are several points to making regarding very wide files and huge datasets.

First, the theoretical SPSS limits are

Number of variables: (2**31) -1
Number of cases: (2**31) - 1

In calculating these limits, count one for each 8 bytes or part
thereof of a string variable.  An a10 variable counts as two
variables, for example.

Approaching the theoretical limit on the number of variables,
however, is a very bad idea in practice for several reasons.

1. These are the theoretical limits in that you absolutely cannot go
beyond them.  But there are other environmentally imposed limits that
you will surely hit first.  For example, Windows applications are
absolutely limited to 2GB of addressable memory, and 1GB is a more
practical limit.  Each dictionary entry requires about 100 bytes of
memory, because in addition to the variable name, other variable
properties also have to be stored.  (On non-Windows platforms, SPSS
Server could, of course, face different environmental
limits.)  Numerical variable values take 8 bytes as they are held as
double precision floating point values.

2. The overhead of reading and writing extremely wide cases when you
are doubtless not using more than a small fraction of them will limit
performance.  And you don't want to be paging the variable
dictionary.  If you have lots of RAM, you can probably reach between
32,000 and 100,000 variables before memory paging degrades
performance seriously.

3. Dialog boxes cannot display very large variable lists.  You can
use variable sets to restrict the lists to the variables you are
really using, but lists with thousands of variables will always be awkward.

4. Memory usage is not just about the dictionary.  The operating
system will almost always be paging code and data between memory and
disk.  (You can look at paging rates via the Windows Task
Manager).  The more you page, the slower things get, but the variable
dictionary is only one among many objects that the operating system
is juggling.  However, there is another effect.  On NT and later,
Windows automatically caches files (code or data) in memory so that
it can retrieve it quickly.  This cache occupies memory that is
otherwise surplus, so if any application needs it, portions of the
cache are discarded to make room.  You can see this effect quite
clearly if you start SPSS or any other large application; then shut
it down and start it again.  It will load much more quickly the
second time, because it is retrieving the code modules needed at
startup from memory rather than disk.  The Windows cache,
unfortunately, will not help data access very much unless most of the
dataset stays in memory, because the cache will generally hold the
most recently accessed data.  If you are reading cases sequentially,
the one you just finished with is the LAST one you will want again.

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Oliver, Richard]

Actually, I think it's theoretically possible to have more than (2**31)-1 cases, but for practical reasons you might not want to go there.

---

From: SPSSX(r) Discussion on behalf of Richard Ristow
Sent: Fri 4/3/2009 3:20 PM
To: [hidden email]
Subject: Re: no. of variables

At 02:00 AM 4/3/2009, Manoj Soni wrote:

>what is the maximum no. of variables in a data file of spss


FAQ: How many variables and cases are allowed in SPSS?

Below is a discussion by Jon Peck of SPSS, Inc., which applies to all
SPSS versions that are likely to be in use - back to release 9, at least.

I add to what Jon wrote,

. For most operations, increasing the number of cases will increase
the running time about in proportion. Usually, SPSS can handle a
great many cases gracefully. Jon, below, notes some operations for
which many cases may slow SPSS badly, but often work-around can be
found even for these.

. Increasing the number of variables will generally increase the
running time about in proportion, even if you're not using them all,
because the running time is dominated by the time to read the file
from disk, i.e. the total file size

. After some point hard to estimate (though larger if the machine has
more RAM), increasing the number of variables will increase the
running time out of all proportion, because putting the whole
dictionary and data for one case in RAM may require paging.

. I emphasize Jon's point that "modern database practice would be to
break up your variables into cohesive subsets", i.e. to restructure
with more cases and fewer variables. A typical example is changing
from one record per entity with data for many years, to one record
per entity per year. I've posted a number of solutions in which data
is given such a 'long' representation with many cases, instead of a
'wide' representation with many variables.

At 10:25 AM 6/5/2003, Peck, Jon [of SPSS, Inc.] wrote:

There are several points to making regarding very wide files and huge datasets.

First, the theoretical SPSS limits are

Number of variables: (2**31) -1
Number of cases: (2**31) - 1

In calculating these limits, count one for each 8 bytes or part
thereof of a string variable.  An a10 variable counts as two
variables, for example.

Approaching the theoretical limit on the number of variables,
however, is a very bad idea in practice for several reasons.

1. These are the theoretical limits in that you absolutely cannot go
beyond them.  But there are other environmentally imposed limits that
you will surely hit first.  For example, Windows applications are
absolutely limited to 2GB of addressable memory, and 1GB is a more
practical limit.  Each dictionary entry requires about 100 bytes of
memory, because in addition to the variable name, other variable
properties also have to be stored.  (On non-Windows platforms, SPSS
Server could, of course, face different environmental
limits.)  Numerical variable values take 8 bytes as they are held as
double precision floating point values.

2. The overhead of reading and writing extremely wide cases when you
are doubtless not using more than a small fraction of them will limit
performance.  And you don't want to be paging the variable
dictionary.  If you have lots of RAM, you can probably reach between
32,000 and 100,000 variables before memory paging degrades
performance seriously.

3. Dialog boxes cannot display very large variable lists.  You can
use variable sets to restrict the lists to the variables you are
really using, but lists with thousands of variables will always be awkward.

4. Memory usage is not just about the dictionary.  The operating
system will almost always be paging code and data between memory and
disk.  (You can look at paging rates via the Windows Task
Manager).  The more you page, the slower things get, but the variable
dictionary is only one among many objects that the operating system
is juggling.  However, there is another effect.  On NT and later,
Windows automatically caches files (code or data) in memory so that
it can retrieve it quickly.  This cache occupies memory that is
otherwise surplus, so if any application needs it, portions of the
cache are discarded to make room.  You can see this effect quite
clearly if you start SPSS or any other large application; then shut
it down and start it again.  It will load much more quickly the
second time, because it is retrieving the code modules needed at
startup from memory rather than disk.  The Windows cache,
unfortunately, will not help data access very much unless most of the
dataset stays in memory, because the cache will generally hold the
most recently accessed data.  If you are reading cases sequentially,
the one you just finished with is the LAST one you will want again.

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD