output limitations in SPSS

[Lin Cassidy]

Hi all

I am trying to run some tests on 2 very large data sets of about 9
million cases each.  I have a mid-high level computer (1 GB RAM, Page
Files set to 2 GB min and max, dual core processors each 3 GHz), SPSS
11.5 over a network.
I'm very much stuck in the the point-and-click stage of SPSS use, have
not tried scripting at all.

Unfortunately, I keep running into limitation problems.  I've worked
with larger databases in the past (18 million cases), but I think part
of my problem with this dataset is that some of my continuous variables
may have more than 1 million unique values.  I keep getting the
following types of error messages:

Warnings
Number of Data Values requested exceeds maximum allowed (1000000)
This command is not executed.

Warnings
There is not enough memory for FREQUENCIES processing. Increase memory
or split the task into several tasks.

Warnings
Too many cases for the available storage. NONPAR CORR cannot continue.
This command is not executed.

Is there any way to overcome these limitations?  I can't find an *.ini
file where I might increase storage options, max no. of data values, etc.

Thank you very much,

Lin

[Beadle, ViAnn]

These limitations are hard to get around, but--

Why you would want to create a frequencies table with a million rows in it?

NONPAR CORR stores the entire array of variables requested in memory to get their rank. Is there some reason why you need nonparametric tests for your continuous variables? If you have very few discrete values for the variables you are including in the NONPAR CORR request, you can get nonparametric correlations from CROSSTABS which doesn't store all the cases in memory. Note that standard Pearson correlations take very little memory.


-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Lin Cassidy
Sent: Friday, May 11, 2007 11:22 AM
To: [hidden email]
Subject: output limitations in SPSS

Hi all

I am trying to run some tests on 2 very large data sets of about 9
million cases each.  I have a mid-high level computer (1 GB RAM, Page
Files set to 2 GB min and max, dual core processors each 3 GHz), SPSS
11.5 over a network.
I'm very much stuck in the the point-and-click stage of SPSS use, have
not tried scripting at all.

Unfortunately, I keep running into limitation problems.  I've worked
with larger databases in the past (18 million cases), but I think part
of my problem with this dataset is that some of my continuous variables
may have more than 1 million unique values.  I keep getting the
following types of error messages:

Warnings
Number of Data Values requested exceeds maximum allowed (1000000)
This command is not executed.

Warnings
There is not enough memory for FREQUENCIES processing. Increase memory
or split the task into several tasks.

Warnings
Too many cases for the available storage. NONPAR CORR cannot continue.
This command is not executed.

Is there any way to overcome these limitations?  I can't find an *.ini
file where I might increase storage options, max no. of data values, etc.

Thank you very much,

Lin

[Zdaniuk, Bozena]

Hello, I was asked to plot individual scores for a variable X but to
adjust the scores for a number of covariates. The only way I can think
of to adjust the scores is to run a regression in which the covariates
predict the X and save and then plot the residuals. Is there another
established way to adjust scores for values of other variables that I am
missing here?
Thanks a lot in advance.
Bozena

Bozena Zdaniuk, Ph.D.

University of Pittsburgh

UCSUR, 6th Fl.

121 University Place

Pittsburgh, PA 15260

Ph.: 412-624-5736

Fax: [phone]

email: [hidden email]

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of
Beadle, ViAnn
Sent: Friday, May 11, 2007 2:34 PM
To: [hidden email]
Subject: Re: output limitations in SPSS

These limitations are hard to get around, but--

Why you would want to create a frequencies table with a million rows in
it?

NONPAR CORR stores the entire array of variables requested in memory to
get their rank. Is there some reason why you need nonparametric tests
for your continuous variables? If you have very few discrete values for
the variables you are including in the NONPAR CORR request, you can get
nonparametric correlations from CROSSTABS which doesn't store all the
cases in memory. Note that standard Pearson correlations take very
little memory.


-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of
Lin Cassidy
Sent: Friday, May 11, 2007 11:22 AM
To: [hidden email]
Subject: output limitations in SPSS

Hi all

I am trying to run some tests on 2 very large data sets of about 9
million cases each.  I have a mid-high level computer (1 GB RAM, Page
Files set to 2 GB min and max, dual core processors each 3 GHz), SPSS
11.5 over a network.
I'm very much stuck in the the point-and-click stage of SPSS use, have
not tried scripting at all.

Unfortunately, I keep running into limitation problems.  I've worked
with larger databases in the past (18 million cases), but I think part
of my problem with this dataset is that some of my continuous variables
may have more than 1 million unique values.  I keep getting the
following types of error messages:

Warnings
Number of Data Values requested exceeds maximum allowed (1000000)
This command is not executed.

Warnings
There is not enough memory for FREQUENCIES processing. Increase memory
or split the task into several tasks.

Warnings
Too many cases for the available storage. NONPAR CORR cannot continue.
This command is not executed.

Is there any way to overcome these limitations?  I can't find an *.ini
file where I might increase storage options, max no. of data values,
etc.

Thank you very much,

Lin