SPSSX Discussion

Insufficient Memory

Classic

List

Threaded

5 messages Options

Marks, Jim

Insufficient Memory

Not directly a SPSS question:

I have a MIXED procedure with a large data file-- 60,000 records across
24 measurements for 3,000 subjects.

I have run it successfully in the past, but it now fails due to
"insufficient memory".

I've tried adjusting the page file, increasing free disk space (over 100
GB unused), and running with a subsample of the data 10% of cases--
ledto system hang).

In March 2007, Jon Peck replied to a memory issue relating to a Python
program that extracted 200+ tables, suggesting insufficient memory was
not the problem.

What can be changed to make the syntax run again?

--jim

====================To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

Jerabek Jindrich

Re:Insufficient Memory

Hello,

This might be SPSS issue. SPSS has a setting for memory workspace.
It is set with:
SET workspace= "a high number, in my old version 4096 default".
You could try to set it to a higher number.

If this is a problem of unsufficient sources reported by Windows then Win write something like "unsufficient memory, Windows now increase the swap file, the operation can take several minutes" - sorry this is not the exact wording Win uses. But usualy Win is realy able to solve the problem.

HTH
Jindra

> ------------ Původní zpráva ------------
> Od: Marks, Jim <[hidden email]>
> Předmět: Insufficient Memory
> Datum: 07.2.2008 19:52:15
> ----------------------------------------
> Not directly a SPSS question:
>
> I have a MIXED procedure with a large data file-- 60,000 records across
> 24 measurements for 3,000 subjects.
>
> I have run it successfully in the past, but it now fails due to
> "insufficient memory".
>
> I've tried adjusting the page file, increasing free disk space (over 100
> GB unused), and running with a subsample of the data 10% of cases--
> ledto system hang).
>
> In March 2007, Jon Peck replied to a memory issue relating to a Python
> program that extracted 200+ tables, suggesting insufficient memory was
> not the problem.
>
> What can be changed to make the syntax run again?
>
> --jim
>
> ???????
> To manage your subscription to SPSSX-L, send a message to
> [hidden email] (not to SPSSX-L), with no body text except the
> command. To leave the list, send the command
> SIGNOFF SPSSX-L
> For a list of commands to manage subscriptions, send the command
> INFO REFCARD
>
>
>

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

ViAnn Beadle

Re: Insufficient Memory

In reply to this post by Marks, Jim

If MIXED procedure works like other linear model procedures, the number of
categories for a nominal variables will exponentially increase the amount of
memory required. Usually this arises from the wrong level of measurement for
a variable which is treated as nominal but really is interval.

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of
Marks, Jim
Sent: Thursday, February 07, 2008 10:34 AM
To: [hidden email]
Subject: Insufficient Memory

Not directly a SPSS question:

I have a MIXED procedure with a large data file-- 60,000 records across
24 measurements for 3,000 subjects.

I have run it successfully in the past, but it now fails due to
"insufficient memory".

I've tried adjusting the page file, increasing free disk space (over 100
GB unused), and running with a subsample of the data 10% of cases--
ledto system hang).

In March 2007, Jon Peck replied to a memory issue relating to a Python
program that extracted 200+ tables, suggesting insufficient memory was
not the problem.

What can be changed to make the syntax run again?

--jim

=======
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

Marks, Jim

Re: Insufficient Memory

Thanks, ViAnn.

The model is pushing that envelope, but it completed successfully in the
past.

When I try a simplified version, I get the iterations, but SPSS hangs
after the iteration counter stops.

--jim

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of
ViAnn Beadle
Sent: Thursday, February 07, 2008 5:06 PM
To: [hidden email]
Subject: Re: Insufficient Memory

If MIXED procedure works like other linear model procedures, the number
of categories for a nominal variables will exponentially increase the
amount of memory required. Usually this arises from the wrong level of
measurement for a variable which is treated as nominal but really is
interval.

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of
Marks, Jim
Sent: Thursday, February 07, 2008 10:34 AM
To: [hidden email]
Subject: Insufficient Memory

Not directly a SPSS question:

I have a MIXED procedure with a large data file-- 60,000 records across
24 measurements for 3,000 subjects.

I have run it successfully in the past, but it now fails due to
"insufficient memory".

I've tried adjusting the page file, increasing free disk space (over 100
GB unused), and running with a subsample of the data 10% of cases--
ledto system hang).

In March 2007, Jon Peck replied to a memory issue relating to a Python
program that extracted 200+ tables, suggesting insufficient memory was
not the problem.

What can be changed to make the syntax run again?

--jim

=======
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command SIGNOFF SPSSX-L For a list
of commands to manage subscriptions, send the command INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command SIGNOFF SPSSX-L For a list
of commands to manage subscriptions, send the command INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

ViAnn Beadle

Re: Insufficient Memory

Does the simplified model hang with an error message or just hang? How do
you shut it down? What does the Task Manager show?

SPSS will attempt to get as much memory as it can. If this same model ran in
the past but doesn't now, it probably means that you've run out of available
memory within Windows. Other applications besides SPSS might be using it or
you've run out of virtual memory because there is not enough disk space for
the swap file. Virtual memory is much slower than real memory and that might
be causing an apparent hang.

Here are various things to try:

1) shut down other applications which are consuming a lot of memory. The
task manager will show what they are. If you're virus checking software is
up to date, some nasty program might be running that you're not aware of.
2) check your disk space and the size of the swap file. Freeing space up
might help.
3) check for excessive disk fragmentation and defrag if necessary. I'm not
sure if Windows gets chunks of disk space for virtual memory with some
minimum contiguous size or not.

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of
Marks, Jim
Sent: Friday, February 08, 2008 8:04 AM
To: [hidden email]
Subject: Re: Insufficient Memory

Thanks, ViAnn.

The model is pushing that envelope, but it completed successfully in the
past.

When I try a simplified version, I get the iterations, but SPSS hangs
after the iteration counter stops.

--jim

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of
ViAnn Beadle
Sent: Thursday, February 07, 2008 5:06 PM
To: [hidden email]
Subject: Re: Insufficient Memory

If MIXED procedure works like other linear model procedures, the number
of categories for a nominal variables will exponentially increase the
amount of memory required. Usually this arises from the wrong level of
measurement for a variable which is treated as nominal but really is
interval.

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of
Marks, Jim
Sent: Thursday, February 07, 2008 10:34 AM
To: [hidden email]
Subject: Insufficient Memory

Not directly a SPSS question:

I have a MIXED procedure with a large data file-- 60,000 records across
24 measurements for 3,000 subjects.

I have run it successfully in the past, but it now fails due to
"insufficient memory".

I've tried adjusting the page file, increasing free disk space (over 100
GB unused), and running with a subsample of the data 10% of cases--
ledto system hang).

In March 2007, Jon Peck replied to a memory issue relating to a Python
program that extracted 200+ tables, suggesting insufficient memory was
not the problem.

What can be changed to make the syntax run again?

--jim

=======
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command SIGNOFF SPSSX-L For a list
of commands to manage subscriptions, send the command INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command SIGNOFF SPSSX-L For a list
of commands to manage subscriptions, send the command INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD