SPSSX Discussion

"SORT CASES BY" memory problem.

Classic

List

Threaded

6 messages Options

Daniel E WILLIAMS

"SORT CASES BY" memory problem.

Hello all,

I'm unable to sort my data sets. I keep getting the message:

>Error # 5828. Command name: SORT CASES BY
>Insufficient memory in which to sort the file. Increase workspace
with SET
>WORKSPACE command .

This occurs both for huge datasets and for datasets as small as 57
cases and 29 variables. I am using Windows XP, have two processors, and
2 gigabytes of RAM. Oh, and I am using SPSS 14.0.1. I have tried
re-starting the computer, altering the WORKSPACE, and altering the
system priority for spsswin.exe.

Do you all have any ideas on what might be going wrong? I've never had
this problem before. Thanks.

Dan Williams
Forecasting, Research and Analysis Office
Finance and Policy Analysis
Department of Human Services
State of Oregon, USA
(503) 947-5354

Barnett, Adrian (HEALTH)

Re: "SORT CASES BY" memory problem.

Hi Daniel
I've become very suspicious of memory management in V14 and believe that
there is a flaw in it somewhere, analogous to a 'memory leak' - I find
the longer SPSS has been running and the more runs it has done, the more
likely it is to throw errors like this. I usually find that shutting
SPSS down and restarting it will stop these complaints from it about not
having sufficient memory.

A separate issue from this conjectural memory leak-style problem, is
that using the WORKSPACE command seems to trigger a permanent setting in
the registry, so that if you've set it to a particular figure, it will
still be at that setting from then on unless you alter it again. You can
check this yourself by running SHOW WORKSPACE. I suspect you've at some
point put in a WORKSPACE value that was too high and SPSS has retained
this and is now giving you these whacko messages because its basically
confused. My suggestion is that you try setting your workspace to a
lower figure and try it again on one of your small files and see if it
still complains. I suspect it will work OK on the small file if you set
it back to its default setting (6148, I think). I'd then try again on
your big file and see how you get on, and maybe slowly build it up again
from there if you keep getting 'insufficient workspace' settings.

I have a similar setup to you at home, with a dual-core system and 2 GB
of RAM and tried using large values of WORKSPACE to see if I could speed
up a sort on a large (700 MB, 1.4 million cases) file. Sorts were taking
a long time because SPSS was spending so much time flogging the hard
drive creating temporary files, with the processor sitting there doing
virtually nothing while it waited for these temporary files to be built.
If more could be done in memory, the sort would run much faster. At a
workspace setting of 600, SPSS complained it could not obtain that
amount of free memory. However the system performance monitor was
showing that SPSS was in fact using 600 MB and there were another 900 MB
of free memory (and of course the CPU utilisation was down around 10%).
So it looks to me like SPSS is trying to do its own memory management
(probably because in this version it is still trying to be compatible
with Windows 98, which does not do its own memory management in a
sensible way), and is getting it wrong.

At values of WORKSPACE between say, 200 and 600, I have observed that
some operations will complain that they could not obtain a memory
segment large enough, and then fail to complete. The effect was more
likely to be observed when the computer had been left running for days
on end with SPSS still open, and after running long programs over and
over. Rebooting seemed to cure the error messages, and lowering
WORKSAPCE settings also helped. These memory-induced failures can have
dire consequences if a file has to have been in a particular sort order
for a subsequent process (e.g. MATCH FILES or transformations involving
LAG) to work properly. The solution has been to wind the WORKSPACE
setting down. The way WORKSPACE is currently implemented, the manual's
advice of not changing it unless the system tells you it doesn't have
enough seems largely to be correct - although performance will not be
optimal unless you have a tiny file.

I notice that in the information about V15, it no longer states it will
run on Win 98, so I hope that means that memory management will now
largely be left to the operating system. I imagine that SPSS usually has
a pretty good idea of the amount of memory that would produce optimal
performance for each operation, so, theoretically it could just ask for
whatever amount it thinks is best and let the operating system give it
whatever it has spare, thus not involving the user. I really hope that
memory management is not going to be as much of pain in the neck under
V15 as it presently is under 14.

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of
Daniel E WILLIAMS
Sent: Tuesday, 29 August 2006 3:38
To: [hidden email]
Subject: "SORT CASES BY" memory problem.

Hello all,

I'm unable to sort my data sets. I keep getting the message:

>Error # 5828. Command name: SORT CASES BY Insufficient memory in which

>to sort the file. Increase workspace
with SET
>WORKSPACE command .

This occurs both for huge datasets and for datasets as small as 57 cases
and 29 variables. I am using Windows XP, have two processors, and
2 gigabytes of RAM. Oh, and I am using SPSS 14.0.1. I have tried
re-starting the computer, altering the WORKSPACE, and altering the
system priority for spsswin.exe.

Do you all have any ideas on what might be going wrong? I've never had
this problem before. Thanks.

Dan Williams
Forecasting, Research and Analysis Office Finance and Policy Analysis
Department of Human Services State of Oregon, USA
(503) 947-5354

Chris Conley

Re: "SORT CASES BY" memory problem.

In reply to this post by Daniel E WILLIAMS

Have you tried incorporating CACHE in your syntax? We've retained it from
some old syntax that addressed memory issues in older versions of SPSS. We
haven't encountered any problems so far. But then, perhaps we aren't
running files as large as you are using? Just a thought.

Chris Conley
Research Analyst
Accountability and Assessment
Durham District School Board

"Barnett, Adrian (HEALTH)" <[hidden email]>
Sent by: "SPSSX(r) Discussion" <[hidden email]>
28/08/2006 10:29 PM
Please respond to
"Barnett, Adrian (HEALTH)" <[hidden email]>

To
[hidden email]
cc

Subject
Re: "SORT CASES BY" memory problem.

Hi Daniel
I've become very suspicious of memory management in V14 and believe that
there is a flaw in it somewhere, analogous to a 'memory leak' - I find
the longer SPSS has been running and the more runs it has done, the more
likely it is to throw errors like this. I usually find that shutting
SPSS down and restarting it will stop these complaints from it about not
having sufficient memory.

A separate issue from this conjectural memory leak-style problem, is
that using the WORKSPACE command seems to trigger a permanent setting in
the registry, so that if you've set it to a particular figure, it will
still be at that setting from then on unless you alter it again. You can
check this yourself by running SHOW WORKSPACE. I suspect you've at some
point put in a WORKSPACE value that was too high and SPSS has retained
this and is now giving you these whacko messages because its basically
confused. My suggestion is that you try setting your workspace to a
lower figure and try it again on one of your small files and see if it
still complains. I suspect it will work OK on the small file if you set
it back to its default setting (6148, I think). I'd then try again on
your big file and see how you get on, and maybe slowly build it up again
from there if you keep getting 'insufficient workspace' settings.

I have a similar setup to you at home, with a dual-core system and 2 GB
of RAM and tried using large values of WORKSPACE to see if I could speed
up a sort on a large (700 MB, 1.4 million cases) file. Sorts were taking
a long time because SPSS was spending so much time flogging the hard
drive creating temporary files, with the processor sitting there doing
virtually nothing while it waited for these temporary files to be built.
If more could be done in memory, the sort would run much faster. At a
workspace setting of 600, SPSS complained it could not obtain that
amount of free memory. However the system performance monitor was
showing that SPSS was in fact using 600 MB and there were another 900 MB
of free memory (and of course the CPU utilisation was down around 10%).
So it looks to me like SPSS is trying to do its own memory management
(probably because in this version it is still trying to be compatible
with Windows 98, which does not do its own memory management in a
sensible way), and is getting it wrong.

At values of WORKSPACE between say, 200 and 600, I have observed that
some operations will complain that they could not obtain a memory
segment large enough, and then fail to complete. The effect was more
likely to be observed when the computer had been left running for days
on end with SPSS still open, and after running long programs over and
over. Rebooting seemed to cure the error messages, and lowering
WORKSAPCE settings also helped. These memory-induced failures can have
dire consequences if a file has to have been in a particular sort order
for a subsequent process (e.g. MATCH FILES or transformations involving
LAG) to work properly. The solution has been to wind the WORKSPACE
setting down. The way WORKSPACE is currently implemented, the manual's
advice of not changing it unless the system tells you it doesn't have
enough seems largely to be correct - although performance will not be
optimal unless you have a tiny file.

I notice that in the information about V15, it no longer states it will
run on Win 98, so I hope that means that memory management will now
largely be left to the operating system. I imagine that SPSS usually has
a pretty good idea of the amount of memory that would produce optimal
performance for each operation, so, theoretically it could just ask for
whatever amount it thinks is best and let the operating system give it
whatever it has spare, thus not involving the user. I really hope that
memory management is not going to be as much of pain in the neck under
V15 as it presently is under 14.

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of
Daniel E WILLIAMS
Sent: Tuesday, 29 August 2006 3:38
To: [hidden email]
Subject: "SORT CASES BY" memory problem.

Hello all,

I'm unable to sort my data sets. I keep getting the message:

>Error # 5828. Command name: SORT CASES BY Insufficient memory in which

>to sort the file. Increase workspace
with SET
>WORKSPACE command .

This occurs both for huge datasets and for datasets as small as 57 cases
and 29 variables. I am using Windows XP, have two processors, and
2 gigabytes of RAM. Oh, and I am using SPSS 14.0.1. I have tried
re-starting the computer, altering the WORKSPACE, and altering the
system priority for spsswin.exe.

Do you all have any ideas on what might be going wrong? I've never had
this problem before. Thanks.

Dan Williams
Forecasting, Research and Analysis Office Finance and Policy Analysis
Department of Human Services State of Oregon, USA
(503) 947-5354

Albert-Jan Roskam

Re: "SORT CASES BY" memory problem.

would it be an idea to split up your SORT command?

So instead of:
SORT CASES BY sex (a) country (d) city (a).

You would use:
SORT CASES BY sex (a).
SORT CASES BY country (d).
SORT CASES BY city (a).

Maybe that's less computer intensive?

Cheers!
Albert-Jan

--- Chris Conley <[hidden email]>
wrote:

> Have you tried incorporating CACHE in your syntax?
> We've retained it from
> some old syntax that addressed memory issues in
> older versions of SPSS. We
> haven't encountered any problems so far. But then,
> perhaps we aren't
> running files as large as you are using? Just a
> thought.
>
> Chris Conley
> Research Analyst
> Accountability and Assessment
> Durham District School Board
>
>
>
>
> "Barnett, Adrian (HEALTH)"
> <[hidden email]>
> Sent by: "SPSSX(r) Discussion"
> <[hidden email]>
> 28/08/2006 10:29 PM
> Please respond to
> "Barnett, Adrian (HEALTH)"
> <[hidden email]>
>
>
> To
> [hidden email]
> cc
>
> Subject
> Re: "SORT CASES BY" memory problem.
>
>
>
>
>
>
> Hi Daniel
> I've become very suspicious of memory management in
> V14 and believe that
> there is a flaw in it somewhere, analogous to a
> 'memory leak' - I find
> the longer SPSS has been running and the more runs
> it has done, the more
> likely it is to throw errors like this. I usually
> find that shutting
> SPSS down and restarting it will stop these
> complaints from it about not
> having sufficient memory.
>
> A separate issue from this conjectural memory
> leak-style problem, is
> that using the WORKSPACE command seems to trigger a
> permanent setting in
> the registry, so that if you've set it to a
> particular figure, it will
> still be at that setting from then on unless you
> alter it again. You can
> check this yourself by running SHOW WORKSPACE. I
> suspect you've at some
> point put in a WORKSPACE value that was too high and
> SPSS has retained
> this and is now giving you these whacko messages
> because its basically
> confused. My suggestion is that you try setting your
> workspace to a
> lower figure and try it again on one of your small
> files and see if it
> still complains. I suspect it will work OK on the
> small file if you set
> it back to its default setting (6148, I think). I'd
> then try again on
> your big file and see how you get on, and maybe
> slowly build it up again
> from there if you keep getting 'insufficient
> workspace' settings.
>
> I have a similar setup to you at home, with a
> dual-core system and 2 GB
> of RAM and tried using large values of WORKSPACE to
> see if I could speed
> up a sort on a large (700 MB, 1.4 million cases)
> file. Sorts were taking
> a long time because SPSS was spending so much time
> flogging the hard
> drive creating temporary files, with the processor
> sitting there doing
> virtually nothing while it waited for these
> temporary files to be built.
> If more could be done in memory, the sort would run
> much faster. At a
> workspace setting of 600, SPSS complained it could
> not obtain that
> amount of free memory. However the system
> performance monitor was
> showing that SPSS was in fact using 600 MB and there
> were another 900 MB
> of free memory (and of course the CPU utilisation
> was down around 10%).
> So it looks to me like SPSS is trying to do its own
> memory management
> (probably because in this version it is still trying
> to be compatible
> with Windows 98, which does not do its own memory
> management in a
> sensible way), and is getting it wrong.
>
> At values of WORKSPACE between say, 200 and 600, I
> have observed that
> some operations will complain that they could not
> obtain a memory
> segment large enough, and then fail to complete. The
> effect was more
> likely to be observed when the computer had been
> left running for days
> on end with SPSS still open, and after running long
> programs over and
> over. Rebooting seemed to cure the error messages,
> and lowering
> WORKSAPCE settings also helped. These memory-induced
> failures can have
> dire consequences if a file has to have been in a
> particular sort order
> for a subsequent process (e.g. MATCH FILES or
> transformations involving
> LAG) to work properly. The solution has been to
> wind the WORKSPACE
> setting down. The way WORKSPACE is currently
> implemented, the manual's
> advice of not changing it unless the system tells
> you it doesn't have
> enough seems largely to be correct - although
> performance will not be
> optimal unless you have a tiny file.
>
> I notice that in the information about V15, it no
> longer states it will
> run on Win 98, so I hope that means that memory
> management will now
> largely be left to the operating system. I imagine
> that SPSS usually has
> a pretty good idea of the amount of memory that
> would produce optimal
> performance for each operation, so, theoretically it
> could just ask for
> whatever amount it thinks is best and let the
> operating system give it
> whatever it has spare, thus not involving the user.
> I really hope that
> memory management is not going to be as much of pain
> in the neck under
> V15 as it presently is under 14.
>
>
>
>
>
>
> -----Original Message-----
> From: SPSSX(r) Discussion
> [mailto:[hidden email]] On Behalf Of
> Daniel E WILLIAMS
> Sent: Tuesday, 29 August 2006 3:38
> To: [hidden email]
> Subject: "SORT CASES BY" memory problem.
>
> Hello all,
>
> I'm unable to sort my data sets. I keep getting the
> message:
>
> >Error # 5828. Command name: SORT CASES BY
> Insufficient memory in which
>
> >to sort the file. Increase workspace
> with SET
> >WORKSPACE command .
>
> This occurs both for huge datasets and for datasets
> as small as 57 cases
> and 29 variables. I am using Windows XP, have two
> processors, and
> 2 gigabytes of RAM. Oh, and I am using SPSS 14.0.1.
> I have tried
> re-starting the computer, altering the WORKSPACE,
> and altering the
> system priority for spsswin.exe.
>
> Do you all have any ideas on what might be going
> wrong? I've never had
> this problem before. Thanks.
>
>
> Dan Williams
> Forecasting, Research and Analysis Office Finance
> and Policy Analysis
> Department of Human Services State of Oregon, USA
> (503) 947-5354
>

=== message truncated ===

__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

Albert-Jan Roskam

Re: "SORT CASES BY" memory problem.

In reply to this post by Chris Conley

sorry, pls ignore my previous mail. It was nonsense
what I was saying.

--- Chris Conley <[hidden email]>
wrote:

=== message truncated ===

__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

Suvra Kundu

Re: "SORT CASES BY" memory problem.

Dear List,

Suvra is no more.
Pls. tell us the process of unsubscribe.

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of
Albert-jan Roskam
Sent: Tuesday, August 29, 2006 6:19 PM
To: [hidden email]
Subject: Re: "SORT CASES BY" memory problem.

sorry, pls ignore my previous mail. It was nonsense
what I was saying.

--- Chris Conley <[hidden email]>
wrote:

=== message truncated ===

__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.