Physical Memory Usage

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

Physical Memory Usage

drfg2008
SPSS 20 / 64bit /16Gig

We run SPSS jobs that need a long time (iteration jobs for about 30-40 minutes). During that time the physical memory usage history shows a linear increase until no physical memory is left.

Even after the job is finished, no physical memory is set free. We have to shut down SPSS to get free physical memory.

Is there a command in SPSS to set physical memory free?
Dr. Frank Gaeth

Reply | Threaded
Open this post in threaded view
|

Re: Physical Memory Usage

David Marso
Administrator
"Is there a command in SPSS to set physical memory free? "
Not to my knowledge.  Maybe JoNoH has thoughts on that.
This sounds like abnormal behavior.
Maybe you should provide a little more information on what you are actually doing?
You have the tendency to provide little or no useful information in your requests for assistance.
In this case I imagine providing some vague idea of the meat behind the madness?
With 16G you must be running some obscenely greedy code to be running out.
Perhaps you should investigate Production Mode for doing this?
As usual, I am providing you with just enough rope to hang yourself ;-)
It is just a bandaid.  You should try to discover the cause by careful analysis of your code and analytical methodology and practices.

drfg2008 wrote
SPSS 20 / 64bit /16Gig

We run SPSS jobs that need a long time (iteration jobs for about 30-40 minutes). During that time the physical memory usage history shows a linear increase until no physical memory is left.

Even after the job is finished, no physical memory is set free. We have to shut down SPSS to get free physical memory.

Is there a command in SPSS to set physical memory free?
Please reply to the list and not to my personal email.
Those desiring my consulting or training services please feel free to email me.
---
"Nolite dare sanctum canibus neque mittatis margaritas vestras ante porcos ne forte conculcent eas pedibus suis."
Cum es damnatorum possederunt porcos iens ut salire off sanguinum cliff in abyssum?"
Reply | Threaded
Open this post in threaded view
|

Re: Physical Memory Usage

Morrell, Ronald
In reply to this post by drfg2008
I'm also interested in this issue.  I'm having to close SPSS after working with very large data sets (1 GB or more) to enable other programs to work correctly.

Thanks in advance for any insight that can be provided.

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of drfg2008
Sent: Thursday, September 27, 2012 9:06 AM
To: [hidden email]
Subject: Physical Memory Usage

SPSS 20 / 64bit /16Gig

We run SPSS jobs that need a long time (iteration jobs for about 30-40 minutes). During that time the physical memory usage history shows a linear increase until no physical memory is left.

Even after the job is finished, no physical memory is set free. We have to shut down SPSS to get free physical memory.

Is there a command in SPSS to set physical memory free?



-----
Dr. Frank Gaeth
FU-Berlin

--
View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Physical-Memory-Usage-tp5715313.html
Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD
Reply | Threaded
Open this post in threaded view
|

Re: Physical Memory Usage

David Marso
Administrator
Well, except for very few procedures, SPSS does NOT keep the data file in memory so the 1G file size is probably not terribly relevant.  What is going on in the code which exhausts memory?
--
Morrell, Ronald wrote
I'm also interested in this issue.  I'm having to close SPSS after working with very large data sets (1 GB or more) to enable other programs to work correctly.

Thanks in advance for any insight that can be provided.

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of drfg2008
Sent: Thursday, September 27, 2012 9:06 AM
To: [hidden email]
Subject: Physical Memory Usage

SPSS 20 / 64bit /16Gig

We run SPSS jobs that need a long time (iteration jobs for about 30-40 minutes). During that time the physical memory usage history shows a linear increase until no physical memory is left.

Even after the job is finished, no physical memory is set free. We have to shut down SPSS to get free physical memory.

Is there a command in SPSS to set physical memory free?



-----
Dr. Frank Gaeth
FU-Berlin

--
View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Physical-Memory-Usage-tp5715313.html
Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD
Please reply to the list and not to my personal email.
Those desiring my consulting or training services please feel free to email me.
---
"Nolite dare sanctum canibus neque mittatis margaritas vestras ante porcos ne forte conculcent eas pedibus suis."
Cum es damnatorum possederunt porcos iens ut salire off sanguinum cliff in abyssum?"
Reply | Threaded
Open this post in threaded view
|

Re: Physical Memory Usage

Mark Miller
The issue might not be from SPSS (or directly so).
If this is on Win64, you might use Task Manager (process view)
to see whether the memory use is attributed to the SPSS process.
I have seen similar cases with other software where the
memory use by the process in question hardly changed at all
but the total memory use continued to increase steadily
over several hours. I was never able to nail down the root
cause but it seemed to be related to conflicting caching behavior
between Windows and a network storage array.

... Mark Miller


On Thu, Sep 27, 2012 at 12:18 PM, David Marso <[hidden email]> wrote:
Well, except for very few procedures, SPSS does NOT keep the data file in
memory so the 1G file size is probably not terribly relevant.  What is going
on in the code which exhausts memory?
--

Morrell, Ronald wrote
> I'm also interested in this issue.  I'm having to close SPSS after working
> with very large data sets (1 GB or more) to enable other programs to work
> correctly.
>
> Thanks in advance for any insight that can be provided.
>
> -----Original Message-----
> From: SPSSX(r) Discussion [mailto:

> SPSSX-L@.UGA

> ] On Behalf Of drfg2008
> Sent: Thursday, September 27, 2012 9:06 AM
> To:

> SPSSX-L@.UGA

> Subject: Physical Memory Usage
>
> SPSS 20 / 64bit /16Gig
>
> We run SPSS jobs that need a long time (iteration jobs for about 30-40
> minutes). During that time the physical memory usage history shows a
> linear increase until no physical memory is left.
>
> Even after the job is finished, no physical memory is set free. We have to
> shut down SPSS to get free physical memory.
>
> Is there a command in SPSS to set physical memory free?
>
>
>
> -----
> Dr. Frank Gaeth
> FU-Berlin
>
> --
> View this message in context:
> http://spssx-discussion.1045642.n5.nabble.com/Physical-Memory-Usage-tp5715313.html
> Sent from the SPSSX Discussion mailing list archive at Nabble.com.
>
> =====================
> To manage your subscription to SPSSX-L, send a message to

> LISTSERV@.UGA

>  (not to SPSSX-L), with no body text except the command. To leave the
> list, send the command SIGNOFF SPSSX-L For a list of commands to manage
> subscriptions, send the command INFO REFCARD
>
> =====================
> To manage your subscription to SPSSX-L, send a message to

> LISTSERV@.UGA

>  (not to SPSSX-L), with no body text except the
> command. To leave the list, send the command
> SIGNOFF SPSSX-L
> For a list of commands to manage subscriptions, send the command
> INFO REFCARD





-----
Please reply to the list and not to my personal email.
Those desiring my consulting or training services please feel free to email me.
--
View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Physical-Memory-Usage-tp5715313p5715322.html
Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

Reply | Threaded
Open this post in threaded view
|

Re: Physical Memory Usage

Jon K Peck
In reply to this post by drfg2008
Measuring actual physical memory usage is rather tricky.  Programs run in virtual memory, and for the most part, the OS controls what physical memory is allocated.  Physical will be less than virtual with SPSS, but since Java memory management is invisible to the OS, and parts of the system are in Java, you can't really see the actual resource usage with standard tools.  There are parameters you can set in the jvmcdg.ini file to limit the maximum memory that Java can use, but reducing that will slow Statistics down.

However, the first question is which processes you are reporting on.  The stats process is holding all the Viewer output, the Data Editor, and the other parts of the user interface.  Closing a Viewer window may deallocate the memory within Java, but it may or may not be returned to the OS.  The spssengine process is the backend and handles computations and data and generates graphics.  Programmability is executed in the startx or startx32 process.

So for starters, see which process or processes are gobbling up memory by watching the Task Manager.  Then we may be able to diagnose what is happening.  There could be OS issues as well, but start with this.

For comprehensive resource measurements, you could use the STATS BENCHMRK extension command to collect a lot of time, memory, and i/o usage measures, but I would start with the basics.

Jon Peck (no "h") aka Kim
Senior Software Engineer, IBM
[hidden email]
new phone: 720-342-5621




From:        drfg2008 <[hidden email]>
To:        [hidden email]
Date:        09/27/2012 07:12 AM
Subject:        [SPSSX-L] Physical Memory Usage
Sent by:        "SPSSX(r) Discussion" <[hidden email]>




SPSS 20 / 64bit /16Gig

We run SPSS jobs that need a long time (iteration jobs for about 30-40
minutes). During that time the physical memory usage history shows a linear
increase until no physical memory is left.

Even after the job is finished, no physical memory is set free. We have to
shut down SPSS to get free physical memory.

Is there a command in SPSS to set physical memory free?



-----
Dr. Frank Gaeth
FU-Berlin

--
View this message in context:
http://spssx-discussion.1045642.n5.nabble.com/Physical-Memory-Usage-tp5715313.html
Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD


Reply | Threaded
Open this post in threaded view
|

Re: Physical Memory Usage

Morrell, Ronald
In reply to this post by David Marso
The syntax I was running was modifying two file to prepare them for matching.  The code would get file A, construct a pseudo ID, sort on the pseudo ID and unduplicate the file on the pseudo ID.  The code with then get the second file and go through the same steps minus the unduplication.  The last part of the syntax matches the files on the pseudo ID and save the resulting file as a new file.  I'll rerun the process and monitor the Task Manager as Mark suggested to see what happens.

Thanks for the insight.
Ron

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of David Marso
Sent: Thursday, September 27, 2012 3:19 PM
To: [hidden email]
Subject: Re: Physical Memory Usage

Well, except for very few procedures, SPSS does NOT keep the data file in memory so the 1G file size is probably not terribly relevant.  What is going on in the code which exhausts memory?
--

Morrell, Ronald wrote
> I'm also interested in this issue.  I'm having to close SPSS after
> working with very large data sets (1 GB or more) to enable other
> programs to work correctly.
>
> Thanks in advance for any insight that can be provided.
>
> -----Original Message-----
> From: SPSSX(r) Discussion [mailto:

> SPSSX-L@.UGA

> ] On Behalf Of drfg2008
> Sent: Thursday, September 27, 2012 9:06 AM
> To:

> SPSSX-L@.UGA

> Subject: Physical Memory Usage
>
> SPSS 20 / 64bit /16Gig
>
> We run SPSS jobs that need a long time (iteration jobs for about 30-40
> minutes). During that time the physical memory usage history shows a
> linear increase until no physical memory is left.
>
> Even after the job is finished, no physical memory is set free. We
> have to shut down SPSS to get free physical memory.
>
> Is there a command in SPSS to set physical memory free?
>
>
>
> -----
> Dr. Frank Gaeth
> FU-Berlin
>
> --
> View this message in context:
> http://spssx-discussion.1045642.n5.nabble.com/Physical-Memory-Usage-tp
> 5715313.html Sent from the SPSSX Discussion mailing list archive at
> Nabble.com.
>
> =====================
> To manage your subscription to SPSSX-L, send a message to

> LISTSERV@.UGA

>  (not to SPSSX-L), with no body text except the command. To leave the
> list, send the command SIGNOFF SPSSX-L For a list of commands to
> manage subscriptions, send the command INFO REFCARD
>
> =====================
> To manage your subscription to SPSSX-L, send a message to

> LISTSERV@.UGA

>  (not to SPSSX-L), with no body text except the command. To leave the
> list, send the command SIGNOFF SPSSX-L For a list of commands to
> manage subscriptions, send the command INFO REFCARD





-----
Please reply to the list and not to my personal email.
Those desiring my consulting or training services please feel free to email me.
--
View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Physical-Memory-Usage-tp5715313p5715322.html
Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD
Reply | Threaded
Open this post in threaded view
|

Re: Physical Memory Usage

Poes, Matthew Joseph
In reply to this post by David Marso
It's my understanding that the iterative processes that rely on "matrix" syntax do hold it in memory and that this can cause a memory problem.  Some of the earlier developed bootstrap add-on programs used a matrix syntax to do the iterations, holding each in memory.  Well written ones will limit the number of cases and variables that can be used, poorly written ones will not account for this, and just lock up the computer.

Matthew J Poes
Research Data Specialist
Center for Prevention Research and Development
University of Illinois
510 Devonshire Dr.
Champaign, IL 61820
Phone: 217-265-4576
email: [hidden email]


-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of David Marso
Sent: Thursday, September 27, 2012 2:19 PM
To: [hidden email]
Subject: Re: Physical Memory Usage

Well, except for very few procedures, SPSS does NOT keep the data file in memory so the 1G file size is probably not terribly relevant.  What is going on in the code which exhausts memory?
--

Morrell, Ronald wrote
> I'm also interested in this issue.  I'm having to close SPSS after
> working with very large data sets (1 GB or more) to enable other
> programs to work correctly.
>
> Thanks in advance for any insight that can be provided.
>
> -----Original Message-----
> From: SPSSX(r) Discussion [mailto:

> SPSSX-L@.UGA

> ] On Behalf Of drfg2008
> Sent: Thursday, September 27, 2012 9:06 AM
> To:

> SPSSX-L@.UGA

> Subject: Physical Memory Usage
>
> SPSS 20 / 64bit /16Gig
>
> We run SPSS jobs that need a long time (iteration jobs for about 30-40
> minutes). During that time the physical memory usage history shows a
> linear increase until no physical memory is left.
>
> Even after the job is finished, no physical memory is set free. We
> have to shut down SPSS to get free physical memory.
>
> Is there a command in SPSS to set physical memory free?
>
>
>
> -----
> Dr. Frank Gaeth
> FU-Berlin
>
> --
> View this message in context:
> http://spssx-discussion.1045642.n5.nabble.com/Physical-Memory-Usage-tp
> 5715313.html Sent from the SPSSX Discussion mailing list archive at
> Nabble.com.
>
> =====================
> To manage your subscription to SPSSX-L, send a message to

> LISTSERV@.UGA

>  (not to SPSSX-L), with no body text except the command. To leave the
> list, send the command SIGNOFF SPSSX-L For a list of commands to
> manage subscriptions, send the command INFO REFCARD
>
> =====================
> To manage your subscription to SPSSX-L, send a message to

> LISTSERV@.UGA

>  (not to SPSSX-L), with no body text except the command. To leave the
> list, send the command SIGNOFF SPSSX-L For a list of commands to
> manage subscriptions, send the command INFO REFCARD





-----
Please reply to the list and not to my personal email.
Those desiring my consulting or training services please feel free to email me.
--
View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Physical-Memory-Usage-tp5715313p5715322.html
Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD
Reply | Threaded
Open this post in threaded view
|

Re: Physical Memory Usage

David Marso
Administrator
Remedy for that is use SPLIT FILE *BUT* you have to run the MATRIX code "batch" .
INCLUDE or INSERT (with appropriate switches).
One thing people should be aware of when doing development with MATRIX  is DISPLAY command within MATRIX for profiling code.
Another is RELEASE .

<OT insightful freebie for MATRiX hounds >
If anyone is creating LARGE MATRIX programs it is handy to grok the following insight.
Rather than have a MATRIX program span several hundred lines, segment it into several macros.
DEFINE Matsub1 (arg defs).
Matrix code for segment 1.
!ENDDEFINE .

DEFINE Matsub2 (arg defs).
Matrix code for segment 2.
!ENDDEFINE .
.....
DEFINE Matsubk (arg defs).
Matrix code for segment k.
!ENDDEFINE .

MATRIX.
GET statements
....
Matsum1 args1.
Matsub2 args2.
.....
Matsubk argsk.
.....
....
Wrapup....
END MATRIX.
------------
I will leave it to the people doing such coding to think about this and hopefully use it in their progs.
It certainly makes debugging easier and any comment code shows up in the left pane of the Sin-tax window
---.
I am currently creating code for a client which implements currently 2 variations (more to come up the road) of a complex calculation.  Much of the code shares certain common processing steps with slightly different parameters.
It makes no sense to create 2 macros with a lot of redundant code.  Rather build in a METHOD parameter and !IF branch appropriately.  AFTER ALL.  Macro just generates code, it doesn't matter if it is regular SPSS syntax or MATRIX code it will simply drop in wherever it is called.  Pretty freaking neato eh?
I have not previously seen this technique espoused or previously documented but it makes absolute sense when one ponders it for a moment or two ;-)
Much of the MATRIX code I have seen over the years is absolutely atrocious WRT any sane concept of modularity or reasonable appreciation of basic concepts of programming.  The consistent lack of commenting is particularly disturbing .
--------
</OT insightful freebie for MATRiX hounds >


Poes, Matthew Joseph wrote
It's my understanding that the iterative processes that rely on "matrix" syntax do hold it in memory and that this can cause a memory problem.  Some of the earlier developed bootstrap add-on programs used a matrix syntax to do the iterations, holding each in memory.  Well written ones will limit the number of cases and variables that can be used, poorly written ones will not account for this, and just lock up the computer.

Matthew J Poes
Research Data Specialist
Center for Prevention Research and Development
University of Illinois
510 Devonshire Dr.
Champaign, IL 61820
Phone: 217-265-4576
email: [hidden email]


-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of David Marso
Sent: Thursday, September 27, 2012 2:19 PM
To: [hidden email]
Subject: Re: Physical Memory Usage

Well, except for very few procedures, SPSS does NOT keep the data file in memory so the 1G file size is probably not terribly relevant.  What is going on in the code which exhausts memory?
--

Morrell, Ronald wrote
> I'm also interested in this issue.  I'm having to close SPSS after
> working with very large data sets (1 GB or more) to enable other
> programs to work correctly.
>
> Thanks in advance for any insight that can be provided.
>
> -----Original Message-----
> From: SPSSX(r) Discussion [mailto:

> SPSSX-L@.UGA

> ] On Behalf Of drfg2008
> Sent: Thursday, September 27, 2012 9:06 AM
> To:

> SPSSX-L@.UGA

> Subject: Physical Memory Usage
>
> SPSS 20 / 64bit /16Gig
>
> We run SPSS jobs that need a long time (iteration jobs for about 30-40
> minutes). During that time the physical memory usage history shows a
> linear increase until no physical memory is left.
>
> Even after the job is finished, no physical memory is set free. We
> have to shut down SPSS to get free physical memory.
>
> Is there a command in SPSS to set physical memory free?
>
>
>
> -----
> Dr. Frank Gaeth
> FU-Berlin
>
> --
> View this message in context:
> http://spssx-discussion.1045642.n5.nabble.com/Physical-Memory-Usage-tp
> 5715313.html Sent from the SPSSX Discussion mailing list archive at
> Nabble.com.
>
> =====================
> To manage your subscription to SPSSX-L, send a message to

> LISTSERV@.UGA

>  (not to SPSSX-L), with no body text except the command. To leave the
> list, send the command SIGNOFF SPSSX-L For a list of commands to
> manage subscriptions, send the command INFO REFCARD
>
> =====================
> To manage your subscription to SPSSX-L, send a message to

> LISTSERV@.UGA

>  (not to SPSSX-L), with no body text except the command. To leave the
> list, send the command SIGNOFF SPSSX-L For a list of commands to
> manage subscriptions, send the command INFO REFCARD





-----
Please reply to the list and not to my personal email.
Those desiring my consulting or training services please feel free to email me.
--
View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Physical-Memory-Usage-tp5715313p5715322.html
Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD
Please reply to the list and not to my personal email.
Those desiring my consulting or training services please feel free to email me.
---
"Nolite dare sanctum canibus neque mittatis margaritas vestras ante porcos ne forte conculcent eas pedibus suis."
Cum es damnatorum possederunt porcos iens ut salire off sanguinum cliff in abyssum?"
Reply | Threaded
Open this post in threaded view
|

Re: Physical Memory Usage

nyclmis
In reply to this post by drfg2008
Have you found any way to correct/limit how much memory is allocated to SPSS v20 64bit? I have this same problem on a 64-bit OS, Windows 7, and 24G RAM - i.e., con tinual increase in memory use until no memory is left for it or anything else.

I am quite sure it is SPSS hogging the memory (as monitored in Task Manager). Routinely, I am running simple routines with 40million (national Census) observations and not any complex matrix routines. The speed of analysis is about the same as when we used a workstation 4GB with Windows XP on a 32-bit machine. This can't be right.

Thanks in advance,
Lesley (NYCLMIS)
Reply | Threaded
Open this post in threaded view
|

Re: Physical Memory Usage

drfg2008
Have you found any way to correct/limit how much memory is allocated to SPSS v20 64bit?

No we haven't (apart from starting SPSS again).
Dr. Frank Gaeth

Reply | Threaded
Open this post in threaded view
|

Re: Physical Memory Usage

Jon K Peck
In reply to this post by nyclmis
As I have posted before, the first step is to understand which SPSS processes - mainly the spssengine backend process or the stats frontend process - are consuming memory.  If it is the frontend process, the immediate thing to consider is the size of the output, i.e., the Viewer contents.  Periodically closing the Viewer via the OUTPUT * command can help organize the output better and allow that memory to be reused for new output.  The backend process is rarely an issue, but very large graphics could consume a large amount of memory.  Also very large sets of value labels can take a lot of memory.  Long running OMS requests could also require a lot of memory.

For the Java portion of memory usage, you can adjust the settings in jvmcfg.ini to limit memory usage.

As for speed, 64-bit apps are not typically faster than 32-bit apps.  Rather, they allow more memory to be used on problems that might not otherwise be possible.  Large numbers of cases don't generally take large amounts of memory except for a few procedures that have to keep the data in memory - especially time series procedures.  64-bit apps can actually be slower, because memory addresses require twice as many bytes and sometimes compiler optimization issues, but this is not typical.

HTH,

Jon Peck (no "h") aka Kim
Senior Software Engineer, IBM
[hidden email]
new phone: 720-342-5621




From:        nyclmis <[hidden email]>
To:        [hidden email]
Date:        10/10/2012 06:46 PM
Subject:        Re: [SPSSX-L] Physical Memory Usage
Sent by:        "SPSSX(r) Discussion" <[hidden email]>




Have you found any way to correct/limit how much memory is allocated to SPSS
v20 64bit? I have this same problem on a 64-bit OS, Windows 7, and 24G RAM -
i.e., con tinual increase in memory use until no memory is left for it or
anything else.

I am quite sure it is SPSS hogging the memory (as monitored in Task
Manager). Routinely, I am running simple routines with 40million (national
Census) observations and not any complex matrix routines. The speed of
analysis is about the same as when we used a workstation 4GB with Windows XP
on a 32-bit machine. This can't be right.

Thanks in advance,
Lesley (NYCLMIS)



--
View this message in context:
http://spssx-discussion.1045642.n5.nabble.com/Physical-Memory-Usage-tp5715313p5715585.html
Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD


Reply | Threaded
Open this post in threaded view
|

Re: Physical Memory Usage

Dan Conlon
In reply to this post by drfg2008
I have experienced similar performance problems with V20.  I checked with SPSS Support and their primary response is to get more RAM.  They also provided this link regarding minimum hardware requirements :
http://www-01.ibm.com/support/docview.wss?uid=swg2148049 .   Based on Lesley's post and experience using 24GB RAM, increased RAM does not appear to resolve the issue.  

I have 4GB RAM. Using Task Manager, I see my memory usage shoot up to 97-98% when doing relatively routine tasks in SPSS such as autorecode, aggregate, sort  (although often with files several GB in size).

One other observation I have (which may or may not be pertinent) regards the operating system.  Our agency recently replaced Windows XP on all of our machines with Windows 7. I did not see these performance problems when using Windows XP. With Windows XP, I could easily run SPSS and have other applications  (e.g., Excel, PL/SQL, Outlook, IE) running at the same time without major performance issues. Now, with Windows 7, it is very difficult to run SPSS simultaneously with other applications.  I don't have benchmark measures to evaluate before and after Windows 7. But the differences are obvious. SPSS appears to use almost all of the RAM and I often cannot tab to other programs without long delays.  For any with insight in decoding Task Manager (not me), it appears that stats.exe and javaw.exe are demanding the most resources.      

Dan
Reply | Threaded
Open this post in threaded view
|

Re: Physical Memory Usage

Dan Conlon
In reply to this post by drfg2008
If the link on my previous posting doesn't work, here is technical note from SPSS:

Hardware recommendations for IBM SPSS Statistics software
.Technote (troubleshooting)

Problem(Abstract)Does IBM have any hardware recommendations for IBM SPSS Statistics software?

Resolving the problemEach IBM SPSS Statistics software program has different hardware and operating system requirements. See the product listing for your software on the IBM website for individual product system requirements.

Be advised that system requirements for an IBM SPSS Statistics software program indicate the minimum requirements for the software program to operate. Meeting the minimum specifications will allow for a minimum of performance. If the tasks that you are performing involve very large data files or very complex analyses or both, it is advised that your hardware platform exceed the minimum specifications.

It is impossible for IBM to provide specific recommendations for hardware platform specifications, as each customer's needs are different. However, the following information attempts to provide general guidelines for IBM SPSS Statistics customers with regard to hardware.

Hard Disk Drive Space
Allow enough disk space. The rule of thumb is you have to have four times the free hard disk space available as your SPSS data file's size is. So if your data file is 2Gig in size then you will need 8Gigs free.

Random Access Memory (RAM)
Consider adding as much RAM as (1) your operating system and/or (2) your hardware supports. For operating system and hardware support information, see the documentation of your vendor.

Processor (CPU) Speed
In general, a faster processor is better than a slower one. This is true whether the processor is single-core or multi-core. Frequently, in processor descriptions, the speed of all of the cores combined is used to report the processor speed. For those SPSS programs that do NOT support multiple core processors, the only relevant speed indicator is the speed of one of the cores.

Generally speaking, get as much resources as your budget allows.

In terms of working with large data files, if you feel you are running into performance issues, it may be better to work on a subset of the data instead of having the entire data file open in one instance of your IBM SPSS Statistics software program. If your data is large, and the analyses you are performing does not require all of the variables or cases, consider creating subsets of the larger dataset.

** Some Specific Recommendations for Microsoft Windows Operating Systems **
In the case of workstation computers using Microsoft Windows operating systems, performance issues can sometimes be addressed by making changes to the manner in which the operating system makes use of available hardware/software resources. The following changes have been known, in some cases, to improve system performance:

Close all open programs.
Open System Properties (Control Panel > System).
Select 'Advanced (tab)
Locate 'Performance' (section).
Select 'Settings' (button).
Select 'Visual Effects' (tab).
Select 'Adjust for best performance' (option).
Select 'Apply' (button).
Select 'Advanced' (tab).
Locate 'Processor scheduling' (section).
Confirm/Select 'Programs' (option).
Locate 'Memory usage' (section).
Confirm/Select 'Programs' (option).
Locate 'Virtual memory' (section).
Select 'Change' (button).
Adjust 'Initial size' to 'Recommended' (or largest allowable value).
Adjust 'Maximum' size' to 'Recommended' (or largest allowable value).
If changes were made, select the 'Set' button.
(You may be warned that the computer will have to be restarted to apply changes. If so, select 'OK'.)
Select 'OK' button until window is closed..
(If changes were made to virtual memory setting, you will be prompted to restart the computer.)
Close all open programs/windows.
Restart computer


.Historical Number48336
.
Reply | Threaded
Open this post in threaded view
|

Re: Physical Memory Usage

Poes, Matthew Joseph
In reply to this post by Dan Conlon
My experience and understanding with this issue is that "routine" really has nothing to do with it.  Very routine procedures (things as simple as sort procedures) can require a lot of memory.  Certain tasks hold portions of the data file in memory until finished.  If you have a data file that is 4 gigs in size, and it needs to hold half of it in memory, then 2 gigs are being dedicated just to holding the file there.  My understanding is that this is far more rare than it used to be, but I do know that the matrix language still does this.

Maybe Jon knows better if the iterative process jobs hold the iterations in memory until it finishes.  That is the only way I could see a computer with 24 gigs of ram using all the ram up to run a job.  Again, if you have a 4 gig data file, you run a job that does 5000 bootstrap resamples, I could see that portions of the data file (while its randomly grabbing cases) could be held in memory, and if it was holding many of these in memory at once while it ran the job, could use up a lot of memory.  My largest data file, which I use on a regular basis, is 107mb's.  If you run something iterative on this, it seems to eat up all the ram and take quite a long time.  More ram I'm sure would help, but if its holding portions of that data file in memory, no amount of ram would fully resolve that.  Again, Jon may know better how this is done, and how likely it is that a really large data file could be held in ram in such a way that would use it up.  For instance, does a bootstrap proc!
 edures attempt to hold my 107mb file in ram 5000 times?  10,000 times?  Some more complicated version of this same principle?

Matthew J Poes
Research Data Specialist
Center for Prevention Research and Development
University of Illinois
510 Devonshire Dr.
Champaign, IL 61820
Phone: 217-265-4576
email: [hidden email]


-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Dan Conlon
Sent: Thursday, October 11, 2012 11:43 AM
To: [hidden email]
Subject: Re: Physical Memory Usage

I have experienced similar performance problems with V20.  I checked with SPSS Support and their primary response is to get more RAM.  They also provided this link regarding minimum hardware requirements :
http://www-01.ibm.com/support/docview.wss?uid=swg2148049 .   Based on
Lesley's post and experience using 24GB RAM, increased RAM does not appear to resolve the issue.

I have 4GB RAM. Using Task Manager, I see my memory usage shoot up to 97-98% when doing relatively routine tasks in SPSS such as autorecode, aggregate, sort  (although often with files several GB in size).

One other observation I have (which may or may not be pertinent) regards the operating system.  Our agency recently replaced Windows XP on all of our machines with Windows 7. I did not see these performance problems when using Windows XP. With Windows XP, I could easily run SPSS and have other applications  (e.g., Excel, PL/SQL, Outlook, IE) running at the same time without major performance issues. Now, with Windows 7, it is very difficult to run SPSS simultaneously with other applications.  I don't have benchmark measures to evaluate before and after Windows 7. But the differences are obvious. SPSS appears to use almost all of the RAM and I often cannot tab to other programs without long delays.  For any with insight in decoding Task Manager (not me), it appears that stats.exe and javaw.exe are demanding the most resources.

Dan




--
View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Physical-Memory-Usage-tp5715313p5715597.html
Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD