SPSS 17 Stalling with large dataset (35000 columns)

[Michael A. Dover]

I have all the latest patches.  As I write, SPSS is stalling as I simply try to cancel out a failed attempt to use Utilities, Variables to create a list of variable names.  Used Task Manager to exit, and noticed that it was saying something crazy like 1.1 gig of memory being used by pawstat.exe!  Why would a 6 MB file need 1.1 gigs of memory?  Just reloaded, and with no data open, paswstat.exe uses 72k of memory, spssengine uses 30k.  Just loaded my dataset, a 7365k file that ran fine in SPSS 10.  Haven't done anything with it, but it uses 128932 k (128kk) of memory for paswstat and 69868 (69kk) of memory of spssengine.  Ran display dictionary info for the working file and it is processing......memory use in spssengine is up to 150kk, 160kk and going up, up, up, now it's at 270kk for paswstate, 482kk, 551kk, wow.....Look at the memory usage!  This is crazy.  Is it possible that the Java-based SPSS PASW 17 no longer lives up to its advertised ability to handle unlimited numbers of variables? It doesn't seem to, at least not in the real computer world I live in, one where the operating system restricts memory to 3 gigs.  Still going up.  You would think someone in SPSS could run an analysis correlating number of columns to memory usage and just inform the customers honestly: "Our software no longer has the functionality of earlier versions" or "Yet again there are bugs and this time it has to do with inordinate memory usage with large files..we hope to fix it".  Ok, it stopped at paswstat.exe 883548k of memory and spssengine 158376 of memory.  Total memory is 2058176, available 570432 (nothing else is running really except a browser) and system cache is 312832.  Kernel memory is 147868, 91232 is paged, 56636 is nonpaged.   Now I'm going to try (once again, it never works) to save the data dictionary as ascii text and memory usage in paswstate.exe is up to 1.1 gigs, spssengine 158kk.  I'm ordering an extra 2 gig ram chip right now to see if this will help.

[SPSS Support]

I did not see the memory usage climbing like this when I tried this, but I was able to replicate the fact that the program seems to freeze when you try to highight the entire list of variable names with this many variables, and I replicated the ability to do this on earlier releases. I've filed a bug on this and for anyone affected here is a workaround for getting a syntax file with a list of variable names for a very large number of variables:

 

Open a syntax window. Type in the following two lines:

 

flip.
list CASE_LBL.

 

In the menus, click Run>All.

 

An output window will open. In it will be a log from the FLIP command and the listing of variable names from the LIST command.

 

Double-click on the list of variable names to activate it. Click OK if a popup warning occurs.

 

In the menus, click Edit>Select All, then Edit>Copy.

 

Go back to the syntax window or open a new one, and paste the text. There is a bit of extraneous text that will need to be deleted, leaving the list of variable names.

 

 

We apologize for the hassle.

 

David Nichols

---

From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Michael A. Dover
Sent: Wednesday, July 15, 2009 2:47 PM
To: [hidden email]
Subject: [SPSSX-L] SPSS 17 Stalling with large dataset (35000 columns)

I have all the latest patches.  As I write, SPSS is stalling as I simply try to cancel out a failed attempt to use Utilities, Variables to create a list of variable names.  Used Task Manager to exit, and noticed that it was saying something crazy like 1.1 gig of memory being used by pawstat.exe!  Why would a 6 MB file need 1.1 gigs of memory?  Just reloaded, and with no data open, paswstat.exe uses 72k of memory, spssengine uses 30k.  Just loaded my dataset, a 7365k file that ran fine in SPSS 10.  Haven't done anything with it, but it uses 128932 k (128kk) of memory for paswstat and 69868 (69kk) of memory of spssengine.  Ran display dictionary info for the working file and it is processing......memory use in spssengine is up to 150kk, 160kk and going up, up, up, now it's at 270kk for paswstate, 482kk, 551kk, wow.....Look at the memory usage!  This is crazy.  Is it possible that the Java-based SPSS PASW 17 no longer lives up to its advertised ability to handle unlimited numbers of variables? It doesn't seem to, at least not in the real computer world I live in, one where the operating system restricts memory to 3 gigs.  Still going up.  You would think someone in SPSS could run an analysis correlating number of columns to memory usage and just inform the customers honestly: "Our software no longer has the functionality of earlier versions" or "Yet again there are bugs and this time it has to do with inordinate memory usage with large files..we hope to fix it".  Ok, it stopped at paswstat.exe 883548k of memory and spssengine 158376 of memory.  Total memory is 2058176, available 570432 (nothing else is running really except a browser) and system cache is 312832.  Kernel memory is 147868, 91232 is paged, 56636 is nonpaged.   Now I'm going to try (once again, it never works) to save the data dictionary as ascii text and memory usage in paswstate.exe is up to 1.1 gigs, spssengine 158kk.  I'm ordering an extra 2 gig ram chip right now to see if this will help.