SPSSX Discussion

Re: Problems with SPSS 15? - Or is it 16? : Re: Set cache reasonable values

Classic

List

Threaded

5 messages Options

Steve Runfeldt

Re: Problems with SPSS 15? - Or is it 16? : Re: Set cache reasonable values

About two weeks ago I tried monitoring memory and CPU levels as I explored
some of the problems I have been having with V16.0. All I needed to do was
to open a pivot table within the Viewer. I did not have to copy, paste,
open another program or do any actual stats. The problem was isolated to
using the pivot tables. When I would open a table to edit in the viewer,
memory would spike at about 50% and then drop back to 0. CPU would increase
somewhat and then stay there. On about the fourth table, as CPU exceeded
about 135 Meg, the memory would spike at 50% and then remain at that level.
At that point I would start having problems with the screen blanking and
SPSS crashing.

I emailed Tech Support about this two weeks ago. Aside from an automatic
message acknowledging receipt of my email, I have not heard back from them
at all.

I considered reinstalling V16.0 but then discovered that there is a separate
installation for Viewer 16.0 as well as an installation on the CD for Viewer
15.0. When I did the original installation, I do not recall installing the
Viewer separately. So I installed Viewer 16.0 and that seemed to help the
situation. I have not been running any major analyses, but the memory
situation did seem better on some trials.

The problem seems to be that the Java is not properly releasing the CPU that
it uses when it opens and closes a pivot table. Perhaps this was fixed in
the separate Viewer installation.

I do wish that someone from SPSS TS would get back to me on this question.
Changing the heap size did not help at all. I doubt that changing the cache
size would work either. Perhaps ensuring that everyone installs the stand
alone viewer is the answer.

Steve Runfeldt

Senior Account Executive/ Quantitative Research
Schwartz Consulting Partners
5027 W. Laurel Street
Tampa, FL 33607
813-207-0332 X 233 (work)
813-431-1840 (mobile)
[hidden email]
http://schwartzconsulting.com/

------------------------------

Date: Wed, 28 Nov 2007 15:56:37 -0500
From: "[hidden email]" <[hidden email]>
Subject: Re: Problems with SPSS 15? - Or is it 16?

As a fellow geezer using SPSS since 1968 (basically freeware).
We have upgraded to 16 from 14 and have had all the problems previously
listed. We set up a computer with only Windows XP and SPSS 16.0. We still
had all of the same problems, ie, files vanish, we can't print output, CPU
runs 100%, etc. We also have found that the export output (from the viewer)
as well as a simple copy and paste also fail. It appears that you can copy
and paste as some type of graphic but not as text. Also the export to
either word or excel doesn't work. Now, this wouldn't be a big deal, but
since we can't print anything...sigh

If we are missing anything, I would love to know what it is.

Thanks

------------------------------

Date: Wed, 28 Nov 2007 19:53:19 -0700
From: Brian Moore <[hidden email]>
Subject: Re: set cache reasonable values

Thanks for the suggestions. As I continue to try to break up this
problem (such as with the file separation idea) it gets more and more
curious.

In fact now I'm getting this warning even with much smaller files.
&have had others run my larger syntax file on their computers without
error

I'm leaning toward this being some kind of hang-up that may have been
initially caused by the overflowing cache; but now SPSS is not even
checking for free space anymore before warning me.

I've shut down and restarted; but haven't tried anything as drastic as
reinstalling (yet).
Anything in between I could try?

Thanks ,
Brian

PS- one last oddity is that I can't find any obvious problems with the
RESULTS I'm getting when warned. (process is one I run every few weeks
on transactional database & levels are roughly as expected)

-----Original Message-----
From: Richard Ristow [mailto:[hidden email]]
Sent: Wednesday, November 28, 2007 1:40 PM
To: Brian Moore; [hidden email]
Cc: Gregory Hildebrandt
Subject: Re: set cache reasonable values

At 02:28 PM 11/28/2007, Brian Moore wrote:

>Thanks to everyone for insights on this issue.
>
>I have succeeded in cleaning out the cache, but still getting warnings.

>Apparently [2097151 bytes] is a software determined maximum [and cannot

>always be reached]:
>
>>Warning # 44
>>The operating system could not allocate a memory segment of the size
>>requested by SPSS.
>
>Other specs that may matter:
>-using version 15 (but are looking at upgrading to 16) -File size is
>420 MB -& I have ~20 gigs total free.

Well, that appears to rule out an *intrinsic* problem with disk space,
although you previously got a message that SPSS couldn't get the disk
space it *thought* it needed:

>>Error. Command name: SORT CASES
>>File write error: file name
>>C:\DOCUME~1\BMoore\LOCALS~1\Temp\spss2064\cache.33: No space left on
>>device (DATA1002) This command not executed.

Goodness knows how SPSS got there, but I doubt we can diagnose that,
certainly not fix it.

>Any other ideas?

Well, the command sequence that blew up before was

>> 222 COMPUTE InDupGrp = MatchSequence > 0.
>> 223 SORT CASES InDupGrp(D).

Assuming that's still the case,

(a) Very wild chance: Do you need to do it at all? The code following
the sort, in your original posting, didn't appear to rely on the data's
being sorted by 'InDupGrp'. But I doubt this'll do it; presumably, you
do need the sort, otherwise.

(b) Still relying on your original posting: It struck me that, earlier
in your code, the command

>> 212 SORT CASES BY cust(A) Order_Date_Overall(A) ProductType(D).

apparently worked, on the same file.

Sorting on the binary variable 'InDupGrp' gives a huge number of ties on
the sort key, and I wonder whether that gives the sorting algorithm some
trouble. (Yes, I can give a good argument why it
shouldn't.) I'd try appending the previous key sequence:

COMPUTE InDupGrp = MatchSequence > 0.
SORT CASES BY InDupGrp(D)
cust(A) Order_Date_Overall(A) ProductType(D).

(c) Have you tried the work-around I suggested? That is,

DO IF InDupGrp EQ 1.
. XSAVE OUTFILE=InDupYes.
ELSE.
. XSAVE OUTFILE=InDupNo.
END IF.
EXECUTE.

ADD FILES
/FILE=InDupYes
/FILE=InDupNo.

(Here, InDupYes and InDupNo are file names or file handles - NOT dataset
names - for scratch files; and the code's still not tested.)
....................
Apologies for any crucial points I've missed. But, well, 'any ideas'
this is.

-Best of luck and best wishes,
Richard

------------------------------

Date: Wed, 28 Nov 2007 20:40:28 -0800
From: Gregory Hildebrandt <[hidden email]>
Subject: Re: set cache reasonable values

Brian,

I think with one million data points and several hundred variables, SPSS
starts to have problems. For example in a similar sized dataset, I create a
chart in the SPSS Viewer, but can't copy into Excel or PowerPoint, which is
my preference for tables rather than editing the SPSS table. The data seems
to be behind the chart. Similar things happen with a large table, that has
less than 65,000 rows. However, I did move a moderate sized table into
"SPSS Pivot Table Object."

The temporary directory in Edit >> Options may fill up vary fast, so I have
replaced the default with C:\SPSS14.0\temp. About every other time I
open SPSS, I first delete all the files in the temporary directory

For sorting, increasing the memory using Set Workspace = 600000 or more has
helped, contrary to what I was told. However, there have been times in the
past when I have had to reduce the memory to permit a procedure like sorting
to work. You may want to start with the default (check "Show Workspace"),
and gradually increase the size.

I wonder if more RAM would help, or if you have used up too high a
proportion of your hard drive.

It may be time to reinstall. Make certain everything is off the hard
drive. It only takes a few minutes. However, once when I went into regedit,
with a member of the SPSS Help desk on the phone, I found remnants of an old
version of SPSS still in this directory, which I manually removed.

With a large file, the SPSS viewer also seems to increase in size very
quickly so one can easily end up with a 20mb viewer file. This might affect
your ability to use the Sort procedure. Contrary to the prevailing wisdon,
I have also found situtations in which the Syntax file is too large, and hae
had to begin a new one. This was with SPSS 11.5, so, perhaps, the problem
has been corrected.

I also wonder if you can copy the file into Access and sort in Access. Then
re-import into SPSS.

Hope this helps.

Greg

On 11/28/07, Brian Moore <[hidden email]> wrote:

>
> Thanks for the suggestions. As I continue to try to break up thiss
> problem (such as with the file separation idea) it gets more and more
> curious.
>
> In fact now I'm getting this warning even with much smaller files.
> &have had others run my larger syntax file on their computers without
> error
>
> I'm leaning toward this being some kind of hang-up that may have been
> initially caused by the overflowing cache; but now SPSS is not even
> checking for free space anymore before warning me.
>
> I've shut down and restarted; but haven't tried anything as drastic as
> reinstalling (yet).
> Anything in between I could try?
>
> Thanks ,
> Brian
>
> PS- one last oddity is that I can't find any obvious problems with the
> RESULTS I'm getting when warned. (process is one I run every few weeks
> on transactional database & levels are roughly as expected)
>
> -----Original Message-----
> From: Richard Ristow [mailto:[hidden email]]
> Sent: Wednesday, November 28, 2007 1:40 PM
> To: Brian Moore; [hidden email]
> Cc: Gregory Hildebrandt
> Subject: Re: set cache reasonable values
>
> At 02:28 PM 11/28/2007, Brian Moore wrote:
>
> >Thanks to everyone for insights on this issue.
> >
> >I have succeeded in cleaning out the cache, but still getting warnings.
>
> >Apparently [2097151 bytes] is a software determined maximum [and cannot
>
> >always be reached]:
> >
> >>Warning # 44
> >>The operating system could not allocate a memory segment of the size
> >>requested by SPSS.
> >
> >Other specs that may matter:
> >-using version 15 (but are looking at upgrading to 16) -File size is
> >420 MB -& I have ~20 gigs total free.
>
> Well, that appears to rule out an *intrinsic* problem with disk space,
> although you previously got a message that SPSS couldn't get the disk
> space it *thought* it needed:
>
> >>Error. Command name: SORT CASES
> >>File write error: file name
> >>C:\DOCUME~1\BMoore\LOCALS~1\Temp\spss2064\cache.33: No space left on
> >>device (DATA1002) This command not executed.
>
> Goodness knows how SPSS got there, but I doubt we can diagnose that,
> certainly not fix it.
>
> >Any other ideas?
>
> Well, the command sequence that blew up before was
>
> >> 222 COMPUTE InDupGrp = MatchSequence > 0.
> >> 223 SORT CASES InDupGrp(D).
>
> Assuming that's still the case,
>
> (a) Very wild chance: Do you need to do it at all? The code following
> the sort, in your original posting, didn't appear to rely on the data's
> being sorted by 'InDupGrp'. But I doubt this'll do it; presumably, you
> do need the sort, otherwise.
>
> (b) Still relying on your original posting: It struck me that, earlier
> in your code, the command
>
> >> 212 SORT CASES BY cust(A) Order_Date_Overall(A) ProductType(D).
>
> apparently worked, on the same file.
>
> Sorting on the binary variable 'InDupGrp' gives a huge number of ties on
> the sort key, and I wonder whether that gives the sorting algorithm some
> trouble. (Yes, I can give a good argument why it
> shouldn't.) I'd try appending the previous key sequence:
>
> COMPUTE InDupGrp = MatchSequence > 0.
> SORT CASES BY InDupGrp(D)
> cust(A) Order_Date_Overall(A) ProductType(D).
>
> (c) Have you tried the work-around I suggested? That is,
>
> DO IF InDupGrp EQ 1.
> . XSAVE OUTFILE=InDupYes.
> ELSE.
> . XSAVE OUTFILE=InDupNo.
> END IF.
> EXECUTE.
>
> ADD FILES
> /FILE=InDupYes
> /FILE=InDupNo.
>
> (Here, InDupYes and InDupNo are file names or file handles - NOT dataset
> names - for scratch files; and the code's still not tested.)
> ....................
> Apologies for any crucial points I've missed. But, well, 'any ideas'
> this is.
>
> -Best of luck and best wishes,
> Richard

ViAnn Beadle

Re: Problems with SPSS 15? - Or is it 16? : Re: Set cache reasonable values

Very large tables will take lots of memory to edit--there is no doubt about
that. The entire table including all the "hidden" cells (not in the
non-selected layer) are stored in memory. It's been my experience that Java
gets a bit flaky as the high-water memory mark approaches the heap size.
Moral of the story--avoid editing very large tables (by large, I'd say that
have more than 10,000 data points). Java also tries to reclaim unused memory
(it's called garbage collection) when it's not busy doing something else.
That's why you will occasionally notice spikes in CPU when nothing visible
seems to be changing.

The viewer is used to read older format viewer files (*.spo) and is a
separate executable. Behavior changes have nothing to do with installing or
un-installing it--that's got to be pure coincidence.

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of
Steve Runfeldt
Sent: Thursday, November 29, 2007 7:38 AM
To: [hidden email]
Subject: Re: Problems with SPSS 15? - Or is it 16? : Re: Set cache
reasonable values

About two weeks ago I tried monitoring memory and CPU levels as I explored
some of the problems I have been having with V16.0. All I needed to do was
to open a pivot table within the Viewer. I did not have to copy, paste,
open another program or do any actual stats. The problem was isolated to
using the pivot tables. When I would open a table to edit in the viewer,
memory would spike at about 50% and then drop back to 0. CPU would increase
somewhat and then stay there. On about the fourth table, as CPU exceeded
about 135 Meg, the memory would spike at 50% and then remain at that level.
At that point I would start having problems with the screen blanking and
SPSS crashing.

I emailed Tech Support about this two weeks ago. Aside from an automatic
message acknowledging receipt of my email, I have not heard back from them
at all.

I considered reinstalling V16.0 but then discovered that there is a separate
installation for Viewer 16.0 as well as an installation on the CD for Viewer
15.0. When I did the original installation, I do not recall installing the
Viewer separately. So I installed Viewer 16.0 and that seemed to help the
situation. I have not been running any major analyses, but the memory
situation did seem better on some trials.

The problem seems to be that the Java is not properly releasing the CPU that
it uses when it opens and closes a pivot table. Perhaps this was fixed in
the separate Viewer installation.

I do wish that someone from SPSS TS would get back to me on this question.
Changing the heap size did not help at all. I doubt that changing the cache
size would work either. Perhaps ensuring that everyone installs the stand
alone viewer is the answer.

Steve Runfeldt

Senior Account Executive/ Quantitative Research
Schwartz Consulting Partners
5027 W. Laurel Street
Tampa, FL 33607
813-207-0332 X 233 (work)
813-431-1840 (mobile)
[hidden email]
http://schwartzconsulting.com/

------------------------------

Date: Wed, 28 Nov 2007 15:56:37 -0500
From: "[hidden email]" <[hidden email]>
Subject: Re: Problems with SPSS 15? - Or is it 16?

As a fellow geezer using SPSS since 1968 (basically freeware).
We have upgraded to 16 from 14 and have had all the problems previously
listed. We set up a computer with only Windows XP and SPSS 16.0. We still
had all of the same problems, ie, files vanish, we can't print output, CPU
runs 100%, etc. We also have found that the export output (from the viewer)
as well as a simple copy and paste also fail. It appears that you can copy
and paste as some type of graphic but not as text. Also the export to
either word or excel doesn't work. Now, this wouldn't be a big deal, but
since we can't print anything...sigh

If we are missing anything, I would love to know what it is.

Thanks

------------------------------

Date: Wed, 28 Nov 2007 19:53:19 -0700
From: Brian Moore <[hidden email]>
Subject: Re: set cache reasonable values

Thanks for the suggestions. As I continue to try to break up this
problem (such as with the file separation idea) it gets more and more
curious.

In fact now I'm getting this warning even with much smaller files.
&have had others run my larger syntax file on their computers without
error

I'm leaning toward this being some kind of hang-up that may have been
initially caused by the overflowing cache; but now SPSS is not even
checking for free space anymore before warning me.

I've shut down and restarted; but haven't tried anything as drastic as
reinstalling (yet).
Anything in between I could try?

Thanks ,
Brian

PS- one last oddity is that I can't find any obvious problems with the
RESULTS I'm getting when warned. (process is one I run every few weeks
on transactional database & levels are roughly as expected)

-----Original Message-----
From: Richard Ristow [mailto:[hidden email]]
Sent: Wednesday, November 28, 2007 1:40 PM
To: Brian Moore; [hidden email]
Cc: Gregory Hildebrandt
Subject: Re: set cache reasonable values

At 02:28 PM 11/28/2007, Brian Moore wrote:

>Thanks to everyone for insights on this issue.
>
>I have succeeded in cleaning out the cache, but still getting warnings.

>Apparently [2097151 bytes] is a software determined maximum [and cannot

>always be reached]:
>
>>Warning # 44
>>The operating system could not allocate a memory segment of the size
>>requested by SPSS.
>
>Other specs that may matter:
>-using version 15 (but are looking at upgrading to 16) -File size is
>420 MB -& I have ~20 gigs total free.

Well, that appears to rule out an *intrinsic* problem with disk space,
although you previously got a message that SPSS couldn't get the disk
space it *thought* it needed:

>>Error. Command name: SORT CASES
>>File write error: file name
>>C:\DOCUME~1\BMoore\LOCALS~1\Temp\spss2064\cache.33: No space left on
>>device (DATA1002) This command not executed.

Goodness knows how SPSS got there, but I doubt we can diagnose that,
certainly not fix it.

>Any other ideas?

Well, the command sequence that blew up before was

>> 222 COMPUTE InDupGrp = MatchSequence > 0.
>> 223 SORT CASES InDupGrp(D).

Assuming that's still the case,

(a) Very wild chance: Do you need to do it at all? The code following
the sort, in your original posting, didn't appear to rely on the data's
being sorted by 'InDupGrp'. But I doubt this'll do it; presumably, you
do need the sort, otherwise.

(b) Still relying on your original posting: It struck me that, earlier
in your code, the command

>> 212 SORT CASES BY cust(A) Order_Date_Overall(A) ProductType(D).

apparently worked, on the same file.

Sorting on the binary variable 'InDupGrp' gives a huge number of ties on
the sort key, and I wonder whether that gives the sorting algorithm some
trouble. (Yes, I can give a good argument why it
shouldn't.) I'd try appending the previous key sequence:

COMPUTE InDupGrp = MatchSequence > 0.
SORT CASES BY InDupGrp(D)
cust(A) Order_Date_Overall(A) ProductType(D).

(c) Have you tried the work-around I suggested? That is,

DO IF InDupGrp EQ 1.
. XSAVE OUTFILE=InDupYes.
ELSE.
. XSAVE OUTFILE=InDupNo.
END IF.
EXECUTE.

ADD FILES
/FILE=InDupYes
/FILE=InDupNo.

(Here, InDupYes and InDupNo are file names or file handles - NOT dataset
names - for scratch files; and the code's still not tested.)
....................
Apologies for any crucial points I've missed. But, well, 'any ideas'
this is.

-Best of luck and best wishes,
Richard

------------------------------

Date: Wed, 28 Nov 2007 20:40:28 -0800
From: Gregory Hildebrandt <[hidden email]>
Subject: Re: set cache reasonable values

Brian,

I think with one million data points and several hundred variables, SPSS
starts to have problems. For example in a similar sized dataset, I create a
chart in the SPSS Viewer, but can't copy into Excel or PowerPoint, which is
my preference for tables rather than editing the SPSS table. The data seems
to be behind the chart. Similar things happen with a large table, that has
less than 65,000 rows. However, I did move a moderate sized table into
"SPSS Pivot Table Object."

The temporary directory in Edit >> Options may fill up vary fast, so I have
replaced the default with C:\SPSS14.0\temp. About every other time I
open SPSS, I first delete all the files in the temporary directory

For sorting, increasing the memory using Set Workspace = 600000 or more has
helped, contrary to what I was told. However, there have been times in the
past when I have had to reduce the memory to permit a procedure like sorting
to work. You may want to start with the default (check "Show Workspace"),
and gradually increase the size.

I wonder if more RAM would help, or if you have used up too high a
proportion of your hard drive.

It may be time to reinstall. Make certain everything is off the hard
drive. It only takes a few minutes. However, once when I went into regedit,
with a member of the SPSS Help desk on the phone, I found remnants of an old
version of SPSS still in this directory, which I manually removed.

With a large file, the SPSS viewer also seems to increase in size very
quickly so one can easily end up with a 20mb viewer file. This might affect
your ability to use the Sort procedure. Contrary to the prevailing wisdon,
I have also found situtations in which the Syntax file is too large, and hae
had to begin a new one. This was with SPSS 11.5, so, perhaps, the problem
has been corrected.

I also wonder if you can copy the file into Access and sort in Access. Then
re-import into SPSS.

Hope this helps.

Greg

On 11/28/07, Brian Moore <[hidden email]> wrote:

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

Steve Runfeldt

Re: Problems with SPSS 15? - Or is it 16? : Re: Set cache reasonable values

ViAnn,

I am not talking about large tables. The same thing happens when I click to
edit a title. It has nothing to do with the size of the table.

Am I correct that the initial install of Base includes an installation of
the Viewer? Because there is a separate installation for just the 16.0
Viewer on the disk. I do not remember installing the viewer separately when
I installed Base. There is also an installation for the V15.0 legacy viewer
on the disk as well, that is, as you say, for .spo files. Do you know
whether the 16.0 viewer installation includes a patch or upgrade for Java
memory management?

And in my case, independent installation of the 16.0 viewer has had a
dramatic effect. When I monitor, I still see CPU gradually going up, but
not as quickly as before. - Same output file, same tables.

Prior to this I also tried opening different output files of very different
sizes and opening different tables within each file also of different sizes.
Each time the problem started as soon as I transitioned between the third
and the fourth "Edit in Viewer". Since I have installed the 16.0 Viewer,
the program has behaved differently.

Steve

-----Original Message-----
From: ViAnn Beadle [mailto:[hidden email]]
Sent: Thursday, November 29, 2007 10:25 AM
To: 'Steve Runfeldt'; [hidden email]
Subject: RE: Problems with SPSS 15? - Or is it 16? : Re: Set cache
reasonable values

Very large tables will take lots of memory to edit--there is no doubt about
that. The entire table including all the "hidden" cells (not in the
non-selected layer) are stored in memory. It's been my experience that Java
gets a bit flaky as the high-water memory mark approaches the heap size.
Moral of the story--avoid editing very large tables (by large, I'd say that
have more than 10,000 data points). Java also tries to reclaim unused memory
(it's called garbage collection) when it's not busy doing something else.
That's why you will occasionally notice spikes in CPU when nothing visible
seems to be changing.

The viewer is used to read older format viewer files (*.spo) and is a
separate executable. Behavior changes have nothing to do with installing or
un-installing it--that's got to be pure coincidence.

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of
Steve Runfeldt
Sent: Thursday, November 29, 2007 7:38 AM
To: [hidden email]
Subject: Re: Problems with SPSS 15? - Or is it 16? : Re: Set cache
reasonable values

About two weeks ago I tried monitoring memory and CPU levels as I explored
some of the problems I have been having with V16.0. All I needed to do was
to open a pivot table within the Viewer. I did not have to copy, paste,
open another program or do any actual stats. The problem was isolated to
using the pivot tables. When I would open a table to edit in the viewer,
memory would spike at about 50% and then drop back to 0. CPU would increase
somewhat and then stay there. On about the fourth table, as CPU exceeded
about 135 Meg, the memory would spike at 50% and then remain at that level.
At that point I would start having problems with the screen blanking and
SPSS crashing.

I emailed Tech Support about this two weeks ago. Aside from an automatic
message acknowledging receipt of my email, I have not heard back from them
at all.

I considered reinstalling V16.0 but then discovered that there is a separate
installation for Viewer 16.0 as well as an installation on the CD for Viewer
15.0. When I did the original installation, I do not recall installing the
Viewer separately. So I installed Viewer 16.0 and that seemed to help the
situation. I have not been running any major analyses, but the memory
situation did seem better on some trials.

The problem seems to be that the Java is not properly releasing the CPU that
it uses when it opens and closes a pivot table. Perhaps this was fixed in
the separate Viewer installation.

I do wish that someone from SPSS TS would get back to me on this question.
Changing the heap size did not help at all. I doubt that changing the cache
size would work either. Perhaps ensuring that everyone installs the stand
alone viewer is the answer.

Steve Runfeldt

Senior Account Executive/ Quantitative Research
Schwartz Consulting Partners
5027 W. Laurel Street
Tampa, FL 33607
813-207-0332 X 233 (work)
813-431-1840 (mobile)
[hidden email]
http://schwartzconsulting.com/

------------------------------

Date: Wed, 28 Nov 2007 15:56:37 -0500
From: "[hidden email]" <[hidden email]>
Subject: Re: Problems with SPSS 15? - Or is it 16?

As a fellow geezer using SPSS since 1968 (basically freeware).
We have upgraded to 16 from 14 and have had all the problems previously
listed. We set up a computer with only Windows XP and SPSS 16.0. We still
had all of the same problems, ie, files vanish, we can't print output, CPU
runs 100%, etc. We also have found that the export output (from the viewer)
as well as a simple copy and paste also fail. It appears that you can copy
and paste as some type of graphic but not as text. Also the export to
either word or excel doesn't work. Now, this wouldn't be a big deal, but
since we can't print anything...sigh

If we are missing anything, I would love to know what it is.

Thanks

------------------------------

Date: Wed, 28 Nov 2007 19:53:19 -0700
From: Brian Moore <[hidden email]>
Subject: Re: set cache reasonable values

Thanks for the suggestions. As I continue to try to break up this
problem (such as with the file separation idea) it gets more and more
curious.

In fact now I'm getting this warning even with much smaller files.
&have had others run my larger syntax file on their computers without
error

I'm leaning toward this being some kind of hang-up that may have been
initially caused by the overflowing cache; but now SPSS is not even
checking for free space anymore before warning me.

I've shut down and restarted; but haven't tried anything as drastic as
reinstalling (yet).
Anything in between I could try?

Thanks ,
Brian

PS- one last oddity is that I can't find any obvious problems with the
RESULTS I'm getting when warned. (process is one I run every few weeks
on transactional database & levels are roughly as expected)

-----Original Message-----
From: Richard Ristow [mailto:[hidden email]]
Sent: Wednesday, November 28, 2007 1:40 PM
To: Brian Moore; [hidden email]
Cc: Gregory Hildebrandt
Subject: Re: set cache reasonable values

At 02:28 PM 11/28/2007, Brian Moore wrote:

>Thanks to everyone for insights on this issue.
>
>I have succeeded in cleaning out the cache, but still getting warnings.

>Apparently [2097151 bytes] is a software determined maximum [and cannot

>always be reached]:
>
>>Warning # 44
>>The operating system could not allocate a memory segment of the size
>>requested by SPSS.
>
>Other specs that may matter:
>-using version 15 (but are looking at upgrading to 16) -File size is
>420 MB -& I have ~20 gigs total free.

Well, that appears to rule out an *intrinsic* problem with disk space,
although you previously got a message that SPSS couldn't get the disk
space it *thought* it needed:

>>Error. Command name: SORT CASES
>>File write error: file name
>>C:\DOCUME~1\BMoore\LOCALS~1\Temp\spss2064\cache.33: No space left on
>>device (DATA1002) This command not executed.

Goodness knows how SPSS got there, but I doubt we can diagnose that,
certainly not fix it.

>Any other ideas?

Well, the command sequence that blew up before was

>> 222 COMPUTE InDupGrp = MatchSequence > 0.
>> 223 SORT CASES InDupGrp(D).

Assuming that's still the case,

(a) Very wild chance: Do you need to do it at all? The code following
the sort, in your original posting, didn't appear to rely on the data's
being sorted by 'InDupGrp'. But I doubt this'll do it; presumably, you
do need the sort, otherwise.

(b) Still relying on your original posting: It struck me that, earlier
in your code, the command

>> 212 SORT CASES BY cust(A) Order_Date_Overall(A) ProductType(D).

apparently worked, on the same file.

Sorting on the binary variable 'InDupGrp' gives a huge number of ties on
the sort key, and I wonder whether that gives the sorting algorithm some
trouble. (Yes, I can give a good argument why it
shouldn't.) I'd try appending the previous key sequence:

COMPUTE InDupGrp = MatchSequence > 0.
SORT CASES BY InDupGrp(D)
cust(A) Order_Date_Overall(A) ProductType(D).

(c) Have you tried the work-around I suggested? That is,

DO IF InDupGrp EQ 1.
. XSAVE OUTFILE=InDupYes.
ELSE.
. XSAVE OUTFILE=InDupNo.
END IF.
EXECUTE.

ADD FILES
/FILE=InDupYes
/FILE=InDupNo.

(Here, InDupYes and InDupNo are file names or file handles - NOT dataset
names - for scratch files; and the code's still not tested.)
....................
Apologies for any crucial points I've missed. But, well, 'any ideas'
this is.

-Best of luck and best wishes,
Richard

------------------------------

Date: Wed, 28 Nov 2007 20:40:28 -0800
From: Gregory Hildebrandt <[hidden email]>
Subject: Re: set cache reasonable values

Brian,

I think with one million data points and several hundred variables, SPSS
starts to have problems. For example in a similar sized dataset, I create a
chart in the SPSS Viewer, but can't copy into Excel or PowerPoint, which is
my preference for tables rather than editing the SPSS table. The data seems
to be behind the chart. Similar things happen with a large table, that has
less than 65,000 rows. However, I did move a moderate sized table into
"SPSS Pivot Table Object."

The temporary directory in Edit >> Options may fill up vary fast, so I have
replaced the default with C:\SPSS14.0\temp. About every other time I
open SPSS, I first delete all the files in the temporary directory

For sorting, increasing the memory using Set Workspace = 600000 or more has
helped, contrary to what I was told. However, there have been times in the
past when I have had to reduce the memory to permit a procedure like sorting
to work. You may want to start with the default (check "Show Workspace"),
and gradually increase the size.

I wonder if more RAM would help, or if you have used up too high a
proportion of your hard drive.

It may be time to reinstall. Make certain everything is off the hard
drive. It only takes a few minutes. However, once when I went into regedit,
with a member of the SPSS Help desk on the phone, I found remnants of an old
version of SPSS still in this directory, which I manually removed.

With a large file, the SPSS viewer also seems to increase in size very
quickly so one can easily end up with a 20mb viewer file. This might affect
your ability to use the Sort procedure. Contrary to the prevailing wisdon,
I have also found situtations in which the Syntax file is too large, and hae
had to begin a new one. This was with SPSS 11.5, so, perhaps, the problem
has been corrected.

I also wonder if you can copy the file into Access and sort in Access. Then
re-import into SPSS.

Hope this helps.

Greg

On 11/28/07, Brian Moore <[hidden email]> wrote:

ViAnn Beadle

Re: Problems with SPSS 15? - Or is it 16? : Re: Set cache reasonable values

How much memory do you have? It may be that the default heap size is larger
than your available memory and you're running into a thrashing problem.

Now with regard to viewers, the Autoplay screen provides two SmartViewer
installs. Which one are you talking about?

The SmartViewer 16.0 IS a separate install on the initial autoplay window
and intended for distribution to others who do not have SPSS. It is not
installed automatically since it duplicates the built-in viewer within SPSS.
The SPSS 15.0 SmartViewer is used to open old .spo files and only works for
Windows. It is installed separately and is a must if you want to read your
old .spo files.

It might be that if you installed SmartViewer 16.0, you got additional java
classes which would help SPSS itself--but it seems to me that the viewer is
probably just using the same code as the built-in viewer.

I have no problems with editing the ordinary run of the mill type output and
noticed no different with the viewer installed, but then I'm just playing
with SPSS and not doing really serious things with it. SPSS does get slower
the larger the viewer file, or number of viewer files open. I have 2GB of
memory.

-----Original Message-----
From: Steve Runfeldt [mailto:[hidden email]]
Sent: Thursday, November 29, 2007 9:15 AM
To: 'ViAnn Beadle'; [hidden email]
Subject: RE: Problems with SPSS 15? - Or is it 16? : Re: Set cache
reasonable values

ViAnn,

I am not talking about large tables. The same thing happens when I click to
edit a title. It has nothing to do with the size of the table.

Am I correct that the initial install of Base includes an installation of
the Viewer? Because there is a separate installation for just the 16.0
Viewer on the disk. I do not remember installing the viewer separately when
I installed Base. There is also an installation for the V15.0 legacy viewer
on the disk as well, that is, as you say, for .spo files. Do you know
whether the 16.0 viewer installation includes a patch or upgrade for Java
memory management?

And in my case, independent installation of the 16.0 viewer has had a
dramatic effect. When I monitor, I still see CPU gradually going up, but
not as quickly as before. - Same output file, same tables.

Prior to this I also tried opening different output files of very different
sizes and opening different tables within each file also of different sizes.
Each time the problem started as soon as I transitioned between the third
and the fourth "Edit in Viewer". Since I have installed the 16.0 Viewer,
the program has behaved differently.

Steve

-----Original Message-----
From: ViAnn Beadle [mailto:[hidden email]]
Sent: Thursday, November 29, 2007 10:25 AM
To: 'Steve Runfeldt'; [hidden email]
Subject: RE: Problems with SPSS 15? - Or is it 16? : Re: Set cache
reasonable values

Very large tables will take lots of memory to edit--there is no doubt about
that. The entire table including all the "hidden" cells (not in the
non-selected layer) are stored in memory. It's been my experience that Java
gets a bit flaky as the high-water memory mark approaches the heap size.
Moral of the story--avoid editing very large tables (by large, I'd say that
have more than 10,000 data points). Java also tries to reclaim unused memory
(it's called garbage collection) when it's not busy doing something else.
That's why you will occasionally notice spikes in CPU when nothing visible
seems to be changing.

The viewer is used to read older format viewer files (*.spo) and is a
separate executable. Behavior changes have nothing to do with installing or
un-installing it--that's got to be pure coincidence.

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of
Steve Runfeldt
Sent: Thursday, November 29, 2007 7:38 AM
To: [hidden email]
Subject: Re: Problems with SPSS 15? - Or is it 16? : Re: Set cache
reasonable values

About two weeks ago I tried monitoring memory and CPU levels as I explored
some of the problems I have been having with V16.0. All I needed to do was
to open a pivot table within the Viewer. I did not have to copy, paste,
open another program or do any actual stats. The problem was isolated to
using the pivot tables. When I would open a table to edit in the viewer,
memory would spike at about 50% and then drop back to 0. CPU would increase
somewhat and then stay there. On about the fourth table, as CPU exceeded
about 135 Meg, the memory would spike at 50% and then remain at that level.
At that point I would start having problems with the screen blanking and
SPSS crashing.

I emailed Tech Support about this two weeks ago. Aside from an automatic
message acknowledging receipt of my email, I have not heard back from them
at all.

I considered reinstalling V16.0 but then discovered that there is a separate
installation for Viewer 16.0 as well as an installation on the CD for Viewer
15.0. When I did the original installation, I do not recall installing the
Viewer separately. So I installed Viewer 16.0 and that seemed to help the
situation. I have not been running any major analyses, but the memory
situation did seem better on some trials.

The problem seems to be that the Java is not properly releasing the CPU that
it uses when it opens and closes a pivot table. Perhaps this was fixed in
the separate Viewer installation.

I do wish that someone from SPSS TS would get back to me on this question.
Changing the heap size did not help at all. I doubt that changing the cache
size would work either. Perhaps ensuring that everyone installs the stand
alone viewer is the answer.

Steve Runfeldt

Senior Account Executive/ Quantitative Research
Schwartz Consulting Partners
5027 W. Laurel Street
Tampa, FL 33607
813-207-0332 X 233 (work)
813-431-1840 (mobile)
[hidden email]
http://schwartzconsulting.com/

------------------------------

Date: Wed, 28 Nov 2007 15:56:37 -0500
From: "[hidden email]" <[hidden email]>
Subject: Re: Problems with SPSS 15? - Or is it 16?

As a fellow geezer using SPSS since 1968 (basically freeware).
We have upgraded to 16 from 14 and have had all the problems previously
listed. We set up a computer with only Windows XP and SPSS 16.0. We still
had all of the same problems, ie, files vanish, we can't print output, CPU
runs 100%, etc. We also have found that the export output (from the viewer)
as well as a simple copy and paste also fail. It appears that you can copy
and paste as some type of graphic but not as text. Also the export to
either word or excel doesn't work. Now, this wouldn't be a big deal, but
since we can't print anything...sigh

If we are missing anything, I would love to know what it is.

Thanks

------------------------------

Date: Wed, 28 Nov 2007 19:53:19 -0700
From: Brian Moore <[hidden email]>
Subject: Re: set cache reasonable values

Thanks for the suggestions. As I continue to try to break up this
problem (such as with the file separation idea) it gets more and more
curious.

In fact now I'm getting this warning even with much smaller files.
&have had others run my larger syntax file on their computers without
error

I'm leaning toward this being some kind of hang-up that may have been
initially caused by the overflowing cache; but now SPSS is not even
checking for free space anymore before warning me.

I've shut down and restarted; but haven't tried anything as drastic as
reinstalling (yet).
Anything in between I could try?

Thanks ,
Brian

PS- one last oddity is that I can't find any obvious problems with the
RESULTS I'm getting when warned. (process is one I run every few weeks
on transactional database & levels are roughly as expected)

-----Original Message-----
From: Richard Ristow [mailto:[hidden email]]
Sent: Wednesday, November 28, 2007 1:40 PM
To: Brian Moore; [hidden email]
Cc: Gregory Hildebrandt
Subject: Re: set cache reasonable values

At 02:28 PM 11/28/2007, Brian Moore wrote:

>Thanks to everyone for insights on this issue.
>
>I have succeeded in cleaning out the cache, but still getting warnings.

>Apparently [2097151 bytes] is a software determined maximum [and cannot

>always be reached]:
>
>>Warning # 44
>>The operating system could not allocate a memory segment of the size
>>requested by SPSS.
>
>Other specs that may matter:
>-using version 15 (but are looking at upgrading to 16) -File size is
>420 MB -& I have ~20 gigs total free.

Well, that appears to rule out an *intrinsic* problem with disk space,
although you previously got a message that SPSS couldn't get the disk
space it *thought* it needed:

>>Error. Command name: SORT CASES
>>File write error: file name
>>C:\DOCUME~1\BMoore\LOCALS~1\Temp\spss2064\cache.33: No space left on
>>device (DATA1002) This command not executed.

Goodness knows how SPSS got there, but I doubt we can diagnose that,
certainly not fix it.

>Any other ideas?

Well, the command sequence that blew up before was

>> 222 COMPUTE InDupGrp = MatchSequence > 0.
>> 223 SORT CASES InDupGrp(D).

Assuming that's still the case,

(a) Very wild chance: Do you need to do it at all? The code following
the sort, in your original posting, didn't appear to rely on the data's
being sorted by 'InDupGrp'. But I doubt this'll do it; presumably, you
do need the sort, otherwise.

(b) Still relying on your original posting: It struck me that, earlier
in your code, the command

>> 212 SORT CASES BY cust(A) Order_Date_Overall(A) ProductType(D).

apparently worked, on the same file.

Sorting on the binary variable 'InDupGrp' gives a huge number of ties on
the sort key, and I wonder whether that gives the sorting algorithm some
trouble. (Yes, I can give a good argument why it
shouldn't.) I'd try appending the previous key sequence:

COMPUTE InDupGrp = MatchSequence > 0.
SORT CASES BY InDupGrp(D)
cust(A) Order_Date_Overall(A) ProductType(D).

(c) Have you tried the work-around I suggested? That is,

DO IF InDupGrp EQ 1.
. XSAVE OUTFILE=InDupYes.
ELSE.
. XSAVE OUTFILE=InDupNo.
END IF.
EXECUTE.

ADD FILES
/FILE=InDupYes
/FILE=InDupNo.

(Here, InDupYes and InDupNo are file names or file handles - NOT dataset
names - for scratch files; and the code's still not tested.)
....................
Apologies for any crucial points I've missed. But, well, 'any ideas'
this is.

-Best of luck and best wishes,
Richard

------------------------------

Date: Wed, 28 Nov 2007 20:40:28 -0800
From: Gregory Hildebrandt <[hidden email]>
Subject: Re: set cache reasonable values

Brian,

I think with one million data points and several hundred variables, SPSS
starts to have problems. For example in a similar sized dataset, I create a
chart in the SPSS Viewer, but can't copy into Excel or PowerPoint, which is
my preference for tables rather than editing the SPSS table. The data seems
to be behind the chart. Similar things happen with a large table, that has
less than 65,000 rows. However, I did move a moderate sized table into
"SPSS Pivot Table Object."

The temporary directory in Edit >> Options may fill up vary fast, so I have
replaced the default with C:\SPSS14.0\temp. About every other time I
open SPSS, I first delete all the files in the temporary directory

For sorting, increasing the memory using Set Workspace = 600000 or more has
helped, contrary to what I was told. However, there have been times in the
past when I have had to reduce the memory to permit a procedure like sorting
to work. You may want to start with the default (check "Show Workspace"),
and gradually increase the size.

I wonder if more RAM would help, or if you have used up too high a
proportion of your hard drive.

It may be time to reinstall. Make certain everything is off the hard
drive. It only takes a few minutes. However, once when I went into regedit,
with a member of the SPSS Help desk on the phone, I found remnants of an old
version of SPSS still in this directory, which I manually removed.

With a large file, the SPSS viewer also seems to increase in size very
quickly so one can easily end up with a 20mb viewer file. This might affect
your ability to use the Sort procedure. Contrary to the prevailing wisdon,
I have also found situtations in which the Syntax file is too large, and hae
had to begin a new one. This was with SPSS 11.5, so, perhaps, the problem
has been corrected.

I also wonder if you can copy the file into Access and sort in Access. Then
re-import into SPSS.

Hope this helps.

Greg

On 11/28/07, Brian Moore <[hidden email]> wrote:

Steve Runfeldt

Re: Problems with SPSS 15? - Or is it 16? : Re: Set cache reasonable values

1.96 G of RAM. So I should have no problem in that regard. I upgraded the
heap to a value of 2, which should be about half the available memory.

The problem is that the Java does not release all the memory as I go from
one table to another in the viewer. It gradually builds up and reaches a
limit beyond which the system fails. I just heard from Devon in Tech
Support. Apparently he had the wrong phone number for me.

He says:

I did want to let you know that the 16.0.1 patch should be out very soon,
and it will address a number of performance issues, especially those
concerning output files. Once the patch is available, you should recieve a
proactive notification. If you have continued problems after applying the
patch, please contact us to troubleshoot further. Thanks!

So that seems to answer that for now.

Regarding the viewers. From what I can see here, the stand alone viewer is
doing something differently than the built-in viewer. CPU still
accumulates, but much more slowly.

Maybe someone else who has been having problems could try the stand-alone
V16 viewer. Let us know what you find.

Steve

-----Original Message-----
From: ViAnn Beadle [mailto:[hidden email]]
Sent: Thursday, November 29, 2007 12:00 PM
To: 'Steve Runfeldt'; [hidden email]
Subject: RE: Problems with SPSS 15? - Or is it 16? : Re: Set cache
reasonable values

How much memory do you have? It may be that the default heap size is larger
than your available memory and you're running into a thrashing problem.

Now with regard to viewers, the Autoplay screen provides two SmartViewer
installs. Which one are you talking about?

The SmartViewer 16.0 IS a separate install on the initial autoplay window
and intended for distribution to others who do not have SPSS. It is not
installed automatically since it duplicates the built-in viewer within SPSS.
The SPSS 15.0 SmartViewer is used to open old .spo files and only works for
Windows. It is installed separately and is a must if you want to read your
old .spo files.

It might be that if you installed SmartViewer 16.0, you got additional java
classes which would help SPSS itself--but it seems to me that the viewer is
probably just using the same code as the built-in viewer.

I have no problems with editing the ordinary run of the mill type output and
noticed no different with the viewer installed, but then I'm just playing
with SPSS and not doing really serious things with it. SPSS does get slower
the larger the viewer file, or number of viewer files open. I have 2GB of
memory.

-----Original Message-----
From: Steve Runfeldt [mailto:[hidden email]]
Sent: Thursday, November 29, 2007 9:15 AM
To: 'ViAnn Beadle'; [hidden email]
Subject: RE: Problems with SPSS 15? - Or is it 16? : Re: Set cache
reasonable values

ViAnn,

I am not talking about large tables. The same thing happens when I click to
edit a title. It has nothing to do with the size of the table.

Am I correct that the initial install of Base includes an installation of
the Viewer? Because there is a separate installation for just the 16.0
Viewer on the disk. I do not remember installing the viewer separately when
I installed Base. There is also an installation for the V15.0 legacy viewer
on the disk as well, that is, as you say, for .spo files. Do you know
whether the 16.0 viewer installation includes a patch or upgrade for Java
memory management?

And in my case, independent installation of the 16.0 viewer has had a
dramatic effect. When I monitor, I still see CPU gradually going up, but
not as quickly as before. - Same output file, same tables.

Prior to this I also tried opening different output files of very different
sizes and opening different tables within each file also of different sizes.
Each time the problem started as soon as I transitioned between the third
and the fourth "Edit in Viewer". Since I have installed the 16.0 Viewer,
the program has behaved differently.

Steve

-----Original Message-----
From: ViAnn Beadle [mailto:[hidden email]]
Sent: Thursday, November 29, 2007 10:25 AM
To: 'Steve Runfeldt'; [hidden email]
Subject: RE: Problems with SPSS 15? - Or is it 16? : Re: Set cache
reasonable values

Very large tables will take lots of memory to edit--there is no doubt about
that. The entire table including all the "hidden" cells (not in the
non-selected layer) are stored in memory. It's been my experience that Java
gets a bit flaky as the high-water memory mark approaches the heap size.
Moral of the story--avoid editing very large tables (by large, I'd say that
have more than 10,000 data points). Java also tries to reclaim unused memory
(it's called garbage collection) when it's not busy doing something else.
That's why you will occasionally notice spikes in CPU when nothing visible
seems to be changing.

The viewer is used to read older format viewer files (*.spo) and is a
separate executable. Behavior changes have nothing to do with installing or
un-installing it--that's got to be pure coincidence.

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of
Steve Runfeldt
Sent: Thursday, November 29, 2007 7:38 AM
To: [hidden email]
Subject: Re: Problems with SPSS 15? - Or is it 16? : Re: Set cache
reasonable values

About two weeks ago I tried monitoring memory and CPU levels as I explored
some of the problems I have been having with V16.0. All I needed to do was
to open a pivot table within the Viewer. I did not have to copy, paste,
open another program or do any actual stats. The problem was isolated to
using the pivot tables. When I would open a table to edit in the viewer,
memory would spike at about 50% and then drop back to 0. CPU would increase
somewhat and then stay there. On about the fourth table, as CPU exceeded
about 135 Meg, the memory would spike at 50% and then remain at that level.
At that point I would start having problems with the screen blanking and
SPSS crashing.

I emailed Tech Support about this two weeks ago. Aside from an automatic
message acknowledging receipt of my email, I have not heard back from them
at all.

I considered reinstalling V16.0 but then discovered that there is a separate
installation for Viewer 16.0 as well as an installation on the CD for Viewer
15.0. When I did the original installation, I do not recall installing the
Viewer separately. So I installed Viewer 16.0 and that seemed to help the
situation. I have not been running any major analyses, but the memory
situation did seem better on some trials.

The problem seems to be that the Java is not properly releasing the CPU that
it uses when it opens and closes a pivot table. Perhaps this was fixed in
the separate Viewer installation.

I do wish that someone from SPSS TS would get back to me on this question.
Changing the heap size did not help at all. I doubt that changing the cache
size would work either. Perhaps ensuring that everyone installs the stand
alone viewer is the answer.

Steve Runfeldt

Senior Account Executive/ Quantitative Research
Schwartz Consulting Partners
5027 W. Laurel Street
Tampa, FL 33607
813-207-0332 X 233 (work)
813-431-1840 (mobile)
[hidden email]
http://schwartzconsulting.com/

------------------------------

Date: Wed, 28 Nov 2007 15:56:37 -0500
From: "[hidden email]" <[hidden email]>
Subject: Re: Problems with SPSS 15? - Or is it 16?

As a fellow geezer using SPSS since 1968 (basically freeware).
We have upgraded to 16 from 14 and have had all the problems previously
listed. We set up a computer with only Windows XP and SPSS 16.0. We still
had all of the same problems, ie, files vanish, we can't print output, CPU
runs 100%, etc. We also have found that the export output (from the viewer)
as well as a simple copy and paste also fail. It appears that you can copy
and paste as some type of graphic but not as text. Also the export to
either word or excel doesn't work. Now, this wouldn't be a big deal, but
since we can't print anything...sigh

If we are missing anything, I would love to know what it is.

Thanks

------------------------------

Date: Wed, 28 Nov 2007 19:53:19 -0700
From: Brian Moore <[hidden email]>
Subject: Re: set cache reasonable values

Thanks for the suggestions. As I continue to try to break up this
problem (such as with the file separation idea) it gets more and more
curious.

In fact now I'm getting this warning even with much smaller files.
&have had others run my larger syntax file on their computers without
error

I'm leaning toward this being some kind of hang-up that may have been
initially caused by the overflowing cache; but now SPSS is not even
checking for free space anymore before warning me.

I've shut down and restarted; but haven't tried anything as drastic as
reinstalling (yet).
Anything in between I could try?

Thanks ,
Brian

PS- one last oddity is that I can't find any obvious problems with the
RESULTS I'm getting when warned. (process is one I run every few weeks
on transactional database & levels are roughly as expected)

-----Original Message-----
From: Richard Ristow [mailto:[hidden email]]
Sent: Wednesday, November 28, 2007 1:40 PM
To: Brian Moore; [hidden email]
Cc: Gregory Hildebrandt
Subject: Re: set cache reasonable values

At 02:28 PM 11/28/2007, Brian Moore wrote:

>Thanks to everyone for insights on this issue.
>
>I have succeeded in cleaning out the cache, but still getting warnings.

>Apparently [2097151 bytes] is a software determined maximum [and cannot

>always be reached]:
>
>>Warning # 44
>>The operating system could not allocate a memory segment of the size
>>requested by SPSS.
>
>Other specs that may matter:
>-using version 15 (but are looking at upgrading to 16) -File size is
>420 MB -& I have ~20 gigs total free.

Well, that appears to rule out an *intrinsic* problem with disk space,
although you previously got a message that SPSS couldn't get the disk
space it *thought* it needed:

>>Error. Command name: SORT CASES
>>File write error: file name
>>C:\DOCUME~1\BMoore\LOCALS~1\Temp\spss2064\cache.33: No space left on
>>device (DATA1002) This command not executed.

Goodness knows how SPSS got there, but I doubt we can diagnose that,
certainly not fix it.

>Any other ideas?

Well, the command sequence that blew up before was

>> 222 COMPUTE InDupGrp = MatchSequence > 0.
>> 223 SORT CASES InDupGrp(D).

Assuming that's still the case,

(a) Very wild chance: Do you need to do it at all? The code following
the sort, in your original posting, didn't appear to rely on the data's
being sorted by 'InDupGrp'. But I doubt this'll do it; presumably, you
do need the sort, otherwise.

(b) Still relying on your original posting: It struck me that, earlier
in your code, the command

>> 212 SORT CASES BY cust(A) Order_Date_Overall(A) ProductType(D).

apparently worked, on the same file.

Sorting on the binary variable 'InDupGrp' gives a huge number of ties on
the sort key, and I wonder whether that gives the sorting algorithm some
trouble. (Yes, I can give a good argument why it
shouldn't.) I'd try appending the previous key sequence:

COMPUTE InDupGrp = MatchSequence > 0.
SORT CASES BY InDupGrp(D)
cust(A) Order_Date_Overall(A) ProductType(D).

(c) Have you tried the work-around I suggested? That is,

DO IF InDupGrp EQ 1.
. XSAVE OUTFILE=InDupYes.
ELSE.
. XSAVE OUTFILE=InDupNo.
END IF.
EXECUTE.

ADD FILES
/FILE=InDupYes
/FILE=InDupNo.

(Here, InDupYes and InDupNo are file names or file handles - NOT dataset
names - for scratch files; and the code's still not tested.)
....................
Apologies for any crucial points I've missed. But, well, 'any ideas'
this is.

-Best of luck and best wishes,
Richard

------------------------------

Date: Wed, 28 Nov 2007 20:40:28 -0800
From: Gregory Hildebrandt <[hidden email]>
Subject: Re: set cache reasonable values

Brian,

I think with one million data points and several hundred variables, SPSS
starts to have problems. For example in a similar sized dataset, I create a
chart in the SPSS Viewer, but can't copy into Excel or PowerPoint, which is
my preference for tables rather than editing the SPSS table. The data seems
to be behind the chart. Similar things happen with a large table, that has
less than 65,000 rows. However, I did move a moderate sized table into
"SPSS Pivot Table Object."

The temporary directory in Edit >> Options may fill up vary fast, so I have
replaced the default with C:\SPSS14.0\temp. About every other time I
open SPSS, I first delete all the files in the temporary directory

For sorting, increasing the memory using Set Workspace = 600000 or more has
helped, contrary to what I was told. However, there have been times in the
past when I have had to reduce the memory to permit a procedure like sorting
to work. You may want to start with the default (check "Show Workspace"),
and gradually increase the size.

I wonder if more RAM would help, or if you have used up too high a
proportion of your hard drive.

It may be time to reinstall. Make certain everything is off the hard
drive. It only takes a few minutes. However, once when I went into regedit,
with a member of the SPSS Help desk on the phone, I found remnants of an old
version of SPSS still in this directory, which I manually removed.

With a large file, the SPSS viewer also seems to increase in size very
quickly so one can easily end up with a 20mb viewer file. This might affect
your ability to use the Sort procedure. Contrary to the prevailing wisdon,
I have also found situtations in which the Syntax file is too large, and hae
had to begin a new one. This was with SPSS 11.5, so, perhaps, the problem
has been corrected.

I also wonder if you can copy the file into Access and sort in Access. Then
re-import into SPSS.

Hope this helps.

Greg

On 11/28/07, Brian Moore <[hidden email]> wrote: