Insufficient Space

[Deepanshu Bhalla]

CONTENTS DELETED

The author has deleted this message.

[David Marso]

Filled up your hard disk?

[Deepanshu Bhalla]

CONTENTS DELETED

The author has deleted this message.

[David Marso]

Hmmm, Spring has sprung:  Maybe time for spring cleaning.
1.  I would move as much as possible from your primary disk to the external.  Delete from primary and then try again.
and/or
2.  If you are creating TONS of variables in the aggregate then possibly break it into 2-3 smaller pieces and merge the files later.
3.  Possibly redirect the TEMP directory to the External.

Deepanshu Bhalla wrote

Hey DM

Hope you are doing awesome !

To resolve the error i added external hard disk to the system and command SPSS to save the file there..Now the situation is that error is not coming up but all the desired output is not coming (i.e all the variables are sysmis out)
 
I have no idea why it is taking more than 40 gb space to save the file . I have been using the syntax for more than a year and i am getting this weird error first time ..Initially there was error comming up "Insufficient memory to run Aggregate command in more than 65k data " so i added "/Presorted " command before */break " to resolve the error .

Note : It saves temporary file of more than 15 GB data in the SPSS temp directory.

[Deepanshu Bhalla]

CONTENTS DELETED

The author has deleted this message.

[Fleming, Katie (DOT)]

I will be out of the office March 26-29, 2012. I will have very limitted access to e-mail, but I will respond as quickly as possible.

 

Thank you,

 

Katie Fleming

 

[David Marso]

Please define "not working".
I suggested 3 possibly overlapping strategies.  Which did you try?
What exactly is the AGGREGATE command you are submitting?
---

Deepanshu Bhalla wrote

Tried these but not working !

[Deepanshu Bhalla]

CONTENTS DELETED

The author has deleted this message.

[Rich Ulrich]

It takes a lot of disk space and computing time because you
have "5 million data" which I assume indicates the number of
lines, and you have (apparently) a whole lot of variables, on
which you are doing a whole lot of aggregates.  You *should*
be using a presorted file, and specifying "presorted" for the
aggregate, or you are wasting time in every run for the simple
Sort that has to be done for Aggregate.

One old-style aid, back when everyone had slow computers
and (therefore) slow computing time, would be to create a
much smaller Test-file.  Since you are doing aggregates, it
could be better to take (say) the first 50 000 lines, rather than
take a 1% sample.  Then you work out your syntax and
procedures with a smaller file and less waiting, before applying
it to the original dataset.

The huge number of Aggregates implies that you are doing
data mining.  The problem you mentioned, having "Sysmis" that
are unexplained, suggests that you may not have cleaned up your
data before starting this phase.  A rule of thumb for data-mining
is that you spend 90% of your time in data-cleanup, for the 10%
in analysis and interpretation.  I suspect that the fraction for
clean-up is even larger when you have an inordinately large set
of miscellaneous variables.  Have you done a full set of
univariate screens on the data? and looked at a few, obvious
bi-variate tabulations, to confirm that the data consist of what
you expected?

 - When I see a very large number of variables, I do wonder
if there is better file organization possible, at least for the
purpose of screening and editing.  For instance, you may want
multiple time periods in one record, for doing repeated measures
testing;  but a vertical, multiple-record format means that you
cut the number of screened variables by 3 or 5 or 10-fold,
whatever the number of repetitions is.

--
Rich Ulrich

> Date: Sun, 25 Mar 2012 09:47:30 -0700

> From: [hidden email]
> Subject: Re: Insufficient Space
> To: [hidden email]
>
> I tried first and second suggested methods but the issue still persist .
>
> In first method the error is not coming up but all the variables are
> sysmissed out and taking hell lot of time to run the code (SYSTEM CONFIG :
> I5 processor ,6 GB DDR3 RAM)
>
> In second method SPSS got hanged after running the syntax for 2 hours.
>
> I am submitting 2472 FIN MEAN N functions in Aggregate command .
>

...