Error #10963 (memory) processing a very large file

[Albert-Jan Roskam]

Hi all,

 

I am trying to aggregate a list with 10**7 cases but I get the error below. The .txt file is a bit over 5Gb and has just 2 columns. I suspect that the syntax causes the computer to run out of temporary disk space. That could be fixed by removing the CACHE-EXE. But since it takes forver to run this syntax (and probably even longer without the cache command), I wanted to ask your advice.

 

Thank you!

 

Albert-Jan

 

 

   INSERT FILE='D:\users\RUN\AJR\TEMP\spssprod.spp' SYNTAX=BATCH ERROR=STOP .
   *>
   *>file handle myfile /name = "<A href="file://\\blah\bleh\bloh\Temp\largefile.txt">\\blah\bleh\bloh\Temp\largefile.txt"
   *>
   *>
   *>GET DATA  /TYPE = TXT
   *> /FILE = myfile
   *> /DELCASE = LINE
   *> /DELIMITERS = "\t
   *> /ARRANGEMENT = DELIMITED
   *> /FIRSTCASE = 2
   *> /IMPORTCASE = ALL
   *> /VARIABLES
   *> ssn       F9.
   *> code A2
   *>
   *>CACHE
   *>EXECUTE
   *>
   *>aggr out = * / break = code / n = n
   *>fre n
   *>Error # 10963
   *>There is not enough memory for all the cases in the aggregated file.  The
   *>aggregated file is missing some cases.  Rerun with more memory.
   *>Note # 213
   *>Due to an error, INSERT and INCLUDE file processing has been terminated.
   *>All transformations since the last procedure command have been discarded.
   *>Note # 236
   *>All outstanding macros have been terminated, all INSERT and INCLUDE
   *>processing has been terminated, and all outstanding PRESERVE commands have
   *>been undone.

Cheers!!
Albert-Jan

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In the face of ambiguity, refuse the temptation to guess.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

[Lemon, John S.]

Albert-Jan

 

I manage to aggregate files of a similar size, actually I have ones with 20 variables in that work but the difference is that I sort before I do the aggregate and I indicate that the file is pre-sorted on the aggregate command. This seems to work better and more reliably.

 

Best Wishes

 

John S. Lemon

DIT ( Directorate of Information Technology ) - Student Liaison Officer

University of Aberdeen

Edward Wright Building: Room G86a ( Please note new room )

 

Tel:  [phone]

Fax: [phone]

 

Diary ( Free / Busy )

 

From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Albert-Jan Roskam
Sent: 14 December 2009 10:11
To: [hidden email]
Subject: Error #10963 (memory) processing a very large file

 

Hi all,

 

I am trying to aggregate a list with 10**7 cases but I get the error below. The .txt file is a bit over 5Gb and has just 2 columns. I suspect that the syntax causes the computer to run out of temporary disk space. That could be fixed by removing the CACHE-EXE. But since it takes forver to run this syntax (and probably even longer without the cache command), I wanted to ask your advice.

 

Thank you!

 

Albert-Jan

 

 

   INSERT FILE='D:\users\RUN\AJR\TEMP\spssprod.spp' SYNTAX=BATCH ERROR=STOP .
   *>
   *>file handle myfile /name = "<a href="file:///\\blah\bleh\bloh\Temp\largefile.txt">\\blah\bleh\bloh\Temp\largefile.txt"
   *>
   *>
   *>GET DATA  /TYPE = TXT
   *> /FILE = myfile
   *> /DELCASE = LINE
   *> /DELIMITERS = "\t
   *> /ARRANGEMENT = DELIMITED
   *> /FIRSTCASE = 2
   *> /IMPORTCASE = ALL
   *> /VARIABLES
   *> ssn       F9.
   *> code A2
   *>
   *>CACHE
   *>EXECUTE
   *>
   *>aggr out = * / break = code / n = n
   *>fre n
   *>Error # 10963
   *>There is not enough memory for all the cases in the aggregated file.  The
   *>aggregated file is missing some cases.  Rerun with more memory.
   *>Note # 213
   *>Due to an error, INSERT and INCLUDE file processing has been terminated.
   *>All transformations since the last procedure command have been discarded.
   *>Note # 236
   *>All outstanding macros have been terminated, all INSERT and INCLUDE
   *>processing has been terminated, and all outstanding PRESERVE commands have
   *>been undone.

Cheers!!
Albert-Jan

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In the face of ambiguity, refuse the temptation to guess.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 

The University of Aberdeen is a charity registered in Scotland, No SC013683.