sort space error but i have adequate space

[devoidx]

Hi guys , sorry that I keep posting about my problems but this database collection that I have is driving me crazy..so I am trying to sort a 350gb database and I have set my external hard drive as the tempdir which has 2.7TB free space and after 2 hours  of running sort I got the error that

 file write error: file name e:/ bla blah not enough space (DATA1002)

how can this be possible when I have 6-7 times more free space than the size of the database?

Thanks

[Albert-Jan Roskam]

----- Original Message -----

> From: devoidx <[hidden email]>
> To: [hidden email]
> Cc:
> Sent: Wednesday, September 25, 2013 1:45 AM
> Subject: [SPSSX-L] sort space error but i have adequate space
>
> Hi guys , sorry that I keep posting about my problems but this database
> collection that I have is driving me crazy..so I am trying to sort a 350gb
> database and I have set my external hard drive as the tempdir which has
> 2.7TB free space and after 2 hours  of running sort I got the error that
>
> file write error: file name e:/ bla blah not enough space (DATA1002)
>
> how can this be possible when I have 6-7 times more free space than the size
> of the database?

Hi, be sure to clean the spss tempdir manually after SPSS has crashed. I happened to me before that there were lots of files (usually lots of small ones, but they may be large) in that dir.
A couple of cache files could eat up a lot of space!

Albert-Jan

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[devoidx]

Thanks, I have been watching out for that, but my external drive is clean as a whistle with 2.7TB free...Iv run the sort 5-6 times and every time i get the space issue...when I get the message my external hard drive has about 2.45TB free space so it cant be the actual limitation ..I can't figure out what is triggering spss to think that there is no space in the tempdir when there is plenty!

[Lemon, John S.]

I wonder whether you need to actually assign the system tempdir as well as the SPSS one to the external drive using good old DOS commands - I remember having to do that once before when doing some large sorts.


Best Wishes

John S. Lemon
IT Services - Student Liaison Officer
University of Aberdeen
Edward Wright Building
Tel:  [phone]

DIT news for Students

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of devoidx
Sent: 25 September 2013 13:53
To: [hidden email]
Subject: Re: sort space error but i have adequate space

Thanks, I have been watching out for that, but my external drive is clean as a whistle with 2.7TB free...Iv run the sort 5-6 times and every time i get the space issue...when I get the message my external hard drive has about 2.45TB free space so it cant be the actual limitation ..I can't figure out what is triggering spss to think that there is no space in the tempdir when there is plenty!



--
View this message in context: http://spssx-discussion.1045642.n5.nabble.com/sort-space-error-but-i-have-adequate-space-tp5722203p5722212.html
Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD




The University of Aberdeen is a charity registered in Scotland, No SC013683.

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[devoidx]

Interesting, thanks, but if the issue would be that spss is writing on to system tempdir then I should see my computer's hard rive filling up which I don't ..definitely a mystery..but ill see if I can try changing the system's temp directory to my external

[Albert-Jan Roskam]

----- Original Message -----

> From: devoidx <[hidden email]>
> To: [hidden email]
> Cc:
> Sent: Wednesday, September 25, 2013 3:55 PM
> Subject: Re: [SPSSX-L] sort space error but i have adequate space
>
> Interesting, thanks, but if the issue would be that spss is writing on to
> system tempdir then I should see my computer's hard rive filling up which I
> don't ..definitely a mystery..but ill see if I can try changing the
> system's
> temp directory to my external

you can set it programmatically like this. Makes your code portable/transferable.

file handle tempdir /name = "\\path\to\alternative\tempdir\on\your\external\drive".
* file handle tempdir /name = "%temp%". /* the default.
begin program.
import SpssClient, spssaux, os
SpssClient.StartClient()
option = SpssClient.PreferenceOptions.TempDir
value = spssaux.FileHandles().resolve("tempdir")
msg = "path must exist and with R/W rights"
assert os.access(value, os.O_RDWR) , msg
SpssClient.SetPreference(option, value)
end program.

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[devoidx]

I just don't understand...I get the error at the time of writing the exact same temporary file every time .."merge.106" that I don't have enough space while i have about 2.5 TB free space and strangely whenever this happens I also get a socketinput error from a torrent application I have which isn't even running....its crazy

[David Marso]

Perhaps at this point consider slicing the file up using XSAVE (within a DO IF block).
Follow with EXECUTE.
then combine using ADD FILES with ID on the BY subcommand.

Perhaps the TEMPDIR is NOT getting set as you presume?
Perhaps the 350G file is COMPRESSED so when it is uncompressed it will be QUITE A LOT LARGER than you believe.
Previously you noted that the thing got hosed on a system missing value.
Why not SELECT IF NOT (SYSMIS(ID)) on both files prior to the merge?
You don't seriously think that these SYSMIS values will be useful in a merge?
I would say you have MUCH MUCH more serious issues than this if your DATABASE has NULLS in a PRIMARY KEY.
---

devoidx wrote

I just don't understand...I get the error at the time of writing the exact same temporary file every time .."merge.106" that I don't have enough space while i have about 2.5 TB free space and strangely whenever this happens I also get a socketinput error from a torrent application I have which isn't even running....its crazy

[devoidx]

The reason that I think I have system missing on my ID variable is that because the database was originally a SAS file with I converted to SPSS using stat/transfer so probably user missing values in SAS got converted to system missing. That actually is the least of my problems because as you said i can just do a select if not, which I did but apparently there are other out of sort values so either way the database needs to be resorted ..which brings me to my current problem....

Tempdir seems to be working as i set it since my external hard drive does get filled up while sort is running and my hard drive is intact .... and when I get the error i still have 2.5 TB free on the external which is what that baffles me

I might have to try your xsave approach and see what happens

[devoidx]

Also can you give me a quick example of putting the xsave into DO IF to break up my database? I think easiest way to break up the database is to divide it based on gender ...something like:

Do if (gender EQ 1).
XSAVE OUTFILE='C:\male.SAV'.
ELSE .
XSAVE OUTFILE='C:\female.SAV'.
END IF.
EXECUTE.

?

[David Marso]

Something like:

DO IF RANGE($CASENUM,1,1000000000).
XSAVE OUTFILE 'F01.sav'.
ELSE IF RANGE(1000000001,2000000000).
XSAVE OUTFILE 'F02.SAV'.
ELSE IF ......
........etc
END IF.

EXECUTE.

GET FILE 'F01.sav'.
SORT CASES BY ID .
DATASET Name File01.
....
....
ADD FILES / FILE File01 / FILE = ........./ BY ID .

devoidx wrote

Also can you give me a quick example of putting the xsave into DO IF to break up my database? I think easiest way to break up the database is to divide it based on gender ...something like:

Do if (gender EQ 1).
XSAVE OUTFILE='C:\male.SAV'.
ELSE .
XSAVE OUTFILE='C:\female.SAV'.
END IF.
EXECUTE.

?

[Rich Ulrich]

Note careless omission of $CASENUM in second RANGE( ) ....

> Date: Wed, 25 Sep 2013 12:20:49 -0700
> From: [hidden email]
> Subject: Re: sort space error but i have adequate space
> To: [hidden email]
>
> Something like:
>
> DO IF RANGE($CASENUM,1,1000000000).
> XSAVE OUTFILE 'F01.sav'.
> ELSE IF RANGE(1000000001,2000000000).
    -->   RANGE($CASENUM,1000000001,2000000000)

> XSAVE OUTFILE 'F02.SAV'.
> ELSE IF ......
> ........etc
> END IF.
>
> *EXECUTE.*
>
> GET FILE 'F01.sav'.
> SORT CASES BY ID .
> DATASET Name File01.
> ....
> ....
> ADD FILES / FILE File01 / FILE = ........./ *BY ID *.
>
>

--
Rich Ulrich

>
> devoidx wrote
> > Also can you give me a quick example of putting the xsave into DO IF to
> > break up my database? I think easiest way to break up the database is to
> > divide it based on gender ...something like:
> >
> > Do if (gender EQ 1).
> > XSAVE OUTFILE='C:\male.SAV'.
> > ELSE .
> > XSAVE OUTFILE='C:\female.SAV'.
> > END IF.
> > EXECUTE.
> >
> > ?
>

[devoidx]

So i divided my 250gb database to 3 parts...and I still cant sort it ...80gb dataset on a 2.7TB external hardware and still tells me not enough space eventhough the hardrive never gets even close to being full...this cant really be an actual space problem..I don't understand what the problem is here

Error I get:
>Error.  Command name: SORT CASES BY
>File read error: file name E:\temp\spss976\merge.82: Not enough space (DATA1003)
>Execution of this command stops.

I used to kept getting write error (DATA1002) now this time it was a read error... DATA1003..not sure if there is any signifcane

[devoidx]

Well I never found out what the space issue was but I finally found a work around...apparently each of my huge  databases were a merge of two other databases which were sorted but were added ontop of eachother instead of being added with /by ID...so I have divided each huge database to its original two databases which are sorted and going to re-add them together with /by ID value to get one humongous sorted database :D

Thanks everyone for all the help..the xsave definitely is making my life easier! im sure ill have more questions coming..its a learning process :)

[devoidx]

meh I spoke to soon...I cant even run a xsave on my database to divide it into smaller ones without getting the exact same "not enough space" error and I tried 2 external hard drives as my output location with same results...looks to me that spss is just simply unable to handle large datasets ....pretty disappointing

[David Marso]

SPSS can handle HUGE files without a problem.  You DO NEED TO HAVE non tinker toy hardware!!!
Sounds like you don't have as much space as you believe and/or the files are MUCH larger than you think.
You did NOT address the point I made earlier regarding compression.
If you review the FM re XSAVE you will see it supports both COMPRESSED and ZCOMPRESSED.  
Perhaps consideration of those options (ZCOMPRESSED in particular) might get you over the coals.
How many variables in the file?  Do you NEED ALL of them for this piece?
Are there LONG strings?  Maybe they can be made shorter?

devoidx wrote

meh I spoke to soon...I cant even run a xsave on my database to divide it into smaller ones without getting the exact same "not enough space" error and I tried 2 external hard drives as my output location with same results...looks to me that spss is just simply unable to handle large datasets ....pretty disappointing

[Albert-Jan Roskam]

----- Original Message -----

> From: David Marso <[hidden email]>
> To: [hidden email]
> Cc:
> Sent: Friday, September 27, 2013 5:48 PM
> Subject: Re: [SPSSX-L] sort space error but i have adequate space
>
> SPSS can handle HUGE files without a problem.  You DO NEED TO HAVE non tinker
> toy hardware!!!
> Sounds like you don't have as much space as you believe and/or the files are
> MUCH larger than you think.
> You did NOT address the point I made earlier regarding compression.
> If you review the FM re XSAVE you will see it supports both COMPRESSED and
> ZCOMPRESSED.

Zcompressed is only available from SPSS v21 and up.

> Perhaps consideration of those options (ZCOMPRESSED in particular) might get
> you over the coals.

But earlier in this thread you mentioned that cache files are not (as) compressed. Perhaps changing the SET CACHE value will also help. And *not* using the CACHE command (but saving it to a real file as the first data pass) also sometimes prevents the tempdir from getting too full.

> How many variables in the file?  Do you NEED ALL of them for this piece?
> Are there LONG strings?  Maybe they can be made shorter?

Yes, ditch as many variables as you can! But for strings, trailing spaces are compressed, so I wonder how much this will help.

> devoidx wrote
>> meh I spoke to soon...I cant even run a xsave on my database to divide it
>> into smaller ones without getting the exact same "not enough
> space" error
>> and I tried 2 external hard drives as my output location with same
>> results...looks to me that spss is just simply unable to handle large
>> datasets ....pretty disappointing
>
>
>
>
>
> -----
> Please reply to the list and not to my personal email.
> Those desiring my consulting or training services please feel free to email me.
> ---
> "Nolite dare sanctum canibus neque mittatis margaritas vestras ante porcos
> ne forte conculcent eas pedibus suis."
> Cum es damnatorum possederunt porcos iens ut salire off sanguinum cliff in
> abyssum?"
> --
> View this message in context:
> http://spssx-discussion.1045642.n5.nabble.com/sort-space-error-but-i-have-adequate-space-tp5722203p5722282.html
> Sent from the SPSSX Discussion mailing list archive at Nabble.com.
>
> =====================
> To manage your subscription to SPSSX-L, send a message to
> [hidden email] (not to SPSSX-L), with no body text except the
> command. To leave the list, send the command
> SIGNOFF SPSSX-L
> For a list of commands to manage subscriptions, send the command
> INFO REFCARD
>

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[devoidx]

What I'v been trying to explain that you guys keep skipping over is that my hard drive never ever even comes  close to being full when I get the error....When I get the error, my external hard drive has about 2.4
TB free space ...so how can it be a space issue? and I have tested out my external hard drive by transferring 2.6 TB into it which it will accept easily...so I am certainly not delusional about the amount of free space I have....

And no matter how much compressed my 250GB dataset is, 2.7 TB free space should be enough for it to at least do the xsave (and sort too),,,

xsave itself saves in a compressed form by default so the end end result should be two other datasets that add up to the original 250GB and there are not massive temporary files created for xsave either...

There must be something else that triggers spss to say I don't have enough space ...

as a FYI, I have about 90 variables and 700 million cases. And as iv said before i even split my database into a 80GB chunk and it still couldn't sort it....i highly doubt a 80gb compressed database magically decompresses to more than 2.7TB

[JKRockStomper]

Try pointing all your temp directories over to the 2.4TB drive.  One of the problems with SPSS during a sort is that it creates multiple temp files and they will total a large size.

[devoidx]

Thanks, I have pointed the main temp directory to the external hardrive through SPSS's edit>option>file locations......any other temp directories that I am not aware of?