Re: loop and do repeat problem with thousands of unique values to insert

Posted by David Marso on
URL: http://spssx-discussion.165.s1.nabble.com/loop-and-do-repeat-problem-with-thousands-of-unique-values-to-insert-tp4268902p4269480.html

"True, sorting is nasty in SPSS "
Especially if you are sorting the entire file 14 times.  The nasty comes in from dragging everything else along in the sort.  In my example you are only sorting on the strvar and carrying along the original ID.
If you already have an ID variable in your file (recommended best practices IMNSHO) you can skip a few expensive steps.
The normalized version (lean-mean-long data structure) is going to run faster than 14 sorts of the entire file (I assume you have much more than just these string variables in your file).

"now you mention normalizing: the variable later in line (name14) the
more blanks there are."
Please complete this last thought...  If you have many blanks then SELECT IF strvar NE " " prior to the sort of the long file.  Any blanks will be unmapped and you cam recode the sysmis later across the 14 vars.
OTOH, varstocases has a NULL clause which might be useful.
HTH, David

Maurice Vergeer wrote

Dear David,

thanks for your suggestion you provided in the earlier email. I need
to let it sink in what really happens in the example.
True, sorting is nasty in SPSS

now you mention normalizing: the variable later in line (name14) the
more blanks there are.


thanks
Maurice


On Tue, Mar 29, 2011 at 22:47, David Marso <[e-mail]> wrote:
> Best bet is these cases is to write syntax once against a properly normalized
> data structure (in this a sorted case long format). � Do the necessary merge
> and then mop up with an aggregate and a final match. � see my example.
> Sorting the huge file multiple times with all of the variables in the file
> in place is going to kill your hard disk. � Trust me, I've been doing this
> stuff for over 20 years!
>
>
> --

=====================
To manage your subscription to SPSSX-L, send a message to
[e-mail] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD