WRITE under TEMPORARY gets entangled with READ in MATRIX - a bug?

[Kirill Orlov]

I've come across the following seeming inconsistency.

Let's have some data

data list list /y x.
begin data
10 0
20 1
30 0
40 1
end data.

The syntax below works as I want. It takes just 2nd and 4th cases into MATRIX and and also reads in previously written-out case numbers.

*temporary.
select if x=1.
comp #id= $casenum.
write outfile= 'd:\exercise\id.sps' /#id (f8). /*Please put your path here
matrix.
get y /var= y.
read id /file= 'd:\exercise\id.sps' /field= 1 to 8 /format= a8 /size= nrow(y).
print y /rnames= id.
end matrix.

However, if you enable the TEMPORARY command above (and I need it because I prefer to keep the dataset unaltered in the end), a catastrophic error occurs and the message saying that READ was used without intervening rewind, for the same file. I see this on both SPSS 15 and 20 (and think other versions will give it too).

We know that MATRIX-END MATRIX block acts like a single procedure - it reads data and executes pending transforms. Why does READ statement works as expected (by me) without TEMPORARY and gives the error with TEMPORARY? What is the mechanics behind and wouldn't it better if SPSS fix it even if it is not a bug, because it so much inconvenient?

[Kirill Orlov]

P.S. Yes, I'm aware that WRITE (or PRINT) isn't listed among the commands that Syntax Reference permits to stand under TEMPORARY. But for all that. If you put EXECUTE or other procedure in place of MATRIX block in my syntax, it works perfectly!

[Kirill Orlov]

P.S. Yes, I'm aware that WRITE (or PRINT) isn't listed among the commands that Syntax Reference permits to stand under TEMPORARY. But for all that. If you put EXECUTE or other procedure in place of MATRIX block in my syntax, it works perfectly!

[David Marso]

I suspect the file on WRITE is in contention because as MATRIX is processing it is indeed passing the data but likely the file is not closed at the point of READ access.  Not sure of the exact sequence maybe JoNoH knows with greater detail.
However one quick fix is EXE following WRITE followed by another

temporary.
select if x=1.
I know it is Rubish (as in Goldberg not trash) but best I can think of pre caffeine.
BTW:  Thank you for the precise description of the error, the context, the desired behavior and exact means of reproduction.  I may use your TEMPORARY idea as a way around some things I am currently working on! Thanks!
HTH DMM.
--

Kirill Orlov wrote

I've come across the following seeming inconsistency.

Let's have some data

data list list /y x.
begin data
10 0
20 1
30 0
40 1
end data.

The syntax below works as I want. It takes just 2nd and 4th cases into
MATRIX and and also reads in previously written-out case numbers.

*temporary.
select if x=1.
comp #id= $casenum.
write outfile= 'd:\exercise\id.sps' /#id (f8)./*Please put your path here
matrix.
get y /var= y.
read id /file= 'd:\exercise\id.sps' /field= 1 to 8 /format= a8 /size=
nrow(y).
print y /rnames= id.
end matrix.

However, if you enable the TEMPORARY command above (and I need it
because I prefer to keep the dataset unaltered in the end), a
catastrophic error occurs and the message saying that READ was used
without intervening rewind, for the same file. I see this on both SPSS
15 and 20 (and think other versions will give it too).

We know that MATRIX-END MATRIX block acts like a single procedure - it
reads data and executes pending transforms. Why does READ statement
works as expected (by me) without TEMPORARY and gives the error with
TEMPORARY? What is the mechanics behind and wouldn't it better if SPSS
fix it even if it is not a bug, because it so much inconvenient?

[Jim Marks]

Does the program work as desired if you use DATASET commands to copy your original data and process the reduced file?

new file.
data list list /y x.
begin data
10 0
20 1
30 0
40 1
end data.

dataset name all_data.
dataset copy x1.
dataset activate x1 window = front.

select if x=1.
comp #id= $casenum.
write outfile= 'c:\spss-test\id.sps' /#id (f8). /*Please put your path here .
matrix.
get y /var= y.
read id /file= 'c:\spss-test\id.sps' /field= 1 to 8 /format= a8 /size=
nrow(y).
print y /rnames= id.
end matrix.

dataset activate all_data window = front.


Jim Marks
Sr Market Research Manager
National Market Research
Kaiser Foundation Health Plan of the Mid-Atlantic States, Inc.
2101 E. Jefferson St.
Rockville, MD 20852
Phone: [phone]
Cell Phone: [phone]

NOTICE TO RECIPIENT:  If you are not the intended recipient of this e-mail, you are prohibited from sharing, copying, or otherwise using or disclosing its contents.  If you have received this e-mail in error, please notify the sender immediately by reply e-mail and permanently delete this e-mail and any attachments without reading, forwarding or saving them.  Thank you.




From:        David Marso <[hidden email]>
To:        [hidden email]
Date:        11/21/2012 09:57 AM
Subject:        Re: WRITE under TEMPORARY gets entangled with READ in MATRIX - a              bug?
Sent by:        "SPSSX(r) Discussion" <[hidden email]>

---

I suspect the file on WRITE is in contention because as MATRIX is processing
it is indeed passing the data but likely the file is not closed at the point
of READ access.  Not sure of the exact sequence maybe JoNoH knows with
greater detail.
However one quick fix is EXE following WRITE followed by another

temporary.
select if x=1.
I know it is Rubish (as in Goldberg not trash) but best I can think of pre
caffeine.
BTW:  Thank you for the precise description of the error, the context, the
desired behavior and exact means of reproduction.  I may use your TEMPORARY
idea as a way around some things I am currently working on! Thanks!
HTH DMM.
--

Kirill Orlov wrote
> I've come across the following seeming inconsistency.
>
> Let's have some data
>
> data list list /y x.
> begin data
> 10 0
> 20 1
> 30 0
> 40 1
> end data.
>
> The syntax below works as I want. It takes just 2nd and 4th cases into
> MATRIX and and also reads in previously written-out case numbers.
>
> *temporary.
> select if x=1.
> comp #id= $casenum.
> write outfile= 'd:\exercise\id.sps' /#id (f8)./*Please put your path here
> matrix.
> get y /var= y.
> read id /file= 'd:\exercise\id.sps' /field= 1 to 8 /format= a8 /size=
> nrow(y).
> print y /rnames= id.
> end matrix.
>
> However, if you enable the TEMPORARY command above (and I need it
> because I prefer to keep the dataset unaltered in the end), a
> catastrophic error occurs and the message saying that READ was used
> without intervening rewind, for the same file. I see this on both SPSS
> 15 and 20 (and think other versions will give it too).
>
> We know that MATRIX-END MATRIX block acts like a single procedure - it
> reads data and executes pending transforms. Why does READ statement
> works as expected (by me) without TEMPORARY and gives the error with
> TEMPORARY? What is the mechanics behind and wouldn't it better if SPSS
> fix it even if it is not a bug, because it so much inconvenient?





-----
Please reply to the list and not to my personal email.
Those desiring my consulting or training services please feel free to email me.
--
View this message in context: http://spssx-discussion.1045642.n5.nabble.com/WRITE-under-TEMPORARY-gets-entangled-with-READ-in-MATRIX-a-bug-tp5716385p5716388.html
Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Kirill Orlov]

Jim,
Thank you, really. Your offer does work.
Yet, it is not was I was searching for. Your solution engages in copying the dataset and then applying to it my code, without TEMPORARY. It is a way to work around. But I'd like to know WHY my code with TEMPORARY produces an error.

And I'd prefer to use TEMPORARY and to not copy the dataset - a clumsy way out.

[David Marso]

OTOH:  The following achieves the same end without the hassle of writing to an external file.
It does not alter the file or require a dataset copy.
--
data list list /y x.
begin data
10 0
20 1
30 0
40 1
end data.


TEMPORARY.
STRING id (A8).
COMPUTE id=STRING($CASENUM,F8).

SELECT if x=1.
matrix.
get y /var= y.
get id /var id.
print y /rnames= id.
end matrix.

[Jon K Peck]

Well, obviously the system should not crash, but there is an ambiguous situation here.  TEMPORARY is terminated by any command that causes a data pass.  WRITE is not supported for TEMPORARY, but it also does not cause a data pass.

However, MATRIX may or may not pass the active dataset.  In this case it does.
Apparently MATRIX is classified as a data passing procedure regardless of whether it actually passes the active data.  In fact, the results of an immediately preceding compute are actually available to MATRIX, but any scratch variables are gone.  Scratch variables are not available to procedures.  Therefore, the MATRIX GET should be synchronized with the WRITE transformation or the transformation block should not be piggybacked on the MATRIX call but should generate an extra data pass.  If you remove the GET, however, the problem still occurs.

Using a dataset would make no difference, since datasets are actually files, but WRITE does not support datasets anyway, since a dataset is a temporary file.

So this needs some thought.


Jon Peck (no "h") aka Kim
Senior Software Engineer, IBM
[hidden email]
new phone: [phone]




From:        Kirill Orlov <[hidden email]>
To:        [hidden email],
Date:        11/21/2012 07:02 AM
Subject:        [SPSSX-L] WRITE under TEMPORARY gets entangled with READ in              MATRIX - a bug?
Sent by:        "SPSSX(r) Discussion" <[hidden email]>

---

I've come across the following seeming inconsistency.

Let's have some data

data list list /y x.
begin data
10 0
20 1
30 0
40 1
end data.

The syntax below works as I want. It takes just 2nd and 4th cases into MATRIX and and also reads in previously written-out case numbers.

*temporary.
select if x=1.
comp #id= $casenum.
write outfile= 'd:\exercise\id.sps' /#id (f8). /*Please put your path here
matrix.
get y /var= y.
read id /file= 'd:\exercise\id.sps' /field= 1 to 8 /format= a8 /size= nrow(y).
print y /rnames= id.
end matrix.

However, if you enable the TEMPORARY command above (and I need it because I prefer to keep the dataset unaltered in the end), a catastrophic error occurs and the message saying that READ was used without intervening rewind, for the same file. I see this on both SPSS 15 and 20 (and think other versions will give it too).

We know that MATRIX-END MATRIX block acts like a single procedure - it reads data and executes pending transforms. Why does READ statement works as expected (by me) without TEMPORARY and gives the error with TEMPORARY? What is the mechanics behind and wouldn't it better if SPSS fix it even if it is not a bug, because it so much inconvenient?

[Rich Ulrich]

I think you answered the question -- WRITE is not a "procedure"
that forces a file-read, or it would be accepted under TEMPORARY.

Instead, it sets a flag.  I remember when this was the status of LIST.
(If I recall correctly, LIST was changed to become a Procedure that
forces a read, many versions ago.)

As you say, putting in an EXECUTE or any real procedure makes
the TEMPORARY work correctly.  I suppose the question is whether
the documentation for WRITE should be more explicit than simply
omitting it from the list concerning TEMPORARY, or if the behavior
should be changed.  But it does seem to "work as documented."

--
Rich Ulrich

---

Date: Wed, 21 Nov 2012 18:30:07 +0400
From: [hidden email]
Subject: Re: WRITE under TEMPORARY gets entangled with READ in MATRIX - a bug?
To: [hidden email]

P.S. Yes, I'm aware that WRITE (or PRINT) isn't listed among the commands that Syntax Reference permits to stand under TEMPORARY. But for all that. If you put EXECUTE or other procedure in place of MATRIX block in my syntax, it works perfectly!

[Rick Oliver-3]

WRITE can appear within a block of temporary commands without ending the temporary block (since it doesn't read the data), but there's nothing temporary about the result of the WRITE command. The same is true for XSAVE.

data list free /x.
begin data
1 2 3
end data.
temporary.
compute y=2.
write outfile='c:\temp\temporary.txt' /x y.
xsave outfile='c:\temp\temporary.sav' /keep y.
list. /*this is the first procedure that reads the data; so y still exists.
list.  /*y is gone from the active file but not from the file written by WRITE or XSAVE.



Rick Oliver
Senior Information Developer
IBM Business Analytics (SPSS)
E-mail: [hidden email]



From:        Rich Ulrich <[hidden email]>
To:        [hidden email],
Date:        11/21/2012 12:08 PM
Subject:        Re: WRITE under TEMPORARY gets entangled with READ in MATRIX - a              bug?
Sent by:        "SPSSX(r) Discussion" <[hidden email]>

---

I think you answered the question -- WRITE is not a "procedure"
that forces a file-read, or it would be accepted under TEMPORARY.

Instead, it sets a flag.  I remember when this was the status of LIST.
(If I recall correctly, LIST was changed to become a Procedure that
forces a read, many versions ago.)

As you say, putting in an EXECUTE or any real procedure makes
the TEMPORARY work correctly.  I suppose the question is whether
the documentation for WRITE should be more explicit than simply
omitting it from the list concerning TEMPORARY, or if the behavior
should be changed.  But it does seem to "work as documented."

--
Rich Ulrich

---

Date: Wed, 21 Nov 2012 18:30:07 +0400
From: [hidden email]
Subject: Re: WRITE under TEMPORARY gets entangled with READ in MATRIX - a bug?
To: [hidden email]

P.S. Yes, I'm aware that WRITE (or PRINT) isn't listed among the commands that Syntax Reference permits to stand under TEMPORARY. But for all that. If you put EXECUTE or other procedure in place of MATRIX block in my syntax, it works perfectly!

[Bruce Weaver]

That's a nice little demo.  Thanks Rick.

Rick Oliver wrote

WRITE can appear within a block of temporary commands without ending the
temporary block (since it doesn't read the data), but there's nothing
temporary about the result of the WRITE command. The same is true for
XSAVE.

data list free /x.
begin data
1 2 3
end data.
temporary.
compute y=2.
write outfile='c:\temp\temporary.txt' /x y.
xsave outfile='c:\temp\temporary.sav' /keep y.
list. /*this is the first procedure that reads the data; so y still
exists.
list.  /*y is gone from the active file but not from the file written by
WRITE or XSAVE.



Rick Oliver
Senior Information Developer
IBM Business Analytics (SPSS)
E-mail: [hidden email]



From:   Rich Ulrich <[hidden email]>
To:     [hidden email],
Date:   11/21/2012 12:08 PM
Subject:        Re: WRITE under TEMPORARY gets entangled with READ in
MATRIX - a              bug?
Sent by:        "SPSSX(r) Discussion" <[hidden email]>



I think you answered the question -- WRITE is not a "procedure"
that forces a file-read, or it would be accepted under TEMPORARY.

Instead, it sets a flag.  I remember when this was the status of LIST.
(If I recall correctly, LIST was changed to become a Procedure that
forces a read, many versions ago.)

As you say, putting in an EXECUTE or any real procedure makes
the TEMPORARY work correctly.  I suppose the question is whether
the documentation for WRITE should be more explicit than simply
omitting it from the list concerning TEMPORARY, or if the behavior
should be changed.  But it does seem to "work as documented."

--
Rich Ulrich

Date: Wed, 21 Nov 2012 18:30:07 +0400
From: [hidden email]
Subject: Re: WRITE under TEMPORARY gets entangled with READ in MATRIX - a
bug?
To: [hidden email]

P.S. Yes, I'm aware that WRITE (or PRINT) isn't listed among the commands
that Syntax Reference permits to stand under TEMPORARY. But for all that.
If you put EXECUTE or other procedure in place of MATRIX block in my
syntax, it works perfectly!

[Kirill Orlov]

Colleagues, thank you all for your answers that came so far.

As Rick demonstrates it (and what I knew myself and exploited it from time to time), WRITE or PRINT do effectively work under TEMPORARY (they are like transformation commands, only that their output isn't temporary of course). So, the question remains: why does it go smoothly if the procedure at the end is EXECUTE or LIST or other reading data command, but it produces an error with MATRIX, in particularly, with READ inside MATRIX.

As Jon has admitted, a problem is really there and must be inspected/overhauled. I'd like to encourage SPSS team to do it. As I've noted, even if it is not a bug it'd be better to do something about it in future releases because the current behaviour is bizarre.

David, your solution is good. The inconvenience however is that if TEMPORARY is disabled (and I might want to switch between using  and not using it), your code leaves permanent variable ID in the dataset. And I don't want new variables to appear in it (my specific setting is such that I won't be able to return to the dataset after END MATRIX, to delete the created variable). That's why I preferred in my code to create scratch variable #ID, not ID. But scratch variable cannot be opened by MATRIX GET - it is killed by MATRIX because MATRIX is a procedure. Hence was my idea to use WRITE (it processes scratches) and READ it then within MATRIX. This trick worked as expected without TEMPORARY, but unfortunately caused crash with TEMPORARY.

****************************************************

Rick Oliver:

WRITE can appear within a block of temporary commands without ending the temporary block (since it doesn't read the data), but there's nothing temporary about the result of the WRITE command. The same is true for XSAVE.

data list free /x.
begin data
1 2 3
end data.
temporary.
compute y=2.
write outfile='c:\temp\temporary.txt' /x y.
xsave outfile='c:\temp\temporary.sav' /keep y.
list. /*this is the first procedure that reads the data; so y still exists.
list.  /*y is gone from the active file but not from the file written by WRITE or XSAVE.

*******************************************************
Jon Peck:

Well, obviously the system should not crash, but there is an ambiguous situation here.  TEMPORARY is terminated by any command that causes a data pass.  WRITE is not supported for TEMPORARY, but it also does not cause a data pass.

However, MATRIX may or may not pass the active dataset.  In this case it does.
Apparently MATRIX is classified as a data passing procedure regardless of whether it actually passes the active data.  In fact, the results of an immediately preceding compute are actually available to MATRIX, but any scratch variables are gone.  Scratch variables are not available to procedures.  Therefore, the MATRIX GET should be synchronized with the WRITE transformation or the transformation block should not be piggybacked on the MATRIX call but should generate an extra data pass.  If you remove the GET, however, the problem still occurs.

Using a dataset would make no difference, since datasets are actually files, but WRITE does not support datasets anyway, since a dataset is a temporary file.

So this needs some thought.

********************************************************************
David Marso:

data list list /y x.
begin data
10 0
20 1
30 0
40 1
end data.

TEMPORARY.
STRING id (A8).
COMPUTE id=STRING($CASENUM,F8).

SELECT if x=1.
matrix.
get y /var= y.
get id /var id.
print y /rnames= id.
end matrix.