The agony and the ecstacy: Development of complex MATRIX programs.
Locked
12
12
The agony and the ecstacy: Development of complex MATRIX programs.
 
Administrator
|
This is not so much a question as much as a general comment and I am interested in other people's experiences.
Context: SPSS has had an embedded MATRIX language for over 20 years. For those with the necessary mathematical background and special needs it is a wonderful tool. However: The documentation SUCKS! Error handling is NON EXISTENT so debugging anything more than trivial programs is obscenely difficult. There have been absolutely NO enhancements in the past 20 years. There is no concept of modularity (no support for subroutines/local variables). I would like to see IBM/SPSS development open up the can of worms called MATRIX and reexamine its potential for developers to create specialized solutions. JoNoH... Please don't tell me to use Python. That would be passing the buck. Don't tell me to use R " "... Compared to SAS IML SPSS MATRIX is a crippled dwarf about 10 years behind the game. ----
Please reply to the list and not to my personal email.
Those desiring my consulting or training services please feel free to email me. --- "Nolite dare sanctum canibus neque mittatis margaritas vestras ante porcos ne forte conculcent eas pedibus suis." Cum es damnatorum possederunt porcos iens ut salire off sanguinum cliff in abyssum?" |
Re: The agony and the ecstasy: Development of complex MATRIX programs.
|
Well, MATRIX could use a lot of enhancements,
but I will tell you that using numpy via Python programmability is a thing
of beauty, and it has all the error handling, modularity, expressive power,
and functionality that any user of such a procedure could want. Here's
the top level of the reference manual:
http://docs.scipy.org/doc/numpy/reference/index.html#reference No doubt, the current MATRIX procedure, while powerful, is lacking in a number of ways, but modernizing it would be a major effort - pretty much a rewrite from scratch. And I am sure that a rewrite would look quite different from the current syntax, so it would have to be a different procedure. When there are free, readily available, and efficient add-ins such as numpy, is this really where IBM/SPSS should be spending its scare development resources? It's not a matter of passing the buck: it's an issue of making the best use of SPSS resources. My unofficial two cents worth. Jon Peck (no "h") aka Kim Senior Software Engineer, IBM [hidden email] new phone: 720-342-5621 From: David Marso <[hidden email]> To: [hidden email], Date: 12/10/2012 07:40 AM Subject: [SPSSX-L] The agony and the ecstacy: Development of complex MATRIX programs. Sent by: "SPSSX(r) Discussion" <[hidden email]> This is not so much a question as much as a general comment and I am interested in other people's experiences. Context: SPSS has had an embedded MATRIX language for over 20 years. For those with the necessary mathematical background and special needs it is a wonderful tool. However: The documentation SUCKS! Error handling is NON EXISTENT so debugging anything more than trivial programs is obscenely difficult. There have been absolutely NO enhancements in the past 20 years. There is no concept of modularity (no support for subroutines/local variables). I would like to see IBM/SPSS development open up the can of worms called MATRIX and reexamine its potential for developers to create specialized solutions. JoNoH... Please don't tell me to use Python. That would be passing the buck. Don't tell me to use R " "... Compared to SAS IML SPSS MATRIX is a crippled dwarf about 10 years behind the game. ---- ----- Please reply to the list and not to my personal email. Those desiring my consulting or training services please feel free to email me. -- View this message in context: http://spssx-discussion.1045642.n5.nabble.com/The-agony-and-the-ecstacy-Development-of-complex-MATRIX-programs-tp5716810.html Sent from the SPSSX Discussion mailing list archive at Nabble.com. ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD |
Re: The agony and the ecstasy: Development of complex MATRIX programs.
|
Administrator
|
Translation:
IBS/SPSS could care less about advanced users needs or keeping up with the Joneses in that respect and everyone who wishes a modern programming environment for designing/implementing bleeding edge statistical analytical techniques should learn Python and require the consumers of such developments to install and maintain python. Since only about 1 percent of SPSS users could care less about such capabilities such considerations shall be deferred for the foreseeable future ;-) --- Maybe a middle ground? Place certain useful functionality such as various useful COMPUTE (CDF, IDF,PDF, string manipulations, date functions etc) into a callable library and open up MATRIX to be able to call these functions. That would NOT require a complete rewrite.
Please reply to the list and not to my personal email.
Those desiring my consulting or training services please feel free to email me. --- "Nolite dare sanctum canibus neque mittatis margaritas vestras ante porcos ne forte conculcent eas pedibus suis." Cum es damnatorum possederunt porcos iens ut salire off sanguinum cliff in abyssum?" |
Re: The agony and the ecstasy: Development of complex MATRIX programs.
|
|
I disagree completely with Marso's so-called
"translation".
Jon Peck (no "h") aka Kim Senior Software Engineer, IBM [hidden email] new phone: 720-342-5621 From: David Marso <[hidden email]> To: [hidden email], Date: 12/10/2012 09:03 AM Subject: Re: [SPSSX-L] The agony and the ecstasy: Development of complex MATRIX programs. Sent by: "SPSSX(r) Discussion" <[hidden email]> Translation: IBS/SPSS could care less about advanced users needs or keeping up with the Joneses in that respect and everyone who wishes a modern programming environment for designing/implementing bleeding edge statistical analytical techniques should learn Python and require the consumers of such developments to install and maintain python. Since only about 1 percent of SPSS users could care less about such capabilities such considerations shall be deferred for the foreseeable future ;-) --- Maybe a middle ground? Place certain useful functionality such as various useful COMPUTE (CDF, IDF,PDF, string manipulations, date functions etc) into a callable library and open up MATRIX to be able to call these functions. That would NOT require a complete rewrite. Jon K Peck wrote > Well, MATRIX could use a lot of enhancements, but I will tell you that > using numpy via Python programmability is a thing of beauty, and it has > all the error handling, modularity, expressive power, and functionality > that any user of such a procedure could want. Here's the top level of the > reference manual: > http://docs.scipy.org/doc/numpy/reference/index.html#reference > > No doubt, the current MATRIX procedure, while powerful, is lacking in a > number of ways, but modernizing it would be a major effort - pretty much a > rewrite from scratch. And I am sure that a rewrite would look quite > different from the current syntax, so it would have to be a different > procedure. When there are free, readily available, and efficient add-ins > such as numpy, is this really where IBM/SPSS should be spending its scare > development resources? It's not a matter of passing the buck: it's an > issue of making the best use of SPSS resources. > > My unofficial two cents worth. > > Jon Peck (no "h") aka Kim > Senior Software Engineer, IBM > peck@.ibm > new phone: 720-342-5621 > > > > > From: David Marso < > david.marso@ > > > To: > SPSSX-L@.uga > , > Date: 12/10/2012 07:40 AM > Subject: [SPSSX-L] The agony and the ecstacy: Development of > complex MATRIX programs. > Sent by: "SPSSX(r) Discussion" < > SPSSX-L@.uga > > > > > > This is not so much a question as much as a general comment and I am > interested in other people's experiences. > Context: > SPSS has had an embedded MATRIX language for over 20 years. > For those with the necessary mathematical background and special needs it > is > a wonderful tool. > However: > The documentation SUCKS! > Error handling is NON EXISTENT so debugging anything more than trivial > programs is obscenely difficult. > There have been absolutely NO enhancements in the past 20 years. > There is no concept of modularity (no support for subroutines/local > variables). > I would like to see IBM/SPSS development open up the can of worms called > MATRIX and reexamine its potential for developers to create specialized > solutions. > JoNoH... Please don't tell me to use Python. That would be passing the > buck. > Don't tell me to use R " "... > Compared to SAS IML SPSS MATRIX is a crippled dwarf about 10 years behind > the game. > ---- > > > > > > > ----- > Please reply to the list and not to my personal email. > Those desiring my consulting or training services please feel free to > email me. > -- > View this message in context: > http://spssx-discussion.1045642.n5.nabble.com/The-agony-and-the-ecstacy-Development-of-complex-MATRIX-programs-tp5716810.html > > Sent from the SPSSX Discussion mailing list archive at Nabble.com. > > ===================== > To manage your subscription to SPSSX-L, send a message to > LISTSERV@.UGA > (not to SPSSX-L), with no body text except the > command. To leave the list, send the command > SIGNOFF SPSSX-L > For a list of commands to manage subscriptions, send the command > INFO REFCARD ----- Please reply to the list and not to my personal email. Those desiring my consulting or training services please feel free to email me. -- View this message in context: http://spssx-discussion.1045642.n5.nabble.com/The-agony-and-the-ecstacy-Development-of-complex-MATRIX-programs-tp5716810p5716819.html Sent from the SPSSX Discussion mailing list archive at Nabble.com. ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD |
Re: The agony and the ecstacy: Development of complex MATRIX programs.
|
In reply to this post by David Marso
David you are smart. I support you fully with regard to MATRIX. I
take your side and not Jon's, this time.
More generally, SPSS Inc seem to have an annoying habit to drift new, discordly bedded features over old functionality. They shun the delicate and noble work to reconsider and enhance older procedures. The package has thus become a bloated layered snowdrift. Python and other Extensions give great power - who argues? - but using these totally alien languages is esthetically sickening. Now that SPSS had brought them in as they wanted it - they should feel pleased. And stop and look back. Today time is coming to turn back to native SPSS devices such as MATRIX and macros and to upgrade them cleverly. 10.12.2012 18:39, David Marso пишет:
This is not so much a question as much as a general comment and I am interested in other people's experiences. Context: SPSS has had an embedded MATRIX language for over 20 years. For those with the necessary mathematical background and special needs it is a wonderful tool. However: The documentation SUCKS! Error handling is NON EXISTENT so debugging anything more than trivial programs is obscenely difficult. There have been absolutely NO enhancements in the past 20 years. There is no concept of modularity (no support for subroutines/local variables). I would like to see IBM/SPSS development open up the can of worms called MATRIX and reexamine its potential for developers to create specialized solutions. JoNoH... Please don't tell me to use Python. That would be passing the buck. Don't tell me to use R " "... Compared to SAS IML SPSS MATRIX is a crippled dwarf about 10 years behind the game. ---- ----- Please reply to the list and not to my personal email. Those desiring my consulting or training services please feel free to email me. -- View this message in context: http://spssx-discussion.1045642.n5.nabble.com/The-agony-and-the-ecstacy-Development-of-complex-MATRIX-programs-tp5716810.html Sent from the SPSSX Discussion mailing list archive at Nabble.com. ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD |
|
|
In reply to this post by David Marso
David,
It doesn't seem like many people are responding, but I want to put in my two cent's worth and echo your frustration. I work almost exclusively in matrix, and the functionality has fallen increasingly behind with each new version release. While admittedly, those of us who make significant use of the matrix functionality are surely in the minority, I think we deserve at least some enhancements to the process, e.g., those you mention in the previous post. Brian ----- Please reply to the list and not to my personal email. Those desiring my consulting or training services please feel free to email me. -- View this message in context: http://spssx-discussion.1045642.n5.nabble.com/The-agony-and-the-ecstacy- Development-of-complex-MATRIX-programs-tp5716810p5716819.html Sent from the SPSSX Discussion mailing list archive at Nabble.com. ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD |
Re: The agony and the ecstacy: Development of complex MATRIX programs.
Administrator
|
In reply to this post by Kirill Orlov
Here's some data concerning the description of SPSS as "bloated". On my laptop:
IBM SPSS Statistics 20: Size = 722 MB* Stata 12: Size = 270 MB * Not including Python and R and whatever else might be "required" in future. ;-)
--
Bruce Weaver bweaver@lakeheadu.ca http://sites.google.com/a/lakeheadu.ca/bweaver/ "When all else fails, RTFM." PLEASE NOTE THE FOLLOWING: 1. My Hotmail account is not monitored regularly. To send me an e-mail, please use the address shown above. 2. The SPSSX Discussion forum on Nabble is no longer linked to the SPSSX-L listserv administered by UGA (https://listserv.uga.edu/). |
Re: The agony and the ecstasy: Development of complex MATRIX programs.
|
Administrator
|
In reply to this post by Jon K Peck
You are perhaps too close to the vat of Python Kool Aid...
The consistent paradigm seems to be to defer to Python to do the heavy lifting and hence offload the economics of maintaining and or enhancing traditional methods of custom implementation from internal developers to python extensions... How long before SPSS itself is merely a collection of python extensions? Is the internal design of MATRIX so intractible so as to make calling external libraries prohibited or providing for localized subroutines/local variables? Is the internal design of the SPSS core transformation funstionality in such a disarray that coordinating a callable library of COMPUTE functions constitutes a herculean undertaking? MACRO is deprecated! Why is that so? Why should it be necessary to resort to external development tools rather than have those tools available in the SPSS application itself?
Please reply to the list and not to my personal email.
Those desiring my consulting or training services please feel free to email me. --- "Nolite dare sanctum canibus neque mittatis margaritas vestras ante porcos ne forte conculcent eas pedibus suis." Cum es damnatorum possederunt porcos iens ut salire off sanguinum cliff in abyssum?" |
Re: The agony and the ecstacy: Development of complex MATRIX programs.
|
Administrator
|
In reply to this post by Bruce Weaver
SPSS 11.5:
ALL MODULES!!! 168 MB --- Also AFAIK: SPSS installs a bunch of stuff in other places other than the SPSS folder. I recently installed a SPSS 20 trial and it wanted something like 2G! --
Please reply to the list and not to my personal email.
Those desiring my consulting or training services please feel free to email me. --- "Nolite dare sanctum canibus neque mittatis margaritas vestras ante porcos ne forte conculcent eas pedibus suis." Cum es damnatorum possederunt porcos iens ut salire off sanguinum cliff in abyssum?" |
Re: The agony and the ecstasy: Development of complex MATRIX programs.
|
|
In reply to this post by David Marso
Again, I can't help agreeing with almost every David's angry lunge.
Because it is all just, and so long sore.
And macros, too. The idea of a macro is basically beautiful: a language that generates code. It *can* be developed and enhanced, I believe, if development is willing and thoughtful, not lazy or urgent. SPSS development has become - to my subjective feeling - so much negligent that they didn't allowed using GPL within macros - just because GPL doesn't support quoting with apostrophes (which macro function !quote does). Also, SPSS stopped developing Chart Builder (which yet never borrowed some really good features of Interactive Graphs dialog box); instead, they added yet a new feature, Graphboard Template. Now we have to wander between the Builder and the Template. Etc etc. One can continue pulling examples of this compoundness without unity, a characteristic of modern SPSS Statistics. 10.12.2012 20:54, David Marso пишет:
You are perhaps too close to the vat of Python Kool Aid... The consistent paradigm seems to be to defer to Python to do the heavy lifting and hence offload the economics of maintaining and or enhancing traditional methods of custom implementation from internal developers to python extensions... How long before SPSS itself is merely a collection of python extensions? Is the internal design of MATRIX so intractible so as to make calling external libraries prohibited or providing for localized subroutines/local variables? Is the internal design of the SPSS core transformation funstionality in such a disarray that coordinating a callable library of COMPUTE functions constitutes a herculean undertaking? MACRO is deprecated! Why is that so? Why should it be necessary to resort to external development tools rather than have those tools available in the SPSS application itself? Jon K Peck wroteI disagree completely with Marso's so-called "translation". Jon Peck (no "h") aka Kim Senior Software Engineer, IBM[hidden email]new phone: 720-342-5621 From: David Marso <david.marso@> To:[hidden email], Date: 12/10/2012 09:03 AM Subject: Re: [SPSSX-L] The agony and the ecstasy: Development of complex MATRIX programs. Sent by: "SPSSX(r) Discussion" <[hidden email]> Translation: IBS/SPSS could care less about advanced users needs or keeping up with the Joneses in that respect and everyone who wishes a modern programming environment for designing/implementing bleeding edge statistical analytical techniques should learn Python and require the consumers of such developments to install and maintain python. Since only about 1 percent of SPSS users could care less about such capabilities such considerations shall be deferred for the foreseeable future ;-) --- Maybe a middle ground? Place certain useful functionality such as various useful COMPUTE (CDF, IDF,PDF, string manipulations, date functions etc) into a callable library and open up MATRIX to be able to call these functions. That would NOT require a complete rewrite. Jon K Peck wroteWell, MATRIX could use a lot of enhancements, but I will tell you that using numpy via Python programmability is a thing of beauty, and it has all the error handling, modularity, expressive power, and functionality that any user of such a procedure could want. Here's the top level ofthereference manual: http://docs.scipy.org/doc/numpy/reference/index.html#reference No doubt, the current MATRIX procedure, while powerful, is lacking in a number of ways, but modernizing it would be a major effort - pretty mucharewrite from scratch. And I am sure that a rewrite would look quite different from the current syntax, so it would have to be a different procedure. When there are free, readily available, and efficientadd-inssuch as numpy, is this really where IBM/SPSS should be spending itsscaredevelopment resources? It's not a matter of passing the buck: it's an issue of making the best use of SPSS resources. My unofficial two cents worth. Jon Peck (no "h") aka Kim Senior Software Engineer, IBM[hidden email]new phone: 720-342-5621 From: David Marso <david.marso@> To:[hidden email], Date: 12/10/2012 07:40 AM Subject: [SPSSX-L] The agony and the ecstacy: Development of complex MATRIX programs. Sent by: "SPSSX(r) Discussion" <[hidden email]> This is not so much a question as much as a general comment and I am interested in other people's experiences. Context: SPSS has had an embedded MATRIX language for over 20 years. For those with the necessary mathematical background and special needsitis a wonderful tool. However: The documentation SUCKS! Error handling is NON EXISTENT so debugging anything more than trivial programs is obscenely difficult. There have been absolutely NO enhancements in the past 20 years. There is no concept of modularity (no support for subroutines/local variables). I would like to see IBM/SPSS development open up the can of worms called MATRIX and reexamine its potential for developers to create specialized solutions. JoNoH... Please don't tell me to use Python. That would be passing the buck. Don't tell me to use R " "... Compared to SAS IML SPSS MATRIX is a crippled dwarf about 10 yearsbehindthe game. ---- ----- Please reply to the list and not to my personal email. Those desiring my consulting or training services please feel free to email me. -- View this message in context:http://spssx-discussion.1045642.n5.nabble.com/The-agony-and-the-ecstacy-Development-of-complex-MATRIX-programs-tp5716810.htmlSent from the SPSSX Discussion mailing list archive at Nabble.com. ===================== To manage your subscription to SPSSX-L, send a message to[hidden email](not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD----- Please reply to the list and not to my personal email. Those desiring my consulting or training services please feel free to email me. -- View this message in context: http://spssx-discussion.1045642.n5.nabble.com/The-agony-and-the-ecstacy-Development-of-complex-MATRIX-programs-tp5716810p5716819.html Sent from the SPSSX Discussion mailing list archive at Nabble.com. ===================== To manage your subscription to SPSSX-L, send a message to[hidden email](not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD----- Please reply to the list and not to my personal email. Those desiring my consulting or training services please feel free to email me. -- View this message in context: http://spssx-discussion.1045642.n5.nabble.com/The-agony-and-the-ecstacy-Development-of-complex-MATRIX-programs-tp5716810p5716829.html Sent from the SPSSX Discussion mailing list archive at Nabble.com. ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD |
Re: The agony and the ecstasy: Development of complex MATRIX programs.
|
Administrator
|
I don't know if I would call it 'angry'. It seems historically that there has been a sort of uneasy coexistence between two distinct factions of SPSS users. On one hand there is the "Real Stats Real Easy" menu clicking What's a manual" contingent and the "Hard Core SPSS nothing is impossible just give me the tools" minority and a vast middle ground of "just give me the dog food and I'll eat it".- When it comes down to the economics of development the majority wins out every time and the needs of the minority bleeding edge "to go where no one has gone before" have been relegated to a 'let them eat python' rather than enhance functional familiar tools. My client base is primarily academic researchers who contract with me to implement new statistical tools which will be distributed to a diverse user base.
To require their entire distribution to install Python is not an acceptable prerequisite. Hence my solutions rely upon MACRO and MATRIX (sometimes internal Basic scripting) .. ie complete solution without any external dependencies. I have developed a set of techniques and code design which have facilitated modular MATRIX programming and some level of debugging but it is a far call from what I have advocated in my post. For those who could care here is an insight : Segment the MATRIX code into logically/functionally distinct modules. Wrap each module in a DEFINE !ENDDEFINE block. At the head of each DEFINE block add PRINT /TITLE="module name" . This approach has saved me what remaining hair I have left after all these years. It also allows one to 'compile' individual macros independently. One solution I am currently working on is roughly 1000 lines of code. Each of the 'MATRIX modules' spans between 50 and 100 lines. Several modules are used more than once over the overall solution ;-)) Hope this helps anyone who is struggling with building large MATRIX/MACRO solutions .
Please reply to the list and not to my personal email.
Those desiring my consulting or training services please feel free to email me. --- "Nolite dare sanctum canibus neque mittatis margaritas vestras ante porcos ne forte conculcent eas pedibus suis." Cum es damnatorum possederunt porcos iens ut salire off sanguinum cliff in abyssum?" |
Re: The agony and the ecstasy: Development of complex MATRIX programs.
|
Administrator
|
There were a number of off list requests for an example of my development
technique WRT MATRIX and MACRO. Here is the simplest possible example. One can imagine the utility when the code begins to span some 20 pages of uber tight MATRIX code and conditional logic / looping and other madness ;-) Look ma, I still have hair!!! ---- data list free / x y. begin data 1 3 2 6 3 5 6 1 3 6 1 5 6 3 5 1 6 7 3 5 1 2 6 3 5 1 6 3 5 6 1 3 1 2 5 7 end data. DEFINE MATALL (). MATRIX . GET x / file * / var x. GET y / file * / var y. COMPUTE NR=NROW(X). COMPUTE one=MAKE(NR,1,1). COMPUTE augX={one,X}. COMPUTE b=GINV(T(augX)*augX)*T(augX)*Y. COMPUTE pred=augX*b. COMPUTE resid=Y-Pred. COMPUTE SSE=SSCP(resid). COMPUTE SST=SSCP(y). COMPUTE SSP=SST-SSE. PRINT b. PRINT {SST;SSP;SSE}. END MATRIX. !ENDDEFINE. MATALL. DEFINE GETDATA (). DO IF (debug GT 0). PRINT / TITLE "getdata". END IF. GET x / file * / var x. GET y / file * / var y. !ENDDEFINE. DEFINE COMP () . DO IF (debug GT 0). PRINT / TITLE "comp". END IF. COMPUTE NR=NROW(X). COMPUTE one=MAKE(NR,1,1). COMPUTE augX={one,X}. COMPUTE b=GINV(T(augX)*augX)*T(augX)*Y. COMPUTE pred=augX*b. COMPUTE resid=Y-Pred. COMPUTE SSE=SSCP(resid). COMPUTE SST=SSCP(y). COMPUTE SSP=SST-SSE. !ENDDEFINE. DEFINE REP (). DO IF (debug GT 0). PRINT / TITLE "rep". END IF. PRINT b. PRINT {SST;SSP;SSE}. !ENDDEFINE . DEFINE MATMAIN ( DEBUG !TOKENS(1) !DEFAULT (0)). MATRIX. COMPUTE debug=!DEBUG. GETDATA . COMP . REP . END MATRIX. !ENDDEFINE. MATMAIN . MATMAIN DEBUG 1.
Please reply to the list and not to my personal email.
Those desiring my consulting or training services please feel free to email me. --- "Nolite dare sanctum canibus neque mittatis margaritas vestras ante porcos ne forte conculcent eas pedibus suis." Cum es damnatorum possederunt porcos iens ut salire off sanguinum cliff in abyssum?" |
Re: The agony and the ecstasy: Development of complex MATRIX programs.
|
|
In reply to this post by Jon K Peck
David, please take a look at R or NumPy.
IMHO it's easier to write (and read) algorithms in them. Their growth has been driven by skilled users (people like you) who create packages out of their own code. Perhaps IBM/SPSS has realized that they can't compete for that type of user. ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD |
Re: The agony and the ecstasy: Development of complex MATRIX programs.
|
Administrator
|
From one of David's posts in this thread:
"My client base is primarily academic researchers who contract with me to implement new statistical tools which will be distributed to a diverse user base. To require their entire distribution to install Python is not an acceptable prerequisite. Hence my solutions rely upon MACRO and MATRIX (sometimes internal Basic scripting) .. ie complete solution without any external dependencies." This describes to a tee virtually all of the academic users I know. For some reason, academic users seem to think that it should be possible to get the job done using SPSS alone. ;-)
--
Bruce Weaver bweaver@lakeheadu.ca http://sites.google.com/a/lakeheadu.ca/bweaver/ "When all else fails, RTFM." PLEASE NOTE THE FOLLOWING: 1. My Hotmail account is not monitored regularly. To send me an e-mail, please use the address shown above. 2. The SPSSX Discussion forum on Nabble is no longer linked to the SPSSX-L listserv administered by UGA (https://listserv.uga.edu/). |
Re: The agony and the ecstasy: Development of complex MATRIX programs.
|
But is it relevant whether "it's possible to get the job done using SPSS alone"? I mean, it's also possible for me to walk to China. I think a more relevant question is how to get things done with minimal effort and maximal precision. And that often involves Python. Then all this functionality would be available to everybody "without any external dependencies". Everybody happy! Ruben > Date: Mon, 10 Dec 2012 18:20:37 -0800 > From: [hidden email] > Subject: Re: The agony and the ecstasy: Development of complex MATRIX programs. > To: [hidden email] > > From one of David's posts in this thread: > > "My client base is primarily academic researchers who contract with me to > implement new statistical tools which will be distributed to a diverse user > base. To require their entire distribution to install Python is not an > acceptable prerequisite. Hence my solutions rely upon MACRO and MATRIX > (sometimes internal Basic scripting) .. ie complete solution without any > external dependencies." > > This describes to a tee virtually all of the academic users I know. For > some reason, academic users seem to think that it should be possible to get > the job done using SPSS alone. ;-) > > > > Zuluaga, Juan wrote > > David, please take a look at R or NumPy. > > IMHO it's easier to write (and read) algorithms in them. > > Their growth has been driven by skilled users (people like you) who create > > packages out of their own code. > > Perhaps IBM/SPSS has realized that they can't compete for that type of > > user. > > > > ===================== > > To manage your subscription to SPSSX-L, send a message to > > > LISTSERV@.UGA > > > (not to SPSSX-L), with no body text except the > > command. To leave the list, send the command > > SIGNOFF SPSSX-L > > For a list of commands to manage subscriptions, send the command > > INFO REFCARD > > > > > > ----- > -- > Bruce Weaver > [hidden email] > http://sites.google.com/a/lakeheadu.ca/bweaver/ > > "When all else fails, RTFM." > > NOTE: My Hotmail account is not monitored regularly. > To send me an e-mail, please use the address shown above. > > -- > View this message in context: http://spssx-discussion.1045642.n5.nabble.com/The-agony-and-the-ecstacy-Development-of-complex-MATRIX-programs-tp5716810p5716855.html > Sent from the SPSSX Discussion mailing list archive at Nabble.com. > > ===================== > To manage your subscription to SPSSX-L, send a message to > [hidden email] (not to SPSSX-L), with no body text except the > command. To leave the list, send the command > SIGNOFF SPSSX-L > For a list of commands to manage subscriptions, send the command > INFO REFCARD |
Re: The agony and the ecstasy: Development of complex MATRIX programs.
|
|
In reply to this post by Kirill Orlov
Hi Kiril, long time
El 10/12/2012 18:39, Kirill Orlov escribió: ... SPSS development has become - to my subjective feeling - so much negligent that they didn't allowed using GPL within macros - just because GPL doesn't support quoting with apostrophes (which macro function !quote does). ....But I DO have a macro with GPL code inside, thanks to ViAnn, who helped a lot with the quote-unquote-double quote-single quote waltz needed. Maybe it is another example of the bunblebee flying although it should not be possible (http://www.sciencenews.org/view/generic/id/5400/title/Math_Trek__Flight_of_the_Bumblebee). Here is the macro, just in case it helps you. BTW, fiddling with my old macros is like looking at an old photo album, full of faded pictures... Warmest regards, Marta GG ***************************************************************** * KALBFLEISCH-PRENTICE 95% CI INTERVALS FOR SURVIVAL CURVES * ***************************************************************** * Kalbfleisch JD and Prentice RL(1980). The Statistical * * Analysis of Failure Time Data. New York: John Wiley & Sons * ***************************************************************** * (c) Marta Garcia-Granero (june 2005 - original code) modified * * (turnt into macro) september 2008. [hidden email] * ***************************************************************** * Downloaded from: http://gjyp.nl/marta/ * * Feel free to use or modify this code, but * * acknowledge the author and the web site * *********************************************** * Unlike Greenwood's, the Kalbfleisch–Prentice approach has the attractive property that the upper and lower bounds are always between 0 and 1. * MACRO definition *. DEFINE KPSURVIVAL(!POS=!TOKENS(1)/!POS=!CHAREND('(')/!POS=!CHAREND(')')). * Creating the chart template *. PRESERVE. SET RESULTS=NONE MESSAGES=NONE ERRORS=NONE. HOST COMMAND=['IF not exist C:\Temp MD C:\Temp']. HOST COMMAND=['DEL C:\Temp\*.sgt']. RESTORE. DO IF $casenum EQ 1. * Dynamically create the template for the chart *. WRITE OUTFILE='C:\Temp\KPTemplate.sgt' /'<?xml version="1.0" encoding="UTF-8" standalone="no"?>' /'<template SPSS-Version="2.2" selectPath="12 47 "' /' xmlns="http://xml.spss.com/spss/visualization"' /' xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"' /' xsi:schemaLocation="http://xml.spss.com/spss/visualization' /' http://xml.spss.com/spss/visualization/vizml-template-3.0.xsd">' /' <setAxisMargin categorical="false" lowerMargin="0%" role="x" upperMargin="5%"/>' /' <setAxisMargin categorical="false" lowerMargin="0%" role="y" upperMargin="5%"/>' /' <addFrame count="1" styleOnly="true" type="visualization">' /' <style color="#ffffff" color2="transparent" number="0" visible="true"/>' /' <style font-family="SansSerif" font-size="8pt" number="1" pattern="0" stroke-linecap="butt" text-fit="true" visible="true"/>' /' </addFrame>' /' <addFrame count="1" styleOnly="true" type="graph">' /' <style color="transparent" color2="transparent" visible="true"/>' /' <style color="#ffffff" color2="transparent" number="1" visible="true"/>' /' </addFrame>' /' <addFrame count="1" styleOnly="true" type="title">' /' <style color="transparent" color2="transparent" visible="true"/>' /' </addFrame>' /'</template>'. END IF. EXECUTE. DATASET NAME OriginalData. * Copy data to leave original unmodified *. DATASET COPY WorkingData1 WINDOW=MINIMIZED. DATASET DECLARE AuxData WINDOW=HIDDEN. DATASET ACTIVATE WorkingData1. COMPUTE status=(!2 EQ !3). RENAME VARIABLES (!1 = time). * Get rid of every other variable *. MATCH FILES FILE=* /KEEP=time status. * Save sorted file to match CI limits later *. SORT CASES BY time(A). DATASET COPY WorkingData2 WINDOW=MINIMIZED. * Saving KM survival estimates *. PRESERVE. SET ERRORS=NONE RESULTS=NONE. KM time /STATUS=status(1) /PRINT NONE /PLOT SURVIVAL /SAVE SURVIVAL . * Total sample size *. RANK VARIABLES=time(A) /N INTO totaln/PRINT=NO. * Preparing data to compute pj and Std Error of Log(survival) *. AGGREGATE /OUTFILE = * MODE=REPLACE /BREAK = time /aj = SUM(status) /sj = FIRST(SUR_1) /totaln = FIRST(totaln) /N = n. * Auxiliary file #1 *. MATRIX. COMPUTE time=0. COMPUTE aj=0. COMPUTE sj=1. COMPUTE totaln=0. COMPUTE n=0. COMPUTE vnames={'time','aj','sj','totaln','n'}. COMPUTE newdata={time,aj,sj,totaln,n}. SAVE newdata /OUTFILE='AuxData'/NAMES=vnames. END MATRIX. * Adding time 0 data (from auxiliary file)*. ADD FILES /FILE=* /FILE='AuxData' /IN=source. IF source EQ 1 totaln=LAG(totaln). SORT CASES BY time(A). * Computing auxiliary statistics *. COMPUTE rj=totaln. DO IF $casenum GT 1. - COMPUTE n = n+LAG(n) . - COMPUTE rj= totaln-LAG(n). END IF. EXECUTE ./* Don't eliminate it (necessary) *. * Getting rid of censored data *. SELECT IF NOT MISSING(sj). * More statistics and reordering dataset *. COMPUTE j= $casenum-1. COMPUTE pj = (rj-aj)/rj . MATCH FILES FILE = * /KEEP = j time aj rj pj sj . * 95% CI limits *. COMPUTE sum = ((1-pj)/(pj*rj)). DO IF $casenum GT 1. - COMPUTE sum=sum+LAG(sum). - COMPUTE sdlog = SQRT(sum/((LN(sj))**2)). * 95% Limits using Log(-Log(Sj)) & back transform * Replace 1.959964 by 2.575829 if 99%CI are wanted *. - COMPUTE lower = EXP(-EXP(LN(-LN(sj))+1.959964*sdlog)). - COMPUTE upper = EXP(-EXP(LN(-LN(sj))-1.959964*sdlog)). END IF. RESTORE. * Report *. FORMAT j TO rj (F8.0) sj pj lower upper (F8.3). SUMMARIZE /TABLES=time sj lower upper /FORMAT=LIST NOCASENUM NOTOTAL /TITLE='Kaplan-Meier Estimates and Kalbfleisch-Prentice 95% CI' /CELLS=NONE. * Adding survival & CI limits to original dataset *. MATCH FILES FILE=* /KEEP = time sj lower upper. MATCH FILES /FILE='WorkingData2' /TABLE=* /BY time. * Adding time=0 survival data (auxiliary file #2) *. PRESERVE. SET ERRORS=NONE RESULTS=NONE. MATRIX. COMPUTE time=0. COMPUTE sj=1. COMPUTE lower=1. COMPUTE upper=1. COMPUTE vnames={'time','sj','lower','upper'}. COMPUTE newdata={time,sj,lower,upper}. SAVE newdata /OUTFILE='AuxData'/NAMES=vnames. END MATRIX. ADD FILES /FILE=* /FILE='AuxData'. SORT CASES BY time(a). * Filling data for censored cases after the first event *. DO IF MISSING(sj). - COMPUTE sj=LAG(sj). - COMPUTE lower=LAG(lower). - COMPUTE upper=LAG(upper). END IF. RESTORE. * Survival plot (fully formatted, thanks to ViAnn Beadle) *. FORMAT sj lower upper (F8.1). !LET !gdataset=!CONCAT('"','graphdataset','"'). !LET !time=!CONCAT('"','time','"'). !LET !sj=!CONCAT('"','sj','"'). !LET !lower=!CONCAT('"','lower','"'). !LET !upper=!CONCAT('"','upper','"'). !LET !ylabel=!CONCAT('"','Cumulative Survival','"'). !LET !xlabel=!CONCAT('"','Follow-up Time','"'). !LET !title=!CONCAT('"','KM estimate with Kalbfleish-Prentice 95%CI','"'). !LET !px=!CONCAT('"','2px','"'). GGRAPH /GRAPHDATASET NAME=!gdataset VARIABLES=time sj lower upper MISSING=LISTWISE REPORTMISSING=NO /GRAPHSPEC SOURCE=INLINE TEMPLATE="C:\Temp\KPTemplate.sgt". BEGIN GPL SOURCE: s=userSource(id(!gdataset)) DATA: time=col(source(s), name(!time)) DATA: sj=col(source(s), name(!sj)) DATA: lower=col(source(s), name(!lower)) DATA: upper=col(source(s), name(!upper)) SCALE: linear(dim(2), include(0)) GUIDE: axis(dim(1), label(!xlabel)) GUIDE: axis(dim(2), label(!ylabel), delta(0.1)) GUIDE: legend(aesthetic(aesthetic.shape.interior), null()) GUIDE: text.title(label(!title)) ELEMENT: line(position(smooth.step.left(time*lower)), missing.wings(), shape.interior(shape.dash), color(color.black)) ELEMENT: line(position(smooth.step.left(time*upper)), missing.wings(), shape.interior(shape.dash), color(color.black)) ELEMENT: line(position(smooth.step.left(time*sj)), missing.wings(), shape.interior(shape.solid), size(size.!px)) END GPL. * Tidying up *. DATASET ACTIVATE OriginalData WINDOW=ASIS. DATASET CLOSE AuxData. DATASET CLOSE WorkingData1. DATASET CLOSE WorkingData2. !ENDDEFINE. ***************************************************************** * EXAMPLE DATASET * * Table 9.1 Biostatistical Methods in Epidemiology (pp 177-178) * * Stephen C Newman (2001); JOHN WILEY&SONS * *****************************************************************. DATA LIST FREE/FuTime Event stage reclevel (4 F8.0). BEGIN DATA 50 0 1 1 51 1 1 1 51 0 1 1 53 0 1 1 53 0 1 1 54 0 1 1 54 0 1 1 55 0 1 1 56 1 1 1 56 0 1 1 57 0 1 1 60 0 1 1 10 1 1 2 34 1 1 2 34 0 1 2 47 1 1 2 47 1 1 2 49 0 1 2 49 0 1 2 50 0 1 2 50 0 1 2 50 0 1 2 50 0 1 2 50 0 1 2 50 0 1 2 50 0 1 2 51 0 1 2 51 0 1 2 51 0 1 2 51 0 1 2 51 0 1 2 51 0 1 2 52 0 1 2 52 0 1 2 52 0 1 2 52 0 1 2 52 0 1 2 53 0 1 2 53 0 1 2 53 0 1 2 53 0 1 2 53 0 1 2 53 0 1 2 54 0 1 2 54 0 1 2 54 0 1 2 54 0 1 2 54 0 1 2 55 0 1 2 55 0 1 2 56 0 1 2 56 0 1 2 57 0 1 2 57 0 1 2 57 0 1 2 57 0 1 2 57 0 1 2 58 0 1 2 58 0 1 2 58 0 1 2 58 0 1 2 58 0 1 2 59 0 1 2 59 0 1 2 59 0 1 2 59 0 1 2 60 0 1 2 60 0 1 2 60 0 1 2 4 0 2 1 9 1 2 1 13 1 2 1 21 1 2 1 29 1 2 1 29 1 2 1 40 1 2 1 46 1 2 1 49 0 2 1 49 0 2 1 52 0 2 1 52 0 2 1 53 0 2 1 54 0 2 1 55 0 2 1 55 0 2 1 56 0 2 1 57 1 2 1 57 0 2 1 58 0 2 1 58 0 2 1 59 0 2 1 60 0 2 1 11 1 2 2 16 1 2 2 21 1 2 2 23 1 2 2 23 1 2 2 24 1 2 2 33 1 2 2 33 1 2 2 36 1 2 2 36 1 2 2 36 0 2 2 37 1 2 2 45 1 2 2 46 1 2 2 49 0 2 2 49 0 2 2 50 0 2 2 50 0 2 2 50 0 2 2 50 0 2 2 50 0 2 2 50 0 2 2 51 0 2 2 51 0 2 2 51 0 2 2 51 0 2 2 52 0 2 2 52 0 2 2 52 0 2 2 52 0 2 2 52 0 2 2 53 0 2 2 53 0 2 2 53 0 2 2 53 0 2 2 53 0 2 2 54 0 2 2 54 0 2 2 54 0 2 2 54 0 2 2 55 0 2 2 55 0 2 2 55 0 2 2 55 0 2 2 56 0 2 2 56 0 2 2 56 0 2 2 56 0 2 2 56 0 2 2 56 0 2 2 57 0 2 2 57 0 2 2 57 0 2 2 57 0 2 2 58 1 2 2 58 1 2 2 58 0 2 2 58 0 2 2 58 0 2 2 58 0 2 2 58 0 2 2 58 0 2 2 58 0 2 2 58 0 2 2 59 0 2 2 59 0 2 2 59 0 2 2 59 0 2 2 59 0 2 2 60 0 2 2 60 0 2 2 60 0 2 2 60 0 2 2 60 0 2 2 60 0 2 2 9 1 3 1 12 1 3 1 14 1 3 1 15 1 3 1 15 0 3 1 17 1 3 1 21 1 3 1 22 1 3 1 23 1 3 1 23 1 3 1 31 1 3 1 34 1 3 1 35 1 3 1 53 0 3 1 60 0 3 1 7 0 3 2 9 1 3 2 17 1 3 2 21 0 3 2 22 1 3 2 22 1 3 2 34 1 3 2 34 1 3 2 41 1 3 2 49 0 3 2 52 0 3 2 55 1 3 2 56 0 3 2 58 0 3 2 58 0 3 2 59 0 3 2 59 0 3 2 END DATA. VAR LABEL stage' Tumour Stage'/ reclevel 'Receptor level' /FuTime 'Survival time (months)' /Event 'Censoring Status'. VALUE label stage 1'I' 2'II' 3'III' /reclevel 1'Low' 2'High' /Event 0'Censored' 1'Event'. KPSURVIVAL FuTime Event(1). |
Re: The agony and the ecstasy: Development of complex MATRIX programs.
|
|
In reply to this post by Ruben Geert van den Berg
Ruben, you entirely forget of a subset of users who are rather
creative individuals than individual consumers. Those ones would
defy "doing things with minimal effort". They ultimately dream to
rewrite all the libraries existent in Python or Matlab now in SPSS
native syntax. They think that SPSS deserves going its own proud way
and to have its own community of users-developers, not Python's
community.
And now that Python *is* already in SPSS, people who like Python - the rational "consumers" - should feel happy and settled (nobody votes to delete Python or R back from SPSS, it'd be idiotic). Why not now to satisfy the others - the "creatives" nuts - and to give them a bit enriched programmability tools within old macro and matrix facilities, just because these people rely on them and don't like Python or R? 11.12.2012 9:26, Ruben van den Berg
пишет:
|
Re: The agony and the ecstasy: Development of complex MATRIX programs.
|
|
In reply to this post by Marta Garcia-Granero
Hello Marta, yeah long time to see.
Yes, it is possible to perform amusing pas like !LET !gdataset=!CONCAT('"','graphdataset','"'). But it is outside of GPL block. It looks so awkward that one should feel himself uneasy. The GPL block *must* itself permit using macro functions and macro statements inside itself - like any usual syntax does permit it. But somebody in SPSS company is too lazy *to get inside GPL and overhaul some aspects of it*. Not they. They prefer to take GPL "as is" even if it can't get on smoothly with SPSS old residents such as macros. SPSS team do not esteem macro facility , their own creature. |
Re: The agony and the ecstasy: Development of complex MATRIX programs.
|
Administrator
|
In reply to this post by Bruce Weaver
One interesting fact is that a number of my recent projects have already been prototyped in R and as part of the research grants the principals were required to provide a native SPSS programming solution as a supporting implementation (ie publish a new analytical technique and provide a means of interesting consumers to create analyses of their own data sets). One thing I learned from doing these projects is that it seems almost "natural" to create virtually incomprehensible, bRamaging, retinal hemorrhage inducing code in R . Somewhat more complicated to do the same in SPSS MATRIX (I must plead guilty to possessing this marvelous ability to one-line some rather scary MATRIX pearls -but despite all my efforts towards deep obsfucation they still maintain a humanly readable character- Of course such eye bleeders are surrounded by ample comments (I already have job security in that regard). I am not at liberty due to NDA considerations to post any of these R beauties (or SPSS attempts) but some of them rival APL http://langexplr.blogspot.com/2011/03/quick-look-at-apl.html in their ease of parsability . Some of the more 'clever?' almost ALWAYS uncommented Python code I have seen posted on this list can vie for a similar claim to fame.
--
Please reply to the list and not to my personal email.
Those desiring my consulting or training services please feel free to email me. --- "Nolite dare sanctum canibus neque mittatis margaritas vestras ante porcos ne forte conculcent eas pedibus suis." Cum es damnatorum possederunt porcos iens ut salire off sanguinum cliff in abyssum?" |
Re: The agony and the ecstasy: Development of complex MATRIX programs.
|
Administrator
|
In reply to this post by Kirill Orlov
One thing I suspect is that the uber brilliant programmer (I knew the man) who wrote MACRO has retired and everyone else is scared s#!^|<$$ to open it up and poke around lest they rouse a dragon or demon and completely FUBAR the rest of the central system. Same goes with MATRIX which as I understand it was developed externally from SPSS (from the FM "The MATRIX procedure was originally developed at the Madison Academic Computing Center, University of Wisconsin.".
--
Please reply to the list and not to my personal email.
Those desiring my consulting or training services please feel free to email me. --- "Nolite dare sanctum canibus neque mittatis margaritas vestras ante porcos ne forte conculcent eas pedibus suis." Cum es damnatorum possederunt porcos iens ut salire off sanguinum cliff in abyssum?" |
«
Return to SPSSX Discussion
|
1 view|%1 views
| Free forum by Nabble | Edit this page |