Re: Historical query regarding SPSS functionality

Posted by Jon Peck on Dec 31, 2020; 10:24pm
URL: http://spssx-discussion.165.s1.nabble.com/Historical-query-regarding-SPSS-functionality-tp5740087p5740092.html

Mr Pack Rat!

The copyright dates are an artifact of the IBM acquisition of SPSS.  IBM dictated that everything be marked with a 1989 copyright notice - I have no idea why, but there was a lot of SPSS code and other materials that didn't have a copyright date in it at the time, so that may have been an approximation.   IBM required a copyright notice in every single thing that SPSS owned.

The Cytel code was written in C, but it used a number of not so good practices that made it hard to port or adapt.  SPSS was using both Fortran and C code at the time - maybe C++ by then, but had stopped doing new development in Fortran (it used a Ratprep preprocessor for it).  There is still plenty of good Fortran code in the product, but Fortran wasn't the problem for the StatXact code.

On Thu, Dec 31, 2020 at 2:55 PM Michael Palij <[hidden email]> wrote:

Hilbe (2004, see full reference below) goes into a fair amount of detail about SPSS ver 12.01

and goes over StatXact's contribution.  I have the feeling that it is unlikely that StatXact

code was included in earlier versions.

On a related note, I dug up my copy of the SPSS Exact Tests ver 21 manual which

has the curious copyright dates of 1989 and 2012 -- 1989 is before the release of

StatXact version 1 (both copyrights of IBM).  I have no idea why 1989 is used especially

since the reference list has no publication for 1989 (does it refer to an unpublished

document?).  Anyway, in the front matter of the manual is the following Acknowledgment:

Acknowledgments
Exact Tests is the result of a collaboration between Cytel Software Corporation and SPSS Inc. The
exact algorithms were developed by Cytel. Integrating the exact engines into the user interface and
documenting the statistical methods in a comprehensive user manual were tasks shared by both
organizations. We would like to thank our fellow developers, Yogesh Gajjar, Hemant Govil, Pralay
Senchaudhuri, and Shailesh Vasundhara of Cytel.

We owe a special debt to Professor Marvin Zelen for creating an exciting intellectual

environment in the Department of Biostatistics at Harvard. He encouraged us to work on a
number of challenging research problems in computational statistics, and this research has
culminated in the development of Exact Tests.

Cyrus R. Mehta and Nitin R. Patel
Cytel Software Corporation and Harvard School of Public Health
Cambridge, Massachusetts

****************************************

Final point:  one of the references is for a 1986 paper that describes a Fortran program

titled FEXACT for the Fisher test of contingency tables.  This leads me to think that

most of StatXact was also probably written in Fortran.  But by the 1990s I would assume

that SPSS was no longer using Fortran code -- wasn't there a conversion to some

version of C by that time (I think SAS had overhauled its code around this time),

If this is true, then I can imagine converting the StatXact Fortran into SPSS C code

would have been a difficult task.  Wasn't this the problem with BMDP?  Just curious.

Take care and have a Better New Year,

-Mike Palij

New York University

On Thu, Dec 31, 2020 at 4:25 PM Jon Peck <[hidden email]> wrote:

It is true that the Fisher-Freeman-Halton and McNeman-Bowker tests depend on the StatXact code via the METHOD subcommand of CROSSTABS and require the Exact Tests option.  The Command Syntax Reference manual started listing new and updated syntax as of SPSS version 13.  This subcommand is not mentioned as a change, so it would have been added in version 12 or earlier, but I don't have an exact date.  I no longer have access to the SPSS code base, which would have an exact record of the changes.  My recollection is that integrating the StatXact code, which was fortunately not my task, was a nightmare due to the way that code was written.

On Thu, Dec 31, 2020 at 1:49 PM Michael Palij <[hidden email]> wrote:

I think that Jon Peck will have to join this thread in order to correct errors or clarify

points in my presentation.  That being said, consider the following:

(1)  If I am not mistaken, prior to the 1990s, SPSS was not able to perform "exact tests"

that are represented by the McNemar-Bowker test and the Fisher-Freeman-Halton.

This was a general limitation of the major statistical software packages (e.g., SAS)

because of hardware and programming limitations.

(2)  The statisticians Cyrus Mehta and Nitin Patel had worked on computer programming

of exact tests and developed the software StatXact (misc statistical tests) and LogXact

(exact logistic regression).  Their algorithms were able to overcome the limitations of

previous implementations and was used in the development of commercial statistical

software.  Version 1 of the PC version of StatXact was released around 1990 and here

are a couple reviews of the early software:

Mehta, C. (1991). StatXact: A Statistical Package for Exact Nonparametric Inference.

The American Statistician, 45(1), 74-75. doi:10.2307/2685246

Sprent, P., & Mehta, C. (1990). Journal of the Royal Statistical Society. Series C

(Applied Statistics), 39(3), 391-395. doi:10.2307/2347397

StatXact was developed on mainframe and similar computers but StatXact was

developed specifically for MS-DOS systems.  Subsequent versions continued

to be available across different computer platforms.

(3)  What is significant about Mehta and Patel's work is that it was easier for

the major statistical packages to either provide the capability to call StatXact

(if it were on the computer system) or the algorithms could be licensed into

the software.  I believe that SAS was the first system to incorporate M&P's

exact test procedure.  By 1995 SPSS provide an exact tests module and

incorporated some of the tests into existing procedures (e.g., crosstabs;

not sure which version of SPSS this was).  Hilbe (2004) wrote a review of

SPSS ver 12 in which he focuses on the logistic regression procedure

(apparently the LogXact software was not used by SPSS) and the Exact

tests module -- he covers some of the history between SPSS and StatXact

which might be useful.  See:

Hilbe, J. (2004). A Review of SPSS 12.01, Part 2. The American Statistician,

58(2), 168-171. Retrieved December 31, 2020, from http://www.jstor.org/stable/27643530

(4)  Jon Peck might know if StatXact code was used in SPSS in version before

version 12 and may be able to give a better/more accurate/exact? date (I bet

it is between 1993 to 1995).

I am curious though as to why you are interested in this question.  I would also

suggest searching Jstor (which has ASA journal and other relevant statistical

journals) for reviews of software and related issues.

-Mike Palij

New York University

On Thu, Dec 31, 2020 at 1:26 PM MACDOUGALL Margaret <[hidden email]> wrote:

Season’s greetings!

 

I have a historical question! Is anyone able to advise me on when functionality to run the following tests first became available in SPSS:

 

• the McNemar-Bowker test as an extension to the McNemar test

 

and

 

• the Fisher-Freeman-Halton test as an extension to Fisher’s Exact test?

 

Thanks in advance

Best wishes
Margaret

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Dr Margaret MacDougall
Medical Statistician and Researcher in Education
(Senior Lecturer)
Centre for Population Health Sciences
Usher Institute
University of Edinburgh Medical School
Teviot Place
Edinburgh EH8 9AG
Email: [hidden email]

https://www.ed.ac.uk/profile/margaret-macdougall

 

The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD

===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD

--

Jon K Peck
[hidden email]

--
===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD