Variance-Covariance Matrix

[fevziesen]

Hi all,

Do we have any function to calculate variance covariance matrix in SPSS?

[Bruce Weaver]

Assuming you mean the covariance matrix for a set of variables (rather than the covariance matrix for some model parameters), just add XPROD to the STATISTICS sub-command for CORRELATIONS.  E.g.,

CORRELATIONS
  /VARIABLES=X1 to X5
  /STATISTICS XPROD
  /MISSING=PAIRWISE.

If you need the covariance matrix in a data set, call OMS before using CORRELATIONS.  E.g.,

DATASET DECLARE  Corr.
OMS
  /SELECT TABLES
  /IF COMMANDS=['Correlations'] SUBTYPES=['Correlations']
  /DESTINATION FORMAT=SAV NUMBERED=TableNumber_
   OUTFILE='Corr'.

CORRELATIONS
  /VARIABLES=X1 to X5
  /STATISTICS XPROD
  /MISSING=PAIRWISE.

OMSEND.
DATASET ACTIVATE Corr.
TEMPORARY.
SELECT IF Var2 EQ "Covariance".
LIST Var2 X1 to X5.

OUTPUT (for some data I made up):

Var2                                    X1        X2        X3        X4        X5
 
Covariance                          74.924    12.282    -2.090     2.277    -3.079
Covariance                          12.282    93.014   -16.735   -16.009   -13.267
Covariance                          -2.090   -16.735   124.661    18.242    -8.632
Covariance                           2.277   -16.009    18.242    90.076     1.744
Covariance                          -3.079   -13.267    -8.632     1.744    93.260
 
HTH.

fevziesen wrote

Hi all,

Do we have any function to calculate variance covariance matrix in SPSS?

[David Marso]

Use MATRIX OUT subcommand in CORRELATIONS.  
You then follow that with MCONVERT command.

fevziesen wrote

Hi all,

Do we have any function to calculate variance covariance matrix in SPSS?

[Bruce Weaver]

Right!  I forgot about MCONVERT--and I've used it before!  This is much easier than the method I suggested earlier (XPROD + OMS, etc).  

David Marso wrote

Use MATRIX OUT subcommand in CORRELATIONS.  
You then follow that with MCONVERT command.

fevziesen wrote

Hi all,

Do we have any function to calculate variance covariance matrix in SPSS?

[Bruce Weaver]

Here's another easy way if all you want is a table in the output viewer.  Replace X1 to X5 with your variable list:

RELIABILITY VARIABLES X1 to X5 /STATISTICS=COVARIANCES.

Bruce Weaver wrote

Right!  I forgot about MCONVERT--and I've used it before!  This is much easier than the method I suggested earlier (XPROD + OMS, etc).  

David Marso wrote

Use MATRIX OUT subcommand in CORRELATIONS.  
You then follow that with MCONVERT command.

fevziesen wrote

Hi all,

Do we have any function to calculate variance covariance matrix in SPSS?

[Kirill Orlov]

An excerpt from my MATRIX functions (sorry, my web-page is temporarily off)

******** COVARIANCE MATRIX ********.
*/*!cov(data%name)*/*.
*Version 1.
*Computes covariances between colums of DATA.
*Returns square symmetric matrix NAME.

define !cov(!pos= !token(1) /!pos= !charend('%') /!pos= !charend(')'))
comp !3= !2.
comp @sum= csum(!3).
comp !3= (sscp(!3)-t(@sum)*@sum/nrow(!3))/(nrow(!3)-1).
release @sum.
!enddefine.


******** CORRELATION MATRIX ********.
*/*!corr(data%name)*/*.
*Version 1.
*Computes Pearson correlations between colums of DATA.
*Returns square symmetric matrix NAME.

define !corr(!pos= !token(1) /!pos= !charend('%') /!pos= !charend(')'))
comp !3= !2.
comp @sum= csum(!3).
comp @ssdev= cssq(!3)-@sum&**2/nrow(!3).
comp !3= sscp(!3)-t(@sum)*@sum/nrow(!3).
comp !3= !3/sqrt(t(@ssdev)*@ssdev).
call setdiag(!3,1).
release @sum,@ssdev.
!enddefine.

EXAMPLE.

matrix.
get vars /variables= v1 v2 v3 /names= names.
!cov(vars%cov).
print cov /rnames= names /cnames= names.
end matrix.

===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD

[Andy W]

FACTOR works the same way as well. I thought maybe those procedures would produce a warning with a non-positive definite covariance matrix (or for reliabilities a negative correlation) but they seem to be fine. (Reliabilities just gives a warning in a footnote of a table.)

*************************************.
MATRIX.
SAVE UNIFORM(100,5) /OUTFILE = *.
END MATRIX.
DATASET NAME D.
COMPUTE COL6 = -1*COL5.
COMPUTE COL7 = COL1 + COL3.
DATASET DECLARE CovMat.
FACTOR VARIABLES = COL1 TO COL7 /MATRIX OUT(COV='CovMat').

DATASET ACTIVATE D.
DATASET DECLARE CovMat2.
RELIABILITY VARIABLES COL1 to COL7 /STATISTICS=COVARIANCES /MATRIX=OUT('CovMat2').
*************************************.

[Maguin, Eugene]

So far as I know, there is no function (but there's python). But there's Matrix-end matrix, which I think Bruce had a post about yesterday. There's commands (Correlation is one but other people have pointed out other commands) that can output a correlation matrix that can be converted to covariances by the Mconvert command.
Gene Maguin

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of fevziesen
Sent: Wednesday, February 18, 2015 1:12 PM
To: [hidden email]
Subject: Variance-Covariance Matrix

Hi all,

Do we have any function to calculate variance covariance matrix in SPSS?



--
View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Variance-Covariance-Matrix-tp5728717.html
Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Ware, William B]

If you have the R add-in, try cov( )...

wbw

William B. Ware, Ph.D.
McMichael Term Professor of Education, 2011-2013
Educational Psychology, Measurement, and Evaluation
CB #3500 - 118 Peabody Hall 
University of North Carolina at Chapel Hill
Chapel Hill, NC     27599-3500
Office: (919)-962-0132
Fax:    [phone]
Office:  118 Peabody Hall
EMAIL: [hidden email]
Adjunct Professor, School of Social Work
Academy of Distinguished Teaching Scholars at UNC-Chapel Hill

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Maguin, Eugene
Sent: Thursday, February 19, 2015 11:19 AM
To: [hidden email]
Subject: Re: Variance-Covariance Matrix

So far as I know, there is no function (but there's python). But there's Matrix-end matrix, which I think Bruce had a post about yesterday. There's commands (Correlation is one but other people have pointed out other commands) that can output a correlation matrix that can be converted to covariances by the Mconvert command.
Gene Maguin

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of fevziesen
Sent: Wednesday, February 18, 2015 1:12 PM
To: [hidden email]
Subject: Variance-Covariance Matrix

Hi all,

Do we have any function to calculate variance covariance matrix in SPSS?



--
View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Variance-Covariance-Matrix-tp5728717.html
Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Art Kendall]

Did I miss something? Did the OP say why the covariance matrix from RELIABILITY is not what is needed?

[Bruce Weaver]

Nope, you've not missed anything Art.  The OP has not been heard from since the OP.  ;-)

(For anyone who is confused, the first OP above = original poster, and the second OP = original post.)

Art Kendall wrote

Did I miss something? Did the OP say why the covariance matrix from RELIABILITY is not what is needed?

[fevziesen]

Thanks for all posts. But I meant variance-covariance matrix.In R, there is a command vcov in stats pack but it is useless when you reach a limit of data like a million..

[Maguin, Eugene]

What's wrong with the suggestions offered? Is it that none of the methods can process millions of records? You asked for a function but there is no function. People offered alternatives.
Gene Maguin


-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of fevziesen
Sent: Thursday, February 19, 2015 2:32 PM
To: [hidden email]
Subject: Re: Variance-Covariance Matrix

Thanks for all posts. But I meant variance-covariance matrix.In R, there is a command vcov in stats pack but it is useless when you reach a limit of data like a million..



--
View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Variance-Covariance-Matrix-tp5728717p5728737.html
Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Bruce Weaver]

The syntax offered in Andy W's post still works on my machine with 10 million records.  I.e., I changed this...

SAVE UNIFORM(100,5) /OUTFILE = *.

...to this:

SAVE UNIFORM(10000000,5) /OUTFILE = *.

But when I tried 100 million records, I got the following error message:

Run MATRIX procedure:
Bytes requested = 4000000000
>Error encountered in source line #  3361
 
>Error # 12477
>MATRIX could not allocate memory for an object.  Reduce problem size, or
>release unused matrices using RELEASE statement.  Use DISPLAY statement to
>list all allocated objects.  .
>Execution of this command stops.

HTH.

Maguin, Eugene wrote

What's wrong with the suggestions offered? Is it that none of the methods can process millions of records? You asked for a function but there is no function. People offered alternatives.
Gene Maguin


-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of fevziesen
Sent: Thursday, February 19, 2015 2:32 PM
To: [hidden email]
Subject: Re: Variance-Covariance Matrix

Thanks for all posts. But I meant variance-covariance matrix.In R, there is a command vcov in stats pack but it is useless when you reach a limit of data like a million..



--
View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Variance-Covariance-Matrix-tp5728717p5728737.html
Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[David Marso]

Maybe you need to indicate what in the elephant you need to do with this matrix in order for anyone to provide better guidance.  I do recall your going on about some variant of mahalanobis (sic) distance awhile back.  Perhaps this is related?  In that case you would do well to aspire towards some level of mastery of the matrix language.
Each of the solutions provided create such a matrix.
Perhaps you aren't paying attention to the specifics of what has been provided to you?

fevziesen wrote

Thanks for all posts. But I meant variance-covariance matrix.In R, there is a command vcov in stats pack but it is useless when you reach a limit of data like a million..

[David Marso]

Bruce, are you still running a 32 bit version of Windows or SPSS? ;-(
---

Bruce Weaver wrote

The syntax offered in Andy W's post still works on my machine with 10 million records.  I.e., I changed this...

SAVE UNIFORM(100,5) /OUTFILE = *.

...to this:

SAVE UNIFORM(10000000,5) /OUTFILE = *.

But when I tried 100 million records, I got the following error message:

Run MATRIX procedure:
Bytes requested = 4000000000
>Error encountered in source line #  3361
 
>Error # 12477
>MATRIX could not allocate memory for an object.  Reduce problem size, or
>release unused matrices using RELEASE statement.  Use DISPLAY statement to
>list all allocated objects.  .
>Execution of this command stops.

HTH.

Maguin, Eugene wrote

What's wrong with the suggestions offered? Is it that none of the methods can process millions of records? You asked for a function but there is no function. People offered alternatives.
Gene Maguin


-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of fevziesen
Sent: Thursday, February 19, 2015 2:32 PM
To: [hidden email]
Subject: Re: Variance-Covariance Matrix

Thanks for all posts. But I meant variance-covariance matrix.In R, there is a command vcov in stats pack but it is useless when you reach a limit of data like a million..



--
View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Variance-Covariance-Matrix-tp5728717p5728737.html
Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Bruce Weaver]

Good guess, David.  Last year, I had a 64-bit version of v21.  But this year, the university (in its wisdom) decided we would have access to 32-bit versions only (v22).  Apparently, a few too many folks had problems with the 64-bit version last year, and the IT folks decided they'd have fewer complaints to deal with if everyone had the 32-bit version!  

David Marso wrote

Bruce, are you still running a 32 bit version of Windows or SPSS? ;-(
---

Bruce Weaver wrote

The syntax offered in Andy W's post still works on my machine with 10 million records.  I.e., I changed this...

SAVE UNIFORM(100,5) /OUTFILE = *.

...to this:

SAVE UNIFORM(10000000,5) /OUTFILE = *.

But when I tried 100 million records, I got the following error message:

Run MATRIX procedure:
Bytes requested = 4000000000
>Error encountered in source line #  3361
 
>Error # 12477
>MATRIX could not allocate memory for an object.  Reduce problem size, or
>release unused matrices using RELEASE statement.  Use DISPLAY statement to
>list all allocated objects.  .
>Execution of this command stops.

HTH.

Maguin, Eugene wrote

What's wrong with the suggestions offered? Is it that none of the methods can process millions of records? You asked for a function but there is no function. People offered alternatives.
Gene Maguin


-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of fevziesen
Sent: Thursday, February 19, 2015 2:32 PM
To: [hidden email]
Subject: Re: Variance-Covariance Matrix

Thanks for all posts. But I meant variance-covariance matrix.In R, there is a command vcov in stats pack but it is useless when you reach a limit of data like a million..



--
View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Variance-Covariance-Matrix-tp5728717p5728737.html
Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Kirill Orlov]

To keep 100 million X 5 matrix in MATRIX you need roughtly 5 Gb memory, and better - 8 Gb because MATRIX, when it can cope not liberelly, may start work very slow.

20.02.2015 1:03, Bruce Weaver пишет:

The syntax offered in Andy W's post still works on my machine with 10 million
records.  I.e., I changed this...

SAVE UNIFORM(100,5) /OUTFILE = *.

...to this:

SAVE UNIFORM(10000000,5) /OUTFILE = *.

But when I tried 100 million records, I got the following error message:

Run MATRIX procedure: 
Bytes requested = 4000000000 

===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD

[fevziesen]

I have 50 million cases and looking for something faster and effective. I have tried so much, but it always stopped working. And finally my macbook pro is dead. I am not joking. Yesterday it's gone unexpectedly while processing data. Screen went black and gone...Today I got service report, ssd is gone and must be replaced. I think programmers should talk about hardware-integrated enviroments for performing statistical tests and data processing.

[Andy W]

Below with 100,000,000 rows and 7 variables works on my machine in a little under 3 minutes. For processing in MATRIX yes if you plop in the original data of 5e7 rows it will be problematic. You can get the correlation matrix from the procedures others have mentioned though which will clearly be smaller and then pipe that to MATRIX for whatever manipulations you want.

Also worth mentioning for just correlations you can work out a sample to obtain the needed precision.

*************************************.
SET SEED 10.
INPUT PROGRAM.
LOOP N = 1 TO 1e8.
END CASE.
END LOOP.
END FILE.
END INPUT PROGRAM.
DATASET NAME D.
VECTOR COL(5).
DO REPEAT C = COL1 TO COL5.
COMPUTE C = RV.UNIFORM(0,1).
END REPEAT.
COMPUTE COL6 = -1*COL5.
COMPUTE COL7 = COL1 + COL3.
DATASET DECLARE CovMat.
FACTOR VARIABLES = COL1 TO COL7 /MATRIX OUT(COV='CovMat').
*************************************.