Variance-Covariance Matrix

[Art Kendall]

If I recall correctly, MATRIX works in memory.  I guess that it is in the creation of the data to work on that you are running into problems.

It is hard to think of what one would do with a variance-covariance matrix on millions of variables.

[Art Kendall]

(1) how many variables do you have?
(2) what syntax are you using?
(3) what are your goals?  what are you going to use the variance covariance matrix for?

[David Marso]

That would NEVER fly!
OP obviously means millions of cases.
To get this thread back on track I think OP needs to clarify intentions!!!
Is this related to the cat in the tree suggested by the recent thread http://spssx-discussion.1045642.n5.nabble.com/Minimum-Covariance-Determinant-Estimator-Implementation-td5728643.html
or have you let that rest in peace?
LMGTFY!
http://search.us.com/serp?k=+minimum+covariance+determinant+&guid={A64B828F-7DB3-48FD-ADC6-1CE487A91F58}&pub=&campaign=&noredirect=1
As a subtle hint, please note that there are updating formulas for variance/covariance (seek out provisional means algorithm -I believe you will find it in the REGRESSION algos-
-----

Art Kendall wrote

If I recall correctly, MATRIX works in memory.  I guess that it is in the creation of the data to work on that you are running into problems.

It is hard to think of what one would do with a variance-covariance matrix on millions of variables.

[Bruce Weaver]

The problem raised in this thread jogged my memory.  I think the incompatibility of 64-bit SPSS with 32-bit Office drivers was another reason the university decided that everyone would use 32-bit SPSS.  (Going to 64-bit Office was not an option, apparently.)  

Bruce Weaver wrote

Good guess, David.  Last year, I had a 64-bit version of v21.  But this year, the university (in its wisdom) decided we would have access to 32-bit versions only (v22).  Apparently, a few too many folks had problems with the 64-bit version last year, and the IT folks decided they'd have fewer complaints to deal with if everyone had the 32-bit version!  

David Marso wrote

Bruce, are you still running a 32 bit version of Windows or SPSS? ;-(
---

Bruce Weaver wrote

The syntax offered in Andy W's post still works on my machine with 10 million records.  I.e., I changed this...

SAVE UNIFORM(100,5) /OUTFILE = *.

...to this:

SAVE UNIFORM(10000000,5) /OUTFILE = *.

But when I tried 100 million records, I got the following error message:

Run MATRIX procedure:
Bytes requested = 4000000000
>Error encountered in source line #  3361
 
>Error # 12477
>MATRIX could not allocate memory for an object.  Reduce problem size, or
>release unused matrices using RELEASE statement.  Use DISPLAY statement to
>list all allocated objects.  .
>Execution of this command stops.

HTH.

Maguin, Eugene wrote

What's wrong with the suggestions offered? Is it that none of the methods can process millions of records? You asked for a function but there is no function. People offered alternatives.
Gene Maguin


-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of fevziesen
Sent: Thursday, February 19, 2015 2:32 PM
To: [hidden email]
Subject: Re: Variance-Covariance Matrix

Thanks for all posts. But I meant variance-covariance matrix.In R, there is a command vcov in stats pack but it is useless when you reach a limit of data like a million..



--
View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Variance-Covariance-Matrix-tp5728717p5728737.html
Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[David Marso]

So they only license the 32 bit version?
I would think they would be able to make both 32 and 64 bit versions available and simply caveat to 64 bit users that they won't support f'd up ODBC isues etc...
I have Vista 32 on bootcamp on my MacBookPro and 64 bit 8.1 on my HP Envy so I run both 32 bit and 64 bit versions under my personal license.
Check with your peeps and see about that?
--

Bruce Weaver wrote

The problem raised in this thread jogged my memory.  I think the incompatibility of 64-bit SPSS with 32-bit Office drivers was another reason the university decided that everyone would use 32-bit SPSS.  (Going to 64-bit Office was not an option, apparently.)  

Bruce Weaver wrote

Good guess, David.  Last year, I had a 64-bit version of v21.  But this year, the university (in its wisdom) decided we would have access to 32-bit versions only (v22).  Apparently, a few too many folks had problems with the 64-bit version last year, and the IT folks decided they'd have fewer complaints to deal with if everyone had the 32-bit version!  

David Marso wrote

Bruce, are you still running a 32 bit version of Windows or SPSS? ;-(
---

Bruce Weaver wrote

The syntax offered in Andy W's post still works on my machine with 10 million records.  I.e., I changed this...

SAVE UNIFORM(100,5) /OUTFILE = *.

...to this:

SAVE UNIFORM(10000000,5) /OUTFILE = *.

But when I tried 100 million records, I got the following error message:

Run MATRIX procedure:
Bytes requested = 4000000000
>Error encountered in source line #  3361
 
>Error # 12477
>MATRIX could not allocate memory for an object.  Reduce problem size, or
>release unused matrices using RELEASE statement.  Use DISPLAY statement to
>list all allocated objects.  .
>Execution of this command stops.

HTH.

Maguin, Eugene wrote

What's wrong with the suggestions offered? Is it that none of the methods can process millions of records? You asked for a function but there is no function. People offered alternatives.
Gene Maguin


-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of fevziesen
Sent: Thursday, February 19, 2015 2:32 PM
To: [hidden email]
Subject: Re: Variance-Covariance Matrix

Thanks for all posts. But I meant variance-covariance matrix.In R, there is a command vcov in stats pack but it is useless when you reach a limit of data like a million..



--
View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Variance-Covariance-Matrix-tp5728717p5728737.html
Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD