Inclusion of IDF functions in matrix language

[Bruce Weaver]

I've just discovered that IDF functions are not available in the matrix language, and I also see there is a Technote about this at the IBM website (link below).  Just wondering if anyone can tell me when IDFs might be expected to appear.  

   https://www-304.ibm.com/support/docview.wss?uid=swg21486071

I was hoping to use critical t-values to compute 95% confidence intervals, but with no IDF.T function, it's a bit tricky.  I suppose I can find a fairly close approximation to the critical value by sticking the TCDF function inside a loop.  But that seems awfully clumsy.  

Thanks,
Bruce

[David Marso]

I created some SPSS Macros for a client awhile back and found the same issue and was not a happy camper.  Being the intrepid beast I am, I hacked my way through by computing the critical T in SPSS before entering MATRIX and then fetching it as a matrix from the MATRIX language.  If you can determine the df ahead of time independent of MATRIX then it should be easy to do (fugly hack, but if you can't bring the mountain to Marso, bring Marso to the mountain or something like that).  You could also likely find some reasonable approximation formulas in  JAS algorithms eg http://lib.stat.cmu.edu/apstat/27 (or other source). Shouldn't be difficult to translate that ugly FORTRAN in AS27.  I didn't feel justified spending the client's money billing for the translation, hence the hack.

[David Marso]

If you don't know the DF ahead of time then...
Before MATRIX.
Untested in current form:
COMPUTE @T_CRIT@=IDF.T(.975,$CASENUM).
*Could save calcs by conditioning on $CASENUM < say 200 and then Lagging the IDF.Normal for larger df..
....
MATRIX.
......
.....
GET TCRITICAL  / VARIABLES=@T_CRIT@           / FILE * .
COMPUTE DF= whatever....
COMPUTE TCrVal=TCRITICAL(DF,1).
COMPUTE HalfInterval=TCrVal * StandardErrorOfWhateverYouAreComputing.
............
END MATRIX.
HTH, David

[Bruce Weaver]

In this case, the df are known in advance, so a variation on this theme will work.  Thanks David.

David Marso wrote

If you don't know the DF ahead of time then...
Before MATRIX.
Untested in current form:
COMPUTE @T_CRIT@=IDF.T(.975,$CASENUM).
*Could save calcs by conditioning on $CASENUM < say 200 and then Lagging the IDF.Normal for larger df..
....
MATRIX.
......
.....
GET TCRITICAL  / VARIABLES=@T_CRIT@           / FILE * .
COMPUTE DF= whatever....
COMPUTE TCrVal=TCRITICAL(DF,1).
COMPUTE HalfInterval=TCrVal * StandardErrorOfWhateverYouAreComputing.
............
END MATRIX.
HTH, David

[Marta Garcia-Granero]

El 09/08/2011 15:02, Bruce Weaver escribió:
> In this case, the df are known in advance, so a variation on this theme will
> work.  Thanks David.

Hi Bruce:

This is the second approach (using a loop to approximate the value):

MATRIX.
blah
blah

loop tvalue95=1960 to 12706.
. compute t95=tvalue95/1000.
. compute onetail=1-tcdf(t95,df1).
. do if abs(onetail-0.025) lt 0.00005.
.  break.
. end if.
end loop.


blah
blah


END MATRIX.


It computes the critical t value with low error

HTH,
Marta GG

d the command INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[David Marso]

Bruce and Marta,
Maybe even better would be to just use a look-up and then if necessary loop to interpolate the next value?
MATRIX.
Blah blah...

* Storing Integer df 1...30,40,50,60,80,100,120, inf.
COMPUTE TCrit975= {12.710,4.303,3.182,2.776,2.571,2.447,2.365,2.306,2.262,2.228,2.201,2.179,2.160,2.145,2.131,2.120,2.110,2.101,2.093,2.086,2.080,2.074,2.069,2.064,2.060,2.056,2.052,2.048,2.045,2.042,2.021,2.009,2.000,1.990,1.984,1.980,1.960 }.
Do IF df <=30.
COMPUTE tval975=TCrit975(df).
ElseIf blahblah
....
....
Do a loop for any interpolated values...
Left as exercise to resolve remaining moving parts.
David

Marta García-Granero-2 wrote

El 09/08/2011 15:02, Bruce Weaver escribió:
> In this case, the df are known in advance, so a variation on this theme will
> work.  Thanks David.

Hi Bruce:

This is the second approach (using a loop to approximate the value):

MATRIX.
blah
blah

loop tvalue95=1960 to 12706.
. compute t95=tvalue95/1000.
. compute onetail=1-tcdf(t95,df1).
. do if abs(onetail-0.025) lt 0.00005.
.  break.
. end if.
end loop.


blah
blah


END MATRIX.


It computes the critical t value with low error

HTH,
Marta GG

d the command INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Bruce Weaver]

Thanks Marta.  I was going to have the user work out the critical t and hand it to the macro.  But I'm changing my mind as I write this.  Using the looping method removes one potential source of user-supplied error!  And in some tests I've done, it gives results that are certainly close enough -- or CEFGW, as David might put it.  ;-)

Cheers,
Bruce

Marta García-Granero-2 wrote

El 09/08/2011 15:02, Bruce Weaver escribió:
> In this case, the df are known in advance, so a variation on this theme will
> work.  Thanks David.

Hi Bruce:

This is the second approach (using a loop to approximate the value):

MATRIX.
blah
blah

loop tvalue95=1960 to 12706.
. compute t95=tvalue95/1000.
. compute onetail=1-tcdf(t95,df1).
. do if abs(onetail-0.025) lt 0.00005.
.  break.
. end if.
end loop.


blah
blah


END MATRIX.


It computes the critical t value with low error

HTH,
Marta GG

d the command INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[David Marso]

Marta's code as it stands seems rather computationally intensive (especially for small df) .  Using a loop-up table to locate the closest tabled value followed by a loop between the next tabled value should get you very close.  A little more logic and code involved but cleaner in some ways.
--

Bruce Weaver wrote

Thanks Marta.  I was going to have the user work out the critical t and hand it to the macro.  But I'm changing my mind as I write this.  Using the looping method removes one potential source of user-supplied error!  And in some tests I've done, it gives results that are certainly close enough -- or CEFGW, as David might put it.  ;-)

Cheers,
Bruce

Marta García-Granero-2 wrote

El 09/08/2011 15:02, Bruce Weaver escribió:
> In this case, the df are known in advance, so a variation on this theme will
> work.  Thanks David.

Hi Bruce:

This is the second approach (using a loop to approximate the value):

MATRIX.
blah
blah

loop tvalue95=1960 to 12706.
. compute t95=tvalue95/1000.
. compute onetail=1-tcdf(t95,df1).
. do if abs(onetail-0.025) lt 0.00005.
.  break.
. end if.
end loop.


blah
blah


END MATRIX.


It computes the critical t value with low error

HTH,
Marta GG

d the command INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[David Marso]

Implementation using my previous Table look-up ideas and Marta's LOOP idea.

DEFINE T_IDF (df !TOKENS(1)).
PRESERVE.
SET MXLOOPS = 10000.
MATRIX.
COMPUTE df=!df.
COMPUTE tval975=0.
COMPUTE Start=0.
COMPUTE TCrit975= {12.710,4.303,3.182,2.776,2.571,2.447,2.365,2.306,2.262,2.228,2.201,2.179,2.160,2.145,2.131,2.120,2.110,2.101,2.093,2.086,2.080,2.074,2.069,2.064,2.060,2.056,2.052,2.048,2.045,2.042,2.021,2.009,2.000,1.990,1.984,1.980,1.960 }.
DO IF df <=30.
+  COMPUTE tval975=TCrit975(df).
ELSE IF df=40.
+  COMPUTE tval975=TCrit975(31).
ELSE IF df=50.
+  COMPUTE tval975=TCrit975(32).
ELSE IF df=60.
+  COMPUTE tval975=TCrit975(33).
ELSE IF df=80.
+  COMPUTE tval975=TCrit975(34).
ELSE IF df=100.
+  COMPUTE tval975=TCrit975(35).
ELSE IF df=120.
+  COMPUTE tval975=TCrit975(36).
ELSE IF (df GE 31) AND (df LE 39).
+  COMPUTE Start=30.
ELSE IF (df GE 41) AND (df LE 49).
+  COMPUTE Start=31.
ELSE IF (df GE 51) AND (df LE 59).
+  COMPUTE Start=32.
ELSE IF (df GE 61) AND (df LE 79).
+  COMPUTE Start=33.
ELSE IF (df GE 81) AND (df LE 99).
+  COMPUTE Start=34.
ELSE IF (df GE 101) AND (df LE 119).
+  COMPUTE Start=35.
ELSE IF (df > 120).
+  COMPUTE Start=36.
END IF.
* Increase Multiplier and or reduce tolerance check for greater accuracy .
DO IF (Start NE 0).
+  LOOP TCheck=TCrit975(START+1)*10000  TO TCrit975(START)*10000.
+    COMPUTE tval=1-tcdf(tCheck/10000,df).
+    DO IF abs(tval-0.025) lt 0.000005.
+      COMPUTE tval975=TCheck/10000.
+      BREAK.
+    END IF.
+  END LOOP.
END IF.
PRINT tval975.
END MATRIX.
RESTORE.
!ENDDEFINE.
T_IDF df=63.

[David Marso]

Another version which is a little more concise and more exact.  I'm not sure this is necessarily more readable but I find long lists of conditionals to be sometimes boring.  Wish there were a way of doing something like the following more efficiently.  As a final thought, one could write an outer loop to enclose the inner loop and increase the multiplier/decrease the tolerance and achieve an arbitrary degree of accuracy.
Note the use of MAX(list of boolean*index).  Rather clever if I don't say so myself ;-)
----
DEFINE T_IDF (df !TOKENS(1)).
PRESERVE.
SET MXLOOPS = 10000.
MATRIX.
COMPUTE TCrit975= {12.70620,4.30265,3.18245,2.77645,2.57058,2.44691,2.36462,2.30600,2.26216,2.22814,2.20099,2.17881,2.16037,2.14479,2.13145,2.11991,2.10982,2.10092,2.09302,2.08596,2.07961,2.07387,2.06866,2.06390,2.05954,2.05553,2.05183,2.04841,2.04523,2.04227,2.02108,2.00856,2.00030,1.99006,1.98397,1.97993,1.96000 }.
COMPUTE df=!df.
COMPUTE Start=0.
DO IF df <=30.
+  COMPUTE tval975=TCrit975(df).
ELSE IF (df=40 OR df=50 OR df=60 OR df=80 OR df=100 OR df=120).
+  COMPUTE tval975=MMAX({TCrit975(31)*(df=40),TCrit975(32)*(df=50),TCrit975(33)*(df=60),TCrit975(34)*(df=80),TCrit975(35)*(df=100),TCrit975(36)*(df=120)}).
ELSE.
+  COMPUTE START=MMAX({(df GE 31 AND df LE 39)*30,(df GE 41 AND df LE 49)*31,(df GE 51 AND df LE 59)*32,(df GE 61 AND df LE 79)*33, (df GE 81 AND df LE 99)*34,(df GE 101 AND df LE 119)*35,(df > 120)*36}).
END IF.

DO IF (Start NE 0).
* Loop code adapted from Marta GarcÌa-Granero -I increased Multiplier (10000) and decreased tolerance (.000005).
* http://spssx-discussion.1045642.n5.nabble.com/Inclusion-of-IDF-functions-in-matrix-language-td4680040.html.
+  LOOP TCheck=TCrit975(START+1)*10000  TO TCrit975(START)*10000.
+    DO IF abs(tcdf(tCheck/10000,df)-0.975) lt 0.000005.
+      COMPUTE tval975=TCheck/10000.
+      BREAK.
+    END IF.
+  END LOOP.
END IF.
PRINT tval975.
END MATRIX.
RESTORE.
!ENDDEFINE.
T_IDF df=66.

[David Marso]

Thinking further on this the following occurred to me:
MAX (Exclusive List of Bln*Indexes)=SUM(Exclusive List of Bln*Indexes)=VectorOfBln * VectorOfIndexes.
Hmmm, why not do this as  MATRIX products? ;-)))
Bearing the following fruit:
---------------------------------------
DEFINE T_IDF (df !TOKENS(1)).
PRESERVE.
SET MXLOOPS = 10000.
MATRIX.
* Loop code adapted from Marta García-Granero -I increased Multiplier (10000) and decreased tolerance (.000005).
* http://spssx-discussion.1045642.n5.nabble.com/Inclusion-of-IDF-functions-in-matrix-language-td4680040.html.

COMPUTE MULT=10000.
COMPUTE TOL=.0000005.
COMPUTE TCrit975={12.70620,4.30265,3.18245,2.77645,2.57058,2.44691,2.36462,2.30600,2.26216,2.22814,2.20099,2.17881,2.16037,2.14479,2.13145,2.11991,2.10982,2.10092,2.09302,2.08596,2.07961,2.07387,2.06866,2.06390,2.05954,2.05553,2.05183,2.04841,2.04523,2.04227,2.02108,2.00856,2.00030,1.99006,1.98397,1.97993,1.96000 }.
COMPUTE df=!df.
COMPUTE Start=0.
COMPUTE BLNTBLDF=T({df EQ 40, df EQ 50, df EQ 60, df EQ 80, df EQ 100, df EQ 120}).
DO IF (df LE 30).
+  COMPUTE tval975=TCrit975(df).
ELSE IF MMAX(BLNTBLDF) GT 0.
+  COMPUTE tval975=TCrit975(31:36) * BLNTBLDF.
ELSE.
+  COMPUTE START= {30:36} * {df GE 31 AND df LE 39; df GE 41 AND df LE 49; df GE 51 AND df LE 59; df GE 61 AND df LE 79; df GE 81 AND df LE 99; df GE 101 AND df LE 119; df > 120}  .
END IF.
******************.
DO IF (Start NE 0).
+  LOOP TCheck=TCrit975(START+1)*MULT  TO TCrit975(START)*MULT.
+    DO IF abs(tcdf(tCheck/MULT,df)-0.975) lt TOL.
+      COMPUTE tval975=TCheck/MULT.
+      BREAK.
+    END IF.
+  END LOOP.
END IF.
PRINT tval975.
END MATRIX.
RESTORE.
!ENDDEFINE.
T_IDF df=43.

David Marso wrote

Another version which is a little more concise and more exact.  I'm not sure this is necessarily more readable but I find long lists of conditionals to be sometimes boring.  Wish there were a way of doing something like the following more efficiently.  As a final thought, one could write an outer loop to enclose the inner loop and increase the multiplier/decrease the tolerance and achieve an arbitrary degree of accuracy.
Note the use of MAX(list of boolean*index).  Rather clever if I don't say so myself ;-)
----
DEFINE T_IDF (df !TOKENS(1)).
PRESERVE.
SET MXLOOPS = 10000.
MATRIX.
COMPUTE TCrit975= {12.70620,4.30265,3.18245,2.77645,2.57058,2.44691,2.36462,2.30600,2.26216,2.22814,2.20099,2.17881,2.16037,2.14479,2.13145,2.11991,2.10982,2.10092,2.09302,2.08596,2.07961,2.07387,2.06866,2.06390,2.05954,2.05553,2.05183,2.04841,2.04523,2.04227,2.02108,2.00856,2.00030,1.99006,1.98397,1.97993,1.96000 }.
COMPUTE df=!df.
COMPUTE Start=0.
DO IF df <=30.
+  COMPUTE tval975=TCrit975(df).
ELSE IF (df=40 OR df=50 OR df=60 OR df=80 OR df=100 OR df=120).
+  COMPUTE tval975=MMAX({TCrit975(31)*(df=40),TCrit975(32)*(df=50),TCrit975(33)*(df=60),TCrit975(34)*(df=80),TCrit975(35)*(df=100),TCrit975(36)*(df=120)}).
ELSE.
+  COMPUTE START=MMAX({(df GE 31 AND df LE 39)*30,(df GE 41 AND df LE 49)*31,(df GE 51 AND df LE 59)*32,(df GE 61 AND df LE 79)*33, (df GE 81 AND df LE 99)*34,(df GE 101 AND df LE 119)*35,(df > 120)*36}).
END IF.

DO IF (Start NE 0).
* Loop code adapted from Marta GarcÌa-Granero -I increased Multiplier (10000) and decreased tolerance (.000005).
* http://spssx-discussion.1045642.n5.nabble.com/Inclusion-of-IDF-functions-in-matrix-language-td4680040.html.
+  LOOP TCheck=TCrit975(START+1)*10000  TO TCrit975(START)*10000.
+    DO IF abs(tcdf(tCheck/10000,df)-0.975) lt 0.000005.
+      COMPUTE tval975=TCheck/10000.
+      BREAK.
+    END IF.
+  END LOOP.
END IF.
PRINT tval975.
END MATRIX.
RESTORE.
!ENDDEFINE.
T_IDF df=66.

[Marta Garcia-Granero]

Very clever. Thanks for sharing :-)

Best regards,
Marta GG

El 10/08/2011 14:28, David Marso escribió:

> Bearing the following fruit:
> ---------------------------------------
> DEFINE T_IDF (df !TOKENS(1)).
> PRESERVE.
> SET MXLOOPS = 10000.
> MATRIX.
> * Loop code adapted from Marta García-Granero -I increased Multiplier
> (10000) and decreased tolerance (.000005).
> *
> http://spssx-discussion.1045642.n5.nabble.com/Inclusion-of-IDF-functions-in-matrix-language-td4680040.html.
>
> COMPUTE MULT=10000.
> COMPUTE TOL=.0000005.
> COMPUTE
> TCrit975={12.70620,4.30265,3.18245,2.77645,2.57058,2.44691,2.36462,2.30600,2.26216,2.22814,2.20099,2.17881,2.16037,2.14479,2.13145,2.11991,2.10982,2.10092,2.09302,2.08596,2.07961,2.07387,2.06866,2.06390,2.05954,2.05553,2.05183,2.04841,2.04523,2.04227,2.02108,2.00856,2.00030,1.99006,1.98397,1.97993,1.96000
> }.
> COMPUTE df=!df.
> COMPUTE Start=0.
> COMPUTE BLNTBLDF=T({df EQ 40, df EQ 50, df EQ 60, df EQ 80, df EQ 100, df EQ
> 120}).
> DO IF (df LE 30).
> +  COMPUTE tval975=TCrit975(df).
> ELSE IF MMAX(BLNTBLDF) GT 0.
> +  COMPUTE tval975=TCrit975(31:36) * BLNTBLDF.
> ELSE.
> +  COMPUTE START= {30:36} * {df GE 31 AND df LE 39; df GE 41 AND df LE 49;
> df GE 51 AND df LE 59; df GE 61 AND df LE 79; df GE 81 AND df LE 99; df GE
> 101 AND df LE 119; df>  120}  .
> END IF.
> ******************.
> DO IF (Start NE 0).
> +  LOOP TCheck=TCrit975(START+1)*MULT  TO TCrit975(START)*MULT.
> +    DO IF abs(tcdf(tCheck/MULT,df)-0.975) lt TOL.
> +      COMPUTE tval975=TCheck/MULT.
> +      BREAK.
> +    END IF.
> +  END LOOP.
> END IF.
> PRINT tval975.
> END MATRIX.
> RESTORE.
> !ENDDEFINE.
> T_IDF df=43.
>
>
>
>
> David Marso wrote:
>> Another version which is a little more concise and more exact.  I'm not
>> sure this is necessarily more readable but I find long lists of
>> conditionals to be sometimes boring.  Wish there were a way of doing
>> something like the following more efficiently.  As a final thought, one
>> could write an outer loop to enclose the inner loop and increase the
>> multiplier/decrease the tolerance and achieve an arbitrary degree of
>> accuracy.
>> Note the use of MAX(list of boolean*index).  Rather clever if I don't say
>> so myself ;-)
>> ----
>> DEFINE T_IDF (df !TOKENS(1)).
>> PRESERVE.
>> SET MXLOOPS = 10000.
>> MATRIX.
>> COMPUTE TCrit975=
>> {12.70620,4.30265,3.18245,2.77645,2.57058,2.44691,2.36462,2.30600,2.26216,2.22814,2.20099,2.17881,2.16037,2.14479,2.13145,2.11991,2.10982,2.10092,2.09302,2.08596,2.07961,2.07387,2.06866,2.06390,2.05954,2.05553,2.05183,2.04841,2.04523,2.04227,2.02108,2.00856,2.00030,1.99006,1.98397,1.97993,1.96000
>> }.
>> COMPUTE df=!df.
>> COMPUTE Start=0.
>> DO IF df<=30.
>> +  COMPUTE tval975=TCrit975(df).
>> ELSE IF (df=40 OR df=50 OR df=60 OR df=80 OR df=100 OR df=120).
>> +  COMPUTE
>> tval975=MMAX({TCrit975(31)*(df=40),TCrit975(32)*(df=50),TCrit975(33)*(df=60),TCrit975(34)*(df=80),TCrit975(35)*(df=100),TCrit975(36)*(df=120)}).
>> ELSE.
>> +  COMPUTE START=MMAX({(df GE 31 AND df LE 39)*30,(df GE 41 AND df LE
>> 49)*31,(df GE 51 AND df LE 59)*32,(df GE 61 AND df LE 79)*33, (df GE 81
>> AND df LE 99)*34,(df GE 101 AND df LE 119)*35,(df>  120)*36}).
>> END IF.
>>
>> DO IF (Start NE 0).
>> * Loop code adapted from Marta GarcÌa-Granero -I increased Multiplier
>> (10000) and decreased tolerance (.000005).
>> *
>> http://spssx-discussion.1045642.n5.nabble.com/Inclusion-of-IDF-functions-in-matrix-language-td4680040.html.
>> +  LOOP TCheck=TCrit975(START+1)*10000  TO TCrit975(START)*10000.
>> +    DO IF abs(tcdf(tCheck/10000,df)-0.975) lt 0.000005.
>> +      COMPUTE tval975=TCheck/10000.
>> +      BREAK.
>> +    END IF.
>> +  END LOOP.
>> END IF.
>> PRINT tval975.
>> END MATRIX.
>> RESTORE.
>> !ENDDEFINE.
>> T_IDF df=66.
>>
>
> --
> View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Inclusion-of-IDF-functions-in-matrix-language-tp4680040p4685640.html
> Sent from the SPSSX Discussion mailing list archive at Nabble.com.
>
> =====================
> To manage your subscription to SPSSX-L, send a message to
> [hidden email] (not to SPSSX-L), with no body text except the
> command. To leave the list, send the command
> SIGNOFF SPSSX-L
> For a list of commands to manage subscriptions, send the command
> INFO REFCARD
>

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD