Bootstrap sampling for evaluating hypothesis tests

[Margaret MacDougall]

Hello

 

I would be grateful to learn from any list users who have used SPSS successfully for bootstrap sampling to assess robustness to Type I errors of a proposed new hypothesis test.

 

Many thanks in advance

 

Best wishes

 

Margaret

[Poes, Matthew Joseph]

I’ve used bootstrap resampling quite a bit, and I’ve used it in SPSS.  I would need to know a bit more about what you are thinking here.  Basically the way to accomplish what you are trying to accomplish is to compare the findings of the bootrstrapped CI to the standard estimate and if they are consistent, then you can be more assured the findings are robust to type I error.  In general bootstrapping is more robust in its own right to type I error, so there is an argument that inconsistent results are meaningless, other than to say that the standard estimates were subject to bias and failed to meet certain assumptions.  In other words, the bootstrapped estimate is always a better estimate, so even if its inconsistent with the standard estimate, it still can be trusted in its own right. 

 

-Matt

 

From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Margaret MacDougall
Sent: Wednesday, March 13, 2013 10:14 AM
To: [hidden email]
Subject: Bootstrap sampling for evaluating hypothesis tests

 

Hello

 

I would be grateful to learn from any list users who have used SPSS successfully for bootstrap sampling to assess robustness to Type I errors of a proposed new hypothesis test.

 

Many thanks in advance

 

Best wishes

 

Margaret

[Art Kendall]

Open SPSS
click <help> <topics>
type "bootstrap" into the edit box
click <BOOTSTRAP>
click <overview>
On 3/13/2013 11:13 AM, Margaret MacDougall wrote:

Hello

 

I would be grateful to learn from any list users who have used SPSS successfully for bootstrap sampling to assess robustness to Type I errors of a proposed new hypothesis test.

 

Many thanks in advance

 

Best wishes

 

Margaret

===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD

[Margaret MacDougall]

Dear Matt

Thanks for your message. My interest in bootsrapping stems from the fact that I have developed a hyothesis test which can be justified on theoretical grounds for sufficiently large sample size and I would like to see how well it behaves for a range of smaller sample sizes. I expect that this would entail generating multiple samples for which the null hypothesis is actually true and using these as a basis for assessing the probability of a Type I error.

I have not used SPSS previously for bootstrapping but I am aware that the standard package offers some level of functionality in this area, via the existing menu commands and that there is a bootstrapping add-on for SPSS. I am unclear how powerful the add-on is in erms of functionality or whether SPSS syntax could cover my needs.

Best wishes

Margaret

From: "Poes, Matthew Joseph" <[hidden email]>
To: [hidden email]
Sent: Wednesday, 13 March 2013, 15:26
Subject: Re: Bootstrap sampling for evaluating hypothesis tests

I’ve used bootstrap resampling quite a bit, and I’ve used it in SPSS.  I would need to know a bit more about what you are thinking here.  Basically the way to accomplish what you are trying to accomplish is to compare the findings of the bootrstrapped CI to the standard estimate and if they are consistent, then you can be more assured the findings are robust to type I error.  In general bootstrapping is more robust in its own right to type I error, so there is an argument that inconsistent results are meaningless, other than to say that the standard estimates were subject to bias and failed to meet certain assumptions.  In other words, the bootstrapped estimate is always a better estimate, so even if its inconsistent with the standard estimate, it still can be trusted in its own right. 

 

-Matt

 

From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Margaret MacDougall
Sent: Wednesday, March 13, 2013 10:14 AM
To: [hidden email]
Subject: Bootstrap sampling for evaluating hypothesis tests

 

Hello

 

I would be grateful to learn from any list users who have used SPSS successfully for bootstrap sampling to assess robustness to Type I errors of a proposed new hypothesis test.

 

Many thanks in advance

 

Best wishes

 

Margaret

[Maguin, Eugene]

Matt and others are more knowledgeable about this question, I’d guess, but this seems to me to be a MonteCarlo problem rather than a bootstrap problem. It sounds like you might fix the numerator of effect size (I’m presuming that the ‘betterness’ of your statistic is related to the standard error calculation) and vary the sample size and the hypothesis is that the test statistic is better than the ‘standard’ statistic. So you need adequate number of replications at each sample size point to assess power.

 

Gene Maguin

 

From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Margaret MacDougall
Sent: Thursday, March 14, 2013 3:18 PM
To: [hidden email]
Subject: Re: Bootstrap sampling for evaluating hypothesis tests

 

Dear Matt

Thanks for your message. My interest in bootsrapping stems from the fact that I have developed a hyothesis test which can be justified on theoretical grounds for sufficiently large sample size and I would like to see how well it behaves for a range of smaller sample sizes. I expect that this would entail generating multiple samples for which the null hypothesis is actually true and using these as a basis for assessing the probability of a Type I error.

I have not used SPSS previously for bootstrapping but I am aware that the standard package offers some level of functionality in this area, via the existing menu commands and that there is a bootstrapping add-on for SPSS. I am unclear how powerful the add-on is in erms of functionality or whether SPSS syntax could cover my needs.

Best wishes

Margaret

 

From: "Poes, Matthew Joseph" <[hidden email]>
To: [hidden email]
Sent: Wednesday, 13 March 2013, 15:26
Subject: Re: Bootstrap sampling for evaluating hypothesis tests

 

I’ve used bootstrap resampling quite a bit, and I’ve used it in SPSS.  I would need to know a bit more about what you are thinking here.  Basically the way to accomplish what you are trying to accomplish is to compare the findings of the bootrstrapped CI to the standard estimate and if they are consistent, then you can be more assured the findings are robust to type I error.  In general bootstrapping is more robust in its own right to type I error, so there is an argument that inconsistent results are meaningless, other than to say that the standard estimates were subject to bias and failed to meet certain assumptions.  In other words, the bootstrapped estimate is always a better estimate, so even if its inconsistent with the standard estimate, it still can be trusted in its own right. 

 

-Matt

 

From: SPSSX(r) Discussion [[hidden email]] On Behalf Of Margaret MacDougall
Sent: Wednesday, March 13, 2013 10:14 AM
To: [hidden email]
Subject: Bootstrap sampling for evaluating hypothesis tests

 

Hello

 

I would be grateful to learn from any list users who have used SPSS successfully for bootstrap sampling to assess robustness to Type I errors of a proposed new hypothesis test.

 

Many thanks in advance

 

Best wishes

 

Margaret

 

[Rich Ulrich]

Gene,
I think you are right is saying that it is a Monte Carlo problem.

But she is assessing the robustness for small samples, not looking
for the power.  That is:  She needs to show that the test does not
reject too many times (or, less problematically, too few) for small
samples.  That is the test under the null.

Of course, if the test itself is a version of bootstrapping, then
there's still a question of bootstrapping.

--
Rich Ulrich

---

Date: Thu, 14 Mar 2013 15:46:05 -0400
From: [hidden email]
Subject: Re: Bootstrap sampling for evaluating hypothesis tests
To: [hidden email]

Matt and others are more knowledgeable about this question, I’d guess, but this seems to me to be a MonteCarlo problem rather than a bootstrap problem. It sounds like you might fix the numerator of effect size (I’m presuming that the ‘betterness’ of your statistic is related to the standard error calculation) and vary the sample size and the hypothesis is that the test statistic is better than the ‘standard’ statistic. So you need adequate number of replications at each sample size point to assess power.

 

Gene Maguin

 

From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Margaret MacDougall
Sent: Thursday, March 14, 2013 3:18 PM
To: [hidden email]
Subject: Re: Bootstrap sampling for evaluating hypothesis tests

 

Dear Matt

Thanks for your message. My interest in bootsrapping stems from the fact that I have developed a hyothesis test which can be justified on theoretical grounds for sufficiently large sample size and I would like to see how well it behaves for a range of smaller sample sizes. I expect that this would entail generating multiple samples for which the null hypothesis is actually true and using these as a basis for assessing the probability of a Type I error.

I have not used SPSS previously for bootstrapping but I am aware that the standard package offers some level of functionality in this area, via the existing menu commands and that there is a bootstrapping add-on for SPSS. I am unclear how powerful the add-on is in erms of functionality or whether SPSS syntax could cover my needs.

Best wishes

Margaret

 ... [snip, previous]

[Art Kendall]

SPSS can be used for many kinds of simulation, bootstrapping, monte carlo, etc.   The vocabulary for this is not very consistent across disciplines.

There are many different kinds of functions for generating random values "*.rv.something(arg1, arg2...).
There are many cumulative and inverse distribution functions.
it also have a vast array of hypothesis tests that you could contrast with your test.


What is the conceptual type of hypothesis you are testing? 
What is the conventional way to test this kind of hypothesis  that you want to show improvement over?
Are there specific assumptions that you want to relax?

What is the part of the test that you want to show is an improvement?


Have you implemented your test in SPSS syntax?




On 3/13/2013 11:13 AM, Margaret MacDougall wrote:

Hello

 

I would be grateful to learn from any list users who have used SPSS successfully for bootstrap sampling to assess robustness to Type I errors of a proposed new hypothesis test.

 

Many thanks in advance

 

Best wishes

 

Margaret

===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD

[David Marso]

FWIW:  We had a rather interesting thread last week on generation of Bootstrap samples.
http://spssx-discussion.1045642.n5.nabble.com/Sampling-WITH-replacement-used-in-bootstrapping-complex-sampling-etc-Demo-syntax-td5718495.html#a5718522

@ Margaret:If you have implemented your Hyp test in SPSS or know MATRIX you could use what is in the cited thread and wrap the whole thing on a macro or python extension.

Art Kendall wrote

SPSS can be used for many kinds of
      simulation, bootstrapping, monte carlo, etc.   The vocabulary for
      this is not very consistent across disciplines.
     
      There are many different kinds of functions for generating random
      values "*.rv.something(arg1, arg2...).
      There are many cumulative and inverse distribution functions.
      it also have a vast array of hypothesis tests that you could
      contrast with your test.
     
     
      What is the conceptual type of hypothesis you are testing? 
      What is the conventional way to test this kind of hypothesis  that
      you want to show improvement over?
      Are there specific assumptions that you want to relax?
     
      What is the part of the test that you want to show is an
      improvement?
     
     
      Have you implemented your test in SPSS syntax?
     
     
     
     
      Art Kendall
Social Research Consultants
      On 3/13/2013 11:13 AM, Margaret MacDougall wrote:
   
   
     
        Hello
         
        I would be grateful to learn from any list users who have
          used SPSS successfully for bootstrap sampling to assess
          robustness to Type I errors of a proposed new hypothesis test.
         
        Many thanks in advance
         
        Best wishes
         
        Margaret
     
   
   
 


=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Margaret MacDougall]

Thanks for your thoughts on the add-on Matt. I am limited to version 19.0 of SPSS and unlikely to have a upgrade to version 21.0 for a while. However, it was valulable to receive your insight on functionality.

 

Best wishes

 

Margaet

From: "Poes, Matthew Joseph" <[hidden email]>
To: "'[hidden email]'" <[hidden email]>; "'[hidden email]'" <[hidden email]>
Sent: Thursday, 14 March 2013, 19:25
Subject: RE: Bootstrap sampling for evaluating hypothesis tests

The Bootstrapping add on was practically rendered pointless when in version 21 they added bootstrapping nearly across the board.  In addition, they offer a simulation system that allows for montecarlo analysis.  I would actually consider the simulation package myself.  It’s now included in version 21 (at least in my version). 

 

Everything that can be done in the menu system can typically be done in the syntax.  Here is an example I generated for bootstrapping with a bias corrected confidence interval and stratified by race.

 

BOOTSTRAP

  /SAMPLING METHOD=STRATIFIED(STRATA=racem ) 

  /VARIABLES TARGET=peaq1r INPUT=  exp07 exp08 

  /CRITERIA CILEVEL=95 CITYPE=BCA  NSAMPLES=10000

  /MISSING USERMISSING=EXCLUDE.

REGRESSION

  /MISSING LISTWISE

  /STATISTICS COEFF OUTS R ANOVA

  /CRITERIA=PIN(.05) POUT(.10)

  /NOORIGIN

  /DEPENDENT peaq1r

  /METHOD=ENTER exp07 exp08.

 

From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Margaret MacDougall
Sent: Thursday, March 14, 2013 2:18 PM
To: [hidden email]
Subject: Re: Bootstrap sampling for evaluating hypothesis tests

 

Dear Matt

Thanks for your message. My interest in bootsrapping stems from the fact that I have developed a hyothesis test which can be justified on theoretical grounds for sufficiently large sample size and I would like to see how well it behaves for a range of smaller sample sizes. I expect that this would entail generating multiple samples for which the null hypothesis is actually true and using these as a basis for assessing the probability of a Type I error.

I have not used SPSS previously for bootstrapping but I am aware that the standard package offers some level of functionality in this area, via the existing menu commands and that there is a bootstrapping add-on for SPSS. I am unclear how powerful the add-on is in erms of functionality or whether SPSS syntax could cover my needs.

Best wishes

Margaret

 

From: "Poes, Matthew Joseph" <[hidden email]>
To: [hidden email]
Sent: Wednesday, 13 March 2013, 15:26
Subject: Re: Bootstrap sampling for evaluating hypothesis tests

 

I’ve used bootstrap resampling quite a bit, and I’ve used it in SPSS.  I would need to know a bit more about what you are thinking here.  Basically the way to accomplish what you are trying to accomplish is to compare the findings of the bootrstrapped CI to the standard estimate and if they are consistent, then you can be more assured the findings are robust to type I error.  In general bootstrapping is more robust in its own right to type I error, so there is an argument that inconsistent results are meaningless, other than to say that the standard estimates were subject to bias and failed to meet certain assumptions.  In other words, the bootstrapped estimate is always a better estimate, so even if its inconsistent with the standard estimate, it still can be trusted in its own right. 

 

-Matt

 

From: SPSSX(r) Discussion [[hidden email]] On Behalf Of Margaret MacDougall
Sent: Wednesday, March 13, 2013 10:14 AM
To: [hidden email]
Subject: Bootstrap sampling for evaluating hypothesis tests

 

Hello

 

I would be grateful to learn from any list users who have used SPSS successfully for bootstrap sampling to assess robustness to Type I errors of a proposed new hypothesis test.

 

Many thanks in advance

 

Best wishes

 

Margaret

 

[Margaret MacDougall]

Dear Ulrich

Thanks for chipping in. I understand that a Monte Carlo approach would be of interest in producing a confidence interval for the proportion of replications in which the test statistic is more extreme than that of the original data. However, in terms of my original query re use of bootstrapping, I have also discovered very recently that Stata covers my needs extrenely well., so much so that I think that this is going to be my choice of package for the tasks in hand.

Best wishes

Margaret

From: Rich Ulrich <[hidden email]>
To: [hidden email]
Sent: Thursday, 14 March 2013, 20:22
Subject: Re: Bootstrap sampling for evaluating hypothesis tests

Gene,
I think you are right is saying that it is a Monte Carlo problem.

But she is assessing the robustness for small samples, not looking
for the power.  That is:  She needs to show that the test does not
reject too many times (or, less problematically, too few) for small
samples.  That is the test under the null.

Of course, if the test itself is a version of bootstrapping, then
there's still a question of bootstrapping.

--
Rich Ulrich

Date: Thu, 14 Mar 2013 15:46:05 -0400
From: [hidden email]
Subject: Re: Bootstrap sampling for evaluating hypothesis tests
To: [hidden email]

Matt and others are more knowledgeable about this question, I’d guess, but this seems to me to be a MonteCarlo problem rather than a bootstrap problem. It sounds like you might fix the numerator of effect size (I’m presuming that the ‘betterness’ of your statistic is related to the standard error calculation) and vary the sample size and the hypothesis is that the test statistic is better than the ‘standard’ statistic. So you need adequate number of replications at each sample size point to assess power.

 

Gene Maguin

 

From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Margaret MacDougall
Sent: Thursday, March 14, 2013 3:18 PM
To: [hidden email]
Subject: Re: Bootstrap sampling for evaluating hypothesis tests

 

Dear Matt

Thanks for your message. My interest in bootsrapping stems from the fact that I have developed a hyothesis test which can be justified on theoretical grounds for sufficiently large sample size and I would like to see how well it behaves for a range of smaller sample sizes. I expect that this would entail generating multiple samples for which the null hypothesis is actually true and using these as a basis for assessing the probability of a Type I error.

I have not used SPSS previously for bootstrapping but I am aware that the standard package offers some level of functionality in this area, via the existing menu commands and that there is a bootstrapping add-on for SPSS. I am unclear how powerful the add-on is in erms of functionality or whether SPSS syntax could cover my needs.

Best wishes

Margaret

 ... [snip, previous]