Overview Bootstrapping, also known as resampling or Monte Carlo estimation, uses brute computer power to estabish confidence intervals for any test statistic, based not on assumptions such as multivariate normal distribution, but rather based on repeated samples from the researcher's own data. As such it is a nonparametric method of statistical inference. That is, rather than use generic distribution tables (ex., normal distribution tables) to compute approximate p probability values, resampling generates a unique sampling distribution based on the actual data at hand and uses experimental rather than analytic methods. Unlike approximation with generic distribution tables, resampling yields unbiased estimates because it is based on unbiased samples of all possible outcomes in the data being studied. However, there is an increased danger of overfitting to noise in the data, a problem which may be addressed by combining bootstrapping methods with cross-validation.

Contents Key concepts and terms Assumptions Frequently asked questions Bibliography




Key Concepts and Terms

• Bootstrapping uses repeated samples from the same original data sample to compute some test statistic. The distribution of this statistic as computed for a very large number of runs of the resampling process is used to estimate the variance of the statistic in the underlying population, thereby allowing the significance of the statistic to be estimated.

• Resampling is the method used in bootstrapping to draw repeated samples from the original data sample. The usual method is to select random cases with replacement from the original data sample, such that each drawn sample has the same number of cases as the original data sample. Because of replacement, the drawn samples used by resampling methods will include repeated cases.

• Monte Carlo estimation. Monte Carlo simulation methods can be used to obtain resampling results. In SPSS, with the Exact Tests module installed, the researcher enters the desired confidence level, the number of samples to be computed, and a random number seed. To duplicate Monte Carlo results, enter the same random seed each time. In SPSS, the random number seed must be set in syntax (ex., SET RNC=MT MTINDEX=-12345.678.)

• Histogram of z values. Computer packages like Resampling Stats will run a thousand or more samples of a researcher-supplied dataset, each time generating a statistic (ex., correlation) requested by the researcher. It will then present the results in a bar chart (the histogram) which shows the sampling distribution of that statistic for the data at hand. Each bar may be called a "bin." There will be a table of the values of the statistic (ex., correlation) at bin centers, with percent and cumulative percent. As with a normal distribution table, one may find the value of the statistic at the point at which the cumulative percent is .05. Suppose that point was r = .30. We could then state that any correlation higher than .30 would be significant at the .05 level.




Assumptions

• Nonparametric. This is an experimental approach which sidesteps all parametric assumptions about the nature of the underlying data stribution.

• Sample size. While there is no specific sample size requirement, the larger the sample, the more reliable the confidence intervals generated by resampling. Likewise, the smaller the sample size, the more likely that confidence limits will be fitted misleadingly to noise in the data rather than to a true underlying distribution.




Frequently Asked Questions

• What software is available for bootstrapping?
  A convenient package for resampling a wide range of test statistics is Resampling Statistics (Resampling Stats Inc., 612 N. Jackson Ave., Arlington, VA 22201; email stats@resample.com; or visit http://www.statistics.com). This is reviewed in Social Science Computer Review, Vol. 15, No. 3 (Fall, 1997:: 305-308.

• Where is bootstrapping in SPSS?
  1. Exact tests. If your installation includes the add-on Exact Tests module, the dialog for most tests (ex., the runs test or the one-sample Kolmogorov-Smirnov goodness-of-fit test) displays an "Exact": button, allowing a choice of types of significance estimates, one of which is "Monte Carlo", which is a bootstrapping method.
     

  2. Nonlinear regression. The nonlinear regression procedure contains a BOOTSTRAP subcommand This subcommand generates 10*p*(p+1)/2 samples, where p = number of parameters, generating nonlinear regression parameter estimates for these repeated samples.

     Example
     MODEL PROGRAM A=.5 B=1.6.
     COMPUTE PSTOP=A*SPEED**B.
     **** Generate nonlinear parameter estimates with standard errors, confidenc intervals and parameter correlations
     **** Save the estimates in a file called PARAM
     CNLR STOP /BOOTSTRAP /OUTFILE=PARAM.
     **** GET retrieves the saved PARAM system file
     GET FILE=PARAM.
     **** LIST lists the sample estimates
     LIST.
     COMPUTE ID=$CASENUM.
     SELECT IF (ID > 1).
     **** FREQUENCIES generates histograms for the bootstrapped parameter estimates
     FREQUENCIES A B /FORMAT=NOTABLE /HISTOGRAM.
     

  3. Macros. The SPSS macro language can be used to generate bootstrap estimates for any test statistic, as discussed below.

  
  

• What is the command code for doing bootstrapping in SPSS on other test statistics?
  The code below, supplied by SPSS Inc., computes bootstrap estimates for regression as a test statistic, using employee data.sav, an example file supplied with SPSS. The code draws repeated random samples with replacement, runs the regression command on each random sample, uses the OMS command to save regression coefficients to a data file, and then uses the frequencies command to create histographs of regression coefficient distributions along with a table of confidence intervals.

  ***oms_bootstrapping.sps***.
  ***if c:\temp is not a valid drive\path, replace all instances of c:\temp
     with a valid drive\path.
  
  PRESERVE.
  SET TVARS NAMES.
  
  *first OMS command just suppresses Viewer output.
  OMS /DESTINATION VIEWER=NO /TAG='suppressall'.
  
  *select regression coefficients tables and write to data file.
  OMS /SELECT TABLES
      /IF COMMANDS=['Regression'] SUBTYPES=['Coefficients']
      /DESTINATION FORMAT=SAV OUTFILE='c:\temp\temp.sav'
     /COLUMNS DIMNAMES=[ 'Variables'  'Statistics']
     /TAG='reg_coeff'.
  
  *define a macro to draw samples with replacement and
   run Regression commands.
  DEFINE regression_bootstrap (samples=!TOKENS(1)
                                       /depvar=!TOKENS(1)
                                       /indvars=!CMDEND)
  
  COMPUTE dummyvar=1.
  AGGREGATE
    /OUTFILE = * MODE = ADDVARIABLES
    /BREAK=dummyvar
    /filesize=N.
  !DO !other=1 !TO !samples
  SET SEED RANDOM.
  WEIGHT OFF.
  FILTER OFF.
  DO IF $casenum=1.
  - COMPUTE #samplesize=filesize.
  - COMPUTE #filesize=filesize.
  END IF.
  DO IF (#samplesize>0 and #filesize>0).
  - COMPUTE sampleWeight=rv.binom(#samplesize, 1/#filesize).
  - COMPUTE #samplesize=#samplesize-sampleWeight.
  - COMPUTE #filesize=#filesize-1.
  ELSE.
  - COMPUTE sampleWeight=0.
  END IF.
  WEIGHT BY sampleWeight.
  FILTER BY sampleWeight.
  REGRESSION
    /STATISTICS COEFF
    /DEPENDENT !depvar
    /METHOD=ENTER !indvars.
  !DOEND
  !ENDDEFINE.
  
  g***insert any valid path\data file name***.
  GET FILE='c:\Program Files\SPSS13\Employee data.sav'.
  
  ***Call the macro, and specify number of samples,
     dependent variable, and independent variables.
  regression_bootstrap
     samples=100
     depvar=salary
     indvars=salbegin  jobtime .
  
  OMSEND.
  
  GET FILE 'c:\temp\temp.sav'.
  
  FREQUENCIES
    VARIABLES=salbegin_B salbegin_Beta jobtime_B jobtime_Beta
    /FORMAT NOTABLE
    /PERCENTILES= 2.5 97.5
    /HISTOGRAM NORMAL.
  
  RESTORE.
  




Bibliography

• Cirincione, C. and G. A. Gurrieri (1997). Research methodology: Computer intensive methods in the social iences. Social Science Computer Review, Vol. 15, No. 1: 83-97. A good overview of resampling and related methods.

• Efron, Bradley (1982). The jackknife, the bootstrap, and other resampling plans. SIAM. This is the classic reference.

Copyright 1998, 2008 by G. David Garson.
Last updated 6/21/08.