Key Takeaways: The Bootstrap and Simulation-Based Inference

One-Sentence Summary

The bootstrap approximates the sampling distribution of any statistic by resampling from the original data with replacement, enabling confidence intervals and standard errors for statistics (like medians, ratios, and correlations) that have no formula-based solutions — while permutation tests provide a distribution-free alternative to hypothesis testing by simulating the null hypothesis directly through random label shuffling.

Core Concepts at a Glance

The Bootstrap Procedure

Step by Step

1. Compute the statistic of interest from the original sample: $\hat{\theta}$
2. Resample $n$ observations from the sample with replacement to create a bootstrap sample
3. Compute the statistic from the bootstrap sample: $\hat{\theta}^*$
4. Repeat steps 2–3 a total of $B = 10{,}000$ times
5. Analyze the bootstrap distribution: - Bootstrap SE = standard deviation of $\hat{\theta}^*_1, \ldots, \hat{\theta}^*_B$ - 95% CI = 2.5th to 97.5th percentile of the bootstrap distribution

Key Python Code

import numpy as np

# Bootstrap CI for ANY statistic
data = np.array([...])
n_boot = 10000

boot_stats = np.array([
    np.median(np.random.choice(data, size=len(data), replace=True))
    for _ in range(n_boot)
])

ci_lower = np.percentile(boot_stats, 2.5)
ci_upper = np.percentile(boot_stats, 97.5)
boot_se = np.std(boot_stats)

The Permutation Test Procedure

Step by Step

1. Compute the observed difference between groups: $\hat{\theta}_{obs}$
2. Combine all observations into one pool
3. Shuffle and randomly split into two groups of the original sizes
4. Compute the test statistic for the shuffled data
5. Repeat steps 3–4 for $B = 10{,}000$ shuffles
6. P-value = proportion of shuffled statistics $\geq \hat{\theta}_{obs}$ (one-sided) or $|\text{shuffled}| \geq |\hat{\theta}_{obs}|$ (two-sided)

Key Python Code

# Permutation test for two groups
combined = np.concatenate([group1, group2])
n1 = len(group1)
obs_diff = np.mean(group2) - np.mean(group1)

perm_diffs = np.array([
    np.mean(np.random.permutation(combined)[n1:]) -
    np.mean(np.random.permutation(combined)[:n1])
    for _ in range(10000)
])

p_value = np.mean(np.abs(perm_diffs) >= np.abs(obs_diff))  # two-sided

Bootstrap CI Methods

Formula-Based vs. Simulation-Based: Quick Comparison

When to Use What

Bootstrap Shines

• CIs for complex statistics: medians, ratios, correlations, percentiles, IQR
• Non-normal data with moderate samples ($15 \leq n < 30$)
• When you want to avoid distributional assumptions
• Checking formula-based results for robustness

Bootstrap Struggles

• Very small samples ($n < 15$) — sample may not represent the population
• Heavily biased samples — bootstrap can't fix systematic errors
• Extreme quantiles (min, max, 99th percentile) — poorly represented in sample
• Time series / dependent data — standard bootstrap assumes independence

Common Misconceptions

How This Chapter Connects

The Key Themes

Theme 3: Modern computational approaches. The bootstrap and permutation test represent a philosophical shift in statistics — from deriving formulas to simulating distributions. Both methods became practical only with the rise of computing power in the 1980s and 1990s. Today, they are standard tools in every data scientist's toolkit.

Theme 1: Computers make statistics more intuitive. For many students, the bootstrap is easier to understand than the CLT-based approach. Instead of abstract theorems about sampling distributions, the bootstrap says: "just resample and see what happens." The idea is concrete, visual, and programmable. If formula-based inference felt mysterious, simulation-based inference may feel like solid ground.

The Threshold Concept: Resampling

The sample is our best model of the population. By resampling from the sample with replacement, we can approximate the sampling distribution of any statistic. This isn't circular — it's the same insight that underlies all of inference: the sample contains information about its own reliability.

The One Thing to Remember

If you forget everything else from this chapter, remember this:

The bootstrap works by resampling from your data with replacement, computing your statistic each time, and using the resulting distribution to build a confidence interval. It works for ANY statistic — means, medians, ratios, correlations, anything — because it doesn't need a formula for the standard error. The permutation test works by shuffling group labels to simulate what "no difference" looks like, then checking whether your observed difference is extreme. Both methods are computational alternatives to formula-based inference: they make fewer assumptions and handle more statistics, but they require a computer. When formula-based and simulation-based methods both apply, they should give similar answers. Use the bootstrap when you need a CI for a non-standard statistic. Use the permutation test when you're not sure whether the $t$-test assumptions hold.

Key Terms