---
title: "Sampling distributions, CLT, & Bootstrapping (Notes)"
subtitle: "STAT 155"
author: "Your Name"
format:
  html:
    toc: true
    toc-depth: 2
    embed-resources: true
---


```{r setup}
#| include: false
knitr::opts_chunk$set(
  collapse = TRUE, 
  warning = FALSE,
  message = FALSE,
  error = TRUE,
  fig.height = 2.75, 
  fig.width = 4.25,
  fig.env = 'figure',
  fig.pos = 'h',
  fig.align = 'center')
```


::: {.callout-note title = "Learning goals"}

Let $\beta$ be some *population parameter* and $\hat{\beta}$ be a *sample estimate* of $\beta$.
Our goals for the day are to:

- use simulation to solidify our understanding of **sampling distributions** and **standard errors**
- explore and compare two approaches to approximating the sampling distribution of $\hat{\beta}$: 
    - Central Limit Theorem (CLT)
    - bootstrapping
- explore the impact of **sample size** on sampling distributions and standard errors

![](https://ajohns24.github.io/155_Fall_2025/images/concept_map_8.png)


:::

::: {.callout-note title="Additional resources"}

**Required videos**

- [sampling distributions](https://youtu.be/Md0TlOoBW38)
- [Central Limit Theorem](https://youtu.be/_Sz0IDI-apk)

**Optional**

- Watch: [bootstrapping](https://voicethread.com/myvoice/thread/15611638/97563140)
- Read: Section 6.7 in the [STAT 155 Notes](https://mac-stat.github.io/Stat155Notes/)


:::



\
\
\
\


# References {-}

Let $\beta$ be some **population parameter** that we want to estimate with **sample statistic** $\hat{\beta}$.

\


**Sampling distributions help us understand the potential error in our sample estimate  $\hat{\beta}$**    

The sampling distribution of $\hat{\beta}$ is a collection of all possible estimates we could observe based on *all* possible samples of the *same size n* from the population.
A sampling distribution captures the degree to which a sample estimate can vary from the actual population value, hence the potential error in a sample estimate.
    

\


**The snag: we can't know exactly how wrong our estimate is.**    

In practice, we only take ONE sample, thus can't actually observe the sampling distribution.

\



**Though we can't observe the sampling distribution, we can APPROXIMATE it using the Central Limit Theorem.**    

So long as certain conditions are met (e.g. our sample size n is "big enough"), then the sampling distribution of $\hat{\beta}$ is Normally distributed around the actual population value $\beta$ with some standard error:

$$\hat{\beta} \sim N(\beta, \text{(standard error)}^2)$$
    
The **standard error**, i.e. standard deviation among all possible estimates $\hat{\beta}$, measures the typical estimation error (i.e. the typical difference between $\hat{\beta}$ and $\beta$).




\
\
\
\



# Warm-Up {-}

\


**WHERE ARE WE?**

- Sample vs population
    - We have data on a *sample* that we want to use to learn about the broader *population*.
    - Eg: We can use a *sample statistic* $\hat{\beta}$ to *estimate* *population parameter* $\beta$.
    
- Sampling variability    
    Ours is one of *many* samples we might have gotten from the population.
    
- Sampling error    
    Since our sample isn't the population itself, there's *error* in using our data to estimate features of the population.
    
- Ethics    
    We must *understand and report* the potential error in our estimate.
    
- Sampling distributions    
    - A sampling distribution describes the different sample estimates we might get, i.e. how $\hat{\beta}$ might vary from sample to sample.
    - Hence they provide a sense of the *potential* error in *our* sample estimate.
    


\
\
\
\


**SAMPLING DISTRIBUTION**

![](https://ajohns24.github.io/155_Fall_2025/images/sampling_dist_flow.png)




\
\
\
\



**ELECTION EXAMPLE**

Recall our election example where Y is the Republican candidate's (Trump's) 2024 county-level vote percentage, and X is the Republican candidate's (Mitt Romney's) 2012 vote percentage.
Based on data from the entire *population* of U.S. counties (except Alaska), we can obtain the "true" population model of Y vs X:

$$
E[Y|X] = \beta_0 + \beta_1 X = 9.699 + 0.957 X
$$


```{r}
library(tidyverse)
elections <- read_csv("https://ajohns24.github.io/data/election_2024.csv") %>% 
  mutate(repub_24 = round(100*per_gop_2024, 2),
         repub_20 = round(100*per_gop_2020, 2),
         repub_16 = round(100*per_gop_2016, 2), 
         repub_12 = round(100*per_gop_2012, 2)) %>% 
  select(state_name, state_abbr, county_name, county_fips, repub_24, repub_20, repub_16, repub_12)

head(elections)

population_model <- lm(repub_24 ~ repub_12, data = elections)
summary(population_model)
```


\
\
\
\



**EXAMPLE 1: Simulating the sampling distribution (500 samples of size 50)**

In the previous activity, each student took *1* sample of 50 counties *without replacement*:

```{r}
# Run this chunk a few times to explore the different sample models you get
elections %>% 
  sample_n(size = 50, replace = FALSE)
```
    
We can also take a sample and then use it to *estimate* the population model:
    
```{r}
# Run this chunk a few times to explore the different sample models you get
elections %>% 
  sample_n(size = 50, replace = FALSE) %>% 
  with(lm(repub_24 ~ repub_12))
```

In the previous class, we obtained a *collection* of sample estimates by having each student take 1 sample and share their results.
BUT we *each* have the power to simulate this process on our own:
    
```{r}
# Set the random number seed
set.seed(155)

# Obtain  *500* separate samples of *50 counties each*
# Store the sample models from each
election_models_50 <- mosaic::do(500)*(
  elections %>% 
    sample_n(size = 50, replace = FALSE) %>% 
    with(lm(repub_24 ~ repub_12))
)
```


```{r}
# Check it out
head(election_models_50)
dim(election_models_50)
```


**Questions**

- What's the point of the `do()` function?!? If you've taken any COMP classes, what process do you think `do()` is a shortcut for?

- What is stored in the `Intercept`, `repub_12`, and `r.squared` columns of the results?
    
    


\
\
\
\



**EXAMPLE 2: Checking out the sampling distribution**

Recall the actual *population* model:

$$
E[Y|X] = \beta_0 + \beta_1 X = 9.699 + 0.957 X
$$


```{r}
# Plot the 500 sample plots (gray) relative to the actual population model (red):
elections %>% 
  ggplot(aes(x = repub_12, y = repub_24)) + 
  geom_smooth(method = "lm", se = FALSE) +
  geom_abline(data = election_models_50, 
              aes(intercept = Intercept, slope = repub_12), 
              color = "gray", size = 0.25) + 
  geom_smooth(method = "lm", color = "red", se = FALSE)

```

Check out the (approximate) **sampling distribution** of the estimated slope coefficient $\hat{\beta}_1$, relative to the actual slope coefficient $\beta_1 = 0.957$ (red):

```{r}
# Approximate sampling distribution of the slope estimate
election_models_50 %>% 
  ggplot(aes(x = repub_12)) + 
  geom_density() +
  geom_vline(xintercept = 0.957, color = "red")
```

Recall that the *standard deviation* of the estimated sample slope coefficients $\hat{\beta}_1$ is referred to as the **standard error** of $\hat{\beta}_1$:

$$
s.e.(\hat{\beta}_1)
$$

It measures the *typical distance* of a sample slope from the actual population slope $\beta_1$, i.e. the *standard error*.
In our simulation, the *standard error* of the sample slope coefficients is 0.076:

```{r}
# Approximate standard error of the slope estimate
election_models_50 %>% 
  summarize(sd(repub_12))
```


**Reflection:** Interpret the standard error. 


\
\
\
\


**BUT! What actually happens in practice?**

We only take *1 sample* of, say, 50 counties.
(We don't go out and take 500 different samples of 50 counties.)

- Thus we don't have access to the sampling distribution...
- Thus we don't actually know the *standard error* of the sample estimates...
- Thus we need to *approximate* the sampling distribution / standard error using our 1 sample.

We'll take 2 different approaches: 

1. Central Limit Theorem (formula-based approach)
2. Bootstrapping (simulation-based approach)


\
\
\
\



**EXAMPLE 3: 1 sample of fish**

Rivers contain small concentrations of mercury which can accumulate in fish.
Scientists studied this phenomenon among largemouth bass in the Wacamaw and Lumber rivers of North Carolina.
One goal was to explore the relationship of a fish's mercury concentration (`Concen`) in parts per million (ppm) with its size, as measured by `Length` (cm):


$$
E(Concen | Length) = \beta_0 + \beta_1 Length
$$

To *estimate* this relationship, they caught and evaluated 171 fish:

```{r}
# Load the data
fish <- read_csv("https://Mac-STAT.github.io/data/Mercury.csv")
```




- **Superpopulation**: The true *underlying process* that governs the relationship between mercury concentration and length of fish, for every fish that has ever existed and ever will exist in these two rivers!

- **Finite Population:** At the time the data was collected, the true *relationship* between mercury concentration and length of fish, for all fish in the two rivers *at that time*. 

- **Sample**: Our sample of 171 fish!!!





\
\
\
\


**EXAMPLE 4: Approximate the sampling distribution via the CLT**


```{r}
# Plot the SAMPLE relationship along with the SAMPLE model estimate
fish %>% 
  ggplot(aes(y = Concen, x = Length)) + 
  geom_point() + 
  geom_smooth(method = "lm", se = FALSE)
```

```{r}
# Obtain the SAMPLE model
fish_model <- lm(Concen ~ Length, data = fish)
coef(summary(fish_model))
```


a. What's $\beta_1$, the *actual* population `Length` coefficient?

b. What's $\hat{\beta}_1$, the *sample estimate* of the population `Length` coefficient?

c. The table above reports an *approximated* standard error of $\hat{\beta}_1$. This was calculated via some formula $c / \sqrt{n}$ (using mathematical theory, not simulation). What is it?

d. State the CLT in this setting, and sketch it (on paper). Mark it according to the "68-95-99.7 Rule"

e. Where does our estimate $\hat{\beta}_1$ fall on this picture (e.g. is it above / below / close to / far from $\beta_1$?)? Or is this a trick question?



    


\
\
\
\








**Approximate the sampling distribution via Bootstrapping**

Recall the *sampling distribution*:

![](https://ajohns24.github.io/155_Fall_2025/images/sampling_dist_flow.png)

Whereas the CLT approximates the sampling distribution via *theoretical assumptions*, the **bootstrapping technique** approximates the sampling distribution via *simulation*.
In practice, we can't simulate the sampling distribution by repeatedly taking samples (as in the election example) -- we only take 1 sample of size n!!
But we *can* **REsample** our sample:   

- we sample our sample **WITH replacement** (you'll explore why below)
- each REsample is of size n, the same size as the original sample (so that we can assess the potential error associated with samples the same size as ours)

This helps us approximate the *shape* of the sampling distribution and the *standard error* of our estimate:


![](https://ajohns24.github.io/155_Fall_2025/images/bootstrapping_flow.png)


The saying "*to pull oneself up by the bootstraps*" is often attributed to Rudolf Erich Raspe's 1781 *The Surprising Adventures of Baron Munchausen* in which the character pulls himself out of a swamp by his hair (not bootstraps).
In short, it means to get something from nothing, through your own effort:

![](https://larspsyll.wordpress.com/wp-content/uploads/2015/12/muenchhausen_herrfurth_7_500x789.jpg?w=300&h=500)


In this spirit, statistical bootstrapping **doesn't make any probability model assumptions**.
It uses only the information from our one sample to approximate standard errors.




\
\
\
\





**Approximating the sampling distribution: CLT vs Bootstrapping**

- CLT   
    - The *quality* of this approximation hinges upon the validity of the Central Limit *Theorem* which hinges upon the validity of the *theoretical* model assumptions, as well as a "large enough" sample size n.
    - The CLT uses theoretical formulas for the standard error estimates, thus can feel a little mysterious without a solid foundation in probability theory.
    - BUT: When the theory is "valid", nothing beats a CLT approximation!!!
    
- Bootstrapping
    - The statistical theory behind bootstrapping is quite complicated, and there are certain obscure cases (none that we will encounter in STAT 155) where the assumptions underlying bootstrapping fail to hold.
    - It cannot and should not *replace* the CLT. BUT, it but gives us some nice intuition behind the idea of *resampling*, which is fundamental for hypothesis testing (which we'll get to shortly!).












\
\
\
\






# Exercises {-}

**Goals:**

- Learn how to implement the bootstrapping technique.
- Use bootstrapping to explore the impact of sample size on standard errors, and the sampling distribution more generally.



\
\
\


## Exercise 1: Why "resampling" (`replace = TRUE`)? {-}

Recall that in bootstrapping, we **REsample** our sample -- we sample it without replacement.
Let's wrap our minds around this idea using a small example of just the first 4 fish in our data:

```{r}
small_sample <- fish %>% 
  select(Concen, Length) %>% 
  head(4)

small_sample
```

This sample produces an estimated `Length` coefficient of 0.06532:

```{r}
lm(Concen ~ Length, data = small_sample)
```


### Part a {-}

The chunk below samples 4 fish from the `small_sample` *without* replacement (`replace = FALSE`).
Run it several times, then discuss:

- What do you observe about the samples themselves?
- If we used each one of these samples to estimate the model of `Concen` by `Length`, what do you anticipate will happen?

```{r}
sample_n(small_sample, size = 4, replace = FALSE)
```


### Part b {-}

Test your intuition.
The chunk below samples 4 fish from the `small_sample` *without* replacement (`replace = FALSE`), then uses it to estimate the model of `Concen` by `Length`.
Run it several times, then discuss:

- What do you observe about the sample models?
- What's the problem with this?

```{r}
sample_n(small_sample, size = 4, replace = FALSE) %>% 
  with(lm(Concen ~ Length))
```


### Part c {-}

Sampling our sample *without* replacement merely returns our original sample, thus does NOT give us any sense of sampling variability!
Instead, *REsample* 4 fish from our `small_sample` *with* replacement.
Run it several times.
What do you notice about the samples themselves?


```{r}
sample_n(small_sample, size = 4, replace = TRUE)
```

Similarly, what do you notice about the sample models built from the REsamples?

```{r}
sample_n(small_sample, size = 4, replace = TRUE) %>% 
  with(lm(Concen ~ Length))
```

\
\
\
\



::: {.callout-note title = "Reflect: Resampling"}

*REsampling* our sample provides insight into sampling variability, hence potential error in our sample estimates.
Each REsample might include some original sample subjects several times, and others not at all.
But this is a good thing!

- Observing a subject multiple times in a REsample mimics the idea that there are several *separate but similar* subjects in the population!
- Sampling with replacement ensures that our resampled observations are *independent*, which we need in order for bootstrapping to "work"!


:::


\
\
\
\


## Exercise 2: Run a bootstrap simulation {-}
    
We'll obtain a **bootstrapping distribution** of $\hat{\beta}_1$ by taking many (500, in this case) different samples of *every* fish in our dataset (171 of them) and exploring the degree to which $\hat{\beta}_1$ varies from sample to sample.

Edit the code below to obtain a bootstrapping distribution.

```{r}
# Set the seed so that we all get the same results
set.seed(155)

# Store the sample models
sample_models_171 <- mosaic::do(___)*(
  fish %>% 
    ___(size = ___, replace = ___) %>% 
    with(___(Concen ~ Length))
)

# Check out the first 6 results
# Make sure that your first Intercept estimate is -1.1802375
head(sample_models_171)
```


\
\
\
\


## Exercise 3: Bootstrap vs CLT sampling distribution {-}

Check out the resulting **500** bootstrapped sample models.
The *red line* represents the model estimate calculated from our *original* sample of 171 fish (not the population model, which we don't know!):

```{r}
fish %>% 
  ggplot(aes(x = Length, y = Concen)) + 
  geom_smooth(method = "lm", se = FALSE) +
  geom_abline(data = sample_models_171, 
              aes(intercept = Intercept, slope = Length), 
              color = "gray", size = 0.25) + 
  geom_smooth(method = "lm", color = "red", se = FALSE)
```    

Let's focus on the *slopes* of these 500 sample models.

### Part a {-}

A plot of our 500 bootstrap slopes *approximates* the actual **sampling distribution**:

```{r}
sample_models_171 %>% 
  ggplot(aes(x = Length)) + 
  geom_density()
```    

Describe the sampling distribution: 

- What's its general shape? 

- Where is it roughly centered? Is this value the "true" population slope $\beta_1$ or our sample slope estimate $\hat{\beta}_1$?

- Roughly what's its spread, i.e. what's the range of estimates you observed?


### Part b {-}

For a more rigorous assessment of the spread among the bootstrap slopes, calculate their standard deviation.
This provides a bootstrap *approximation* of the **standard error** of the slope estimates:

```{r}
sample_models_171 %>% 
  ___(sd(Length))
```

### Part c {-}

Recall that the bootstrap and CLT both aim to approximate features of the sampling distribution of sample slope $\hat{\beta}_1$.
In the warm-up, we noted that the CLT assumed that the sample estimates are *Normally* distributed around the "true" population slope $\beta_1$ with an estimated standard error of 0.005.
In contrast, since they're obtained from resampling the original sample of 171 fish, the bootstrapping results are distributed around the original sample slope.
However, and importantly:

- Is the *shape* of the bootstrapping results in Part a consistent with the CLT assumptions of Normality?

- Is the *spread* / *standard error* of the bootstrapping results in Part b consistent with the CLT approximation of the standard error (0.005)? 






\
\
\
\


## Exercise 4: Increasing sample size -- intuition {-}

Now that we trust bootstrapping simulations to provide reasonable insight into sampling distributions, let's use them to explore the impact of **sample size** on the quality of our sample estimates.
Recall our different (bootstrap) sample estimates when we started with a sample of 171 fish:

```{r}
# All bootstrap sample models
fish %>% 
  ggplot(aes(x = Length, y = Concen)) + 
  geom_smooth(method = "lm", se = FALSE) +
  geom_abline(data = sample_models_171, 
              aes(intercept = Intercept, slope = Length), 
              color = "gray", size = 0.25)
```    

```{r}
# Slopes of all bootstrap sample models
sample_models_171 %>% 
  ggplot(aes(x = Length)) + 
  geom_density()
```    


Suppose that instead of starting with n = 171 fish, we only had a sample of n = 50 or n = 20 fish!
What impact do you *anticipate* this having on our sample estimates:          

- If we had a smaller sample, how would it impact the sample model lines (top plot): Do you expect there to be more or less variability among the sample model lines?

- If we had a smaller sample, how would it impact the sampling distribution of the sample slopes (bottom plot):   
    - Around what value to you expect it be centered?
    - What general shape do you expect it to have?
    - Do you expect it to be narrower (with smaller standard error) or wider (with larger standard error)?



\
\
\
\



## Exercise 5: 500 samples of size n {-}

Let's decrease the sample size in our simulation.
First, take 500 REsamples from `fish`, but this time make each resample just 50 fish.
Then build a sample model from each sample:

```{r}
set.seed(155)
sample_models_50 <- mosaic::___(500)*(
  fish %>% 
    ___(size = ___, replace = TRUE) %>% 
    ___(___(Concen ~ Length))
)

# Check it out
# Make sure that your first Intercept estimate is -1.1602313
head(sample_models_50)
```


Similarly, take 500 REsamples of size *20* from `fish`, then build a sample model from each sample:


```{r}
set.seed(155)
sample_models_20 <- mosaic::___(500)*(
  fish %>% 
    ___(size = ___, replace = TRUE) %>% 
    ___(___(Concen ~ Length))
)

# Check it out
# Make sure that your first Intercept estimate is -2.3961270
head(sample_models_20)
```



\
\
\
\



## Exercise 6: Impact of sample size (part I) {-}

Use the 3 plots below to compare and contrast the 500 sets of sample models when using samples of size 20, 50, and 171.
What happens as we increase sample size?!
Was this what you expected?

```{r}
# 500 sample models using samples of size 20
fish %>% 
  ggplot(aes(x = Length, y = Concen)) + 
  geom_smooth(method = "lm", se = FALSE) + 
  geom_abline(data = sample_models_20, 
              aes(intercept = Intercept, slope = Length), 
              color = "gray", size = 0.25) + 
  lims(x = c(25, 65), y = c(-1, 5))
```


```{r}
# 500 sample models using samples of size 50
fish %>% 
  ggplot(aes(x = Length, y = Concen)) + 
  geom_smooth(method = "lm", se = FALSE) + 
  geom_abline(data = sample_models_50, 
              aes(intercept = Intercept, slope = Length), 
              color = "gray", size = 0.25) + 
  lims(x = c(25, 65), y = c(-1, 5))
```


```{r}
# 500 sample models using samples of size 171
fish %>% 
  ggplot(aes(x = Length, y = Concen)) + 
  geom_smooth(method = "lm", se = FALSE) + 
  geom_abline(data = sample_models_171, 
              aes(intercept = Intercept, slope = Length), 
              color = "gray", size = 0.25) + 
  lims(x = c(25, 65), y = c(-1, 5))
```






\
\
\
\



## Exercise 7: Impact of sample size (part II) {-}

Using the 3 plots below, let's focus on just the sampling distributions of our 500 slope estimates $\hat{\beta}_1$, i.e. the slopes of the lines in the above plots.
How do the shapes, centers, and spreads of these sampling distributions compare?
Was this what you expected?

```{r}
# 500 sample slopes using samples of size 20
sample_models_20 %>% 
  ggplot(aes(x = Length)) + 
  geom_density() +
  lims(x = c(-0.01, 0.13), y = c(0, 70))
```

```{r}
# 500 sample slopes using samples of size 50
sample_models_50 %>% 
  ggplot(aes(x = Length)) + 
  geom_density() +
  lims(x = c(-0.01, 0.13), y = c(0, 70))
```

```{r}
# 500 sample slopes using samples of size 171
sample_models_171 %>% 
  ggplot(aes(x = Length)) + 
  geom_density() +
  lims(x = c(-0.01, 0.13), y = c(0, 70))
```





\
\



## Exercise 8: Properties of sampling distributions {-}

In light of your observations, complete the following statements about the sampling distribution of the sample slope.    
  
a. For all sample sizes, the **shape** of the sampling distribution is roughly ___ and is roughly centered around the same location. (NUANCE: Here the estimates vary around the original sample slope since we're resampling from the *sample.* But in theory, different samples from the *population* will produce sample slopes that vary around the population slope!)  

b. As sample size increases:    
    The average sample slope estimate INCREASES / DECREASES / IS FAIRLY STABLE.    
    The standard error of the sample slopes INCREASES / DECREASES / IS FAIRLY STABLE.
    
c. Thus, as sample size increases, estimates of the slope become MORE RELIABLE / LESS RELIABLE.
  


\
\
\
\


## Just for fun {-}

Want more intuition into sampling distributions and the CLT?
Watch this video explanation using bunnies and dragons: [https://www.youtube.com/watch?v=jvoxEYmQHNM](https://www.youtube.com/watch?v=jvoxEYmQHNM)







## Done!

- Finalize your notes: (1) Render your notes to an HTML file; (2) Inspect this HTML in your Viewer -- check that your work translated correctly; and (3) Outside RStudio, navigate to your 'Activities' subfolder within your 'STAT155' folder and locate the HTML file. Open and inspect this in a web browser.
- Clean up your RStudio session: End the rendering process by clicking the 'Stop' button in the 'Background Jobs' tab in the lower left pane.
- Check the wrap-up announcements and solutions in the online course manual, at the bottom of the corresponding chapter.
- Work on homework!


