---
title: "F-Tests & Nested models (Notes)"
subtitle: "STAT 155"
author: "Your Name"
format:
  html:
    toc: true
    toc-depth: 2
    embed-resources: true
---


```{r setup}
#| include: false
knitr::opts_chunk$set(
  collapse = TRUE, 
  warning = FALSE,
  message = FALSE,
  error = TRUE,
  fig.height = 2.75, 
  fig.width = 4.25,
  fig.env = 'figure',
  fig.pos = 'h',
  fig.align = 'center')
```


::: {.callout-note title = "Learning goals"}

- Determine if one model is nested within another
- Determine which null and alternative hypotheses require an F-test
- Determine which F-tests require the use of the `anova` function in R vs. the overall F-test given in regular regression output
- Interpret the results of an F-test in context


:::

::: {.callout-note title = "Additional resources"}

**Required video**

- [F-Tests](https://youtu.be/XIHtAGXhlzQ)


**Optional**

- Read: Section 7.3.4 in the [STAT 155 Notes](https://mac-stat.github.io/Stat155Notes/)

:::

\
\
\
\

# References {-}

![](https://ajohns24.github.io/155_Fall_2025/images/Ftest_table.png)


::: {.callout-note title = "F-test test statistic"}

The F-test test statistic is *not* calculated like the t-test test statistic, thus cannot be interpreted the same way!
Check out the optional math box below to learn how this statistic is calculated.

:::


::: {.callout-note title = "OPTIONAL: Calculating the F-test test statistic"}


Consider 2 models which we'll estimate using $n$ data points:

- Model 1 (Big model)   
    - $E[Y|X_1,...,X_k,...,X_p] = \beta_0 + \beta_1 X_1 + \cdots + \beta_k X_k + \cdots + \beta_p X_p$
    - "Residual Sum of Squares": $RSS_1 = \sum_{i=1}^n (Y_i - \hat{Y}_i)^2$ where $\hat{Y}_i$ are the predictions calculated from Model 1
    
- Model 2 (Small model)   
    - $E[Y|X_1,...,X_k] = \beta_0 + \beta_1 X_1 + \cdots + \beta_k X_k$
    - "Residual Sum of Squares": $RSS_2 = \sum_{i=1}^n (Y_i - \hat{Y}_i)^2$ where $\hat{Y}_i$ are the predictions calculated from Model 2

And consider the F-test:

$H_0$: $\beta_{k+1} = \beta_{k+2} = ... = \beta_p = 0$

$H_a$: at least one of $\beta_{k+1}$, $\beta_{k+2}$, ..., $\beta_p$ is non-0 


Then the F-test test statistic measures the additional residual error when using Model 2 vs Model 1, while accounting for the complexity of each model.
The greater the test statistic, the greater the residuals when we simplify the model (go from Model 1 to Model 2), thus the more evidence we have to reject $H_0$: 

$$
F = \frac{RSS_2 - RSS_1}{RSS_1} \frac{n - (p + 1)}{p - k}
$$



:::





\
\
\
\


# Warm-up {-}

**DIRECTIONS**

- Throughout this activity, test hypotheses at the 0.05 significance level.
- Make all conclusions and interpretations in context.

\


To explore the relationship of casual bikeshare ridership with the day of week (weekend or weekday), feels like temperature (F), and actual temperature (F), let's build 3 sample models:

```{r}
bikes <- read_csv("https://mac-stat.github.io/data/bikeshare.csv") %>% 
  rename(rides = riders_casual)

# Build sample models
bike_model_1 <- lm(rides ~ weekend + temp_feel + temp_actual, bikes)
bike_model_2 <- lm(rides ~ temp_feel + temp_actual, bikes)
bike_model_3 <- lm(rides ~ weekend, bikes)
```



\
\


**EXAMPLE 1: Nested models**

Which of these models are *nested* in any other?





\
\
\
\




**EXAMPLE 2: "overall" F-tests**

The following population model

E[`rides` | `weekend`, `temp_feel`, `temp_actual`] = $\beta_0$ + $\beta_1$ `weekendTRUE` + $\beta_2$ `temp_feel` + $\beta_3$ `temp_actual`

is estimated by `bike_model_1`:

```{r}
summary(bike_model_1)
```

The "overall" **F-test** is reported in the bottom line of the `summary()`.
State the hypotheses, p-value, and your conclusion.
NOTE: The F-test test statistic is *not* calculated like the t-test test statistic.





\
\
\
\



**EXAMPLE 3: t-tests for individual model coefficients**


State the hypotheses, p-value, and your conclusion about the **t-test** in the `temp_feel` row of the `summary()`.
(Similarly, the t-test in the `temp_actual` row has a p-value of 0.0935 (> 0.05).)





\
\
\
\


**EXAMPLE 4: F-tests for nested models**

The following F-test compares `bike_model_3` to `bike_model_1`.
State the hypotheses, p-value, and your conclusion.

```{r}
# Put the smaller model first!!!
anova(bike_model_3, bike_model_1)
```




\
\
\
\


**EXAMPLE 5: Different angles**

A *series* of models and tests can provide more insight than one model or test alone!
What did we learn about the relationship of `rides` with `temp_feel` and `temp_actual` from Examples 3 and 4 *combined*?
Why do you think this happened?
What would you do next?






\
\
\
\


# Exercises {-}

The `MacGrades` dataset contains a sub-sample (to help preserve anonymity) of every grade assigned to a former Macalester graduating class.
We have 6414 observations that include `sessionID` (section), `sid` (student), `sem` (semester), `iid` (instructor), as well as the following variables that we'll use in our analysis:

- `grade`: the grade obtained, as a numerical value (i.e. an A is a 4, an A- is a 3.67, etc.)
- `dept`: department identifier (anonymized)
- `level`: course level (e.g. 100-, 200-, 300-, and 600-)
- `enroll`: section enrollment

```{r}
# Load packages & data
library(tidyverse)
MacGrades <- read_csv("https://mac-stat.github.io/data/MacGrades.csv") %>% 
  mutate(level = factor(level)) # make level a factor variable
head(MacGrades)
```




## Exercise 1: Review {-}

Do average grades differ by department?
Let's explore.
For simplicity, we'll focus on just 6 of 40 departments: A, b, B, M, N, q.

### Part a {-}

Which is our response / outcome variable: `grade` or `dept`?

### Part b {-}

Which type of model is appropriate for exploring this relationship: (Normal) linear regression or logistic regression?

### Part c {-}

Interpret the sample estimate of the intercept coefficient:

```{r}
MacGrades %>% 
  filter(dept %in% c("A", "b", "B", "M", "N", "q")) %>% 
  with(lm(grade ~ dept)) %>% 
  summary()
```


### Part d {-}

Which department has the lowest average grade and what is that grade?
What about the highest average grade?

### Part e {-}

Let $\beta_1$ be the true but unknown `deptb` coefficient and consider the t-test:

$H_0$: $\beta_1 = 0$    
$H_a$: $\beta_1 \ne 0$

How can we interpret $H_0$?

- The average grade in Department b is 0.
- The average grade in Department b is different than in Department A.
- The average grade in Department b is not different than in Department A.

### Part f {-}

The p-value for this test is 0.589.
Interpret the p-value in context.
(Your answer should make clear what 0.589 actually means or measures, not just how it compares to 0.05!)


### Part g {-}

What's your conclusion about this hypothesis test?

- We (do / do not) have statistically significant evidence that the average grade in Department b is 0.
- We (do / do not) have statistically significant evidence that the average grade in Department b is different than in Department A.



\
\
\
\



## Exercise 2: F-test and t-tests {-}

Let's continue to explore the population model of grades vs department:

$$
E[grades | dept] = \beta_0 + \beta_1 deptb + \beta_2 deptB + \beta_3 deptM + \beta_4 deptN + \beta_5 deptq
$$


### Part a {-}

Recall our research question: Do average grades differ by department?
State the relevant hypotheses using notation.



### Part b {-}

State the p-value for this test, and what conclusion you make from this p-value.


### Part c {-}

Now check out the t-tests for the individual department coefficients.
Are any of these statistically significant?
What does this mean in context?


### Part d {-}

Explain why Parts b and c are *not* contradictory!







\
\
\
\



## Exercise 3: grades vs enrollment and level {-}

Next, let's explore the relationship of grades with course enrollment (a quantitative predictor) and level (a categorical predictor with levels 100, 200, 300, 400, and 600).
NOTE: We'll use data on all departments now, not just the 6 used above.

Complete the below model statement of this relationship.
For consistency from student to student, put any `enroll` terms before any `level` terms.
This is the notation you'll refer to throughout the next exercises!


E[grades | enroll, level] = $\beta_0$ + 



\
\
\
\


## Exercise 4: Some useful and not useful R output {-}

Below are some models and summaries of various relationships of grades with enrollment and level:

```{r}
grades_model_2 <- lm(grade ~ enroll + level, MacGrades)
grades_model_3 <- lm(grade ~ enroll, MacGrades)
summary(grades_model_2)
summary(grades_model_3)
anova(grades_model_3, grades_model_2)
```




### Part a {-}

`grades_model_2` is the sample estimate of the model you wrote out in Exercise 3.
Interpret the estimated `enroll` coefficient.


### Part b {-}

For classes of the same level, are course grades associated with enrollments? 
Using your notation from Exercise 3, state the relevant hypotheses that address this question.
THINK: Which coefficient(s) is/are relevant?


### Part c {-}

What do you need to test these hypotheses?

- some t-test reported in `summary(grades_model_2)`
- the overall F-test reported in `summary(grades_model_2)`
- some t-test reported in `summary(grades_model_3)`
- the overall F-test reported in `summary(grades_model_3)`
- to F-test comparison of `grades_model_3` to `grades_model_2` reported using `anova()`

### Part d {-}

Using the approach you identified in Part c, report a p-value and state your corresponding conclusion.




\
\
\
\





## Exercise 5: continuing with grades vs enrollment and level {-}

Let's continue exploring the relationship of grades with enrollment and level.

### Part a {-}

For classes with the same enrollments, are course grades associated with the level of the course? 
Using your notation from Exercise 3, state the relevant hypotheses that address this question.
THINK: Which coefficient(s) is/are relevant?


### Part b {-}

What do you need to test these hypotheses?

- some t-test reported in `summary(grades_model_2)`
- the overall F-test reported in `summary(grades_model_2)`
- some t-test reported in `summary(grades_model_3)`
- the overall F-test reported in `summary(grades_model_3)`
- to F-test comparison of `grades_model_3` to `grades_model_2` reported using `anova()`

### Part c {-}

Using the approach you identified in Part b, report a p-value and state your corresponding conclusion.


\
\
\
\



## Exercise 6: Guess the p-value {-}

Consider a new model of grades:

```{r}
grades_model_4 <- lm(grade ~ level, MacGrades)
```

### Part a {-}

Using your notation from Exercise 3, state the hypotheses that would be tested by running `anova(grades_model_4, grades_model_2)`.
**DO NOT YET RUN THIS CODE!**


### Part b {-}

**WITHOUT RUNNING THE anova() CODE:** What will be the p-value reported by `anova(grades_model_4, grades_model_2)`?
What's your reasoning?


### Part c {-}

Check your intuition:

```{r}
anova(grades_model_4, grades_model_2)
```




\
\
\
\




## Exercise 7: But is it a "good" model? {-}

In the above exercises, you should have concluded that, when controlling for the other, both course enrollments and course level are significantly associated with grades.
But are they "good" predictors?
Let's explore `grades_model_2` in more depth.


### Part a {-}

The relationships of `grade` with `enroll` and `level` are *statistically* significant, but are they *practically significant* / contextually meaningful?


### Part b {-}

Grades vary from student to student within and across courses.
What percentage of this variation is explained by course enrollments and level?


### Part c {-}

Putting this all together, what's your overall conclusion about the relationship here?


\
\
\
\




## Up next {-}

Work on the project!!!
Remember to work in the same Google Doc you shared with me for Milestone 2.



## Done!

- Finalize your notes: (1) Render your notes to an HTML file; (2) Inspect this HTML in your Viewer -- check that your work translated correctly; and (3) Outside RStudio, navigate to your 'Activities' subfolder within your 'STAT155' folder and locate the HTML file. Open and inspect this in a web browser.
- Clean up your RStudio session: End the rendering process by clicking the 'Stop' button in the 'Background Jobs' tab in the lower left pane.
- Check the wrap-up announcements and solutions in the online course manual, at the bottom of the corresponding chapter.
- Work on homework!


