---
title: "P-values + other considerations (Notes)"
subtitle: "STAT 155"
author: "Your Name"
format:
  html:
    toc: true
    toc-depth: 2
    embed-resources: true
---


```{r setup}
#| include: false
knitr::opts_chunk$set(
  collapse = TRUE, 
  warning = FALSE,
  message = FALSE,
  error = TRUE,
  fig.height = 2.75, 
  fig.width = 4.25,
  fig.env = 'figure',
  fig.pos = 'h',
  fig.align = 'center')
```


# Warm-up {-}

\


**p-value / testing cautionary tales: review from previous activity**


["Still not significant" article](https://mchankins.wordpress.com/2013/04/21/still-not-significant-2/)   

- 0.05 is not a magic number below which things are meaningful and above which they aren't!
- don't try to fit a hypothesis test into our narrative. state a significance level and stick with it.


\

[XKCD comic](https://imgs.xkcd.com/comics/significant.png)   

- fishing around for significance can lead to misleading conclusions. this is referred to as **multiple testing**.
- the more things we test, the more likely something will appear to be "significant" just by chance (even if it's not). this is a **false positive** or **Type I error**.

\

[p-hacking interactive tool](https://stats.andrewheiss.com/hack-your-way/)   

- as with part b: fishing around for significance can lead to misleading conclusions
- as with part a: don't try to fit a hypothesis test into our narrative


\

Spotify example

- when sample size n is large, results might be *statistically significant* but not *practically significant*. that is, we might detect the *existence* of an effect, but the *magnitude* of the effect might be too small to be contextually meaningful.
- why? standard errors decrease as n increases, and the smaller the s.e. the easier it is to detect coefficient deviations from 0.

```{r eval = TRUE}
# Load the data
spotify_big <- read.csv("https://ajohns24.github.io/data/spotify_example_big.csv") %>% 
  select(track_artist, track_name, duration_ms, latin_genre) %>% 
  mutate(duration = duration_ms / 1000)
nrow(spotify_big)

# Plot the relationship
spotify_big %>% 
  ggplot(aes(x = duration, color = latin_genre)) + 
  geom_density()

# Build the model
spotify_model_2 <- lm(duration ~ latin_genre, spotify_big)
coef(summary(spotify_model_2))
```   


\
\
\
\

**The p-value debate**

The p-value is very commonly misinterpreted and misused (usually unintentionally!).
The following papers highlight a debate around the p-value: should we use them at all? and if so, what are best practices?
I recommend skimming them outside of class!

-   [The ASA Statement on p-values](https://www.tandfonline.com/doi/full/10.1080/00031305.2016.1154108)   
-   [Scientific method: Statistical error](https://www.nature.com/news/scientific-method-statistical-errors-1.14700)
-   [Moving to a World Beyond "p \< 0.05"](https://www.tandfonline.com/doi/full/10.1080/00031305.2019.1583913)
-   [Statistical tests, p-values, confidence intervals, and power: a guide to misinterpretations](https://drive.google.com/file/d/1dW6G8gO4PGAZP-cTf7w2EC29hmHbvdOq/view?usp=sharing)



\
\
\
\



**EXAMPLE 1: [You name this example]**

Researchers report that people who regularly eat oatmeal for breakfast are significantly happier on a 0-10 scale (p-value = 0.015).

a. Interpret the p-value.


b. What statistical follow-up questions should you ask before taking this result seriously?





\
\
\
\



**EXAMPLE 2: Never report a p-value alone**

We should never report a p-value alone!
It really means nothing on its own -- we need the estimated *magnitude* of the effect and the *error* associated with this estimate.
A few approaches for what to report / look for:

- CI + p-value
- estimate + standard error + p-value
- estimate + CI + p-value

Let's practice.
Report what you learn about the relationship between the `Price` ($) of a home and its `Age` (years).
NOTE: This data is from 2006!

```{r eval = TRUE}
homes <- read_csv("https://mac-stat.github.io/data/homes.csv")
price_model <- lm(Price ~ Living.Area + Age, data = homes)
summary(price_model)
```




\
\
\
\



**EXAMPLE 3 (OPTIONAL): Power**

Statistical power is the probability of rejecting the null hypothesis when the alternative hypothesis is true (a "true positive").
For example, in the tests we've been doing, statistical power is the probability of detecting a relationship when there truly *is* a relationship.

Check out this interactive visualization of the factors that influence statistical power:

[https://rpsychologist.com/d3/nhst/](https://rpsychologist.com/d3/nhst/)

Under "Settings", next to the "Solve for?" text, click "Power".
You will vary the 3 different parameters (significance level, sample size, and effect size) one at a time to understand how these factors affect power.
Some (OPTIONAL) context behind this interactive visualization:

- Visualization is based on a one sample Z-test:    
    This is a test for whether the true population mean equals a particular value. (e.g., true mean = 30)
    
- The effect size slider is measured with a metric called Cohen's d:
    - Cohen's d = magnitude of effect/standard deviation of response variable
    - Here: how far is the true mean from the null value in units of SD?
    - e.g., If the null value is 30, true mean is 40, and the true population SD of the quantity is 5, the Cohen’s d effect size is (40-30)/5 = 2.

a. What is your intuition about how changing the significance level will change power? Check your intuition with the visualization and explain why this happens.

b. Repeat Part a for the sample size.

c. Repeat Part a for the effect size.




\
\
\
\


# Project time {-}

- Discuss Project Milestone 2 feedback.
    
- Discuss what needs to be done for the final project submission.
    - Everyone should have a hand in every aspect of the project (eg: planning the analysis, writing code, writing up the report, etc). But it would be helpful to divvy up who will write the first draft of each section in the report.
    - *Everyone* is expected to review and provide feedback on the entire report + qmd code file.   
        - Pro tip: use the provided rubric in your editing and feedback process!


## Done!

- Finalize your notes: (1) Render your notes to an HTML file; (2) Inspect this HTML in your Viewer -- check that your work translated correctly; and (3) Outside RStudio, navigate to your 'Activities' subfolder within your 'STAT155' folder and locate the HTML file. Open and inspect this in a web browser.
- Clean up your RStudio session: End the rendering process by clicking the 'Stop' button in the 'Background Jobs' tab in the lower left pane.
- Check the wrap-up announcements and solutions in the online course manual, at the bottom of the corresponding chapter.
- Work on homework!


