16 Welcome back: regression & classification review

16.1 General goals

The overarching goal of the past few weeks has been to model some response variable \(Y\) by predictors \(X\):

  • When \(Y\) is quantitative and bell-shaped (whether continuous or discrete), we might utilize a Normal regression model which assumes that
    \[Y_i | \beta_0, \beta_1, \sigma \stackrel{ind}{\sim} N(\mu_i, \sigma^2) \;\; \text{ where } \mu_i = \beta_0 + \beta_1 X_i\]

  • When \(Y\) is a quantitative count, we might utilize a Poisson (or Negative Binomial) regression model which assumes that
    \[Y_i | \beta_0, \beta_1 \stackrel{ind}{\sim} Pois(\lambda_i) \;\; \text{ where } \log(\lambda_i) = \beta_0 + \beta_1 X_i\]

  • When \(Y\) is a binary categorical variable, we might utilize a logistic regression model which assumes that
    \[Y_i | \beta_0, \beta_1 \stackrel{ind}{\sim} Bern(\pi_i) \;\; \text{ where } \log\left(\frac{\pi_i}{1 - \pi_i}\right) = \beta_0 + \beta_1 X_i\]

  • When \(Y\) is a categorical variable with 2+ categories, we might utilize a naive Bayes classification algorithm. This is unlike the other models here in that it doesn’t make “parametric” assumptions about the relationship with the predictors (eg: \(\beta_0 + \beta_1 X\)).



16.2 How these models are connected

Normal, Poisson, Negative Binomial, and logistic regression are all part of the broader generalized linear model family. For all GLMs, the dependence of \(E(Y|...)\) on a predictor \(X\) is

\[g(E(Y|...)) = \beta_0 + \beta_1 X\]

where the appropriate link function \(g()\) depends upon the data structure:

  • Normal: Identity link function \(g(z) = z\)
    \(Y_i|\beta_0,\beta_1,\sigma \sim N(\mu_i, \sigma^2)\)
    \(E(Y|\beta_0,\beta_1,\sigma) = \mu_i\)
    \(g(\mu_i) = \mu_i = \beta_0 + \beta_1 X\)

  • Poisson: Log link function \(g(z) = log(z)\)
    \(Y_i|\beta_0,\beta_1 \sim Pois(\lambda_i)\)
    \(E(Y|\beta_0,\beta_1) = \lambda_i\)
    \(g(\lambda_i) = \log(\lambda_i) = \beta_0 + \beta_1 X\)

  • Logistic: Logit link function \(g(z) = log(z / (1-z))\)
    \(Y_i|\beta_0,\beta_1 \sim Bern(\pi_i)\)
    \(E(Y|\beta_0,\beta_1) = \pi_i\)
    \(g(\pi_i) = \log\left(\frac{\pi_i}{1-\pi_i}\right) = \beta_0 + \beta_1 X\)



16.3 Modeling workflow