A summary of GLMs

The three distributions we’ve covered above are:

  1. Linear regression: \(Y_i \sim \text{Normal}(\mu_i, \sigma^2)\) where \(\mu_i = \alpha + \beta_1 x_i\)
  2. Poisson regression: \(Y_i \sim \text{Poisson}(\mu_i)\) where \(\text{log}(\mu_i) = \alpha + \beta_1 x_i\)
  3. Logistic regression: \(Y_i \sim \text{Binomial}(n_i, p_i)\) where \(\text{logit}(p_i) = \alpha + \beta_1 x_i\)

What would happen if we wanted to add extra explanatory terms (e.g., \(z_i\))? Then,

  1. Linear regression: \(Y_i \sim \text{Normal}(\mu_i, \sigma^2)\) where \(\mu_i = \alpha + \beta_1 x_i + \beta_2 z_i\)
  2. Poisson regression: \(Y_i \sim \text{Poisson}(\mu_i)\) where \(\text{log}(\mu_i) = \alpha + \beta_1 x_i+ \beta_2 z_i\)
  3. Logistic regression: \(Y_i \sim \text{Binomial}(n_i, p_i)\) where \(\text{logit}(p_i) = \alpha + \beta_1 x_i+ \beta_2 z_i\)

What about interactions (e.g., \(x_iz_i\))?

  1. Linear regression: \(Y_i \sim \text{Normal}(\mu_i, \sigma^2)\) where \(\mu_i = \alpha + \beta_1 x_i + \beta_2 z_i + \beta_3 x_iz_i\)
  2. Poisson regression: \(Y_i \sim \text{Poisson}(\mu_i)\) where \(\text{log}(\mu_i) = \alpha + \beta_1 x_i+ \beta_2 z_i + \beta_3 x_iz_i\)
  3. Logistic regression: \(Y_i \sim \text{Binomial}(n_i, p_i)\) where \(\text{logit}(p_i) = \alpha + \beta_1 x_i+ \beta_2 z_i + \beta_3 x_iz_i\)

Building a GLM

Assume the observations are independent of one another, then,

  1. Choose a distribution for the response. For example, Normal, Poisson, or Binomial.

  2. Choose a parameter to relate to explanatory terms. For example, \(\mu_i\), \(\mu_i\), or \(p_i\).

  3. Choose a link function. For example, identity, log, or logit.

  4. Choose explanatory terms

  5. Estimate additional parameters. For example, \(\sigma^2\).

Other distributions

We are not restricted to the three distributions above. Many others exist:

  • Gamma and inverse-Gaussian, for continuous responses on the interval \([0,\infty)\)
  • Beta, for continuous responses on the interval \([0,1]\)
  • Quasi-Poisson\(^{+}\), typically for modelling overdispersed count data (i.e., when \(\text{Var}(Y) > E(Y)\)) where \(\text{Var}(Y)\) is a linear function of \(E(Y)\)
  • Negative binomial, for discrete responses on \((0,1,2,\cdots)\) (can also be used for overdispersed count data), with \(\text{Var}(Y) \geq E(Y)\) and where \(\text{Var}(Y)\) is a quadratic function of \(E(Y)\)
  • Quasi-binomial\(^{+}\), a model for situation when \(\text{Var}(Y) > \text{Var}(Y_{Y \sim \text{Binomial}})\)
  • Beta-binomial, a Binomial-type model where \(p \sim \text{Beta}(a, b)\)
  • …
Distribution Notation Mean \(\mu = E(Y)\) Variance \(\sigma^2 = \text{Var}(Y)\) Linear predictor (w. a typical link function)
Gaussian \(Y\sim\textbf{Normal}(\mu, \sigma^2)\) \(\mu\) \(\sigma^2\) I\((\mu) = \alpha + \Sigma_{j=1}^{n_\text{covariates}}\beta_j x_j\)
Poisson \(Y \sim \textbf{Poisson}(\mu)\) where \(\mu = \text{rate}\) \(\mu\) \(\mu\) log\((\mu) = \alpha + \Sigma_{j=1}^{n_\text{covariates}}\beta_j x_j\)
Binomial \(Y \sim \textbf{Bonomial}(\text{n}, p)\) where \(\text{n} = \text{number of trials}\) and \(p = \text{probability of sucess}\) \(\text{n}p\) \(\text{n}p (1-p)\) logit\((p) = \alpha + \Sigma_{j=1}^{n_\text{covariates}}\beta_j x_j\)
Gamma \(Y \sim \textbf{Gamma}(k, \theta = \frac{1}{\text{rate}})\) where \(k = \text{shape}\) and \(\theta = \text{scale}\) \(k\theta\) \(k\theta^2\) log\((E(Y)) = \alpha + \Sigma_{j=1}^{n_\text{covariates}}\beta_j x_j\)
Beta \(Y \sim \textbf{Beta}(a, b)\) where \(a = \text{shape}\) and \(b = \text{scale}\) \(\frac{a}{a+b}\) \(\frac{ab}{(a+b)^2(a+b+1)}\) log\((E(Y)) = \alpha + \Sigma_{j=1}^{n_\text{covariates}}\beta_j x_j\)
Negative binomial \(Y \sim \textbf{NB}(r, p)\) where \(\text{r} = \text{number of successes until the experiment is stopped}\) and \(p = \text{probability of sucess}\) or \(Y \sim \textbf{NB}(k, p)\) where \(\text{k} = \text{number of failures}\) given \(p = \text{probability of sucess}\) \(\frac{r(1-p)}{p}\) or \(\mu = k\frac{p}{1-p}\) \(\frac{r(1-p)}{p^2}\) or \(\mu + \frac{\mu^2}{k}\) log\((E(Y)) = \alpha + \Sigma_{j=1}^{n_\text{covariates}}\beta_j x_j\)
Beta-binomial \(Y \sim \textbf{BetaBin}(\text{n},a, b)\) where \(\text{n} = \text{number of trials}\) and \(p = \frac{a}{a + b}, \text{the probability of success}\) \(\frac{\text{n} a}{a+b}= \text{n}p\) \(\frac{\text{n} a b(a + b + \text{n})}{(a+b)^2(a + b + 1)}\) logit\((p) = \alpha + \Sigma_{j=1}^{n_\text{covariates}}\beta_j x_j\)

\(^{+}\) Note that the Quasi-… distributions are not really proper distributions; they are simply models describing a mean-variance relationship beyond that offered by the base distribution.

TASK Study this cheatsheet and find relevant examples of response variables from your field of study for each distribution.

Note glm() is only the beginning.

R function Use
gam() Fit a generalised additive model. The R package mgcv must be loaded
nlme() Fit linear and non-linear mixed effects models. The R package nlme must be loaded
gls() Fit generalised least squares models. The R package nlme must be loaded