Key Takeaways: Correlation and Simple Linear Regression

One-Sentence Summary

The Pearson correlation coefficient measures the strength and direction of a linear relationship between two numerical variables, while simple linear regression fits a least squares line that predicts one variable from another — but correlation does not imply causation, $R^2$ measures the proportion of variability explained, residual plots reveal whether a linear model is appropriate, and regression to the mean explains why extreme observations tend to be followed by less extreme ones.

Core Concepts at a Glance

The Regression Procedure

Step by Step

1. Always plot first: Create a scatterplot. Look for direction, form, strength, and outliers.

2. Calculate the correlation: $$r = \frac{1}{n-1}\sum\left(\frac{x_i - \bar{x}}{s_x}\right)\left(\frac{y_i - \bar{y}}{s_y}\right)$$

3. Fit the regression line: - Slope: $b_1 = r \cdot \frac{s_y}{s_x}$ - Intercept: $b_0 = \bar{y} - b_1\bar{x}$ - Equation: $\hat{y} = b_0 + b_1 x$

4. Interpret slope and intercept in context.

5. Assess model fit: - $R^2 = r^2$ — proportion of variability explained - Residual plot — check for patterns

6. Check LINE conditions (for inference): - Linearity - Independence - Normality of residuals - Equal variance of residuals

7. Discuss causation: Is this an experiment (causal) or observational study (association only)? Identify lurking variables.

Key Python Code

import numpy as np
from scipy import stats
import statsmodels.api as sm
import seaborn as sns
import matplotlib.pyplot as plt

# Correlation
r, p = stats.pearsonr(x, y)

# Also: np.corrcoef(x, y)[0, 1]

# Simple regression
result = stats.linregress(x, y)
slope = result.slope
intercept = result.intercept
r_value = result.rvalue
r_squared = result.rvalue**2

# Full output with statsmodels
X = sm.add_constant(x)
model = sm.OLS(y, X).fit()
print(model.summary())

# Visualization with seaborn
sns.regplot(x=x, y=y, ci=95)

# Residual plot
predicted = intercept + slope * x
residuals = y - predicted
plt.scatter(predicted, residuals)
plt.axhline(y=0, color='red', linestyle='--')

Excel Functions

The Threshold Concept: Regression to the Mean

Extreme observations tend to be followed by less extreme observations — not because of any causal force, but because the correlation between measurements is less than perfect.

Mathematical explanation: In z-score terms, predicted $z_y = r \times z_x$. Since $|r| < 1$, the prediction is always less extreme than the observation it's based on.

Correlation vs. Causation

Four Explanations for a Correlation

How to Establish Causation

Key Formulas

Properties of $r$

Interpreting Slope and Intercept

Slope Template

"For each one-unit increase in [explanatory variable], the predicted [response variable] changes by $b_1$ [units], on average."

Intercept Interpretation

"When [explanatory variable] = 0, the predicted [response variable] is $b_0$."

Caution: The intercept only has practical meaning when $x = 0$ is within or near the range of observed data. If $x = 0$ is implausible (shoe size = 0, years of experience = 0 for a CEO study), the intercept is just a mathematical anchor.

Common Mistakes

Connections