Key Takeaways: Multiple Regression

One-Sentence Summary

Multiple regression extends simple regression to multiple predictors, enabling you to estimate the effect of each variable "holding all others constant" — thereby partially controlling for confounders — while adjusted $R^2$ penalizes unnecessary complexity, VIF detects multicollinearity, indicator variables incorporate categorical predictors, and residual diagnostics verify assumptions.

Core Concepts at a Glance

The Multiple Regression Procedure

Step by Step

1. Define the question. What outcome? What predictors? Why these predictors (theory-driven)?

2. Explore the data: - Scatterplots of $y$ vs. each $x$ - Correlation matrix among predictors (watch for multicollinearity) - Descriptive statistics

3. Fit the model using smf.ols('y ~ x1 + x2 + x3', data=df).fit() in Python or the Data Analysis ToolPak in Excel.

4. Check overall fit: - $R^2$ and adjusted $R^2$ - F-test (is at least one predictor useful?)

5. Interpret individual predictors: - Coefficient: "For each one-unit increase in $x_i$, predicted $y$ changes by $b_i$, holding all other variables constant" - t-test / p-value: Is this specific predictor significant? - 95% CI: Plausible range for the true effect

6. Check multicollinearity: Calculate VIF for each predictor

7. Diagnose residuals: - Residuals vs. predicted → linearity, equal variance - QQ-plot → normality - Independence → study design

8. Report carefully: Discuss causation, unmeasured confounders, and limitations

Key Python Code

import statsmodels.formula.api as smf
from statsmodels.stats.outliers_influence import variance_inflation_factor
import matplotlib.pyplot as plt
from scipy import stats

# Fit model
model = smf.ols('y ~ x1 + x2 + C(category)', data=df).fit()
print(model.summary())

# VIF
X = df[['x1', 'x2']]
for i, col in enumerate(X.columns):
    print(f"VIF for {col}: "
          f"{variance_inflation_factor(X.values, i):.2f}")

# Residual diagnostics
fig, axes = plt.subplots(1, 3, figsize=(15, 5))
axes[0].scatter(model.fittedvalues, model.resid, alpha=0.6)
axes[0].axhline(y=0, color='red', linestyle='--')
axes[0].set_title('Residuals vs. Predicted')
stats.probplot(model.resid, plot=axes[1])
axes[1].set_title('QQ-Plot')
axes[2].hist(model.resid, bins=15, edgecolor='navy', alpha=0.7)
axes[2].set_title('Residual Distribution')
plt.tight_layout()
plt.show()

Excel Procedure

The Threshold Concept: "Holding Other Variables Constant"

Each partial regression coefficient estimates the effect of one predictor on the outcome, assuming all other predictors remain unchanged. This is how regression statistically approximates what an experiment achieves through randomization: isolating the effect of one variable by controlling for others.

Key Formulas

Adjusted $R^2$ vs. $R^2$

Multicollinearity Reference

Key distinction: Multicollinearity affects individual coefficient interpretation but does NOT affect overall model predictions.

Dummy Variables Quick Reference

The dummy variable trap: Including all $k$ dummies creates perfect multicollinearity. Python/R handle this automatically; in Excel, you must create $k-1$ columns manually.

Residual Diagnostics (LINE)

Interaction Terms

• What they capture: The effect of one variable depends on the level of another
• In the model: Add $x_1 \times x_2$ as a predictor; in Python: y ~ x1 * x2 (includes both main effects and interaction)
• Interpretation: When the interaction is significant, you cannot interpret main effects in isolation — the "effect" of $x_1$ changes depending on $x_2$
• Caution: Don't add interactions without theoretical justification; with $k$ predictors, there are $k(k-1)/2$ possible pairwise interactions

Model Building Strategy

Common Mistakes

Connections