Multiple Linear Regression: Predicting with More Power

Moving Beyond Simple: Multiple Linear Regression

In Simple Linear Regression (SLR), we saw how to predict an outcome (like house price) using just one input factor (like house size). But reality is often more complex! House prices usually depend on size, location, number of bedrooms, age, and more.

That's where Multiple Linear Regression (MLR) comes in. It's a powerful extension of SLR that allows us to use multiple independent variables (inputs) to predict a single dependent variable (output). It helps us build more realistic and often more accurate models.

Important Rules (Assumptions) for MLR

MLR shares some assumptions with SLR, but adds a crucial new one related to the input variables themselves:

1. Linearity: The relationship between the dependent variable (Y) and *each* independent variable (Xᵢ) should be linear, holding other variables constant.
2. Independence of Errors: The errors (residuals) should be independent of each other (similar to SLR).
3. Homoscedasticity: The errors should have constant variance across all levels of the independent variables (similar to SLR).
4. Normality of Errors: The errors should be normally distributed (similar to SLR).
5. Lack of Multicollinearity: This is new and very important for MLR! The independent variables (X's) should not be highly correlated with *each other*. If two inputs are highly correlated (e.g., 'Years of Experience' and 'Age'), it's hard for the model to tell which one is truly influencing the output, leading to unstable and unreliable coefficient estimates (b values).

Checking for multicollinearity often involves looking at correlation matrices between independent variables or calculating Variance Inflation Factors (VIFs).

Building an MLR Model (Python Workflow)

Here’s a typical workflow using Python libraries like `pandas` and `scikit-learn`:

1. Load & Prepare Data:
   • Import your dataset (e.g., using `pd.read_csv`).
   • Separate your features (X - potentially multiple columns) and target (y - one column).
   • Handle missing values (e.g., using `SimpleImputer`).
2. Encode Categorical Features:
   • Identify categorical columns in X.
   • Use `ColumnTransformer` with `OneHotEncoder` to create dummy variables. Remember to set `drop='first'` or manually drop one column per category to avoid the dummy variable trap.
3. Split Data:
   • Divide X and y into training and testing sets using `train_test_split`.
4. Feature Scaling (Optional but Recommended):
   • Apply `StandardScaler` or `MinMaxScaler` to the numerical features *after* splitting. Fit only on the training data (`fit_transform`) and then transform the test data (`transform`).
5. Train the Model:
   • Create an instance of `LinearRegression` from `sklearn.linear_model`.
   • Fit the model to the prepared training data: `model.fit(X_train, y_train)`.
6. Make Predictions:
   • Use the trained model to predict on the prepared test set: `y_pred = model.predict(X_test)`.
7. Evaluate the Model:
   • Compare `y_pred` with the actual `y_test` values.
   • Calculate metrics like Mean Squared Error (MSE) and R-squared (R²) using functions from `sklearn.metrics`. R² tells you the proportion of variance in Y explained by the X variables (closer to 1 is better).
8. Interpret & Refine (Optional):
   • Examine the learned coefficients (`model.coef_`) to understand the influence of each feature.
   • Consider feature selection techniques (like Backward Elimination, Forward Selection, or using Lasso regularization) to potentially remove less important features and build a simpler, more robust model.

Conceptual Python Snippet

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score

# --- Assume df is loaded and X, y are separated ---
# Assume X has shape (n_samples, n_features)
# Assume categorical features are at specific indices, e.g., [3] for 'State'
categorical_features_indices = [3]
numerical_features_indices = [0, 1, 2] # Example

# 1. Preprocessing Step (Encoding) using ColumnTransformer
preprocessor = ColumnTransformer(
    transformers=[
        ('cat', OneHotEncoder(drop='first'), categorical_features_indices)], # drop='first' avoids trap
    remainder='passthrough') # Keep other columns (numerical)

X = preprocessor.fit_transform(X) # Apply encoding

# 2. Split Data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# 3. Feature Scaling (on numerical features *after* potential encoding shifts indices)
#    Need to identify the new indices of numerical features after one-hot encoding.
#    This part requires careful index management or applying scaling within the preprocessor.
#    Simplified example: Assuming all columns after split need scaling
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)

# 4. Train Model
mlr_model = LinearRegression()
mlr_model.fit(X_train, y_train)

# 5. Predict
y_pred = mlr_model.predict(X_test)

# 6. Evaluate
mse = mean_squared_error(y_test, y_pred)
r2 = r2_score(y_test, y_pred)

print(f"Intercept: {mlr_model.intercept_}")
print(f"Coefficients: {mlr_model.coef_}")
print(f"MSE: {mse:.4f}")
print(f"R-squared: {r2:.4f}")
                                    

Summary: MLR Key Points

• Multiple Linear Regression (MLR) predicts a dependent variable (Y) using two or more independent variables (X₁, X₂, ...).
• The equation is y = b₀ + b₁x₁ + ... + bₙxₙ.
• Key assumptions include Linearity, Independence, Homoscedasticity, Normality of Errors, and crucially, Lack of Multicollinearity among predictors.
• Categorical predictors must be converted to Dummy Variables (usually via One-Hot Encoding).
• Avoid the Dummy Variable Trap by dropping one dummy column per original categorical feature.
• Evaluation often uses MSE (how large are errors on average) and R² (how much variance is explained).
• Feature selection methods can help refine the model by keeping only significant predictors.