First time studying this topic?

Launch the interactive walkthrough to discover each component, including the 3D Q-Sphere Lab & Python in Colab.

Topic 2.7

Overfitting, Underfitting & Regularization

Diagnose the Bias-Variance tradeoff and tame complex models using L1 (Lasso) and L2 (Ridge) regularization.

Learning Objectives

✓Define Overfitting (High Variance) and Underfitting (High Bias)
✓Analyze train vs validation learning curves to detect overfitting
✓Apply L1 (Lasso) and L2 (Ridge) weight regularization penalties
✓Understand early stopping and dropout techniques

📖Let's Begin with a Story

Imagine a tailor making a custom suit for a client. Underfitting: The tailor cuts a generic square sack with two armholes. It fits nobody well—it is too simple and rigid. Overfitting: The tailor creates a suit molded down to the exact millimeter of the client's posture on a cold Tuesday morning while they were slouching slightly to the left with their wallet in their back pocket. When the client stands up straight on Wednesday, the suit rips at the seams! It was memorized to one fleeting instance and cannot adapt. Good Fit: The tailor cuts a suit that drapes naturally, accommodating normal body movement while looking sharp. That balance is the holy grail of machine learning.

Why Should You Learn Overfitting, Underfitting & Regularization?

🎯 Practical & Engineering Stakes

The central struggle of all ML

Any model can memorize its training data to achieve 100% training accuracy. The true engineering skill lies in preventing overfitting so the model performs reliably in the wild.

What Is It?

Simple Explanation

Underfitting (High Bias): The model is too dumb/simple. It fails on both practice homework and the final exam. Overfitting (High Variance): The model is a parrot. It memorized every practice question word-for-word, but flunks the real exam when questions change slightly. Regularization: A leash placed on the model to keep it from getting too complicated.

Technical Explanation

Generalization error decomposes into three additive terms:

\\ext{Error} = \\ext{Bias}^2 + \\ext{Variance} + \sigma^2_{irreducible}

. High bias denotes underfitting due to overly restrictive hypothesis space. High variance denotes overfitting where

\\ext{Var}(\hat{f}) \gg 0

. Regularization adds penalty terms

\lambda \|\mathbf{w}\|_2^2

(Ridge) or

\lambda \|\mathbf{w}\|_1

(Lasso) to constrain the model parameter norm.

📚 Key Terms

High Bias (Underfitting)

Simple: The model is too simple to capture the underlying pattern.

Technical: Error stemming from erroneous assumptions in the learning algorithm; manifests as high training error and high validation error.

High Variance (Overfitting)

Simple: The model memorized noise and quirks in the training data.

Technical: Error stemming from extreme sensitivity to small fluctuations in training set; manifests as near-zero training error but high validation error.

L2 Regularization (Ridge)

Simple: Penalizing large weights to make the model smoother and simpler.

Technical: Adds squared Euclidean norm

\lambda \sum w_j^2

to the loss function, shrinking weights toward zero.

L1 Regularization (Lasso)

Simple: Driving unimportant feature weights completely to zero for automatic feature selection.

Technical: Adds Manhattan norm

\lambda \sum |w_j|

to the loss function, inducing sparsity in weight vectors.

📐 Mathematical Foundation

LaTeX KaTeX

\mathcal{L}_{Ridge} = \text{MSE} + \lambda \sum_{j=1}^D w_j^2, \quad \mathcal{L}_{Lasso} = \text{MSE} + \lambda \sum_{j=1}^D |w_j|

What does it mean?

Ridge (L2) and Lasso (L1) loss functions: augment standard MSE with penalty on weight magnitudes controlled by hyperparameter lambda.

Understanding the Symbols

Symbol / Term	Meaning	Beginner Interpretation
$\lambda$	Regularization strength	Higher lambda enforces simpler, smoother models
$w_j$	Model weights	Coefficients of features

📊 Numerical Example

Large weight w=10 with lambda=0.1. Ridge penalty is 0.1 * 100 = 10.0 added to loss.

\text{Penalty} = 0.1 \times 10^2 = 10.0

Result: Optimizer forces weight down to avoid high penalty

⚙️ How Does It Work?

Monitor Learning Curves

Plot training loss and validation loss over model complexity or epochs.

Detect Divergence

When training loss keeps falling but validation loss starts rising, overfitting has begun.

Apply Regularization

Increase lambda, prune decision trees, apply dropout, or collect more training data.

🌍 Where Is It Used?

Genomic Disease Prediction

Problem: Dataset has 500 patients but 20,000 gene features (high risk of extreme overfitting).

Solution: Lasso (L1) regularization forces 19,950 uninformative gene weights to 0, isolating the 50 truly causal mutations.

Financial Asset Valuation

Problem: High-degree polynomial regression overfits noisy historical stock price spikes.

Solution: Ridge regularization shrinks coefficients, producing a smooth generalized economic trend.

🎯 Try It Yourself

Your deep learning model gets 99.8% accuracy on training data, but only 64.2% accuracy on validation data. What is happening?

Interactive activity component

🐍 Python Hands-On: Taming Overfitting with Ridge Regularization

See a high-degree polynomial overfit and how Ridge regression restores generalization.

Open Colab

1Install Required Packages

For Google Colab or Jupyter notebooks, run this shell command to install the required dependencies:

Terminal / Notebook Install Commandbash

!pip install scikit-learn numpy

Packages:

scikit-learnnumpy

2Import Libraries

Import Statementspython

1 from sklearn.preprocessing import PolynomialFeatures
2 from sklearn.linear_model import LinearRegression, Ridge
3 import numpy as np

from sklearn.preprocessing import PolynomialFeatures → Create polynomial features

from sklearn.linear_model import LinearRegression, Ridge → Linear & Ridge models

import numpy as np → NumPy arrays

3Demonstration Code & Logic

Python Execution Codepython

1 # Noisy sine wave data (only 6 points)
2 X = np.array([[0.1], [0.3], [0.5], [0.7], [0.85], [1.0]])
3 y = np.sin(X.ravel() * 3.14) + np.random.normal(0, 0.05, 6)
4  
5 # Expand to 5th-degree polynomial features
6 poly = PolynomialFeatures(degree=5)
7 X_poly = poly.fit_transform(X)
8  
9 # 1. Unregularized Model (Wild Overfitting!):
10 overfit_model = LinearRegression().fit(X_poly, y)
11 print("Unregularized Max Weight:", max(abs(overfit_model.coef_)).round(1))
12  
13 # 2. Ridge Regularized Model (Constrained weights!):
14 ridge_model = Ridge(alpha=1.0).fit(X_poly, y)
15 print("Ridge Regularized Max Weight:", max(abs(ridge_model.coef_)).round(1))

Step-by-Step Code Breakdown:

# Noisy sine wave data (only 6 points)

Small dataset

X = np.array([[0.1], [0.3], [0.5], [0.7], [0.85], [1.0]])

Input points

y = np.sin(X.ravel() * 3.14) + np.random.normal(0, 0.05, 6)

Noisy targets

# Expand to 5th-degree polynomial features

High complexity

poly = PolynomialFeatures(degree=5)

Creates x, x^2, x^3, x^4, x^5

X_poly = poly.fit_transform(X)

Transform features

# 1. Unregularized Model (Wild Overfitting!):

Standard OLS

overfit_model = LinearRegression().fit(X_poly, y)

Fit without leash

print("Unregularized Max Weight:", max(abs(overfit_model.coef_)).round(1))

Exploding weights

# 2. Ridge Regularized Model (Constrained weights!):

L2 penalty

ridge_model = Ridge(alpha=1.0).fit(X_poly, y)

Fit with alpha=1.0

print("Ridge Regularized Max Weight:", max(abs(ridge_model.coef_)).round(1))

Tamed weights

📋 Complete Script (All-in-One)

Click "Copy for Colab", open Google Colab, paste into a single cell, and run.

Google Colab Notebook Scriptpython

1 # =========================================================
2 # Taming Overfitting with Ridge Regularization
3 # Prepared for Google Colab (Run with Shift + Enter)
4 # =========================================================
5  
6 # Step 1: Install required packages
7 !pip install scikit-learn numpy
8  
9 # Step 2: Import libraries
10 from sklearn.preprocessing import PolynomialFeatures
11 from sklearn.linear_model import LinearRegression, Ridge
12 import numpy as np
13  
14 # Step 3: Run Demonstration
15 # Noisy sine wave data (only 6 points)
16 X = np.array([[0.1], [0.3], [0.5], [0.7], [0.85], [1.0]])
17 y = np.sin(X.ravel() * 3.14) + np.random.normal(0, 0.05, 6)
18  
19 # Expand to 5th-degree polynomial features
20 poly = PolynomialFeatures(degree=5)
21 X_poly = poly.fit_transform(X)
22  
23 # 1. Unregularized Model (Wild Overfitting!):
24 overfit_model = LinearRegression().fit(X_poly, y)
25 print("Unregularized Max Weight:", max(abs(overfit_model.coef_)).round(1))
26  
27 # 2. Ridge Regularized Model (Constrained weights!):
28 ridge_model = Ridge(alpha=1.0).fit(X_poly, y)
29 print("Ridge Regularized Max Weight:", max(abs(ridge_model.coef_)).round(1))

⚡ How to Run in Google Colab (3 Quick Steps)

Click "Copy for Google Colab" above.
Click Open New Google Colab Notebook (opens in a new tab).
Click on the first empty code cell, press Ctrl + V (or Cmd + V) to paste, then press Shift + Enter to execute!

Launch Blank Colab Notebook

🔄 Algorithmic Execution Flow

Unregularized Fit

Weights explode to massive magnitudes (+/- 800) to pass through every noisy sample.

→

Ridge Fit

L2 penalty keeps weights small (< 2.0), producing a smooth generalized curve.

📥 Input / Initial State

6 noisy sample points fitted with 5th degree polynomial

📤 Expected Output

Unregularized Max Weight: ~840.2 (Overfitting)
Ridge Regularized Max Weight: ~1.8 (Tamed & Generalized)

💡 What Does This Demonstration Show?

Ridge regularization squashed runaway parameter explosion, preventing the model from oscillating wildly between points.

Course Learning Resource • Slide DeckLesson 417 Slides

Lesson 4: Python, Data Preparation and Classical ML Foundations

NumPy arrays, pandas DataFrames, train-test splits, and scikit-learn models.

Author: Dr. Syed Muzamil Basha• FDP Approved Curriculum

Open in New Tab

📝 Learning Check

Test your understanding with these questions. Remember, this is for learning, not examination!

Question 1: Which regularization method has the property of driving some feature coefficients to exactly ZERO, effectively acting as automatic feature selection?

Question 2: What causes overfitting?

Question 3: What is regularization in machine learning?

Question 4: How can you detect overfitting?

Question 5: What is the bias-variance tradeoff?

Previous: Model Evaluation: Beyond Simple Accuracy Next: Introduction to Scikit-Learn: The Industry Workhorse

1	from sklearn.preprocessing import PolynomialFeatures
2	from sklearn.linear_model import LinearRegression, Ridge
3	import numpy as np

1	# =========================================================
2	# Taming Overfitting with Ridge Regularization
3	# Prepared for Google Colab (Run with Shift + Enter)
4	# =========================================================
5
6	# Step 1: Install required packages
7	!pip install scikit-learn numpy
8
9	# Step 2: Import libraries
10	from sklearn.preprocessing import PolynomialFeatures
11	from sklearn.linear_model import LinearRegression, Ridge
12	import numpy as np
13
14	# Step 3: Run Demonstration
15	# Noisy sine wave data (only 6 points)
16	X = np.array([[0.1], [0.3], [0.5], [0.7], [0.85], [1.0]])
17	y = np.sin(X.ravel() * 3.14) + np.random.normal(0, 0.05, 6)
18
19	# Expand to 5th-degree polynomial features
20	poly = PolynomialFeatures(degree=5)
21	X_poly = poly.fit_transform(X)
22
23	# 1. Unregularized Model (Wild Overfitting!):
24	overfit_model = LinearRegression().fit(X_poly, y)
25	print("Unregularized Max Weight:", max(abs(overfit_model.coef_)).round(1))
26
27	# 2. Ridge Regularized Model (Constrained weights!):
28	ridge_model = Ridge(alpha=1.0).fit(X_poly, y)
29	print("Ridge Regularized Max Weight:", max(abs(ridge_model.coef_)).round(1))