First time studying this topic?

Launch the interactive walkthrough to discover each component, including the 3D Q-Sphere Lab & Python in Colab.

Topic 1.7

Supervised Learning: Classification & Regression

Master the most commercially dominant form of AI: mapping inputs to targets via classification and regression.

Learning Objectives

✓Distinguish between discrete classification and continuous regression tasks
✓Examine common supervised algorithms: Linear Regression, Logistic Regression, Decision Trees, SVMs
✓Understand train/test splits and generalization error

📖Let's Begin with a Story

Imagine working at an emergency hospital triage center. Every minute, patients walk in with varying symptoms, blood pressures, and heart rates. The triage AI must make two distinct kinds of predictions: 1. Classification: Is this patient having an acute cardiac infarction? (Binary answer: YES or NO; or Category: High, Medium, Low risk). 2. Regression: What is this patient's expected recovery time in days? (Continuous number: 2.4 days, 5.8 days). Both models learn from historical hospital records where patient outcomes are known with certainty. Supervised learning powers the vast majority of industrial AI today.

Why Should You Learn Supervised Learning: Classification & Regression?

🎯 Practical & Engineering Stakes

Why Supervised Learning is foundational:

1. Direct business utility: Almost all enterprise AI initiatives (churn prediction, credit scoring, lead scoring) are supervised classification or regression problems. 2. Clear evaluation metrics: Accuracy, precision, recall, RMSE, and

R^2

provide mathematically concrete benchmarks of model quality.

What Is It?

Simple Explanation

Supervised learning is like learning with an answer key. If you want the computer to predict a category (like "Dog" vs "Cat"), that is Classification. If you want it to predict a continuous number (like house price or tomorrow's temperature), that is Regression.

Technical Explanation

Given an empirical sample

\mathcal{D} = \{(\mathbf{x}_i, y_i)\}_{i=1}^N \stackrel{i.i.d.}{\sim} \mathcal{P}_{\mathcal{X}\mathcal{Y}}

, supervised learning estimates conditional expectation

\mathbb{E}[Y|X=\mathbf{x}]

(for regression

y \in \mathbb{R}

) or class posterior

P(Y=c|X=\mathbf{x})

(for classification

y \in \{1, \dots, C\}

📚 Key Terms

$Classification$

Simple: Predicting a categorical label or group membership (e.g. Yes/No, Red/Green/Blue).

Technical: Mapping feature space to a discrete set of class identifiers:

f: \mathbb{R}^d \rightarrow \{0, 1, \dots, C-1\}

$Regression$

Simple: Predicting a continuous numerical quantity (e.g. price, speed, temperature).

Technical: Mapping feature space to a continuous real-valued scalar:

f: \mathbb{R}^d \rightarrow \mathbb{R}

📐 Mathematical Foundation

LaTeX KaTeX

\text{MSE} = \frac{1}{N} \sum_{i=1}^N (y_i - \hat{y}_i)^2

What does it mean?

Mean Squared Error (MSE): the standard loss function for regression tasks, penalizing larger errors quadratically.

Understanding the Symbols

Symbol / Term	Meaning	Beginner Interpretation
$y_i$	True value	Actual recorded outcome
$\hat{y}_i$	Predicted value	Model prediction
$N$	Number of samples	Sample size

📊 Numerical Example

Errors for two predictions: true [10, 20], predicted [12, 19].

\text{MSE} = \frac{(10 - 12)^2 + (20 - 19)^2}{2} = \frac{4 + 1}{2} = 2.5

Result: $MSE = 2.5$

⚙️ How Does It Work?

Feature Matrix Construction

Assemble tabular matrix X (N samples, D features) and target vector y.

Loss Formulation

Define MSE for regression or Cross-Entropy for classification.

Model Optimization

Backpropagation or convex solver minimizes error over parameter space.

🌍 Where Is It Used?

Loan Default Prediction (Classification)

Problem: Banks need to classify applicants as Low Risk vs High Risk.

Solution: Logistic regression or XGBoost predicts probability of repayment.

Weather Forecasting (Regression)

Problem: Predicting peak temperature tomorrow in degrees Celsius.

Solution: Regression model trained on atmospheric pressure and satellite metrics.

🎯 Try It Yourself

An algorithm estimates the exact market rental price in dollars for apartments in New York. Is this Classification or Regression?

Interactive activity component

🐍 Python Hands-On: Classification vs Regression in Scikit-Learn

Train a classifier and a regressor in under 15 lines of code.

Open Colab

1Install Required Packages

For Google Colab or Jupyter notebooks, run this shell command to install the required dependencies:

Terminal / Notebook Install Commandbash

!pip install scikit-learn

Packages:

scikit-learn

2Import Libraries

Import Statementspython

1 from sklearn.linear_model import LinearRegression, LogisticRegression
2 import numpy as np

from sklearn.linear_model import LinearRegression, LogisticRegression → Import both models

import numpy as np → NumPy arrays

3Demonstration Code & Logic

Python Execution Codepython

1 # Hours studied vs Exam Score (0-100) -> REGRESSION
2 X_study = np.array([[1], [2], [4], [6], [8]])
3 y_score = np.array([35, 50, 70, 85, 95])
4 reg = LinearRegression().fit(X_study, y_score)
5 print(f"Predicted Score for 5 hours: {reg.predict([[5]])[0]:.1f}")
6  
7 # Hours studied vs Pass/Fail (0=Fail, 1=Pass) -> CLASSIFICATION
8 y_pass = np.array([0, 0, 1, 1, 1])
9 clf = LogisticRegression().fit(X_study, y_pass)
10 print(f"Pass/Fail Prediction for 3 hours: {clf.predict([[3]])[0]} (Probability: {clf.predict_proba([[3]])[0][1]:.2f})")

Step-by-Step Code Breakdown:

# Hours studied vs Exam Score (0-100) -> REGRESSION

Regression data

X_study = np.array([[1], [2], [4], [6], [8]])

Hours

y_score = np.array([35, 50, 70, 85, 95])

Continuous scores

reg = LinearRegression().fit(X_study, y_score)

Fit regressor

print(f"Predicted Score for 5 hours: {reg.predict([[5]])[0]:.1f}")

Regression output

# Hours studied vs Pass/Fail (0=Fail, 1=Pass) -> CLASSIFICATION

Classification data

y_pass = np.array([0, 0, 1, 1, 1])

Discrete classes

clf = LogisticRegression().fit(X_study, y_pass)

Fit classifier

print(f"Pass/Fail Prediction for 3 hours: {clf.predict([[3]])[0]} (Probability: {clf.predict_proba([[3]])[0][1]:.2f})")

Classification output

📋 Complete Script (All-in-One)

Click "Copy for Colab", open Google Colab, paste into a single cell, and run.

Google Colab Notebook Scriptpython

1 # =========================================================
2 # Classification vs Regression in Scikit-Learn
3 # Prepared for Google Colab (Run with Shift + Enter)
4 # =========================================================
5  
6 # Step 1: Install required packages
7 !pip install scikit-learn
8  
9 # Step 2: Import libraries
10 from sklearn.linear_model import LinearRegression, LogisticRegression
11 import numpy as np
12  
13 # Step 3: Run Demonstration
14 # Hours studied vs Exam Score (0-100) -> REGRESSION
15 X_study = np.array([[1], [2], [4], [6], [8]])
16 y_score = np.array([35, 50, 70, 85, 95])
17 reg = LinearRegression().fit(X_study, y_score)
18 print(f"Predicted Score for 5 hours: {reg.predict([[5]])[0]:.1f}")
19  
20 # Hours studied vs Pass/Fail (0=Fail, 1=Pass) -> CLASSIFICATION
21 y_pass = np.array([0, 0, 1, 1, 1])
22 clf = LogisticRegression().fit(X_study, y_pass)
23 print(f"Pass/Fail Prediction for 3 hours: {clf.predict([[3]])[0]} (Probability: {clf.predict_proba([[3]])[0][1]:.2f})")

⚡ How to Run in Google Colab (3 Quick Steps)

Click "Copy for Google Colab" above.
Click Open New Google Colab Notebook (opens in a new tab).
Click on the first empty code cell, press Ctrl + V (or Cmd + V) to paste, then press Shift + Enter to execute!

Launch Blank Colab Notebook

🔄 Algorithmic Execution Flow

Regression Fit

Fits best straight line predicting exact score.

→

Classification Fit

Fits sigmoid curve calculating probability of passing.

📥 Input / Initial State

Study hours: 5 for score, 3 for pass/fail

📤 Expected Output

Predicted Score for 5 hours: 77.0
Pass/Fail Prediction for 3 hours: 1 (Probability: 0.73)

💡 What Does This Demonstration Show?

Regression outputs an unbounded continuous score (77.0), while classification outputs a discrete class outcome (1 = Pass) with confidence 73%.

Course Learning Resource • Slide DeckLesson 211 Slides

Lesson 2: Classical Machine Learning as the Starting Point

Data, prediction, and the bridge to modern artificial intelligence.

Author: Dr. Syed Muzamil Basha• FDP Approved Curriculum

Open in New Tab

📝 Learning Check

Test your understanding with these questions. Remember, this is for learning, not examination!

Question 1: Which loss function is most widely used for training linear regression models?

Question 2: A hospital uses an AI model to predict whether a tumor is malignant or benign from MRI scans. Is this a classification or regression task?

Question 3: You compute MSE = (1/2) × [(10-12)² + (20-19)²] = 2.5 for a model. What does this value tell you?

Question 4: Why does supervised learning require a train/test split rather than evaluating the model on the same data it was trained on?

Question 5: In logistic regression for pass/fail prediction, the output probability for 3 study hours is 0.73. What does this mean practically?

Question 6: For predicting next month's electricity bill in dollars, which supervised learning approach is most appropriate?

Previous: The Three Main Types of Machine Learning Next: Unsupervised Learning: Clustering & Dimensionality Reduction

1	from sklearn.linear_model import LinearRegression, LogisticRegression
2	import numpy as np

1	# =========================================================
2	# Classification vs Regression in Scikit-Learn
3	# Prepared for Google Colab (Run with Shift + Enter)
4	# =========================================================
5
6	# Step 1: Install required packages
7	!pip install scikit-learn
8
9	# Step 2: Import libraries
10	from sklearn.linear_model import LinearRegression, LogisticRegression
11	import numpy as np
12
13	# Step 3: Run Demonstration
14	# Hours studied vs Exam Score (0-100) -> REGRESSION
15	X_study = np.array([[1], [2], [4], [6], [8]])
16	y_score = np.array([35, 50, 70, 85, 95])
17	reg = LinearRegression().fit(X_study, y_score)
18	print(f"Predicted Score for 5 hours: {reg.predict([[5]])[0]:.1f}")
19
20	# Hours studied vs Pass/Fail (0=Fail, 1=Pass) -> CLASSIFICATION
21	y_pass = np.array([0, 0, 1, 1, 1])
22	clf = LogisticRegression().fit(X_study, y_pass)
23	print(f"Pass/Fail Prediction for 3 hours: {clf.predict([[3]])[0]} (Probability: {clf.predict_proba([[3]])[0][1]:.2f})")