First time studying this topic?

Launch the interactive walkthrough to discover each component, including the 3D Q-Sphere Lab & Python in Colab.

Topic 2.4

Classification Algorithms & Decision Boundaries

Explore fundamental classification algorithms: Logistic Regression, Support Vector Machines (SVM), and Decision Trees.

Learning Objectives

✓Understand how Logistic Regression maps inputs to probabilities using the Sigmoid function
✓Visualize decision boundaries separating classes in feature space
✓Compare linear classifiers with non-linear kernel SVMs and decision trees

📖Let's Begin with a Story

In 1936, British statistician Ronald Fisher measured the length and width of petals and sepals from 150 Iris flowers belonging to three species: Setosa, Versicolor, and Virginica. When you plot petal length against petal width on a 2D graph, Setosa forms a tight cluster completely isolated from the other two species. A single straight line can separate Setosa from the rest. This line is a "decision boundary." Classification is the mathematical science of drawing boundaries through feature space—whether straight lines, curving polynomial contours, or hyperplanes in 1,000 dimensions—to cleanly separate distinct categories.

Why Should You Learn Classification Algorithms & Decision Boundaries?

🎯 Practical & Engineering Stakes

Classification is the core of AI decision-making

Whether deciding if an email is spam, an X-ray has cancer, or an autonomous car sees a pedestrian, classification algorithms provide the discrete answers that drive automated actions.

What Is It?

Simple Explanation

Classification is like drawing a fence between different groups of animals on a field. If you have sheep on one side and goats on the other, the fence is the decision boundary. When a new animal enters the field, you check which side of the fence it is on to decide whether it is a sheep or a goat.

Technical Explanation

Classification partitions feature space

\mathbb{R}^d

into

C

decision regions

\mathcal{R}_1, \dots, \mathcal{R}_C

separated by decision surfaces

\{\mathbf{x} : g_k(\mathbf{x}) = g_j(\mathbf{x})\}

. In binary classification, Logistic Regression models the posterior log-odds as a linear combination:

\ln\\frac{P(Y=1|\mathbf{x})}{1-P(Y=1|\mathbf{x})} = \mathbf{w}^T \mathbf{x} + b

📚 Key Terms

Sigmoid Function ( $\sigma$ )

Simple: An S-shaped curve that squashes any number into a probability between 0 and 1.

Technical: The activation

\sigma(z) = \frac{1}{1 + e^{-z}}

, mapping

\mathbb{R} \rightarrow (0, 1)

with derivative

\sigma'(z) = \sigma(z)(1 - \sigma(z))

Decision Boundary

Simple: The border separating one class from another in the feature space.

Technical: The geometric hypersurface where predicted class probabilities are equal (

P(Y=1|\mathbf{x}) = 0.5

Support Vector Machine (SVM)

Simple: An algorithm that draws the widest possible road between two groups.

Technical: Max-margin classifier that finds the separating hyperplane maximizing the margin distance

\frac{2}{\|\mathbf{w}\|}

to the nearest support vectors.

📐 Mathematical Foundation

LaTeX KaTeX

P(Y=1|x) = \sigma(\mathbf{w}^T \mathbf{x} + b) = \frac{1}{1 + e^{-(\mathbf{w}^T \mathbf{x} + b)}}

What does it mean?

The Logistic Regression hypothesis: passes linear combination through the sigmoid function to output a calibrated probability between 0 and 1.

Understanding the Symbols

Symbol / Term	Meaning	Beginner Interpretation
$\sigma(z)$	Sigmoid activation	S-shaped probability mapping
$\mathbf{w}$	Weight vector	Feature importances
$b$	$Bias$	Threshold shift

📊 Numerical Example

Linear score z = 0.0 results in exactly 50% probability.

\sigma(0) = \frac{1}{1 + e^{-0}} = \frac{1}{1 + 1} = 0.50

Result: P = 0.50 (Exact decision boundary threshold)

⚙️ How Does It Work?

Linear Combination

Compute dot product of features with learned weights: z = w1*x1 + w2*x2 + b.

Probability Transformation

Pass z through sigmoid function to obtain probability p = 1 / (1 + exp(-z)).

Thresholding

If p >= 0.5, predict Class 1; otherwise predict Class 0.

🌍 Where Is It Used?

Spam Detection

Problem: Classifying incoming emails as spam or inbox.

Solution: Logistic regression calculates probability based on token frequency features.

Credit Card Fraud Prevention

Problem: Flagging fraudulent card transactions.

Solution: SVM with RBF kernel detects non-linear fraud clusters in high-dimensional spend spaces.

🎯 Try It Yourself

If a logistic regression model computes z = w*x + b = 2.0, what does the sigmoid function output?

Interactive activity component

🐍 Python Hands-On: Classification with Logistic Regression & Train-Test Split

Train a Logistic Regression classifier on labeled feature data and evaluate generalization accuracy on unseen test data.

Open Colab

1Install Required Packages

For Google Colab or Jupyter notebooks, run this shell command to install the required dependencies:

Terminal / Notebook Install Commandbash

!pip install scikit-learn

Packages:

scikit-learn

2Import Libraries

Import Statementspython

1 from sklearn.model_selection import train_test_split
2 from sklearn.linear_model import LogisticRegression
3 from sklearn.metrics import accuracy_score

from sklearn.model_selection import train_test_split → Dataset split utility

from sklearn.linear_model import LogisticRegression → Logistic regression classifier

from sklearn.metrics import accuracy_score → Accuracy score metric

3Demonstration Code & Logic

Python Execution Codepython

1 # Sample feature data and class labels
2 X = [[1], [2], [3], [4], [5], [1], [2], [3], [4], [5], [1], [2], [3], [4], [5], [1], [2], [3], [4], [5]]
3 y = [1, 2, 3, 4, 5, 1, 3, 3, 4, 5, 1, 2, 3, 4, 5, 1, 1, 3, 4, 5]  # Labels
4  
5 # Split data into training (80%) and testing (20%) sets
6 X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
7  
8 # Train a Logistic Regression model
9 model = LogisticRegression()
10 model.fit(X_train, y_train)
11  
12 # Make predictions on unseen test data
13 y_pred = model.predict(X_test)
14  
15 # Evaluate model accuracy
16 accuracy = accuracy_score(y_test, y_pred)
17 print("Accuracy:", accuracy)

Step-by-Step Code Breakdown:

# Sample feature data and class labels

Dataset definition

X = [[1], [2], [3], [4], [5], [1], [2], [3], [4], [5], [1], [2], [3], [4], [5], [1], [2], [3], [4], [5]]

Feature values

y = [1, 2, 3, 4, 5, 1, 3, 3, 4, 5, 1, 2, 3, 4, 5, 1, 1, 3, 4, 5] # Labels

Target classes

# Split data into training (80%) and testing (20%) sets

Train/test split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

Split datasets

# Train a Logistic Regression model

Model creation

model = LogisticRegression()

Instantiate classifier

model.fit(X_train, y_train)

Fit model on training set

# Make predictions on unseen test data

Inference

y_pred = model.predict(X_test)

Predict classes

# Evaluate model accuracy

Evaluation

accuracy = accuracy_score(y_test, y_pred)

Compute accuracy

print("Accuracy:", accuracy)

Display accuracy score

📋 Complete Script (All-in-One)

Click "Copy for Colab", open Google Colab, paste into a single cell, and run.

Google Colab Notebook Scriptpython

1 # =========================================================
2 # Classification with Logistic Regression & Train-Test Split
3 # Prepared for Google Colab (Run with Shift + Enter)
4 # =========================================================
5  
6 # Step 1: Install required packages
7 !pip install scikit-learn
8  
9 # Step 2: Import libraries
10 from sklearn.model_selection import train_test_split
11 from sklearn.linear_model import LogisticRegression
12 from sklearn.metrics import accuracy_score
13  
14 # Step 3: Run Demonstration
15 # Sample feature data and class labels
16 X = [[1], [2], [3], [4], [5], [1], [2], [3], [4], [5], [1], [2], [3], [4], [5], [1], [2], [3], [4], [5]]
17 y = [1, 2, 3, 4, 5, 1, 3, 3, 4, 5, 1, 2, 3, 4, 5, 1, 1, 3, 4, 5]  # Labels
18  
19 # Split data into training (80%) and testing (20%) sets
20 X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
21  
22 # Train a Logistic Regression model
23 model = LogisticRegression()
24 model.fit(X_train, y_train)
25  
26 # Make predictions on unseen test data
27 y_pred = model.predict(X_test)
28  
29 # Evaluate model accuracy
30 accuracy = accuracy_score(y_test, y_pred)
31 print("Accuracy:", accuracy)

⚡ How to Run in Google Colab (3 Quick Steps)

Click "Copy for Google Colab" above.
Click Open New Google Colab Notebook (opens in a new tab).
Click on the first empty code cell, press Ctrl + V (or Cmd + V) to paste, then press Shift + Enter to execute!

Launch Blank Colab Notebook

🔄 Algorithmic Execution Flow

Data Partitioning

Splits 20 samples into 16 training instances and 4 held-out testing instances.

→

Model Fitting

LogisticRegression optimizes class decision boundaries on X_train and y_train.

→

Unseen Testing

Calculates classification accuracy on X_test.

📥 Input / Initial State

20 feature-label pairs split 80/20

📤 Expected Output

Accuracy: 0.75

💡 What Does This Demonstration Show?

The model correctly generalizes to unseen test cases by discovering the relationship between numerical inputs and target class labels.

Course Learning Resource • Slide DeckLesson 417 Slides

Lesson 4: Python, Data Preparation and Classical ML Foundations

Supervised classification pipelines, train-test splits, and logistic decision boundaries.

Author: Dr. Syed Muzamil Basha• FDP Approved Curriculum

Open in New Tab

📝 Learning Check

Test your understanding with these questions. Remember, this is for learning, not examination!

Question 1: What is the mathematical threshold typically used in binary logistic regression to assign Class 1?

Question 2: What type of output does a classification algorithm produce?

Question 3: What is the difference between binary and multi-class classification?

Question 4: Which algorithm finds the optimal hyperplane separating two classes?

Question 5: What is a confusion matrix used for?

Previous: Training and Testing: The Golden Rule of ML Next: Regression: Continuous Value Prediction

1	# =========================================================
2	# Classification with Logistic Regression & Train-Test Split
3	# Prepared for Google Colab (Run with Shift + Enter)
4	# =========================================================
5
6	# Step 1: Install required packages
7	!pip install scikit-learn
8
9	# Step 2: Import libraries
10	from sklearn.model_selection import train_test_split
11	from sklearn.linear_model import LogisticRegression
12	from sklearn.metrics import accuracy_score
13
14	# Step 3: Run Demonstration
15	# Sample feature data and class labels
16	X = [[1], [2], [3], [4], [5], [1], [2], [3], [4], [5], [1], [2], [3], [4], [5], [1], [2], [3], [4], [5]]
17	y = [1, 2, 3, 4, 5, 1, 3, 3, 4, 5, 1, 2, 3, 4, 5, 1, 1, 3, 4, 5] # Labels
18
19	# Split data into training (80%) and testing (20%) sets
20	X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
21
22	# Train a Logistic Regression model
23	model = LogisticRegression()
24	model.fit(X_train, y_train)
25
26	# Make predictions on unseen test data
27	y_pred = model.predict(X_test)
28
29	# Evaluate model accuracy
30	accuracy = accuracy_score(y_test, y_pred)
31	print("Accuracy:", accuracy)