First time studying this topic?

Launch the interactive walkthrough to discover each component, including the 3D Q-Sphere Lab & Python in Colab.

Topic 3.7

Gradient Descent & Backpropagation

Unravel the mathematical core of deep learning: computing analytical gradients via the multivariable Chain Rule and stepping down loss landscapes.

Learning Objectives

✓Understand gradient vectors as directions of steepest ascent
✓Trace Backpropagation using the calculus Chain Rule
✓Compare Batch GD, Stochastic GD (SGD), and Adam optimizers
✓Tune learning rate hyperparameters and avoid saddle points

📖Let's Begin with a Story

Imagine being dropped blindfolded onto a rugged mountain range in dense fog. Your mission is to find the deepest valley (the minimum loss) as quickly as possible. You cannot see ahead. What do you do? You extend your foot and feel the slope of the ground directly beneath you. Whichever direction slopes downward most steeply, you take one cautious step in that direction. You repeat this step after step. In machine learning, this is Gradient Descent. The slope beneath your feet is the Gradient vector (calculated via the calculus Chain Rule / Backpropagation), and your stride length is the Learning Rate (

\alpha

Why Should You Learn Gradient Descent & Backpropagation?

🎯 Practical & Engineering Stakes

The core algorithm of modern AI

Without gradient descent and backpropagation, training billion-parameter neural networks would be computationally impossible. It is the universal optimization engine powering all deep learning.

What Is It?

Simple Explanation

Gradient descent is walking down a hill to find the bottom of the valley: 1. The Gradient: An arrow pointing uphill. To go down, we step in the exact opposite direction! 2. Learning Rate: How big of a step you take. Too big, and you stumble over the valley; too small, and you take 100 years to reach the bottom. 3. Backpropagation: The math trick (the Chain Rule) that calculates how to tweak every single weight in the network.

Technical Explanation

Gradient descent updates parameter vector

\boldsymbol{\theta} \in \mathbb{R}^P

along the negative gradient of the empirical risk:

\boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t - \alpha \nabla_\theta \mathcal{L}(\boldsymbol{\theta}_t)

. Backpropagation evaluates the gradient efficiently in

O(P)

operations by applying reverse-mode automatic differentiation across the computational graph using the multivariate chain rule

\frac{\partial \mathcal{L}}{\partial w_{ij}^{(l)}} = \delta_i^{(l)} a_j^{(l-1)}

📚 Key Terms

$Backpropagation$

Simple: Calculating who is to blame for the mistake, working backwards from the output to the first layer.

Technical: Reverse-mode automatic differentiation propagating error sensitivities

\delta^{(l)}

backwards through transposed weight matrices.

Learning Rate ( $\alpha$ )

Simple: The size of the step taken down the hill on each update.

Technical: Scalar hyperparameter scaling the gradient step size during parameter updates.

Adam Optimizer

Simple: A smart version of gradient descent that gives each parameter its own custom, adaptive step size.

Technical: Adaptive Moment Estimation combining first-order momentum (running average of gradients) and second-order RMSprop (running average of squared gradients).

📐 Mathematical Foundation

LaTeX KaTeX

\boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t - \alpha \nabla_{\boldsymbol{\theta}} \mathcal{L}(\boldsymbol{\theta}_t)

What does it mean?

The Gradient Descent update rule: subtract the learning rate alpha times the gradient vector from the current parameter weights.

Understanding the Symbols

Symbol / Term	Meaning	Beginner Interpretation
$\boldsymbol{\theta}_t$	Current weights	Where we are on the mountain
$\alpha$	Learning rate	Step size
$\nabla \mathcal{L}$	Gradient vector	Slope of steepest ascent

📊 Numerical Example

Current weight w = 5.0, gradient dL/dw = 4.0, learning rate alpha = 0.1.

w_{new} = 5.0 - (0.1 \times 4.0) = 5.0 - 0.4 = 4.6

Result: Weight moves from 5.0 down to 4.6

⚙️ How Does It Work?

Forward Pass

Compute predictions and evaluate scalar loss L.

Backward Pass (Backprop)

Traverse graph in reverse, computing partial derivatives dL/dw for every weight.

Optimizer Step

Update parameters using gradient descent or Adam step.

Zero Gradients

Reset accumulated gradient buffers before the next iteration.

🌍 Where Is It Used?

Training Large Language Models

Problem: Optimizing 175 billion weights simultaneously across GPU clusters.

Solution: Distributed AdamW optimizer synchronizes gradients across thousands of tensor cores.

🎯 Try It Yourself

What happens if your learning rate (alpha) is set way too high (e.g., alpha = 100.0)?

Interactive activity component

🐍 Python Hands-On: PyTorch Automatic Differentiation (Autograd)

Witness PyTorch calculate exact analytical derivatives automatically using computational graphs.

Open Colab

1Install Required Packages

For Google Colab or Jupyter notebooks, run this shell command to install the required dependencies:

Terminal / Notebook Install Commandbash

!pip install torch

Packages:

torch

2Import Libraries

Import Statementspython

1 import torch

import torch → PyTorch autograd engine

3Demonstration Code & Logic

Python Execution Codepython

1 # Define a learnable weight tensor w with requires_grad=True
2 w = torch.tensor(3.0, requires_grad=True)
3  
4 # Define a computational function: Loss = w^3 - 2*w + 5
5 loss = w**3 - 2*w + 5
6 print("Forward Pass Loss Value:", loss.item())
7  
8 # Trigger Backpropagation via the Chain Rule!
9 loss.backward()
10  
11 # Calculus check: dLoss/dw = 3*w^2 - 2 = 3*(3^2) - 2 = 27 - 2 = 25
12 print("PyTorch Autograd Computed Gradient dLoss/dw:", w.grad.item())
13  
14 # Take one Gradient Descent step with learning rate 0.01:
15 with torch.no_grad():
16     w -= 0.01 * w.grad
17 print("Updated Weight w:", round(w.item(), 3))

Step-by-Step Code Breakdown:

# Define a learnable weight tensor w with requires_grad=True

Track gradients

w = torch.tensor(3.0, requires_grad=True)

Variable w

# Define a computational function: Loss = w^3 - 2*w + 5

Function

loss = w**3 - 2*w + 5

Compute forward pass

print("Forward Pass Loss Value:", loss.item())

Loss = 3^3 - 6 + 5 = 26

# Trigger Backpropagation via the Chain Rule!

Backward pass

loss.backward()

Calculates dLoss/dw automatically!

# Calculus check: dLoss/dw = 3*w^2 - 2 = 3*(3^2) - 2 = 27 - 2 = 25

Analytical derivative

print("PyTorch Autograd Computed Gradient dLoss/dw:", w.grad.item())

Print gradient

# Take one Gradient Descent step with learning rate 0.01:

Optimizer step

with torch.no_grad():

Disable tracking during update

w -= 0.01 * w.grad

w = 3.0 - (0.01 * 25) = 2.75

print("Updated Weight w:", round(w.item(), 3))

Print updated weight

📋 Complete Script (All-in-One)

Click "Copy for Colab", open Google Colab, paste into a single cell, and run.

Google Colab Notebook Scriptpython

1 # =========================================================
2 # PyTorch Automatic Differentiation (Autograd)
3 # Prepared for Google Colab (Run with Shift + Enter)
4 # =========================================================
5  
6 # Step 1: Install required packages
7 !pip install torch
8  
9 # Step 2: Import libraries
10 import torch
11  
12 # Step 3: Run Demonstration
13 # Define a learnable weight tensor w with requires_grad=True
14 w = torch.tensor(3.0, requires_grad=True)
15  
16 # Define a computational function: Loss = w^3 - 2*w + 5
17 loss = w**3 - 2*w + 5
18 print("Forward Pass Loss Value:", loss.item())
19  
20 # Trigger Backpropagation via the Chain Rule!
21 loss.backward()
22  
23 # Calculus check: dLoss/dw = 3*w^2 - 2 = 3*(3^2) - 2 = 27 - 2 = 25
24 print("PyTorch Autograd Computed Gradient dLoss/dw:", w.grad.item())
25  
26 # Take one Gradient Descent step with learning rate 0.01:
27 with torch.no_grad():
28     w -= 0.01 * w.grad
29 print("Updated Weight w:", round(w.item(), 3))

⚡ How to Run in Google Colab (3 Quick Steps)

Click "Copy for Google Colab" above.
Click Open New Google Colab Notebook (opens in a new tab).
Click on the first empty code cell, press Ctrl + V (or Cmd + V) to paste, then press Shift + Enter to execute!

Launch Blank Colab Notebook

🔄 Algorithmic Execution Flow

Graph Construction

PyTorch dynamically constructs computational DAG linking w to loss.

→

Backward Sweep

loss.backward() evaluates reverse-mode gradient: 3*(3^2) - 2 = 25.0.

→

Weight Step

Subtracts alpha * gradient, stepping weight from 3.0 down to 2.75 toward minimum.

📥 Input / Initial State

Scalar variable w = 3.0

📤 Expected Output

Forward Pass Loss Value: 26.0
PyTorch Autograd Computed Gradient dLoss/dw: 25.0
Updated Weight w: 2.75

💡 What Does This Demonstration Show?

PyTorch Autograd computes exact calculus derivatives across arbitrary complex graphs without manual derivation.

Course Learning Resource • Slide DeckLesson 512 Slides

Lesson 5: Classical Deep Learning with PyTorch

CNN, MNIST dataset, backpropagation, loss functions, and optimization.

Author: Dr. Syed Muzamil Basha• FDP Approved Curriculum

Open in New Tab

📝 Learning Check

Test your understanding with these questions. Remember, this is for learning, not examination!

Question 1: What is the key advantage of the Adam optimizer over basic Stochastic Gradient Descent (SGD)?

Question 2: What is the gradient in gradient descent?

Question 3: What is the learning rate (η) in gradient descent?

Question 4: What is the difference between batch, stochastic, and mini-batch gradient descent?

Question 5: What is backpropagation?

Previous: Loss Functions: Cross-Entropy & Mean Squared Error Next: Introduction to TensorFlow and Keras

1	# =========================================================
2	# PyTorch Automatic Differentiation (Autograd)
3	# Prepared for Google Colab (Run with Shift + Enter)
4	# =========================================================
5
6	# Step 1: Install required packages
7	!pip install torch
8
9	# Step 2: Import libraries
10	import torch
11
12	# Step 3: Run Demonstration
13	# Define a learnable weight tensor w with requires_grad=True
14	w = torch.tensor(3.0, requires_grad=True)
15
16	# Define a computational function: Loss = w^3 - 2*w + 5
17	loss = w*3 - 2w + 5
18	print("Forward Pass Loss Value:", loss.item())
19
20	# Trigger Backpropagation via the Chain Rule!
21	loss.backward()
22
23	# Calculus check: dLoss/dw = 3w^2 - 2 = 3(3^2) - 2 = 27 - 2 = 25
24	print("PyTorch Autograd Computed Gradient dLoss/dw:", w.grad.item())
25
26	# Take one Gradient Descent step with learning rate 0.01:
27	with torch.no_grad():
28	w -= 0.01 * w.grad
29	print("Updated Weight w:", round(w.item(), 3))