First time studying this topic?

Launch the interactive walkthrough to discover each component, including the 3D Q-Sphere Lab & Python in Colab.

Back to Deep Learning
Topic 3.9

Introduction to PyTorch: Dynamic Graphs & Research Standard

Master PyTorch, the favorite framework of AI researchers worldwide: dynamic computation graphs (Eager Mode), Tensors, and nn.Module.

Learning Objectives

  • ✓Understand PyTorch Tensors, GPU acceleration (.to("cuda")), and Autograd
  • ✓Build modular architectures by subclassing nn.Module
  • ✓Write clean, explicit PyTorch training loops (zero_grad, backward, step)

📖Let's Begin with a Story

In 2016, Soumith Chintala and the Meta AI Research team released PyTorch. At the time, other frameworks required compiling static computation graphs: you had to declare all operations in advance, and debugging was notoriously difficult. PyTorch did something revolutionary: it implemented Dynamic Computation Graphs ("Define-by-Run"). A PyTorch tensor acts exactly like a NumPy array, but with two superpowers: 1. It runs seamlessly on NVIDIA GPUs with 50x acceleration. 2. It tracks every mathematical operation dynamically for instant automatic differentiation. Today, over 80% of top academic research papers at CVPR, NeurIPS, and ICML, as well as models from OpenAI, Anthropic, and Hugging Face, are built in PyTorch.

Why Should You Learn Introduction to PyTorch: Dynamic Graphs & Research Standard?

🎯 Practical & Engineering Stakes

The AI Research & Generative AI Lingua Franca

If you want to read cutting-edge AI research, fine-tune open-source LLMs (Llama 3, Mistral), or understand Diffusion models, PyTorch is the essential framework you must master.

What Is It?

Simple Explanation

PyTorch is Pythonic: it feels like standard Python and NumPy, but with GPU rocket fuel. The standard PyTorch training loop has 5 steps you will write every time: 1. optimizer.zero_grad() (Clear old gradients) 2. y_pred = model(X) (Make predictions) 3. loss = criterion(y_pred, y) (Grade error) 4. loss.backward() (Calculate gradients) 5. optimizer.step() (Update weights!)

Technical Explanation

PyTorch centers on dynamic computation graphs constructed on the fly via C++ ATen tensor library and LibTorch backend. Memory is dynamically allocated via caching CUDA allocators. Subclassing torch.nn.Module provides state encapsulation and parameter registration.

📚 Key Terms

Dynamic Computation Graph

Simple: The graph of math operations is built on the fly as code runs, making debugging simple.
Technical: Define-by-run tape-based automatic differentiation recording operations as they are executed.

nn.Module

Simple: The master blueprint class in PyTorch used to build all models and layers.
Technical: Base class for all neural network modules, managing parameter buffers, sub-modules, and forward hooks.

optimizer.zero_grad()

Simple: Clearing the slate so old gradients don’t pile up on top of new ones.
Technical: Zeroes the .grad attributes of all registered parameters to prevent additive gradient accumulation across steps.

⚙️ How Does It Work?

1

Tensor Allocation

Create tensors and move to device (CPU or GPU CUDA).

2

Forward Evaluation

Pass input through model instance: outputs = model(inputs).

3

Autograd Backprop

loss.backward() computes partial derivatives across leaf tensors.

4

Optimizer Update

optimizer.step() shifts parameters along negative gradient.

🌍 Where Is It Used?

Generative AI & LLM Training

Problem: Training modern Transformer models like LLaMA and Stable Diffusion.

Solution: PyTorch native support for FlashAttention and Distributed Data Parallel (DDP) powers frontier AI research.

🎯 Try It Yourself

A student forgets to write "optimizer.zero_grad()" inside their PyTorch training loop. What will happen?

Interactive activity component

🐍 Python Hands-On: Complete PyTorch Training Loop from Scratch

Write a full 5-step PyTorch training loop and verify loss reduction.

Open Colab

1Install Required Packages

For Google Colab or Jupyter notebooks, run this shell command to install the required dependencies:

Terminal / Notebook Install Commandbash
!pip install torch
Packages:
torch

2Import Libraries

Import Statementspython
1import torch
2import torch.nn as nn
3import torch.optim as optim

import torch → Core PyTorch

import torch.nn as nn → Neural network modules

import torch.optim as optim → Optimization algorithms

3Demonstration Code & Logic

Python Execution Codepython
1# Synthetic dataset: y = 2.5 * x + 1.0
2X = torch.randn(100, 1)
3y = 2.5 * X + 1.0 + torch.randn(100, 1) * 0.1 # With noise
4 
5# Define linear model: 1 input -> 1 output
6model = nn.Linear(1, 1)
7criterion = nn.MSELoss()
8optimizer = optim.SGD(model.parameters(), lr=0.1)
9 
10# The 5-Step PyTorch Training Loop:
11for epoch in range(50):
12    optimizer.zero_grad()         # 1. Zero gradients
13    predictions = model(X)        # 2. Forward pass
14    loss = criterion(predictions, y) # 3. Calculate loss
15    loss.backward()               # 4. Backward pass
16    optimizer.step()              # 5. Optimizer step
17 
18learned_w = model.weight.item()
19learned_b = model.bias.item()
20print(f"Learned equation: y = {learned_w:.2f} * x + {learned_b:.2f} (Target: 2.50 * x + 1.00)")
Step-by-Step Code Breakdown:

# Synthetic dataset: y = 2.5 * x + 1.0

Synthetic data

X = torch.randn(100, 1)

100 input samples

y = 2.5 * X + 1.0 + torch.randn(100, 1) * 0.1 # With noise

Target labels

# Define linear model: 1 input -> 1 output

Model

model = nn.Linear(1, 1)

Single linear layer

criterion = nn.MSELoss()

MSE loss

optimizer = optim.SGD(model.parameters(), lr=0.1)

SGD optimizer

# The 5-Step PyTorch Training Loop:

Standard loop

for epoch in range(50):

Run 50 epochs

optimizer.zero_grad() # 1. Zero gradients

Step 1

predictions = model(X) # 2. Forward pass

Step 2

loss = criterion(predictions, y) # 3. Calculate loss

Step 3

loss.backward() # 4. Backward pass

Step 4

optimizer.step() # 5. Optimizer step

Step 5

learned_w = model.weight.item()

Get weight

learned_b = model.bias.item()

Get bias

print(f"Learned equation: y = {learned_w:.2f} * x + {learned_b:.2f} (Target: 2.50 * x + 1.00)")

Display result

📋 Complete Script (All-in-One)

Click "Copy for Colab", open Google Colab, paste into a single cell, and run.

Google Colab Notebook Scriptpython
1# =========================================================
2# Complete PyTorch Training Loop from Scratch
3# Prepared for Google Colab (Run with Shift + Enter)
4# =========================================================
5 
6# Step 1: Install required packages
7!pip install torch
8 
9# Step 2: Import libraries
10import torch
11import torch.nn as nn
12import torch.optim as optim
13 
14# Step 3: Run Demonstration
15# Synthetic dataset: y = 2.5 * x + 1.0
16X = torch.randn(100, 1)
17y = 2.5 * X + 1.0 + torch.randn(100, 1) * 0.1 # With noise
18 
19# Define linear model: 1 input -> 1 output
20model = nn.Linear(1, 1)
21criterion = nn.MSELoss()
22optimizer = optim.SGD(model.parameters(), lr=0.1)
23 
24# The 5-Step PyTorch Training Loop:
25for epoch in range(50):
26    optimizer.zero_grad()         # 1. Zero gradients
27    predictions = model(X)        # 2. Forward pass
28    loss = criterion(predictions, y) # 3. Calculate loss
29    loss.backward()               # 4. Backward pass
30    optimizer.step()              # 5. Optimizer step
31 
32learned_w = model.weight.item()
33learned_b = model.bias.item()
34print(f"Learned equation: y = {learned_w:.2f} * x + {learned_b:.2f} (Target: 2.50 * x + 1.00)")
⚡ How to Run in Google Colab (3 Quick Steps)
  1. Click "Copy for Google Colab" above.
  2. Click Open New Google Colab Notebook (opens in a new tab).
  3. Click on the first empty code cell, press Ctrl + V (or Cmd + V) to paste, then press Shift + Enter to execute!
Launch Blank Colab Notebook

🔄 Algorithmic Execution Flow

1
Zero Grad

Clears gradient buffers.

→
2
Forward & Loss

Generates predictions and evaluates MSE against noisy targets.

→
3
Backward & Step

Computes analytical gradients and updates weights; converges to w ≈ 2.50, b ≈ 1.00.

📥 Input / Initial State
100 synthetic data points following y = 2.5x + 1.0
📤 Expected Output
Learned equation: y = 2.50 * x + 1.00 (Target: 2.50 * x + 1.00)
💡 What Does This Demonstration Show?

The canonical 5-step PyTorch loop converged on the exact true physical relationship in 50 iterations.

Course Learning Resource • Slide DeckLesson 512 Slides

Lesson 5: Classical Deep Learning with PyTorch

CNN, MNIST dataset, backpropagation, loss functions, and optimization.

Author: Dr. Syed Muzamil Basha• FDP Approved Curriculum
Open in New Tab

📝 Learning Check

Test your understanding with these questions. Remember, this is for learning, not examination!

Question 1: Which device string is used in PyTorch to move a model and tensors to an NVIDIA GPU?

Question 2: What is PyTorch primarily used for?

Question 3: What is a PyTorch tensor?

Question 4: What is torch.nn.Module in PyTorch?

Question 5: What is the difference between PyTorch and TensorFlow?

Previous: Introduction to TensorFlow and KerasNext: Building a Deep Neural Network: Module 3 Capstone