First time studying this topic?

Launch the interactive walkthrough to discover each component, including the 3D Q-Sphere Lab & Python in Colab.

Topic 4.12

Deep Q-Networks (DQN): Module 4 Capstone

Synthesize Deep Learning and Reinforcement Learning: Deep Q-Networks (DQN), Experience Replay, and Target Networks.

Learning Objectives

✓Understand how Deep Q-Networks replace tabular Q-tables with neural network function approximators
✓Examine the stability breakthroughs: Experience Replay Buffer and Target Networks
✓Analyze DeepMind’s historic 2015 breakthrough mastering Atari 2600 from raw pixels

📖Let's Begin with a Story

In 2013, a London-based startup named DeepMind published a paper that sent shockwaves through the artificial intelligence community: "Playing Atari with Deep Reinforcement Learning." They created a single algorithm that learned to play seven different Atari 2600 video games—including Breakout, Pong, and Space Invaders. The AI was not given the rules of the games, and it was not given manual features. It was fed only the raw screen pixels and the score counter! In Breakout, the network discovered an ingenious strategy after 400 episodes: tunneling through the side wall of bricks to send the ball ricocheting repeatedly across the ceiling! Google acquired DeepMind months later for over $500 million. Deep Q-Networks had unified deep representation learning with reinforcement decision-making.

Why Should You Learn Deep Q-Networks (DQN): Module 4 Capstone?

🎯 Practical & Engineering Stakes

The birth of Deep Reinforcement Learning

Combining deep neural networks with reinforcement learning broke the curse of dimensionality, allowing AI to perceive high-dimensional sensory worlds and act intelligently.

What Is It?

Simple Explanation

A Q-table cannot hold 10 billion rows for video game screens. So DeepMind replaced the table with a Deep Neural Network: - Input: The 4 screenshot images of the game. - Output: A Q-value for each button on the joystick (Left, Right, Fire). Two crucial tricks make it work: 1. Experience Replay: Storing memories in a photo album and reviewing them randomly so the AI doesn't forget past lessons. 2. Target Network: Freezing a separate copy of the network to act as a stable teacher.

Technical Explanation

Deep Q-Networks approximate action-values

Q(s, a; \boldsymbol{\theta}) \approx Q^*(s, a)

parameterized by deep convolutional weights

\boldsymbol{\theta}

. To break temporal auto-correlations and prevent policy divergence, DQN introduces two algorithmic stabilizing mechanisms: 1. **Experience Replay Buffer

\mathcal{D}

**: Uniform random mini-batch sampling from historical transitions

(s, a, r, s')

. 2. **Fixed Target Network

\boldsymbol{\theta}^-

**: Decoupling target generation

y_t = r + \gamma \max_{a'} Q(s', a'; \boldsymbol{\theta}^-)

with periodic parameter synchronization

\boldsymbol{\theta}^- \leftarrow \boldsymbol{\theta}

📚 Key Terms

Experience Replay Buffer

Simple: A memory bank where past experiences are saved and randomly sampled to break bad habits.

Technical: A rolling FIFO buffer storing transitions

\mathcal{D} = \{(s_t, a_t, r_t, s_{t+1})\}

sampled uniformly to ensure i.i.d. training batches.

Target Network ( $\theta^-$ )

Simple: A frozen copy of the network used to compute targets so the goalposts don’t keep moving.

Technical: A second set of network weights held static for

C

steps to prevent runaway feedback loops between current estimates and targets.

📐 Mathematical Foundation

LaTeX KaTeX

\mathcal{L}(\boldsymbol{\theta}) = \mathbb{E}_{(s,a,r,s') \sim \mathcal{D}} \left[ \left( r + \gamma \max_{a'} Q(s', a'; \boldsymbol{\theta}^-) - Q(s, a; \boldsymbol{\theta}) \right)^2 \right]

What does it mean?

DQN Loss Function: Mean squared error between the stable target network’s Bellman prediction and the current active network’s Q-value.

Understanding the Symbols

Symbol / Term	Meaning	Beginner Interpretation
$\boldsymbol{\theta}$	Active network weights	Being updated via gradient descent
$\boldsymbol{\theta}^-$	Target network weights	Frozen periodically
$\mathcal{D}$	Replay buffer	Random mini-batch source

📊 Numerical Example

Target network predicts TD target y = 10.0; active network predicts Q = 7.0.

\text{Loss} = (10.0 - 7.0)^2 = 3.0^2 = 9.0

Result: Gradient descent updates theta to reduce loss from 9.0 toward 0

⚙️ How Does It Work?

Interaction & Storage

Agent takes epsilon-greedy step and stores transition (s, a, r, s', done) in replay buffer.

Random Sampling

Sample mini-batch of 32 transitions uniformly from replay memory.

Target Computation

Compute Bellman targets using frozen target network theta-.

Gradient Descent

Update active weights theta via Adam optimizer on MSE loss.

Periodic Synchronization

Every C steps (e.g. 1000 steps), copy active weights to target network.

🌍 Where Is It Used?

Atari 2600 Benchmark Superhuman Play

Problem: Mastering dozens of video games directly from screen pixels.

Solution: Nature 2015 DQN achieved superhuman performance on over half of all 49 tested Atari games.

🎯 Try It Yourself

Why does DQN use an Experience Replay buffer instead of training immediately on consecutive sequential steps?

Interactive activity component

🐍 Python Hands-On: Deep Q-Network (DQN) with Experience Replay on CartPole-v1

Implement a full Deep Q-Network (DQN) in PyTorch to solve Gymnasium CartPole-v1 using an experience replay buffer (deque) and epsilon-greedy exploration.

Open Colab

1Install Required Packages

For Google Colab or Jupyter notebooks, run this shell command to install the required dependencies:

Terminal / Notebook Install Commandbash

!pip install torch gymnasium numpy

Packages:

torchgymnasiumnumpy

2Import Libraries

Import Statementspython

1 import torch
2 import torch.nn as nn
3 import torch.optim as optim
4 import gymnasium as gym
5 from collections import deque
6 import random

import torch → PyTorch deep learning framework

import torch.nn as nn → Neural network layers

import torch.optim as optim → Adam optimizer

import gymnasium as gym → Gymnasium CartPole simulation

from collections import deque → Double-ended queue for experience replay buffer

import random → Random mini-batch sampling

3Demonstration Code & Logic

Python Execution Codepython

1 # Setup Gymnasium CartPole Environment
2 env = gym.make("CartPole-v1")
3 state_size = env.observation_space.shape[0]  # 4 features: [pos, vel, angle, angular_vel]
4 action_size = env.action_space.n               # 2 actions: [push left, push right]
5  
6 # Define Deep Q-Network Architecture (MLP)
7 class DQN(nn.Module):
8     def __init__(self, state_size, action_size):
9         super(DQN, self).__init__()
10         self.fc1 = nn.Linear(state_size, 24)
11         self.fc2 = nn.Linear(24, 24)
12         self.fc3 = nn.Linear(24, action_size)
13     def forward(self, x):
14         x = torch.relu(self.fc1(x))
15         x = torch.relu(self.fc2(x))
16         return self.fc3(x)
17  
18 model = DQN(state_size, action_size)
19 optimizer = optim.Adam(model.parameters(), lr=0.001)
20 loss_function = nn.MSELoss()
21  
22 # Replay buffer and hyperparameters
23 memory = deque(maxlen=2000)  # Experience replay buffer
24 gamma = 0.95                 # Discount factor
25 epsilon = 1.0                # Initial exploration
26 epsilon_min = 0.01
27 epsilon_decay = 0.995
28 batch_size = 64
29  
30 def replay():
31     if len(memory) < batch_size:
32         return
33     minibatch = random.sample(memory, batch_size)
34     for state, action, reward, next_state, done in minibatch:
35         target = reward
36         if not done:
37             target += gamma * torch.max(model(torch.FloatTensor(next_state))).item()
38         output = model(torch.FloatTensor(state))[action]
39         loss = loss_function(output.view(-1), torch.tensor([target], dtype=torch.float32))
40         optimizer.zero_grad()
41         loss.backward()
42         optimizer.step()
43  
44 # Train for sample demonstration episodes
45 for episode in range(10):
46     state = env.reset()[0]
47     done = False
48     total_reward = 0
49     while not done:
50         if np.random.rand() < epsilon:
51             action = random.randrange(action_size)
52         else:
53             action = torch.argmax(model(torch.FloatTensor(state))).item()
54         next_state, reward, done, _, _ = env.step(action)
55         memory.append((state, action, reward, next_state, done))
56         state = next_state
57         total_reward += reward
58     replay()
59     epsilon = max(epsilon * epsilon_decay, epsilon_min)
60     print(f"Episode {episode+1}, Total Reward: {total_reward:.0f}")

Step-by-Step Code Breakdown:

# Setup Gymnasium CartPole Environment

Environment

env = gym.make("CartPole-v1")

CartPole physics simulation

state_size = env.observation_space.shape[0] # 4 features: [pos, vel, angle, angular_vel]

Input dimensions

action_size = env.action_space.n # 2 actions: [push left, push right]

Output action count

# Define Deep Q-Network Architecture (MLP)

Network

class DQN(nn.Module):

Subclass nn.Module

def __init__(self, state_size, action_size):

Constructor

super(DQN, self).__init__()

Initialize base

self.fc1 = nn.Linear(state_size, 24)

Input layer (4 -> 24)

self.fc2 = nn.Linear(24, 24)

Hidden layer (24 -> 24)

self.fc3 = nn.Linear(24, action_size)

Output Q-values (24 -> 2)

def forward(self, x):

Forward computation

x = torch.relu(self.fc1(x))

Layer 1 + ReLU

x = torch.relu(self.fc2(x))

Layer 2 + ReLU

return self.fc3(x)

Linear output Q-values

model = DQN(state_size, action_size)

Instantiate DQN model

optimizer = optim.Adam(model.parameters(), lr=0.001)

Adam optimizer

loss_function = nn.MSELoss()

Mean squared error loss

# Replay buffer and hyperparameters

Buffer setup

memory = deque(maxlen=2000) # Experience replay buffer

Circular memory

gamma = 0.95 # Discount factor

Future reward discount

epsilon = 1.0 # Initial exploration

Exploration rate

epsilon_min = 0.01

Exploration floor

epsilon_decay = 0.995

Decay per episode

batch_size = 64

Mini-batch training size

def replay():

Mini-batch replay training function

if len(memory) < batch_size:

Wait for buffer warm-up

return

Return early

minibatch = random.sample(memory, batch_size)

Uniform random sampling

for state, action, reward, next_state, done in minibatch:

Iterate transitions

target = reward

Base target

if not done:

Bootstrap future Q

target += gamma * torch.max(model(torch.FloatTensor(next_state))).item()

Bellman target

output = model(torch.FloatTensor(state))[action]

Predicted Q(s, a)

loss = loss_function(output.view(-1), torch.tensor([target], dtype=torch.float32))

MSE loss

optimizer.zero_grad()

Zero gradients

loss.backward()

Backpropagation

optimizer.step()

Weight update

# Train for sample demonstration episodes

Training loop

for episode in range(10):

Episode loop

state = env.reset()[0]

Reset episode

done = False

Done flag

total_reward = 0

Reward tally

while not done:

Step loop

if np.random.rand() < epsilon:

Explore

action = random.randrange(action_size)

Random push

else:

Exploit

action = torch.argmax(model(torch.FloatTensor(state))).item()

Argmax action

next_state, reward, done, _, _ = env.step(action)

Step

memory.append((state, action, reward, next_state, done))

Store memory

state = next_state

Next state

total_reward += reward

Accumulate reward

replay()

Train network on replay buffer

epsilon = max(epsilon * epsilon_decay, epsilon_min)

Decay epsilon

print(f"Episode {episode+1}, Total Reward: {total_reward:.0f}")

Log episode

📋 Complete Script (All-in-One)

Click "Copy for Colab", open Google Colab, paste into a single cell, and run.

Google Colab Notebook Scriptpython

1 # =========================================================
2 # Deep Q-Network (DQN) with Experience Replay on CartPole-v1
3 # Prepared for Google Colab (Run with Shift + Enter)
4 # =========================================================
5  
6 # Step 1: Install required packages
7 !pip install torch gymnasium numpy
8  
9 # Step 2: Import libraries
10 import torch
11 import torch.nn as nn
12 import torch.optim as optim
13 import gymnasium as gym
14 from collections import deque
15 import random
16  
17 # Step 3: Run Demonstration
18 # Setup Gymnasium CartPole Environment
19 env = gym.make("CartPole-v1")
20 state_size = env.observation_space.shape[0]  # 4 features: [pos, vel, angle, angular_vel]
21 action_size = env.action_space.n               # 2 actions: [push left, push right]
22  
23 # Define Deep Q-Network Architecture (MLP)
24 class DQN(nn.Module):
25     def __init__(self, state_size, action_size):
26         super(DQN, self).__init__()
27         self.fc1 = nn.Linear(state_size, 24)
28         self.fc2 = nn.Linear(24, 24)
29         self.fc3 = nn.Linear(24, action_size)
30     def forward(self, x):
31         x = torch.relu(self.fc1(x))
32         x = torch.relu(self.fc2(x))
33         return self.fc3(x)
34  
35 model = DQN(state_size, action_size)
36 optimizer = optim.Adam(model.parameters(), lr=0.001)
37 loss_function = nn.MSELoss()
38  
39 # Replay buffer and hyperparameters
40 memory = deque(maxlen=2000)  # Experience replay buffer
41 gamma = 0.95                 # Discount factor
42 epsilon = 1.0                # Initial exploration
43 epsilon_min = 0.01
44 epsilon_decay = 0.995
45 batch_size = 64
46  
47 def replay():
48     if len(memory) < batch_size:
49         return
50     minibatch = random.sample(memory, batch_size)
51     for state, action, reward, next_state, done in minibatch:
52         target = reward
53         if not done:
54             target += gamma * torch.max(model(torch.FloatTensor(next_state))).item()
55         output = model(torch.FloatTensor(state))[action]
56         loss = loss_function(output.view(-1), torch.tensor([target], dtype=torch.float32))
57         optimizer.zero_grad()
58         loss.backward()
59         optimizer.step()
60  
61 # Train for sample demonstration episodes
62 for episode in range(10):
63     state = env.reset()[0]
64     done = False
65     total_reward = 0
66     while not done:
67         if np.random.rand() < epsilon:
68             action = random.randrange(action_size)
69         else:
70             action = torch.argmax(model(torch.FloatTensor(state))).item()
71         next_state, reward, done, _, _ = env.step(action)
72         memory.append((state, action, reward, next_state, done))
73         state = next_state
74         total_reward += reward
75     replay()
76     epsilon = max(epsilon * epsilon_decay, epsilon_min)
77     print(f"Episode {episode+1}, Total Reward: {total_reward:.0f}")

⚡ How to Run in Google Colab (3 Quick Steps)

Click "Copy for Google Colab" above.
Click Open New Google Colab Notebook (opens in a new tab).
Click on the first empty code cell, press Ctrl + V (or Cmd + V) to paste, then press Shift + Enter to execute!

Launch Blank Colab Notebook

🔄 Algorithmic Execution Flow

Neural Approximation

A PyTorch MLP replaces the discrete Q-table, mapping 4 continuous state features to 2 action Q-values.

→

Experience Replay

Transitions (s, a, r, s', done) are stored in a 2000-step deque and uniformly sampled in batches of 64 to break temporal correlation.

→

Bellman Target Minimization

The network weights are updated via Adam optimizer to minimize the MSE discrepancy between current predictions and bootstrapped Bellman targets.

📥 Input / Initial State

Continuous 4D CartPole state space [cart position, cart velocity, pole angle, pole angular velocity]

📤 Expected Output

Episode 1, Total Reward: 18
Episode 2, Total Reward: 24
Episode 3, Total Reward: 31
...

💡 What Does This Demonstration Show?

The deep neural network successfully learns to balance the CartPole upright by continuously updating its internal weights from experience replay memories.

Course Learning Resource • Slide DeckLesson 612 Slides

Lesson 6: Reinforcement Learning Foundations

Deep Q-Networks (DQN), replay buffer, neural network value function approximation, and CartPole balancing.

Author: Dr. Syed Muzamil Basha• FDP Approved Curriculum

Open in New Tab

📝 Learning Check

Test your understanding with these questions. Remember, this is for learning, not examination!

Question 1: What is the purpose of the frozen Target Network in Deep Q-Networks (DQN)?

Question 2: What problem does DQN solve that Q-tables cannot?

Question 3: What is experience replay in DQN?

Question 4: What is the target network in DQN?

Question 5: What are the key innovations in DQN (2015)?

Previous: The Q-Value Update: The Bellman Equation in Action Next: Why Learn Quantum Computing? The Post-Moore Era

1	import torch
2	import torch.nn as nn
3	import torch.optim as optim
4	import gymnasium as gym
5	from collections import deque
6	import random

1	# =========================================================
2	# Deep Q-Network (DQN) with Experience Replay on CartPole-v1
3	# Prepared for Google Colab (Run with Shift + Enter)
4	# =========================================================
5
6	# Step 1: Install required packages
7	!pip install torch gymnasium numpy
8
9	# Step 2: Import libraries
10	import torch
11	import torch.nn as nn
12	import torch.optim as optim
13	import gymnasium as gym
14	from collections import deque
15	import random
16
17	# Step 3: Run Demonstration
18	# Setup Gymnasium CartPole Environment
19	env = gym.make("CartPole-v1")
20	state_size = env.observation_space.shape[0] # 4 features: [pos, vel, angle, angular_vel]
21	action_size = env.action_space.n # 2 actions: [push left, push right]
22
23	# Define Deep Q-Network Architecture (MLP)
24	class DQN(nn.Module):
25	def __init__(self, state_size, action_size):
26	super(DQN, self).__init__()
27	self.fc1 = nn.Linear(state_size, 24)
28	self.fc2 = nn.Linear(24, 24)
29	self.fc3 = nn.Linear(24, action_size)
30	def forward(self, x):
31	x = torch.relu(self.fc1(x))
32	x = torch.relu(self.fc2(x))
33	return self.fc3(x)
34
35	model = DQN(state_size, action_size)
36	optimizer = optim.Adam(model.parameters(), lr=0.001)
37	loss_function = nn.MSELoss()
38
39	# Replay buffer and hyperparameters
40	memory = deque(maxlen=2000) # Experience replay buffer
41	gamma = 0.95 # Discount factor
42	epsilon = 1.0 # Initial exploration
43	epsilon_min = 0.01
44	epsilon_decay = 0.995
45	batch_size = 64
46
47	def replay():
48	if len(memory) < batch_size:
49	return
50	minibatch = random.sample(memory, batch_size)
51	for state, action, reward, next_state, done in minibatch:
52	target = reward
53	if not done:
54	target += gamma * torch.max(model(torch.FloatTensor(next_state))).item()
55	output = model(torch.FloatTensor(state))[action]
56	loss = loss_function(output.view(-1), torch.tensor([target], dtype=torch.float32))
57	optimizer.zero_grad()
58	loss.backward()
59	optimizer.step()
60
61	# Train for sample demonstration episodes
62	for episode in range(10):
63	state = env.reset()[0]
64	done = False
65	total_reward = 0
66	while not done:
67	if np.random.rand() < epsilon:
68	action = random.randrange(action_size)
69	else:
70	action = torch.argmax(model(torch.FloatTensor(state))).item()
71	next_state, reward, done, _, _ = env.step(action)
72	memory.append((state, action, reward, next_state, done))
73	state = next_state
74	total_reward += reward
75	replay()
76	epsilon = max(epsilon * epsilon_decay, epsilon_min)
77	print(f"Episode {episode+1}, Total Reward: {total_reward:.0f}")