First time studying this topic?

Launch the interactive walkthrough to discover each component, including the 3D Q-Sphere Lab & Python in Colab.

Topic 1.9

Introduction to Reinforcement Learning

Understand how intelligent agents learn through trial, error, states, actions, and environmental rewards.

Learning Objectives

✓Define the Reinforcement Learning loop (Agent, Environment, State, Action, Reward)
✓Understand the Exploration vs Exploitation dilemma
✓Recognize differences between passive prediction and sequential decision-making

📖Let's Begin with a Story

When a human toddler learns to walk, nobody hands them an operating manual or a database of 100,000 labeled walking videos. The child stands up, loses balance, and falls (a painful negative reward). They adjust their foot angle and shift weight (action in state). Gradually, muscles coordinate, balance stabilizes, and the child walks across the room to grab a toy (positive reward). This is Reinforcement Learning: an agent situated in a dynamic world learns optimal behaviors through real-time experiential consequences.

Why Should You Learn Introduction to Reinforcement Learning?

🎯 Practical & Engineering Stakes

Why Reinforcement Learning is the frontier of autonomous systems:

1. Active decision making: Unlike supervised learning which only predicts labels, RL takes actions that permanently alter future states of the world. 2. Superhuman achievement: RL powers systems like AlphaZero, robotic manipulation, automated algorithmic trading, and ChatGPT’s RLHF alignment.

What Is It?

Simple Explanation

Reinforcement Learning is like training a puppy. When the puppy sits on command, you give it a treat (positive reward). When it chews the sofa, you say "No!" (negative reward). Over time, the puppy figures out what behaviors earn the most treats.

Technical Explanation

Reinforcement Learning is formal decision-theoretic learning where an agent interacts with a Markov Decision Process

\mathcal{M} = \langle \mathcal{S}, \mathcal{A}, \mathcal{P}, \mathcal{R}, \gamma \rangle

to learn a policy

\pi^*(a|s) = \arg\max_\pi \mathbb{E}[\sum_{t=0}^\infty \gamma^t R_t]

📚 Key Terms

Agent & Environment

Simple: The Agent is the learner/decision-maker; the Environment is the world it lives in.

Technical: The agent selects actions

a_t \in \mathcal{A}

; the environment transitions states

s_{t+1} \sim P(s_{t+1}|s_t, a_t)

and emits scalar reward

r_t

Reward Signal

Simple: A number telling the agent how well or poorly it did on the latest step.

Technical: A scalar feedback metric

R_t \in \mathbb{R}

defining the formal optimization goal.

Policy ( $\pi$ )

Simple: The strategy or brain of the agent mapping situations to actions.

Technical: A probability distribution

\pi(a|s) = P(A_t = a | S_t = s)

defining action selection.

📐 Mathematical Foundation

LaTeX KaTeX

G_t = \sum_{k=0}^{\infty} \gamma^k R_{t+k+1}

What does it mean?

Discounted Return: the total cumulative reward an agent expects to receive from time t onwards, where future rewards are discounted by gamma.

Understanding the Symbols

Symbol / Term	Meaning	Beginner Interpretation
$G_t$	Discounted Return	Total long-term payoff
$\gamma$	Discount factor (0 to 1)	How much the agent cares about future vs immediate rewards
$R_{t+k+1}$	Reward at future step	Reward received k steps later

📊 Numerical Example

Agent gets reward 0 today, 10 tomorrow, with discount factor gamma = 0.9.

G_t = 0 + (0.9 \times 10) = 9.0

Result: Discounted return value is 9.0

⚙️ How Does It Work?

Observe State

Agent inspects current environment state s_t.

Execute Action

Agent chooses action a_t based on current policy.

Receive Feedback

Environment yields scalar reward r_t and transitions to next state s_{t+1}.

Policy Update

Agent updates action values to increase probability of high-reward actions.

🌍 Where Is It Used?

Autonomous Drone Flight

Problem: Navigating turbulent wind gusts in unknown terrain.

Solution: RL agent learns micro-second propeller thrust adjustments in simulation before physical deployment.

Datacenter Cooling (DeepMind / Google)

Problem: Optimizing industrial HVAC chilling to minimize electricity waste.

Solution: RL model reduced cooling energy consumption by 40% in Google datacenters.

🎯 Try It Yourself

You visit your favorite restaurant. Choosing your usual favorite meal is EXPLOITATION. What is EXPLORATION?

Interactive activity component

🐍 Python Hands-On: Simulating a Multi-Armed Bandit Agent

Implement a minimal epsilon-greedy reinforcement agent in pure Python.

Open Colab

1Install Required Packages

For Google Colab or Jupyter notebooks, run this shell command to install the required dependencies:

Terminal / Notebook Install Commandbash

!pip install numpy

Packages:

numpy

2Import Libraries

Import Statementspython

1 import numpy as np

import numpy as np → NumPy for random choices

3Demonstration Code & Logic

Python Execution Codepython

1 # Two slot machines with win probabilities [0.2, 0.8]
2 win_probs = [0.2, 0.8]
3 q_values = [0.0, 0.0] # Estimated quality of each action
4 counts = [0, 0] # Times each machine was pulled
5 epsilon = 0.2 # 20% exploration, 80% exploitation
6  
7 for step in range(50):
8     # Epsilon-greedy action selection:
9     if np.random.rand() < epsilon:
10         action = np.random.choice([0, 1]) # Random explore
11     else:
12         action = int(np.argmax(q_values)) # Exploit best known
13  
14     reward = 1.0 if np.random.rand() < win_probs[action] else 0.0
15     counts[action] += 1
16     q_values[action] += (reward - q_values[action]) / counts[action] # Update Q
17 print("Estimated Win Rates:", [round(q, 2) for q in q_values])
18 print("Times Each Pulled:", counts)

Step-by-Step Code Breakdown:

# Two slot machines with win probabilities [0.2, 0.8]

Environment definition

win_probs = [0.2, 0.8]

Machine 1 is poor; Machine 2 is generous

q_values = [0.0, 0.0] # Estimated quality of each action

Agent memory

counts = [0, 0] # Times each machine was pulled

Action counter

epsilon = 0.2 # 20% exploration, 80% exploitation

Exploration parameter

for step in range(50):

50 trials

# Epsilon-greedy action selection:

Decision rule

if np.random.rand() < epsilon:

Explore

action = np.random.choice([0, 1]) # Random explore

Try random

else:

Exploit

action = int(np.argmax(q_values)) # Exploit best known

Pick highest Q

reward = 1.0 if np.random.rand() < win_probs[action] else 0.0

Simulate payout

counts[action] += 1

Increment count

q_values[action] += (reward - q_values[action]) / counts[action] # Update Q

Incremental average

print("Estimated Win Rates:", [round(q, 2) for q in q_values])

Print learned values

print("Times Each Pulled:", counts)

Print action distribution

📋 Complete Script (All-in-One)

Click "Copy for Colab", open Google Colab, paste into a single cell, and run.

Google Colab Notebook Scriptpython

1 # =========================================================
2 # Simulating a Multi-Armed Bandit Agent
3 # Prepared for Google Colab (Run with Shift + Enter)
4 # =========================================================
5  
6 # Step 1: Install required packages
7 !pip install numpy
8  
9 # Step 2: Import libraries
10 import numpy as np
11  
12 # Step 3: Run Demonstration
13 # Two slot machines with win probabilities [0.2, 0.8]
14 win_probs = [0.2, 0.8]
15 q_values = [0.0, 0.0] # Estimated quality of each action
16 counts = [0, 0] # Times each machine was pulled
17 epsilon = 0.2 # 20% exploration, 80% exploitation
18  
19 for step in range(50):
20     # Epsilon-greedy action selection:
21     if np.random.rand() < epsilon:
22         action = np.random.choice([0, 1]) # Random explore
23     else:
24         action = int(np.argmax(q_values)) # Exploit best known
25  
26     reward = 1.0 if np.random.rand() < win_probs[action] else 0.0
27     counts[action] += 1
28     q_values[action] += (reward - q_values[action]) / counts[action] # Update Q
29 print("Estimated Win Rates:", [round(q, 2) for q in q_values])
30 print("Times Each Pulled:", counts)

⚡ How to Run in Google Colab (3 Quick Steps)

Click "Copy for Google Colab" above.
Click Open New Google Colab Notebook (opens in a new tab).
Click on the first empty code cell, press Ctrl + V (or Cmd + V) to paste, then press Shift + Enter to execute!

Launch Blank Colab Notebook

🔄 Algorithmic Execution Flow

Exploration

Agent randomly samples both slot machines.

→

Exploitation

Once machine 1 proves superior (~80% payout), agent directs majority of pulls to it.

📥 Input / Initial State

50 interaction episodes with 2 bandit arms

📤 Expected Output

Estimated Win Rates: [0.18, 0.79]
Times Each Pulled: [8, 42]

💡 What Does This Demonstration Show?

The agent quickly converged on Machine 2, pulling it 42 times versus only 8 for Machine 1, demonstrating autonomous value discovery.

Course Learning Resource • Slide DeckLesson 612 Slides

Lesson 6: Reinforcement Learning Foundations

Q-learning, DQN, exploration, rewards, and optimization intuition.

Author: Dr. Syed Muzamil Basha• FDP Approved Curriculum

Open in New Tab

📝 Learning Check

Test your understanding with these questions. Remember, this is for learning, not examination!

Question 1: What is the role of the discount factor gamma (γ) in reinforcement learning?

Question 2: DeepMind's RL system reduced Google datacenter cooling energy by 40%. At each decision step, what does the RL agent do?

Question 3: The discounted return formula G_t = Σ γ^k R_{t+k+1} is computed with γ=0.9, R_1=0, R_2=10. What is G_t?

Question 4: In an epsilon-greedy bandit agent with epsilon=0.2, what happens when np.random.rand() returns 0.15?

Question 5: What is the fundamental difference between supervised learning and reinforcement learning in terms of feedback?

Question 6: A drone RL agent is trained in simulation before physical deployment. After 100,000 simulated flights, it achieves stable hovering. What RL principle does the self-play simulation demonstrate?

Previous: Unsupervised Learning: Clustering & Dimensionality Reduction Next: Python for AI: Essential Libraries

1	# =========================================================
2	# Simulating a Multi-Armed Bandit Agent
3	# Prepared for Google Colab (Run with Shift + Enter)
4	# =========================================================
5
6	# Step 1: Install required packages
7	!pip install numpy
8
9	# Step 2: Import libraries
10	import numpy as np
11
12	# Step 3: Run Demonstration
13	# Two slot machines with win probabilities [0.2, 0.8]
14	win_probs = [0.2, 0.8]
15	q_values = [0.0, 0.0] # Estimated quality of each action
16	counts = [0, 0] # Times each machine was pulled
17	epsilon = 0.2 # 20% exploration, 80% exploitation
18
19	for step in range(50):
20	# Epsilon-greedy action selection:
21	if np.random.rand() < epsilon:
22	action = np.random.choice([0, 1]) # Random explore
23	else:
24	action = int(np.argmax(q_values)) # Exploit best known
25
26	reward = 1.0 if np.random.rand() < win_probs[action] else 0.0
27	counts[action] += 1
28	q_values[action] += (reward - q_values[action]) / counts[action] # Update Q
29	print("Estimated Win Rates:", [round(q, 2) for q in q_values])
30	print("Times Each Pulled:", counts)