First time studying this topic?

Launch the interactive walkthrough to discover each component, including the 3D Q-Sphere Lab & Python in Colab.

Topic 4.1

What is Reinforcement Learning? Sequential Decision Making

Understand the science of learning to act: how agents navigate dynamic worlds to maximize long-term cumulative reward.

Learning Objectives

✓Contrast Reinforcement Learning with Supervised and Unsupervised Learning
✓Understand sequential decision-making where actions influence future observations
✓Formulate problems as Markov Decision Processes (MDPs)
✓Grasp the concept of the Credit Assignment problem

📖Let's Begin with a Story

In March 2016 in Seoul, South Korea, 200 million people watched as Lee Sedol, the legendary 18-time world champion of Go, sat down across from a computer program named AlphaGo. Go is an ancient game with more legal board configurations than there are atoms in the observable universe (

10^{170}

). Brute-force search was impossible. Expert consensus believed computers were decades away from defeating a grandmaster. AlphaGo did not have a human grandmaster whispering moves. It learned by playing millions of games against copies of itself using Reinforcement Learning. On Move 37 of Game 2, AlphaGo placed a stone on the shoulder of the board—a move no human professional would ever play. Commentators gasped, calling it a mistake. But 50 moves later, that exact stone decided the game. AlphaGo had discovered strategies unknown to 3,000 years of human Go history through pure trial, error, and reinforcement.

Why Should You Learn What is Reinforcement Learning? Sequential Decision Making?

🎯 Practical & Engineering Stakes

The pinnacle of autonomous agency

While supervised learning passively answers questions, reinforcement learning interacts with reality. From robotics to drug discovery and autonomous trading, RL builds agents that take action in an uncertain world.

What Is It?

Simple Explanation

Reinforcement Learning is like learning to ride a bicycle. Nobody can explain the physics to you in words. You get on the bike, wobble, lean too far right, and fall (ouch!). You try again, lean slightly left, pedal faster, and suddenly you stay upright (success!). Through trial and error, your brain figures out the sequence of muscle actions needed to balance.

Technical Explanation

Reinforcement Learning formalizes optimal control within a Markov Decision Process (MDP)

\mathcal{M} = \langle \mathcal{S}, \mathcal{A}, \mathcal{P}, \mathcal{R}, \gamma \rangle

. An agent samples transitions

(s_t, a_t, r_t, s_{t+1})

, seeking a policy

\pi(a|s)

that maximizes expected cumulative discounted return:

\mathbb{E}_\pi [\sum_{t=0}^\infty \gamma^t r_t]

📚 Key Terms

Markov Decision Process (MDP)

Simple: The mathematical sandbox that defines the world the agent lives in.

Technical: A 5-tuple

\langle \mathcal{S}, \mathcal{A}, \mathcal{P}, \mathcal{R}, \gamma \rangle

satisfying the Markov property

P(s_{t+1}|s_t, a_t, \dots, s_0) = P(s_{t+1}|s_t, a_t)

Credit Assignment Problem

Simple: Figuring out which specific past action was responsible for winning or losing later.

Technical: The temporal challenge of attributing delayed reward outcomes to intermediate causal actions across long horizons.

Return ( $G_t$ )

Simple: The grand total of points collected from now until the end of the game.

Technical: The discounted sum of future rewards:

G_t = \sum_{k=0}^\infty \gamma^k R_{t+k+1}

📐 Mathematical Foundation

LaTeX KaTeX

G_t = R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \dots = \sum_{k=0}^{\infty} \gamma^k R_{t+k+1}

What does it mean?

Discounted Return: the total cumulative reward an agent optimizes, where immediate rewards are valued more than distant future rewards.

Understanding the Symbols

Symbol / Term	Meaning	Beginner Interpretation
$G_t$	Total discounted return	Target payoff
$\gamma$	Discount factor (0 to 1)	Temporal patience dial
$R_t$	Reward at time t	Immediate feedback score

📊 Numerical Example

Receiving rewards [1, 2, 10] at steps 1, 2, 3 with discount factor gamma = 0.9.

G_0 = 1 + (0.9 \times 2) + (0.9^2 \times 10) = 1 + 1.8 + (0.81 \times 10) = 1 + 1.8 + 8.1 = 10.9

Result: Discounted return = 10.9

⚙️ How Does It Work?

Observation

Agent inspects current environment state s_t.

Action Selection

Agent executes action a_t selected by policy pi(a|s).

World Reaction

Environment computes new state s_{t+1} and returns reward r_{t+1}.

Policy Optimization

Agent updates its internal value estimates to favor successful paths.

🌍 Where Is It Used?

ChatGPT RLHF (Reinforcement Learning from Human Feedback)

Problem: Raw language models generating harmful, hallucinated, or unhelpful answers.

Solution: RL reward models train policy to maximize human satisfaction and conversational alignment.

Industrial Robotics

Problem: Assembly robot inserting delicate cables into circuit boards without snapping wires.

Solution: RL policy trained with tactile force sensors learns micro-adjustments via trial-and-error.

🎯 Try It Yourself

How does Reinforcement Learning fundamentally differ from Supervised Learning?

Interactive activity component

🐍 Python Hands-On: Simulating a Basic 1D GridWorld in Python

Build a minimal grid environment where an agent steps forward and backward to reach a goal.

Open Colab

1Install Required Packages

For Google Colab or Jupyter notebooks, run this shell command to install the required dependencies:

Terminal / Notebook Install Commandbash

!pip install numpy

Packages:

numpy

2Import Libraries

Import Statementspython

1 import numpy as np

import numpy as np → NumPy for random choices

3Demonstration Code & Logic

Python Execution Codepython

1 # 1D GridWorld: States 0, 1, 2, 3, 4 (Goal is state 4)
2 state = 0 # Start at position 0
3 goal = 4  # Win state
4 steps = 0
5  
6 # Run random walk agent until goal is reached:
7 while state < goal:
8     # Actions: 0 = Left (-1), 1 = Right (+1)
9     action = np.random.choice([0, 1])
10     if action == 1:
11         state = min(goal, state + 1)
12     else:
13         state = max(0, state - 1)
14     steps += 1
15  
16 print(f"Goal reached at state {state} in {steps} random exploration steps!")

Step-by-Step Code Breakdown:

# 1D GridWorld: States 0, 1, 2, 3, 4 (Goal is state 4)

Setup

state = 0 # Start at position 0

Initial state

goal = 4 # Win state

Target

steps = 0

Step counter

# Run random walk agent until goal is reached:

Agent loop

while state < goal:

Loop until win

# Actions: 0 = Left (-1), 1 = Right (+1)

Action space

action = np.random.choice([0, 1])

Sample action

if action == 1:

Right step

state = min(goal, state + 1)

Move right

else:

Left step

state = max(0, state - 1)

Move left

steps += 1

Increment steps

print(f"Goal reached at state {state} in {steps} random exploration steps!")

Display outcome

📋 Complete Script (All-in-One)

Click "Copy for Colab", open Google Colab, paste into a single cell, and run.

Google Colab Notebook Scriptpython

1 # =========================================================
2 # Simulating a Basic 1D GridWorld in Python
3 # Prepared for Google Colab (Run with Shift + Enter)
4 # =========================================================
5  
6 # Step 1: Install required packages
7 !pip install numpy
8  
9 # Step 2: Import libraries
10 import numpy as np
11  
12 # Step 3: Run Demonstration
13 # 1D GridWorld: States 0, 1, 2, 3, 4 (Goal is state 4)
14 state = 0 # Start at position 0
15 goal = 4  # Win state
16 steps = 0
17  
18 # Run random walk agent until goal is reached:
19 while state < goal:
20     # Actions: 0 = Left (-1), 1 = Right (+1)
21     action = np.random.choice([0, 1])
22     if action == 1:
23         state = min(goal, state + 1)
24     else:
25         state = max(0, state - 1)
26     steps += 1
27  
28 print(f"Goal reached at state {state} in {steps} random exploration steps!")

⚡ How to Run in Google Colab (3 Quick Steps)

Click "Copy for Google Colab" above.
Click Open New Google Colab Notebook (opens in a new tab).
Click on the first empty code cell, press Ctrl + V (or Cmd + V) to paste, then press Shift + Enter to execute!

Launch Blank Colab Notebook

🔄 Algorithmic Execution Flow

State Transition

Agent takes discrete steps across 1D line.

→

Boundary Constraints

Agent cannot walk left of 0, and terminates upon reaching 4.

📥 Input / Initial State

1D grid of 5 states

📤 Expected Output

Goal reached at state 4 in ~14 random exploration steps!

💡 What Does This Demonstration Show?

A random exploratory agent eventually reaches the goal, demonstrating the baseline exploration dynamics of RL environments.

Course Learning Resource • Slide DeckLesson 612 Slides

Lesson 6: Reinforcement Learning Foundations

Q-learning, Q-table updates, DQN architectures, and exploration-exploitation trade-offs.

Author: Dr. Syed Muzamil Basha• FDP Approved Curriculum

Open in New Tab

📝 Learning Check

Test your understanding with these questions. Remember, this is for learning, not examination!

Question 1: What is the "Credit Assignment Problem" in Reinforcement Learning?

Question 2: What is the key difference between reinforcement learning and supervised learning?

Question 3: Which of these is a famous RL success story?

Question 4: What is the Markov Decision Process (MDP) in RL?

Question 5: What does an RL agent aim to maximize?

Previous: Building a Deep Neural Network: Module 3 Capstone Next: The RL Agent: Internal Architecture

1	# =========================================================
2	# Simulating a Basic 1D GridWorld in Python
3	# Prepared for Google Colab (Run with Shift + Enter)
4	# =========================================================
5
6	# Step 1: Install required packages
7	!pip install numpy
8
9	# Step 2: Import libraries
10	import numpy as np
11
12	# Step 3: Run Demonstration
13	# 1D GridWorld: States 0, 1, 2, 3, 4 (Goal is state 4)
14	state = 0 # Start at position 0
15	goal = 4 # Win state
16	steps = 0
17
18	# Run random walk agent until goal is reached:
19	while state < goal:
20	# Actions: 0 = Left (-1), 1 = Right (+1)
21	action = np.random.choice([0, 1])
22	if action == 1:
23	state = min(goal, state + 1)
24	else:
25	state = max(0, state - 1)
26	steps += 1
27
28	print(f"Goal reached at state {state} in {steps} random exploration steps!")