First time studying this topic?

Launch the interactive walkthrough to discover each component, including the 3D Q-Sphere Lab & Python in Colab.

Topic 4.2

The RL Agent: Internal Architecture

Examine the components of an intelligent agent: the policy, the value function, and the internal environment model.

Learning Objectives

✓Dissect the three key components of an agent: Policy, Value Function, and Model
✓Differentiate Model-Free vs Model-Based RL
✓Differentiate Value-Based vs Policy-Based methods

📖Let's Begin with a Story

Imagine three chess players with radically different thinking styles: 1. The Policy Player: Plays purely on instinct. When they see a pinned knight, their policy immediately dictates: "Attack the pin!" No mental simulation needed. 2. The Value Player: Looks at the board and mentally scores every move: "If I move my bishop here, my board score is +2.4. If I move my rook, my score is +1.1. I will pick the +2.4 move." 3. The Model-Based Player: Sits quietly and plans ahead: "If I move pawn to d4, black will respond with knight to f6, then I can push c4..." They maintain a mental simulator of the rules of chess. All three are valid RL agent architectures!

Why Should You Learn The RL Agent: Internal Architecture?

🎯 Practical & Engineering Stakes

Navigating the RL algorithm zoo

Understanding whether an agent is model-free or model-based, policy-based or value-based, allows you to pick the right algorithm (PPO, DQN, SAC, or MuZero) for your problem.

What Is It?

Simple Explanation

An RL Agent is the "brain" of the system. Inside the brain, it can have: 1. A Policy: "When I am in this situation, do this action." 2. A Value Function: "Being in this situation is worth 50 points." 3. A Model: "If I push this button, I predict the door will open."

Technical Explanation

An RL agent is characterized by three core sub-components: - **Policy

\pi(a|s)

: The mapping from states to action probabilities. - Value Function

V(s) = \mathbb{E}[G_t|S_t=s]

**: Expected cumulative discounted return from state

s

. - **Model

\mathcal{P}_{ss'}^a, \mathcal{R}_s^a

**: Internal representation approximating environment transitions and expected immediate rewards.

📚 Key Terms

Model-Free RL

Simple: The agent learns directly from experience without trying to build a simulator of the world.

Technical: Algorithms (like Q-Learning and Policy Gradients) that optimize policies directly without learning transition dynamics

P(s'|s,a)

Model-Based RL

Simple: The agent learns how the world works so it can plan moves in its head before acting.

Technical: Algorithms that approximate the environment transition model

\hat{P}(s'|s,a)

to perform tree search or trajectory optimization.

Value Function $V(s)$

Simple: How good it is to be in a given situation.

Technical: State-value function measuring expected discounted return under policy

\pi

📐 Mathematical Foundation

LaTeX KaTeX

V^\pi(s) = \mathbb{E}_\pi \left[ \sum_{k=0}^\infty \gamma^k R_{t+k+1} \;\middle|\; S_t = s \right]

What does it mean?

State-Value Function: the expected return if the agent starts in state s and follows policy pi forever after.

Understanding the Symbols

Symbol / Term	Meaning	Beginner Interpretation
$V^\pi(s)$	Value of state s	Desirability of being in state s
$\mathbb{E}_\pi$	Expected value under policy pi	Average outcome over stochastic runs

📊 Numerical Example

State near the goal has high expected value V=9.2; state trapped in a maze corner has low value V=0.1.

V(\text{near\_goal}) = 9.2 > V(\text{trapped}) = 0.1

Result: Agent prefers states with higher V(s)

⚙️ How Does It Work?

State Evaluation

Evaluate current state s using value function V(s) or Q(s,a).

Action Generation

Sample action from stochastic policy or pick argmax of Q-values.

Experience Ingestion

Collect transition tuple (s, a, r, s') into memory buffer.

🌍 Where Is It Used?

MuZero (DeepMind)

Problem: Mastering games like Chess, Go, and Atari without ever being given the rules of the game.

Solution: MuZero learns its own latent internal model of the environment dynamics.

🎯 Try It Yourself

A self-driving car agent learns to drive purely through trial and error in a simulator without any internal physics engine predicting how friction works. What type of RL is this?

Interactive activity component

🐍 Python Hands-On: Evaluating State Values in Python

Calculate and compare state values for a simple 3-state chain.

Open Colab

1Install Required Packages

For Google Colab or Jupyter notebooks, run this shell command to install the required dependencies:

Terminal / Notebook Install Commandbash

!pip install numpy

Packages:

numpy

2Import Libraries

Import Statementspython

1 import numpy as np

import numpy as np → NumPy math

3Demonstration Code & Logic

Python Execution Codepython

1 # 3 States: [Start (0), Step (1), Goal (2)]
2 # Reward +10 only upon reaching Goal
3 gamma = 0.9 # Discount factor
4  
5 # Value of Goal (Terminal state) is 0
6 V_goal = 0.0
7 # Value of Step: immediate reward 10 to reach Goal
8 V_step = 10.0 + gamma * V_goal
9 # Value of Start: immediate reward 0 to reach Step
10 V_start = 0.0 + gamma * V_step
11  
12 print(f"Value of Start State V(0): {V_start:.1f}")
13 print(f"Value of Step State  V(1): {V_step:.1f}")

Step-by-Step Code Breakdown:

# 3 States: [Start (0), Step (1), Goal (2)]

Environment setup

# Reward +10 only upon reaching Goal

Reward vector

gamma = 0.9 # Discount factor

Discount parameter

# Value of Goal (Terminal state) is 0

Terminal

V_goal = 0.0

V[2] = 0

# Value of Step: immediate reward 10 to reach Goal

Step state

V_step = 10.0 + gamma * V_goal

V[1] = 10 + 0.9 * 0 = 10.0

# Value of Start: immediate reward 0 to reach Step

Start state

V_start = 0.0 + gamma * V_step

V[0] = 0 + 0.9 * 10 = 9.0

print(f"Value of Start State V(0): {V_start:.1f}")

Display V(0)

print(f"Value of Step State V(1): {V_step:.1f}")

Display V(1)

📋 Complete Script (All-in-One)

Click "Copy for Colab", open Google Colab, paste into a single cell, and run.

Google Colab Notebook Scriptpython

1 # =========================================================
2 # Evaluating State Values in Python
3 # Prepared for Google Colab (Run with Shift + Enter)
4 # =========================================================
5  
6 # Step 1: Install required packages
7 !pip install numpy
8  
9 # Step 2: Import libraries
10 import numpy as np
11  
12 # Step 3: Run Demonstration
13 # 3 States: [Start (0), Step (1), Goal (2)]
14 # Reward +10 only upon reaching Goal
15 gamma = 0.9 # Discount factor
16  
17 # Value of Goal (Terminal state) is 0
18 V_goal = 0.0
19 # Value of Step: immediate reward 10 to reach Goal
20 V_step = 10.0 + gamma * V_goal
21 # Value of Start: immediate reward 0 to reach Step
22 V_start = 0.0 + gamma * V_step
23  
24 print(f"Value of Start State V(0): {V_start:.1f}")
25 print(f"Value of Step State  V(1): {V_step:.1f}")

⚡ How to Run in Google Colab (3 Quick Steps)

Click "Copy for Google Colab" above.
Click Open New Google Colab Notebook (opens in a new tab).
Click on the first empty code cell, press Ctrl + V (or Cmd + V) to paste, then press Shift + Enter to execute!

Launch Blank Colab Notebook

🔄 Algorithmic Execution Flow

Backward Value Bellman Backup

Calculates future returns backwards: V(1)=10.0, V(0)=9.0.

→

Gradient of Desire

Agent naturally seeks states with higher values: 0 -> 1 -> Goal.

📥 Input / Initial State

3-state chain with reward at goal

📤 Expected Output

Value of Start State V(0): 9.0
Value of Step State  V(1): 10.0

💡 What Does This Demonstration Show?

The discounted value function creates an uphill gradient toward the goal that the agent can climb.

Course Learning Resource • Slide DeckLesson 612 Slides

Lesson 6: Reinforcement Learning Foundations

Q-learning, Q-table updates, DQN architectures, and exploration-exploitation trade-offs.

Author: Dr. Syed Muzamil Basha• FDP Approved Curriculum

Open in New Tab

📝 Learning Check

Test your understanding with these questions. Remember, this is for learning, not examination!

Question 1: In reinforcement learning, what is a "Policy"?

Question 2: What are the main components of an RL agent?

Question 3: What is the difference between model-based and model-free RL?

Question 4: What is an on-policy vs off-policy RL algorithm?

Question 5: What is the credit assignment problem in RL?

Previous: What is Reinforcement Learning? Sequential Decision Making Next: The Environment: Dynamics, Observability & Gym

1	# =========================================================
2	# Evaluating State Values in Python
3	# Prepared for Google Colab (Run with Shift + Enter)
4	# =========================================================
5
6	# Step 1: Install required packages
7	!pip install numpy
8
9	# Step 2: Import libraries
10	import numpy as np
11
12	# Step 3: Run Demonstration
13	# 3 States: [Start (0), Step (1), Goal (2)]
14	# Reward +10 only upon reaching Goal
15	gamma = 0.9 # Discount factor
16
17	# Value of Goal (Terminal state) is 0
18	V_goal = 0.0
19	# Value of Step: immediate reward 10 to reach Goal
20	V_step = 10.0 + gamma * V_goal
21	# Value of Start: immediate reward 0 to reach Step
22	V_start = 0.0 + gamma * V_step
23
24	print(f"Value of Start State V(0): {V_start:.1f}")
25	print(f"Value of Step State V(1): {V_step:.1f}")