First time studying this topic?

Launch the interactive walkthrough to discover each component, including the 3D Q-Sphere Lab & Python in Colab.

Topic 4.10

The Q-Table: Tabular Representation & Memory

Understand the Q-Table lookup structure: mapping discrete states (rows) and actions (columns) to estimated cumulative rewards.

Learning Objectives

✓Construct and initialize a 2D Tabular Q-Matrix
✓Understand memory complexity and the Curse of Dimensionality
✓Perform state-action value queries and greedy policy extraction

📖Let's Begin with a Story

Imagine a taxi driver navigating a small village with 10 intersections. At each intersection, the taxi can turn Left, Right, or go Straight. The driver carries a small notebook with a 10x3 grid: - 10 rows (one for each intersection) - 3 columns (Left, Right, Straight) In each box of the grid, the driver writes a score in pencil showing how quickly that turn leads to the train station. When the driver arrives at intersection 4, they look at Row 4: - Left: 2.1 - Right: 8.9 - Straight: 4.0 The driver immediately turns Right! This notebook is a Q-Table. In environments with a small number of discrete states, a Q-Table is the cleanest, most transparent form of reinforcement learning memory.

Why Should You Learn The Q-Table: Tabular Representation & Memory?

🎯 Practical & Engineering Stakes

The bedrock of tabular RL

Before neural networks were applied to RL, the Q-table was the standard method to prove convergence to optimal policies. Mastering Q-tables demystifies how values are stored and retrieved.

What Is It?

Simple Explanation

A Q-Table is a spreadsheet: - Rows: The places or situations the agent can be in (States). - Columns: The moves the agent can make (Actions). - Cells: How good that move is (Q-values). To find the best move, look up your current row and pick the column with the highest number!

Technical Explanation

A Q-Table represents the discrete mapping

Q: \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}

as a 2D tensor

\mathbf{Q} \in \mathbb{R}^{|\mathcal{S}| \times |\mathcal{A}|}

. The greedy policy is extracted analytically via

\pi(s) = \arg\max_{a} Q(s, a)

. The tabular approach suffers from the Curse of Dimensionality: memory scales as

O(|\mathcal{S}| \cdot |\mathcal{A}|)

, breaking down when state spaces become continuous.

📚 Key Terms

Q-Table

Simple: A grid storing the expected score for every action in every state.

Technical: A 2D array storing state-action values

Q(s,a)

indexed by discrete state indices and action indices.

Curse of Dimensionality

Simple: When states explode into billions of combinations, making the table too huge to fit in memory.

Technical: Exponential growth of state space cardinality

|\mathcal{S}| = K^d

as the dimensionality of continuous state variables

d

increases.

📐 Mathematical Foundation

LaTeX KaTeX

a^* = \arg\max_{a \in \mathcal{A}} Q(s, a)

What does it mean?

Greedy Policy Extraction: given a state s, look across the row in the Q-table and select the action a* with the maximum Q-value.

Understanding the Symbols

Symbol / Term	Meaning	Beginner Interpretation
$a^*$	Optimal action	Best move
$Q(s, a)$	Q-value in row s, column a	Expected reward

📊 Numerical Example

Row for State 2: Q(2, Left) = 4.2, Q(2, Right) = 9.8. Max is 9.8.

a^* = \arg\max([4.2, 9.8]) = \text{Right}

Result: Greedy policy picks Right

⚙️ How Does It Work?

Initialization

Create zero matrix of size (num_states, num_actions).

Lookup

Current state index s identifies row Q[s, :].

Selection

Apply epsilon-greedy selection across row entries.

🌍 Where Is It Used?

GridWorld Navigation / FrozenLake

Problem: Solving 4x4 slippery grid mazes.

Solution: A 16x4 Q-table learns optimal paths avoiding ice holes and reaching the frisbee.

🎯 Try It Yourself

An environment has 50 discrete states and 4 discrete actions. How many numbers are stored in its Q-table?

Interactive activity component

🐍 Python Hands-On: Q-Table Optimization on Gymnasium FrozenLake-v1

Train an agent to navigate the FrozenLake environment using a 16x4 Q-table, epsilon-greedy exploration, and the Bellman temporal difference update equation.

Open Colab

1Install Required Packages

For Google Colab or Jupyter notebooks, run this shell command to install the required dependencies:

Terminal / Notebook Install Commandbash

!pip install gymnasium numpy

Packages:

gymnasiumnumpy

2Import Libraries

Import Statementspython

1 import numpy as np
2 import gymnasium as gym

import numpy as np → NumPy for 2D Q-matrix math

import gymnasium as gym → Gymnasium reinforcement learning environment suite

3Demonstration Code & Logic

Python Execution Codepython

1 # Create FrozenLake environment (4x4 grid = 16 states, 4 actions)
2 env = gym.make("FrozenLake-v1", is_slippery=False)
3 state_size = env.observation_space.n
4 action_size = env.action_space.n
5  
6 # Initialize Q-table with zeros
7 Q_table = np.zeros((state_size, action_size))
8  
9 # Hyperparameters
10 alpha = 0.1        # Learning rate
11 gamma = 0.99       # Discount factor
12 epsilon = 1.0      # Initial exploration rate
13 epsilon_decay = 0.995
14 epsilon_min = 0.01
15 episodes = 2000    # Number of games
16  
17 # Training Loop
18 for episode in range(episodes):
19     state = env.reset()[0]
20     done = False
21  
22     while not done:
23         # Epsilon-greedy action selection
24         if np.random.rand() < epsilon:
25             action = np.random.choice(action_size)
26         else:
27             action = np.argmax(Q_table[state])
28  
29         # Take action and observe transition
30         next_state, reward, done, _, _ = env.step(action)
31  
32         # Update Q-table using the Bellman equation
33         Q_table[state, action] = Q_table[state, action] + alpha * (
34             reward + gamma * np.max(Q_table[next_state]) - Q_table[state, action]
35         )
36  
37         state = next_state
38  
39     # Decay epsilon to shift from exploration to exploitation
40     epsilon = max(epsilon * epsilon_decay, epsilon_min)
41  
42 print("Training complete! Non-zero learned Q-values:\n", np.round(Q_table[Q_table > 0][:5], 4))

Step-by-Step Code Breakdown:

# Create FrozenLake environment (4x4 grid = 16 states, 4 actions)

Environment creation

env = gym.make("FrozenLake-v1", is_slippery=False)

Deterministic 4x4 grid

state_size = env.observation_space.n

16 discrete states

action_size = env.action_space.n

4 discrete actions: Left, Down, Right, Up

# Initialize Q-table with zeros

Initialize Q-table

Q_table = np.zeros((state_size, action_size))

16x4 matrix

# Hyperparameters

Parameters

alpha = 0.1 # Learning rate

Alpha step size

gamma = 0.99 # Discount factor

Gamma future discount

epsilon = 1.0 # Initial exploration rate

Exploration probability

epsilon_decay = 0.995

Decay rate per episode

epsilon_min = 0.01

Minimum exploration floor

episodes = 2000 # Number of games

Training duration

# Training Loop

Training loop

for episode in range(episodes):

Loop over episodes

state = env.reset()[0]

Reset environment to start state

done = False

Episode flag

while not done:

Step through episode

# Epsilon-greedy action selection

Action choice

if np.random.rand() < epsilon:

Explore

action = np.random.choice(action_size)

Sample random action

else:

Exploit

action = np.argmax(Q_table[state])

Greedy action from Q-table

# Take action and observe transition

Environment step

next_state, reward, done, _, _ = env.step(action)

Step transition

# Update Q-table using the Bellman equation

Q-value update

Q_table[state, action] = Q_table[state, action] + alpha * (

Temporal difference update

reward + gamma * np.max(Q_table[next_state]) - Q_table[state, action]

TD error calculation

)

Close update

state = next_state

Transition to next state

# Decay epsilon to shift from exploration to exploitation

Epsilon decay

epsilon = max(epsilon * epsilon_decay, epsilon_min)

Reduce epsilon

print("Training complete! Non-zero learned Q-values:\n", np.round(Q_table[Q_table > 0][:5], 4))

Display learned values

📋 Complete Script (All-in-One)

Click "Copy for Colab", open Google Colab, paste into a single cell, and run.

Google Colab Notebook Scriptpython

1 # =========================================================
2 # Q-Table Optimization on Gymnasium FrozenLake-v1
3 # Prepared for Google Colab (Run with Shift + Enter)
4 # =========================================================
5  
6 # Step 1: Install required packages
7 !pip install gymnasium numpy
8  
9 # Step 2: Import libraries
10 import numpy as np
11 import gymnasium as gym
12  
13 # Step 3: Run Demonstration
14 # Create FrozenLake environment (4x4 grid = 16 states, 4 actions)
15 env = gym.make("FrozenLake-v1", is_slippery=False)
16 state_size = env.observation_space.n
17 action_size = env.action_space.n
18  
19 # Initialize Q-table with zeros
20 Q_table = np.zeros((state_size, action_size))
21  
22 # Hyperparameters
23 alpha = 0.1        # Learning rate
24 gamma = 0.99       # Discount factor
25 epsilon = 1.0      # Initial exploration rate
26 epsilon_decay = 0.995
27 epsilon_min = 0.01
28 episodes = 2000    # Number of games
29  
30 # Training Loop
31 for episode in range(episodes):
32     state = env.reset()[0]
33     done = False
34  
35     while not done:
36         # Epsilon-greedy action selection
37         if np.random.rand() < epsilon:
38             action = np.random.choice(action_size)
39         else:
40             action = np.argmax(Q_table[state])
41  
42         # Take action and observe transition
43         next_state, reward, done, _, _ = env.step(action)
44  
45         # Update Q-table using the Bellman equation
46         Q_table[state, action] = Q_table[state, action] + alpha * (
47             reward + gamma * np.max(Q_table[next_state]) - Q_table[state, action]
48         )
49  
50         state = next_state
51  
52     # Decay epsilon to shift from exploration to exploitation
53     epsilon = max(epsilon * epsilon_decay, epsilon_min)
54  
55 print("Training complete! Non-zero learned Q-values:\n", np.round(Q_table[Q_table > 0][:5], 4))

⚡ How to Run in Google Colab (3 Quick Steps)

Click "Copy for Google Colab" above.
Click Open New Google Colab Notebook (opens in a new tab).
Click on the first empty code cell, press Ctrl + V (or Cmd + V) to paste, then press Shift + Enter to execute!

Launch Blank Colab Notebook

🔄 Algorithmic Execution Flow

Environment Setup

Creates a 16-state FrozenLake grid and initializes a 16x4 Q-table with zeros.

→

Epsilon-Greedy Interaction

Agent balances exploratory random actions with exploitative argmax actions, decaying epsilon from 1.0 to 0.01.

→

Bellman Propagation

Whenever the agent reaches the goal (reward=1), the reward propagates backwards across the Q-table, revealing the optimal trajectory.

📥 Input / Initial State

16x4 FrozenLake environment trained over 2,000 episodes

📤 Expected Output

Training complete! Non-zero learned Q-values:
[0.5832 0.6591 0.7451 0.8428 0.9510]

💡 What Does This Demonstration Show?

The Q-table converges to the optimal navigation policy, ensuring the agent takes the safest, shortest route across the frozen lake to reach the goal.

Course Learning Resource • Slide DeckLesson 612 Slides

Lesson 6: Reinforcement Learning Foundations

Q-learning, Bellman optimality equation, Q-table updates, and exploration-exploitation balance.

Author: Dr. Syed Muzamil Basha• FDP Approved Curriculum

Open in New Tab

📝 Learning Check

Test your understanding with these questions. Remember, this is for learning, not examination!

1	# =========================================================
2	# Q-Table Optimization on Gymnasium FrozenLake-v1
3	# Prepared for Google Colab (Run with Shift + Enter)
4	# =========================================================
5
6	# Step 1: Install required packages
7	!pip install gymnasium numpy
8
9	# Step 2: Import libraries
10	import numpy as np
11	import gymnasium as gym
12
13	# Step 3: Run Demonstration
14	# Create FrozenLake environment (4x4 grid = 16 states, 4 actions)
15	env = gym.make("FrozenLake-v1", is_slippery=False)
16	state_size = env.observation_space.n
17	action_size = env.action_space.n
18
19	# Initialize Q-table with zeros
20	Q_table = np.zeros((state_size, action_size))
21
22	# Hyperparameters
23	alpha = 0.1 # Learning rate
24	gamma = 0.99 # Discount factor
25	epsilon = 1.0 # Initial exploration rate
26	epsilon_decay = 0.995
27	epsilon_min = 0.01
28	episodes = 2000 # Number of games
29
30	# Training Loop
31	for episode in range(episodes):
32	state = env.reset()[0]
33	done = False
34
35	while not done:
36	# Epsilon-greedy action selection
37	if np.random.rand() < epsilon:
38	action = np.random.choice(action_size)
39	else:
40	action = np.argmax(Q_table[state])
41
42	# Take action and observe transition
43	next_state, reward, done, _, _ = env.step(action)
44
45	# Update Q-table using the Bellman equation
46	Q_table[state, action] = Q_table[state, action] + alpha * (
47	reward + gamma * np.max(Q_table[next_state]) - Q_table[state, action]
48	)
49
50	state = next_state
51
52	# Decay epsilon to shift from exploration to exploitation
53	epsilon = max(epsilon * epsilon_decay, epsilon_min)
54
55	print("Training complete! Non-zero learned Q-values:\n", np.round(Q_table[Q_table > 0][:5], 4))