First time studying this topic?

Launch the interactive walkthrough to discover each component, including the 3D Q-Sphere Lab & Python in Colab.

Topic 4.8

Exploration vs Exploitation: Epsilon-Greedy & UCB

Master the fundamental trade-off: exploiting known high-value actions versus exploring uncertain actions to discover even greater rewards.

Learning Objectives

✓Define the Exploration vs Exploitation dilemma in Multi-Armed Bandits
✓Implement Epsilon-Greedy and Epsilon-Decay strategies
✓Understand Upper Confidence Bound (UCB) and optimism in the face of uncertainty

📖Let's Begin with a Story

Imagine you move to a new city with 50 restaurants. On your first week, you try a small pizzeria near your apartment. The pizza is great—an 8 out of 10! Now it is Friday night. What do you do? - Exploit: You go back to the pizzeria. You are guaranteed a solid 8/10 meal. - Explore: You try a sushi restaurant down the road. It might be a terrible 2/10 (risk!). But it might be an extraordinary 10/10 dining experience you would have missed forever. If you only exploit, you miss the best restaurant in town. If you only explore, you suffer through terrible meals every night. Balancing exploration and exploitation is a universal challenge of life and AI.

Why Should You Learn Exploration vs Exploitation: Epsilon-Greedy & UCB?

🎯 Practical & Engineering Stakes

The dilemma that defines learning

Without exploration, agents get trapped in mediocre local optima. With too much exploration, agents waste energy making foolish random mistakes. Balancing them correctly is essential for rapid learning.

What Is It?

Simple Explanation

Exploit: Choose your favorite known choice to get a guaranteed good reward. Explore: Try an unfamiliar choice to see if it might be even better. **Epsilon-Greedy (

\epsilon

)**: - Roll a 100-sided die. - If it rolls below

\epsilon

(e.g., 10%), pick a random crazy move to explore! - Otherwise, pick your best-known move to exploit!

Technical Explanation

In multi-armed bandit settings, an agent seeks to minimize cumulative regret

R_T = T \mu^* - \sum_{t=1}^T \mathbb{E}[r_t]

. The Upper Confidence Bound (UCB1) algorithm chooses

A_t = \arg\max_a \left[ Q_t(a) + c \sqrt{\frac{\ln t}{N_t(a)}} \right]

, adding an uncertainty bonus that embodies "optimism in the face of uncertainty."

📚 Key Terms

Epsilon-Greedy

Simple: A strategy where the agent explores randomly with probability epsilon, and picks the best move with probability 1-epsilon.

Technical:

\pi(a|s) = 1-\epsilon + \frac{\epsilon}{|\mathcal{A}|}

for

a = \arg\max Q(s,a)

, and

\frac{\epsilon}{|\mathcal{A}|}

for other actions.

Upper Confidence Bound (UCB)

Simple: Choosing actions that have either high scores OR high uncertainty.

Technical: Deterministic action selection maximizing estimated value plus an exploration bonus proportional to the inverse frequency of visitation.

Epsilon Decay

Simple: Starting with lots of exploration when the AI is young, and slowly exploring less as it becomes an expert.

Technical: Annealing schedule

\epsilon_{t+1} = \max(\epsilon_{min}, \epsilon_t \times d)

where

d < 1

📐 Mathematical Foundation

LaTeX KaTeX

A_t = \arg\max_{a} \left[ Q_t(a) + c \sqrt{\frac{\ln t}{N_t(a)}} \right]

What does it mean?

The Upper Confidence Bound (UCB1) formula: selects the action that maximizes current average reward plus an uncertainty bonus based on how rarely it has been tried.

Understanding the Symbols

Symbol / Term	Meaning	Beginner Interpretation
$Q_t(a)$	Estimated reward	Exploitation term
$c \sqrt{\frac{\ln t}{N_t(a)}}$	Uncertainty bonus	Exploration term
$N_t(a)$	Count of times action a was tried	Less tried = higher bonus

📊 Numerical Example

Action 1 tried 100 times with Q=8. Action 2 tried only 2 times with Q=7. Action 2 gets a massive uncertainty bonus and gets chosen.

\text{Bonus}_2 \gg \text{Bonus}_1 \implies \text{Agent explores Action 2}

Result: Optimism in the face of uncertainty

⚙️ How Does It Work?

Generate Random Float

Draw uniform random number r between 0.0 and 1.0.

Branch Decision

If r < epsilon, pick random action (Explore); otherwise pick argmax Q(s,a) (Exploit).

Decay Epsilon

Multiply epsilon by decay factor (e.g. 0.995) after each episode.

🌍 Where Is It Used?

A/B Testing in Digital Marketing

Problem: Showing 5 different website landing page designs to find the highest conversion rate.

Solution: Multi-Armed Bandit algorithms route more traffic to winning designs while continuing to explore others.

🎯 Try It Yourself

Why do reinforcement learning practitioners start with epsilon = 1.0 (100% exploration) and gradually decay it to epsilon = 0.01?

Interactive activity component

🐍 Python Hands-On: Epsilon-Decay Implementation in Python

Simulate epsilon decay over 100 episodes and visualize action choices.

Open Colab

1Install Required Packages

For Google Colab or Jupyter notebooks, run this shell command to install the required dependencies:

Terminal / Notebook Install Commandbash

!pip install numpy

Packages:

numpy

2Import Libraries

Import Statementspython

1 import numpy as np

import numpy as np → NumPy for random choices

3Demonstration Code & Logic

Python Execution Codepython

1 epsilon = 1.0      # Start with 100% exploration
2 epsilon_min = 0.05 # Keep 5% minimum exploration
3 decay_rate = 0.95  # Decay 5% every episode
4  
5 exploration_count = 0
6 exploitation_count = 0
7  
8 for episode in range(100):
9     # Epsilon-greedy check:
10     if np.random.rand() < epsilon:
11         exploration_count += 1
12     else:
13         exploitation_count += 1
14     # Decay epsilon after episode:
15     epsilon = max(epsilon_min, epsilon * decay_rate)
16  
17 print(f"Final Epsilon: {epsilon:.3f}")
18 print(f"Total Explores: {exploration_count}, Total Exploits: {exploitation_count}")

Step-by-Step Code Breakdown:

epsilon = 1.0 # Start with 100% exploration

Initial epsilon

epsilon_min = 0.05 # Keep 5% minimum exploration

Floor

decay_rate = 0.95 # Decay 5% every episode

Decay factor

exploration_count = 0

Counter

exploitation_count = 0

Counter

for episode in range(100):

100 episodes

# Epsilon-greedy check:

Check

if np.random.rand() < epsilon:

Explore branch

exploration_count += 1

Count explore

else:

Exploit branch

exploitation_count += 1

Count exploit

# Decay epsilon after episode:

Decay step

epsilon = max(epsilon_min, epsilon * decay_rate)

Apply decay

print(f"Final Epsilon: {epsilon:.3f}")

Show final epsilon

print(f"Total Explores: {exploration_count}, Total Exploits: {exploitation_count}")

Show breakdown

📋 Complete Script (All-in-One)

Click "Copy for Colab", open Google Colab, paste into a single cell, and run.

Google Colab Notebook Scriptpython

1 # =========================================================
2 # Epsilon-Decay Implementation in Python
3 # Prepared for Google Colab (Run with Shift + Enter)
4 # =========================================================
5  
6 # Step 1: Install required packages
7 !pip install numpy
8  
9 # Step 2: Import libraries
10 import numpy as np
11  
12 # Step 3: Run Demonstration
13 epsilon = 1.0      # Start with 100% exploration
14 epsilon_min = 0.05 # Keep 5% minimum exploration
15 decay_rate = 0.95  # Decay 5% every episode
16  
17 exploration_count = 0
18 exploitation_count = 0
19  
20 for episode in range(100):
21     # Epsilon-greedy check:
22     if np.random.rand() < epsilon:
23         exploration_count += 1
24     else:
25         exploitation_count += 1
26     # Decay epsilon after episode:
27     epsilon = max(epsilon_min, epsilon * decay_rate)
28  
29 print(f"Final Epsilon: {epsilon:.3f}")
30 print(f"Total Explores: {exploration_count}, Total Exploits: {exploitation_count}")

⚡ How to Run in Google Colab (3 Quick Steps)

Click "Copy for Google Colab" above.
Click Open New Google Colab Notebook (opens in a new tab).
Click on the first empty code cell, press Ctrl + V (or Cmd + V) to paste, then press Shift + Enter to execute!

Launch Blank Colab Notebook

🔄 Algorithmic Execution Flow

Early Phase

High epsilon drives aggressive exploration.

→

Late Phase

Epsilon drops to 0.05, and the agent exploits 95% of the time.

📥 Input / Initial State

100 episodes with initial epsilon 1.0 and decay 0.95

📤 Expected Output

Final Epsilon: 0.050
Total Explores: ~24, Total Exploits: ~76

💡 What Does This Demonstration Show?

Epsilon decay seamlessly transitions the agent from a curious explorer into an efficient optimizer.

Course Learning Resource • Slide DeckLesson 612 Slides

Lesson 6: Reinforcement Learning Foundations

Q-learning, Q-table updates, DQN architectures, and exploration-exploitation trade-offs.

Author: Dr. Syed Muzamil Basha• FDP Approved Curriculum

Open in New Tab

📝 Learning Check

Test your understanding with these questions. Remember, this is for learning, not examination!

1	# =========================================================
2	# Epsilon-Decay Implementation in Python
3	# Prepared for Google Colab (Run with Shift + Enter)
4	# =========================================================
5
6	# Step 1: Install required packages
7	!pip install numpy
8
9	# Step 2: Import libraries
10	import numpy as np
11
12	# Step 3: Run Demonstration
13	epsilon = 1.0 # Start with 100% exploration
14	epsilon_min = 0.05 # Keep 5% minimum exploration
15	decay_rate = 0.95 # Decay 5% every episode
16
17	exploration_count = 0
18	exploitation_count = 0
19
20	for episode in range(100):
21	# Epsilon-greedy check:
22	if np.random.rand() < epsilon:
23	exploration_count += 1
24	else:
25	exploitation_count += 1
26	# Decay epsilon after episode:
27	epsilon = max(epsilon_min, epsilon * decay_rate)
28
29	print(f"Final Epsilon: {epsilon:.3f}")
30	print(f"Total Explores: {exploration_count}, Total Exploits: {exploitation_count}")