First time studying this topic?

Launch the interactive walkthrough to discover each component, including the 3D Q-Sphere Lab & Python in Colab.

Topic 4.7

The Policy: Deterministic vs Stochastic Strategies

Deep dive into the agent’s decision function: deterministic argmax policies versus stochastic probability distributions.

Learning Objectives

✓Define Deterministic Policies: a = pi(s)
✓Define Stochastic Policies: pi(a|s) = P(A=a|S=s)
✓Understand why stochastic policies are required in partially observable games (like Poker)

📖Let's Begin with a Story

Imagine playing Rock-Paper-Scissors against an opponent who plays a Deterministic Policy: - Every time they are in State 1, they throw Rock with 100% certainty. How long will it take you to crush them? Exactly one turn! Once you observe their deterministic pattern, you throw Paper every time and win 100% of games. To be unexploitable in competitive, partially observable games, an agent must adopt a Stochastic Policy: playing Rock with 33.3%, Paper with 33.3%, and Scissors with 33.3% probability. Stochastic policies enable optimal mixed strategies (Nash Equilibria).

Why Should You Learn The Policy: Deterministic vs Stochastic Strategies?

🎯 Practical & Engineering Stakes

The core output of RL

Everything in reinforcement learning ultimately serves to optimize the policy

\pi^*

. Understanding deterministic vs stochastic representations is crucial when choosing between Q-learning and Policy Gradient methods.

What Is It?

Simple Explanation

A Policy is the agent’s rulebook: - Deterministic Policy: "If red light, STOP. No debate, no randomness." - Stochastic Policy: "If exploring a maze, go Left with 70% chance and Right with 30% chance." It allows for creativity, exploration, and unpredictability.

Technical Explanation

A deterministic policy is a function mapping state space to action space:

\mu: \mathcal{S} \rightarrow \mathcal{A}

. A stochastic policy is a conditional probability distribution

\pi: \mathcal{S} \times \mathcal{A} \rightarrow [0, 1]

satisfying

\sum_{a \in \mathcal{A}} \pi(a|s) = 1

. The Policy Gradient Theorem computes

\nabla_\theta J(\theta) = \mathbb{E}_\pi [\nabla_\theta \log \pi_\theta(a|s) Q^\pi(s, a)]

📚 Key Terms

Deterministic Policy

Simple: A fixed rule that always picks the exact same action in a situation.

Technical: Mapping

\mu(s)

yielding identical action

a

whenever evaluated on state

s

Stochastic Policy

Simple: A rule that assigns probabilities to different actions, picking randomly based on those odds.

Technical: Distribution

\pi(a|s)

enabling exploration and optimal mixed strategy formulation under uncertainty.

📐 Mathematical Foundation

LaTeX KaTeX

\pi(a \mid s) = \frac{e^{Q(s, a) / \tau}}{\sum_{b} e^{Q(s, b) / \tau}}

What does it mean?

Boltzmann (Softmax) Exploration Policy: converts Q-values into action selection probabilities scaled by temperature parameter tau.

Understanding the Symbols

Symbol / Term	Meaning	Beginner Interpretation
$Q(s, a)$	Action quality value	Higher Q yields higher probability
$\tau$	Temperature hyperparameter	High tau = random exploration; low tau = greedy exploitation

📊 Numerical Example

Actions with Q-values [2.0, 1.0] with tau = 1.0.

e^2 / (e^2 + e^1) = 7.39 / (7.39 + 2.72) = 7.39 / 10.11 = 0.73

Result: 73% probability of picking action 1, 27% probability for action 2

⚙️ How Does It Work?

State Ingestion

Agent takes current state s as input.

Action Probability Distribution

Network outputs vector of probabilities summing to 1.

Sampling vs Greedy

During training, sample randomly according to probabilities; during testing, pick argmax.

🌍 Where Is It Used?

Algorithmic Poker AI (Pluribus / Libratus)

Problem: Playing multi-player No-Limit Texas Hold’em against top human pros.

Solution: Stochastic mixed policies randomize bluffing frequencies to remain mathematically unexploitable.

🎯 Try It Yourself

Why would a deterministic policy fail in Rock-Paper-Scissors?

Interactive activity component

🐍 Python Hands-On: Sampling from a Softmax Boltzmann Policy

Convert Q-values into an action probability distribution and sample actions.

Open Colab

1Install Required Packages

For Google Colab or Jupyter notebooks, run this shell command to install the required dependencies:

Terminal / Notebook Install Commandbash

!pip install numpy

Packages:

numpy

2Import Libraries

Import Statementspython

1 import numpy as np

import numpy as np → NumPy math

3Demonstration Code & Logic

Python Execution Codepython

1 # Q-values for 3 possible actions: [Left, Stay, Right]
2 q_values = np.array([1.5, 3.0, 0.5])
3 tau = 1.0 # Temperature
4  
5 # Compute Boltzmann Softmax probabilities:
6 exp_q = np.exp(q_values / tau)
7 probs = exp_q / np.sum(exp_q)
8 print("Action Probabilities [Left, Stay, Right]:", probs.round(3))
9  
10 # Sample an action based on the stochastic policy:
11 actions = [0, 1, 2]
12 sampled_action = np.random.choice(actions, p=probs)
13 print("Sampled Action according to policy:", sampled_action)

Step-by-Step Code Breakdown:

# Q-values for 3 possible actions: [Left, Stay, Right]

Q values

q_values = np.array([1.5, 3.0, 0.5])

Action 1 (Stay) is best

tau = 1.0 # Temperature

Temperature

# Compute Boltzmann Softmax probabilities:

Softmax calculation

exp_q = np.exp(q_values / tau)

Exponentials

probs = exp_q / np.sum(exp_q)

Normalize

print("Action Probabilities [Left, Stay, Right]:", probs.round(3))

Print probs

# Sample an action based on the stochastic policy:

Sampling

actions = [0, 1, 2]

Action indices

sampled_action = np.random.choice(actions, p=probs)

Sample with weights

print("Sampled Action according to policy:", sampled_action)

Print chosen action

📋 Complete Script (All-in-One)

Click "Copy for Colab", open Google Colab, paste into a single cell, and run.

Google Colab Notebook Scriptpython

1 # =========================================================
2 # Sampling from a Softmax Boltzmann Policy
3 # Prepared for Google Colab (Run with Shift + Enter)
4 # =========================================================
5  
6 # Step 1: Install required packages
7 !pip install numpy
8  
9 # Step 2: Import libraries
10 import numpy as np
11  
12 # Step 3: Run Demonstration
13 # Q-values for 3 possible actions: [Left, Stay, Right]
14 q_values = np.array([1.5, 3.0, 0.5])
15 tau = 1.0 # Temperature
16  
17 # Compute Boltzmann Softmax probabilities:
18 exp_q = np.exp(q_values / tau)
19 probs = exp_q / np.sum(exp_q)
20 print("Action Probabilities [Left, Stay, Right]:", probs.round(3))
21  
22 # Sample an action based on the stochastic policy:
23 actions = [0, 1, 2]
24 sampled_action = np.random.choice(actions, p=probs)
25 print("Sampled Action according to policy:", sampled_action)

⚡ How to Run in Google Colab (3 Quick Steps)

Click "Copy for Google Colab" above.
Click Open New Google Colab Notebook (opens in a new tab).
Click on the first empty code cell, press Ctrl + V (or Cmd + V) to paste, then press Shift + Enter to execute!

Launch Blank Colab Notebook

🔄 Algorithmic Execution Flow

Probability Calibration

Q-values [1.5, 3.0, 0.5] map to probabilities [16.8%, 75.3%, 7.9%].

→

Action Sampling

Action 1 is chosen 75% of the time, while still exploring Actions 0 and 2.

📥 Input / Initial State

Q-values [1.5, 3.0, 0.5]

📤 Expected Output

Action Probabilities [Left, Stay, Right]: [0.168 0.753 0.079]
Sampled Action according to policy: 1

💡 What Does This Demonstration Show?

Stochastic policies balance exploitation of the best-known move (Action 1) with probabilistic exploration of alternatives.

Course Learning Resource • Slide DeckLesson 612 Slides

Lesson 6: Reinforcement Learning Foundations

Q-learning, Q-table updates, DQN architectures, and exploration-exploitation trade-offs.

Author: Dr. Syed Muzamil Basha• FDP Approved Curriculum

Open in New Tab

📝 Learning Check

Test your understanding with these questions. Remember, this is for learning, not examination!

Question 1: What happens to a Boltzmann softmax policy as temperature tau approaches zero (tau -> 0)?

Question 2: What is a deterministic policy?

Question 3: What is a stochastic policy?

Question 4: What is the optimal policy π*?

Question 5: What is a value-based vs policy-based RL method?

Previous: The Reward Signal: Reward Engineering & Shaping Next: Exploration vs Exploitation: Epsilon-Greedy & UCB

1	# =========================================================
2	# Sampling from a Softmax Boltzmann Policy
3	# Prepared for Google Colab (Run with Shift + Enter)
4	# =========================================================
5
6	# Step 1: Install required packages
7	!pip install numpy
8
9	# Step 2: Import libraries
10	import numpy as np
11
12	# Step 3: Run Demonstration
13	# Q-values for 3 possible actions: [Left, Stay, Right]
14	q_values = np.array([1.5, 3.0, 0.5])
15	tau = 1.0 # Temperature
16
17	# Compute Boltzmann Softmax probabilities:
18	exp_q = np.exp(q_values / tau)
19	probs = exp_q / np.sum(exp_q)
20	print("Action Probabilities [Left, Stay, Right]:", probs.round(3))
21
22	# Sample an action based on the stochastic policy:
23	actions = [0, 1, 2]
24	sampled_action = np.random.choice(actions, p=probs)
25	print("Sampled Action according to policy:", sampled_action)