First time studying this topic?

Launch the interactive walkthrough to discover each component, including the 3D Q-Sphere Lab & Python in Colab.

Topic 4.5

Action Spaces: Discrete vs Continuous

Explore the action space: discrete choices (buttons, moves) versus continuous actuator controls (steering angles, voltages).

Learning Objectives

✓Distinguish between Discrete and Continuous action spaces
✓Map action spaces in Gymnasium: Discrete(N) vs Box(low, high, shape)
✓Understand how action representations dictate algorithm selection (DQN vs DDPG/PPO)

📖Let's Begin with a Story

Imagine controlling a video game character versus driving a real car. In a retro video game like Super Mario, you have an NES controller with 4 buttons: Left, Right, Jump, Run. At any frame, you pick one of a finite set of discrete button combinations. Now step into a Formula 1 race car. You don't have discrete buttons for steering. You turn the steering wheel continuously between -180.0° and +180.0° with infinite precision (e.g. 14.372°). You press the throttle pedal with continuous foot pressure between 0% and 100%. This fundamental divide—Discrete vs Continuous Action Spaces—determines whether you choose algorithms like Q-learning (built for discrete choices) or Policy Gradients (built for continuous torque and angles).

Why Should You Learn Action Spaces: Discrete vs Continuous?

🎯 Practical & Engineering Stakes

Matching algorithm to physics

Trying to use standard Deep Q-Networks (DQN) on a continuous robotic arm requires artificially discretizing thousands of angles, leading to an explosion of actions. Understanding action spaces ensures you deploy the right agent.

What Is It?

Simple Explanation

Discrete Actions: Choosing from a menu of distinct options: - Go Left, Go Right, Jump, Shoot. Continuous Actions: Turning a knob or pressing a pedal with smooth, continuous numbers: - Steer 12.5 degrees left, apply 45.2% braking force.

Technical Explanation

A discrete action space is a finite set

\mathcal{A} = \{a_1, a_2, \dots, a_K\}

where policies output categorical probabilities

\pi(a_k|s)

. A continuous action space is a compact subset of Euclidean space

\mathcal{A} \subseteq \mathbb{R}^m

where policies parameterize continuous probability distributions, typically Gaussian

\pi(a|s) = \mathcal{N}(\mu_\\heta(s), \sigma_\\heta^2(s))

📚 Key Terms

Discrete Action Space

Simple: A finite list of separate choices (e.g., UP, DOWN, LEFT, RIGHT).

Technical: Represented in Gymnasium as gym.spaces.Discrete(n) with cardinality

n

Continuous Action Space

Simple: Smooth values with infinite possible decimal numbers (e.g., steering angle from -1.0 to +1.0).

Technical: Represented in Gymnasium as gym.spaces.Box(low, high, shape) representing an

m

-dimensional real bounding box.

📐 Mathematical Foundation

LaTeX KaTeX

a \sim \pi_\theta(a \mid s) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left( -\frac{(a - \mu_\theta(s))^2}{2\sigma^2} \right)

What does it mean?

Gaussian Policy for continuous actions: neural network outputs mean action mu(s) and variance sigma^2, from which the continuous action is sampled.

Understanding the Symbols

Symbol / Term	Meaning	Beginner Interpretation
$\mu_\theta(s)$	Mean action output	The ideal steering angle or torque
$\sigma^2$	$Variance$	Exploration noise around the mean
$a$	Sampled continuous action	Executed control signal

📊 Numerical Example

Network predicts mean steering angle mu = 15.0 degrees with exploration noise sigma = 1.0 degree.

a = 15.0 + 1.0 \times \mathcal{N}(0, 1) = 15.24^{\circ}

Result: Smooth continuous actuator command

⚙️ How Does It Work?

Discrete Handling

Network outputs K raw logits; apply Softmax or take argmax for discrete decisions.

Continuous Handling

Network outputs continuous vector mu; apply Tanh to clamp values within physical actuator limits [-1.0, +1.0].

🌍 Where Is It Used?

Quadruped Robot Locomotion (Boston Dynamics / ANYmal)

Problem: Controlling 12 continuous motor torques 500 times per second.

Solution: Continuous action Actor-Critic networks output smooth torque values directly to joint actuators.

🎯 Try It Yourself

A smart thermostat controls an electric furnace by setting the heat output continuously between 0.0% and 100.0%. What type of action space is this?

Interactive activity component

🐍 Python Hands-On: Sampling Discrete vs Continuous Actions in Python

Demonstrate sampling discrete choices vs continuous Gaussian control signals.

Open Colab

1Install Required Packages

For Google Colab or Jupyter notebooks, run this shell command to install the required dependencies:

Terminal / Notebook Install Commandbash

!pip install numpy

Packages:

numpy

2Import Libraries

Import Statementspython

1 import numpy as np

import numpy as np → NumPy random module

3Demonstration Code & Logic

Python Execution Codepython

1 # 1. Discrete Action Space (e.g. 4 joystick directions):
2 discrete_actions = ["UP", "DOWN", "LEFT", "RIGHT"]
3 chosen_discrete = np.random.choice(discrete_actions)
4 print("Discrete Action Sampled:", chosen_discrete)
5  
6 # 2. Continuous Action Space (e.g. Steering Angle [-30 to +30 deg]):
7 mean_steering = 5.2 # Network predicts 5.2 degrees right
8 noise_std = 0.5     # Exploration noise
9 continuous_action = np.random.normal(mean_steering, noise_std)
10 continuous_action = np.clip(continuous_action, -30.0, 30.0) # Enforce physical bounds
11 print(f"Continuous Steering Sampled: {continuous_action:.2f} degrees")

Step-by-Step Code Breakdown:

# 1. Discrete Action Space (e.g. 4 joystick directions):

Discrete

discrete_actions = ["UP", "DOWN", "LEFT", "RIGHT"]

Choices

chosen_discrete = np.random.choice(discrete_actions)

Sample discrete

print("Discrete Action Sampled:", chosen_discrete)

Print discrete

# 2. Continuous Action Space (e.g. Steering Angle [-30 to +30 deg]):

Continuous

mean_steering = 5.2 # Network predicts 5.2 degrees right

Predicted mean

noise_std = 0.5 # Exploration noise

Exploration noise

continuous_action = np.random.normal(mean_steering, noise_std)

Sample Gaussian

continuous_action = np.clip(continuous_action, -30.0, 30.0) # Enforce physical bounds

Clip limits

print(f"Continuous Steering Sampled: {continuous_action:.2f} degrees")

Print continuous

📋 Complete Script (All-in-One)

Click "Copy for Colab", open Google Colab, paste into a single cell, and run.

Google Colab Notebook Scriptpython

1 # =========================================================
2 # Sampling Discrete vs Continuous Actions in Python
3 # Prepared for Google Colab (Run with Shift + Enter)
4 # =========================================================
5  
6 # Step 1: Install required packages
7 !pip install numpy
8  
9 # Step 2: Import libraries
10 import numpy as np
11  
12 # Step 3: Run Demonstration
13 # 1. Discrete Action Space (e.g. 4 joystick directions):
14 discrete_actions = ["UP", "DOWN", "LEFT", "RIGHT"]
15 chosen_discrete = np.random.choice(discrete_actions)
16 print("Discrete Action Sampled:", chosen_discrete)
17  
18 # 2. Continuous Action Space (e.g. Steering Angle [-30 to +30 deg]):
19 mean_steering = 5.2 # Network predicts 5.2 degrees right
20 noise_std = 0.5     # Exploration noise
21 continuous_action = np.random.normal(mean_steering, noise_std)
22 continuous_action = np.clip(continuous_action, -30.0, 30.0) # Enforce physical bounds
23 print(f"Continuous Steering Sampled: {continuous_action:.2f} degrees")

⚡ How to Run in Google Colab (3 Quick Steps)

Click "Copy for Google Colab" above.
Click Open New Google Colab Notebook (opens in a new tab).
Click on the first empty code cell, press Ctrl + V (or Cmd + V) to paste, then press Shift + Enter to execute!

Launch Blank Colab Notebook

🔄 Algorithmic Execution Flow

Discrete Choice

Selects one discrete string from fixed set.

→

Continuous Sampling

Samples continuous floating-point angle with Gaussian exploration noise.

📥 Input / Initial State

Action definitions

📤 Expected Output

Discrete Action Sampled: LEFT
Continuous Steering Sampled: 5.48 degrees

💡 What Does This Demonstration Show?

Discrete actions are finite integers/labels; continuous actions are real-valued scalars representing physical commands.

Course Learning Resource • Slide DeckLesson 612 Slides

Lesson 6: Reinforcement Learning Foundations

Q-learning, Q-table updates, DQN architectures, and exploration-exploitation trade-offs.

Author: Dr. Syed Muzamil Basha• FDP Approved Curriculum

Open in New Tab

📝 Learning Check

Test your understanding with these questions. Remember, this is for learning, not examination!

Question 1: Which reinforcement learning algorithm is natively designed for discrete action spaces?

Question 2: What is an action space in RL?

Question 3: What is a discrete vs continuous action space?

Question 4: What is action masking?

Question 5: What is an action-value function Q(s,a)?

Previous: State Representation & The Markov Property Next: The Reward Signal: Reward Engineering & Shaping

1	# =========================================================
2	# Sampling Discrete vs Continuous Actions in Python
3	# Prepared for Google Colab (Run with Shift + Enter)
4	# =========================================================
5
6	# Step 1: Install required packages
7	!pip install numpy
8
9	# Step 2: Import libraries
10	import numpy as np
11
12	# Step 3: Run Demonstration
13	# 1. Discrete Action Space (e.g. 4 joystick directions):
14	discrete_actions = ["UP", "DOWN", "LEFT", "RIGHT"]
15	chosen_discrete = np.random.choice(discrete_actions)
16	print("Discrete Action Sampled:", chosen_discrete)
17
18	# 2. Continuous Action Space (e.g. Steering Angle [-30 to +30 deg]):
19	mean_steering = 5.2 # Network predicts 5.2 degrees right
20	noise_std = 0.5 # Exploration noise
21	continuous_action = np.random.normal(mean_steering, noise_std)
22	continuous_action = np.clip(continuous_action, -30.0, 30.0) # Enforce physical bounds
23	print(f"Continuous Steering Sampled: {continuous_action:.2f} degrees")