First time studying this topic?

Launch the interactive walkthrough to discover each component, including the 3D Q-Sphere Lab & Python in Colab.

Topic 4.3

The Environment: Dynamics, Observability & Gym

Understand the world the agent inhabits: deterministic vs stochastic worlds, full vs partial observability, and the Gymnasium API.

Learning Objectives

✓Differentiate between Fully Observable (MDP) and Partially Observable (POMDP) environments
✓Compare deterministic vs stochastic transition dynamics
✓Master the standard Gymnasium (OpenAI Gym) API: reset(), step(), reward, done

📖Let's Begin with a Story

Imagine playing Chess versus playing Poker. In Chess, both players can see every single piece on the 64 squares at all times. Nothing is hidden. This is a Fully Observable Environment (MDP). In Poker, you cannot see your opponents' hidden hole cards, and you cannot see the future cards remaining in the deck. You must infer what cards they might hold based on their betting patterns under deep uncertainty. This is a Partially Observable Markov Decision Process (POMDP). Building an AI for a fully observable world is vastly different from building one that can thrive in a world of hidden information.

Why Should You Learn The Environment: Dynamics, Observability & Gym?

🎯 Practical & Engineering Stakes

The interface of RL

Every RL benchmark on earth—from robotics simulators to Atari games—uses the Gymnasium API standard. Mastering this interface is essential to train your own agents.

What Is It?

Simple Explanation

The environment is the game or world the AI lives in: - Full Observability: You can see everything (like chess). - Partial Observability: You only see what’s in front of you through fog (like a first-person shooter or poker). - The Gym API: The universal game controller: - reset(): Restart the game. - step(action): Take a move and see what happens!

Technical Explanation

In a POMDP

\langle \mathcal{S}, \mathcal{A}, \mathcal{O}, \mathcal{P}, \Omega, \mathcal{R}, \gamma \rangle

, the agent does not receive the true state

s_t

, but rather an observation

o_t \sim \Omega(o|s_t)

, necessitating belief-state tracking

b(s) = P(S_t=s | h_t)

. Gymnasium abstracts this with discrete/continuous observation spaces (

Box

Discrete

📚 Key Terms

Gymnasium (OpenAI Gym)

Simple: The universal Python toolkit used to build and benchmark RL environments.

Technical: The standardized Python interface exposing env.reset() and env.step(action) yielding (obs, reward, terminated, truncated, info).

Fully Observable (MDP)

Simple: The agent can see the entire state of the world.

Technical: An environment where observation

o_t

equals true environmental state

s_t

Partially Observable (POMDP)

Simple: The agent can only see part of the world; key information is hidden.

Technical: An environment where observations provide incomplete sensory projections of the underlying Markov state.

📐 Mathematical Foundation

LaTeX KaTeX

s_{t+1} \sim P(s_{t+1} \mid s_t, a_t), \quad r_{t+1} = R(s_t, a_t, s_{t+1})

What does it mean?

Environmental Transition Dynamics: given current state s_t and chosen action a_t, the environment probabilistically determines the next state and returns a scalar reward.

Understanding the Symbols

Symbol / Term	Meaning	Beginner Interpretation
$P(s_{t+1}\|s_t, a_t)$	Transition probability distribution	How the world changes
$R$	Reward function	Points awarded for transition

📊 Numerical Example

A robot tries to move forward with slippery wheels: 80% chance it succeeds, 20% chance it slips and stays in place.

P(s+1 | s, \text{forward}) = 0.8, \quad P(s | s, \text{forward}) = 0.2

Result: Stochastic environment transition

⚙️ How Does It Work?

Environment Reset

obs, info = env.reset() initializes the world and returns initial sensory observation.

Action Execution

obs, reward, terminated, truncated, info = env.step(action) updates physics.

Termination Check

If terminated or truncated is True, episode ends and environment resets.

🌍 Where Is It Used?

Robot Manipulation in Isaac Sim / MuJoCo

Problem: Training physical bipedal humanoid robots without breaking expensive hardware.

Solution: High-fidelity physics environments simulate gravity and joint friction at 10,000x real-time.

🎯 Try It Yourself

A robot equipped only with a forward-facing camera navigates a maze. It cannot see behind itself or through walls. Is this environment Fully Observable or Partially Observable?

Interactive activity component

🐍 Python Hands-On: Interacting with a Gymnasium Environment

Run the standard Gymnasium interaction loop with random actions.

Open Colab

1Install Required Packages

For Google Colab or Jupyter notebooks, run this shell command to install the required dependencies:

Terminal / Notebook Install Commandbash

!# Standard Gymnasium pattern simulation

Packages:

numpy

2Import Libraries

Import Statementspython

1 import numpy as np

import numpy as np → Array processing

3Demonstration Code & Logic

Python Execution Codepython

1 # A minimal Gymnasium-compliant Environment class:
2 class SimpleCartPole:
3     def __init__(self):
4         self.pole_angle = 0.0
5     def reset(self):
6         self.pole_angle = np.random.uniform(-0.05, 0.05)
7         return np.array([self.pole_angle])
8     def step(self, action):
9         # Action: 0 = Push Left, 1 = Push Right
10         force = 0.1 if action == 1 else -0.1
11         self.pole_angle += force + np.random.normal(0, 0.02)
12         reward = 1.0 # +1 for every step balanced
13         terminated = abs(self.pole_angle) > 0.5 # Fell over!
14         return np.array([self.pole_angle]), reward, terminated
15  
16 env = SimpleCartPole()
17 obs = env.reset()
18 total_reward = 0
19  
20 for t in range(50):
21     action = 1 if obs[0] < 0 else 0 # Simple heuristic: push opposite to tilt
22     obs, reward, terminated = env.step(action)
23     total_reward += reward
24     if terminated:
25         print(f"Pole fell over at step {t+1}!")
26         break
27 print(f"Total Episode Reward: {total_reward}")

Step-by-Step Code Breakdown:

# A minimal Gymnasium-compliant Environment class:

Class structure

class SimpleCartPole:

Simulated environment

def __init__(self):

Constructor

self.pole_angle = 0.0

State: angle in degrees

def reset(self):

Reset method

self.pole_angle = np.random.uniform(-0.05, 0.05)

Slight random start

return np.array([self.pole_angle])

Return observation

def step(self, action):

Step method

# Action: 0 = Push Left, 1 = Push Right

Action space

force = 0.1 if action == 1 else -0.1

Apply force

self.pole_angle += force + np.random.normal(0, 0.02)

Physics update

reward = 1.0 # +1 for every step balanced

Reward

terminated = abs(self.pole_angle) > 0.5 # Fell over!

Terminal condition

return np.array([self.pole_angle]), reward, terminated

Return tuple

env = SimpleCartPole()

Instantiate env

obs = env.reset()

Reset world

total_reward = 0

Accumulator

for t in range(50):

Run up to 50 steps

action = 1 if obs[0] < 0 else 0 # Simple heuristic: push opposite to tilt

Reflex policy

obs, reward, terminated = env.step(action)

Step environment

total_reward += reward

Add reward

if terminated:

Check game over

print(f"Pole fell over at step {t+1}!")

Terminated message

break

Exit loop

print(f"Total Episode Reward: {total_reward}")

Display total points

📋 Complete Script (All-in-One)

Click "Copy for Colab", open Google Colab, paste into a single cell, and run.

Google Colab Notebook Scriptpython

1 # =========================================================
2 # Interacting with a Gymnasium Environment
3 # Prepared for Google Colab (Run with Shift + Enter)
4 # =========================================================
5  
6 # Step 1: Install required packages
7 !# Standard Gymnasium pattern simulation
8  
9 # Step 2: Import libraries
10 import numpy as np
11  
12 # Step 3: Run Demonstration
13 # A minimal Gymnasium-compliant Environment class:
14 class SimpleCartPole:
15     def __init__(self):
16         self.pole_angle = 0.0
17     def reset(self):
18         self.pole_angle = np.random.uniform(-0.05, 0.05)
19         return np.array([self.pole_angle])
20     def step(self, action):
21         # Action: 0 = Push Left, 1 = Push Right
22         force = 0.1 if action == 1 else -0.1
23         self.pole_angle += force + np.random.normal(0, 0.02)
24         reward = 1.0 # +1 for every step balanced
25         terminated = abs(self.pole_angle) > 0.5 # Fell over!
26         return np.array([self.pole_angle]), reward, terminated
27  
28 env = SimpleCartPole()
29 obs = env.reset()
30 total_reward = 0
31  
32 for t in range(50):
33     action = 1 if obs[0] < 0 else 0 # Simple heuristic: push opposite to tilt
34     obs, reward, terminated = env.step(action)
35     total_reward += reward
36     if terminated:
37         print(f"Pole fell over at step {t+1}!")
38         break
39 print(f"Total Episode Reward: {total_reward}")

⚡ How to Run in Google Colab (3 Quick Steps)

Click "Copy for Google Colab" above.
Click Open New Google Colab Notebook (opens in a new tab).
Click on the first empty code cell, press Ctrl + V (or Cmd + V) to paste, then press Shift + Enter to execute!

Launch Blank Colab Notebook

🔄 Algorithmic Execution Flow

Reset

Initializes pole with slight tilt angle.

→

Step Loop

Agent observes angle, pushes cart, and earns +1 reward per balanced timestep.

📥 Input / Initial State

Simulated CartPole physics environment

📤 Expected Output

Total Episode Reward: 50.0 (Survived full episode!)

💡 What Does This Demonstration Show?

The standard env.reset() and env.step() contract enables testing arbitrary algorithms across standardized environments.

Course Learning Resource • Slide DeckLesson 612 Slides

Lesson 6: Reinforcement Learning Foundations

Q-learning, Q-table updates, DQN architectures, and exploration-exploitation trade-offs.

Author: Dr. Syed Muzamil Basha• FDP Approved Curriculum

Open in New Tab

📝 Learning Check

Test your understanding with these questions. Remember, this is for learning, not examination!

Question 1: What are the primary return values of the standard Gymnasium env.step(action) method?

Question 2: What defines an RL environment?

Question 3: What is a deterministic vs stochastic environment?

Question 4: What is a fully observable vs partially observable environment?

Question 5: What is OpenAI Gym?

Previous: The RL Agent: Internal Architecture Next: State Representation & The Markov Property

1	# =========================================================
2	# Interacting with a Gymnasium Environment
3	# Prepared for Google Colab (Run with Shift + Enter)
4	# =========================================================
5
6	# Step 1: Install required packages
7	!# Standard Gymnasium pattern simulation
8
9	# Step 2: Import libraries
10	import numpy as np
11
12	# Step 3: Run Demonstration
13	# A minimal Gymnasium-compliant Environment class:
14	class SimpleCartPole:
15	def __init__(self):
16	self.pole_angle = 0.0
17	def reset(self):
18	self.pole_angle = np.random.uniform(-0.05, 0.05)
19	return np.array([self.pole_angle])
20	def step(self, action):
21	# Action: 0 = Push Left, 1 = Push Right
22	force = 0.1 if action == 1 else -0.1
23	self.pole_angle += force + np.random.normal(0, 0.02)
24	reward = 1.0 # +1 for every step balanced
25	terminated = abs(self.pole_angle) > 0.5 # Fell over!
26	return np.array([self.pole_angle]), reward, terminated
27
28	env = SimpleCartPole()
29	obs = env.reset()
30	total_reward = 0
31
32	for t in range(50):
33	action = 1 if obs[0] < 0 else 0 # Simple heuristic: push opposite to tilt
34	obs, reward, terminated = env.step(action)
35	total_reward += reward
36	if terminated:
37	print(f"Pole fell over at step {t+1}!")
38	break
39	print(f"Total Episode Reward: {total_reward}")