Launch the interactive walkthrough to discover each component, including the 3D Q-Sphere Lab & Python in Colab.
Master the engine of reinforcement: sparse vs dense rewards, reward shaping, and avoiding perverse alignment incentives.
| Symbol / Term | Meaning | Beginner Interpretation |
|---|---|---|
| Potential of state s | Heuristic desirability (e.g. negative distance to goal) | |
| Discount factor | Discount parameter |
Robot gets closer to goal: Phi(s) = -5, Phi(s') = -3 with gamma = 1. Auxiliary reward is (-3) - (-5) = +2.
Result: Safe reward encouraging movement toward goal
Assign large positive reward (+100) for success and penalty (-100) for failure.
Add small negative reward (-0.1 per step) to incentivize speed.
Add difference of potentials to guide agent through sparse search spaces.
Problem: Randomly searching a 3D space to insert a peg into a hole has a 0.0001% random success rate.
Solution: Dense reward proportional to negative Euclidean distance drives robot toward the target hole.
Interactive activity component
Compare raw sparse rewards with shaped dense rewards in a 1D environment.
For Google Colab or Jupyter notebooks, run this shell command to install the required dependencies:
!pip install numpy1 import numpy as np
import numpy as np → NumPy math
1 # Position x along 1D line from 0 to 10 (Goal is 10) 2 def potential(x): 3 return x # Closer to 10 has higher potential 4 5 gamma = 0.99 6 current_x = 3.0 7 next_x = 4.0 # Step forward 8 9 # Sparse reward: 0 until goal 10 r_sparse = 0.0 11 12 # Potential-based shaped reward: r + gamma * Phi(s') - Phi(s) 13 r_shaped = r_sparse + (gamma * potential(next_x) - potential(current_x)) 14 15 print(f"Raw Sparse Reward: {r_sparse:.2f}") 16 print(f"Shaped Dense Reward: {r_shaped:.2f} (Provides immediate positive gradient!)")
# Position x along 1D line from 0 to 10 (Goal is 10)
Setup
def potential(x):
Define potential function
return x # Closer to 10 has higher potential
Phi(x) = x
gamma = 0.99
Discount
current_x = 3.0
Start at x=3
next_x = 4.0 # Step forward
Move forward
# Sparse reward: 0 until goal
Sparse baseline
r_sparse = 0.0
No intermediate reward
# Potential-based shaped reward: r + gamma * Phi(s') - Phi(s)
Shaping formula
r_shaped = r_sparse + (gamma * potential(next_x) - potential(current_x))
Calculate shaping
print(f"Raw Sparse Reward: {r_sparse:.2f}")
Print sparse
print(f"Shaped Dense Reward: {r_shaped:.2f} (Provides immediate positive gradient!)")
Print shaped
Click "Copy for Colab", open Google Colab, paste into a single cell, and run.
1 # ========================================================= 2 # Potential-Based Reward Shaping in Python 3 # Prepared for Google Colab (Run with Shift + Enter) 4 # ========================================================= 5 6 # Step 1: Install required packages 7 !pip install numpy 8 9 # Step 2: Import libraries 10 import numpy as np 11 12 # Step 3: Run Demonstration 13 # Position x along 1D line from 0 to 10 (Goal is 10) 14 def potential(x): 15 return x # Closer to 10 has higher potential 16 17 gamma = 0.99 18 current_x = 3.0 19 next_x = 4.0 # Step forward 20 21 # Sparse reward: 0 until goal 22 r_sparse = 0.0 23 24 # Potential-based shaped reward: r + gamma * Phi(s') - Phi(s) 25 r_shaped = r_sparse + (gamma * potential(next_x) - potential(current_x)) 26 27 print(f"Raw Sparse Reward: {r_sparse:.2f}") 28 print(f"Shaped Dense Reward: {r_shaped:.2f} (Provides immediate positive gradient!)")
Zero feedback provided for moving from 3 to 4.
Yields +0.96 reward signal, instantly validating the positive step.
Transition from x=3 to x=4
Raw Sparse Reward: 0.00 Shaped Dense Reward: 0.96 (Provides immediate positive gradient!)
Potential-based shaping provides immediate guiding signals while mathematically preserving optimal policy invariance.
Q-learning, Q-table updates, DQN architectures, and exploration-exploitation trade-offs.
Test your understanding with these questions. Remember, this is for learning, not examination!