Q-Learning

import numpy as np → NumPy provides numerical array operations for our Q-table and calculations.

import matplotlib.pyplot as plt → Matplotlib helps us visualize the learning progress and Q-values.

# ─── SECTION 1: ENVIRONMENT SETUP ───────────────────────────────────────

📌 Section 1 begins here. We first describe the world the agent lives in — a 5×5 grid with 25 cells. This is the "environment" in reinforcement learning.

# Define the environment

💬 Comment: This block sets up the rules of the grid world — its size, where the agent starts, and where the goal is.

GRID_SIZE = 5

📐 Sets the grid to 5 columns × 5 rows = 25 total cells. States are numbered 0 to 24 (top-left = 0, bottom-right = 24). UPPER_CASE naming signals this is a constant that should not change during the run.

START_STATE = 0

🚀 The agent always begins at state 0 (top-left corner, row=0, col=0). This is reset at the start of every episode so the agent re-learns from the same starting point each time.

GOAL_STATE = 24

🎯 The destination is state 24 (bottom-right corner, row=4, col=4). When the agent reaches this state, the episode ends with a large reward. The entire purpose of training is to learn the fastest path from state 0 → state 24.

# ─── SECTION 2: HYPERPARAMETERS ────────────────────────────────────────────

📌 Section 2: Hyperparameters are knobs you tune to control how the agent learns. These are NOT learned — you choose them before training.

# Q-learning parameters

💬 Comment: These four numbers control the speed, depth, curiosity, and duration of learning.

LEARNING_RATE = 0.1

📈 Alpha (α) = 0.1. This controls HOW FAST the agent updates its Q-values when it learns something new. Think of it as "how seriously do I take new information?" • α = 0.1 → 10% of new info replaces old belief each step (stable, gradual learning) • α = 1.0 → completely replaces old belief (too fast, unstable) • α = 0.0 → agent never learns at all ✅ 0.1 is a safe, standard choice for small environments.

DISCOUNT_FACTOR = 0.9

⏳ Gamma (γ) = 0.9. Controls HOW MUCH the agent values FUTURE rewards versus IMMEDIATE rewards. • γ = 0.9 → future reward is worth 90% of an equal immediate reward (forward-thinking agent) • γ = 0.0 → agent is completely greedy, only cares about the next immediate reward • γ = 1.0 → agent values all future rewards equally (can cause instability) ✅ 0.9 makes the agent plan ahead — essential for navigating a grid where the goal is many steps away.

EPSILON = 0.1

🎲 Epsilon (ε) = 0.1. Controls the EXPLORATION vs EXPLOITATION trade-off. • With 10% probability, the agent tries a RANDOM action (explore — discover new paths) • With 90% probability, the agent picks the BEST known action (exploit — use what it learned) ✅ This prevents the agent from getting stuck always doing the same thing. Without exploration, it might never find the optimal path.

EPISODES = 500

🔁 The agent will train for 500 complete episodes. One episode = one attempt to go from START → GOAL (or max 100 steps). More episodes = more practice = better Q-values. 500 is enough to converge on a good policy for this small 5×5 grid.

# ─── SECTION 3: Q-TABLE INITIALIZATION ─────────────────────────────────────

📌 Section 3: The Q-table is the agent's "brain" — a memory of how good each action is in each state.

# Initialize Q-table

💬 Comment: Before training, the agent knows nothing, so all Q-values start at zero.

num_states = GRID_SIZE * GRID_SIZE

🧮 Calculates total number of states: 5 × 5 = 25. Each cell in the grid is one state. This tells us how many rows the Q-table needs.

num_actions = 4 # up, down, left, right

🧭 The agent has 4 possible moves at every state: • Action 0 = UP (move to the row above) • Action 1 = DOWN (move to the row below) • Action 2 = LEFT (move to the column left) • Action 3 = RIGHT (move to the column right) The comment reminds us of this mapping. This tells us how many columns the Q-table needs.

Q = np.zeros((num_states, num_actions))

🗃️ Creates the Q-table: a 25×4 matrix (25 states × 4 actions), filled entirely with zeros. • Each row = one state (a grid cell) • Each column = one action (up/down/left/right) • Each cell Q[s, a] = estimated quality of taking action a in state s → At start, Q[0,0]=0, Q[0,1]=0, Q[0,2]=0, Q[0,3]=0 ... all zeros → After training, Q[0,3] might be large because "go right" from start is a good idea.

# ─── SECTION 4: HELPER FUNCTIONS ───────────────────────────────────────────

📌 Section 4: Two small utility functions that the training loop will call thousands of times.

# Helper function: get next state

💬 Comment: This function simulates the physics of the grid — what happens when the agent moves.

def get_next_state(state, action):

🔧 Defines a function that takes the current state (0–24) and chosen action (0–3), and returns the resulting next state. It handles wall collisions — if the agent tries to move off the grid, it stays in place.

row, col = state // GRID_SIZE, state % GRID_SIZE

🗺️ Converts the flat state number into 2D grid coordinates: • row = state // 5 (integer division: state 7 → row 1) • col = state % 5 (remainder: state 7 → col 2) Example: state 12 → row=2, col=2 (center of the grid) This lets us check if moving would go out of bounds.

if action == 0 and row > 0: row -= 1 # up

⬆️ Action 0 = UP. Subtracts 1 from the row (moves toward the top). The guard "row > 0" ensures the agent doesn't go above the top edge. If at row=0 and tries to go up → stays at row=0 (wall collision).

elif action == 1 and row < GRID_SIZE-1: row += 1 # down

⬇️ Action 1 = DOWN. Adds 1 to the row (moves toward the bottom). Guard "row < 4" prevents going below the last row. If at row=4 (bottom edge) and tries to go down → stays at row=4.

elif action == 2 and col > 0: col -= 1 # left

⬅️ Action 2 = LEFT. Subtracts 1 from the column (moves toward the left). Guard "col > 0" prevents going off the left edge. If at col=0 (left edge) and tries to go left → stays at col=0.

elif action == 3 and col < GRID_SIZE-1: col += 1 # right

➡️ Action 3 = RIGHT. Adds 1 to the column (moves toward the right). Guard "col < 4" prevents going off the right edge. If at col=4 (right edge) and tries to go right → stays at col=4.

return row * GRID_SIZE + col

🔄 Converts 2D coordinates back to a flat state number: next_state = row × 5 + col Example: row=2, col=3 → state = 2×5+3 = 13 This reverses the earlier decomposition and gives a single integer the Q-table can use as an index.

# Helper function: get reward

💬 Comment: Defines the reward signal — the feedback the agent receives after each action.

def get_reward(state):

🏆 Defines a reward function. Takes the state the agent just moved into and returns a number: • Positive number = good outcome • Negative number = bad/costly outcome This is how the environment "tells" the agent whether it did something worthwhile.

if state == GOAL_STATE: return 100

✅ If the agent reaches state 24 (the goal), it gets a reward of +100. This large positive signal teaches the agent that reaching the goal is highly desirable. The agent will update Q-values to make goal-reaching actions much more attractive.

else: return -1

⏱️ Every step that is NOT the goal gives a reward of -1. This "step penalty" encourages efficiency — the agent is penalized for wasting time. Without this, the agent might wander aimlessly and still get the +100 eventually. With -1 per step, the agent learns to reach the goal in as few moves as possible.

# ─── SECTION 5: TRAINING LOOP ──────────────────────────────────────────────

📌 Section 5: The core of Q-learning — repeatedly running episodes and updating the Q-table after every step.

# Training loop

💬 Comment: This is where the actual learning happens over 500 episodes.

rewards_per_episode = []

📊 Creates an empty Python list to record the total reward earned in each episode. After training, we'll plot this list to see how the agent improved over time. If rewards increase episode-by-episode, the agent is learning!

for episode in range(EPISODES):

🔁 Outer loop: repeats 500 times (episode = 0, 1, 2, ... 499). Each iteration is one complete training attempt — the agent starts fresh at state 0 and tries to reach state 24.

state = START_STATE

📍 Resets the agent's position to state 0 (top-left corner) at the start of every episode. This ensures each episode is a fresh attempt, just like a player starting a new game from the beginning.

total_reward = 0

💰 Initializes a running counter for the total reward accumulated during this episode. Starts at 0 and we add to it after each step. At the end of the episode, this tells us how well the agent performed.

steps = 0

👟 Counter for how many steps taken in this episode. Used to enforce the max-100-step limit. Prevents infinite loops in case the agent gets stuck in a cycle without reaching the goal.

while state != GOAL_STATE and steps < 100:

🔄 Inner loop: runs until EITHER the goal is reached (state == 24) OR 100 steps have been taken. This inner loop is one episode's decision-action-learn cycle. The 100-step cap prevents the agent from running forever early in training when it's mostly lost.

# Choose action (epsilon-greedy)

💬 Comment: The ε-greedy strategy decides whether to explore or exploit on each step.

if np.random.random() < EPSILON:

🎲 np.random.random() generates a float between 0.0 and 1.0 uniformly at random. If it falls below EPSILON (0.1), we explore. Since EPSILON=0.1, this happens ~10% of the time. The other 90% we exploit what we already know.

action = np.random.randint(0, num_actions)

🃏 EXPLORE: picks a random action integer from {0, 1, 2, 3} with equal probability. This is the exploration step — the agent tries something it might not normally choose, which might reveal a better path it didn't know about.

else:

🧠 EXPLOIT: we skip the random choice and use what the agent has learned.

action = np.argmax(Q[state])

🏹 Q[state] is a 1D array of 4 Q-values (one per action) for the current state. np.argmax() returns the index of the LARGEST value — the action the agent currently believes is best. Example: if Q[5] = [2.1, 4.7, 1.3, 3.9], then argmax = 1 (DOWN has the highest Q-value → take action DOWN).

# Take action

💬 Comment: Execute the chosen action and observe what happens in the environment.

next_state = get_next_state(state, action)

🌍 Calls our helper function to simulate the action: given the current state and chosen action, what cell does the agent land in? The result is the "next_state" — the agent's new position on the grid.

reward = get_reward(next_state)

🏅 Asks the environment: "What reward do I get for being in next_state?" Returns either +100 (goal reached!) or -1 (any other cell). This reward will be used to update the Q-value for the action just taken.

# Q-learning update

💬 Comment: This is the heart of the algorithm — updating the Q-table using the Bellman equation.

best_next_q = np.max(Q[next_state])

🔭 Looks ahead: "What is the BEST Q-value I could get from the next_state, regardless of which action I would take?" np.max(Q[next_state]) scans all 4 action Q-values in the next state and returns the maximum. This is the max Q(s',a') term from the Bellman equation.

Q[state, action] = Q[state, action] + LEARNING_RATE * (reward + DISCOUNT_FACTOR * best_next_q - Q[state, action])

⚡ THE BELLMAN UPDATE — the single most important line in Q-learning! Breaking it down step-by-step: ① reward + DISCOUNT_FACTOR * best_next_q → "What is this action actually worth?" = immediate reward + (0.9 × best future value) → This is called the TD Target (Temporal Difference Target) ② TD Target - Q[state, action] → "How surprised are we?" = difference between what we expected vs what actually happened → This is called the TD Error ③ LEARNING_RATE * TD_Error → "How much do we adjust?" = 0.1 × TD Error (small, gradual adjustment) ④ Q[state, action] = Q[state, action] + adjustment → Update the Q-value: nudge it toward reality If TD Error is positive → we underestimated, Q goes UP If TD Error is negative → we overestimated, Q goes DOWN If TD Error is zero → our estimate was perfect, Q stays SAME

state = next_state

📍 The agent "moves" — updates its current state to the new state. Now the agent is at the cell it just moved to. The next iteration of the while loop will start decision-making from this new position.

total_reward += reward

💰 Adds the step's reward to the episode's running total. After the episode ends, this sum tells us how well the agent did overall. Early episodes: very negative (many -1 steps, rarely reaches +100). Late episodes: near +100 - (few steps × 1) = efficient paths.

steps += 1

👟 Increments the step counter by 1 so the while-loop condition (steps < 100) can eventually stop an episode where the agent is wandering. Without this counter, a lost agent would loop forever.

rewards_per_episode.append(total_reward)

📈 At the END of each episode, save the total reward to the list. After 500 episodes, this list has 500 numbers. Plotting them reveals the learning curve: should trend upward from very negative (lost/wandering) to near positive (efficient goal-reaching).

# ─── SECTION 6: VISUALIZATION ──────────────────────────────────────────────

📌 Section 6: Plot the learning curve to visually confirm the agent improved over time.

# Visualize learning progress

💬 Comment: Create a graph showing how the agent's total reward per episode changed over training.

plt.figure(figsize=(10, 5))

🖼️ Creates a new matplotlib figure (plot canvas) with width=10 inches and height=5 inches. Always call this before plt.plot() to control the output size. A wider figure makes the learning curve easier to read.

plt.plot(rewards_per_episode)

📉📈 Plots the 500 episode rewards as a line graph: • X-axis = episode number (0 to 499) • Y-axis = total reward that episode • You should see the line start very low (negative, many wasted steps) and gradually climb higher (agent finds shorter paths and collects less penalties).

plt.xlabel('Episode')

🏷️ Labels the X-axis "Episode" so readers know each point on the x-axis represents one complete training attempt.

plt.ylabel('Total Reward')

🏷️ Labels the Y-axis "Total Reward" — the sum of all rewards in an episode. Negative values = agent was slow. Values close to +100 = agent found the goal quickly with few wasted steps.

plt.title('Q-Learning Progress: Reward per Episode')

📋 Adds a descriptive title to the chart. This makes the plot self-explanatory when shared or included in a report.

plt.grid(True)

📏 Adds a light grey grid to the plot background. Grids make it much easier to read values off the chart and spot trends, especially whether the curve is flat or rising.

plt.show()

👁️ Renders and displays the finished plot. In Google Colab, this appears inline below the cell. In a local script, it opens a pop-up window. Without plt.show(), nothing appears on screen.

print(f"Training completed over {EPISODES} episodes")

📢 Prints a confirmation message after all training is done. The f-string inserts the value of EPISODES (500) into the string automatically → "Training completed over 500 episodes".

print(f"Final average reward (last 100 episodes): {np.mean(rewards_per_episode[-100:]):.2f}")

📊 Evaluates and prints the agent's final performance: • rewards_per_episode[-100:] → slices the LAST 100 values from the list • np.mean(...) → computes the average of those 100 values • :.2f → formats the float to 2 decimal places Why the last 100 episodes? Because early training is chaotic — the agent is still learning. The last 100 episodes represent the agent after it has mostly converged to a good policy. A high average here confirms successful learning.