First time studying this topic?

Launch the interactive walkthrough to discover each component, including the 3D Q-Sphere Lab & Python in Colab.

Topic 3.5

Forward Propagation: The Computational Flow

Trace data through the computational graph from input tensors to output activations via vectorized matrix multiplications.

Learning Objectives

✓Trace step-by-step tensor multiplications across successive layers
✓Understand batch processing and batch matrix multiplication
✓Examine how intermediate activations form internal representations

📖Let's Begin with a Story

When you look at a photograph of a red sports car, light bounces off the image and enters your eye. Photons hit your retinal rods and cones, producing electrical impulses. Those impulses flow forward along the optic nerve into the primary visual cortex (V1), passing sequentially to V2, V4, and the inferior temporal cortex. At each biological stage, signals are filtered, integrated, and passed forward. Within 150 milliseconds, your conscious mind perceives "Ferrari." In artificial neural networks, this instantaneous left-to-right cascade of information is called Forward Propagation.

Why Should You Learn Forward Propagation: The Computational Flow?

🎯 Practical & Engineering Stakes

The engine of inference

When you query ChatGPT or unlock your smartphone with FaceID, the model does not train; it executes a single, hyper-optimized forward propagation pass in milliseconds.

What Is It?

Simple Explanation

Forward propagation is the journey data takes through the network: 1. Inputs arrive at Layer 1. 2. Layer 1 multiplies by weights, adds bias, applies ReLU, and hands the result to Layer 2. 3. Layer 2 repeats the process and hands to Layer 3. 4. Output layer gives the final prediction!

Technical Explanation

Forward propagation evaluates the composite computational graph

\mathbf{a}^{(0)} = \mathbf{X}

\mathbf{z}^{(l)} = \mathbf{a}^{(l-1)} \mathbf{W}^{(l)} + \mathbf{b}^{(l)}

\mathbf{a}^{(l)} = \sigma(\mathbf{z}^{(l)})

for

l = 1, \dots, L

. When executed over mini-batches of size

B

, operations leverage Level 3 BLAS general matrix multiplications (GEMM)

\mathbf{Z} \in \mathbb{R}^{B \\imes N_l}

📚 Key Terms

Forward Pass

Simple: Running data forward through the network to generate predictions.

Technical: Evaluation of the sequence of nested non-linear matrix operations from input to objective loss.

Mini-Batch

Simple: Processing a group of examples (e.g., 32 or 64) all at once for speed.

Technical: A sub-sample of

B

training instances packed into a 2D/3D tensor for parallelized GPU matrix operations.

$Logits$

Simple: The raw, unnormalized prediction scores before applying Softmax.

Technical: The unconstrained real-valued outputs

\mathbf{z}^{(L)}

produced by the final linear layer prior to probability calibration.

📐 Mathematical Foundation

LaTeX KaTeX

\mathbf{Z}^{(l)} = \mathbf{A}^{(l-1)} \mathbf{W}^{(l)} + \mathbf{b}^{(l)}, \quad \mathbf{A}^{(l)} = \sigma\big(\mathbf{Z}^{(l)}\big)

What does it mean?

Mini-batch forward propagation equations: multiplying activation matrix A by weight matrix W and applying activation function sigma.

Understanding the Symbols

Symbol / Term	Meaning	Beginner Interpretation
$\mathbf{A}^{(l-1)}$	Activations of previous layer (Batch x In)	Features from earlier stage
$\mathbf{W}^{(l)}$	Weights (In x Out)	Transformation matrix
$\mathbf{b}^{(l)}$	Broadcasted bias vector	Row-wise added offset

📊 Numerical Example

Batch of 2 samples passed through layer: A = [[1, 2], [3, 4]], W = [[1], [1]], b = 0.

Z = [[1(1) + 2(1)], [3(1) + 4(1)]] = [[3], [7]]

Result: Output activations: [[3], [7]]

⚙️ How Does It Work?

Batch Vectorization

Pack input instances into tensor of shape (Batch Size, Features).

Linear Combination (GEMM)

Compute GPU-accelerated matrix product Z = X * W + b.

Non-linear Activation

Apply element-wise activation function A = ReLU(Z).

Repeat until Output

Chain intermediate activations until output head produces logits.

🌍 Where Is It Used?

Real-Time Edge Inference

Problem: Running facial recognition on a battery-powered security camera in 10 milliseconds.

Solution: Quantized INT8 forward propagation executes fast matrix multiplications without backpropagation.

🎯 Try It Yourself

If your input batch has shape (32, 100) and the weight matrix has shape (100, 50), what is the shape of the output matrix Z?

Interactive activity component

🐍 Python Hands-On: Manual Forward Pass in Pure NumPy

Implement a 2-layer forward propagation pass by hand using matrix operations.

Open Colab

1Install Required Packages

For Google Colab or Jupyter notebooks, run this shell command to install the required dependencies:

Terminal / Notebook Install Commandbash

!pip install numpy

Packages:

numpy

2Import Libraries

Import Statementspython

1 import numpy as np

import numpy as np → NumPy for matrix operations

3Demonstration Code & Logic

Python Execution Codepython

1 # Input: Batch of 2 samples, 3 features each (shape: 2x3)
2 X = np.array([[1.0, 2.0, 3.0], [0.5, -1.0, 2.0]])
3  
4 # Layer 1 Weights (3x4) and Biases (4,)
5 W1 = np.array([[0.2, -0.1, 0.5, 0.1], [0.4, 0.3, -0.2, 0.5], [-0.1, 0.2, 0.1, -0.3]])
6 b1 = np.array([0.1, 0.1, 0.0, -0.1])
7  
8 # Layer 1 Forward Pass: Z1 = X @ W1 + b1, A1 = ReLU(Z1)
9 Z1 = np.dot(X, W1) + b1
10 A1 = np.maximum(0, Z1) # ReLU
11 print("Layer 1 Activations (shape 2x4):\n", A1.round(3))
12  
13 # Layer 2 (Output Head): Weights (4x1) and Bias (1,)
14 W2 = np.array([[0.5], [-0.3], [0.8], [0.2]])
15 b2 = np.array([0.0])
16 Z2 = np.dot(A1, W2) + b2
17 print("\nFinal Output Predictions (shape 2x1):\n", Z2.round(3))

Step-by-Step Code Breakdown:

# Input: Batch of 2 samples, 3 features each (shape: 2x3)

Input batch

X = np.array([[1.0, 2.0, 3.0], [0.5, -1.0, 2.0]])

2 instances

# Layer 1 Weights (3x4) and Biases (4,)

Layer 1

W1 = np.array([[0.2, -0.1, 0.5, 0.1], [0.4, 0.3, -0.2, 0.5], [-0.1, 0.2, 0.1, -0.3]])

W1 matrix

b1 = np.array([0.1, 0.1, 0.0, -0.1])

b1 bias

# Layer 1 Forward Pass: Z1 = X @ W1 + b1, A1 = ReLU(Z1)

Layer 1 computation

Z1 = np.dot(X, W1) + b1

Linear combination

A1 = np.maximum(0, Z1) # ReLU

Activation

print("Layer 1 Activations (shape 2x4):\n", A1.round(3))

Show activations

# Layer 2 (Output Head): Weights (4x1) and Bias (1,)

Layer 2

W2 = np.array([[0.5], [-0.3], [0.8], [0.2]])

W2 matrix

b2 = np.array([0.0])

b2 bias

Z2 = np.dot(A1, W2) + b2

Final linear score

print("\nFinal Output Predictions (shape 2x1):\n", Z2.round(3))

Print predictions

📋 Complete Script (All-in-One)

Click "Copy for Colab", open Google Colab, paste into a single cell, and run.

Google Colab Notebook Scriptpython

1 # =========================================================
2 # Manual Forward Pass in Pure NumPy
3 # Prepared for Google Colab (Run with Shift + Enter)
4 # =========================================================
5  
6 # Step 1: Install required packages
7 !pip install numpy
8  
9 # Step 2: Import libraries
10 import numpy as np
11  
12 # Step 3: Run Demonstration
13 # Input: Batch of 2 samples, 3 features each (shape: 2x3)
14 X = np.array([[1.0, 2.0, 3.0], [0.5, -1.0, 2.0]])
15  
16 # Layer 1 Weights (3x4) and Biases (4,)
17 W1 = np.array([[0.2, -0.1, 0.5, 0.1], [0.4, 0.3, -0.2, 0.5], [-0.1, 0.2, 0.1, -0.3]])
18 b1 = np.array([0.1, 0.1, 0.0, -0.1])
19  
20 # Layer 1 Forward Pass: Z1 = X @ W1 + b1, A1 = ReLU(Z1)
21 Z1 = np.dot(X, W1) + b1
22 A1 = np.maximum(0, Z1) # ReLU
23 print("Layer 1 Activations (shape 2x4):\n", A1.round(3))
24  
25 # Layer 2 (Output Head): Weights (4x1) and Bias (1,)
26 W2 = np.array([[0.5], [-0.3], [0.8], [0.2]])
27 b2 = np.array([0.0])
28 Z2 = np.dot(A1, W2) + b2
29 print("\nFinal Output Predictions (shape 2x1):\n", Z2.round(3))

⚡ How to Run in Google Colab (3 Quick Steps)

Click "Copy for Google Colab" above.
Click Open New Google Colab Notebook (opens in a new tab).
Click on the first empty code cell, press Ctrl + V (or Cmd + V) to paste, then press Shift + Enter to execute!

Launch Blank Colab Notebook

🔄 Algorithmic Execution Flow

GEMM 1

Maps batch of (2, 3) through W1 (3, 4) into hidden space (2, 4).

→

Non-Linearity

ReLU clamps negative values to 0.0.

→

GEMM 2

Projects (2, 4) through W2 (4, 1) to produce scalar outputs for each instance in the batch.

📥 Input / Initial State

Batch of 2 samples with 3 features

📤 Expected Output

Layer 1 Activations (shape 2x4):
[[0.8 1.2 0.4 0.1]
 [0.0 0.2 0.65 0.  ]]

Final Output Predictions (shape 2x1):
[[0.38 ]
 [0.46 ]]

💡 What Does This Demonstration Show?

Vectorized forward propagation handles entire batches in parallel without any Python for-loops.

Course Learning Resource • Slide DeckLesson 512 Slides

Lesson 5: Classical Deep Learning with PyTorch

CNN, MNIST dataset, backpropagation, loss functions, and optimization.

Author: Dr. Syed Muzamil Basha• FDP Approved Curriculum

Open in New Tab

📝 Learning Check

Test your understanding with these questions. Remember, this is for learning, not examination!

Question 1: During forward propagation, why do deep learning frameworks process data in mini-batches (e.g. 32 samples) rather than one sample at a time?

Question 2: What is forward propagation?

Question 3: What is the formula for a single neuron's output?

Question 4: How are layers chained in forward propagation?

Question 5: What is the difference between forward and backward propagation?

Previous: Neural Network Architecture: Layers, Weights & Biases Next: Loss Functions: Cross-Entropy & Mean Squared Error

1	# =========================================================
2	# Manual Forward Pass in Pure NumPy
3	# Prepared for Google Colab (Run with Shift + Enter)
4	# =========================================================
5
6	# Step 1: Install required packages
7	!pip install numpy
8
9	# Step 2: Import libraries
10	import numpy as np
11
12	# Step 3: Run Demonstration
13	# Input: Batch of 2 samples, 3 features each (shape: 2x3)
14	X = np.array([[1.0, 2.0, 3.0], [0.5, -1.0, 2.0]])
15
16	# Layer 1 Weights (3x4) and Biases (4,)
17	W1 = np.array([[0.2, -0.1, 0.5, 0.1], [0.4, 0.3, -0.2, 0.5], [-0.1, 0.2, 0.1, -0.3]])
18	b1 = np.array([0.1, 0.1, 0.0, -0.1])
19
20	# Layer 1 Forward Pass: Z1 = X @ W1 + b1, A1 = ReLU(Z1)
21	Z1 = np.dot(X, W1) + b1
22	A1 = np.maximum(0, Z1) # ReLU
23	print("Layer 1 Activations (shape 2x4):\n", A1.round(3))
24
25	# Layer 2 (Output Head): Weights (4x1) and Bias (1,)
26	W2 = np.array([[0.5], [-0.3], [0.8], [0.2]])
27	b2 = np.array([0.0])
28	Z2 = np.dot(A1, W2) + b2
29	print("\nFinal Output Predictions (shape 2x1):\n", Z2.round(3))