First time studying this topic?

Launch the interactive walkthrough to discover each component, including the 3D Q-Sphere Lab & Python in Colab.

Topic 3.4

Neural Network Architecture: Layers, Weights & Biases

Understand the anatomy of Multi-Layer Perceptrons (MLP): input layers, hidden layers, output layers, and parameter dimensions.

Learning Objectives

✓Define Input, Hidden, and Output layers
✓Calculate weight tensor shapes and total parameter counts
✓Contrast Dense (Fully Connected) layers with Convolutional and Recurrent layers

📖Let's Begin with a Story

When engineers designed the Boeing 777, they didn't carve it out of a single piece of steel. They assembled it from standardized components: wings, turbines, hydraulic actuators, and avionics computers. A deep neural network is built with the same modular philosophy: - An Input Layer that matches your data dimensions (e.g. 784 pixels or 30 medical measurements) - Multiple Hidden Layers that transform representation space - An Output Layer that formats the final answer The art of deep learning architecture is selecting how many layers to stack, how wide each layer should be, and how information flows through the computational graph.

Why Should You Learn Neural Network Architecture: Layers, Weights & Biases?

🎯 Practical & Engineering Stakes

Architectural intuition

The layout of layers determines a model’s expressive capacity and computational footprint. Designing an architecture that is too small leads to underfitting; making it needlessly huge causes overfitting and explodes training costs.

What Is It?

Simple Explanation

Think of a neural network as an assembly line in a factory: 1. Input Station: Unpacks raw materials (pixels, numbers). 2. Hidden Stations: Workers reshape, combine, and polish the materials into parts. 3. Output Station: Packages the finished product into a final decision.

Technical Explanation

A Multi-Layer Perceptron (MLP) is a directed acyclic graph structured in discrete topological strata. A layer

l

containing

N_l

neurons connected to layer

l-1

containing

N_{l-1}

neurons is parameterized by weight matrix

\mathbf{W}^{(l)} \in \mathbb{R}^{N_l \\imes N_{l-1}}

and bias vector

\mathbf{b}^{(l)} \in \mathbb{R}^{N_l}

, yielding parameter complexity

\sum_{l=1}^L (N_l \cdot N_{l-1} + N_l)

📚 Key Terms

Dense (Fully Connected) Layer

Simple: A layer where every neuron is connected to every neuron in the previous layer.

Technical: A linear mapping

\mathbf{y} = \mathbf{W}\mathbf{x} + \mathbf{b}

where the connectivity graph between adjacent layer sets is a complete bipartite graph.

Width vs Depth

Simple: Width is how many neurons are in a layer; Depth is how many layers are stacked.

Technical: Width denotes layer cardinality

N_l

; Depth denotes graph length

L

of computational path.

Feedforward Network

Simple: Information moves in only one direction—from inputs to outputs—with no loops.

Technical: A network graph with acyclic topology containing no recurrent feedback loops.

📐 Mathematical Foundation

LaTeX KaTeX

\text{Parameters}^{(l)} = (N_{\text{in}} \times N_{\text{out}}) + N_{\text{out}}

What does it mean?

Formula to calculate the total number of learnable parameters (weights + biases) in a fully connected dense layer.

Understanding the Symbols

Symbol / Term	Meaning	Beginner Interpretation
$N_{\text{in}}$	Number of input neurons	Incoming connections
$N_{\text{out}}$	Number of output neurons	Neurons in current layer

📊 Numerical Example

Connecting layer with 100 inputs to a hidden layer of 50 neurons.

\text{Params} = (100 \times 50) + 50 = 5000 + 50 = 5050

Result: 5,050 learnable parameters

⚙️ How Does It Work?

Input Layer Mapping

Dimensionality matches input feature vector length D.

Hidden Layer Transformation

Compresses or expands feature dimensions using non-linear activations.

Output Layer Head

1 neuron for regression/binary classification; C neurons with Softmax for multi-class.

🌍 Where Is It Used?

Tabular Fraud Detection

Problem: Analyzing 50 banking customer features.

Solution: 3-layer MLP [50 -> 64 -> 32 -> 1] achieves high-throughput low-latency inference.

🎯 Try It Yourself

A dense layer receives 10 inputs and produces 5 outputs. How many total learnable parameters (weights + biases) does this layer possess?

Interactive activity component

🐍 Python Hands-On: Inspecting Model Architecture in PyTorch

Print layer-by-layer parameter summaries of a deep architecture.

Open Colab

1Install Required Packages

For Google Colab or Jupyter notebooks, run this shell command to install the required dependencies:

Terminal / Notebook Install Commandbash

!pip install torch

Packages:

torch

2Import Libraries

Import Statementspython

1 import torch.nn as nn

import torch.nn as nn → PyTorch neural network module

3Demonstration Code & Logic

Python Execution Codepython

1 class MultiLayerPerceptron(nn.Module):
2     def __init__(self):
3         super().__init__()
4         self.fc1 = nn.Linear(20, 64) # 20 inputs -> 64 neurons
5         self.relu = nn.ReLU()
6         self.fc2 = nn.Linear(64, 1)  # 64 neurons -> 1 output
7  
8     def forward(self, x):
9         return self.fc2(self.relu(self.fc1(x)))
10  
11 net = MultiLayerPerceptron()
12 print("Network Architecture:\n", net)
13 for name, param in net.named_parameters():
14     print(f"{name}: shape {list(param.shape)} -> {param.numel()} parameters")

Step-by-Step Code Breakdown:

class MultiLayerPerceptron(nn.Module):

Define network class

def __init__(self):

Constructor

super().__init__()

Initialize base module

self.fc1 = nn.Linear(20, 64) # 20 inputs -> 64 neurons

Hidden layer 1

self.relu = nn.ReLU()

Activation

self.fc2 = nn.Linear(64, 1) # 64 neurons -> 1 output

Output layer

def forward(self, x):

Forward computation graph

return self.fc2(self.relu(self.fc1(x)))

Chain layers

net = MultiLayerPerceptron()

Instantiate network

print("Network Architecture:\n", net)

Print structure

for name, param in net.named_parameters():

Loop over layers

print(f"{name}: shape {list(param.shape)} -> {param.numel()} parameters")

Print tensor shapes

📋 Complete Script (All-in-One)

Click "Copy for Colab", open Google Colab, paste into a single cell, and run.

Google Colab Notebook Scriptpython

1 # =========================================================
2 # Inspecting Model Architecture in PyTorch
3 # Prepared for Google Colab (Run with Shift + Enter)
4 # =========================================================
5  
6 # Step 1: Install required packages
7 !pip install torch
8  
9 # Step 2: Import libraries
10 import torch.nn as nn
11  
12 # Step 3: Run Demonstration
13 class MultiLayerPerceptron(nn.Module):
14     def __init__(self):
15         super().__init__()
16         self.fc1 = nn.Linear(20, 64) # 20 inputs -> 64 neurons
17         self.relu = nn.ReLU()
18         self.fc2 = nn.Linear(64, 1)  # 64 neurons -> 1 output
19  
20     def forward(self, x):
21         return self.fc2(self.relu(self.fc1(x)))
22  
23 net = MultiLayerPerceptron()
24 print("Network Architecture:\n", net)
25 for name, param in net.named_parameters():
26     print(f"{name}: shape {list(param.shape)} -> {param.numel()} parameters")

⚡ How to Run in Google Colab (3 Quick Steps)

Click "Copy for Google Colab" above.
Click Open New Google Colab Notebook (opens in a new tab).
Click on the first empty code cell, press Ctrl + V (or Cmd + V) to paste, then press Shift + Enter to execute!

Launch Blank Colab Notebook

🔄 Algorithmic Execution Flow

Layer 1 Parameters

Weights [64, 20] = 1,280 + Biases [64] = 1,344 parameters.

→

Layer 2 Parameters

Weights [1, 64] = 64 + Bias [1] = 65 parameters.

→

Total Architecture

1,409 total parameters ready for gradient descent.

📥 Input / Initial State

20-dimensional input vector

📤 Expected Output

Network Architecture:
MultiLayerPerceptron(
  (fc1): Linear(in_features=20, out_features=64, bias=True)
  (relu): ReLU()
  (fc2): Linear(in_features=64, out_features=1, bias=True)
)
fc1.weight: shape [64, 20] -> 1280 parameters
fc1.bias: shape [64] -> 64 parameters
fc2.weight: shape [1, 64] -> 64 parameters
fc2.bias: shape [1] -> 1 parameters

💡 What Does This Demonstration Show?

PyTorch automatically instantiates weight matrices and bias vectors matching exact input and output dimensions.

Course Learning Resource • Slide DeckLesson 512 Slides

Lesson 5: Classical Deep Learning with PyTorch

CNN, MNIST dataset, backpropagation, loss functions, and optimization.

Author: Dr. Syed Muzamil Basha• FDP Approved Curriculum

Open in New Tab

📝 Learning Check

Test your understanding with these questions. Remember, this is for learning, not examination!

Question 1: What is the function of the bias vector in a neural network layer?

Question 2: What is the difference between shallow and deep neural networks?

Question 3: What is a fully connected (dense) layer?

Question 4: What is the purpose of the output layer?

Question 5: What is ResNet's key innovation?

Previous: Activation Functions: Sigmoid, Tanh, ReLU & Softmax Next: Forward Propagation: The Computational Flow

1	# =========================================================
2	# Inspecting Model Architecture in PyTorch
3	# Prepared for Google Colab (Run with Shift + Enter)
4	# =========================================================
5
6	# Step 1: Install required packages
7	!pip install torch
8
9	# Step 2: Import libraries
10	import torch.nn as nn
11
12	# Step 3: Run Demonstration
13	class MultiLayerPerceptron(nn.Module):
14	def __init__(self):
15	super().__init__()
16	self.fc1 = nn.Linear(20, 64) # 20 inputs -> 64 neurons
17	self.relu = nn.ReLU()
18	self.fc2 = nn.Linear(64, 1) # 64 neurons -> 1 output
19
20	def forward(self, x):
21	return self.fc2(self.relu(self.fc1(x)))
22
23	net = MultiLayerPerceptron()
24	print("Network Architecture:\n", net)
25	for name, param in net.named_parameters():
26	print(f"{name}: shape {list(param.shape)} -> {param.numel()} parameters")