First time studying this topic?

Launch the interactive walkthrough to discover each component, including the 3D Q-Sphere Lab & Python in Colab.

Topic 3.6

Loss Functions: Cross-Entropy & Mean Squared Error

Understand how mathematical loss functions measure prediction error and guide neural network optimization.

Learning Objectives

✓Differentiate between Mean Squared Error (MSE) and Cross-Entropy Loss
✓Understand the information-theoretic meaning of Cross-Entropy and Surprisal
✓Examine Binary Cross-Entropy (BCE) vs Categorical Cross-Entropy (CCE)

📖Let's Begin with a Story

Imagine coaching an archer. If the archer is practicing target shooting where the arrow lands 10 centimeters from the bullseye, you measure the physical Euclidean distance. The error is a continuous distance in centimeters. This is Mean Squared Error. Now imagine a meteorologist who announces: "There is a 99% chance of a bright sunny day today." An hour later, a ferocious blizzard blankets the city! The meteorologist wasn't just wrong; they were supremely confident and wrong. In probability theory, that massive error is called Surprisal. Cross-Entropy Loss heavily penalizes models that are confidently wrong, driving them to calibrate their probabilities honestly.

Why Should You Learn Loss Functions: Cross-Entropy & Mean Squared Error?

🎯 Practical & Engineering Stakes

The compass of learning

The network never sees your accuracy score; it only sees the loss gradient. Choosing the correct loss function is critical to prevent training stagnation and numerical divergence.

What Is It?

Simple Explanation

A loss function is the teacher grading the network’s exam: - Mean Squared Error (MSE): Used when predicting numbers (like house prices or temperatures). Measures how far off the guess was. - Cross-Entropy Loss: Used when predicting categories (like cat vs dog). It measures how surprised or confident the model was in its choice.

Technical Explanation

In maximum likelihood estimation, minimizing Cross-Entropy is equivalent to minimizing the Kullback-Leibler (KL) divergence

D_{KL}(P \parallel Q)

between the empirical data distribution

P

and model distribution

Q_\\heta

. For discrete multi-class classification, Categorical Cross-Entropy is

\mathcal{L} = -\sum_{c=1}^C y_c \log(\hat{y}_c)

📚 Key Terms

Categorical Cross-Entropy

Simple: The standard loss for multi-class problems; penalizes wrong probability guesses.

Technical: Negative log-likelihood of the true class under the model distribution:

\mathcal{L} = -\log \hat{y}_{true}

Binary Cross-Entropy (Log Loss)

Simple: Cross-entropy specialized for two-choice (binary) problems.

Technical:

\mathcal{L} = -[y \log \hat{y} + (1-y) \log(1-\hat{y})]

Mean Squared Error (MSE)

Simple: The average squared distance between true and predicted numbers.

Technical:

\frac{1}{N} \sum (y - \hat{y})^2

, optimal for targets with Gaussian noise distributions.

📐 Mathematical Foundation

LaTeX KaTeX

\mathcal{L}_{BCE} = -\frac{1}{N} \sum_{i=1}^N \Big[ y_i \ln(\hat{y}_i) + (1 - y_i) \ln(1 - \hat{y}_i) \Big]

What does it mean?

Binary Cross-Entropy Loss: penalizes discrepancies between true binary label y (0 or 1) and predicted probability y_hat.

Understanding the Symbols

Symbol / Term	Meaning	Beginner Interpretation
$y_i$	Ground truth (0 or 1)	Actual class label
$\hat{y}_i$	Predicted probability	Model confidence between 0 and 1

📊 Numerical Example

True label y=1. If model predicts high confidence 0.90, loss is small: -ln(0.90) = 0.105. If model confidently predicts 0.01, loss explodes: -ln(0.01) = 4.605!

\text{Loss}_{\text{good}} = -\ln(0.90) = 0.105, \quad \text{Loss}_{\text{bad}} = -\ln(0.01) = 4.605

Result: Confident errors are penalized 44x more severely

⚙️ How Does It Work?

Output Generation

Network outputs continuous probabilities via Sigmoid or Softmax.

Log-Likelihood Calculation

Evaluate negative log probability assigned to the ground-truth class.

Batch Averaging

Average loss scalars across the mini-batch to form scalar objective.

🌍 Where Is It Used?

Image Classification (ImageNet)

Problem: Evaluating classification across 1,000 competing categories.

Solution: Categorical Cross-Entropy computes loss based solely on the probability assigned to the correct category.

Autonomous Vehicle Steering Control

Problem: Predicting continuous steering wheel steering angle in degrees.

Solution: MSE loss penalizes angular deviations from human driver demonstrations.

🎯 Try It Yourself

You are training a neural network to predict the exact price of Bitcoin in dollars. Which loss function is appropriate?

Interactive activity component

🐍 Python Hands-On: Computing Loss Functions in PyTorch

Calculate MSE and Cross-Entropy loss using built-in PyTorch loss functions.

Open Colab

1Install Required Packages

For Google Colab or Jupyter notebooks, run this shell command to install the required dependencies:

Terminal / Notebook Install Commandbash

!pip install torch

Packages:

torch

2Import Libraries

Import Statementspython

1 import torch
2 import torch.nn as nn

import torch → PyTorch library

import torch.nn as nn → Neural network loss modules

3Demonstration Code & Logic

Python Execution Codepython

1 # 1. Mean Squared Error (Regression Example):
2 mse_loss = nn.MSELoss()
3 y_pred = torch.tensor([10.5, 20.0, 31.0])
4 y_true = torch.tensor([10.0, 22.0, 30.0])
5 loss_reg = mse_loss(y_pred, y_true)
6 print(f"MSE Loss: {loss_reg.item():.4f}")
7  
8 # 2. Cross-Entropy Loss (Classification Example):
9 ce_loss = nn.CrossEntropyLoss() # Combines Softmax + NLLLoss internally
10 logits = torch.tensor([[2.0, 0.5, 0.1]]) # Unnormalized logits for 3 classes
11 target_class = torch.tensor([0])         # True answer is Class 0
12 loss_clf = ce_loss(logits, target_class)
13 print(f"Cross-Entropy Loss: {loss_clf.item():.4f}")

Step-by-Step Code Breakdown:

# 1. Mean Squared Error (Regression Example):

Regression loss

mse_loss = nn.MSELoss()

MSE loss criterion

y_pred = torch.tensor([10.5, 20.0, 31.0])

Continuous predictions

y_true = torch.tensor([10.0, 22.0, 30.0])

Ground truth targets

loss_reg = mse_loss(y_pred, y_true)

Compute MSE

print(f"MSE Loss: {loss_reg.item():.4f}")

Print MSE

# 2. Cross-Entropy Loss (Classification Example):

Classification loss

ce_loss = nn.CrossEntropyLoss() # Combines Softmax + NLLLoss internally

Cross-entropy criterion

logits = torch.tensor([[2.0, 0.5, 0.1]]) # Unnormalized logits for 3 classes

Model raw scores

target_class = torch.tensor([0]) # True answer is Class 0

Correct class index

loss_clf = ce_loss(logits, target_class)

Compute Cross-Entropy

print(f"Cross-Entropy Loss: {loss_clf.item():.4f}")

Print CE loss

📋 Complete Script (All-in-One)

Click "Copy for Colab", open Google Colab, paste into a single cell, and run.

Google Colab Notebook Scriptpython

1 # =========================================================
2 # Computing Loss Functions in PyTorch
3 # Prepared for Google Colab (Run with Shift + Enter)
4 # =========================================================
5  
6 # Step 1: Install required packages
7 !pip install torch
8  
9 # Step 2: Import libraries
10 import torch
11 import torch.nn as nn
12  
13 # Step 3: Run Demonstration
14 # 1. Mean Squared Error (Regression Example):
15 mse_loss = nn.MSELoss()
16 y_pred = torch.tensor([10.5, 20.0, 31.0])
17 y_true = torch.tensor([10.0, 22.0, 30.0])
18 loss_reg = mse_loss(y_pred, y_true)
19 print(f"MSE Loss: {loss_reg.item():.4f}")
20  
21 # 2. Cross-Entropy Loss (Classification Example):
22 ce_loss = nn.CrossEntropyLoss() # Combines Softmax + NLLLoss internally
23 logits = torch.tensor([[2.0, 0.5, 0.1]]) # Unnormalized logits for 3 classes
24 target_class = torch.tensor([0])         # True answer is Class 0
25 loss_clf = ce_loss(logits, target_class)
26 print(f"Cross-Entropy Loss: {loss_clf.item():.4f}")

⚡ How to Run in Google Colab (3 Quick Steps)

Click "Copy for Google Colab" above.
Click Open New Google Colab Notebook (opens in a new tab).
Click on the first empty code cell, press Ctrl + V (or Cmd + V) to paste, then press Shift + Enter to execute!

Launch Blank Colab Notebook

🔄 Algorithmic Execution Flow

MSE Computation

Computes [(10.5-10)^2 + (20-22)^2 + (31-30)^2] / 3 = [0.25 + 4.0 + 1.0] / 3 = 1.75.

→

Cross-Entropy Computation

Applies internal Softmax and computes negative log-likelihood of class 0 (~0.28).

📥 Input / Initial State

Regression targets and classification logits

📤 Expected Output

MSE Loss: 1.7500
Cross-Entropy Loss: 0.2800

💡 What Does This Demonstration Show?

PyTorch loss modules evaluate numerical errors efficiently on GPU/CPU tensors with full autograd tracking.

Course Learning Resource • Slide DeckLesson 512 Slides

Lesson 5: Classical Deep Learning with PyTorch

CNN, MNIST dataset, backpropagation, loss functions, and optimization.

Author: Dr. Syed Muzamil Basha• FDP Approved Curriculum

Open in New Tab

📝 Learning Check

Test your understanding with these questions. Remember, this is for learning, not examination!

Question 1: What happens to Cross-Entropy loss when the model predicts a probability near 0.001 for the class that is actually true?

Question 2: What is Mean Squared Error (MSE) used for?

Question 3: What is Cross-Entropy loss used for?

Question 4: Why do we need loss functions?

Question 5: What is regularization in the loss function?

Previous: Forward Propagation: The Computational Flow Next: Gradient Descent & Backpropagation

1	# =========================================================
2	# Computing Loss Functions in PyTorch
3	# Prepared for Google Colab (Run with Shift + Enter)
4	# =========================================================
5
6	# Step 1: Install required packages
7	!pip install torch
8
9	# Step 2: Import libraries
10	import torch
11	import torch.nn as nn
12
13	# Step 3: Run Demonstration
14	# 1. Mean Squared Error (Regression Example):
15	mse_loss = nn.MSELoss()
16	y_pred = torch.tensor([10.5, 20.0, 31.0])
17	y_true = torch.tensor([10.0, 22.0, 30.0])
18	loss_reg = mse_loss(y_pred, y_true)
19	print(f"MSE Loss: {loss_reg.item():.4f}")
20
21	# 2. Cross-Entropy Loss (Classification Example):
22	ce_loss = nn.CrossEntropyLoss() # Combines Softmax + NLLLoss internally
23	logits = torch.tensor([[2.0, 0.5, 0.1]]) # Unnormalized logits for 3 classes
24	target_class = torch.tensor([0]) # True answer is Class 0
25	loss_clf = ce_loss(logits, target_class)
26	print(f"Cross-Entropy Loss: {loss_clf.item():.4f}")