First time studying this topic?

Launch the interactive walkthrough to discover each component, including the 3D Q-Sphere Lab & Python in Colab.

Topic 3.3

Activation Functions: Sigmoid, Tanh, ReLU & Softmax

Master the mathematical non-linearities that allow neural networks to bend space: Sigmoid, Tanh, ReLU, Leaky ReLU, and Softmax.

Learning Objectives

✓Understand why non-linear activations are mathematically necessary in deep networks
✓Diagnose the Vanishing Gradient problem in Sigmoid and Tanh
✓Examine why Rectified Linear Units (ReLU) became the dominant standard
✓Implement Softmax for multi-class probability normalization

📖Let's Begin with a Story

In the early 1990s, training deep neural networks with 5 or more layers was nearly impossible. Gradients would shrink exponentially as they backpropagated from the output layer toward the input layer, completely freezing the early layers. For years, researchers blamed computer hardware. But in 2010, Xavier Glorot and Yoshua Bengio discovered the culprit: the beloved Sigmoid activation function! Sigmoid saturates at both ends (near 0 and 1), where its derivative is practically zero (

< 0.05

). Multiplying these tiny fractions across 10 layers caused the gradients to vanish into nothingness. The solution was shockingly simple: ReLU (Rectified Linear Unit),

f(x) = \max(0, x)

. Its derivative for positive numbers is always exactly 1.0! This simple change unlocked the ability to train networks with hundreds of layers.

Why Should You Learn Activation Functions: Sigmoid, Tanh, ReLU & Softmax?

🎯 Practical & Engineering Stakes

Choosing the right activation

The choice of activation function dictates gradient flow, convergence speed, and numerical stability. Using Sigmoid in deep hidden layers will ruin your training; using Softmax on the output layer guarantees calibrated probabilities.

What Is It?

Simple Explanation

Activation functions decide whether a neuron should fire and how strongly: 1. Sigmoid: Squashes numbers between 0 and 1 (good for binary yes/no output). 2. Tanh: Squashes numbers between -1 and +1 (centered around 0). 3. ReLU: "If negative, turn to 0. If positive, leave it alone!" (The king of hidden layers). 4. Softmax: Turns a list of scores into a percentage pie chart that sums to 100%.

Technical Explanation

Activation functions

\sigma: \mathbb{R} \rightarrow \mathbb{R}

introduce point-wise non-linear mappings. Sigmoid

\sigma(z) = \frac{1}{1+e^{-z}}

and Tanh

\tanh(z) = \frac{e^z - e^{-z}}{e^z + e^{-z}}

have saturated regimes where

\sigma'(z) \rightarrow 0

, causing vanishing gradients. ReLU

f(z) = \max(0, z)

provides non-saturating gradients

\frac{\partial f}{\partial z} = 1

for

z > 0

. Softmax generalizes sigmoid to

C

classes:

\text{Softmax}(\mathbf{z})_i = \frac{e^{z_i}}{\sum_{j=1}^C e^{z_j}}

📚 Key Terms

ReLU (Rectified Linear Unit)

Simple: The most popular activation: passes positive numbers through, blocks negatives.

Technical:

f(x) = \max(0, x)

, non-saturating non-linear activation with constant unit derivative on positive domain.

Vanishing Gradient Problem

Simple: When learning signals become so tiny that the first layers stop learning completely.

Technical: Exponential decay of gradient magnitudes as backpropagation propagates through saturated activation derivatives.

$Softmax$

Simple: Converts raw scores into probabilities that add up to 1.0 (100%).

Technical: Normalized exponential function mapping unbounded logits

\mathbf{z} \in \mathbb{R}^C

to probability simplex

\Delta^C

📐 Mathematical Foundation

LaTeX KaTeX

\text{Softmax}(z_i) = \frac{e^{z_i}}{\sum_{j=1}^C e^{z_j}}, \quad \text{ReLU}(z) = \max(0, z)

What does it mean?

Softmax (left) transforms C raw logit scores into a valid probability distribution summing to 1. ReLU (right) provides linear pass-through for positive inputs.

Understanding the Symbols

Symbol / Term	Meaning	Beginner Interpretation
$z_i$	Raw output logit for class i	Unbounded score
$C$	Number of classes	Total target categories

📊 Numerical Example

Applying Softmax to logits [2.0, 1.0, 0.1].

e^2 = 7.39, e^1 = 2.72, e^{0.1} = 1.11 \implies \text{Sum} = 11.22 \implies P = [0.658, 0.242, 0.099]

Result: Probabilities: 65.8%, 24.2%, 9.9% (Sum = 100%)

⚙️ How Does It Work?

Hidden Layers

Use ReLU (or Leaky ReLU / GELU) in all hidden layers to ensure healthy gradient flow.

Binary Output Head

Use Sigmoid on a single output neuron for binary classification (outputs probability between 0 and 1).

Multi-Class Output Head

Use Softmax across C output neurons for multi-class classification (outputs probability distribution summing to 1).

🌍 Where Is It Used?

Computer Vision (CNNs)

Problem: Deep 50-layer networks suffering from vanishing gradient failure.

Solution: ReLU activations enable training deep ResNet architectures without gradient attenuation.

Large Language Models (Transformers)

Problem: Selecting the most likely next word out of a 50,000-token vocabulary.

Solution: Softmax temperature layer normalizes output token logits into next-word sampling probabilities.

🎯 Try It Yourself

You are building a neural network to classify photos into 10 mutually exclusive dog breeds. Which activation should be used on the FINAL output layer?

Interactive activity component

🐍 Python Hands-On: Comparing Activation Functions in NumPy

Implement Sigmoid, ReLU, and Softmax and visualize their numerical outputs.

Open Colab

1Install Required Packages

For Google Colab or Jupyter notebooks, run this shell command to install the required dependencies:

Terminal / Notebook Install Commandbash

!pip install numpy

Packages:

numpy

2Import Libraries

Import Statementspython

1 import numpy as np

import numpy as np → NumPy for vector math

3Demonstration Code & Logic

Python Execution Codepython

1 # Raw input logits from negative to positive
2 z = np.array([-3.0, -1.0, 0.0, 1.0, 3.0])
3  
4 # 1. Sigmoid Function: 1 / (1 + exp(-z))
5 sigmoid = 1 / (1 + np.exp(-z))
6 print("Sigmoid Outputs (0 to 1):\n", sigmoid.round(3))
7  
8 # 2. ReLU Function: max(0, z)
9 relu = np.maximum(0, z)
10 print("\nReLU Outputs (0 for negatives):\n", relu)
11  
12 # 3. Softmax Function on 3-class logits [2.0, 1.0, 0.1]:
13 logits = np.array([2.0, 1.0, 0.1])
14 exp_scores = np.exp(logits - np.max(logits)) # Subtract max for numerical stability
15 softmax_probs = exp_scores / np.sum(exp_scores)
16 print("\nSoftmax Class Probabilities:\n", softmax_probs.round(3))
17 print("Softmax Sum Check:", round(np.sum(softmax_probs), 3))

Step-by-Step Code Breakdown:

# Raw input logits from negative to positive

Sample inputs

z = np.array([-3.0, -1.0, 0.0, 1.0, 3.0])

Range of test values

# 1. Sigmoid Function: 1 / (1 + exp(-z))

Sigmoid formula

sigmoid = 1 / (1 + np.exp(-z))

Compute sigmoid

print("Sigmoid Outputs (0 to 1):\n", sigmoid.round(3))

Print sigmoid

# 2. ReLU Function: max(0, z)

ReLU formula

relu = np.maximum(0, z)

Compute ReLU

print("\nReLU Outputs (0 for negatives):\n", relu)

Print ReLU

# 3. Softmax Function on 3-class logits [2.0, 1.0, 0.1]:

Softmax formula

logits = np.array([2.0, 1.0, 0.1])

Logit scores

exp_scores = np.exp(logits - np.max(logits)) # Subtract max for numerical stability

Exp step

softmax_probs = exp_scores / np.sum(exp_scores)

Normalize to 1.0

print("\nSoftmax Class Probabilities:\n", softmax_probs.round(3))

Print probs

print("Softmax Sum Check:", round(np.sum(softmax_probs), 3))

Verify sum is 1.0

📋 Complete Script (All-in-One)

Click "Copy for Colab", open Google Colab, paste into a single cell, and run.

Google Colab Notebook Scriptpython

1 # =========================================================
2 # Comparing Activation Functions in NumPy
3 # Prepared for Google Colab (Run with Shift + Enter)
4 # =========================================================
5  
6 # Step 1: Install required packages
7 !pip install numpy
8  
9 # Step 2: Import libraries
10 import numpy as np
11  
12 # Step 3: Run Demonstration
13 # Raw input logits from negative to positive
14 z = np.array([-3.0, -1.0, 0.0, 1.0, 3.0])
15  
16 # 1. Sigmoid Function: 1 / (1 + exp(-z))
17 sigmoid = 1 / (1 + np.exp(-z))
18 print("Sigmoid Outputs (0 to 1):\n", sigmoid.round(3))
19  
20 # 2. ReLU Function: max(0, z)
21 relu = np.maximum(0, z)
22 print("\nReLU Outputs (0 for negatives):\n", relu)
23  
24 # 3. Softmax Function on 3-class logits [2.0, 1.0, 0.1]:
25 logits = np.array([2.0, 1.0, 0.1])
26 exp_scores = np.exp(logits - np.max(logits)) # Subtract max for numerical stability
27 softmax_probs = exp_scores / np.sum(exp_scores)
28 print("\nSoftmax Class Probabilities:\n", softmax_probs.round(3))
29 print("Softmax Sum Check:", round(np.sum(softmax_probs), 3))

⚡ How to Run in Google Colab (3 Quick Steps)

Click "Copy for Google Colab" above.
Click Open New Google Colab Notebook (opens in a new tab).
Click on the first empty code cell, press Ctrl + V (or Cmd + V) to paste, then press Shift + Enter to execute!

Launch Blank Colab Notebook

🔄 Algorithmic Execution Flow

Sigmoid

Compresses [-3, 3] smoothly into [0.047, 0.953].

→

ReLU

Zeroes out negative values completely, keeping positives untouched.

→

Softmax

Converts raw logits [2.0, 1.0, 0.1] into clean probabilities [0.658, 0.242, 0.099] summing to exactly 1.0.

📥 Input / Initial State

Test vectors z = [-3, -1, 0, 1, 3] and logits = [2.0, 1.0, 0.1]

📤 Expected Output

Sigmoid Outputs:
[0.047 0.269 0.5   0.731 0.953]

ReLU Outputs:
[0. 0. 0. 1. 3.]

Softmax Class Probabilities:
[0.658 0.242 0.099]
Softmax Sum Check: 1.0

💡 What Does This Demonstration Show?

Each activation serves a distinct mathematical role: ReLU for deep hidden feature extraction, Sigmoid for binary probability, and Softmax for multi-class probability.

Course Learning Resource • Slide DeckLesson 512 Slides

Lesson 5: Classical Deep Learning with PyTorch

CNN, MNIST dataset, backpropagation, loss functions, and optimization.

Author: Dr. Syed Muzamil Basha• FDP Approved Curriculum

Open in New Tab

📝 Learning Check

Test your understanding with these questions. Remember, this is for learning, not examination!

Question 1: Why is ReLU preferred over Sigmoid in deep hidden layers?

Question 2: Why are activation functions necessary in neural networks?

Question 3: What is the ReLU activation function?

Question 4: What is the sigmoid activation function used for?

Question 5: What is the softmax activation function?

Previous: The Artificial Neuron: Perceptron & Math Foundations Next: Neural Network Architecture: Layers, Weights & Biases

1	# =========================================================
2	# Comparing Activation Functions in NumPy
3	# Prepared for Google Colab (Run with Shift + Enter)
4	# =========================================================
5
6	# Step 1: Install required packages
7	!pip install numpy
8
9	# Step 2: Import libraries
10	import numpy as np
11
12	# Step 3: Run Demonstration
13	# Raw input logits from negative to positive
14	z = np.array([-3.0, -1.0, 0.0, 1.0, 3.0])
15
16	# 1. Sigmoid Function: 1 / (1 + exp(-z))
17	sigmoid = 1 / (1 + np.exp(-z))
18	print("Sigmoid Outputs (0 to 1):\n", sigmoid.round(3))
19
20	# 2. ReLU Function: max(0, z)
21	relu = np.maximum(0, z)
22	print("\nReLU Outputs (0 for negatives):\n", relu)
23
24	# 3. Softmax Function on 3-class logits [2.0, 1.0, 0.1]:
25	logits = np.array([2.0, 1.0, 0.1])
26	exp_scores = np.exp(logits - np.max(logits)) # Subtract max for numerical stability
27	softmax_probs = exp_scores / np.sum(exp_scores)
28	print("\nSoftmax Class Probabilities:\n", softmax_probs.round(3))
29	print("Softmax Sum Check:", round(np.sum(softmax_probs), 3))