First time studying this topic?

Launch the interactive walkthrough to discover each component, including the 3D Q-Sphere Lab & Python in Colab.

Topic 2.5

Regression: Continuous Value Prediction

Master Simple Linear Regression, Multiple Regression, and Polynomial Regression to forecast continuous outcomes.

Learning Objectives

✓Derive the Ordinary Least Squares (OLS) objective
✓Interpret regression coefficients, intercepts, and residuals
✓Expand linear models into non-linear spaces using Polynomial features

📖Let's Begin with a Story

In 1886, Sir Francis Galton measured the heights of 928 adult children and their parents. He noticed an intriguing phenomenon: exceptionally tall parents had children who were tall, but on average slightly shorter than themselves. Similarly, very short parents had children who were slightly taller than them. The heights of the children were "regressing toward the mediocrity" (the population mean). Galton termed this statistical line the "regression line." Today, regression refers to any mathematical method that predicts continuous numbers along a trend line.

Why Should You Learn Regression: Continuous Value Prediction?

🎯 Practical & Engineering Stakes

Continuous predictions drive the economy

Predicting stock prices, corporate revenues, fuel consumption, CO2 emissions, and battery degradation all require continuous regression models.

What Is It?

Simple Explanation

Regression is finding the best-fit line through a cloud of scatter points. Once you have that line, you can predict what happens between the points or forecast into the future. For example, if you know a person's height, you can use the regression line to estimate their weight.

Technical Explanation

Multiple Linear Regression models the conditional expectation as an inner product:

\hat{y} = \mathbf{x}^T \\boldsymbol{\\beta} + \epsilon

, where

\epsilon \sim \mathcal{N}(0, \sigma^2)

. The Ordinary Least Squares (OLS) closed-form analytic solution solves the normal equations:

\hat{\\boldsymbol{\\beta}} = (\mathbf{X}^T \mathbf{X})^{-1} \mathbf{X}^T \mathbf{y}

📚 Key Terms

$Residual$

Simple: The vertical distance between a real data point and the prediction line.

Technical: The sample error

e_i = y_i - \hat{y}_i

representing unexplained variance.

Ordinary Least Squares (OLS)

Simple: The math trick that finds the line that makes the squared vertical distances as small as possible.

Technical: The convex optimization method that minimizes the sum of squared residuals

\|\mathbf{y} - \mathbf{X}\boldsymbol{\beta}\|^2

R-Squared ( $R^2$ )

Simple: A score from 0% to 100% showing how well the line explains the data.

Technical: Coefficient of determination

R^2 = 1 - \frac{SS_{res}}{SS_{tot}}

measuring the proportion of variance explained by features.

📐 Mathematical Foundation

LaTeX KaTeX

\boldsymbol{\beta} = (\mathbf{X}^T \mathbf{X})^{-1} \mathbf{X}^T \mathbf{y}

What does it mean?

The Normal Equation: the exact closed-form linear algebra solution for optimal OLS regression coefficients.

Understanding the Symbols

Symbol / Term	Meaning	Beginner Interpretation
$\boldsymbol{\beta}$	Optimal weight vector	Best-fit slopes and intercept
$\mathbf{X}$	Design matrix	Table of input features
$\mathbf{y}$	Target vector	Actual continuous values

📊 Numerical Example

Simple 1D line through points (1, 2) and (3, 6).

\text{Slope } m = \frac{6 - 2}{3 - 1} = \frac{4}{2} = 2.0

Result: Model: y = 2.0 * x (Residuals = 0, R^2 = 1.0)

⚙️ How Does It Work?

Scatter Plot Inspection

Plot target against candidate predictors to assess linear correlation.

Matrix Solution or Gradient Descent

Compute optimal coefficients using normal equation or iterative optimizer.

Residual Analysis

Verify residuals are normally distributed with constant variance (homoscedasticity).

🌍 Where Is It Used?

Real Estate Valuation

Problem: Estimating market price of houses based on square meters, bedrooms, and transit score.

Solution: Multiple regression outputs estimated dollar valuation with confidence intervals.

Energy Grid Demand Forecasting

Problem: Predicting megawatt electrical load based on temperature forecasts.

Solution: Polynomial regression captures non-linear AC usage spikes during summer heatwaves.

🎯 Try It Yourself

A house price regression model achieves R² = 0.85. What does this mean?

Interactive activity component

🐍 Python Hands-On: Linear Regression: House Price Prediction & Trend Visualization

Fit an Ordinary Least Squares (OLS) linear regression model on house sizes to predict continuous prices, and visualize the regression line with Matplotlib.

Open Colab

1Install Required Packages

For Google Colab or Jupyter notebooks, run this shell command to install the required dependencies:

Terminal / Notebook Install Commandbash

!pip install numpy scikit-learn matplotlib

Packages:

numpyscikit-learnmatplotlib

2Import Libraries

Import Statementspython

1 import numpy as np
2 import matplotlib.pyplot as plt
3 from sklearn.linear_model import LinearRegression

import numpy as np → NumPy for matrix operations

import matplotlib.pyplot as plt → Matplotlib for plotting trend line

from sklearn.linear_model import LinearRegression → Scikit-Learn OLS regressor

3Demonstration Code & Logic

Python Execution Codepython

1 # Data: House sizes (sq ft) and corresponding prices ($)
2 X = np.array([500, 700, 900, 1100, 1300]).reshape(-1, 1)  # Feature (sq ft)
3 y = np.array([150000, 180000, 210000, 250000, 280000])  # Target (price)
4  
5 # Create Linear Regression model
6 model = LinearRegression()
7  
8 # Train the model on the data
9 model.fit(X, y)
10  
11 # Predict price for a house with 1000 sq ft
12 predicted_price = model.predict([[1000]])
13 print(f"Predicted Price for 1000 sq ft house: ${predicted_price[0]:,.2f}")
14  
15 # Plot Data and Regression Line
16 plt.scatter(X, y, color='blue', label='Actual Prices')
17 plt.plot(X, model.predict(X), color='red', label='Regression Line')
18 plt.xlabel("House Size (sq ft)")
19 plt.ylabel("House Price ($)")
20 plt.title("Linear Regression - House Price Prediction")
21 plt.legend()
22 plt.show()

Step-by-Step Code Breakdown:

# Data: House sizes (sq ft) and corresponding prices ($)

Data definition

X = np.array([500, 700, 900, 1100, 1300]).reshape(-1, 1) # Feature (sq ft)

Reshape into column vector

y = np.array([150000, 180000, 210000, 250000, 280000]) # Target (price)

Target house prices

# Create Linear Regression model

Instantiate model

model = LinearRegression()

Create OLS instance

# Train the model on the data

Fitting

model.fit(X, y)

Fit best-fit line

# Predict price for a house with 1000 sq ft

Inference

predicted_price = model.predict([[1000]])

Evaluate prediction at 1000 sqft

print(f"Predicted Price for 1000 sq ft house: ${predicted_price[0]:,.2f}")

Print formatted predicted price

# Plot Data and Regression Line

Visualization

plt.scatter(X, y, color='blue', label='Actual Prices')

Plot data points

plt.plot(X, model.predict(X), color='red', label='Regression Line')

Plot best-fit line

plt.xlabel("House Size (sq ft)")

X-axis label

plt.ylabel("House Price ($)")

Y-axis label

plt.title("Linear Regression - House Price Prediction")

Plot title

plt.legend()

Add legend

plt.show()

Display plot

📋 Complete Script (All-in-One)

Click "Copy for Colab", open Google Colab, paste into a single cell, and run.

Google Colab Notebook Scriptpython

1 # =========================================================
2 # Linear Regression: House Price Prediction & Trend Visualization
3 # Prepared for Google Colab (Run with Shift + Enter)
4 # =========================================================
5  
6 # Step 1: Install required packages
7 !pip install numpy scikit-learn matplotlib
8  
9 # Step 2: Import libraries
10 import numpy as np
11 import matplotlib.pyplot as plt
12 from sklearn.linear_model import LinearRegression
13  
14 # Step 3: Run Demonstration
15 # Data: House sizes (sq ft) and corresponding prices ($)
16 X = np.array([500, 700, 900, 1100, 1300]).reshape(-1, 1)  # Feature (sq ft)
17 y = np.array([150000, 180000, 210000, 250000, 280000])  # Target (price)
18  
19 # Create Linear Regression model
20 model = LinearRegression()
21  
22 # Train the model on the data
23 model.fit(X, y)
24  
25 # Predict price for a house with 1000 sq ft
26 predicted_price = model.predict([[1000]])
27 print(f"Predicted Price for 1000 sq ft house: ${predicted_price[0]:,.2f}")
28  
29 # Plot Data and Regression Line
30 plt.scatter(X, y, color='blue', label='Actual Prices')
31 plt.plot(X, model.predict(X), color='red', label='Regression Line')
32 plt.xlabel("House Size (sq ft)")
33 plt.ylabel("House Price ($)")
34 plt.title("Linear Regression - House Price Prediction")
35 plt.legend()
36 plt.show()

⚡ How to Run in Google Colab (3 Quick Steps)

Click "Copy for Google Colab" above.
Click Open New Google Colab Notebook (opens in a new tab).
Click on the first empty code cell, press Ctrl + V (or Cmd + V) to paste, then press Shift + Enter to execute!

Launch Blank Colab Notebook

🔄 Algorithmic Execution Flow

Data Formulation

Encodes 5 house sizes [500-1300 sq ft] and prices [$150k-$280k].

→

Linear Fitting

Finds optimal slope m and intercept b minimizing squared residuals.

→

Inference & Plotting

Forecasts 1000 sq ft house at $230,000 and plots data points alongside the regression trend line.

📥 Input / Initial State

House sizes: [500, 700, 900, 1100, 1300] sq ft

📤 Expected Output

Predicted Price for 1000 sq ft house: $230,000.00

💡 What Does This Demonstration Show?

The linear regression model accurately captures the ~$162.50 per square foot trend line, providing reliable continuous price estimations.

Course Learning Resource • Slide DeckLesson 211 Slides

Lesson 2: Classical Machine Learning as the Starting Point

Data, prediction, linear regression, and the bridge to modern artificial intelligence.

Author: Dr. Syed Muzamil Basha• FDP Approved Curriculum

Open in New Tab

📝 Learning Check

Test your understanding with these questions. Remember, this is for learning, not examination!

Question 1: What happens to the regression line if an extreme outlier data point is added far from the trend?

Question 2: What is the difference between classification and regression?

Question 3: What does the slope (coefficient) in linear regression y = mx + b represent?

Question 4: What loss function is typically minimized in linear regression?

Question 5: What is polynomial regression?

Previous: Classification Algorithms & Decision Boundaries Next: Model Evaluation: Beyond Simple Accuracy

1	import numpy as np
2	import matplotlib.pyplot as plt
3	from sklearn.linear_model import LinearRegression

1	# =========================================================
2	# Linear Regression: House Price Prediction & Trend Visualization
3	# Prepared for Google Colab (Run with Shift + Enter)
4	# =========================================================
5
6	# Step 1: Install required packages
7	!pip install numpy scikit-learn matplotlib
8
9	# Step 2: Import libraries
10	import numpy as np
11	import matplotlib.pyplot as plt
12	from sklearn.linear_model import LinearRegression
13
14	# Step 3: Run Demonstration
15	# Data: House sizes (sq ft) and corresponding prices ($)
16	X = np.array([500, 700, 900, 1100, 1300]).reshape(-1, 1) # Feature (sq ft)
17	y = np.array([150000, 180000, 210000, 250000, 280000]) # Target (price)
18
19	# Create Linear Regression model
20	model = LinearRegression()
21
22	# Train the model on the data
23	model.fit(X, y)
24
25	# Predict price for a house with 1000 sq ft
26	predicted_price = model.predict([[1000]])
27	print(f"Predicted Price for 1000 sq ft house: ${predicted_price[0]:,.2f}")
28
29	# Plot Data and Regression Line
30	plt.scatter(X, y, color='blue', label='Actual Prices')
31	plt.plot(X, model.predict(X), color='red', label='Regression Line')
32	plt.xlabel("House Size (sq ft)")
33	plt.ylabel("House Price ($)")
34	plt.title("Linear Regression - House Price Prediction")
35	plt.legend()
36	plt.show()