Title: Gradient Descent Algorithm

Student Name: Gautam Dhodi

Student Enrollment: 0827CS221095

Overview:
Gradient Descent is a fundamental optimization algorithm used extensively in
machine learning to minimize a function. This function typically represents a cost
or loss function, quantifying the error between a model's predictions and the actual
data. By iteratively adjusting the model's parameters in the direction opposite to the
gradient of this function, Gradient Descent aims to find the parameter values that
minimize the error, thereby improving the model's performance. It's a cornerstone
of many machine learning algorithms due to its effectiveness and relative
simplicity.

Technical Background:
Gradient Descent operates on the principle of iteratively moving towards the
minimum of a function. The gradient of a function is a vector that contains the
partial derivatives of the function with respect to each of its input variables.

For a function J(θ1, θ2, ..., θn), the gradient is defined as:

∇J(θ) =
[
∂J/∂θ1
∂J/∂θ2
...
∂J/∂θn
]
The gradient vector points in the direction of the steepest ascent of the function. To
find the minimum, Gradient Descent takes steps in the opposite direction of the
gradient. The size of these steps is determined by the learning rate, denoted by α.
The iterative update rule for the parameters θ is:

θk+1 = θk - α∇J(θk)

Where:
●​ θk represents the parameter vector at the k-th iteration.
●​ α is the learning rate, controlling the step size.
The algorithm continues updating the parameters until a stopping criterion is met.
Common stopping criteria include:
●​ The change in the cost function or parameter values falls below a specified
threshold.
●​ A maximum number of iterations is reached.
The choice of learning rate (α) significantly impacts the algorithm's performance.
A small learning rate may lead to slow convergence, while a large learning rate can
cause oscillations or divergence.

Insights:
●​ Direction of Steepest Descent: Gradient Descent leverages the fact that the
negative gradient provides the direction of the steepest decrease of the
function.
●​ Iterative Process: It's an iterative process that progressively refines parameter
estimates to approach the minimum.
●​ Sensitivity to Cost Function: The shape of the cost function's landscape
affects how efficiently Gradient Descent finds the minimum.
●​ Learning Rate Importance: Selecting an appropriate learning rate is essential
for both the speed and stability of convergence.
Methodology:
The Gradient Descent algorithm follows these general steps:
1.​ Initialization: Initialize the parameter vector θ with some starting values.
2.​ Compute Gradient: Calculate the gradient of the cost function J(θ) at the
current parameter values.
 3.​ Update Parameters: Update the parameters using the update rule: θnew =
θold - α∇J(θold).
4.​ Iteration: Repeat steps 2 and 3 until a stopping criterion is met.
Different variations of Gradient Descent exist, primarily differing in how the
gradient is computed:
●​ Batch Gradient Descent: Computes the gradient using the entire training
dataset. This provides an accurate gradient estimate but can be computationally
expensive for large datasets.
●​ Stochastic Gradient Descent (SGD): Computes the gradient using a single,
randomly selected data point. SGD is faster per iteration but introduces more
noise, leading to a potentially less stable path towards the minimum.
●​ Mini-Batch Gradient Descent: Computes the gradient using a small,
randomly selected subset (mini-batch) of the data. This balances the
computational efficiency of SGD with the more stable gradient estimates of
Batch Gradient Descent.
Architecture:
Gradient Descent, as an optimization algorithm, is applied within the architecture
of a machine learning system. It is a key component in training various models,
including:
●​ Linear Regression: To minimize the mean squared error and find optimal
weights and bias.
●​ Logistic Regression: To minimize the cost function (e.g., cross-entropy) and
determine the decision boundary.
●​ Neural Networks: In deep learning, Gradient Descent and its variants are used
to update the weights and biases of network layers through backpropagation.
The "architecture" here refers to the combination of the model, the cost function,
and the optimization process (Gradient Descent) used to train the model.

Appropriate Diagrams:
J(θ)​
|​
| * (Initial θ)​
| \​
| \ α * dJ/dθ​
| \​
| * (Updated θ)​
+-------------------> θ​
 θ_min​

This diagram illustrates how Gradient Descent iteratively moves towards the
minimum of a function in one dimension.

J(θ1, θ2)​
^​
|​
 Level Curves​
|​
* (Initial θ)​
/​
/ Step 1​
*​
/​
/ Step 2​
* . . . -> θ_min​
/​
/​
*​
+----------------> θ1​
θ2​

This 2D contour plot visualizes Gradient Descent's iterative steps towards the
minimum of a function with two parameters.
(A conceptual diagram showing the different paths taken by Batch GD (smooth),
SGD (erratic), and Mini-Batch GD (intermediate) towards the minimum.)

Facts and Figures:

●​ Convergence Rate: The convergence rate depends on factors like the learning
rate, cost function shape, and the number of parameters.
●​ Computational Cost: Batch GD is computationally expensive for large
datasets; SGD is faster per iteration but may require more iterations.
●​ Learning Rate Sensitivity: Improper learning rates can lead to slow
convergence, oscillations, or divergence.
●​ Local Minima: In non-convex functions, Gradient Descent may get stuck in
local minima.
Comparative Studies:
Gradient Descent serves as a foundation, and other optimization algorithms
improve upon it or offer alternatives. Some comparisons include:
●​ Newton's Method: Uses second-order derivatives (Hessian) for faster
convergence but is computationally expensive for large problems.
●​ Conjugate Gradient: Employs conjugate directions for more efficient
convergence in certain problems.
●​ Adam (Adaptive Moment Estimation): An adaptive learning rate method
that combines the advantages of RMSprop and momentum, often leading to
faster convergence.
●​ RMSprop (Root Mean Square Propagation): Adaptively scales learning
rates for each parameter.
●​ L-BFGS (Limited-memory BFGS): A quasi-Newton method that
approximates the Hessian, more efficient for large-scale problems.
While these advanced algorithms often offer benefits, Gradient Descent remains a
core concept.

Application Areas:
Gradient Descent is a fundamental algorithm across various domains, particularly
in machine learning:
●​ Machine Learning Model Training: Used to train a wide variety of models
like linear regression, logistic regression, and support vector machines.
●​ Deep Learning: Essential for training neural networks through
backpropagation and its variants (SGD, Adam, RMSprop).
●​ General Optimization: Applied to solve optimization problems in fields like
engineering, finance, and operations research.
●​ Image Processing and Computer Vision: Used in image registration and
training Convolutional Neural Networks for image tasks.
●​ Natural Language Processing (NLP): Employed in training word
embeddings and language models.
Additional Information:
Code Snippet (Python - Linear Regression with Gradient Descent):

import numpy as np​

​
def cost_function(X, y, w, b):​
m = len(y)​
y_pred = np.dot(X, w) + b​
cost = (1 / (2 * m)) * np.sum((y_pred - y) ** 2)​
return cost​
​
def gradient_descent(X, y, w_init, b_init, alpha, num_iters):​
m = len(y)​
w = w_init​
b = b_init​
history = []​
​
for i in range(num_iters):​
y_pred = np.dot(X, w) + b​
dw = (1 / m) * np.dot(X.T, (y_pred - y))​
db = (1 / m) * np.sum(y_pred - y)​
​
w = w - alpha * dw​
b = b - alpha * db​
cost = cost_function(X, y, w, b)​
history.append(cost)​
​
if i % 100 == 0:​
print(f"Iteration {i}, Cost: {cost:.4f}")​
​
return w, b, history​
​
# Example​
X = np.array([[1], [2], [3], [4]])​
y = np.array([2, 4, 5, 4])​
w_init = 0​
b_init = 0​
alpha = 0.01​
iterations = 1000​
​
w_trained, b_trained, cost_history = gradient_descent(X, y, w_init, b_init, alpha,
iterations)​
print(f"Trained w: {w_trained[0]:.4f}")​
print(f"Trained b: {b_trained:.4f}")​

Real-world Problem Statement:

Consider predicting housing prices based on factors like square footage and the
number of bedrooms. Linear regression with Gradient Descent can be used to
determine the optimal weights for these features and the bias that minimize the
error between predicted and actual prices.

References:
●​ Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press.
●​ Bishop, C. M. (2006). Pattern Recognition and Machine Learning. Springer.
●​ Hastie, T., Tibshirani, R., & Friedman, J. (2009). The Elements of Statistical
Learning. Springer.
●​ Ruder, S. (2016). An overview of gradient descent optimization algorithms.
arXiv preprint arXiv:1609.04747.