Lec 08 Varients of Gradient descent and Momentum based techniques

The lecture introduces various gradient descent optimizers including batch, mini-batch, and stochastic gradient descent (SGD). It covers the mathematical formulation of weight updates and compares the convergence characteristics of each method.

Discussion on why standard gradient descent struggles with plateau regions. Momentum-based optimization is introduced as a way to accelerate convergence by incorporating a history of gradients, similar to a ball rolling down a hill.

Explanation of Nesterov Accelerated Gradient, which improves upon standard momentum by using a 'look-ahead' mechanism to make more informed weight updates, thereby reducing oscillations near the minima.

Explores the limitations of using a fixed learning rate and introduces adaptive learning rate strategies. The concept of sparse vs. dense data sets is explained to motivate adaptive algorithms.

Details the Adagrad optimizer which tunes learning rates based on previous updates, and RMSprop, which uses exponentially decaying averages of squared gradients to avoid the aggressive learning rate decay observed in Adagrad.

Introduction to Adam, which combines momentum and RMSprop, and Adadelta, which removes the need for an initial learning rate by dynamically adjusting the learning rate based on the slope of the loss landscape.

Final summary of all discussed optimizers, their pros and cons, and concluding remarks regarding their application in neural networks.