Lec 05 Activation and Loss Functions

An explanation of activation functions as gates that determine neuron output and introduce necessary nonlinearity to capture complex relationships in data, preventing networks from behaving like simple linear regression models.

Overview of various functions including Linear, Sigmoid (for binary classification), Tanh (for zero-centered hidden layer outputs), Softmax (for multiclass classification), and ReLU/Leaky ReLU (for computational efficiency and mitigating the dying ReLU problem).

A detailed look at Mean Squared Error (MSE) and Mean Absolute Error (MAE), highlighting how MSE is highly sensitive to outliers due to the squaring of errors.

Discussion of Binary Cross Entropy for binary classification, Categorical Cross Entropy for multiclass problems, and strategies for handling multi-label classification.

Introduction to KL Divergence as a measure of how one probability distribution differs from a reference distribution, commonly used in generative models.

Guidelines on setting the number of neurons in input, hidden, and output layers based on input features, task complexity, and the specific classification requirements.

A recap of key concepts covered during the lecture and a look ahead to upcoming sessions on optimization algorithms and backpropagation.