Challenges That Arise as Deep Learning Models Get Deeper

As deep learning models increase in depth, they have the capacity to learn more complex patterns.

However, as neural networks become deeper, several challenges may arise, and failing to address these can lead to decreased performance of the AI model.

In this lesson, we will explore the primary problems that occur as deep learning models become deeper.

When the layers of a neural network become deeper, it can lead to a problem where the weights in the earlier layers (closer to the input) are not adjusted properly.

This happens because the gradient continues to shrink during backpropagation, resulting in little to no weight updates.

Conversely to vanishing gradients, as neural networks deepen, gradients can become excessively large, causing the weights to be updated to very large values.

When exploding gradients occur, the model can become unstable and learning may fail.

With an increased number of layers, there is a risk of the model fitting too closely to the training data, leading to overfitting. This means the model's ability to generalize to new data decreases.

As the depth of the network increases, the computational workload grows, slowing the training process.

Typically, as computations increase, it requires longer training times and more hardware resources like GPUs.

Though deeper models can learn more complex patterns, these issues can degrade their performance.

To address these challenges, we can use techniques such as adjusting activation functions to mitigate the vanishing gradient problem, or employing normalization and standardization.

In the next lesson, we'll explore the "Dropout" technique, which helps prevent overfitting by randomly excluding some neurons during training.