Everything about Vanishing gradient problem
Machine learning model training problem
In machine learning, the vanishing gradient problem is the problem of greatly diverging gradient magnitudes between earlier and later layers encountered when training neural networks with backpropagation. In such methods, neural network weights are updated proportional to their partial derivative of the loss function. As the number of forward propagation steps in a network increases, for instance due to greater network depth, the gradients of earlier weights are calculated with increasingly many multiplications. These multiplications shrink the gradient magnitude. Consequently, the gradients of earlier weights will be exponentially smaller than the gradients of later weights. This difference in gradient magnitude might introduce instability in the training process, slow it, or halt it entirely. For instance, consider the hyperbolic tangent activation function. The gradients of this function are in range [0,1]. The product of repeated multiplication with such gradients decreases exponentially. The inverse problem, when weight gradients at earlier layers get exponentially larger, is called the exploding gradient problem. Wikipedia →
🧭 Start here: a 6-step reading path
- Read the overview: Vanishing gradient problemA 5-minute orientation
- Skim a recent paper: The vanishing gradient problem for stiff neural differential equationsInfluential work since 2022
- Pick up a classic book: Neural networksThe most-reprinted book on the subject
- Read the landmark paper: The Vanishing Gradient Problem During Learning Recurrent Neural Nets and Problem SolutionsCited 2,481 times
- Go sideways: Highway networkA closely related topic
- Go sideways: Long short-term memoryA closely related topic
📖 Definitions
No dictionary entry for this exact phrase.
🗂️ The subject at a glance
🏆 Landmark papers (most cited)
The Vanishing Gradient Problem During Learning Recurrent Neural Nets and Problem Solutions
A new approach for the vanishing gradient problem on sigmoid activation
On the vanishing and exploding gradient problem in Gated Recurrent Units
Handling Vanishing Gradient Problem Using Artificial Derivative
Preventing Vanishing Gradient Problem of Hardware Neuromorphic System by Implementing Imidazole‐Based Memristive ReLU Activation Neuron
An improved autoencoder for recommendation to alleviate the vanishing gradient problem
Overcoming the Vanishing Gradient Problem during Learning Recurrent Neural Nets (RNN)
The Monge problem with vanishing gradient penalization: Vortices and asymptotic profile
🆕 Influential since 2022
The vanishing gradient problem for stiff neural differential equations
Decentralized Laplacian Gradient Flows With Vanishing Anchors for Resource Allocation Problems With Arbitrary Initialization
Mitigating the Vanishing Gradient Problem Using a Pseudo-Normalizing Method
Vanishing Gradient Problem: An Activation Function that Seeks to Improve VGP
The Hybrid Residual–BiLSTM Model for Mitigating the Vanishing Gradient Problem in Synthetic Image Detection
📚 Books worth reading
Neural networksSimon S. Haykin · 1994
Intelligent Science and Intelligent Data EngineeringJian Yang · 2013
Fundamentals of Deep LearningNikhil Buduma · 2017
Ranked by number of editions on Open Library, a rough measure of how lasting a book is.
🔗 Related topics
- Highway network In machine learning, the Highway Network was a feedforward neural network that uses skip connections modulated by learned gating mechanisms to regulate…
- Long short-term memory Long short-term memory (LSTM) is a type of recurrent neural network (RNN) aimed at mitigating the vanishing gradient problem commonly encountered by traditional…
- Recurrent neural network In artificial neural networks, recurrent neural networks (RNNs) are designed for processing sequential data, such as text, speech, and time series, where…
- Residual neural network A residual neural network (also referred to as a residual network or ResNet) is a deep learning architecture in which the layers learn residual functions…
- Gating mechanism In neural networks, the gating mechanism is an architectural motif for controlling the flow of activation and gradient signals. They are most prominently…
- Weight initialization In deep learning, weight initialization or parameter initialization describes the initial step in creating a neural network. A neural network contains…
- Deep backward stochastic differential equation method Deep backward stochastic differential equation method is a numerical method that combines deep learning with Backward stochastic differential equation…
- Jürgen Schmidhuber Jürgen Schmidhuber (born 17 January 1963) is a German computer scientist in the field of artificial intelligence, specifically artificial neural networks…