Live briefing
Everything about Gradient descent
Optimization algorithm
Gradient descent is a method for unconstrained mathematical optimization. It is a first-order iterative algorithm for minimizing a differentiable multivariate function. Wikipedia →
🧭 Start here: a 6-step reading path
- Read the overview: Gradient descentA 5-minute orientation
- Skim a recent paper: Recent Advances in Stochastic Gradient Descent in Deep LearningInfluential work since 2022
- Pick up a classic book: Nonlinear performance seeking control using fuzzy model reference learning control and the method of steepest descentThe most-reprinted book on the subject
- Read the landmark paper: Learning long-term dependencies with gradient descent is difficultCited 7,186 times
- Go sideways: Backtracking line searchA closely related topic
- Go sideways: Subgradient methodA closely related topic
📖 Definitions
Noun A first-order iterative optimization algorithm for finding a local minimum of a differentiable function.
🏆 Landmark papers (most cited)
Learning long-term dependencies with gradient descent is difficult
On the momentum term in gradient descent learning algorithms
Exponentiated Gradient versus Gradient Descent for Linear Predictors
Accelerated Distributed Nesterov Gradient Descent
On Nonconvex Decentralized Gradient Descent
Preconditioned Stochastic Gradient Descent
Online Gradient Descent Learning Algorithms
Graph Drawing by Stochastic Gradient Descent
🆕 Influential since 2022
Recent Advances in Stochastic Gradient Descent in Deep Learning
Byzantine-Resilient Decentralized Stochastic Gradient Descent
BRIDGE: Byzantine-Resilient Decentralized Gradient Descent
Stochastic gradient descent for wind farm optimization
LSPIA, (stochastic) gradient descent, and parameter correction
📚 Books worth reading
- 📕Nonlinear performance seeking control using fuzzy model reference learning control and the method of steepest descentGeorge Kopaskis · 1997
Ranked by number of editions on Open Library, a rough measure of how lasting a book is.
🔗 Related topics
- Backtracking line search In (unconstrained) mathematical optimization, a backtracking line search is a line search method to determine the amount to move along a given search direction…
- Subgradient method Subgradient methods are convex optimization methods which use subderivatives. Originally developed by Naum Z. Shor and others in the 1960s and 1970s, subgradien…
- Online machine learning In computer science, online machine learning is a method of machine learning in which data becomes available in a sequential order and is used to update…
- Hill climbing In numerical analysis, hill climbing is a mathematical optimization technique which belongs to the family of local search. It is an iterative algorithm…
- Stochastic gradient descent Stochastic gradient descent (often abbreviated SGD) is an iterative method for optimizing an objective function with suitable smoothness properties (e…
- Newton's method in optimization In calculus, Newton's method (also called Newton–Raphson) is an iterative method for finding the roots of a differentiable function f, which are solutions…
- Stochastic variance reduction (Stochastic) variance reduction is an algorithmic approach to minimizing functions that can be decomposed into finite sums. By exploiting the finite sum…
- Gauss–Newton algorithm In numerical methods, the Gauss–Newton algorithm is an iterative algorithm to solve non-linear least squares problems. It is an extension of Newton's method…