Backpropagation

In <a href="/facts/Machine_learning/e0w0XJTu">machine learning</a>, backpropagation is a <a href="/facts/Gradient/TsR7uukj">gradient</a> estimation method commonly used for training a <a href="/facts/Neural_network_(machine_learning)/6V1jMlkx">neural network</a> to compute its parameter updates.
It is an efficient application of the <a href="/facts/Chain_rule/aMLYxP0x">chain rule</a> to neural networks. Backpropagation computes the <a href="/facts/Gradient/TsR7uukj">gradient</a> of a <a href="/facts/Loss_function/xv5ozuhl">loss function</a> with respect to the <a href="/facts/Glossary_of_graph_theory_terms/W0MqKkyE">weights</a> of the network for a single input–output example, and does so <a href="/facts/Algorithmic_efficiency/VutjvPTd">efficiently</a>, computing the gradient one layer at a time, <a href="/facts/Iteration/AMc1p7BL">iterating</a> backward from the last layer to avoid redundant calculations of intermediate terms in the chain rule; this can be derived through <a href="/facts/Dynamic_programming/CLcabtmk">dynamic programming</a>.
Strictly speaking, the term backpropagation refers only to an algorithm for efficiently computing the gradient, not how the gradient is used; but the term is often used loosely to refer to the entire learning algorithm – including how the gradient is used, such as by <a href="/facts/Stochastic_gradient_descent/HbcaYqQP">stochastic gradient descent</a>, or as an intermediate step in a more complicated optimizer, such as <a href="/facts/Stochastic_gradient_descent/HbcaYqQP">Adaptive Moment Estimation</a>. The local minimum convergence, exploding gradient, vanishing gradient, and weak control of learning rate are main disadvantages of these optimization algorithms. The <a href="/facts/Hessian_matrix/RguMNr3m">Hessian</a> and quasi-Hessian optimizers solve only local minimum convergence problem, and the backpropagation works longer. These problems caused researchers to develop hybrid and fractional optimization algorithms. 
Backpropagation had multiple discoveries and partial discoveries, with a tangled history and terminology. See the history section for details. Some other names for the technique include "reverse mode of <a href="/facts/Automatic_differentiation/dQcdhBAM">automatic differentiation</a>" or "<a href="/facts/Reverse_accumulation/dQcdhBAM">reverse accumulation</a>".

Backpropagation open-in-new

Backpropagation