Menu
Home
People
Places
Arts
History
Plants & Animals
Science
Life & Culture
Technology
Reference.org
Delta rule
open-in-new
<h2 id="derivation-of-the-delta-rule">Derivation of the delta rule</h2> <p>The delta rule is derived by attempting to minimize the error in the output of the neural network through <a href="/facts/Gradient_descent/pFFrek0F">gradient descent</a>. The error for a neural network with j {\displaystyle j} outputs can be measured as E = ∑ j 1 2 ( t j − y j ) 2 . {\displaystyle E=\sum _{j}{\tfrac {1}{2}}\left(t_{j}-y_{j}\right)^{2}.} </p><p>In this case, we wish to move through "weight space" of the neuron (the space of all possible values of all of the neuron's weights) in proportion to the gradient of the error function with respect to each weight. In order to do that, we calculate the <a href="/facts/Partial_derivative/JWHsBkAu">partial derivative</a> of the error with respect to each weight. For the i {\displaystyle i} th weight, this derivative can be written as ∂ E ∂ w j i . {\displaystyle {\frac {\partial E}{\partial w_{ji}}}.} </p><p>Because we are only concerning ourselves with the j {\displaystyle j} -th neuron, we can substitute the error formula above while omitting the summation: ∂ E ∂ w j i = ∂ ∂ w j i [ 1 2 ( t j − y j ) 2 ] {\displaystyle {\frac {\partial E}{\partial w_{ji}}}={\frac {\partial }{\partial w_{ji}}}\left[{\frac {1}{2}}\left(t_{j}-y_{j}\right)^{2}\right]} </p><p>Next we use the <a href="/facts/Chain_rule/aMLYxP0x">chain rule</a> to split this into two derivatives: ∂ E ∂ w j i = ∂ ( 1 2 ( t j − y j ) 2 ) ∂ y j ∂ y j ∂ w j i {\displaystyle {\frac {\partial E}{\partial w_{ji}}}={\frac {\partial \left({\frac {1}{2}}\left(t_{j}-y_{j}\right)^{2}\right)}{\partial y_{j}}}{\frac {\partial y_{j}}{\partial w_{ji}}}} </p><p>To find the left derivative, we simply apply the <a href="/facts/Power_rule/V6wJxLlp">power rule</a> and the chain rule: ∂ E ∂ w j i = − ( t j − y j ) ∂ y j ∂ w j i {\displaystyle {\frac {\partial E}{\partial w_{ji}}}=-\left(t_{j}-y_{j}\right){\frac {\partial y_{j}}{\partial w_{ji}}}} </p><p>To find the right derivative, we again apply the chain rule, this time differentiating with respect to the total input to j {\displaystyle j} , h j {\displaystyle h_{j}} : ∂ E ∂ w j i = − ( t j − y j ) ∂ y j ∂ h j ∂ h j ∂ w j i {\displaystyle {\frac {\partial E}{\partial w_{ji}}}=-\left(t_{j}-y_{j}\right){\frac {\partial y_{j}}{\partial h_{j}}}{\frac {\partial h_{j}}{\partial w_{ji}}}} </p><p>Note that the output of the j {\displaystyle j} th neuron, y j {\displaystyle y_{j}} , is just the neuron's activation function g {\displaystyle g} applied to the neuron's input h j {\displaystyle h_{j}} . We can therefore write the derivative of y j {\displaystyle y_{j}} with respect to h j {\displaystyle h_{j}} simply as g {\displaystyle g} 's first derivative: ∂ E ∂ w j i = − ( t j − y j ) g ′ ( h j ) ∂ h j ∂ w j i {\displaystyle {\frac {\partial E}{\partial w_{ji}}}=-\left(t_{j}-y_{j}\right)g'(h_{j}){\frac {\partial h_{j}}{\partial w_{ji}}}} </p><p>Next we rewrite h j {\displaystyle h_{j}} in the last term as the sum over all k {\displaystyle k} weights of each weight w j k {\displaystyle w_{jk}} times its corresponding input x k {\displaystyle x_{k}} : ∂ E ∂ w j i = − ( t j − y j ) g ′ ( h j ) ∂ ∂ w j i [ ∑ i x i w j i ] {\displaystyle {\frac {\partial E}{\partial w_{ji}}}=-\left(t_{j}-y_{j}\right)g'(h_{j})\;{\frac {\partial }{\partial w_{ji}}}\!\!\left[\sum _{i}x_{i}w_{ji}\right]} </p><p>Because we are only concerned with the i {\displaystyle i} th weight, the only term of the summation that is relevant is x i w j i {\displaystyle x_{i}w_{ji}} . Clearly, ∂ ( x i w j i ) ∂ w j i = x i . {\displaystyle {\frac {\partial (x_{i}w_{ji})}{\partial w_{ji}}}=x_{i}.} giving us our final equation for the gradient: ∂ E ∂ w j i = − ( t j − y j ) g ′ ( h j ) x i {\displaystyle {\frac {\partial E}{\partial w_{ji}}}=-\left(t_{j}-y_{j}\right)g'(h_{j})x_{i}} </p><p>As noted above, gradient descent tells us that our change for each weight should be proportional to the gradient. Choosing a proportionality constant α {\displaystyle \alpha } and eliminating the minus sign to enable us to move the weight in the negative direction of the gradient to minimize error, we arrive at our target equation: Δ w j i = α ( t j − y j ) g ′ ( h j ) x i . {\displaystyle \Delta w_{ji}=\alpha (t_{j}-y_{j})g'(h_{j})x_{i}.} </p> <h2 id="see-also">See also</h2> <ul><li><a href="/facts/Stochastic_gradient_descent/HbcaYqQP">Stochastic gradient descent</a></li> <li><a href="/facts/Backpropagation/lCsIdKHc">Backpropagation</a></li> <li><a href="/facts/Rescorla%E2%80%93Wagner_model/6ne0Wuq5">Rescorla–Wagner model</a> – the origin of delta rule</li></ul> <h2 id="references">References</h2> <ol> <li id="fn:1"><p>Russell, Ingrid. "The Delta Rule". University of Hartford. Archived from the original on 4 March 2016. Retrieved 5 November 2012. <a href="https://web.archive.org/web/20160304032228/http://uhavax.hartford.edu/compsci/neural-networks-delta-rule.html" target="_blank">https://web.archive.org/web/20160304032228/http://uhavax.hartford.edu/compsci/neural-networks-delta-rule.html</a> <a href="#fnref:1" class="footnote-back-ref">↩</a></p></li> </ol>