Universal approximation theorem

In the <a href="/facts/Mathematics/pxTouaz4">mathematical</a> theory of <a href="/facts/Artificial_neural_networks/6V1jMlkx">artificial neural networks</a>, universal approximation theorems are theorems of the following form: Given a family of neural networks, for each function 
 
 
 
 f
 
 
 {\displaystyle f}
 
 from a certain <a href="/facts/Function_space/lLUE2R1t">function space</a>, there exists a sequence of neural networks 
 
 
 
 
 ϕ
 
 1
 
 
 ,
 
 ϕ
 
 2
 
 
 ,
 …
 
 
 {\displaystyle \phi _{1},\phi _{2},\dots }
 
 from the family, such that 
 
 
 
 
 ϕ
 
 n
 
 
 →
 f
 
 
 {\displaystyle \phi _{n}\to f}
 
 according to some criterion. That is, the family of neural networks is <a href="/facts/Dense_set/nPT9Yxao">dense</a> in the function space.
The most popular version states that <a href="/facts/Feedforward_neural_network/CP0pPGDF">feedforward networks</a> with non-<a href="/facts/Polynomial/Lzak8VVx">polynomial</a> <a href="/facts/Activation_function/S4NImL6L">activation functions</a> are dense in the space of <a href="/facts/Continuous_function/sKbl02pB">continuous functions</a> between two <a href="/facts/Euclidean_space/R2UbzmzM">Euclidean spaces</a>, with respect to the <a href="/facts/Compact_convergence/65F9goFE">compact convergence</a> <a href="/facts/Topology/2EqW47cU">topology</a>.
Universal approximation theorems are existence theorems: They simply state that there exists such a sequence 
 
 
 
 
 ϕ
 
 1
 
 
 ,
 
 ϕ
 
 2
 
 
 ,
 ⋯
 →
 f
 
 
 {\displaystyle \phi _{1},\phi _{2},\dots \to f}
 
, and do not provide any way to actually find such a sequence. They also do not guarantee any method, such as <a href="/facts/Backpropagation/lCsIdKHc">backpropagation</a>, might actually find such a sequence. Any method for searching the space of neural networks, including backpropagation, might find a converging sequence, or not (i.e. the backpropagation might get stuck in a local optimum). 
Universal approximation theorems are limit theorems: They simply state that for any 
 
 
 
 f
 
 
 {\displaystyle f}
 
 and a criterion of closeness 
 
 
 
 ϵ
 >
 0
 
 
 {\displaystyle \epsilon >0}
 
, if there are enough neurons in a neural network, then there exists a neural network with that many neurons that does approximate 
 
 
 
 f
 
 
 {\displaystyle f}
 
 to within 
 
 
 
 ϵ
 
 
 {\displaystyle \epsilon }
 
. There is no guarantee that any finite size, say, 10000 neurons, is enough.

Universal approximation theorem open-in-new

Universal approximation theorem