Neural Networks
Multi-layer parametric function approximators trained via backpropagation.
Overview
The core architecture of modern deep learning is the neural network. Also called Multi-Layer Perceptrons, they consist of a set of layers. Each layer is mathematically represented as a vector. Looking at it this way, one layer is transformed to the next by multiplying by a matrix and adding a bias vector.
In Linear Algebra lingo, this is called an Affine Transformation.
Thus, we say a feedforward neural network computes a composition of affine transformations and non-linear activations:
Universal Approximation
This is a key result about Neural Networks that explains why they are so useful. It goes something like this:
A single hidden-layer network with sufficient width can approximate any continuous function on a compact set to arbitrary precision (Cybenko, 1989).
In supervised deep learning, we are attempting to find a function mapping from inputs () to outputs (). According to universal approximation, given enough training time and data, the neural network will eventually be able to learn the underlying function.
Backpropagation
Gradients are computed efficiently via the chain rule applied layer-by-layer:
Related Topics
Neural networks form the foundation of Transformers and are optimised using Gradient Descent.
Learning Path
What should I learn before Neural Networks?
Start with gradient-descent before studying Neural Networks.
How does Neural Networks connect to other AI topics?
Neural Networks belongs to Deep Learning and connects to deep-learning, representation-learning, backpropagation, activation-functions, optimisation, universal-approximation.