Transformers

Self-attention-based sequence models that power modern LLMs and vision models.

advanced generative deep-learningnlpattention
self-attentionpositional-encodingoptimisationsequence-modelling

Author @Praneeth-Suresh Updated 2026-06-24

Overview

The Transformer architecture replaces recurrence with multi-head self-attention:

Attention(Q,K,V)=softmax ⁣(QK⊤dk)V\text{Attention}(Q, K, V) = \text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V

Multi-Head Attention

Multiple attention heads capture different relationship patterns:

MultiHead(Q,K,V)=Concat(head1,…,headh)WO\text{MultiHead}(Q,K,V) = \text{Concat}(\text{head}_1, \ldots, \text{head}_h)\mathbf{W}^O

Positional Encoding

Since self-attention is permutation-equivariant, positional information is injected:

PE(pos,2i)=sin⁡ ⁣(pos100002i/d)PE_{(pos, 2i)} = \sin\!\left(\frac{pos}{10000^{2i/d}}\right)

Transformers build on Neural Networks and use Gradient Descent for training. They extend to generative models like VAEs and diffusion models.

Learning Path

What should I learn before Transformers?

Start with neural-networks before studying Transformers.

How does Transformers connect to other AI topics?

Transformers belongs to Generative and connects to deep-learning, nlp, attention, self-attention, positional-encoding, optimisation, sequence-modelling.

Further Reading

  1. Attention Is All You Need – Vaswani et al.