Transformers Architecture Explained

1. What Are Transformers?

A Transformer is a neural network architecture that processes sequential data -- most commonly text -- using a mechanism called self-attention rather than recurrence or convolution. Unlike RNNs that read tokens one at a time and carry a hidden state forward, transformers look at all positions in a sequence simultaneously, computing weighted relationships between every pair of tokens in a single matrix operation.

The architecture was introduced by Ashish Vaswani and colleagues at Google in the landmark 2017 paper "Attention Is All You Need." The title was provocative: it declared that attention alone -- without recurrence, convolution, or gating -- was sufficient for state-of-the-art sequence modeling. The claim proved correct, and transformers now underpin virtually every major natural language processing system, from search ranking to machine translation to the large language models (LLMs) that power conversational AI.

At the highest level, a transformer takes a sequence of tokens, converts each into a dense vector embedding, adds positional information, and then passes those vectors through a stack of identical layers. Each layer refines the representations by allowing tokens to attend to one another, then applies a position-wise feed-forward network. After many such layers, the output representations encode rich contextual meaning that can be used for classification, generation, translation, or any downstream task.

Core Properties

• Parallelism: All tokens are processed simultaneously within a layer, enabling massive GPU utilization.
• Long-range dependencies: Any token can directly attend to any other token in a single step, regardless of distance -- no information bottleneck like an RNN hidden state.
• Stacked layers: Depth (number of layers) allows the model to learn progressively more abstract representations.
• Universal approximation of sequence relationships: The attention mechanism can in principle model arbitrary pairwise interactions, limited only by sequence length and model capacity.

2. Why Transformers Revolutionized NLP

Before 2017, the dominant sequence modeling architectures were LSTMs (Long Short-Term Memory networks) and GRUs (Gated Recurrent Units). These models achieved strong results but suffered from fundamental limitations that transformers overcame:

• Sequential processing: RNNs must process tokens one at a time. A sentence of 512 tokens requires 512 sequential operations, making training on large datasets prohibitively slow on modern hardware.
• Vanishing gradients over long distances: Although LSTMs mitigate this with gating, information from early tokens still degrades as it passes through many recurrent steps.
• Limited context window: RNNs struggle to capture dependencies between tokens separated by more than a few hundred positions.
• Difficult to scale: RNNs do not parallelize, so scaling to billions of parameters and trillions of training tokens is impractical.

Transformers solved all of these problems simultaneously. The attention mechanism provides direct access between any two positions, eliminating the information bottleneck. Matrix-based computation parallelizes perfectly on GPUs and TPUs, enabling training on datasets orders of magnitude larger. And the architecture scales smoothly: adding more layers, heads, or parameters reliably improves performance, a property that enabled the era of large language models.

Timeline of Impact

3. Encoder vs Decoder vs Encoder-Decoder

The original transformer was an encoder-decoder architecture designed for sequence-to-sequence tasks like machine translation. The encoder maps an input sequence into a sequence of contextualized representations; the decoder then generates an output sequence auto-regressively, attending both to its own previous outputs (causal self-attention) and to the encoder's representations (cross-attention).

Over time, researchers found that each half of the architecture is useful independently, leading to three major families:

3.1 Encoder-Only (e.g., BERT)

Uses only the encoder stack. Every token attends to every other token (bidirectional / full self-attention). Produces a contextualized embedding for each position. Best suited for understanding tasks: classification, named entity recognition, question answering, sentence embeddings.

• Attention pattern: Full (bidirectional) -- every token sees all others.
• Training objective: Masked language modeling (MLM) -- predict randomly masked tokens using both left and right context.
• Strength: Rich contextual understanding; strong for analysis tasks.
• Limitation: Not designed for open-ended generation; needs a classification head or fine-tuning for specific tasks.

3.2 Decoder-Only (e.g., GPT family, Llama, Claude)

Uses only the decoder stack with causal (masked) self-attention. Each token can only attend to itself and previous tokens, never to future ones. This auto-regressive structure is ideal for generation tasks: given a prefix, predict the next token, then the next, and so on.

• Attention pattern: Causal (lower-triangular) -- token i can only attend to tokens 0..i.
• Training objective: Next-token prediction (causal language modeling).
• Strength: Simple, scalable, naturally generative. This is the dominant architecture for modern LLMs.
• Limitation: Each token only sees left context (unidirectional), which can be suboptimal for tasks that benefit from bidirectional understanding.

3.3 Encoder-Decoder (e.g., T5, BART, original Transformer)

Uses both stacks. The encoder processes the full input sequence with bidirectional attention; the decoder generates output auto-regressively, using causal self-attention for its own outputs and cross-attention to attend over encoder outputs. Ideal for sequence-to-sequence tasks: translation, summarization, and any task framed as input-to-output mapping.

• Attention pattern: Encoder uses full attention; decoder uses causal self-attention plus cross-attention to encoder outputs.
• Training objective: Denoising / span-corruption (T5) or seq2seq reconstruction (BART).
• Strength: Naturally handles input-to-output transformation; the encoder fully understands the source before generation begins.
• Limitation: More parameters needed for two stacks; less commonly used for general-purpose LLMs compared to decoder-only.

Architecture Comparison Summary

4. The Self-Attention Mechanism

Self-attention is the heart of the transformer. Given a sequence of token embeddings, self-attention computes a new representation for each token by taking a weighted average of all other token representations, where the weights reflect how relevant each other token is to the current one.

4.1 Query, Key, and Value Matrices

The mechanism borrows terminology from information retrieval. For each token, the model produces three vectors via learned linear projections:

• Query (Q): "What am I looking for?" -- a vector representing what information the current token wants to retrieve.
• Key (K): "What do I contain?" -- a vector representing what information this token offers to others.
• Value (V): "Here is my content." -- the actual content vector that will be mixed into the output if this token is attended to.

For an input matrix X of shape (sequence_length × d_model), three learned weight matrices W_Q, W_K, and W_V (each of shape d_model × d_k) produce:

Here, n is the sequence length, d_model is the embedding dimension, and d_k / d_v are the per-head attention dimensions (typically d_model / num_heads).

4.2 Scaled Dot-Product Attention

The attention output for the entire sequence is computed in a single matrix operation:

Breaking this down step by step:

1. Dot product Q · K^T: Compute the similarity (dot product) between every query and every key. This produces an n × n matrix of raw attention scores, where entry (i, j) measures how much token i should attend to token j.
2. Scaling by 1/sqrt(d_k): Divide each score by the square root of the key dimension. This prevents the dot products from growing large in high dimensions, which would push the softmax into regions with extremely small gradients.
3. Softmax: Apply softmax along each row (over the key dimension) to convert raw scores into a probability distribution. Each row now sums to 1, representing how the current token distributes its attention across all positions.
4. Multiply by V: Multiply the n × n attention weight matrix by the n × d_v value matrix to produce the final n × d_v output. Each output token is a weighted sum of all value vectors.

4.3 Causal Masking

In decoder models, we must prevent tokens from attending to future positions. This is achieved by adding a causal mask to the attention scores before softmax. The mask sets all entries above the diagonal to negative infinity (or a very large negative number), so that after softmax, those positions receive exactly zero attention weight:

4.4 Complexity Analysis

The O(n²) memory for the attention matrix is the primary bottleneck for long sequences. A 128K-token context window requires storing a 128K × 128K attention matrix, which is why techniques like FlashAttention, sliding window attention, and linear attention variants have become critical at scale.

5. Multi-Head Attention

Instead of performing a single attention function with d_model-dimensional keys and values, the transformer runs h attention functions (heads) in parallel, each with reduced dimension d_k = d_model / h. The outputs are concatenated and projected back to d_model.

5.1 Why Multiple Heads?

Different heads learn to attend to different types of relationships. Research on attention head analysis has shown that individual heads specialize:

• Some heads focus on syntax -- attending to the head word, determiner, or subject of a verb.
• Some heads capture coreference -- linking pronouns to their referents.
• Some heads are positional -- attending to the immediately preceding or following token.
• Some heads learn rare / semantic patterns -- connecting entities across long distances.

By running multiple heads in parallel and concatenating, the model captures a richer set of relationships than any single attention function could.

5.2 Typical Head Counts

6. Positional Encoding

Self-attention is permutation-invariant -- it computes the same result regardless of the order of tokens, because the attention weights depend only on the content of Q, K, V, not on position. This is a problem: "the dog chased the cat" and "the cat chased the dog" would produce identical representations without positional information.

Positional encoding solves this by injecting position-dependent signals into the input embeddings so that the attention mechanism can distinguish token order. Several approaches have been developed:

6.1 Sinusoidal Positional Encoding (Original)

The original transformer uses fixed sine and cosine functions of different frequencies:

Where pos is the position (0, 1, 2, ...) and i is the dimension index. The resulting positional encoding vector is added to the token embedding before the first layer.

• Pros: No learned parameters; can extrapolate to longer sequences than seen in training (in theory).
• Cons: Extrapolation is poor in practice; fixed encoding may not be optimal.

6.2 Learned Positional Embeddings

Instead of a fixed formula, the model learns a lookup table of position embeddings -- one vector per position, up to a maximum sequence length. BERT and GPT-2 use this approach.

• Pros: Flexible; the model learns optimal position representations for the training distribution.
• Cons: Cannot generalize beyond the maximum sequence length seen in training. If trained with max_len=2048, the model has no embeddings for position 2049+.

6.3 Rotary Position Embedding (RoPE)

RoPE, introduced by Su et al. in 2021, encodes position by rotating the query and key vectors in a multi-dimensional space. The rotation angle depends on position, and the relative position between two tokens is implicitly encoded in the angle difference of their rotations.

• Pros: Encodes relative position; good extrapolation with techniques like NTK-aware scaling; widely adopted in Llama, Mistral, Qwen, and most modern LLMs.
• Cons: Requires careful implementation; extrapolation still degrades beyond a few times the training context length without scaling adjustments.

6.4 ALiBi (Attention with Linear Biases)

ALiBi, introduced by Press et al. in 2021, takes a radically different approach: it does not add positional embeddings to the input at all. Instead, it adds a static, non-learned bias to the attention scores that penalizes attention to distant tokens:

• Pros: Excellent extrapolation -- models trained at 1024 context perform well at 2048+ without retraining; no positional embedding parameters.
• Cons: Inductive bias toward local attention may not be ideal for all tasks; less widely adopted than RoPE.

6.5 Comparison of Positional Encoding Methods

7. Feed-Forward Networks, Layer Normalization & Residual Connections

Self-attention is only one of two sub-layers in each transformer block. The second is a position-wise feed-forward network (FFN). Additionally, residual connections and layer normalization are applied around both sub-layers. These components are essential for training deep transformer stacks.

7.1 Position-Wise Feed-Forward Network

After multi-head attention, each position's vector is independently passed through a two-layer MLP with a non-linear activation:

The hidden dimension d_ff is typically 4× the model dimension (e.g., d_model=4096 → d_ff=16384). This makes the FFN the most parameter-heavy component of a standard transformer -- approximately 2/3 of the total parameters are in the FFN layers.

Activation functions have evolved:

• ReLU (original transformer): max(0, x). Simple, effective.
• GELU (BERT, GPT-2): x · Φ(x), where Φ is the Gaussian CDF. Smoother than ReLU.
• SwiGLU (Llama, PaLM): Uses a gating mechanism with Swish activation. Improves performance at similar parameter counts.
• GeGLU (T5 v1.1): GELU-based gating variant.

7.2 Residual Connections

Each sub-layer (attention and FFN) is wrapped with a residual connection:

Residual connections (from the ResNet architecture) allow gradients to flow directly through the network during backpropagation, mitigating the vanishing gradient problem in deep stacks. Without them, training transformers beyond a few layers is extremely difficult.

7.3 Layer Normalization

Layer normalization normalizes the activations across the feature dimension (d_model) for each token independently, stabilizing training:

Two placement strategies exist:

• Post-LN (original transformer): LayerNorm is applied after the residual addition: output = LayerNorm(x + Sublayer(x)). Training is less stable at large scale without learning rate warmup.
• Pre-LN (modern standard): LayerNorm is applied before the sub-layer: output = x + Sublayer(LayerNorm(x)). More stable training; no warmup needed. Used by GPT-2, Llama, and most modern LLMs.

7.4 Complete Transformer Layer Formula

8. The Transformer Block Diagram

Below is a text-based diagram of the complete encoder-decoder transformer architecture from the original paper, showing how data flows through the layers.

Decoder-Only Block (GPT / Llama style)

9. How Tokens Flow Through a Transformer (Step by Step)

Let us trace exactly what happens when the input sentence "The cat sat" is processed by a decoder-only transformer (the most common modern architecture). We assume a model with d_model=512, 8 heads, 12 layers.

Step 1: Tokenization

The input string is split into tokens using a subword tokenizer (BPE, WordPiece, or SentencePiece). Each token maps to an integer ID from the vocabulary.

Step 2: Token Embedding

Each integer ID is looked up in an embedding matrix of shape (V × d_model), producing a d_model-dimensional vector for each token.

Step 3: Positional Encoding

Positional information is added. For RoPE (used in modern models), this is applied inside the attention computation by rotating Q and K vectors. For learned or sinusoidal embeddings, a position vector is added to the embedding here.

Step 4: Layer 1 -- Multi-Head Self-Attention

The input passes through the first transformer layer. First, RMSNorm/LayerNorm is applied, then the normalized output is projected into Q, K, V for each of the 8 attention heads.

Step 5: Layer 1 -- Feed-Forward Network

Step 6: Layers 2 through N

The same process repeats for each of the remaining 11 layers. Each layer refines the representations:

Early layers tend to capture syntactic and local patterns; middle layers capture semantic relationships; later layers produce task-relevant representations. This hierarchical feature emergence is a well-documented phenomenon in transformer interpretability research.

Step 7: Output Projection & Softmax

The final hidden states are projected back to vocabulary size via the output (unembedding) matrix, then softmax converts to probabilities:

10. Popular Transformer Models: GPT, BERT & T5

10.1 GPT (Decoder-Only, Generative Pre-Training)

The GPT family (Generative Pre-trained Transformer) from OpenAI uses a decoder-only architecture with causal self-attention. The training objective is simple: next-token prediction. Given a sequence of tokens, predict the next one. This unsupervised objective can be applied to any text data, enabling training on vast web-scale corpora.

• GPT-1 (2018): 117M parameters, 12 layers, d_model=768. Demonstrated that generative pre-training + task fine-tuning beats task-specific architectures.
• GPT-2 (2019): 1.5B parameters, 48 layers, d_model=1600. Showed zero-shot task performance from a purely generative model. Initially deemed too dangerous to release.
• GPT-3 (2020): 175B parameters, 96 layers, d_model=12288. Introduced in-context learning: the model performs new tasks from examples in the prompt, without weight updates.
• GPT-4 (2023): Architecture details not publicly confirmed; widely believed to be a Mixture of Experts model with ~1.8T total parameters. Multimodal (vision + text).

The key insight of the GPT line: a decoder-only transformer trained on next-token prediction at sufficient scale develops emergent capabilities -- few-shot learning, code generation, reasoning, instruction following -- that were not explicitly trained.

10.2 BERT (Encoder-Only, Bidirectional Understanding)

BERT (Bidirectional Encoder Representations from Transformers) from Google uses an encoder-only architecture with full (bidirectional) self-attention. It is trained with two objectives:

• Masked Language Modeling (MLM): Randomly mask 15% of tokens and predict them using both left and right context. This forces the model to build truly bidirectional representations.
• Next Sentence Prediction (NSP): Predict whether two sentences appear consecutively in the source text. (Later research found NSP contributes little; RoBERTa dropped it.)

BERT is not generative -- it produces contextual embeddings, not token sequences. It excels at understanding tasks and is typically fine-tuned with a task-specific classification head. BERT and its variants (RoBERTa, ALBERT, DeBERTa) dominated NLP benchmarks from 2018-2021 and remain widely used for embeddings, classification, and retrieval.

• BERT-Base: 110M parameters, 12 layers, 12 heads, d_model=768.
• BERT-Large: 340M parameters, 24 layers, 16 heads, d_model=1024.
• RoBERTa: Same architecture as BERT-Base; trained with more data, longer, larger batches, and dynamic masking. Significantly outperforms BERT.
• DeBERTa: Adds disentangled attention (separating content and position). SOTA on SuperGLUE for its parameter range.

10.3 T5 (Encoder-Decoder, Text-to-Text)

T5 (Text-to-Text Transfer Transformer) from Google uses a full encoder-decoder architecture and frames every NLP task as a text-to-text problem: the input is text, the output is text. Translation, summarization, classification, QA, and even regression tasks are all cast as generating a text sequence.

• T5-Small: 60M parameters, 6+6 layers.
• T5-Base: 220M parameters, 12+12 layers.
• T5-Large: 770M parameters, 24+24 layers.
• T5-3B / T5-11B: 3B and 11B parameters. T5-11B was among the largest models at its release in 2019.

T5's training objective is span corruption: random spans of text are replaced with sentinel tokens, and the model must generate the original text. This is a denoising objective that trains both the encoder (to understand corrupted input) and the decoder (to generate the correct output).

11. Model Scaling: Parameters, Layers, Hidden Size & Heads

One of the most important properties of transformers is that they scale predictably. The Kaplan et al. (2020) and Hoffmann et al. (2022, "Chinchilla") scaling laws established that model performance (loss) follows a power-law relationship with model size, dataset size, and compute. This predictability is what enabled the LLM era: you can estimate how good a 100B model will be by measuring a 1B model.

11.1 Parameter Count Breakdown

For a decoder-only transformer with L layers, d_model hidden size, h heads, d_ff = 4·d_model, and vocabulary V:

11.2 Scaling Relationships

The Kaplan scaling law states that the loss L scales as a power law with parameters N, data D, and compute C:

11.3 Scaling Configurations of Major Models

11.4 Depth vs Width

When scaling, there is a choice between increasing depth (more layers) and increasing width (larger d_model). Empirical research suggests:

• Depth and width both matter, but depth is generally preferred for reasoning and compositional tasks.
• Very wide but shallow models (few layers, huge d_model) can be less parameter-efficient.
• Very deep but narrow models face optimization challenges despite residual connections.
• The ratio d_model / layers typically ranges from 64 to 128 for well-tuned models. GPT-3 has 12288/96 = 128; Llama-2-70B has 8192/80 = ~102.

12. Mixture of Experts (MoE) Architecture

A Mixture of Experts (MoE) model replaces the standard feed-forward network in (some or all) transformer layers with a set of multiple expert FFNs and a router (gate) that dynamically selects which expert(s) to use for each token. This dramatically increases total parameter count while keeping active compute per token roughly constant.

12.1 How MoE Works

Typical configurations use k=2 (top-2 routing) with 8-64 experts. This means each token uses 2 expert FFNs, but the model has 8-64 FFNs worth of parameters available. A 67B MoE model might have only ~13B active parameters per token, making it as fast as a 13B dense model but with the capacity of a much larger model.

12.2 Benefits of MoE

• Parameter efficiency: Total parameters can be 4-16× the active parameters, giving the model more "knowledge capacity" without proportional inference cost.
• Faster training and inference: Active FLOPs per token are similar to a much smaller dense model.
• Specialization: Different experts can learn different domains (code, math, languages, factual knowledge), improving performance on diverse tasks.
• Scalability: Adding more experts increases capacity with minimal compute increase -- a different scaling axis than depth or width.

12.3 Challenges

• Memory cost: All expert parameters must reside in GPU memory, even though only a few are used per token. A 1T-parameter MoE model requires enormous VRAM.
• Load balancing: Without regularization, the router may send most tokens to a few experts, leaving others underutilized. An auxiliary load-balancing loss is typically added to encourage even distribution.
• Communication overhead: In distributed training, tokens routed to experts on different GPUs require all-to-all communication, which can bottleneck training.
• Fine-tuning instability: MoE models can overfit during fine-tuning; experts may collapse (some experts stop being used).

12.4 Notable MoE Models

12.5 MoE vs Dense: Trade-off Summary

13. Why Transformers Replaced RNNs / LSTMs

The replacement of RNNs by transformers was not merely incremental -- it was a paradigm shift. Here is a detailed comparison of the fundamental limitations of RNNs and how transformers address each one.

13.1 Parallelism

RNNs: Process tokens sequentially. For a sequence of length n, n sequential matrix multiplications are required. This cannot be parallelized across the time dimension, making training on long sequences extremely slow on GPUs/TPUs.

Transformers: Process all tokens in parallel within each layer. The attention computation is a single batched matrix multiplication. This achieves near-peak hardware utilization and enables training on datasets orders of magnitude larger.

13.2 Long-Range Dependencies

RNNs: Information from token 1 must pass through every intermediate hidden state to reach token 100. Despite LSTM gating, the effective path length is O(n), and distant information degrades.

Transformers: Any token can directly attend to any other token in a single attention operation. The path length between any two positions is O(1) within a layer. This fundamentally changes the model's ability to capture long-range dependencies.

13.3 Gradient Flow

RNNs: Backpropagation through time (BPTT) requires unrolling the network over the full sequence. Gradients must traverse every time step, making vanishing/exploding gradients a persistent challenge.

Transformers: Gradients flow through residual connections directly across layers. The architecture is effectively a deep feed-forward network (not recurrent), so standard backpropagation applies without time-unrolling.

13.4 Context Length

RNNs: Practical context is limited to a few hundred tokens. Beyond that, the hidden state becomes an information bottleneck, and training becomes unstable.

Transformers: Context is limited by the O(n²) attention cost and memory, but modern techniques (FlashAttention, ring attention, sparse attention) have extended practical context to 128K-1M+ tokens -- orders of magnitude beyond what RNNs could achieve.

13.5 Transfer Learning

RNNs: Transfer learning was limited. Pre-trained RNN embeddings existed (ELMo, ULMFiT), but the architecture did not scale well enough for massive pre-training to produce general-purpose capabilities.

Transformers: Scale beautifully with data and parameters. Pre-training a transformer on web-scale text produces general-purpose capabilities (reasoning, translation, coding) that transfer to virtually any text task through prompting or light fine-tuning.

13.6 Direct Comparison Table

14. Comparison Table of Major Transformer Models

The following table provides a comprehensive comparison of the most influential transformer models, their architectures, training data, and key characteristics.

Key Architectural Trends Over Time

• 2017-2019: Encoder-decoder and encoder-only models dominate (original Transformer, BERT, T5). Decoder-only is a niche choice (GPT-1/2).
• 2020-2022: Decoder-only becomes dominant (GPT-3). Scale becomes the primary lever. Scaling laws published.
• 2022-2023: RMSNorm, RoPE, and SwiGLU become standard (Llama). Open-weight models proliferate. MoE goes mainstream (Mixtral).
• 2024+: MoE is the frontier architecture (DeepSeek-V3, Mixtral, rumored GPT-4). Context lengths reach 128K-1M+. MLA and sparse attention reduce memory. Data quality and quantity (15T+ tokens) become the key differentiator.

Summary & Key Takeaways

The transformer architecture, now nearly a decade old, remains the foundation of modern AI. Its key design principles are:

1. Self-attention enables direct, O(1)-path interactions between any two tokens, replacing the sequential bottleneck of RNNs.
2. Parallelism across the sequence dimension enables efficient GPU/TPU utilization and training on web-scale data.
3. Multi-head attention allows the model to capture multiple types of relationships simultaneously.
4. Positional encoding injects order information, with RoPE being the modern standard for its extrapolation properties.
5. Residual connections and normalization (Pre-LN or RMSNorm) enable training of deep stacks (80-120+ layers).
6. Scaling laws provide predictable performance improvements with more parameters, data, and compute -- with the Chinchilla ratio of ~20 tokens/parameter being the key calibration.
7. Mixture of Experts decouples parameter count from compute, enabling trillion-parameter-class models with efficient inference.
8. Decoder-only has won the architecture war for general-purpose LLMs, due to simplicity, scalability, and the power of next-token prediction as a universal training objective.

Understanding these mechanisms is essential for anyone working with LLMs -- whether fine-tuning, building retrieval pipelines, optimizing inference, or developing new architectures. The transformer is not just a model; it is the scaffolding upon which the entire modern AI ecosystem is built.