Interview Prep · ML Infra

Model architectures:
from N-gram to Transformer

First a timeline showing what each architecture fixed in its predecessor, then the Transformer taken apart in ten steps, each with a diagram, and attention you can compute by hand. After that come shapes and parameter counts, a comparison table, staff-level interview questions, an exercise you can run, and a one-page cheat sheet.

Timeline

From counting to attention: what each step fixed in the one before

1. From the 1950sN-gram language modelsStatisticalCount which word most often follows the previous few.▼
2. 1990RNNRecurrentRead one word at a time, with a hidden state that remembers what has been read.▼
3. 1997 / 2014LSTM / GRURecurrentAdd "gates" to the RNN so that information can travel further unchanged.▼
4. 2012 / 2015CNN and ResNetConvolutionalThe vision branch. What it left the Transformer is the residual connection.▼
5. 2013Word2VecRepresentationTurn every word into a vector, so that words with similar meanings have similar vectors.▼
6. 2014Seq2Seq and attentionRecurrentOne RNN reads and another writes; attention lets the writer look back at the source.▼
7. 2017TransformerAttentionDrop the RNN, use only attention, and compute all positions at once.▼

   Origin

   Google, the paper "Attention Is All You Need", Vaswani and seven co-authors

   The problem it set out to solve

   RNNs cannot be parallelized and are slow to train; information between distant words has to travel step by step. The task at the time was machine translation.

   • Inputone vector per token
   • First sublayer
     • Norm
     • masked self-attentioneach position sees only itself and earlier ones
     • Add back the inputresidual connection
   • Second sublayer
     • Norm
     • MLPwiden to 4x, then narrow back
     • Add back the inputresidual connection
   • Outputsame shape as the input, passed to the next layer

   How it works

   • Self-attention: every word looks directly at every word in the sentence and takes in their information weighted by relevance. Any two words are one step apart.
   • The computations for different positions do not depend on each other and can be done in one go, which suits GPUs.
   • Attention itself does not know word order, so a positional encoding is added for each position.
   • Each layer is attention followed by an MLP. Both are wrapped in a residual connection and LayerNorm.
   • The original paper is an encoder plus a decoder. The base model has 6 layers, 512-dimensional vectors, 8 heads and about 65 million parameters.

   Training, step by step

   1. Feed in all positions of a passage at once.
   2. Use a mask so that no position sees what comes after it.
   3. Every position predicts the next word.
   4. Add up the error over all positions and update the parameters.

   Inference, step by step

   1. Feed in the text so far and get the last position's probabilities for the next word.
   2. Choose a word by probability.
   3. Append it and compute again.
   4. The keys and values already computed are kept in a cache and not recomputed.

   Strengths

   • All positions are computed at once, so training makes full use of GPUs.
   • Any two words are one step apart, so long-range dependencies are easy to learn.
   • The structure is uniform, and results improve as the model and the data grow.

   Weaknesses

   • The cost of attention grows with the square of the sequence length, so long texts are expensive.
   • It has no built-in notion of order and relies on positional encoding to supply one.
   • It brings few built-in assumptions and needs a great deal of data to show its strength. On images, with little data, it does worse than a CNN.

   Where it is used, and where it still is

   Nearly all large language models today, and most vision and speech models.

   What replaced it, and why

   Not replaced. Newer structures are mostly modifications of it or are mixed with it.

   In an interview, one sentence

   A sequence model using only attention: parallel computation, and any two positions directly connected.

   In an interview, two minutes

   The Transformer removes the RNN and uses only self-attention. Every word looks directly at every other word and takes information by relevance, so any two words are one step apart and all positions can be computed at once. Because attention does not know order, positional encoding is added separately. Each layer is attention plus an MLP, wrapped in residual connections and normalization. The price is a cost that grows with the square of the length. It became mainstream because parallelism let it train on large data, and its results keep improving with scale.

   Worth adding

   A whole section below takes it apart. The three things interviewers most often press on are: why divide by the square root of d, why positional encoding is needed, and why the cost is quadratic.

8. 2018BERTAttentionEncoder only, able to see in both directions, good at understanding.▼
9. 2018GPTAttentionDecoder only, able to see only what came before, good at generating. Nearly all of today's large models belong to this branch.▼
10. 2019–2024Changes in modern LLMsAttentionThe skeleton is still 2017's, but nearly every part has been replaced.▼
11. 2020ViTAttentionCut the image into patches, treat them as words, and use a Transformer directly.▼
12. 2020 / 2022Diffusion and DiTGenerativeStart from pure noise and remove it step by step to get an image.▼
13. 2023Mamba (state space models)Next generationCost proportional to length, aiming to replace attention on long sequences.▼

Five concepts first

Every step on the timeline trades off these five things

Token and embedding

Text is first cut into tokens (words or pieces of words), and each token is looked up in a table to get a vector.

A model can only compute on numbers. All later computation happens on these vectors, and the model's "width", d_model, is the dimension of this vector.

How far the context reaches

When predicting a word, how far back the model can reach for information.

For an N-gram it is a few words; for an RNN it is unlimited in theory, but in practice it forgets; for a Transformer it is the whole context window. This thread runs through the entire timeline.

Whether it can be parallelized

Whether the positions in a sentence must be computed one after another, or can be computed at the same time.

GPUs are good at computing many things at once. An RNN must compute in order and a Transformer can compute them all at once, which is the direct reason the Transformer can make use of large data.

Residual connection

A layer's output = the layer's input + the change the layer computed.

It leaves a straight path for gradients; without it a network of dozens of layers cannot be trained. You can picture the whole model as one main line to which every layer adds a little.

Autoregressive

Generate one token at a time, append it to the input, then generate the next.

This is how GPT-style models generate text, and it is why longer outputs are slower and a KV cache is needed.

Main topic · the figure from the paper first

The classic 2017 structure: encoder plus decoder

This is Figure 1 of "Attention Is All You Need". When interviews and textbooks say "the structure of the Transformer", this is usually what they mean. The task then was translation: the encoder on the left reads the source, and the decoder on the right writes the translation. Read the figure from the bottom up.

The figure is wide; scroll sideways to see all of it.

In the paper's figure, each Add & Norm also has a line beside it that goes around the sublayer. That is the residual connection. It is left out here for clarity and written in the boxes' notes instead.

Translating one sentence: how the data moves

1. The source "我 爱 猫" (I love cats) enters at the bottom left, becomes vectors, and gets positional encoding added.
2. It passes through 6 encoder layers. Each does self-attention first (the three words look at each other), then an MLP. Out come three vectors, one per word, now carrying information from the whole sentence. The encoder runs only once.
3. The decoder starts at the bottom right. At the first step its only input is a "start" symbol.
4. Each decoder layer does three things: self-attention among the words written so far, with a mask so it sees only earlier ones; then cross-attention, looking back at the three vectors the encoder output; and finally an MLP. All 6 decoder layers look at the output of the encoder's last layer.
5. The Linear and Softmax at the top give the probability of the next word, from which "I" is chosen. The paper uses beam search, keeping several candidates at once and not just the single most probable word at each step.
6. "I" is appended to the decoder's input, and steps 4 and 5 repeat until the end symbol is output.

How cross-attention differs from self-attention

The formula is exactly the same; the only difference is where Q, K and V come from. In self-attention all three come from the same sequence of words. In cross-attention, Q comes from the vector at each position in this decoder layer (just after the masked self-attention), and K and V come from the encoder's output. The meaning: the decoder takes "what I am about to write" and goes to the source to find the related words. This is the Transformer version of the 2014 attention; follow-up 4 below has an animation.

Where each part of the figure is covered below

Remove the left half of this figure and the attention in the middle of the decoder, and what remains is the skeleton of today's large language models; the changes to the parts are in the table above. The ten steps below follow this reduced structure, because it has fewer parts and is the shape actually in use today. The part each step covers is the same part as in this figure.

Main topic · the Transformer taken apart

Ten steps, from one sentence to the next token

Four follow-ups

Points the timeline passed over in a sentence, expanded

A walk through the shapes

What shape a 5-token sentence takes inside GPT-2 small

Getting the shape right at every step in an interview shows real understanding. GPT-2 small: 12 layers, d_model 768, 12 heads, vocabulary 50,257.

Note the row for attention weights: it is 5×5. With a sentence of length n it is n×n. That is where "the cost grows with the square of the length" comes from.

Parameter count

Where the parameters are

Pick a model size and see how the parameters are distributed. For all four sizes this formula gives exactly the parameter count obtained from Hugging Face's GPT-2 implementation; small is 124,439,808. OpenAI's paper says 117M, which differs from what the released weights count to.

A way to do it in your head: each layer is about 12 × d² (attention is 4 matrices of d×d and the MLP the equivalent of 8), plus the embedding of vocabulary × d.

Why it won

Side by side with RNNs and CNNs

This table comes from the original paper. n is the sequence length, d the vector width, and k the size of the convolution window.

The last two columns are the key. "Steps that must be done in order" being 1 means it can be parallelized; "furthest distance" being 1 means even the most distant two words influence each other directly. The price is in the first column: when n is very large, the computation of self-attention exceeds that of RNNs and convolution. The log_k(n) in the convolution row refers to dilated convolution; ordinary convolution needs n/k stacked layers for the two most distant words to meet.

Three ways of using it

The 2017 original and today's LLMs

Side by side

The same dimensions, in one table

Where the industry is heading · checked October 2026

What people are working on now

• MoE has become common practice in large models

  Replace each layer's MLP with many experts, and send each token through only a few. Total parameters can be very large while each computation uses few. DeepSeek-V3 and the largest Qwen3 model, at 235B, both have this structure.

• Mixing attention with linear-cost layers

  For longer contexts, some models no longer use full attention in every layer; most layers use a linear-cost structure, with a full attention layer every few layers. Sebastian Raschka's list of papers from June 2026 says this alternating hybrid structure was one of the more popular directions that year; the examples in his November 2025 article are Qwen3-Next and Kimi Linear: three linear attention layers to one full attention layer.

• Making attention itself cheaper

  GQA lets several heads share keys and values and shrinks the KV cache. FlashAttention avoids writing the whole attention matrix to GPU memory. These do not change the quadratic order, but they make long contexts feasible in practice.

• Generating text with diffusion

  Autoregression produces one token at a time. Diffusion language models aim to generate many tokens in parallel in one go and then revise repeatedly. They have not replaced autoregression, and they are an active research direction.

Common misconceptions

These sound right but are not

• Misconception: The Transformer invented attention.

  Attention was already used in RNN-based machine translation in 2014. The Transformer's contribution was removing the RNN and using only attention.

• Misconception: Attention weights are the model's explanation; a large weight means that word is important.

  A weight only says where this one head in this one layer took information from. A model has many layers and many heads, with MLPs and residuals after them, and one head's weights cannot be taken as an explanation of the model's decision.

• Misconception: The Transformer is parallel, so generation is parallel too.

  Parallelism refers to training, and to processing input that is already given. Generation still goes one token at a time, each token depending on the one before.

• Misconception: BERT and GPT are two different architectures.

  They use the same kind of layer. The difference is whether attention can see later words, and whether the training task is filling in blanks or predicting the next word.

• Misconception: Diffusion is an alternative to the Transformer.

  One is a method of generation and the other a network structure. DiT uses a Transformer as the denoising network in diffusion.

• Misconception: Most of the parameters are in attention.

  In a standard layer the MLP has about twice the parameters of attention: attention is 4 matrices of d×d, and the MLP is the equivalent of 8.

• Misconception: Positional encoding is a fixed vector added to the input.

  That is the original paper. The mainstream today is RoPE: nothing is added to the input, and the query and key are rotated inside attention at every layer.

Staff-level interview questions

Answer first, then open

Hands-on exercise · about 5 minutes

Write self-attention once in NumPy

Requires Python and NumPy. Save the code below as attention.py and run it. The core is only four lines.

import numpy as np

np.set_printoptions(precision=3, suppress=True)

tokens = ["cat", "chases", "mouse", "it"]

# The query, key and value vector of each word. These are hand-written
# 2-dimensional vectors so the arithmetic can be done in your head; in a real
# model they come from the embedding times three trained matrices.
Q = np.array([[1, 0], [1, 1], [0, 1], [2, 0.5]])
K = np.array([[2, 0], [0, 1], [1, 1], [0, 0.5]])
V = np.array([[1, 0], [0, 1], [0.5, 0.5], [0, 0]])


def softmax(x):
    e = np.exp(x - x.max(axis=-1, keepdims=True))
    return e / e.sum(axis=-1, keepdims=True)


def attention(Q, K, V, causal):
    d = K.shape[-1]
    scores = Q @ K.T / np.sqrt(d)          # step 1: similarity of every query with every key
    if causal:                              # step 2: a word sees only itself and earlier words
        hidden = np.triu(np.ones_like(scores), k=1).astype(bool)
        scores = np.where(hidden, -np.inf, scores)
    weights = softmax(scores)               # step 3: each row becomes weights that sum to 1
    return weights, weights @ V             # step 4: add up the values by weight


weights, output = attention(Q, K, V, causal=False)
print("Weights without a mask (each row is what one word attends to)")
print(weights)
print('The row for "it":', dict(zip(tokens, weights[3].round(3).tolist())))
print('Output vector for "it":', output[3])

weights, output = attention(Q, K, V, causal=True)
print("\nWeights with the causal mask")
print(weights)
print("Sum of each row:", weights.sum(axis=1))

Actual output on NumPy 1.26:

Weights without a mask (each row is what one word attends to)
[[0.505 0.123 0.249 0.123]
 [0.352 0.174 0.352 0.122]
 [0.154 0.313 0.313 0.22 ]
 [0.666 0.056 0.231 0.047]]
The row for "it": {'cat': 0.666, 'chases': 0.056, 'mouse': 0.231, 'it': 0.047}
Output vector for "it": [0.782 0.171]

Weights with the causal mask
[[1.    0.    0.    0.   ]
 [0.67  0.33  0.    0.   ]
 [0.198 0.401 0.401 0.   ]
 [0.666 0.056 0.231 0.047]]
Sum of each row: [1. 1. 1. 1.]

• The numbers here are exactly those in the worked attention example in step 4 above, from the same vectors. You can check them cell by cell against that table.
• With the mask, the upper right of the matrix is all 0: each word sees only itself and earlier words. The first row has only itself left, so its weight is 1.
• "it" is the last word and nothing comes after it, so its row is the same with or without the mask.
• Try changing things: remove the division by np.sqrt(d) and see whether the row for "it" concentrates more on "cat"; change the vector for "it" in Q to [0.5, 2] and see which word it turns to.

Cheat sheet

Ten minutes before the interview

N-gram

Counting. Sees only the previous few words; no similarity between words.

Word2Vec

Words become vectors. One vector per word, ignoring context.

RNN

Reads in order with one hidden state. Vanishing gradients, no parallelism.

LSTM / GRU

Gates and a straight path; remembers further. Still no parallelism.

Seq2Seq + Attention

The encoder reads, the decoder writes, attention looks back at the source.

ResNet

Output = input + change. What makes deep networks trainable.

Transformer

Attention only. Parallel, any two words one step apart, at a cost that grows with the square of the length.

self-attention

softmax(QKᵀ / √d_k) · V. Dividing by √d_k keeps the softmax from saturating and the gradient from vanishing.

BERT

Encoder-only, bidirectional, fill-in-the-blank pre-training. Understanding and retrieval.

GPT

Decoder-only, causal mask, predict the next word. Generation.

Modern LLMs

Pre-Norm, RMSNorm, RoPE, SwiGLU, GQA, MoE, KV cache.

Parameter count

About 12 × layers × d² + vocabulary × d.

Diffusion

Denoise step by step. A method of generation; the network can be a Transformer.

Mamba

Cost proportional to length; chooses what to keep in its state based on the input. Often mixed with attention.

Sources

"Where the industry is heading" was checked in October 2026. It may have changed since. The interactive numbers on the page (attention weights, positional encoding, parameter counts) are all computed on the spot and not hard-coded.

• Attention Is All You Need (Vaswani et al., 2017)
• Neural Machine Translation by Jointly Learning to Align and Translate (Bahdanau et al., 2014)
• Sequence to Sequence Learning with Neural Networks (Sutskever et al., 2014)
• Long Short-Term Memory (Hochreiter and Schmidhuber, 1997)
• Efficient Estimation of Word Representations in Vector Space (Word2Vec, 2013)
• Deep Residual Learning for Image Recognition (ResNet, 2015)
• BERT (Devlin et al., 2018)
• Improving Language Understanding by Generative Pre-Training (GPT, 2018)
• GPT-2 model card (source of the parameter counts, Hugging Face)
• An Image is Worth 16x16 Words (ViT, 2020)
• LLaMA (Touvron et al., 2023; the combination of Pre-Norm, SwiGLU and RoPE)
• RoFormer: Rotary Position Embedding (Su et al., 2021)
• GQA: Grouped-Query Attention (Ainslie et al., 2023)
• FlashAttention (Dao et al., 2022)
• Denoising Diffusion Probabilistic Models (Ho et al., 2020)
• Scalable Diffusion Models with Transformers (DiT, 2022)
• Mamba (Gu and Dao, 2023)
• DeepSeek-V3 technical report (MoE)
• An Empirical Study of Smoothing Techniques for Language Modeling (Chen and Goodman, 1998)
• On the difficulty of training recurrent neural networks (gradient clipping; Pascanu et al., 2013)
• Qwen3 technical report
• Finding Structure in Time (Elman, 1990)
• Learning to Forget: adding the forget gate to the LSTM (Gers et al., 2000)
• The source of the GRU (Cho et al., 2014)
• ImageNet Classification with Deep Convolutional Neural Networks (AlexNet, 2012)
• Google's Neural Machine Translation System (2016)
• Language Models are Unsupervised Multitask Learners (GPT-2, 2019)
• RMSNorm (Zhang and Sennrich, 2019)
• GLU Variants Improve Transformer (source of SwiGLU; Shazeer, 2020)
• Attention is not Explanation (Jain and Wallace, 2019)
• LLM Research Papers: The 2026 List (Sebastian Raschka, June 2026)
• Beyond Standard LLMs (Sebastian Raschka, November 2025)