How Artificial Intelligence Actually Works

1. What is artificial intelligence?

How can a computer answer a question it has never seen before? A calculator never does this: every input it accepts was anticipated by whoever wrote it. A spam filter, by contrast, classifies emails nobody had written when the filter was built, and a translation system turns sentences it was never shown into another language. Seeing how that becomes possible is what this whole study is about, and it starts with knowing what "artificial intelligence" covers.

A family of capabilities, not one mechanism

Google's machine learning glossary defines artificial intelligence broadly: a non-human program or model that can solve sophisticated tasks, such as translating text or spotting disease in radiology images . NIST's AI Risk Management Framework is more operational. It calls an AI system an engineered or machine-based system that, for a given set of objectives, produces outputs such as predictions, recommendations or decisions that influence real or virtual environments . NIST adapted that definition from the OECD and ISO/IEC 22989, and adds that AI systems work with varying degrees of autonomy . NIST's site also says AI RMF 1.0 is being revised, so the wording may change .

Neither definition names a technique, and that is deliberate. "Intelligence" here is a set of capabilities built from different mechanisms:

Mechanism — What it does — Typical example; Rules written by people — Applies conditions a developer specified — Filtering out publishers that have sent spam before; Search and planning — Explores possible states or moves to find a good path — Route finding over a road graph; game look-ahead; Statistical learning (machine learning) — Fits a model to example data and predicts on new data — Deciding whether an email is spam or a photo contains a cat; Neural networks and deep learning — Machine learning with models that have hidden layers — Image recognisers that pass pixels through a deep network; Generative models — Machine learning that produces new content — Auto-completing sentences, generating images

The OECD's explanation covers both ends of that table: an AI system's objectives can be explicit, programmed directly by a developer, or implicit, set by rules a human wrote or learned from data . A rule-based expert system and a neural network can both be AI systems.

The terms nest. Machine learning is formally a sub-field of AI, though some organisations now use the two words interchangeably . Deep learning is a family of machine learning techniques: a neural network with more than one hidden layer is a deep model . Large language models are one kind of deep model. So not every AI system is machine learning, not every machine learning model is a neural network, and not every neural network is a language model.

The mechanism under the hood is arithmetic

The OECD describes a deep network as a parameterised algebraic expression made of addition, multiplication and certain nonlinear operations . An image recogniser "infers how to generate its output" by passing the pixels through that expression . That is how a system answers questions it has never seen: it does not look the answer up, it computes one from the input with numbers fixed during training. Whether the answer is right depends on whether the new input resembles the training data. The glossary puts this precisely: a trained model makes useful predictions on new data drawn from the same distribution as its training data .

Prediction is not decision

Most learned models produce a prediction: a number, a probability or a ranked list. A decision is what the surrounding system does with it. NIST's definition lists predictions, recommendations and decisions as separate outputs , and keeping them apart matters in engineering. The model says "0.93 probability of spam"; a threshold, chosen by people, turns that into "move to the spam folder" .

A worked example

A toy spam score uses two features: x₁ = how many times the word "free" appears, and x₂ = number of links. Suppose training has produced weights w₁ = 0.8, w₂ = 0.5 and bias b = −3. These numbers are invented for the arithmetic, not taken from any real filter. For an email with x₁ = 3 and x₂ = 2:

``text z = b + w₁·x₁ + w₂·x₂ = −3 + 0.8·3 + 0.5·2 = −3 + 2.4 + 1.0 = 0.4 p = σ(z) = 1 / (1 + e^(−z)) = 1 / (1 + 0.6703) = 0.599 `

Here σ is the sigmoid function, which maps any real number into the interval (0, 1); logistic regression uses exactly this form . The prediction is 0.599. With a threshold of 0.5 the decision is "spam"; with a threshold of 0.7, chosen because a lost legitimate email costs more than a missed spam, the same prediction leads to "inbox". The 0.5 default is not automatically correct when error costs differ . The model did not change; the decision did.

Compare a Celsius-to-Fahrenheit converter: F = C × 9/5 + 32`, so 21 °C is 69.8 °F. The relationship is known exactly, nothing is learned from data and nothing is uncertain. That is ordinary deterministic automation. Calling it AI would add nothing.

Try it

The lab below gives nine problems, each with its inputs and outputs. For each one, pick the mechanism you would reach for first: fixed rules or a formula, search and planning, supervised learning, unsupervised learning, reinforcement learning, or a generative model. The feedback explains the conventional choice and flags the cases that are plain automation. Some problems accept more than one answer, because strong game-playing systems combine search with learned values. The "best fit" labels are teaching judgements, not measurements.

Where this shows up

Google lists everyday uses of machine learning: predicting weather, estimating travel times, recommending songs, auto-completing sentences, summarising articles and generating images . Spam classification is the textbook classification task . Fraud detection follows the same pattern: a learned score per transaction, with a threshold and a review process deciding what happens next. Recommendation systems may keep adapting after deployment, changing as they learn individual preferences . Modern chat assistants, and many translation systems, are built on language models, the subject of most of this study.

Common misconceptions

• "AI means a chatbot." Chat assistants are one product built on one model family. The definitions above include route planners, spam filters, recommenders and rule-based systems .
• "AI and machine learning are the same thing." Formally, machine learning is a sub-field of AI, even though the terms are often used interchangeably .
• "A system that answers fluently understands like a person." The mechanism is a parameterised arithmetic expression applied to numbers . None of the primary sources used here (Google's crash course, NIST's AI RMF, the OECD) says anything about AI consciousness either way , and this study makes no claim about it. What can be said precisely is what the system computes, and the rest of this study does that.
• "Anything automated is AI." Sorting invoices by due date and emailing overdue customers is a fully specified workflow. Nothing in it is inferred.

Narrow and general capability

Google's glossary defines artificial general intelligence as a non-human mechanism showing a broad range of problem solving, creativity and adaptability. It sets that against task-specific AI, and does not claim any existing system qualifies . In practice, every system in this study is built and evaluated for particular tasks, even when one model is used for many of them. The primary sources here do not use the phrase "narrow AI", so this study uses "task-specific" instead.

Start with rules, move to learning when you have data

Google's Rules of Machine Learning advises against using machine learning before you have data, and recommends launching with a simple heuristic . It also warns that reusing data from a different problem will probably do worse than a basic heuristic . Machine learning becomes the better choice once the heuristic grows complex, because complex heuristics become unmaintainable . A learned model is then easier to update and maintain than a large tangle of rules . This is ordinary engineering advice. A team that cannot state the problem as data and a target is not ready for a learned model.

Why AI software is harder to test

NIST lists ways in which AI risk differs from traditional software risk, and several of them change how you test:

Difference — Consequence; Training data may not represent the intended use, and the ground truth may not exist or be available — A test set can pass while real inputs fail; Systems can contain billions or even trillions of decision points, housed inside conventional applications — You cannot review the logic line by line; AI systems are not subject to the same controls as traditional code, so regular testing, or even deciding what to test, is harder — Testing becomes statistical: error rates on held-out data, not pass/fail per path

Two consequences matter for engineers. First, a model sits inside an ordinary program: the application parses input, calls the model, applies thresholds and acts. Most production failures can be traced to that boundary. Second, the behaviour you can test is the distribution of outputs over representative inputs. That is why the next chapters spend so much time on data splits and evaluation.

Architecture differences that matter later

Even inside deep learning, "the model" can mean very different things. Convolutional networks dominate many image tasks , recurrent networks process sequences step by step , and transformers replace recurrence with attention . Chapter 5 compares the families. Chapter 19 shows that transformers come in encoder-only, decoder-only and encoder–decoder forms, so "transformer" is not a synonym for "chat model".

2. Traditional programming vs machine learning

In conventional programming a developer writes the rules and the computer applies them:

``text Rules + Input → Output `

Supervised machine learning moves the rules into data. A developer supplies examples with their correct outputs and a learning algorithm, and the algorithm produces numbers:

`text Examples + Target outputs + Learning algorithm → Learned parameters New input + Learned parameters → Prediction `

The problem: rules you cannot write down

Google's crash course contrasts two ways to predict rainfall. The traditional approach builds a physics-based model of the atmosphere and surface and computes enormous numbers of fluid-dynamics equations . The machine learning approach gives a model large amounts of weather data until it learns the mathematical relationship between weather patterns and rainfall . Both are programs. The difference is where the relationship comes from: in the first, people derive it; in the second, it is estimated from examples.

A model, in that vocabulary, is a mathematical relationship derived from data that a system uses to make predictions . In supervised learning it is "the complex collection of numbers" that maps input feature patterns to output labels, and those numbers are found by training .

The mechanism: parameters

Linear regression shows the idea in its simplest form:

`text y′ = b + w₁·x₁ `

y′ is the prediction, x₁ is a feature (an input value), w₁ is a weight and b is the bias. The weight and the bias are parameters, calculated during training rather than typed in . Google's fuel-efficiency example fits a line to car weight versus miles per gallon and gets a bias of 34 and a weight of −4.6 . Nobody chose −4.6; the data did.

For classification, logistic regression computes the same kind of weighted sum, z = b + w₁x₁ + … + wₙxₙ, and passes it through the sigmoid σ(z) = 1 / (1 + e^(−z)) to get a probability between 0 and 1 . The sigmoid output is always strictly between 0 and 1, and σ(0) = 0.5 . A threshold then turns the probability into a class: above it, positive; below it, negative . What happens exactly at the threshold depends on the library: Keras predicts the negative class, and other tools may differ .

The points where z = 0, and therefore p = 0.5, form a straight line (with two features) or a hyperplane (with more). That line is the decision boundary, the separator a classifier learns between classes . In a hand-written rule you place the boundary yourself. In a learned model, training places it.

A worked example: thermal comfort

Imagine reports from an office: temperature in °C, relative humidity in %, and whether the person said they were uncomfortable. A rule a facilities engineer might write:

`text uncomfortable if temperature > 27 `

A logistic model can use both features. Take these weights, chosen by hand so the arithmetic is easy (they are not what the lab learns):

`text z = 0.5·temp + 0.05·humidity − 16 p = σ(z) ``

Report — Label — Rule (temp > 27) — z — p — Model (p > 0.5); 24 °C, 88 % — uncomfortable — comfortable ✗ — 12 + 4.4 − 16 = 0.40 — 0.599 — uncomfortable ✓; 25 °C, 45 % — comfortable — comfortable ✓ — 12.5 + 2.25 − 16 = −1.25 — 0.223 — comfortable ✓; 30 °C, 35 % — comfortable — uncomfortable ✗ — 15 + 1.75 − 16 = 0.75 — 0.679 — uncomfortable ✗

The rule ignores humidity, so it misses the hot, humid report. The model uses humidity and gets that report right. Both fail on the dry 30 °C report, which sits on the "wrong" side of any sensible boundary. On the lab's 17 starting points the rule at 27 °C gets 13 right (76%), and these hand-picked weights get 16 right (94%). Those are training accuracies: they count points the boundary was fitted to, or in the rule's case points the engineer could see.

Try it

The lab below plots the 17 invented reports and shows the two classifiers side by side. On the left, drag the rule's temperature threshold. On the right, logistic regression is trained in your browser by 600 steps of full-batch gradient descent on standardised features, and the lab prints its learned equation. Pick a label, then click the chart to add a report, or click a point to remove it. The learned boundary moves when the data changes; the rule moves only when you move it. Both accuracies are measured on the points shown, which flatters both.

Where this shows up

Most production systems mix the two. Google's Rules of Machine Learning suggests launching with a simple heuristic, such as filtering out publishers that have sent spam before , and switching to a learned model once the heuristic becomes too complex to maintain . The learned model is easier to update and maintain than a large set of rules . Hard constraints, such as legal requirements, usually stay as code around the model.

Common misconceptions

• "A trained model is correct because it learned from data." Training fits the data it was given. Generalisation, the ability to make correct predictions on new data, is a separate property, and it is the opposite of overfitting . The lab's training accuracy says nothing about a report it has not seen.
• "The probability is the decision." A probability of 0.932 means a 93.2% estimated probability of spam . What to do with it is a threshold choice. A threshold of 0.5 is a poor choice when one kind of error costs more than the other, or when the classes are imbalanced .
• "Rules are obsolete." Rules are exact, explainable and need no data. They are the right tool when the relationship is known.
• "Machine learning finds the true rule." It finds parameters that minimise a loss on the training examples, within the shapes the model can represent. A logistic model can only draw a straight boundary in the feature space it is given.

Regression and classification

Regression models predict a number, such as rainfall in inches. Classification models predict a category: binary when there are two classes, multiclass when there are more . The comfort example is binary classification. Chapter 3 works through a regression example, and Chapter 11 shows the losses each kind of task uses.

Why training cannot guarantee correct predictions

Even with perfect optimisation, three gaps remain:

• The data are a sample. The parameters fit these reports, not every report that will ever be filed. A model predicts reliably only on new data drawn from the same distribution as its training data .
• The model has a fixed shape. A straight boundary cannot fit a curved relationship. Choosing a more flexible model trades this error for a risk of fitting noise (Chapter 13).
• The labels can be wrong or ambiguous. Two people in the same room can report different comfort. No boundary separates contradictory labels.

So a deployed classifier has an error rate, and engineering is about measuring it on data the model never saw and deciding what each kind of error costs.

Rules and learned models as software

Where the logic lives — Source code — Parameter values produced by training; How it changes — A developer edits code — Retrain on new data; the code may not change; Behaviour on unseen inputs — Exactly what the code says — Estimated from the training distribution; Testing — Unit tests per branch — Statistical evaluation on held-out data; Failure mode — Missing case, brittle thresholds — Plausible but wrong outputs, sensitivity to data shift

Engineering details

The lab's learned model hides two practical details. First, it standardises the features before training, so temperature (around 20 to 30) and humidity (around 30 to 90) are on similar scales. Normalisation like this helps gradient descent converge faster . The equation shown has been converted back to raw units. Second, logistic regression is a linear model with a convex loss, so gradient descent heads for the same global minimum from any start ; the lab adds a very small L2 penalty on the weights, which keeps the fitted weights finite when the classes can be separated perfectly. Neural networks do not have that guarantee, which is one reason deep learning training needs more care.

3. How machines learn from data

Learning, for a machine, means adjusting numbers until predictions on known examples are as close as possible to the known answers, then checking whether that holds on examples it has never seen. Every supervised system, from a house-price model to the pretraining of a language model, follows the same loop.

The vocabulary

A dataset is a set of examples, each one like a row in a spreadsheet, holding features and a label . Features are the values the model uses to predict; the label is the answer it should predict . A labeled example has both; an unlabeled example has only features, and the trained model predicts its label . Datasets are described by size, the number of examples, and diversity, the range they cover; good datasets are both large and diverse .

The lifecycle

Step — What happens; 1. Collect representative data — Size alone is not enough: 100 years of July rainfall predicts January poorly; 2. Define the target — Decide exactly what the label is, and use only features that will exist at prediction time; 3. Clean and prepare — Validate raw data against a schema, for example ratings always between 1 and 5 . More features do not always help; 4. Split — Training, validation and test sets , shuffled well before splitting; 5. Initialise — Weights start random and near zero, or at zero for simple models; 6. Predict — Run the model on training examples; 7. Compute loss — One number for how wrong the predictions are; 8. Compute gradients — The direction in which each parameter would reduce the loss; 9. Update — Move each parameter a small step in that direction; 10. Repeat — Many passes over the data; one full pass is an epoch; 11. Evaluate on unseen data — Give the model only features, compare with true labels; 12. Deploy and monitor — Inference on new data ; watch for drift

Steps 6 to 10 are training. Training compares each prediction with the real label, and the difference, the loss, guides gradual updates . Google's example: predicting 1.15 inches of rain when the true value was 0.75 nudges the model toward 0.75 .

The mathematics

For a linear model y′ = b + w·x and N examples, the mean squared error is:

``text L(w, b) = (1/N) · Σᵢ (y′ᵢ − yᵢ)² `

yᵢ is the true label of example i and y′ᵢ the prediction. Loss measures distance, not direction, so the sign is removed, here by squaring . MSE penalises outliers more heavily than mean absolute error . The partial derivatives, which tell you how the loss changes when each parameter changes, are:

`text ∂L/∂w = (2/N) · Σᵢ (y′ᵢ − yᵢ) · xᵢ ∂L/∂b = (2/N) · Σᵢ (y′ᵢ − yᵢ) `

Gradient descent moves each parameter against its derivative, scaled by a learning rate η:

`text w ← w − η · ∂L/∂w b ← b − η · ∂L/∂b `

The learning rate is a hyperparameter, something you set; the weight and bias are parameters, something training learns .

A worked example: house size to price

Three past sales, with size in hundreds of square metres and price in hundreds of thousands (of any currency):

House — Size x — Price y; A — 1 — 2; B — 2 — 4; C — 3 — 5

Step 0. Start at w = 0, b = 0, as Google's walkthrough does . Every prediction is 0, so the errors y′ − y are −2, −4, −5.

`text L = (4 + 16 + 25) / 3 = 15 ∂L/∂w = (2/3) · (−2·1 − 4·2 − 5·3) = (2/3) · (−25) = −16.667 ∂L/∂b = (2/3) · (−2 − 4 − 5) = (2/3) · (−11) = −7.333 `

Step 1. With η = 0.05: w = 0 − 0.05 · (−16.667) = 0.833 and b = 0 − 0.05 · (−7.333) = 0.367. The predictions become 1.200, 2.033 and 2.867, the errors −0.800, −1.967 and −2.133, and

`text L = (0.640 + 3.868 + 4.551) / 3 = 3.020 `

One step cut the loss from 15 to about 3.02. Repeating converges to the least-squares solution w = 1.5, b = 0.667, with predictions 2.167, 3.667 and 5.167 and a training MSE of 0.056. Because a linear model's loss is convex, gradient descent finds this global minimum .

Evaluation. A fourth house, D, is held out: size 2.5, price 4.3. The model predicts 0.667 + 1.5 · 2.5 = 4.417`, an error of 0.117. That held-out error, not the training error, estimates how the model will do in use.

Where this shows up

The loop is the same at every scale. Large language models replace three houses with trillions of tokens, the straight line with billions of parameters and MSE with cross-entropy, but they still predict, compute a loss, compute gradients and update. The rest of this study builds each piece: neurons (Chapter 7), loss functions (Chapter 11), backpropagation (Chapter 12) and generalisation (Chapter 13).

Common misconceptions

• "Low training loss means a good model." Overfitting means matching, in effect memorising, the training set so closely that the model fails on new data . Training loss cannot detect it.
• "Testing on the test set many times is fine." Tuning against the test set makes the model implicitly fit it ("teaching to the test") . Validation and test sets wear out with repeated use and should be refreshed with new data .
• "The model keeps learning as it is used." In the standard setup, training and inference are separate phases: inference applies fixed parameters to new inputs . Some systems are retrained or adapt after deployment , but that is a deliberate pipeline, not a side effect of answering.

Memorisation, made concrete

A quadratic y = a + bx + cx² has three parameters and can pass through the three houses exactly. The curve through (1, 2), (2, 4) and (3, 5) is y = 2 + 2(x − 1) − 0.5(x − 1)(x − 2), with zero training error. It looks better than the line on training data.

Model — Training MSE — Prediction for D (2.5) — Error on D — Prediction at size 5; Line, 2 parameters — 0.056 — 4.417 — 0.117 — 8.167; Quadratic, 3 parameters — 0 — 4.625 — 0.325 — 4.000

The quadratic has memorised the three points, noise included. Its error on the held-out house is nearly three times larger. Asked about a house of size 5, it predicts a lower price than for size 3. Too complex a model for the data is one of the two broad causes of overfitting; the other is a training set that does not represent real data .

Underfitting is the opposite failure: the model predicts poorly even on its training data . Causes include the wrong features, too few epochs, too low a learning rate, too much regularisation and too few hidden layers . On a generalisation curve, overfitting shows as training loss falling or holding steady while validation loss starts to rise .

What generalisation assumes

Held-out scores predict real-world performance only under assumptions. Examples must be independent and identically distributed, the data must be stationary, and every partition must come from the same distribution . Time breaks these assumptions most often. Google's Rules of Machine Learning recommends testing on data gathered after the training data, for example training until 5 January and testing from 6 January, because that better reflects production .

Data leakage

Leakage lets information about the label into the features, so evaluation looks better than reality:

• Label leakage. A feature acts as a proxy for the label. Google's example is a "spoke to customer agent" flag that is set only after a purchase . It can be very hard to detect .
• Duplicates across splits. Copies of training examples in the test set inflate scores; delete them, because the only fair test is on new examples .
• Inconsistent transformations. Whatever you do to training features must also be done to validation, test and real-world data . Computing normalisation statistics on the full dataset before splitting is a common leak, because the training data then carries information from the test set.

Training-serving skew and monitoring

Training-serving skew means the inputs at serving time differ from those at training time, for example because the two pipelines run different feature code . Rules of Machine Learning names three causes: different data handling in the two pipelines, data that changes between training and serving, and feedback loops . Google's golden rule is that training and production should mimic each other as closely as possible .

After deployment, quality declines if serving data changes and the model is not retrained, so Google recommends tracking model age and alerting past a threshold . Pipelines should also check numerically that weights and layer outputs never become NaN or Inf, and that more than half of a layer's outputs are not zero . These are the same habits as monitoring any service: the model is a dependency whose behaviour drifts.

Epochs and batches

Real datasets are too large to compute the loss over every example for each update. Training splits them into batches. One epoch, a full pass over N examples, takes N / batch size iterations: 1,000 examples with a batch size of 50 take 20 iterations . Chapter 12 shows how batch size and learning rate interact.

4. Categories of machine learning

Machine learning methods are usually grouped by where the training signal comes from. Google's crash course uses four groups, supervised learning, unsupervised learning, reinforcement learning and generative AI, and notes that one system can belong to more than one . Several other terms, such as self-supervised, transfer and representation learning, describe how the signal is built or reused rather than a separate world. Modern language models use almost all of them in sequence.

Where does the target come from?

Category — Training signal — Output — Example; Supervised — Correct answers supplied with the data — A predictor — Spam / not spam; Unsupervised — None; structure in unlabeled data — Groups, structure — Clustering weather data, later named as seasons; Self-supervised — Surrogate labels made from unlabeled data — A pretrained model — BERT's masked tokens; Reinforcement — Rewards or penalties for actions in an environment — A policy — Robots learning to walk, AlphaGo playing Go; Generative — Mimicking the training data — New, similar data — Text, images, audio

Supervised learning is studying old exams that include the answers . Chapters 2 and 3 covered it.

Unsupervised learning finds patterns in data that are typically unlabeled; its most common use is clustering similar examples into groups . Clustering differs from classification because you do not define the categories in advance. A person may name the clusters afterwards .

Reinforcement learning learns from rewards or penalties earned by actions in an environment, and its product is a policy, the strategy that earns the most reward . The glossary puts it formally: algorithms that learn an optimal policy to maximise return while interacting with an environment . Nobody labels the correct action at each step. The signal is delayed and has to be credited to the actions that earned it.

Self-supervised learning turns an unsupervised problem into a supervised one by creating surrogate labels from unlabeled examples . This is the key to modern language models.

Targets from raw text

A labeled question-answer dataset is expensive. Raw text is not, and it already contains targets: every token is the "answer" to the tokens before it. Causal language modelling trains a model to predict the next token from those before it, and it never sees future tokens . The target is computed from the text itself, so no human labelling is needed .

Take the five-token text why is the sky blue. It yields four training examples at once:

Input (context) — Target; why — is; why is — the; why is the — sky; why is the sky — blue

The first token is never a target, and the last has nothing after it to predict . One forward pass scores all four positions . Real systems work on token IDs, not words (Chapter 14), but the principle is the same.

BERT builds a different surrogate task. It picks 15% of token positions; each chosen token becomes [MASK] 80% of the time, a random token 10% of the time and stays unchanged 10% of the time . The model must predict the original token from context on both sides . For a 200-token passage that means 30 positions chosen: on average 24 masked, 3 randomised and 3 left alone. Across many passages the counts average out to those proportions, though each passage varies.

A worked clustering example

Unsupervised learning has no targets, so the arithmetic looks different. Group four one-dimensional points, 1, 2, 9 and 10, into two clusters with k-means, starting with cluster centres at 1 and 2:

• Assign each point to its nearest centre: 1 goes to centre 1; 2, 9 and 10 go to centre 2.
• Update each centre to the mean of its points: centre 1 = 1, centre 2 = (2 + 9 + 10) / 3 = 7.
• Assign again: point 2 is now 1 away from centre 1 and 5 away from centre 2, so it moves. Clusters {1, 2} and {9, 10}.
• Update: centres 1.5 and 9.5. A further assignment changes nothing, so the algorithm stops.

Nobody said "small" and "large"; the structure came from the data. Whether those clusters mean anything is a human judgement afterwards .

Try it

This chapter has no lab of its own. You can see self-supervised targets directly in Chapter 28, where every position of a toy model's input is scored against the next token, and preference optimisation in Chapter 30. The lab in Chapter 1 asks you to tell supervised, unsupervised, reinforcement and generative problems apart.

Where this shows up

A modern assistant combines several categories in sequence. InstructGPT starts from GPT-3, an autoregressive language model pretrained on text . It then fine-tunes on demonstrations written by labelers (supervised), and uses rankings of model outputs for reinforcement learning from human feedback . The OECD notes that the objectives of such systems are partly learned, through imitation of human-written text and RLHF, rather than programmed .

Common misconceptions

• "Language models are trained on question-answer pairs." Pretraining uses raw text with targets derived automatically . Curated demonstrations come later, in a much smaller fine-tuning stage .
• "RLHF is reinforcement learning like AlphaGo's." The algorithmic machinery overlaps, but the signal differs. In RLHF it comes from human judgements of the model's outputs, rankings or ratings, not from a reward built into an environment such as a game score .
• "The categories are exclusive." One system can fall into several . Generative models are typically trained first without supervision to mimic their data, and sometimes further with supervised or reinforcement learning .
• "A bigger model follows instructions better." The InstructGPT paper says scale alone does not make models follow a user's intent better. In human evaluations, outputs from the 1.3B-parameter InstructGPT were preferred to those of the 175B GPT-3 .

Transfer and representation learning

Transfer learning moves knowledge from one task to another, for example from a task with plenty of data to one with little . Computer vision had already shown its value by fine-tuning models pretrained on ImageNet . BERT applies the recipe to language in two steps: pretraining on unlabeled data over several tasks , then initialising from the pretrained parameters and fine-tuning all of them on labeled data for the downstream task .

"Representation learning" describes what pretraining is for. The point is not the surrogate task itself, since nobody needs masked words filled in, but the internal vectors the model learns along the way. Those vectors are useful inputs for many other tasks. Chapter 16 shows learned word vectors directly.

GPT-3 goes further. It is a 175-billion-parameter autoregressive model, and its paper describes the usual recipe as pretraining followed by task-specific fine-tuning . GPT-3 itself was evaluated with no gradient updates at all, with the task and a few examples given purely as text . The weights did not change; the prompt did. Few-shot prompting conditions a fixed model. It does not train it.

Preference optimisation compared with classical RL

Signal — Reward from the environment — Human rankings of outputs — Human preference pairs; Reward model — Usually not needed — Trained on rankings — None; Episode — Many steps of state and action — Prompt, one response, reward, end — No sampling during training; Optimisation — RL algorithm — PPO with a KL penalty to the SFT model — A classification loss on preference pairs

Both RLHF and DPO target the same objective: maximise expected reward while a KL term, weighted by β, limits how far the policy moves from the reference model . DPO's authors describe RLHF as complex and often unstable . DPO rewrites the problem so it reduces to a simple loss on preference pairs, with no reward model and no RL loop . It is called preference optimisation, but it is no longer reinforcement learning in the classical sense: nothing explores an environment. Chapter 30 works through the DPO loss with numbers.

Generative modelling as a category

Generative models learn the patterns in data in order to produce new but similar data . "Generative" describes the output, not the training signal. A language model is generative and self-supervised; a diffusion model (Chapter 5) is generative and is trained on a denoising task. Producing plausible new data is not the same as producing true statements. Chapter 41 returns to that gap.

5. Types of AI models

A model family is a set of assumptions about how inputs relate to outputs. A linear model assumes a weighted sum. A tree assumes that a sequence of yes/no questions works. A convolutional network assumes nearby pixels matter together. When the assumption fits the data, the family learns well from fewer examples; when it does not, no amount of tuning fully compensates. That is why no family wins everywhere.

The families

Linear and logistic regression. The prediction is a linear combination of the features, ŷ = w₀ + w₁x₁ + … + wₚxₚ; scikit-learn stores w₁…wₚ as coef_ and w₀ as intercept_ . Despite its name, logistic regression is a linear model for classification. scikit-learn turns its probability into a class with a default threshold of 0.5 .

Decision trees. A non-parametric supervised method that learns simple if-then-else rules from the features . Trees need little data preparation, with no normalisation, and prediction cost grows only logarithmically with the number of training points . Finding the optimal tree is NP-complete, so practical learners split greedily . Trees overfit by growing too complex; pruning, a minimum number of samples per leaf or a maximum depth guards against it .

Random forests. An ensemble combines several models to generalise better than any single one . In a random forest, each tree is trained on a bootstrap sample of the data and considers only a random subset of features at each split . The two kinds of randomness decouple the trees' errors, so averaging cancels some of them and reduces variance .

Gradient-boosted trees. Weak models, such as small trees, are trained one after another to reduce the loss of the ensemble, and the final model is the sum of them all . XGBoost trains this way, adding one tree at a time, because fitting all the trees jointly is intractable . scikit-learn calls gradient-boosted trees excellent for regression and classification, especially on tabular data .

Feedforward neural networks. Layers of weighted sums, each followed by a nonlinear function. A network has at least one hidden layer, and more than one makes it deep . Chapters 7 to 13 build one from scratch.

Convolutional networks (CNNs). At least one layer is convolutional, typically combined with pooling and dense layers, and they have done especially well on image recognition . A convolution slides a small filter, for example 3×3, over the input and sums element-wise products at each step . The same filter weights are used at every position. ResNet's residual connections made networks of up to 152 layers trainable .

Recurrent networks (RNNs, LSTMs). The network runs once per step, feeding its hidden state into the next step, which suits sequences . LSTM cells keep history in an internal memory state to fight vanishing gradients on long sequences .

Transformers. Attention replaces recurrence and convolution entirely, which makes training far more parallel . Recurrent models' sequential nature prevents parallelism within a training example, and that matters more as sequences grow longer . Chapters 19 to 26 take the transformer apart.

Autoencoders and VAEs. An encoder maps the input to a typically lossy, lower-dimensional form, and a decoder rebuilds the input from it . The variational autoencoder adds a learned approximate inference model and a reparameterised lower bound that ordinary stochastic gradient methods can optimise .

Diffusion models. Ho et al.'s denoising diffusion models synthesise high-quality images, reaching an FID of 3.17 on unconditional CIFAR-10 . They are trained on a weighted variational bound linked to denoising score matching . In outline, generation starts from noise and removes it over many steps.

A worked example: lines, trees and boosting

Four training points: x = 1, 2, 3, 4 with y = 2, 4, 6, 8.

• Linear regression fits ŷ = 2x exactly. At x = 10 it predicts 20.
• A depth-1 tree (a "stump") splits at x < 2.5. Each leaf predicts the mean of its points: left (2 + 4) / 2 = 3, right (6 + 8) / 2 = 7. At x = 10 it predicts 7, because no leaf has ever seen a value above 8. Tree predictions are piecewise constant, so trees extrapolate poorly .
• Boosting, two rounds. Round 0 predicts the mean, 5, everywhere. The residuals y − ŷ are −3, −1, 1, 3. Round 1 fits a stump to the residuals: split at 2.5, leaves −2 and +2. The sum is now 3, 3, 7, 7, and the residuals shrink to −1, 1, −1, 1. A third stump splitting at 1.5 or 3.5 would cut them further. Each weak model corrects what the sum so far gets wrong .

Here the line wins because the data really are linear. Bend the data (y = 2, 4, 4, 2, say) and the line fails while the stump-based models adapt. Which family is "better" depends on the data.

Try it

The lab below lists nine families: linear and logistic regression, decision tree, random forest, gradient-boosted trees, CNN, RNN/LSTM, transformer, diffusion model and autoencoder/VAE. Select one to see the kind of data it usually takes, how it computes, one strength, one limitation and a small schematic. The descriptions are simplified summaries, not benchmarks.

Where this shows up

On tabular business data, such as transactions, sensor readings and customer records, tree ensembles are a common first choice. Images use CNNs and, increasingly, vision transformers (Chapter 40). Language models are transformers. Image generators are often diffusion models. One product may contain several: a recommender might use a gradient-boosted model for ranking and a transformer for text features.

Common misconceptions

• "Deep learning beats everything." On a benchmark of 45 tabular datasets, tree-based models such as XGBoost and random forests stayed state of the art on medium-sized data, around 10,000 samples . The same paper notes that deep learning drove large gains on text and images, but its advantage on tabular data is not clear .
• "Every neural network is a transformer." CNNs, RNNs, autoencoders and plain feedforward networks are all neural networks, and all are in production use.
• "A learned model always beats a heuristic." Often, not always. Google's glossary gives an illustrative case of 86% accuracy with a heuristic against 98% with a deep network , but it is an example, not a law. A baseline is how you find out.

Inductive bias: what each family assumes

Family — Built-in assumption — Consequence; Linear — Effects add up — Needs engineered features for curves; coefficients are inspectable; Tree — Thresholds on single features — Invariant to feature scale ; flat outside the training range; Forest / boosting — Many trees reduce error — Lower variance or lower bias; CNN — Local patterns, same everywhere — Shared filters slide over the image; RNN / LSTM — Order matters, step by step — Sequential computation; Transformer — Any position may relate to any other — Parallel training ; attention cost grows with sequence length squared (Chapter 21)

Grinsztajn et al. explain why trees do well on tables: a tabular neural network would need to be robust to uninformative features, preserve the orientation of the data and learn irregular functions easily . Trees have those properties by construction; standard neural networks have to learn them.

Implementation details that change results

Libraries differ in ways that matter. scikit-learn's random forest averages the trees' probability estimates, whereas the original algorithm let each tree vote for a single class . Its histogram-based gradient boosting first bins the inputs into integer bins, typically 256. That can be orders of magnitude faster beyond tens of thousands of samples, and it handles missing values natively . Trees are also unstable: small changes in data can produce a completely different tree , one reason ensembles are preferred.

PyTorch's nn.LSTM computes input, forget, cell and output gates for every element of the sequence. Its defaults include batch_first=False, so the time axis comes first, dropout=0.0 and bidirectional=False . Getting batch_first wrong is a classic silent bug: the shapes may still line up while the model reads the wrong axis as time.

Depth is not free

ResNet's paper begins from the observation that deeper networks are harder to train. Residual learning made 152-layer networks trainable, and an ensemble reached 3.57% error on the ImageNet test set . Residual connections reappear in every transformer block (Chapter 24). Different families borrow from each other, so the boundaries in the lab are useful but not rigid.

Choosing a model

• Start with a baseline: a heuristic or a linear model.
• Match the family to the data type: tables to trees or linear models, grids to CNNs or ViTs, sequences to transformers or RNNs.
• Weigh the constraints: latency, memory, interpretability, the amount of labelled data.
• Compare on held-out data, not on intuition (Chapter 3).

6. How data becomes numbers

A model is arithmetic, so everything it receives must be numbers. Google's crash course says it plainly: models can only manipulate floating-point values, so strings must be converted first . How a photograph, a sentence or a recording becomes an array of numbers decides what the model can learn from it. Getting it wrong is one of the most common sources of silent failure.

Scalars, vectors, matrices, tensors

• A scalar is one number: a temperature, 21.5.
• A vector is a list of numbers. The feature vector for one example, such as [0.92, 0.56], is the model's input in both training and inference .
• A matrix is a 2-D grid: a batch of feature vectors, or a grayscale image.
• A tensor generalises to any number of dimensions. In PyTorch it is a multi-dimensional matrix whose elements all share one data type . You build one from a list with torch.tensor() or create one with a chosen type, such as torch.zeros([2, 4], dtype=torch.int32) .

The shape lists the size of each dimension. A batch of 16 RGB images of 32 × 32 pixels is a tensor of shape [16, 3, 32, 32] in PyTorch's channels-first layout.

How each kind of data is encoded

Data — Typical numeric form; Numbers — Floats, normalised to similar scales; Categories — One-hot vectors , or embeddings when there are many categories; Text — Token IDs from a tokenizer , looked up as vectors in an embedding table; Images — Integers 0–255 per channel , converted to floats; Audio — A float32 waveform of shape [channel, time] plus a sample rate , often turned into a spectrogram; Time series — Fixed-length windows of past values, with time features such as hour or weekday

Categories. Encoding colours as red = 0, green = 1, blue = 2 misleads the model, because it treats larger indices as larger values. One-hot encoding uses an N-element vector with exactly one 1.0 and all other entries 0.0 , so red is [1, 0, 0] and blue [0, 0, 1]. The one-hot vector, not the string or index, enters the feature vector, and the model learns a weight for each element .

Text. A tokenizer turns text into a sequence of tokens , each identified by an integer ID. GPT-2's vocabulary has 50,257 entries , and its tokenizer turns "Why is the sky blue?" into the six IDs [5195, 318, 262, 6766, 4171, 30] (Chapter 15's lab runs the real tokenizer). An embedding layer is a lookup table from integer IDs to stored vectors . The model never sees letters.

Audio. A waveform is a long sequence of amplitudes. At 16,000 Hz, one second is 16,000 numbers per channel. torchaudio's Spectrogram converts that into frequency bins: n_fft defaults to 400, which gives 400 // 2 + 1 = 201 bins, and the hop length defaults to win_length // 2 . With a 400-sample window that is a hop of 200 samples, about 80 frames per second. MelSpectrogram defaults to 16,000 Hz and 128 mel filterbanks .

Images, exactly

An 8-bit RGB pixel has three integers between 0 and 255, the range of torch.uint8 . A 32 × 32 colour image therefore holds 32 × 32 × 3 = 3,072 channel values. CIFAR-10 stores each image as exactly those 3,072 uint8 values: 1,024 red, then 1,024 green, then 1,024 blue, each channel in row-major order .

Two transforms usually follow. torchvision's ToTensor converts an H × W × C image in [0, 255] into a float tensor of shape C × H × W in [0.0, 1.0] . Normalize then standardises each channel:

``text output[c] = (input[c] − mean[c]) / std[c] `

where c is the channel and mean[c], std[c] are statistics chosen per channel . A Z-score is the number of standard deviations a value sits from the mean .

A worked example: one orange pixel

The pixel (255, 107, 61), with the per-channel statistics the lab uses (mean 0.485, 0.456, 0.406; standard deviation 0.229, 0.224, 0.225):

Channel — Raw (uint8) — Scaled = raw / 255 — Standardised = (scaled − mean) / std; R — 255 — 1.0000 — (1.0000 − 0.485) / 0.229 = 2.2489; G — 107 — 0.4196 — (0.4196 − 0.456) / 0.224 = −0.1625; B — 61 — 0.2392 — (0.2392 − 0.406) / 0.225 = −0.7413

After standardisation the red channel ranges from −2.1179 (raw 0) to 2.2489 (raw 255), so values are no longer bounded by 0 and 1. That is intended: each channel is centred near zero with similar spread. The same transform, with the same statistics, must be applied at prediction time .

Try it

The lab below holds an 8 × 8 RGB image, starting as a simple face. Pick a colour and click pixels to paint them; the clicked pixel becomes the selected one. Switch the value view between raw 0–255, scaled 0–1 and standardised (with the constants above) to see the selected pixel's channels and a whole row of the red channel as the model would receive them. The lab reports the tensor shape [3, 8, 8]: 192 numbers, 192 bytes as uint8 and 768 bytes as float32. Storage counts only pixel values, not file headers or compression.

Where this shows up

CIFAR-10 has 60,000 32 × 32 colour images in 10 classes, 50,000 for training and 10,000 for testing , and its labels are integers from 0 to 9 . As uint8 the 60,000 images take 60,000 × 3,072 = 184,320,000` bytes, about 184 MB; as float32, four times that, 737,280,000 bytes. Google notes that practitioners spend far more time evaluating, cleaning and transforming data than building models .

Common misconceptions

• "The model reads text." It receives integer token IDs, which an embedding table turns into vectors . Whether "sky" is one token or several depends on the tokenizer (Chapter 15).
• "Category indices are fine as numbers." An index implies an order and a distance that do not exist .
• "Normalisation is optional polish." Features with ranges as different as 154 to 24,917,482 and 5 to 22 slow gradient descent; scaling them to similar ranges helps it converge faster . Mismatched normalisation between training and serving is a classic bug .
• "ToTensor is harmless on everything." It rescales to [0, 1], so torchvision warns against using it on segmentation masks, where pixel values are class labels .

Precision: how many bits per number

A float splits its bits into sign, exponent and mantissa. More exponent bits give more range; more mantissa bits give finer steps.

dtype — Bits — Layout (S-E-M) — Machine epsilon; float32 — 32 — 1-8-23 — 2⁻²³ ≈ 1.19 × 10⁻⁷; float16 — 16 — 1-5-10 — 2⁻¹⁰ ≈ 0.000977; bfloat16 — 16 — 1-8-7 — 2⁻⁷ = 0.0078125; uint8 — 8 — integer 0–255 — —

float32 is IEEE 754 single precision and is typically PyTorch's default floating-point type . Its 1-8-23 layout follows from the 8 exponent bits it shares with bfloat16 . Epsilon is the gap between 1.0 and the next larger representable float . For a normal float with m mantissa bits it is 2⁻ᵐ, which is how the column above was computed. bfloat16 keeps float32's exponent range and gives up precision, whereas float16 keeps more precision in a much smaller range . The PyTorch docs do not print numeric ranges such as float16's maximum; they are available at runtime through torch.finfo , which reports bits, eps, max, min and the smallest normal value .

Does an 8-bit pixel survive in bfloat16? Adjacent pixel levels differ by 1/255 ≈ 0.003922. In the interval [0.5, 1), bfloat16's spacing is 2⁻¹ × 2⁻⁷ = 2⁻⁸ = 0.00390625: just smaller, so the 256 levels stay distinct, but only just. float16's spacing there is 2⁻¹¹ ≈ 0.000488, with room to spare. Training in lower precision is a trade of exactly this kind (Chapter 31).

Layout matters as much as values

The same 3,072 numbers can be arranged in different ways. CIFAR-10's files are channel-planar: all red, then all green, then all blue . Image libraries usually give height × width × channels, interleaving R, G, B per pixel, and ToTensor reorders that to channels × height × width . Feeding a model the wrong layout often still runs, because the element count matches, and the model then learns or predicts nonsense. Checking shapes and a few values by hand, as the lab does, is cheap insurance.

Preprocessing depends on the model

• Tree models need no normalisation ; neural networks generally benefit from it .
• One-hot vectors suit a handful of categories. For high-cardinality features, Google recommends embeddings or hashing instead .
• scikit-learn's OneHotEncoder learns categories from the training data and by default raises an error on a category it has not seen (handle_unknown='error') . That is a sensible default, because production data contains categories training never saw.
• A model pretrained with particular normalisation statistics expects them at inference. Changing them changes every input.

From a vector to a tensor of examples

Putting it together for a batch: 16 images give [16, 3, 32, 32], 16 token sequences of length 128 give integer IDs of shape [16, 128] and, after embedding lookup with width d, floats of shape [16, 128, d]. Every later chapter manipulates tensors like these. Chapter 7 starts with the smallest operation on them, one neuron's weighted sum.

7. The artificial neuron

An artificial neuron is the smallest unit of computation in a neural network, and it is much simpler than the name suggests. It takes a handful of numbers, multiplies each one by its own weight, adds the products together with one more number called the bias, and passes the total through a fixed function. That is the whole of it. A neuron in a hidden layer computes the same thing as the linear model from earlier chapters, multiply each input by its weight, sum, add a bias, with one extra step at the end .

The problem it solves

Chapter 2 contrasted hand-written rules with a model whose behaviour is set by numbers learned from data. To build large learned functions we need a building block that is tiny, adjustable and differentiable: tiny so that millions of copies are affordable, adjustable so that training has something to change, and differentiable so that training can work out which way to change it (chapter 12). The weighted sum followed by a simple function meets all three. Weights and biases are the parameters the model learns; settings such as the learning rate are hyperparameters chosen by the developer .

The mechanism

For a neuron with n inputs:

``text z = w₁x₁ + w₂x₂ + … + wₙxₙ + b a = f(z) `

Symbol — Name — Meaning; xᵢ — input — the i-th number fed in: a pixel value, a feature, or the output of a neuron in the layer before; wᵢ — weight — how strongly, and in which direction, input i pushes the sum; positive weights excite, negative weights inhibit; b — bias — a constant added regardless of the inputs; it shifts where the neuron switches on; z — weighted sum (pre-activation) — the result of the linear part; f — activation function — a fixed function applied to z, usually nonlinear (chapter 8); a — activation — the neuron's output, passed on to the next layer

In vector notation the same thing is a = f(w · x + b), where w · x is the dot product of the weight vector and the input vector. Google's Machine Learning Crash Course writes the value of a node exactly this way, as the activation function applied to the weighted sum plus the bias .

A worked example

Take two inputs and a ReLU activation, ReLU(z) = max(0, z) :

`text x₁ = 2 w₁ = 0.4 x₂ = 3 w₂ = 0.6 b = −0.5

contribution of input 1: 2 × 0.4 = 0.8 contribution of input 2: 3 × 0.6 = 1.8 z = 0.8 + 1.8 − 0.5 = 2.1 a = ReLU(2.1) = max(0, 2.1) = 2.1 `

Both contributions are positive and the bias pulls the sum down by 0.5. Because z is positive, ReLU passes it through unchanged. With a different activation the same z gives a different output: sigmoid gives 1 / (1 + e^(−2.1)) = 0.890903, and tanh gives 0.970452.

Now change only the bias to b = −3. Then z = 0.8 + 1.8 − 3 = −0.4, and ReLU(−0.4) = 0: the neuron is silent for this input. The bias decides how much evidence the weighted inputs must supply before the neuron produces anything. Geometrically, the set of inputs where w · x + b = 0 is a straight line (with two inputs) or a flat hyperplane (with more); the weights set its orientation and the bias slides it.

Try it

The lab below is a three-input neuron computed exactly in your browser. It starts at x = (0.8, −0.4, 0.3), w = (1.2, 0.7, −1.5), b = 0.1 with ReLU, which gives z = 0.96 − 0.28 − 0.45 + 0.1 = 0.33 and an output of 0.33. Every input, weight and the bias has a slider from −2 to 2, and the activation can be switched between none (linear), ReLU, Leaky ReLU, sigmoid, tanh, GELU and SiLU. Each input has a signed bar for its contribution xᵢ·wᵢ: bars to the right push the sum up, bars to the left push it down. The equation underneath rewrites itself with every change. Try making the third input larger: its negative weight drags z below zero, and with ReLU the output snaps to exactly 0. The weights here are set by hand; in a real network nobody sets them, training does.

Where it appears

Frameworks do not compute neurons one at a time. PyTorch's nn.Linear layer applies the affine transformation y = xAᵀ + b to its input, so a whole layer of neurons is one matrix multiplication plus a bias vector . Each row of the weight matrix is one neuron's weight vector: the matrix has shape (out_features, in_features) and the bias has shape (out_features)` . The toy language model on this page has a feed-forward layer of 192 units in each of its two blocks; every one of those units is this calculation with 48 inputs followed by GELU, and the same pattern, at much larger widths, sits inside the feed-forward layers of transformer language models.

Common misconceptions

"An artificial neuron is a model of a brain cell." The name is historical. What the network computes is the arithmetic above: multiplications, additions and a fixed function. Nothing in this chapter depends on biology, and the behaviour of real networks should be explained from the mathematics, not from analogies to neurons in a brain. This page also makes no claim about the history of the perceptron, because no primary source for it was checked .

"Someone chooses the weights." In the lab you do, because the point is to see the arithmetic. In practice the weights and biases are learned . MLCC describes gradient descent as starting from randomised weights and biases near zero , and PyTorch initialises nn.Linear weights and biases from a uniform distribution U(−√k, √k) with k = 1/in_features . For a neuron with three inputs that range is about ±0.577.

"The bias is a minor detail." Without a bias, w · x = 0 always passes through the origin, so a neuron could never be active for every input near zero or silent for all of them. The bias is a learned parameter like any weight. It can be switched off (nn.Linear(…, bias=False)), but it is on by default ; layers are sometimes built without one when a later operation would cancel it anyway.

"A neuron's output means something." A single unit's activation is a number. It may correlate with something a person can name, but a network is not built from labelled units, and an individual value has no meaning outside the weights that read it.

Advanced insight: the dot product as a similarity test

Write the weighted sum as w · x = ‖w‖ ‖x‖ cos θ, where ‖·‖ is the length of a vector and θ the angle between the two. For inputs of a given length the sum is largest when x points the same way as w. A neuron is therefore a template matcher: it responds most strongly to inputs that look like its weight vector, and the bias sets how close the match must be before the activation lets the signal through. This view carries forward: attention in chapter 21 scores tokens with the same dot product between learned vectors.

A single neuron draws one straight boundary. It can separate points on either side of a line, but not a pattern such as exclusive-or, where the two classes sit on opposite corners of a square. No choice of w₁, w₂ and b puts both (0, 1) and (1, 0) on one side and both (0, 0) and (1, 1) on the other. Two layers of neurons can, which is why networks have layers (chapter 9) and why the activation function must be nonlinear (chapter 8).

Engineering details

A batch of inputs goes through a layer in one call. nn.Linear accepts any number of leading dimensions and transforms only the last one: an input of shape 128 × 20 through Linear(20, 30) gives 128 × 30 .

``python import torch from torch import nn

neuron = nn.Linear(in_features=2, out_features=1) # one neuron: 2 weights + 1 bias with torch.no_grad(): neuron.weight.copy_(torch.tensor([[0.4, 0.6]])) # shape (out_features, in_features) neuron.bias.copy_(torch.tensor([-0.5])) z = neuron(torch.tensor([[2.0, 3.0]])) # tensor([[2.1000]]) a = torch.relu(z) # tensor([[2.1000]]) `

Two practical points follow from the arithmetic. First, the sum is only as precise as the number format: in 32-bit floating point the 2.1 above is stored as the nearest representable value, so printed results are rounded. Second, the magnitude of z grows with the number of inputs: a neuron with 4,096 inputs adds up 4,096 products, which is why initialisation scales the weights by the input count (the 1/in_features` above) to keep sums in a range where the activation still has a useful slope.

8. Activation functions

The activation function is the bend in an otherwise straight calculation. A neuron's weighted sum is linear: double the inputs and the sum doubles. If every layer were linear, a hundred layers would still be one linear function, however many parameters they had. An activation function is a nonlinear transform applied to a neuron's output before it is passed to the next layer , and it is what makes depth worth having.

The problem: linear layers collapse

Adding hidden layers alone does not let a network learn nonlinear patterns, because linear operations on linear operations stay linear . The algebra is short. Two linear layers without an activation compute

``text y = W₂(W₁x + b₁) + b₂ = (W₂W₁)x + (W₂b₁ + b₂) `

which is a single layer with weight matrix W₂W₁ and bias W₂b₁ + b₂. With single numbers: if the first layer computes h = 2x + 1 and the second y = −3h + 4, the pair computes y = −3(2x + 1) + 4 = −6x + 1, a straight line. No stack of such layers can represent a curve, a threshold or the exclusive-or pattern from chapter 7. Once a nonlinearity sits between the layers, stacking them lets the network model very complicated relationships, with each layer learning a higher-level function of the raw inputs .

The common functions

Function — Definition — Output range — Derivative; Linear (none) — f(z) = z — all reals — 1; ReLU — max(0, z) — [0, ∞) — 0 for z < 0, 1 for z > 0; Leaky ReLU — max(0, z) + 0.01·min(0, z); 0.01 is PyTorch's default slope — all reals — 0.01 for z < 0, 1 for z > 0; Sigmoid — σ(z) = 1 / (1 + e^(−z)), output between 0 and 1 — (0, 1) — σ(z)(1 − σ(z)), at most 0.25; Tanh — (e^z − e^(−z)) / (e^z + e^(−z)), range −1 to 1 — (−1, 1) — 1 − tanh²(z), at most 1; GELU — z·Φ(z), with Φ the standard Gaussian cumulative distribution function — about [−0.17, ∞) — smooth, slightly negative for some z < 0; SiLU / Swish — z·σ(z), also known as swish — about [−0.28, ∞) — smooth

Here e is Euler's number and z is the neuron's weighted sum. PyTorch's GELU computes the exact form by default and offers the approximation 0.5·z·(1 + tanh(√(2/π)·(z + 0.044715·z³))) with approximate='tanh' .

Worked values

The same five weighted sums through each function (four decimal places; GELU uses the tanh approximation):

z — ReLU — Leaky ReLU — Sigmoid — Tanh — GELU — SiLU; −2 — 0 — −0.02 — 0.1192 — −0.9640 — −0.0454 — −0.2384; −0.5 — 0 — −0.005 — 0.3775 — −0.4621 — −0.1543 — −0.1888; 0 — 0 — 0 — 0.5 — 0 — 0 — 0; 0.5 — 0.5 — 0.5 — 0.6225 — 0.4621 — 0.3457 — 0.3112; 2 — 2 — 2 — 0.8808 — 0.9640 — 1.9546 — 1.7616

The tanh column matches the example in PyTorch's documentation, which maps [−2, −0.5, 0, 0.5, 2] to [−0.9640, −0.4621, 0, 0.4621, 0.9640] . Read across the rows: ReLU discards all negative information, Leaky ReLU keeps a trace of it, sigmoid and tanh squash large values towards fixed limits, and GELU and SiLU behave like ReLU for large positive inputs but bend smoothly near zero and let small negative values through.

Softmax is different

Softmax does not act on one number at a time. It takes a whole vector of scores z and returns a vector of the same length:

`text softmax(z)ᵢ = exp(zᵢ) / Σⱼ exp(zⱼ) `

where i indexes the output element and the sum runs over every element j. Every output lies in [0, 1] and the outputs sum to 1 . For z = (2, 1, 0.1): e² = 7.389056, e¹ = 2.718282, e^0.1 = 1.105171, total 11.212509, so the output is (0.659001, 0.242433, 0.098566), which sums to 1. Softmax is typically applied at the output layer of a multi-class classifier to turn raw scores (logits) into probabilities that add up to 1.0 , and it assumes each example belongs to exactly one class . It is not a typical hidden-layer activation. In a language model, softmax over the vocabulary is the step that turns the final layer's scores into next-token probabilities (chapter 26).

Try it

The lab plots one activation at a time over inputs from −4 to 4, with its derivative as a dashed line (untick "Show derivative" to hide it). Choose among none (linear), ReLU, Leaky ReLU (slope 0.01), sigmoid, tanh, GELU and SiLU, and move the input slider (it starts at 0.8) to read f(z) and f′(z)` exactly. GELU is drawn with the tanh approximation and its slope is computed numerically. Push sigmoid or tanh out to ±4 and watch the derivative flatten towards zero: that is saturation, and chapter 12 shows why it matters. Softmax is not in this lab because it needs a whole vector; the forward-pass lab in chapter 10 applies it.

Where they are used

MLCC names sigmoid, tanh and ReLU as the three commonly used activation functions . It says ReLU often works a little better than sigmoid or tanh because it is less prone to vanishing gradients and is cheaper to compute, and it recommends starting with ReLU . Transformer language models commonly use smoother functions in their feed-forward layers. LLaMA replaces ReLU with SwiGLU , a gated unit built on Swish . The toy model on this page uses GELU in its feed-forward layers.

Common misconceptions

"Softmax is just another activation." It is a normalisation over a vector, used almost always at the output. Its outputs depend on every input: raising one score lowers every other probability. Hidden layers use element-wise functions such as ReLU; PyTorch's nn.ReLU applies max(0, x) to each element separately and returns a tensor of the same shape .

"Several sigmoid outputs are a probability distribution." Independent sigmoid outputs (one-vs.-all) need not sum to 1; in MLCC's fruit example they sum to 1.43 . That is correct for multi-label problems, where an image can contain both a cat and a dog, and wrong for single-label ones.

"ReLU is linear, so it cannot add nonlinearity." It is linear on each side of zero, but the kink at zero is enough. A weighted sum of shifted ReLUs can trace any continuous piecewise-linear curve of one variable, and a network with many units has many kinks to place.

"Any function will do." MLCC notes that in practice any mathematical function can serve as an activation function, and different layers may use different ones . What separates good choices from bad ones is mostly the derivative: training passes gradients back through every activation, so a function whose slope is near zero over most of its range starves earlier layers of signal.

"GELU, SiLU and Swish are three different inventions." The naming is tangled. PyTorch credits the GELU paper with first coining the name SiLU, and Elfwing et al. and Ramachandran et al. with experimenting with it later . Elfwing et al. proposed the sigmoid-weighted linear unit for reinforcement learning . Ramachandran et al. found x·σ(βx) with an automated search, named it Swish, and report that it tends to beat ReLU on deeper models; with β = 1 it is SiLU .

Advanced insight: saturation and dead units

The derivative column of the table is the part that matters for training. Sigmoid's derivative σ(z)(1 − σ(z)) is 0.25 at z = 0, 0.1050 at z = 2 and 0.006648 at z = 5. Tanh's derivative is 1 at zero but 0.0707 at z = 2. When a unit's weighted sum is large in either direction it saturates: its output barely changes, so almost no gradient flows back through it. Stacked layers multiply these slopes together, so ten sigmoid layers can scale the gradient by at most 0.25¹⁰ ≈ 9.5 × 10⁻⁷. This is one route to the vanishing gradients of chapter 12.

ReLU has the opposite problem. Its slope is exactly 1 for positive inputs, so it does not saturate there, but it is 0 for negative inputs. A ReLU unit whose weighted sum stays below zero for every training example outputs 0 and passes no gradient, so it can never recover; a lower learning rate or a variant such as Leaky ReLU helps . The GELU paper frames the difference as weighting versus gating: GELU weights each input by its value, while ReLU gates inputs by their sign . Unlike ReLU, GELU can output negative values as well as positive ones .

Engineering details

• Cost. Exact GELU needs the Gaussian CDF. The GELU paper gives two cheaper approximations, the tanh form and x·σ(1.702x), for when speed matters more than exactness . The difference is small but not zero, so a model should be run with the variant it was trained with.
• Non-differentiable points. ReLU has no derivative at exactly 0. PyTorch defines a convention for the gradient at such points .
• Softmax and the loss. PyTorch's Softmax normalises along one chosen dimension, and every slice along it sums to 1 . It warns against feeding Softmax output to NLLLoss; LogSoftmax is faster and numerically better .
• Overflow. exp(1000) overflows 64-bit floating point. Because softmax(z) = softmax(z − c) for any constant c (the factor e^(−c) cancels top and bottom), implementations subtract the largest element first: (1000, 999) becomes (0, −1) and gives (0.731059, 0.268941).
• Large outputs. Full softmax is cheap with a few classes but very expensive with many; candidate sampling computes probabilities only for the positive classes and a random sample of negatives .

9. Building a multilayer network

One neuron draws one boundary. A layer is many neurons reading the same inputs side by side, each with its own weights, and a network is several layers in a row, each reading the outputs of the one before. The first layer turns raw inputs into a set of new numbers; the next layer combines those into something more useful; the last layer produces the answer. MLCC describes a standard neural network as three parts: nodes arranged in layers, learned weights and biases connecting each layer to the one below it, and an activation function on each node .

The problem it solves

A linear model needs a person to invent useful combinations of the inputs, such as products of two features ("feature crosses"), before it can fit curved patterns. MLCC frames neural networks as models that learn the useful combinations during training instead , while noting that they are not always better than hand-made feature crosses, only a flexible alternative . The hidden layer is where those learned combinations live.

The vocabulary

Term — Meaning; Input layer — the input values themselves. It has no weights and computes nothing; "2 input neurons" means the input vector has 2 numbers; Hidden layer — a layer between input and output; in MLCC's terminology its nodes are the neurons; Output layer — the last layer; its values are the prediction (or the raw scores that become one); Dense (fully connected) — every unit in a layer reads every unit of the layer before; Weight matrix W — one row per unit in the layer, one column per input; the weights and biases are the connections between consecutive layers; Bias vector b — one bias per unit in the layer; Width — the number of units in a layer; Depth — the number of layers with weights; MLCC calls networks with more than one hidden layer deep networks; Parameters — every weight and bias; a Linear(in, out) layer has in·out + out of them

Different layers may use different activation functions ; a common arrangement is ReLU in the hidden layers and none, sigmoid or softmax at the output, depending on the task.

A 2 → 3 → 1 network, connection by connection

Two inputs, one hidden layer of three ReLU units, one linear output. The input is x = (1, 2).

Hidden layer (weights W₁, shape 3 × 2, and bias b₁, length 3). Each of the six connections carries one weight and contributes weight × input:

Connection — Weight — Contribution; x₁ → h₁ — 0.5 — 0.5 × 1 = 0.5; x₂ → h₁ — −0.2 — −0.2 × 2 = −0.4; x₁ → h₂ — 0.3 — 0.3 × 1 = 0.3; x₂ → h₂ — 0.8 — 0.8 × 2 = 1.6; x₁ → h₃ — −1.0 — −1.0 × 1 = −1.0; x₂ → h₃ — 0.4 — 0.4 × 2 = 0.8

The same numbers as one matrix-vector product for the entire layer:

``text W₁ x b₁ z₁ a₁ = ReLU(z₁) [ 0.5 −0.2 ] [ 0.1 ] [ 0.5 − 0.4 + 0.1 ] [ 0.2 ] [ 0.2 ] [ 0.3 0.8 ] · [ 1 ] + [−0.3 ] = [ 0.3 + 1.6 − 0.3 ] = [ 1.6 ] → [ 1.6 ] [−1.0 0.4 ] [ 2 ] [ 0.2 ] [−1.0 + 0.8 + 0.2 ] [ 0.0 ] [ 0.0 ] `

Row i of W₁ dotted with x is unit i's weighted sum, exactly the single-neuron calculation of chapter 7 done three times at once. Unit h₃ lands exactly on 0, so ReLU outputs 0 and h₃ contributes nothing downstream for this input.

Output layer (weights W₂ = (0.7, −0.5, 1.2), shape 1 × 3, bias b₂ = 0.05, no activation):

Connection — Weight — Contribution; h₁ → y — 0.7 — 0.7 × 0.2 = 0.14; h₂ → y — −0.5 — −0.5 × 1.6 = −0.8; h₃ → y — 1.2 — 1.2 × 0 = 0

y = 0.14 − 0.8 + 0 + 0.05 = −0.61. The connection h₃ → y has the largest weight in the layer and yet contributes nothing here, because its source is silent. A weight's influence on one prediction is weight × the value flowing through it, not the weight alone.

Parameter count: hidden layer 3 × 2 + 3 = 9, output layer 1 × 3 + 1 = 4, total 13.

A 3 → 4 → 2 network: shapes and counts

The forward-pass lab in chapter 10 uses three inputs, four hidden ReLU units and two outputs. Its shapes:

Stage — Shape (one example) — Shape (batch of B) — Parameters; input x — [3] — [B, 3] — none; W₁, b₁ — [4, 3], [4] — same — 12 + 4 = 16; hidden z₁, a₁ — [4] — [B, 4] — none; W₂, b₂ — [2, 4], [2] — same — 8 + 2 = 10; output — [2] — [B, 2] — none; total — — — 26

The rule that makes a stack fit together: the number of columns of each weight matrix equals the width of the layer before it. MLCC works the same count for 3 inputs, a hidden layer of 4 and 1 output: 4 × (3 + 1) = 16 for the hidden layer plus 4 + 1 = 5 for the output, 21 parameters in total , against four for a single linear output reading the same three inputs . One hidden layer of four units multiplied the parameter count by more than five.

At a realistic size

PyTorch's beginner network for 28 × 28 images is Linear(784, 512), ReLU, Linear(512, 512), ReLU, Linear(512, 10): two hidden layers of width 512 and 10 outputs . Applying in·out + out to each layer:

`text Linear(784, 512): 784 × 512 + 512 = 401,920 Linear(512, 512): 512 × 512 + 512 = 262,656 Linear(512, 10): 512 × 10 + 10 = 5,130 total = 669,706 parameters `

The toy language model on this page has about 70,500 parameters. Multi-class networks commonly have one output unit per class rather than a separate classifier per class ; with a softmax output, MLCC notes that the layer feeding softmax must have as many nodes as the output layer .

Check it yourself

nn.Sequential is an ordered container that passes data through its modules in the order they were defined , and a module's parameters can be listed with parameters() or named_parameters() :

`python from torch import nn

net = nn.Sequential(nn.Linear(2, 3), nn.ReLU(), nn.Linear(3, 1)) for name, p in net.named_parameters(): print(name, tuple(p.shape)) # 0.weight (3, 2) 0.bias (3,) 2.weight (1, 3) 2.bias (1,) print(sum(p.numel() for p in net.parameters())) # 13 ``

Common misconceptions

"The input layer is a layer of neurons." It is a vector of numbers. Diagrams draw it as circles for symmetry, but it has no weights, no bias and no activation. Conventions differ on whether to count it when people say "a three-layer network", so the shapes are a more reliable description than the layer count.

"Deeper or wider is always better." None of the sources checked for this page gives a rule for choosing depth or width; MLCC treats it as experimentation . For transformer language models, Kaplan et al. found that performance depends strongly on scale and only very weakly on shape, such as depth against width, within reasonable limits . That result is specific to their setting and does not make architecture irrelevant elsewhere.

"A big enough network can represent anything, so it will learn anything." Universal approximation theorems are often quoted here, but they were not checked against a primary source for this page, so no claim is made about them . Whatever such results say about what a network can represent, they say nothing about whether gradient descent will find those weights from a finite training set.

"Neural network" means transformer. The dense multilayer network of this chapter, often called a multilayer perceptron, is the plainest kind. Convolutional, recurrent and transformer networks often contain dense layers too, but they add other structure, and the original transformer's feed-forward sub-layer is two linear layers with a ReLU between them, applied to each position separately and identically (chapter 24).

Advanced insight: what the hidden layer buys

The 2 → 3 → 1 network computes y = W₂ ReLU(W₁x + b₁) + b₂. Each hidden unit is a ReLU "hinge" along a different direction in input space: zero on one side of the line wᵢ · x + bᵢ = 0, rising linearly on the other. The output is a weighted sum of three hinges, a surface made of flat pieces joined along three lines. Every extra hidden unit adds another hinge that training can place; every extra layer folds the already-folded space again, which is why depth can produce many more linear pieces than width alone for the same parameter count. Without the nonlinearity the hinges disappear and, as chapter 8 showed, the whole network collapses to one linear map. The PyTorch tutorial says the same in one line: the nonlinear activations are what create complex mappings between inputs and outputs .

Engineering details

• Parameter growth is quadratic in width. A dense layer from width n to width n has n² + n parameters. Doubling 512 to 1,024 takes Linear(512, 512) from 262,656 to 1,049,600 parameters, almost exactly four times as many.
• Memory. The 669,706 parameters above take 669,706 × 4 = 2,678,824 bytes, about 2.7 MB, in 32-bit floating point. Training needs more: gradients, optimiser state and saved activations (chapters 10, 12 and 31).
• Storage layout. PyTorch stores each layer's weight as (out_features, in_features) and computes y = xWᵀ + b on row vectors . Textbooks often write Wx with column vectors; the numbers are the same, transposed.
• Shape errors. The commonest bug in a hand-built network is a mismatch between one layer's output width and the next layer's in_features. Printing the parameter shapes, as above, catches it before any training starts.

10. Forward propagation

Forward propagation is the act of running a network: put numbers in at the input, compute each layer from the one before, and read the prediction at the end. Nothing is learned during it and nothing is stored in the weights; it is a fixed sequence of matrix multiplications, additions and element-wise functions. Every prediction a network makes, from a spam score to the next token of a language model, is one forward pass.

The problem it solves

Chapter 9 described a network's structure. Using it needs a procedure that is exact (the same input always gives the same output), efficient on hardware that is good at large matrix products, and able to handle many examples at once. The forward pass goes layer by layer: hidden values are computed from the inputs, and the next layer is computed from the hidden values . Each node's output is the activation of a linear combination of the previous layer's nodes, so the pass alternates "weighted sum, then nonlinearity" .

The mechanism

For a network with layers l = 1 … L:

``text a⁽⁰⁾ = x z⁽ˡ⁾ = W⁽ˡ⁾ a⁽ˡ⁻¹⁾ + b⁽ˡ⁾ a⁽ˡ⁾ = f⁽ˡ⁾(z⁽ˡ⁾) `

x is the input vector, W⁽ˡ⁾ and b⁽ˡ⁾ are layer l's weight matrix and bias vector, z⁽ˡ⁾ its weighted sums, f⁽ˡ⁾ its activation function and a⁽ˡ⁾ its output. For a classifier the last layer usually has no activation of its own: it outputs logits, unbounded raw scores, which softmax turns into probabilities between 0 and 1 .

Seven steps through a 2 → 3 → 2 network

A two-class classifier with two inputs, three ReLU hidden units and two outputs (6 + 3 + 6 + 2 = 17 parameters). Every number below can be checked by hand.

• Input vector. x = (2, 1), shape [2].

• First weight matrix and bias. W₁ has one row per hidden unit and one column per input, shape [3, 2]; b₁ has shape [3].

`text W₁ = [ 1 −1 ] b₁ = [ 0 ] [ 0.5 0.5 ] [ −1 ] [ −1 0.5 ] [ 0.5 ] `

• Matrix multiplication. Each row of W₁ dotted with x, shape [3, 2] · [2] → [3]:

`text W₁x = ( 1·2 + (−1)·1, 0.5·2 + 0.5·1, (−1)·2 + 0.5·1 ) = ( 1, 1.5, −1.5 ) `

• Bias addition. z₁ = W₁x + b₁ = (1 + 0, 1.5 − 1, −1.5 + 0.5) = (1, 0.5, −1), shape [3].

• Activation. a₁ = ReLU(z₁) = (1, 0.5, 0). The third unit's negative sum becomes exactly 0.

• Next layer. W₂ has shape [2, 3] and b₂ = (0, 0.5):

`text W₂ = [ 1 2 −1 ] [ −1 0 3 ]

z₂ = W₂a₁ + b₂ = ( 1·1 + 2·0.5 + (−1)·0 + 0, (−1)·1 + 0·0.5 + 3·0 + 0.5 ) = ( 2, −0.5 ) `

These are the logits, shape [2].

• Output prediction. Softmax: e² = 7.389056, e^(−0.5) = 0.606531, total 7.995587, so

`text p = ( 7.389056 / 7.995587, 0.606531 / 7.995587 ) = ( 0.924142, 0.075858 ) `

The network assigns probability 0.924 to class 1. The weights were chosen by hand, so the number says nothing about any real task; it shows only that the computation is exact.

The same pass on a batch

Stack examples as rows of a matrix and the same two products handle all of them. With X of shape [2, 2] holding (2, 1) and (0, 2), row-vector frameworks compute Z₁ = XW₁ᵀ + b₁, shape [2, 3]:

Example — z₁ — a₁ — z₂ — p; (2, 1) — (1, 0.5, −1) — (1, 0.5, 0) — (2, −0.5) — (0.924142, 0.075858); (0, 2) — (−2, 0, 1.5) — (0, 0, 1.5) — (−1.5, 5) — (0.001501, 0.998499)

The bias row is added to every row of the product. Each row is computed independently: the second example does not change the first one's answer. PyTorch's documentation gives the same pattern at a larger size, a batch of 128 inputs of width 20 through Linear(20, 30) giving a 128 × 30 output, one row per example .

Try it

The lab runs a fixed 3 → 4 → 2 network with ReLU and softmax: 4 × 3 + 4 + 2 × 4 + 2 = 26 parameters. Its weights are seeded random numbers, not trained, so it demonstrates the computation rather than a useful prediction. Set the three inputs with sliders (−3 to 3; they start at 0.5, −1 and 2), then step through five stages: the input vector, the weighted sums z₁ = W₁x + b₁, the ReLU activations, the logits z₂ = W₂a₁ + b₂, and the softmax probabilities. Play animates the stages and the speed control sets the pace; next and previous move one stage at a time. At each stage, select a value to see the exact dot product that produced it, and check the shapes listed under the matrices. At the last stage the probabilities sum to 1.

Where it appears

In PyTorch, each nn.Module subclass puts its forward computation in a forward method , and you run it by calling the model object rather than model.forward() directly, because the call also runs background operations . Inputs are first reshaped into vectors; the tutorial's nn.Flatten` turns each 28 × 28 image into 784 values and keeps the batch dimension . A language model's forward pass has the same skeleton with more stages: token IDs become vectors, pass through a stack of blocks, and end as one logit per vocabulary entry. For the toy model on this page that is 233 logits at each of up to 56 positions, all computed in one pass.

Common misconceptions

"Running a model teaches it." A forward pass reads the weights and never writes them. Training is a separate procedure that adds a backward pass and an update (chapter 12). GPT-3 is a clear case: it was evaluated with no gradient updates or fine-tuning, the task supplied only as text in the prompt, and the weights did not change at inference time .

"The network outputs probabilities." The last linear layer outputs logits . Probabilities appear only if softmax (or sigmoid) is applied, and during training it is usually left to the loss function, which combines it with the logarithm for numerical stability (chapter 11).

"Examples in a batch influence each other." In a plain dense network they do not: each row of the batch is multiplied by the same weights independently. Some layers are exceptions, and model.eval() exists for them: it changes only modules that behave differently in training, such as Dropout and BatchNorm, and is the same as train(False) . In a language model, positions within one sequence do interact, through attention, but separate sequences in a batch still do not.

"A larger output number means more confidence." Softmax keeps the order of the logits, but the probabilities depend on the differences between them. Logits (2, −0.5) and (102, 99.5) give exactly the same probabilities.

Advanced insight: what training adds to the forward pass

When gradients are required, the forward pass does extra work. Autograd performs the requested computation and at the same time builds a graph of the operations, which the backward pass later uses to compute gradients . Some operations must also save intermediate results for the backward pass, which is why training uses more memory than inference . In the seven steps above, the backward pass will need x, a₁ and which hidden units were positive; a deep network keeps such values for every layer and every example in the batch until the gradients have been computed.

For inference-only passes, torch.no_grad() turns that tracking off :

``python model.eval() # Dropout and BatchNorm switch to evaluation behaviour with torch.no_grad(): # no graph, no saved activations logits = model(x_batch) # call the module, not model.forward probs = logits.softmax(dim=-1) `

Engineering details

• Cost. A dense layer from n_in to n_out performs n_in × n_out multiply-adds per example. The 3 → 4 → 2 lab network does 12 + 8 = 20; the 784 → 512 → 512 → 10 network from chapter 9 does 401,408 + 262,144 + 5,120 = 668,672 per example. The activations and bias additions are cheap by comparison; the matrix products dominate.
• Batching. Running many examples as one matrix-matrix product is much faster per example than many matrix-vector products, because the weights are loaded once and reused for every row. In the PyTorch tutorial's terms, batch size is the number of samples propagated through the network before the parameters are updated , and in mini-batch SGD the examples are chosen at random, their gradients averaged, and the parameters updated once per batch .
• Conventions. Textbooks write Wx with column vectors; frameworks typically store a batch as rows and compute XWᵀ + b` . The values are identical; only the layout differs.
• Determinism. Mathematically the pass is exact. On real hardware, floating-point addition is not associative, so summing the same products in a different order (different batch sizes, kernels or devices) can change the last digits of a result.

11. Loss functions

A forward pass produces an output; the training data says what the output should have been. A loss function turns the gap between the two into one number. Loss is a numerical measure of how wrong a model's predictions are , and the goal of training is to drive it as low as possible . Everything the network learns, it learns because some change to its weights made this number smaller.

The problem it solves

"Wrong" means different things for different tasks. Predicting a house price 10,000 too high is a matter of degree; calling a cat a dog is a matter of which answer got the probability. Training also needs the measure to be a smooth function of the weights, so that its slope can say which way to move them (chapter 12). Accuracy fails that test: it changes in jumps, and a small weight change usually leaves it exactly the same. So each kind of task gets a loss that is differentiable and that rewards what the task actually cares about. Loss measures distance, not direction, which is why every regression loss removes the sign, by squaring or by taking the absolute value .

Regression: squared and absolute error

For N examples with targets yᵢ and predictions ŷᵢ:

``text MSE = (1/N) Σᵢ (yᵢ − ŷᵢ)² MAE = (1/N) Σᵢ |yᵢ − ŷᵢ| `

Mean squared error averages the squared differences; mean absolute error averages the absolute differences . MLCC's single-point example: predicting 23.1 mpg when the truth is 24 gives a squared loss of (24 − 23.1)² = 0.81 .

Three predictions ŷ = (2.5, 0, 2) against targets y = (3, −1, 2) have errors (−0.5, 1, 0):

`text MSE = (0.25 + 1 + 0) / 3 = 0.416667 MAE = (0.5 + 1 + 0) / 3 = 0.5 `

Add a fourth example with an error of 10, an outlier:

`text MSE = (0.25 + 1 + 0 + 100) / 4 = 25.3125 MAE = (0.5 + 1 + 0 + 10) / 4 = 2.875 `

One point now supplies 99% of the squared loss. Squaring makes large errors larger and errors below 1 smaller, so MSE punishes outliers more than MAE , and a model trained with MSE is pulled towards outliers while MAE is more robust to them . Which is right depends on whether the outliers are errors to ignore or rare cases that matter.

Binary classification: log loss

When the output is a probability y′ between 0 and 1 for a label y ∈ {0, 1}, the loss is

`text Log Loss = −(1/N) Σ [ y·log(y′) + (1 − y)·log(1 − y′) ] `

. Only one of the two terms is active for each example: −log(y′) when the label is 1, −log(1 − y′) when it is 0. With natural logarithms, a correct, confident prediction (y = 1, y′ = 0.9) costs −ln 0.9 = 0.105361. A confident wrong one (y = 1, y′ = 0.1, or y = 0, y′ = 0.9) costs −ln 0.1 = 2.302585, about 22 times as much, and the cost grows without limit as the wrong probability approaches certainty. MLCC explains why squared error is not used here: it suits models whose output changes at a constant rate, and a sigmoid output does not .

Multi-class classification: cross-entropy

With C classes the network outputs C logits, softmax turns them into probabilities, and the loss is minus the log of the probability given to the correct class y:

`text loss = −log( exp(z_y) / Σ_c exp(z_c) ) `

which is PyTorch's CrossEntropyLoss per example (with class weights of 1), equivalent to LogSoftmax followed by NLLLoss . For logits (2, 1, 0.1), softmax gives (0.659001, 0.242433, 0.098566) (chapter 8). If the correct class is the first, the loss is −ln 0.659001 = 0.417030; if it is the third, −ln 0.098566 = 2.317030. A model that knows nothing and spreads probability evenly over four classes scores −ln(1/4) = ln 4 = 1.386294 on every example, a useful baseline: a four-class classifier whose loss is above 1.386 is doing worse than uniform guessing.

Token prediction: cross-entropy at every position

A language model is a classifier whose classes are vocabulary entries, applied at every position of a sequence. A training step scores all positions at once and averages the per-token cross-entropy . Suppose the model gives the true next tokens of a four-token stretch the probabilities 0.5, 0.25, 0.8 and 0.1:

`text position: 1 2 3 4 P(target): 0.5 0.25 0.8 0.1 −ln P: 0.693147 1.386294 0.223144 2.302585 mean loss = 4.605170 / 4 = 1.151293 nats `

The sum is −ln(0.5 × 0.25 × 0.8 × 0.1) = −ln 0.01 = ln 100: averaging log losses is the same as taking the geometric mean of the probabilities. Perplexity is the exponential of the evaluation cross-entropy , here e^1.151293 = 3.162278, the model's average "effective number of choices". For the toy model on this page, with a 233-entry vocabulary, uniform guessing would score ln 233 = 5.451038 per token. These probabilities are illustrative numbers chosen for the arithmetic, not measurements of any model.

Try it

The lab has two modes. In regression, sliders set the prediction (−3 to 5, starting at 2.5) and the target (−2 to 4, starting at 1); the plot shows squared error as a parabola and absolute error as a V, with the current values and gradients written out. At the start the error is 1.5, so the squared error is 2.25 with gradient 2(ŷ − y) = 3, and the absolute error is 1.5 with gradient +1. In classification, one slider sets the model's probability for the correct class (starting at 0.7) and the plot shows −ln p: 0.356675 at the start, a gentle slope near 1 and a wall near 0. The reference line ln 4 = 1.3863 marks uniform guessing among four classes. Losses are in nats (natural logarithm), as PyTorch computes them, for a single example.

Where it appears

PyTorch's MSELoss and L1Loss compute the squared and absolute differences per element and average them by default ; the reduction argument chooses 'none', 'mean' or 'sum' . BCEWithLogitsLoss combines the sigmoid and binary cross-entropy in one step, which is more numerically stable because of the log-sum-exp trick . CrossEntropyLoss takes raw logits, which need not be positive or sum to 1 , and the PyTorch tutorial passes the model's logits straight to it, with no softmax layer in the model . Its defaults are weight=None, ignore_index=-100, reduction='mean' and label_smoothing=0.0` .

Common misconceptions

"Lower loss always means a better model." Lower training loss means the model fits its training data better. Whether it predicts new data better is a separate question, and chapter 13 shows a model whose training loss reaches zero while its error on unseen data grows.

"The loss is the metric." Loss is what the optimiser minimises because it is smooth. The number reported to people is often a different metric (accuracy, error rate, a business measure) that is not differentiable. A model can improve its loss without changing its accuracy, by becoming more confident about answers it already had right.

"Cross-entropy needs probabilities as input." In PyTorch it is the reverse: CrossEntropyLoss expects unnormalised logits and applies log-softmax internally. Feeding it softmax outputs applies softmax twice and quietly weakens the gradients. The binary losses split the other way: BCELoss expects probabilities, for example sigmoid outputs , while BCEWithLogitsLoss expects logits.

"A cross-entropy of zero is the goal." Zero needs probability exactly 1 on every correct answer, which a softmax over finite logits never reaches. Pushing towards it means ever-larger logits and overconfidence, one reason for label smoothing below.

Advanced insight: the gradients are simple

The loss is chosen partly for what its slope looks like, because the slope is what training uses.

Loss — Gradient with respect to the model output; Squared error (ŷ − y)² — 2(ŷ − y): proportional to the error; Absolute error abs(ŷ − y) — ±1: the same push however large the error; Sigmoid + binary cross-entropy — p − y with respect to the logit; Softmax + cross-entropy — p − onehot(y) with respect to the logits

For the logits (2, 1, 0.1) with the first class correct, the gradient is (0.659001 − 1, 0.242433, 0.098566) = (−0.340999, 0.242433, 0.098566). Training will raise the correct logit and lower the others, each in proportion to how much probability it was wrongly given. The components sum to zero, because adding the same constant to every logit does not change softmax. This tidy form is why frameworks fuse the activation with the loss.

Averaging matters too. Using the mean rather than the total log loss decouples batch size from learning rate, so the two can be tuned independently .

Engineering details

• Never take log(0). BCELoss clamps its log outputs to at least −100 so that a probability of exactly 0 does not make the loss infinite .
• Imbalanced classes. BCEWithLogitsLoss takes pos_weight: with 100 positive and 300 negative examples, pos_weight = 300/100 = 3; values above 1 raise recall and values below 1 raise precision . CrossEntropyLoss's weight argument rescales each class's loss for unbalanced training sets .
• Ignored positions. Targets equal to ignore_index (default −100) add nothing to the loss or gradient, and the mean is taken over the targets that remain . Language-model training uses this to leave padding or prompt tokens out of the loss . The toy model on this page uses the same idea for its tool calls: the <result>…</result> tokens are written by the application, not the model, so they are excluded from the loss.
• Soft targets. Targets can be class indices or per-class probabilities; indices are faster and recommended unless soft labels are needed . label_smoothing mixes the true target with a uniform distribution over the classes : with smoothing 0.1 over three classes, the target (1, 0, 0) becomes (0.9 + 0.1/3, 0.1/3, 0.1/3) = (0.933333, 0.033333, 0.033333).
• Older arguments. size_average and reduce are being deprecated in favour of reduction, and passing either overrides it .

12. Backpropagation and gradient descent

Training is a loop of small corrections. Run the network, measure the loss, work out for every weight whether nudging it up would raise or lower the loss and by how much, then nudge every weight a little in the helpful direction. Repeat many times. Gradient descent is the nudging rule; backpropagation is the bookkeeping that finds the "which way and how much" for every weight at once. Backpropagation is the most common training algorithm for neural networks, and it is what makes gradient descent practical for networks with many layers .

The problem it solves

A network has thousands to billions of weights. The naive way to find how the loss depends on one weight is to change it slightly, rerun the forward pass and compare. That costs one extra forward pass per weight, per step: hopeless at scale. Backpropagation gets the slope with respect to every weight from one forward pass and one backward pass of similar cost, by reusing intermediate results with the chain rule.

The ideas, in order

• Partial derivative. ∂L/∂w is how fast the loss L changes when one parameter w changes and everything else is held fixed. Its sign says which direction raises the loss; its size is the parameter's sensitivity.
• Gradient. The vector of all partial derivatives, one per parameter. It points in the direction of steepest increase, so moving against it lowers the loss fastest, locally.
• Chain rule. If L depends on u and u depends on w, then ∂L/∂w = (∂L/∂u) · (∂u/∂w). Slopes along a path multiply.
• Computational graph. The forward pass as a graph of simple operations. Each operation knows its own local derivative; backpropagation walks the graph from the loss back to the inputs, multiplying local derivatives as it goes.
• Update rule. New parameter = old parameter − learning rate × gradient :

``text w_new = w_old − η · ∂L/∂w `

η (eta) is the learning rate, the "small amount" gradient descent moves in the downhill direction . With a gradient of magnitude 2.5 and a learning rate of 0.01, the parameter changes by 0.025 . The loop is: compute the loss, find the direction that reduces it, move the parameters a small amount that way, and repeat .

One parameter, fully worked

A model with a single weight, ŷ = w · x, one training example (x = 2, y = 6), squared-error loss:

`text L(w) = (w·x − y)² = (2w − 6)² dL/dw = 2(w·x − y) · x (chain rule: outer square, then inner 2w − 6) `

Start at w = 1:

`text ŷ = 1 × 2 = 2, residual ŷ − y = −4, L = 16 dL/dw = 2 × (−4) × 2 = −16 `

The negative sign says that increasing w lowers the loss. Check it numerically: L(1.001) = (−3.998)² = 15.984004 and L(0.999) = (−4.002)² = 16.016004, so the slope is (15.984004 − 16.016004) / 0.002 = −16.000. The derivative is right.

Update with η = 0.05:

`text step 1: w = 1 − 0.05 × (−16) = 1.8 ŷ = 3.6 L = (−2.4)² = 5.76 step 2: w = 1.8 − 0.05 × (−9.6) = 2.28 ŷ = 4.56 L = (−1.44)² = 2.0736 `

The best weight is w = 3. Its distance from the current weight goes 2, 1.2, 0.72: each step multiplies it by 1 − 2ηx² = 1 − 0.4 = 0.6. That factor explains the learning rate's behaviour on this problem. For 0 < η < 0.125 the factor is between 0 and 1 and w creeps up steadily; at η = 0.125 it is 0 and one step lands exactly on 3; between 0.125 and 0.25 it is negative, so w overshoots and oscillates while still converging; above 0.25 its magnitude exceeds 1 and every step overshoots by more than the last, so the loss grows without limit.

The chain rule through two layers

Now a network with a hidden unit: z₁ = w₁x + b₁, a = σ(z₁) (sigmoid), ŷ = w₂a + b₂, loss L = ½(ŷ − y)². Take x = 2, y = 1, w₁ = 0.5, b₁ = 0, w₂ = 2, b₂ = −1.

Forward, keeping every intermediate value:

`text z₁ = 0.5 × 2 + 0 = 1 a = σ(1) = 0.731059 ŷ = 2 × 0.731059 − 1 = 0.462117 L = ½(0.462117 − 1)² = 0.144659 `

Backward, from the loss towards the input. Each line multiplies the gradient arriving from above by one local derivative:

`text ∂L/∂ŷ = ŷ − y = −0.537883 ∂L/∂w₂ = ∂L/∂ŷ · a = −0.537883 × 0.731059 = −0.393224 ∂L/∂b₂ = ∂L/∂ŷ · 1 = −0.537883 ∂L/∂a = ∂L/∂ŷ · w₂ = −0.537883 × 2 = −1.075766 ∂a/∂z₁ = a(1 − a) = 0.731059 × 0.268941 = 0.196612 ∂L/∂z₁ = ∂L/∂a · ∂a/∂z₁ = −1.075766 × 0.196612 = −0.211508 ∂L/∂w₁ = ∂L/∂z₁ · x = −0.211508 × 2 = −0.423017 ∂L/∂b₁ = ∂L/∂z₁ · 1 = −0.211508 `

Written out in one line, ∂L/∂w₁ = (ŷ − y) · w₂ · σ′(z₁) · x: one factor per step back through the graph. A finite-difference check on w₁ gives −0.423017 as well. Notice the reuse: ∂L/∂ŷ is computed once and shared by four gradients, and ∂L/∂z₁ once and shared by two. In a deep network that sharing is the whole saving.

Update all four parameters with η = 0.5: w₁ = 0.711508, b₁ = 0.105754, w₂ = 2.196612, b₂ = −0.731059. A new forward pass gives ŷ = 1.074175 and L = 0.002751, down from 0.144659. The prediction slightly overshoots the target, a hint that this learning rate is on the large side.

Try it

Two labs. The first is a loss landscape with one parameter. On the convex bowl L = ½(w − 3)² + 1 the gradient is w − 3; you start at w = −2 with η = 0.3, so the first step computes gradient −5, update −0.3 × (−5) = +1.5, next w = −0.5, and the loss falls from 13.5 to 7.125. Use Step or 10 steps, click the curve to start somewhere else, and move the learning-rate slider (0.01 to 2.2). Here each step multiplies the distance to the minimum by 1 − η, so any η below 2 converges, η = 1 lands in one step and η > 2 diverges. The non-convex landscape has two valleys: where you start decides which one you end in, because gradient descent only follows the local slope.

The second lab trains a real network in your browser: 2 inputs, two hidden layers of 8 tanh units and one output, 105 weights in all, on 120 points of two interleaved "moons". Every step is a full forward pass over all 120 points, binary cross-entropy loss, backpropagation by the page's automatic-differentiation engine (tested against finite differences) and a plain gradient-descent update with no momentum. Press Train, or take one step at a time; the panel shows one weight's update written out as w − η × g. Watch the shaded decision regions bend to follow the moons and the loss curve fall. Try learning rates between 0.01 and 3 and the four initial-weight seeds; training stops after 3,000 steps, and the accuracy shown is on the training points.

Where it appears

Libraries run backpropagation for you . PyTorch's autograd is reverse-mode automatic differentiation: it records operations as a directed acyclic graph with inputs as leaves and outputs as roots, and applies the chain rule from the output back to the inputs . Each training step has three calls: optimizer.zero_grad(), because gradients add up by default; loss.backward(), to compute gradients; and optimizer.step(), to update the parameters .

`python for x, y in loader: optimizer.zero_grad() # clear gradients left from the last step loss = loss_fn(model(x), y) # forward pass and loss; autograd records the graph loss.backward() # backward pass: fills p.grad for every parameter optimizer.step() # update: for plain SGD, p = p - lr * p.grad ``

Common misconceptions

"Backpropagation is the learning algorithm." It computes gradients and nothing else. The update is the optimiser's job: optimizer.step() changes the parameters using the gradients that backward() collected . The same gradients can feed plain SGD, momentum or Adam.

"Gradient descent finds the best weights." MLCC states that linear models always have a convex loss surface, so gradient descent finds the global minimum, and it makes that guarantee only for linear models . Neural networks have non-convex losses, like the second landscape in the lab. Training finds a point where more iterations stop lowering the loss, which is what MLCC calls convergence , not a certified optimum.

"There is a correct learning rate." Too small converges slowly; too large never converges and bounces around the minimum or diverges . The ideal value depends on the model and the data , as the factor 1 − 2ηx² above showed even for one weight.

"Inference uses backpropagation." It does not. Gradients are computed only during training; a deployed model runs forward passes with fixed weights.

Optimisers

MLCC's worked example shows plain gradient descent at work: with MSE and learning rate 0.01, starting from weight 0 and bias 0 at loss 303.71, one step gives weight 1.20, bias 0.34 and loss 170.84, and by step 6 the loss is 42.17 . Real training adds three refinements.

Mini-batches. Stochastic gradient descent uses one example per update and is noisy; mini-batch SGD uses batches between one example and the whole dataset . The number of updates is epochs × examples / batch size: 1,000 examples, batch size 100 and 20 epochs give 200 updates . The PyTorch tutorial uses learning rate 1e-3, batch size 64 and 5 epochs .

Momentum. torch.optim.SGD defaults in PyTorch 2.14 are lr=0.001, momentum=0, dampening=0, weight_decay=0, nesterov=False . With momentum it keeps a running buffer b_t = μ·b_(t−1) + (1 − τ)·g_t and steps θ_t = θ_(t−1) − lr·b_t, where μ is the momentum and τ the dampening ; Nesterov momentum works only when momentum is non-zero . The buffer averages recent gradients, so consistent directions build up speed and oscillating ones cancel.

Adam. Adam is a first-order method that adapts each parameter's step size using running estimates of the gradient's first and second moments . PyTorch's update is m_t = β₁m + (1 − β₁)g, v_t = β₂v + (1 − β₂)g², m̂ = m_t/(1 − β₁ᵗ), v̂ = v_t/(1 − β₂ᵗ), θ ← θ − lr·m̂/(√v̂ + ε) , with defaults lr=0.001, betas=(0.9, 0.999), eps=1e-08 ; ε is there for numerical stability . Work the first step for a gradient g = 0.5: m₁ = 0.05, v₁ = 0.00025, m̂ = 0.05/0.1 = 0.5, v̂ = 0.00025/0.001 = 0.25, step = 0.001 × 0.5/0.5 ≈ 0.001. With g = 50 the step is again about 0.001. Adam's steps are scaled by the gradient's recent typical size, so a parameter with tiny gradients and one with huge gradients both move at roughly the learning rate. Its authors say its hyperparameters usually need little tuning .

Vanishing and exploding gradients

The two-layer example's gradient was a product of factors, one per step back. A deep network multiplies one such factor per layer. If the factors are mostly below 1 the product shrinks towards zero: the layers nearest the input learn slowly or not at all. These are vanishing gradients, and ReLU helps . Sigmoid's slope is at most 0.25, so ten sigmoid layers can shrink a gradient by 0.25¹⁰ ≈ 9.5 × 10⁻⁷. If the factors are mostly above 1, as with large weights, the product grows and training cannot converge: exploding gradients, where batch normalisation or a lower learning rate help . Twenty factors of 1.5 multiply to about 3,325. Pascanu et al., writing about recurrent networks, define exploding gradients as a large increase in the gradient norm during training , and vanishing gradients as long-term components shrinking exponentially to norm 0, which stops the model learning links between distant events .

Gradient clipping. Pascanu et al. propose gradient norm clipping against explosion : if the norm reaches a threshold, rescale the gradient to g × threshold / ‖g‖, keeping its direction and capping its size . The gradient (3, 4) has norm 5; with threshold 1 it becomes (0.6, 0.8). Earlier work clipped each element separately , which can change the gradient's direction. PyTorch's clip_grad_norm_ computes one total norm over all parameters' gradients, as if they were one vector, and modifies them in place ; it defaults to the 2-norm and returns the total norm from before clipping . LLaMA's training used clipping at 1.0 .

Engineering details

• Only leaf tensors with requires_grad=True, such as a module's parameters, have gradients stored in .grad ; backward() accumulates into them , which is why forgetting zero_grad() silently sums gradients across steps.
• The graph is rebuilt on every iteration, so ordinary Python control flow can change the computation from step to step .
• When the loss curve oscillates, MLCC advises lowering the learning rate and checking the data for bad examples, not raising it . A sudden spike can come from NaNs in the input, such as a division by zero, or from a batch full of outliers .

13. Generalization, overfitting and regularization

A student who memorises last year's exam answers can score perfectly on last year's exam and fail this year's. Models have the same failure. Training minimises the loss on the training data, but the training data is only a sample; what matters is how the model does on examples it has never seen. Generalisation, making good predictions on new data, is the opposite of overfitting .

The problem

Overfitting means fitting (memorising) the training set so closely that the model fails on new data . Its opposite, underfitting, means doing badly even on the training data . MLCC gives two broad causes of overfitting: a training set that does not represent real data, and a model that is too complex . Training error alone cannot tell these cases apart, because a flexible enough model can drive it to zero. So data is split:

Set — Used for — Error measured on it; Training — fitting the weights — training error; Validation — comparing models and settings during development — validation error; Test — a final check, used as rarely as possible — test error

MLCC recommends exactly this: train on the training set, test on the validation set during development, and keep the test set for a final check . scikit-learn calls training and testing on the same data a methodological mistake, because a model that simply repeated the labels it had seen would score perfectly . The typical sign of overfitting is training loss falling or holding steady while validation loss starts to rise .

A worked example

The true relationship is y = x. Four training points carry noise: (0, 0.2), (1, 0.8), (2, 2.3), (3, 2.9). Four validation points lie on the true line: x = 0.5, 1.5, 2.5, 4. Error is root-mean-square error (RMSE).

A straight line (two coefficients), fitted by least squares, is ŷ = 0.11 + 0.96x. Its training residuals are (0.09, −0.27, 0.27, −0.09):

``text training RMSE = √((0.0081 + 0.0729 + 0.0729 + 0.0081) / 4) = √0.0405 = 0.2012 validation RMSE = √((0.0081 + 0.0025 + 0.0001 + 0.0025) / 4) = √0.0033 = 0.0574 `

A cubic (four coefficients) can pass through all four training points exactly: ŷ = 0.2 − 0.45x + 1.35x² − 0.3x³. Training RMSE is 0. On the validation points it predicts 0.275, 1.55, 2.825 and 0.8 where the truth is 0.5, 1.5, 2.5 and 4:

`text validation RMSE = √((0.050625 + 0.0025 + 0.105625 + 10.24) / 4) = √2.599688 = 1.6124 `

The model with zero training error is 28 times worse on unseen data. Even leaving out x = 4, which lies outside the training range, the cubic's validation RMSE is 0.230 against the line's 0.060. Four coefficients for four points let the cubic fit the noise as if it were signal. Counting makes this general: a polynomial of degree d has d + 1 coefficients, so with n training points degree n − 1 can always interpolate them exactly.

Fixes: regularisation and friends

• Simpler model or more representative data. MLCC's fixes include a simpler model, for example fewer features, a higher regularisation rate, and making sure the training and test sets are statistically equivalent .
• L2 regularisation. Add a complexity penalty, the sum of squared weights w₁² + … + wₙ², in which large weights dominate and weights near zero barely count . Training minimises loss + λ × complexity, where λ is the regularisation rate . For weights (3, −0.5, 0.1) the sum is 9 + 0.25 + 0.01 = 9.26, so λ = 0.01 adds 0.0926 to the loss, 97% of it from the largest weight. L2 pushes weights towards 0 but never all the way ; λ = 0 switches it off and carries the highest risk of overfitting .
• Early stopping. End training before full convergence, for example when validation loss starts to rise . MLCC calls it quick but rarely optimal compared with a well-tuned regularisation rate .
• Dropout. Randomly drop units and their connections during training , which keeps units from co-adapting too much . With drop probability p = 0.5 and activations (2, 4, 6, 8), one random mask might keep the first and third: PyTorch zeroes the others and scales the survivors by 1/(1 − p) = 2, giving (4, 0, 12, 0). Each element's expected value is unchanged: 0.5 × 2 × a = a.
• Data augmentation. Train on altered copies of the examples (an image flipped, cropped or recoloured) so the model must learn what stays the same under those changes. No primary source checked for this page defines it or measures how much it helps, so no number is given here .

Try it

The lab fits polynomials to 12 noisy training points drawn from y = sin(2.5x) + 0.3x` and scores each fit on 40 validation points from the same distribution. The fits are exact least squares. Move the degree slider from 0 to 11 (it starts at 3): the left plot shows the fit against the true curve, and the right plot shows training and validation RMSE for every degree at once. Training error only falls as the degree rises; validation error falls, then climbs. At degree 11 the polynomial has 12 coefficients and passes through all 12 training points, and it swings wildly between them. The noise slider (0 to 0.6, starting at 0.25) and "New sample" change the data; watch which degree the lab reports as best on validation error.

Where it appears

Every serious training run keeps a held-out set and watches the gap between training and validation loss. The toy language model on this page is a deliberate extreme: it is trained on a small corpus until it has memorised it, which is why it answers its own training questions fluently and generalises poorly beyond them. That is acceptable for a demonstration of the mechanism and would be a failure in a real system.

Common misconceptions

"A validation set can be reused for ever." Validation and test sets wear out: the more decisions are based on them, the less they say about new data, so they should be refreshed . Repeatedly tuning against the test set makes the model implicitly fit it, "teaching to the test" .

"The test score proves real-world performance." Good generalisation assumes examples are independent and identically distributed and that the data is stationary, not changing much over time . It also assumes training, validation, test and real-world data come from the same distribution; under distribution shift, held-out performance stops predicting real-world performance . MLCC's example: viewer tastes change, so ten years of history may not predict the next three . Good shuffling before splitting makes the partitions much more likely to match , and validation and test examples that duplicate training examples should be removed, or the evaluation looks too good .

"Bigger models always overfit more, and more data always helps." Nakkiran et al. observe "double descent": as model size grows, test performance first gets worse and then better again, against the classic U-shaped curve . It also happens as training goes on, not only with model size , and in some regimes near the critical point even four times as much training data made test performance worse . The U-shaped curve in the lab is real for polynomials fitted exactly; it is not a law for every model family.

"Dropout runs at inference." In PyTorch it switches off only when the model is put in evaluation mode with model.eval(), the same as train(False) . Forgetting that call leaves units being dropped at random, so the same input can give different predictions.

Advanced insight: L2 regularisation is not always weight decay

Loshchilov and Hutter show that L2 regularisation and weight decay are equivalent for plain SGD (after rescaling by the learning rate) but not for adaptive methods such as Adam . The reason is visible in the Adam update of chapter 12: an L2 term added to the gradient is divided by √v̂ like the rest, so parameters with large gradients are barely decayed. In SGD and plain Adam, PyTorch's weight_decay is added to the gradient as an L2 penalty, g = g + λθ . AdamW decouples the decay from the gradient step, recovering the original form : PyTorch applies θ = θ − lr·λ·θ before the Adam step, so the decay never enters the moment estimates . AdamW's default weight_decay is 0.01, while Adam's and SGD's is 0 . With lr = 0.001 and λ = 0.01, each step multiplies every weight by 1 − 0.00001; over 16,000 steps, the length of the toy model's training run, that alone would shrink a weight to 0.99999¹⁶⁰⁰⁰ ≈ 0.8521 of its value if no gradient pushed back. Learning rate and regularisation pull weights in opposite directions and have to be balanced .

Dropout has a similar implementation subtlety. The dropout paper approximates averaging the many "thinned" networks by using one unthinned network with smaller weights at test time . PyTorch's nn.Dropout instead zeroes each element with probability p, drawn independently on every forward call , and scales the survivors by 1/(1 − p) during training, so in evaluation it is the identity; the expected result is the same . The default p is 0.5 . MLCC notes that a rate of 0.0 means no dropout and 1.0 drops everything, so the model learns nothing; useful values lie between, and higher rates regularise more .

Engineering details: cross-validation

A fixed three-way split leaves less data for training, and results can depend on which random split was drawn . In k-fold cross-validation each of the k folds is used once for evaluation while the other k − 1 train the model, and the reported score is the average . With 1,000 examples and k = 5, five models are trained on 800 examples each and evaluated on the remaining 200, and every example is used for evaluation exactly once. It costs k trainings but wastes little data , which makes it most attractive for small datasets; for a large network, where one training run is already expensive, k runs may be unaffordable. For time series, scikit-learn's TimeSeriesSplit trains on earlier folds and tests on later ones, each training set containing the ones before it , because shuffling time-ordered data would let the model learn from the future.

14. Why language models need tokens

Type "Artificial intelligence is fascinating." into a chat window and you see a sentence. The model never sees that sentence. Models can only process numbers, so a component called a tokenizer first turns the text into numerical data the model can consume . Language models see text as a sequence of numbers called tokens . Everything that follows, from embeddings to sampling, operates on those numbers and on vectors derived from them, not on words.

The problem: choosing the unit

Someone must decide what one input unit is. The set of distinct units is the vocabulary; each unit gets an integer token ID from 0 up to the vocabulary size, and the model identifies tokens only by these IDs . There are three obvious choices, and each fails in its own way.

• Words. English has over 500,000 words, each needing its own ID . Related forms such as "dog" and "dogs" become unrelated tokens, so the model initially has no way of knowing they are similar . Words missing from the vocabulary are mapped to a special unknown token such as [UNK] or <unk>, and every unknown token is lost information . A word-level vocabulary also makes the embedding matrix enormous, which costs memory and compute .
• Characters. The vocabulary is small and unknown tokens are rare, but each character carries little meaning and sequences become much longer: one word can easily become 10 or more tokens .
• Sub-words. Frequent words stay whole and rare words are decomposed into meaningful pieces, for example "annoyingly" into "annoying" + "ly" . This gives good coverage with a small vocabulary and close to no unknown tokens .

The models discussed on this page all use some form of sub-word tokenization, but the exact algorithm and vocabulary differ from model to model, as the deep dive shows.

The mechanism: a pipeline with two directions

Encoding has two steps: split the text into tokens, then convert each token to its ID using the tokenizer's vocabulary, which must be the same one the model was pretrained with . In practice the split is preceded by normalisation (clean-up such as removing needless whitespace, lower-casing, removing accents or Unicode normalisation) and pre-tokenization (splitting into word-like units) . Decoding goes the other way, from IDs back to a string, and regroups sub-word pieces into readable text .

Special tokens are vocabulary entries that do not come from the text at all. A BERT tokenizer adds [CLS] at the start and [SEP] at the end because the model was pretrained with them . Other models add none, or different ones, or only at one end; the tokenizer knows which its model expects . SentencePiece, for example, reserves IDs for <unk>, <s> (beginning of sequence), </s> (end of sequence) and <pad> . This page's toy model reserves ten, including <q>, <a>, <end> and <unk>.

The mathematics

Treat a tokenizer as a pair of functions. Let Σ be the set of all strings and V the vocabulary size.

`` encode : Σ → {0, 1, …, V − 1} text → sequence of token IDs decode : {0, 1, …, V − 1} → Σ* IDs → text `

A tokenizer is lossless when decode(encode(s)) = s for every string s. A useful summary of efficiency is the compression ratio bytes(s) / |encode(s)|, where bytes(s) is the length of s in UTF-8 and |encode(s)| the number of tokens. For OpenAI's encodings, tiktoken reports that a token corresponds to about 4 bytes on average in practice; that figure depends on the tokenizer and on the language .

Worked example: one sentence, three tokenizers

Take the 39-character sentence "Artificial intelligence is fascinating." (39 UTF-8 bytes, since it is all ASCII).

Tokenizer — Tokens — Count; Character-level — A r t i f … n g . (spaces included) — 39; Word-level (split on spaces and punctuation) — Artificial intelligence is fascinating . — 5; GPT-2 byte-level BPE — Art ificial ␣intelligence ␣is ␣fascinating . — 6

The character-level row needs only 14 distinct symbols, but the sequence is 39 long. The word-level split is the shortest, but only if every word is in the vocabulary. The toy model on this page has a word-level vocabulary built from its own small corpus: it lower-cases the text, knows "is" and ".", and maps "artificial", "intelligence" and "fascinating" all to <unk>. Three of the five inputs become the same meaningless ID.

The GPT-2 row was produced with GPT-2's real tokenizer (OpenAI's tiktoken library, encoding gpt2), and the IDs are [8001, 9542, 4430, 318, 13899, 13]. Three things are worth noticing.

• Token boundaries are not word boundaries. "Artificial" at the start of the sentence becomes two tokens, Art and ificial.
• Spaces belong to the following token: ␣intelligence (written here with ␣ for the space) is one token with ID 4430. GPT-2's tokenizer treats spaces as part of tokens, so a word is encoded differently at the start of a sentence than in the middle . The same tokenizer splits "fascinating" with no leading space into three tokens, f asc inating (IDs 69, 3372, 6010), while ␣fascinating` is the single token 13899.
• Compression: 39 bytes in 6 tokens is 6.5 bytes per token.

Try it

This chapter has no lab; the next one runs GPT-2's tokenizer in your browser. Before opening it, predict how many tokens the sentence becomes in lower case, then check.

Where you meet tokens

Context windows are sized in token positions: GPT-2's maximum sequence length defaults to 1,024 . Byte pair encoding is used by many transformer models, including GPT, GPT-2, RoBERTa, BART and DeBERTa , and Hugging Face lists Llama, Gemma and Qwen2 among its users, Unigram for T5, BigBird and Pegasus, and WordPiece for BERT-family models . Because each model has its own vocabulary, a tokenizer must be loaded by model name so the text is split and mapped exactly as it was in pretraining .

Common misconceptions

• "A token is a word." Sometimes it is; often it is not. The worked example shows a word split into two tokens and spaces glued to the following word. Numbers, code and non-English text split very differently (next chapter).
• "The model reads the text." It reads integers. The ID 4430 for ␣intelligence is an index chosen when the tokenizer was built, and neighbouring IDs are not similar tokens. An ID only selects a row of a table (chapter 16).
• "Special tokens are just text." They are reserved vocabulary entries the model was trained to treat specially . Whether a user typing <|endoftext|> produces that token or ordinary characters is decided by the application's tokenization code.

Byte pair encoding

Sub-word units entered neural machine translation with Sennrich et al.: translation is an open-vocabulary problem but the network has a fixed vocabulary, so rare and unknown words are encoded as sequences of sub-word units . Their method, byte pair encoding (BPE), comes from a data-compression technique that repeatedly replaces the most frequent pair of bytes with a new, unused byte; they adapted it to merge characters or character sequences . The final vocabulary size is the initial size plus the number of merges, and the number of merges is the algorithm's only hyperparameter .

Training starts from the characters that occur in the corpus and repeatedly merges the most frequent adjacent pair, adding each merged token to the vocabulary, until the target size is reached . In Hugging Face's worked example, the word counts are hug 10, pug 5, pun 12, bun 4 and hugs 5. The pair (u, g) occurs 20 times and is merged first into "ug"; next (u, n), 16 times, becomes "un"; then (h, ug) becomes "hug" . To tokenize new text, BPE normalises, pre-tokenizes, splits words into characters and applies the learned merges in the order they were learned. A character outside the base vocabulary becomes the unknown token, so "mug" gives [UNK], ug . Byte-level BPE (next chapter) removes that failure .

WordPiece

WordPiece, developed by Google to pretrain BERT, also learns merges, but it chooses the pair with the best score freq(pair) / (freq(first) × freq(second)), which favours pairs whose parts are individually rare . Apply it to the same word counts. The symbol u occurs 36 times (10 + 5 + 12 + 4 + 5), g 20 times (10 + 5 + 5) and s 5 times. The pair (u, g) scores 20 / (36 × 20) = 1/36, but (g, s), which occurs only 5 times, scores 5 / (20 × 5) = 1/20, which is higher. BPE merges the most frequent pair; WordPiece here merges a rarer pair first. BERT's tokenizer marks continuation pieces with ##, so "Transformer" becomes transform and ##er in bert-base-cased . Google never open-sourced WordPiece's training algorithm, so the published descriptions are a best guess from the literature .

Unigram and SentencePiece

Unigram works in the opposite direction: it starts from a large vocabulary and removes tokens until it reaches the target size . Each step removes the p per cent (usually 10 or 20) of symbols whose removal increases the corpus loss least, and base characters are never removed, so any word can still be tokenized . The model assumes each sub-word occurs independently, so a segmentation's probability is the product of its pieces' probabilities; the best segmentation is found with the Viterbi algorithm and the probabilities are estimated with EM . Segmentation is ambiguous even with a fixed vocabulary, and Kudo sampled several segmentations during training (subword regularisation) to make translation models more robust . Unlike BPE's deterministic merge rules, Unigram is probabilistic: it picks the most probable segmentation at inference and can sample alternatives in training .

SentencePiece is a library rather than an algorithm. It is a language-independent tokenizer and detokenizer that trains directly on raw sentences without assuming the text is already split into words , and it implements both BPE and the unigram model . It escapes whitespace as the meta symbol ▁ (U+2581), so spaces survive and detokenization is lossless . It is configured by final vocabulary size rather than number of merges .

Trade-offs at a glance

What training learns — merge rules and a vocabulary — a vocabulary — a vocabulary with a score per token; Direction — grow from characters or bytes — grow from characters — prune from a large set; Segmentation — deterministic, merges in order — deterministic — most probable, or sampled in training; Example users (Hugging Face) — GPT-2, Llama, Gemma, Qwen2 — BERT, DistilBERT, Electra — T5, BigBird, Pegasus

The engineering trade-off is the same in every case: a larger vocabulary gives shorter sequences but a larger embedding table and fewer training examples per entry; a smaller one gives longer sequences, which cost more computation per request.

15. Tokenization step by step

This chapter follows one tokenizer, GPT-2's, through every step on real text. It is a good one to study: its vocabulary and merge list are public, and it shows the byte-level design that RoBERTa shares .

The problem byte-level BPE solves

A character-level base vocabulary that included every Unicode character would be enormous. Byte-level BPE instead uses the 256 possible byte values as its base vocabulary, so every string can be tokenized without an unknown token . GPT-2 and RoBERTa see text as UTF-8 bytes rather than Unicode characters: the base vocabulary is small, yet any character is covered . GPT-2's vocabulary is 50,257 tokens: 256 byte tokens, 50,000 merges and one special end-of-text token . For comparison, the original GPT used character-level BPE with 40,478 entries, 478 base tokens plus 40,000 merges .

The mechanism, in four steps

• Pre-tokenization. A pattern splits the text on whitespace and punctuation but keeps each space attached to the following piece; GPT-2's vocabulary displays that leading space as Ġ, which is how the original spaces are recovered when decoding .
• Bytes. Each piece is converted to its UTF-8 bytes. GPT-2 maps each byte to a printable symbol, which is why a space appears as Ġ.
• Merges. Within each piece, the learned merge rules are applied in the order they were learned : at every step, the adjacent pair whose merge is earliest in the list is joined, until no listed pair remains.
• IDs. Each resulting token is looked up in the vocabulary to get its ID.

Decoding concatenates the tokens' bytes and decodes them as UTF-8. BPE is reversible and lossless , and it works on any text, including text absent from the tokenizer's training data .

The mathematics of the merge loop

Let a piece be the symbol sequence s = (s_1, …, s_n), initially its bytes, and let rank(a, b) be the position of the merge a + b in the learned list (undefined if the pair was never learned). One step is:

`` (a, b) = argmin over adjacent pairs (s_j, s_j+1) of rank(s_j, s_j+1) replace every adjacent occurrence of (a, b) in s by the single symbol ab `

The loop stops when no adjacent pair has a rank. Tokenization is deterministic: the same input always gives the same tokens.

Worked example: "Why is the sky blue?"

The 20-character input (20 bytes) splits into six pieces: Why, ␣is, ␣the, ␣sky, ␣blue and ?. Take ␣blue. It starts as five byte symbols, ␣ b l u e, and four merges apply:

Step — Merge — Symbols afterwards; start — — ␣ b l u e; 1 — ␣ + b — ␣b l u e; 2 — u + e — ␣b l ue; 3 — ␣b + l — ␣bl ue; 4 — ␣bl + ue — ␣blue

Note that step 2 happens to the right of step 1's result: the order is set by the merge list, not by position in the word. Every piece of this sentence ends as a single token, and the result, checked against OpenAI's tiktoken, is:

` "Why is the sky blue?" → [5195, 318, 262, 6766, 4171, 30] characters 20, bytes 20, tokens 6, decode(IDs) = input `

Space handling matters. GPT-2 encodes "Hello world" as [15496, 995] but " Hello world", with a leading space, as [18435, 995] . In the same way, "blue" at the start of a text is ID 17585 while ␣blue is 4171, and ␣Blue is 4518. To the model these are three unrelated IDs.

Try it

The lab runs GPT-2's actual tokenizer in your browser: its published vocabulary and merge list, re-implemented for this page and tested against tiktoken. Type any text, or pick one of the examples (English, a contraction with a curly apostrophe, a long word, numerals, Japanese, emoji and accented words, Python source). Select a pre-tokenized piece and press Apply next merge to step through its merges one at a time; each step shows the pair and its rank in the merge list, with lower ranks applied first. The panel reports characters, UTF-8 bytes, tokens, characters per token, and whether decoding the IDs gives back exactly your input. The vocabulary file (about 1.5 MB) downloads the first time the lab opens.

What the examples show

These results were computed with tiktoken's gpt2 encoding and match the lab.

Input — Tokens — What to notice; The year 2026: 1,234,567 tokens! — 13 — ␣20 26, 234, 5 67: numerals are cut into irregular chunks; 日本語のテキスト (8 characters, 24 bytes) — 10 — some tokens are fragments of one character's bytes; 0.8 characters per token; emoji 🙂 and naïve café — 6 — ␣🙂, ␣naïve and ␣café are each one token; "emoji" is two; def f(x): newline, four spaces, return x**2 — 13 — newline is ID 198; the four spaces become three ␣ tokens (220) and ␣return; Tokenization isn’t word splitting. — 9 — Token ization; the curly apostrophe’s three bytes split across two tokens

Many NLP systems have handled emoji badly because a character missing from the training data became the unknown token; byte-level BPE cannot fail that way . The price is visible in the Japanese row: a single character can be split across tokens, and text in some languages costs more tokens per character than English.

Different tokenizers, different answers

The same text segments differently under different tokenizers. BERT splits on whitespace and punctuation and drops a double space; GPT-2 keeps it; T5's SentencePiece splits only on whitespace and adds a leading ▁ . Because BERT's tokenizer removes repeated spaces it is not reversible, while SentencePiece decoding simply concatenates tokens and turns ▁ back into spaces . The same word can split differently depending on the vocabulary: "annoyingly" might become annoying ly or annoy ing ly . GPT-2 gives a third answer: ␣annoy ingly`, IDs 10072 and 4420.

Common misconceptions

• "The tokenizer is part of the neural network." It is a separate, fixed preprocessing step, built before the model is trained and never updated by it. It must match the model exactly .
• "Sub-words are linguistic morphemes." Sometimes they line up (Token ization); often they do not (Art ificial, f asc inating). Merges are learned from frequency, not grammar.
• "BPE always splits a word the same way." The tiktoken README notes that BPE often splits "encoding" into "encod" + "ing", so the model sees the common sub-word "ing" in many contexts . That illustrates the idea but is not universal: GPT-2 encodes ␣encoding as the single token 21004, and "encoding" with no leading space as enc oding. The answer depends on the vocabulary and on the surrounding space.
• "Segmentation is unique." With a fixed vocabulary, many sequences of sub-words spell the same string; Kudo shows several for "Hello World" . BPE picks one by its merge order; Unigram picks the most probable.
• "Four bytes per token." That is tiktoken's practical average for OpenAI's encodings . The examples above range from 6.5 bytes per token for common English to 2.4 for the Japanese line (24 bytes in 10 tokens).

Pre-tokenization is where much of the behaviour lives

The merge loop only works inside one pre-tokenized piece, so no token in GPT-2 ever spans two pieces. That is why ␣is and ␣the can never merge into one token even if "is the" were common: the pattern cuts between them first. The pattern also decides that digits, letters and punctuation fall into separate pieces, which is part of why numerals split as they do. Tokenizers for other models use different patterns, or, like SentencePiece, none: SentencePiece treats the input as a raw stream including the space character, which suits languages such as Chinese and Japanese that do not separate words with spaces .

GPT-2's special tokens and defaults

In the Hugging Face Transformers GPT2Tokenizer, which is based on byte-level BPE , the unknown, beginning and end tokens all default to the same end-of-text token, and no padding token is set . With byte-level coverage an unknown token never arises from text. The model side matches: GPT2Config defaults to vocab_size 50,257, a maximum of 1,024 positions, an embedding width of 768 and 12 layers .

Character-level BPE tokenizers are not as fragile as the word-level ones in the previous chapter: their base vocabulary covers every character seen in training, so unknown tokens are rare in practice , though a character never seen in training still has nowhere to go.

Engineering notes

• Count with the right tokenizer. Context limits and output limits are in tokens. A character count, or another model's tokenizer, gives the wrong number, and the error varies by language and content.
• Keep untrusted text away from special tokens. If user input containing the literal text of a control token were encoded as that token, a user could forge structure such as end-of-text markers. Make that conversion an explicit decision in your code, never a side effect of encoding user text; this page's lab always treats such strings as ordinary bytes.
• Speed. Tokenization runs on every request and every training document, so implementations are optimised; OpenAI says tiktoken is 3 to 6 times faster than a comparable open-source tokenizer on 1 GB of text with the GPT-2 tokenizer .
• Production tokenizers differ. OpenAI's README example loads the o200k_base encoding used for gpt-4o ; it is a different vocabulary from GPT-2's, so the IDs on this page apply only to GPT-2.

16. Token embeddings

A token ID such as 4171 (␣blue in GPT-2) is just a label. The first thing a transformer does with it is look up a vector: a list of a few hundred or a few thousand real numbers that the network can compute with. This chapter is about that lookup and about what the vectors come to contain.

The problem: integers carry no similarity

Many earlier NLP systems treated words as atomic units, indices in a vocabulary with no notion of similarity between them . If "blue" is 4171 and "red" is some other integer, nothing in the numbers says they are both colours. Feeding the raw integer to a network would be worse than useless, because it would invent an order (4171 is "between" its neighbours) that means nothing.

The mechanism: a lookup table

An embedding layer is a simple lookup table of fixed size: give it token indices and it returns the corresponding vectors . In PyTorch's nn.Embedding, the two required parameters are the vocabulary size num_embeddings and the vector size embedding_dim , and the learnable weight has shape (num_embeddings, embedding_dim), initialised from a standard normal distribution . An input of any shape of integer IDs produces an output with one extra dimension of size embedding_dim .

`` token ID 42 → row 42 of E → [0.12, −0.38, 0.71, …] (illustrative values) `

The vectors start random and are adjusted by gradient descent along with every other weight, so after training, tokens that play similar roles tend to have similar vectors. Nobody assigns meanings to the dimensions; the information is distributed across all of them.

The mathematics

Let V be the vocabulary size, d the embedding width, and E the V × d embedding matrix. Writing token t as a one-hot row vector x_t (length V, a 1 in position t, zeros elsewhere), the embedding is

` e_t = x_t · E = row t of E (a vector of length d) `

The multiplication is never actually performed; indexing gives the same result. Two vectors u and v are compared with the dot product u · v = Σ_i u_i v_i, or with cosine similarity, which divides out their lengths:

` cos(u, v) = (u · v) / max(‖u‖ · ‖v‖, ε) ‖u‖ = sqrt(Σ_i u_i²) `

This is how PyTorch's nn.CosineSimilarity defines it , with ε = 1e-8 by default to avoid division by zero . Cosine ranges from −1 (opposite directions) through 0 (orthogonal) to 1 (same direction) and ignores length; the dot product does not.

Sizes grow quickly. GPT-2's defaults of a 50,257-token vocabulary and width 768 give a 50,257 × 768 token-embedding table , which is 38,597,376 numbers. This page's toy model, with a 233-entry word-level vocabulary and width 48, has a table of 233 × 48 = 11,184.

Worked example

Take three invented 3-dimensional vectors (they are illustrative, not from any model):

` a = [0.12, −0.38, 0.71] b = [0.10, −0.30, 0.80] c = [−0.60, 0.50, 0.10] `

• a · b = 0.012 + 0.114 + 0.568 = 0.694. With ‖a‖ = sqrt(0.6629) ≈ 0.81419 and ‖b‖ = sqrt(0.74) ≈ 0.86023, cos(a, b) = 0.694 / 0.70039 ≈ 0.9909: nearly the same direction.
• a · c = −0.072 − 0.190 + 0.071 = −0.191. With ‖c‖ = sqrt(0.62) ≈ 0.78740, cos(a, c) = −0.191 / 0.64109 ≈ −0.2979: pointing somewhat apart.
• Doubling b doubles a · b to 1.388 but leaves the cosine at 0.9909. That is the practical difference: cosine compares direction only.

Try it

The lab loads real pretrained vectors: GloVe 6B, 50 dimensions, for a 195-word subset grouped into themes such as royalty and people, animals, countries, capitals, colours and numbers. Choose a word to see its eight nearest neighbours by cosine similarity in all 50 dimensions, and its vector drawn as a strip of values. The map projects the subset onto its first two principal components; the caption states how much of the variance those two keep, which for this subset is only about 23%. Toggle the groups shown, click a point to select it, and try the vector-arithmetic presets.

A few results to look for. The nearest neighbours of "king" within the subset are "prince" (cosine 0.824) and "queen" (0.784). The nearest neighbour of "blue" is "red" (0.890): words that appear in similar contexts are close even when their meanings are opposite or merely parallel. king − man + woman lands nearest to "queen" (0.861), and paris − france + japan nearest to "tokyo" (0.919), searching only within the 195 words and excluding the three inputs.

Where embeddings appear

Transformer language models such as GPT-2 start with a token-embedding table, and some reuse it at the output: the original Transformer shares one weight matrix between its embedding layers and the pre-softmax linear layer, and scales embeddings by sqrt(d_model)` ; GPT-2's Transformers configuration ties word embeddings by default . This page's toy model ties them too. Static word embeddings such as GloVe also remain in use on their own, for low-resource settings, efficient models and interpretability work .

Common misconceptions

• "Each dimension means something." The training objective does not ask for interpretable axes, and any rotation of the whole space would serve the network equally well if the next layer were rotated to match. Meaning is in directions and distances, spread across dimensions.
• "The 2-D map shows the space." It shows a shadow of it. On the lab's map the first two principal components keep about 23% of the subset's variance, so two words can look close on the map yet be far apart in 50 dimensions, or the reverse. Trust the cosine list over the picture.
• "Close vectors mean synonyms." Close vectors mean similar contexts in the training text. "blue" sits next to "red", "black" and "green". Words can also be similar in several ways at once, for example sharing a word ending .
• "Analogies prove the vectors understand relations." Vector arithmetic works for some relations and fails for others, and the answer is found by a nearest-neighbour search that excludes the input words . It reveals regularities in the text, including its biases.
• "Embeddings are a dictionary of facts." A token's embedding is one row of learned weights. It is not a stored definition, and the model's behaviour depends on everything that later layers do with it.

Pretrained static vectors versus model embeddings

word2vec (Mikolov et al., 2013) proposed two architectures for computing continuous word vectors from very large data sets and reported learning high-quality vectors from 1.6 billion words in less than a day . CBOW predicts a word from the words around it, averaging their vectors so word order does not matter ; skip-gram does the reverse, predicting the surrounding words from the current one . Their vectors supported analogy arithmetic: vector("King") − vector("Man") + vector("Woman") lands closest to vector("Queen") , and Paris − France + Italy gives Rome, with answers found by cosine distance .

GloVe builds a global word co-occurrence matrix from local context windows and trains on it; the resulting weights are the embeddings . The glove.6B download was trained on Wikipedia 2014 plus Gigaword 5, 6 billion tokens with a 400,000-word uncased vocabulary . The lab uses that release's 50-dimensional vectors; note that the GloVe README describes glove.6B only as 300-dimensional, and the listing of the smaller variants could not be checked against the project page . The vectors are released under the Open Data Commons Public Domain Dedication and License and the code under Apache 2.0 . Larger downloads exist, trained on Common Crawl and Twitter , and Stanford released new vectors in 2024 in 50, 100, 200 and 300 dimensions ; Stanford notes the 2014 models were not carefully documented as to data versions and preprocessing .

These are static embeddings: each word gets one context-independent vector, so a word with several senses gets a single representation . ELMo made embeddings contextual, computing each token's representation as a function of the whole sentence with a bidirectional LSTM language model , so that it could model polysemy ; its higher layers captured context-dependent meaning and lower layers syntax .

A transformer's token embeddings sit between the two. The table itself is static, one row per token ID, but it is learned together with the rest of the model, and from the first layer onward attention mixes each vector with its context. Three kinds of vectors appear on this page, and it is worth keeping them apart:

Unit — lower-cased word — byte-level BPE token — word from its own corpus; How learned — co-occurrence statistics, separately — with the whole language model — with the whole toy model; Width — 50 here — 768 by default — 48; Context-dependent? — no — not the table; yes after attention — not the table; yes after attention

Engineering details

• Padding. If padding_idx is set, that row starts as zeros and receives no gradient, so padding positions contribute nothing .
• Loading pretrained vectors. Embedding.from_pretrained builds a layer from a 2-D tensor and by default freezes it, so the vectors are not updated in training . Freezing is a choice: fine-tuning them adapts them to the task at the cost of drifting from the original space.
• Memory. The table is V × d numbers, and tying it to the output layer avoids storing a second matrix of the same size. For GPT-2 that is about 38.6 million numbers.
• Scale of vectors. Because the dot product grows with vector length, models and retrieval systems that compare vectors often normalise them first, which makes the dot product equal to the cosine.

17. Positional information

"Dog bites man." and "Man bites dog." contain the same words and say opposite things. A model of language has to know which token came first. Recurrent networks get order for free because they read one token at a time. A transformer does not: its core operation, attention, treats its inputs as a set.

The problem

The Transformer contains no recurrence and no convolution, so to use word order it must be given information about the positions of the tokens explicitly . As Shaw et al. put it, unlike RNNs and CNNs it does not model position in its structure and needs position representations added to its inputs .

To see why, run the two sentences through GPT-2's tokenizer with a leading space, so every word carries its space: " dog bites man." is [3290, 26081, 582, 13] and " man bites dog." is [582, 26081, 3290, 13]. Same four IDs, different order. Attention computes, for each position, a weighted average of value vectors with weights from query–key scores. If you permute the inputs, the outputs are permuted in the same way and nothing else changes. Without position, the token ␣dog would produce exactly the same vector in both sentences. (Without the leading space the tokenizer itself would differ, Dog versus ␣dog, but that is an accident of spacing, not a way of representing order.)

The mechanism: four families

• Learned absolute embeddings. A second lookup table with one row per position, added to the token embedding. GPT-2 does this , with a table of n_positions rows, 1,024 by default, which is the longest sequence the model can ever take . This page's toy model does the same with 56 positions.
• Fixed sinusoidal encodings. The original Transformer adds a fixed pattern of sines and cosines to the input embeddings at the bottom of the encoder and decoder stacks; the encodings have the same width d_model so the two can be summed . Encodings can be learned or fixed , and learned embeddings gave nearly identical results in that paper .
• Relative positions. Instead of labelling positions, self-attention is extended to consider the distance between elements .
• Rotary position embeddings (RoPE). Queries and keys are rotated by an angle proportional to position, so absolute position is encoded with a rotation while the attention score depends on relative position . LLaMA, for example, removed absolute position embeddings and applies RoPE at each layer .

The mathematics

Sinusoidal. For position pos and dimension index i,

`` PE(pos, 2i) = sin(pos / 10000^(2i / d_model)) PE(pos, 2i+1) = cos(pos / 10000^(2i / d_model)) `

so each pair of dimensions is a sinusoid , with wavelengths forming a geometric progression from 2π to 10000·2π . The input to the first layer is x_pos = e_token + PE(pos). The authors chose sinusoids because for any fixed offset k, PE(pos + k) is a linear function of PE(pos), which they hypothesised would make attending by relative position easy .

RoPE. Split a query q and key k into pairs of dimensions. For a pair at frequency θ, rotate the query at position m by angle mθ and the key at position n by nθ, where R(φ) is the 2 × 2 rotation matrix [[cos φ, −sin φ], [sin φ, cos φ]]. Because rotations compose, R(a)ᵀ R(b) = R(b − a), so

` (R(mθ) q) · (R(nθ) k) = q · R((n − m)θ) k `

The score depends on the positions only through the offset n − m . RoPE uses a different θ_i for each pair; the paper's Eq. 15 gives θ_i = 10000^(−2(i−1)/d) for i = 1 … d/2, while its Section 3.3 writes 10000^(−2i/d), an indexing difference within the same paper .

Worked example

Sinusoids with d_model = 4. The two frequencies are 1 and 1/10000^(2/4) = 1/100, so PE(pos) = [sin pos, cos pos, sin(pos/100), cos(pos/100)]:

` PE(0) = [0.0000, 1.0000, 0.0000, 1.0000] PE(1) = [0.8415, 0.5403, 0.0100, 1.0000] PE(2) = [0.9093, −0.4161, 0.0200, 0.9998] `

The first pair turns quickly, distinguishing neighbouring positions; the second barely moves, distinguishing positions hundreds apart.

RoPE on one pair. Take q = [1, 0.4], k = [0.7, −0.8] and θ = 0.35. Unrotated, q · k = 0.7 − 0.32 = 0.38, which is also what you get for any m = n. With the query at m = 2 and the key at n = 5, the offset is 3, so the score equals q · R(1.05) k. With cos 1.05 ≈ 0.497571 and sin 1.05 ≈ 0.867423, R(1.05) k ≈ [0.7 × 0.497571 + 0.8 × 0.867423, 0.7 × 0.867423 − 0.8 × 0.497571] = [1.042238, 0.209139], and the score is 1.042238 + 0.4 × 0.209139 ≈ 1.125894. Moving both to m = 5, n = 8, or m = 0, n = 3, gives the same 1.125894.

Try it

The lab has three views. Sinusoidal draws PE as a heat map for positions 0–39 and 32 dimensions (real models use hundreds or thousands), shows PE(pos, 0) and PE(pos, 1) for the selected position, and plots that position's similarity (dot product divided by d`) with every other position: highest at itself and falling with distance. Same word, two positions adds 50-dimensional sinusoidal encodings to the real GloVe vector for "blue" at positions A and B and reports their cosine: at the defaults, positions 2 and 5, it is 0.8634 rather than 1. Rotary (RoPE) draws the worked example's query and key on one 2-D pair; change the query position with the offset fixed and the score does not move.

In real systems

Different models make different choices. GPT-2 uses learned absolute positions, which is why Hugging Face advises padding its inputs on the right . LLaMA uses RoPE . ALiBi adds nothing to the embeddings at all: it penalises each query–key score in proportion to the distance between the two tokens . Position also affects what an inference server caches: the same token's cached keys and values differ depending on its position and everything before it .

Common misconceptions

• "Attention knows word order." Attention on its own is indifferent to order; position has to be supplied . A decoder's causal mask (chapter 22) does break the symmetry, since each position sees a different prefix, but every model discussed here still adds explicit position information.
• "Positional encoding is part of the definition of a transformer." It is a design choice that varies: learned, fixed, relative, rotary or a bias on attention scores. The original paper itself found learned and sinusoidal versions nearly identical .
• "Sinusoids let a model run on longer inputs than it was trained on." The authors preferred them because they may allow extrapolation; that was a hypothesis, not a result shown in the paper .
• "RoPE is added to the embeddings." It is applied to queries and keys inside attention, and it is multiplicative rather than additive . The token's input vector carries no position at all.
• "Position is only in the first layer." For additive schemes the information enters at the bottom and must be carried up through the residual stream. For RoPE and ALiBi it is reapplied in every attention layer.

Relative positions

Shaw et al. extend self-attention with learned representations of the distance between two positions . Distances are clipped to a maximum k, giving 2k + 1 labels, on the hypothesis that precise relative position stops being useful beyond some distance; clipping also lets the model generalise to sequence lengths not seen in training . On WMT 2014 translation they report gains of 1.3 BLEU (English–German) and 0.3 BLEU (English–French) over absolute positions, and no further gain from combining the two . Relative representations are invariant to total sequence length, which absolute ones are not .

RoPE in more detail

RoPE encodes absolute position by rotation but makes the attention score depend explicitly on relative position . Its authors claim flexibility of sequence length, inter-token dependency that decays with distance, and compatibility with linear self-attention ; the transformer enhanced with it is called RoFormer . With many pairs at geometrically spaced frequencies, low-index pairs rotate fast and resolve small offsets, high-index pairs rotate slowly and resolve large ones, the same idea as the sinusoidal table applied to queries and keys instead of inputs.

ALiBi

ALiBi starts from a practical question: how can a model extrapolate at inference to sequences longer than those it was trained on? Its authors show that changing the position method alone can enable this . Rather than adding anything to embeddings, it biases each attention score with a penalty proportional to distance , which gives an inductive bias towards recency . They report a 1.3-billion-parameter model trained on length 1,024 that extrapolates to 2,048 with the same perplexity as a sinusoidal model trained on 2,048, while training 11% faster and using 11% less memory . The exact per-head slopes could not be checked against the full paper for this page .

Comparison

Scheme — Where position enters — Learned? — Example; Learned absolute — added to input embeddings — yes — GPT-2; Sinusoidal — added to input embeddings — no — original Transformer; Relative representations — inside attention, by distance — yes — Shaw et al.; RoPE — rotation of queries and keys — no — LLaMA; ALiBi — distance penalty on scores — no — Press et al.

Engineering details

• A learned table is a hard limit. GPT-2's table has 1,024 rows : 1,024 × 768 = 786,432 numbers at the default width. Position 1,024 has no row, so the model cannot take a longer input without changing the architecture. The toy model's table is 56 × 48 = 2,688 numbers.
• Padding side. With absolute positions, left padding shifts every real token to a different position than in training, which is why right padding is advised for GPT-2 .
• Caching. Because cached keys depend on position , an identical sentence at a different offset in a prompt cannot reuse another request's cache entries.

18. Sequence modelling before transformers

Transformers did not appear from nowhere. Each earlier approach to modelling sequences solved part of the problem and exposed the next limitation, and attention itself was invented to fix a weakness of recurrent networks. This chapter follows that line from counting to recurrence to attention.

The problem

Statistical language modelling aims to learn the joint probability of word sequences . That is hard because of the curse of dimensionality: a test sentence is likely to differ from every sentence seen in training . Any model has to generalise from pieces it has seen to wholes it has not.

N-gram models and the Markov assumption

By the chain rule of probability, a sequence w_1 … w_T has probability

`` P(w_1, …, w_T) = Π_t P(w_t | w_1, …, w_(t−1)) `

which is exact but useless as it stands, since most long prefixes have never been seen. An n-gram model makes the Markov assumption that only the last n − 1 tokens matter:

` P(w_t | w_1, …, w_(t−1)) ≈ P(w_t | w_(t−n+1), …, w_(t−1)) `

and estimates each conditional by counting, P(b | a) = count(a, b) / count(a) for a bigram (n = 2). N-gram models generalise by stitching together very short overlapping sequences seen in training , and they were the popular statistical language model of the era in which words were atomic indices .

Worked example: bigrams from this page's corpus

Join the answers in the toy model's corpus end to end: 256 words, 139 of them distinct. The word "the" occurs 8 times, followed by "sky" twice, "weights" twice, and "way", "light", "gradient" and "loss" once each. So

` P(sky | the) = 2 / 8 = 0.25 P(light | blue) = 4 / 7 ≈ 0.571 ("blue" is followed by light 4, wavelengths 1, "." 1, "," 1) P(sky, . | the) = P(sky | the) · P(. | sky) = 0.25 × 1/2 = 0.125 `

The pair "blue sky" never occurs, so its estimate is exactly 0, however natural it is. Of the 139 × 139 = 19,321 possible bigrams, only 233 appear: about 1.2%. That is data sparsity in miniature, and it gets worse exponentially with n.

Neural language models

Bengio et al. (2003) learned a distributed representation for each word jointly with the probability function over sequences , so an unseen sentence gets high probability if its words are similar to words in sentences that were seen. Their neural model beat state-of-the-art n-gram models and used longer contexts . The counts are replaced by embeddings, which is the step chapter 16 described.

Recurrent networks

A recurrent network processes positions in order, each hidden state h_t being a function of the previous state h_(t−1) and the input at position t : in the simplest form h_t = tanh(W_x x_t + W_h h_(t−1) + b), where x_t is the input vector, W_x and W_h weight matrices and b a bias. The same weights are used at every step, and there is no fixed limit on how far back the state can carry information. In practice, gradients through many repeated multiplications vanish or explode, and gated units such as the LSTM's were designed to address the vanishing problem on long sequences .

• LSTM. PyTorch's nn.LSTM computes input, forget, cell and output gates i_t, f_t, g_t, o_t, then c_t = f_t ⊙ c_(t−1) + i_t ⊙ g_t and h_t = o_t ⊙ tanh(c_t) , where c_t is the cell state, σ the sigmoid that produces i, f and o, and ⊙ element-wise multiplication . The additive path through c_t lets information and gradients survive many steps.
• GRU. nn.GRU uses reset, update and new gates r_t, z_t, n_t, and h_t = (1 − z_t) ⊙ n_t + z_t ⊙ h_(t−1) . The gated unit was introduced by Cho et al., motivated by the LSTM but simpler, with two gates rather than a memory cell and four gates .

Try it

The lab is a bigram model built from the same toy-corpus answers as the worked example. The default context is "the"; the bars show P(next | last word)` for up to ten successors, with their raw counts. Sample next word draws from that distribution with a seeded random number (press New seed for a different draw), Most likely always takes the top word, and Undo removes the last word. You can start from any word in the corpus. If a word were never followed by another, the lab would report a dead end; and since unseen pairs have probability 0, the generator can only ever reproduce word pairs it has seen. Watch how quickly the text drifts: after "the", the model cannot know whether the sentence was about the sky or about gradient descent.

Where this leads

The next step was to read a whole input sequence with one recurrent network and generate an output with another, the encoder–decoder; then to let the decoder look back at every input position, which is attention. The deep dive covers both, and why the Transformer then dropped recurrence altogether.

Encoder–decoder models

Cho et al. (2014) proposed the RNN Encoder–Decoder: one RNN encodes a symbol sequence into a fixed-length vector and another decodes it into a second sequence, both trained jointly to maximise the probability of the target given the source . Sutskever et al. (2014) did the same with deep LSTMs, mapping the input to a fixed-dimensional vector and decoding the target from it . The motivation was that plain feed-forward networks need fixed-size inputs and outputs, while translation and speech recognition involve sequences of unknown length . Their system scored 34.8 BLEU on WMT'14 English-to-French, against 33.3 for a phrase-based statistical system . Decoding stopped at an end-of-sentence token and used a left-to-right beam search .

Two engineering findings from that paper show how much the architecture constrained the data. Reversing the word order of source sentences (not targets) improved the LSTM markedly by creating many short-term dependencies . And although LSTMs tend not to suffer from vanishing gradients, they can have exploding ones, so the gradient was rescaled whenever its norm exceeded 5 .

The bottleneck

Bahdanau et al. conjectured that squeezing the whole source sentence into one fixed-length vector was the bottleneck of the basic encoder–decoder : performance deteriorated rapidly as input length increased, especially beyond the lengths seen in training . However large the vector, it is the same size for a five-word sentence and a fifty-word one.

Attention, before the Transformer

Their fix was to keep all the encoder's states. A bidirectional RNN produces a sequence of vectors h_1 … h_n, and the decoder chooses among them adaptively for each output word . For output step i, with decoder state s_(i−1):

`` e_ij = a(s_(i−1), h_j) alignment score for source position j α_ij = exp(e_ij) / Σ_k exp(e_ik) softmax over source positions c_i = Σ_j α_ij h_j context vector for output word i `

so each target word gets its own context vector, a weighted sum of encoder states . The scoring function a is a small feed-forward network trained jointly with everything else, and because the alignment is soft, gradients flow through it . Their gated recurrent units also let gradients flow backwards without suffering too much from the vanishing effect , and the GRU's update gate plays a role similar to the LSTM's memory cell ; when the reset gate is near 0, the hidden state ignores the previous state and restarts from the current input .

Attention therefore existed three years before the Transformer, as an addition to recurrent networks. What the Transformer changed was to make it the only mechanism.

Why the Transformer became influential

Recurrence is sequential: h_t cannot be computed before h_(t−1), which prevents parallelising within a training example, and that matters most for long sequences, where memory limits batching across examples . The Transformer is based solely on attention, dispensing with recurrence and convolutions, and is more parallelisable and faster to train ; it reached a new state of the art in translation after as little as twelve hours on eight P100 GPUs .

Sequential operations per layer — O(n) — O(1); Path between two distant positions — up to n steps — one step; Cost per layer — O(n · d²) — O(n² · d)

Here n is the sequence length and d the representation width. Self-attention is faster when n is smaller than d, which is usually true for sentence representations . For very long sequences the n² term dominates, and later chapters on inference return to what that costs.

Common misconceptions

• "Transformers made everything before them obsolete." They displaced recurrent networks in many language tasks, but earlier ideas live on inside them (embeddings, attention, encoder–decoder structure) and alongside them: static embeddings remain useful for low-resource and efficient systems .
• "Attention was invented by the Transformer." It was introduced for recurrent translation models .
• "RNNs cannot remember anything long." Gating exists precisely to carry information and gradients over long spans . The practical limits are the fixed-size state and the sequential computation, not an absolute memory horizon.
• "An n-gram model is a weak transformer." It is a different thing: a lookup of counts with a hard context limit and no notion of similarity between words. The lab's zero probability for "blue sky" is a structural property, not a training shortfall.
• "Encoder–decoder means old." The original Transformer is itself an encoder–decoder, and not every modern transformer is decoder-only, as chapter 19 shows.

Engineering notes

• PyTorch's nn.GRU defaults to one layer, biases on, batch_first = False (tensors shaped (sequence, batch, feature)`) and no dropout . The layout default trips up code written for batch-first tensors.
• Recurrent inference keeps a fixed-size state per sequence, while a transformer's per-sequence cache of keys and values grows with every token. The trade-off is memory against direct access to every earlier position.

19. The transformer architecture

A transformer is a stack of identical layers that does two things over and over. First, every position in the sequence gathers information from other positions (attention). Then every position's vector is transformed on its own by a small feed-forward network. Everything else in this module, from queries and keys to the final softmax, is a detail of one of those two steps or of the plumbing that joins them.

The problem it was built to solve

Before 2017 the strongest sequence models were recurrent: each hidden state is computed from the previous hidden state and the current input . That chain is the bottleneck. Vaswani et al. point out that the sequential nature of recurrent models prevents parallelising within a training example, which matters more as sequences get longer . It also means information from token 1 must survive many steps to reach token 50: a recurrent layer needs O(n) sequential operations to connect two positions, while self-attention connects any two positions in a constant number .

The Transformer was proposed as a sequence model built only on attention, with no recurrence and no convolutions . The authors report that the design trains in parallel more easily and needs much less training time than the earlier translation models, while giving better quality .

The original: an encoder–decoder

The 2017 model was built for translation, so it has two halves. The encoder turns the input symbols into a sequence of continuous representations, and the decoder uses them to produce the output . The decoder works one symbol at a time and feeds the symbols it has already produced back in as extra input; it is auto-regressive .

• Encoder: a stack of N = 6 identical layers in the base model, each with multi-head self-attention followed by a position-wise feed-forward network .
• Decoder: also N = 6 layers, with a third sub-layer that attends over the encoder's output .
• One width everywhere: all sub-layers and embedding layers output vectors of size d_model = 512 in the base model, so that residual connections can add them together .

Attention appears in three places: encoder self-attention, decoder self-attention, and encoder–decoder (cross) attention, where the queries come from the decoder and the keys and values come from the encoder's output .

``text ENCODER–DECODER (2017 Transformer, T5)

source tokens target tokens so far (shifted right) │ │ embed + position embed + position │ │ ┌────▼──────────────┐ ×N ┌──────────▼───────────┐ ×N │ self-attention │ │ causal self-attention │ │ (bidirectional) │ │ cross-attention ◄────┼── keys, values from │ feed-forward │───────────────►│ feed-forward │ the encoder output └───────────────────┘ └──────────┬───────────┘ linear + softmax │ next target token `

Three families from one design

The Hugging Face course says most Transformer models use one of three layouts: encoder-only, decoder-only, or encoder–decoder .

`text ENCODER-ONLY (BERT) DECODER-ONLY (GPT family, this page's toy)

tokens tokens │ │ embed + position embed + position │ │ ┌─▼──────────────────┐ ×N ┌─▼──────────────────┐ ×N │ self-attention │ │ causal │ │ (every token sees │ │ self-attention │ │ every token) │ │ (earlier only) │ │ feed-forward │ │ feed-forward │ └─┬──────────────────┘ └─┬──────────────────┘ │ │ one vector per token linear + softmax at each position → classifier / tagger → next-token distribution `

• Encoder-only models let attention see every token of the input ("bi-directional" attention) and suit understanding tasks such as sentence classification, named entity recognition and extractive question answering . BERT is a multi-layer bidirectional Transformer encoder based on the original implementation , trained by masking some input tokens at random and predicting them from the context on both sides .
• Decoder-only models let each token attend only to the tokens before it; they are called auto-regressive and are usually pretrained to predict the next word . BERT's authors describe OpenAI GPT exactly this way: every token can only attend to earlier tokens in self-attention .
• Encoder–decoder models suit tasks that generate new text from an input, such as summarisation, translation or generative question answering; BART and T5 are listed examples . T5's encoder–decoder closely follows the original design and casts every problem as text in, text out .

The mathematics of "decoder-only"

A decoder-only language model scores a sequence of token IDs t₁ … tₙ by the chain rule of probability:

`text P(t₁, t₂, …, tₙ) = P(t₁) · P(t₂ | t₁) · P(t₃ | t₁, t₂) · … · P(tₙ | t₁ … tₙ₋₁) ``

Each factor is a distribution over the vocabulary computed at one position, from that position and the positions before it. The restriction "only earlier positions" is not a convention; it is what makes each factor a genuine prediction. Chapter 22 shows how a mask enforces it.

A small worked example: who may read whom

Take four tokens and count the (query, key) pairs attention is allowed to use in one layer.

Layout — Allowed pairs — Count for n = 4; Encoder self-attention (bidirectional) — every i with every j — 4 × 4 = 16; Decoder self-attention (causal) — j ≤ i — 4 + 3 + 2 + 1 = 10, i.e. n(n + 1)/2; Cross-attention (4 target × 4 source) — every target with every source — 16

An encoder–decoder with four source and four target tokens therefore uses 16 + 10 + 16 = 42 pairs per layer pair; a decoder-only model reading all eight tokens as one sequence uses 8 · 9 / 2 = 36. The counts are not a measure of quality; they show that the three layouts differ in which information can flow, not only in size.

Try it

The illustration below shows four-token examples of each layout. Choose an architecture, then select a token: the arrows show which positions its attention is allowed to read. For encoder–decoder, the selected target position reads every source token through cross-attention and only earlier target tokens through its own self-attention. The arrows show permissions, not learned weights.

Where you meet each one

Encoder-only models produce one contextual vector per token and are used for classification, tagging and search embeddings. Encoder–decoder models remain natural wherever an input is transformed into a different output, such as translation. The Hugging Face course states that most modern large language models use the decoder-only architecture , and the T5 paper already noted in 2019 that much later work uses a single layer stack, such as a decoder-only language model, rather than the original encoder–decoder . The toy model that runs on this page is decoder-only.

Why decoder-only dominates text generation

There is no single published reason, but the mechanics make the choice easy to understand. A decoder-only model has one stack and one objective. Next-token prediction turns every position of raw text into a training example: one forward pass scores every next-token prediction in a sequence at once . Generation then needs no separate machinery, because a decoder model generates text by predicting one token at a time , and a prompt and its answer are simply one sequence. An encoder–decoder has to decide which text is "input" and which is "output"; a decoder-only model does not. These are engineering reasons, not a proof that the layout is better for every task.

Common misconceptions

• "Transformer" means "large language model". Transformers are a family of architectures. BERT is a transformer encoder used for understanding tasks, not text generation , and the Vision Transformer feeds image patches to a standard Transformer encoder . Not every neural network is a transformer, and not every transformer is decoder-only.
• GPT-style models are the 2017 Transformer. The original was an encoder–decoder with cross-attention . Decoder-only models keep the causal self-attention and feed-forward sub-layers and drop the encoder and cross-attention.
• A decoder-only model cannot "understand" its prompt because it cannot look ahead. Every prompt token is processed; each one simply builds its representation from itself and earlier tokens. The last prompt position has seen the entire prompt, which is where the first answer token is predicted.
• The three families are the only options. The course says most models use one of the three . T5's paper also describes a prefix-causal mask, fully visible over a prefix and causal afterwards , which sits between the encoder and decoder patterns.
• Published sizes describe current commercial models. Exact layer counts, widths and parameter counts of closed production models are not stated in any primary source checked for this article .

Two sizes worth knowing

The numbers below are the ones the papers themselves report. They anchor the shapes used throughout the module.

Model — Layout — Layers — Width — Heads — Parameters; Transformer base (2017) — encoder–decoder — 6 + 6 — 512 — 8 — about 65 million; Transformer big (2017) — encoder–decoder — 6 + 6 — 1,024 — 16 — about 213 million; BERT-base — encoder-only — 12 — 768 — 12 — 110 million; BERT-large — encoder-only — 24 — 1,024 — 16 — 340 million; This page's toy — decoder-only — 2 — 48 — 4 — 70,512 (computed in chapter 25)

The base Transformer trained for 100,000 steps, about 12 hours, on 8 NVIDIA P100 GPUs; the big model trained for 300,000 steps over 3.5 days . Against modern language models this is tiny, but it set the template: identical blocks at one fixed width, stacked.

Architecture differences that matter in practice

Question — Encoder-only — Decoder-only — Encoder–decoder; Which positions can a token read? — all — itself and earlier — encoder: all; decoder: earlier target tokens plus all source tokens; Typical pretraining — masked tokens — next token — target given source; Output — a vector per token — a next-token distribution per position — a target sequence; Generates text by itself — no — yes, token by token — yes, conditioned on the encoded input

A practical consequence: an encoder can change every token's representation when a later token is added, because every token reads every other. A causal decoder never revises the representation of a token already processed, which is what allows a key-value cache during generation (chapter 35) .

Engineering details

• Cost. A self-attention layer costs O(n² · d) for sequence length n and width d, compared with O(n · d²) for a recurrent layer . For short sequences attention is cheap; for long ones the n² term dominates, which is the subject of much of the inference and serving material later in the article.
• The width is fixed for a reason. Because every sub-layer outputs d_model values, a block's output can be added to its input and passed to the next block . Chapter 24 builds the block; chapter 25 stacks them.
• Positions are an input, not a property. Attention by itself treats its inputs as a set; the Transformer has no recurrence or convolution and must be given position information explicitly . Chapter 17 covered how.
• Parallel in training, sequential in generation. The parallelism that made the Transformer fast to train applies to processing a known sequence. Producing new text is still one token at a time , which is why the inference chapters later in the article look so different from the training ones.

20. Self-attention: tokens exchanging information

Read the sentence "The animal didn't cross the road because it was tired." You know at once that "it" is the animal, not the road, because "tired" fits animals. The word "it" on its own carries almost no meaning; its meaning comes from other words in the sentence. Self-attention is the mechanism a transformer uses to let a token's vector take in information from other tokens' vectors.

The problem: one vector per token is not enough

After tokenization and embedding, each token is a vector looked up in a table (chapter 16). The vector for "it" is the same in every sentence, whether "it" refers to an animal, a road or the weather. That is the limitation of static embeddings. Contextual embeddings fix it: each token's representation becomes a function of the whole input . ELMo did this with a bidirectional LSTM ; a transformer does it with attention.

The mechanism in words

Self-attention relates the different positions of one sequence to each other in order to compute a representation of that sequence . In a self-attention layer, the queries, keys and values all come from the same place: the previous layer's output . The general attention function maps a query and a set of key–value pairs to an output, and the output is a weighted sum of the values, with weights set by how well the query matches each key . The T5 paper puts it compactly: self-attention replaces each element of a sequence by a weighted average of the rest of the sequence .

The Hugging Face course gives the informal version: the layer tells the model to pay specific attention to certain words, and more or less ignore others, when building each word's representation . The next chapter makes "how well the query matches each key" precise. For now, treat the weights as given.

The mathematics

For position i in a sequence of n tokens, with value vectors v₁ … vₙ (one per token):

``text output_i = Σ_j α_ij · v_j with α_ij ≥ 0 and Σ_j α_ij = 1 `

• α_ij is the attention weight from position i (the one being updated) to position j (the one being read).
• v_j is a vector computed from token j's current representation.
• The weights in each row are non-negative and sum to 1, so the output is a weighted average of the value vectors.

Two consequences follow directly. The output for "it" is a blend of information from many tokens, not a copy of one. And because the weights depend on the vectors, which depend on the sentence, the same token gets different outputs in different sentences.

A small worked example

Suppose position 3 reads three positions whose value vectors are

`text v₁ = [1, 0] v₂ = [0, 1] v₃ = [1, 1] `

and its attention weights are α₃₁ = 0.1, α₃₂ = 0.2, α₃₃ = 0.7 (they sum to 1.0). Then

`text output₃ = 0.1·[1, 0] + 0.2·[0, 1] + 0.7·[1, 1] = [0.1 + 0 + 0.7, 0 + 0.2 + 0.7] = [0.8, 0.9] `

The result lies inside the region spanned by the three values, pulled most strongly towards v₃. Change one weight and the output moves; the arithmetic is no more mysterious than that.

What attention could do with "it"

The weights in the table below are invented to show the shape of the idea. They are not measured from any model.

"it" reads → — The — animal — didn't — cross — the — road — because — it; weight — 0.03 — 0.46 — 0.04 — 0.06 — 0.02 — 0.21 — 0.08 — 0.10

The row sums to 1.00. A head with weights like these would pull "it" towards the representation of "animal", but would also mix in some of "road".

Notice two things. First, in a decoder-only model "it" cannot read "tired", which comes later; only positions after "it" can use "tired" (chapter 22). In an encoder, every position can attend to every position , so "it" could read "tired" directly. Second, even in this invented row, the weight is spread out. Nothing guarantees a single head lands on the grammatically correct referent.

Try it

The lab below runs this page's trained toy transformer (2 layers, 4 heads, width 48) in your browser. Choose a sequence, select a token, and pick a layer and head: the bars show the token's attention weights over itself and earlier tokens, from that one head and averaged over the four heads of the layer. The weights are real for this toy model. The animal sentence is not used because most of its words are outside the toy's small vocabulary and would become <unk>`; the presets are sentences the toy was trained on. Later positions always get exactly zero weight, because the toy model is decoder-only.

Where this shows up

Every transformer layer in every model in this module contains self-attention, whether the model is an encoder like BERT , a decoder like GPT , or the encoder and decoder halves of an encoder–decoder . What differs is which positions each token may read.

Attention is the only place tokens meet

Inside a transformer block there are two kinds of sub-layer. The feed-forward network is applied to each position separately and identically . So the only step in which information moves between positions is attention. If you want to know how "tired" could ever affect the representation used to predict what follows, the answer runs through attention weights, layer after layer.

Information can also travel indirectly. In layer 1, position 5 reads position 2. In layer 2, position 8 reads position 5, whose vector now carries some of position 2. Stacking layers lets information hop across the sequence, which is one reason depth matters (chapter 25).

Common misconceptions

• "The attention weights explain the model's answer." This is contested. Jain and Wallace found that attention weights often do not line up with gradient-based feature importance, and that very different attention patterns can give the same prediction; their experiments used BiLSTM models with attention, not full transformers . They concluded that standard attention modules should not be treated as meaningful explanations . Wiegreffe and Pinter replied that the answer depends on the definition of explanation and that testing it must consider the whole model , and they distinguish plausible explanations from faithful ones . The safe reading: attention weights show where one head copies information from; they are not, on their own, an account of why the model produced an output.
• "One head resolves the pronoun." A model has many heads in many layers, and their outputs are summed into the residual stream. A single head with high weight on "animal" is suggestive, not proof, and a head with low weight on it does not show the model failed.
• "High weight means important." A weight multiplies a value vector. A large weight on a value vector that is nearly zero contributes little; a small weight on a large value vector can contribute a lot. Weights alone omit the values.
• "Self-attention looks at words." It combines vectors at positions that correspond to tokens, and tokens are often pieces of words (chapter 15). The model never sees the string "animal", only the vectors computed from its token IDs.
• "The original paper promised interpretability." It suggested, as a side benefit, that self-attention could yield more interpretable models, and reported that individual heads appear to learn different tasks . That is an observation about some heads, not a guarantee about all of them.

Why short paths matter

A recurrent layer passes information along a chain, so two tokens n positions apart are connected through O(n) sequential steps; self-attention connects them in O(1), and shorter paths make long-range dependencies easier to learn . For the animal sentence, "it" reaches "animal" in one step, however many words are between them.

The price: a square matrix

Every query is compared with every key, so the weights form an n × n matrix per head. A self-attention layer costs O(n² · d) for sequence length n and width d , and the FlashAttention paper states that self-attention's time and memory grow quadratically with sequence length, which makes long sequences slow and memory-hungry . The Hugging Face course notes that this square matrix becomes a bottleneck for long texts, and that models such as Longformer and Reformer use sparse attention instead .

Sequence length n — Entries per head per layer (n²); 10 — 100; 1,024 — 1,048,576; 8,192 — 67,108,864

These are counts, not timings, but the quadratic shape is the point: doubling n quadruples the matrix.

Bidirectional versus causal, concretely

Model type — Can "it" read "tired"? — Where can "tired" influence the result?; Encoder (BERT-style) — yes, in the first layer — at every position, including "it"; Decoder (GPT-style) — no, it comes later — only at "tired" and positions after it

In a decoder, the information "the referent is tired" can only be combined with "it" at positions after "tired", for example at the final position that predicts the next token. Whether a given model actually does this is an empirical question that has to be measured, not assumed.

Engineering details

• Attention weights in each row are produced by softmax, so they are positive wherever a position is allowed and sum to 1 (chapter 21). In a decoder, disallowed positions get exactly zero.
• Libraries often do not materialise the weights at all; PyTorch's nn.MultiheadAttention returns them by default, and setting need_weights=False lets it use the faster fused path . Visualisations like the lab above need the weights explicitly, so they take the slow path on purpose.

21. Queries, keys and values

Attention needs an answer to one question for every pair of positions: how much should position i take from position j? The transformer answers it with three learned views of each token. A query says what a position is looking for. A key says what a position offers to be found by. A value is what a position actually hands over when it is read. Matching queries against keys gives the weights; the weights then average the values.

The problem: one vector cannot play three roles

If attention compared raw token vectors with each other directly, a token's similarity to itself would always be large, and "what I am looking for" would be forced to equal "what I contain". Separate learned projections remove that constraint. Queries, keys and values are produced by learned linear projections of the model's d_model-sized vectors, to d_k, d_k and d_v dimensions, separately for each head .

The mechanism and the shapes

Stack the n token vectors entering the layer as the rows of a matrix X. Then

``text Q = X·W_Q K = X·W_K V = X·W_V `

Matrix — Shape — Meaning; X — n × d_model — one row per position: the layer's input; W_Q, W_K — d_model × d_k — learned weights, shared by all positions; W_V — d_model × d_v — learned weights; Q, K — n × d_k — one query and one key per position; V — n × d_v — one value per position

The paper's name for what follows is scaled dot-product attention, with queries and keys of dimension d_k and values of dimension d_v . For each query, take the dot product with every key, divide each by √d_k, and apply softmax to get the weights that average the values . Processing all queries at once in matrix form gives the paper's equation 1 :

`text Attention(Q, K, V) = softmax( Q·Kᵀ / √d_k ) · V `

where Kᵀ is K transposed (d_k × n), so Q·Kᵀ is n × n: entry (i, j) is the dot product of query i with key j. Softmax is applied to each row separately. The result is n × d_v: one new vector per position.

Seven steps

• Build Q, K, V by three matrix multiplications of X.
• Score: compute S = Q·Kᵀ, every query against every key.
• Scale: divide every score by √d_k.
• Mask where required: in a decoder, scores for later positions are set to −∞ (chapter 22).
• Softmax each row: w_ij = exp(s_ij) / Σ_k exp(s_ik), so each row is non-negative and sums to 1.
• Weight the values: output row i is Σ_j w_ij · v_j, i.e. the matrix product W·V.
• Result: a contextualised vector per position, passed on to the rest of the block.

Why divide by √d_k

If the components of a query and a key are independent with mean 0 and variance 1, their dot product has mean 0 and variance d_k; dividing by √d_k counteracts this . With d_k = 64, as in the base Transformer , unscaled scores would have a standard deviation of √64 = 8. For large d_k, large dot products push softmax into regions with extremely small gradients, which hurts learning . Plain dot-product attention is identical except for that 1/√d_k factor , and without scaling, additive attention outperformed dot-product attention for larger d_k .

A complete worked example

Hand-picked matrices with 3 tokens and d_model = d_k = d_v = 2, chosen so the arithmetic is easy to follow. Real weights are learned, not chosen.

Step 1: inputs and projections. Three tokens, A, B and C:

`text ┌ ┐ ┌ ┐ ┌ ┐ ┌ ┐ │ 1 0 │ A │ 1 0 │ │ 1 0 │ │ 1 0 │ X = │ 0 1 │ B W_Q =│ 1 1 │ W_K = │ 1 2 │ W_V = │ 0 1 │ │ 1 1 │ C └ ┘ └ ┘ └ ┘ └ ┘ `

Each row of Q is a row of X times W_Q. For C: [1, 1]·W_Q = [1·1 + 1·1, 1·0 + 1·1] = [2, 1]. Doing the same for every row:

`text Q = [[1, 0], K = [[1, 0], V = [[1, 0], [1, 1], [1, 2], [0, 1], [2, 1]] [2, 2]] [1, 1]] `

(W_V is the identity here, so V = X.)

Step 2: scores S = Q·Kᵀ. Entry (i, j) = q_i · k_j:

q_A = [1, 0] — 1 — 1 — 2; q_B = [1, 1] — 1 — 3 — 4; q_C = [2, 1] — 2 — 4 — 6

For example q_C · k_B = 2·1 + 1·2 = 4.

Step 3: scale by 1/√2 ≈ 0.70711.

A — 0.70711 — 0.70711 — 1.41421; B — 0.70711 — 2.12132 — 2.82843; C — 1.41421 — 2.82843 — 4.24264

Step 4: causal mask. Position A may read only A; B may read A and B; C may read all three. Entries above the diagonal become −∞:

A — 0.70711 — −∞ — −∞; B — 0.70711 — 2.12132 — −∞; C — 1.41421 — 2.82843 — 4.24264

Step 5: softmax each row. The exponentials are exp(0.70711) = 2.02811, exp(1.41421) = 4.11325, exp(2.12132) = 8.34214, exp(2.82843) = 16.91883, exp(4.24264) = 69.59138, and exp(−∞) = 0.

Row — Exponentials — Sum — Weights; A — 2.02811, 0, 0 — 2.02811 — 1, 0, 0; B — 2.02811, 8.34214, 0 — 10.37026 — 0.19557, 0.80443, 0; C — 4.11325, 16.91883, 69.59138 — 90.62346 — 0.04539, 0.18669, 0.76792

Check row C: 4.11325 / 90.62346 = 0.04539, and 0.04539 + 0.18669 + 0.76792 = 1.00000.

Step 6: weighted sum of values.

`text out_A = 1·[1, 0] = [1.00000, 0.00000] out_B = 0.19557·[1, 0] + 0.80443·[0, 1] = [0.19557, 0.80443] out_C = 0.04539·[1, 0] + 0.18669·[0, 1] + 0.76792·[1, 1] = [0.04539 + 0.76792, 0.18669 + 0.76792] = [0.81331, 0.95461] `

Step 7: result. A's output is its own value, because it can read nothing else. C's output is mostly its own value, with some of B's. Each output row is a new, context-dependent vector for that position.

Try it

The calculator below does exactly these seven steps on one head of the trained toy model, for a short sequence (the default is "<q> why is snow white ? <a>"). It shows Q, K and V (width 12), the masked score matrix scaled by 1/√12, the softmax weights and the outputs. Select any score, weight or output cell to see its computation from the rows of Q, K and V, then switch layer or head. A note at the bottom recomputes the whole thing with an independent implementation and reports the largest difference from the model's own weights.

Where this is used

This is the attention inside every model in the module. PyTorch exposes it directly as scaled_dot_product_attention(query, key, value, attn_mask=None, dropout_p=0.0, is_causal=False, scale=None, enable_gqa=False) ; if scale` is not given it defaults to 1/√E, where E is the query's last dimension, the head size . Its reference code computes query·keyᵀ times the scale, adds the mask bias, applies softmax over the last dimension, applies dropout and multiplies by value , the same order as the steps above.

Common misconceptions

• "Queries, keys and values are three different inputs." In self-attention all three come from the same place, the previous layer's output ; they differ only by the learned matrix that projects them. In cross-attention they really do come from two sources: queries from the decoder, keys and values from the encoder .
• "Attention is a lookup in a database." The vocabulary of keys and values suggests one, but nothing is looked up exactly. Every value contributes in proportion to its weight, and weights are never exactly zero except where a mask forces them to be. It is a soft, differentiable average.
• "The weight matrix depends on sequence length." W_Q, W_K and W_V are d_model × d_k and d_model × d_v regardless of n . Only the score matrix Q·Kᵀ grows with n, as n × n. That is why the same model can process sequences of different lengths up to its context limit.
• "Scaling is a cosmetic constant." Without it, scores grow with d_k and softmax saturates, giving tiny gradients . It costs one multiplication and matters for learning.
• "The key and the value of a token are the same thing." They are separate projections, so what a token is matched on and what it contributes can differ. In the worked example, token C's key [2, 2] and its value [1, 1] point the same way only because W_V was chosen as the identity.

Why dot products, not a small network

Additive attention scores a query against a key with a small feed-forward network instead; dot-product attention is much faster and more memory-efficient because it uses optimised matrix multiplication . The whole score matrix for a head is a single matrix product, which is the operation accelerators are built for.

Numerical stability

Implementations do not compute exp of the raw scores directly. They subtract the row's maximum first, because softmax is unchanged by adding the same constant to every entry of a row:

``text exp(s_j − m) / Σ_k exp(s_k − m) = exp(s_j) / Σ_k exp(s_k) for any constant m `

For row C above, m = 4.24264: the shifted exponentials are exp(−2.82843) = 0.05911, exp(−1.41421) = 0.24312 and exp(0) = 1, which sum to 1.30222 and give the same weights 0.04539, 0.18669, 0.76792. Without the shift, a score of 100 would overflow 32-bit floating point; with it, the largest exponential is always 1. The calculator above shows this shift in its weight explanations.

Engineering details

• Dropout in evaluation. PyTorch's scaled_dot_product_attention always applies dropout at the given dropout_p, so callers must pass 0.0 during evaluation . Modules usually pass dropout_p only when in training mode.
• One projection, many heads. Each head has its own projections , but in code the per-head matrices are usually concatenated into one wide matrix, multiplied once, and the result is split. The toy model's lab describes its Q, K and V as each head's slice of the layer's projection.
• Sharing keys and values. With enable_gqa=True`, PyTorch supports grouped-query attention, where several query heads share key/value heads; the number of query heads must be divisible by the number of key/value heads, and the docs call GQA experimental . Chapter 23 covers why models do this.
• Cost by part. Projections cost O(n · d_model · d_k) per head and grow linearly with n. Scores and the weighted sum cost O(n² · d_k) and grow quadratically, which is the term behind the long-context costs discussed in chapter 20 .
• What changes between architectures. Encoder self-attention skips step 4 except for padding; decoder self-attention always applies the causal mask; cross-attention takes Q from one sequence and K, V from another, so its score matrix is n_target × n_source rather than square.

22. Causal attention and masking

A language model learns by predicting the next token. If, while predicting token 6, it were allowed to look at token 6, the task would be trivial and the model would learn nothing useful. The causal mask is the rule that a position may read itself and earlier positions, never later ones. It is implemented with one line of arithmetic and it shapes everything about how decoder-only models are trained and run.

The problem: training on a whole sequence at once

The efficient way to train a decoder is to feed an entire sequence in and score every position's next-token prediction in one forward pass . But in that pass every token is present at once. Without a restriction, position i could read position i + 1, the very token it is meant to predict. The original paper changed the decoder's self-attention so that a position cannot attend to positions after it , to block leftward information flow (from future to past) and keep the model auto-regressive .

The mechanism

Each decoder position may attend to every decoder position up to and including itself . The mask is implemented inside scaled dot-product attention by setting the scores of illegal connections to −∞ before the softmax, so they receive zero weight . In T5's notation, the causal mask sets attention weight w(i, j) to zero whenever j > i .

For four positions the mask added to the scores is triangular:

``text key position j → 0 1 2 3 query 0 [ 0 −∞ −∞ −∞ ] i 1 [ 0 0 −∞ −∞ ] ↓ 2 [ 0 0 0 −∞ ] 3 [ 0 0 0 0 ] `

A 0 leaves a score unchanged; −∞ removes it. Combined with shifting the output embeddings right by one position, the mask makes the prediction for position i depend only on known outputs before i .

The mathematics: masked scores and row-wise softmax

With masked scores s'_ij = s_ij + M_ij, where M_ij = 0 for j ≤ i and −∞ for j > i:

`text w_ij = exp(s'_ij) / Σ_k exp(s'_ik) = exp(s_ij) / Σ_{k ≤ i} exp(s_ik) for j ≤ i, w_ij = 0 for j > i `

Because exp(−∞) = 0, the masked entries drop out of both numerator and denominator. Each row is renormalised over the positions it may see, so it still sums to 1.

A worked example: the same scores, with and without the mask

Take the scaled scores from chapter 21's example (three tokens A, B, C, d_k = 2):

A — 0.70711 — 0.70711 — 1.41421; B — 0.70711 — 2.12132 — 2.82843; C — 1.41421 — 2.82843 — 4.24264

Without the mask, every row uses all three exponentials:

Row — Exponentials — Sum — Weights — Weight on later positions; A — 2.02811, 2.02811, 4.11325 — 8.16948 — 0.24826, 0.24826, 0.50349 — 0.75174; B — 2.02811, 8.34214, 16.91883 — 27.28909 — 0.07432, 0.30570, 0.61999 — 0.61999; C — 4.11325, 16.91883, 69.59138 — 90.62346 — 0.04539, 0.18669, 0.76792 — 0

With the mask, row A becomes 1, 0, 0 and row B becomes 0.19557, 0.80443, 0, as in chapter 21. Row C is unchanged: the last position has no future to hide. Unmasked, position A puts about 75% of its attention on tokens it should not yet know, including the token B it would be asked to predict.

Of the n² entries of the score matrix, the mask keeps n(n + 1)/2: here 6 of 9, and for a 10-token sequence 55 of 100.

Try it

The lab below applies the mask, or removes it, on the trained toy model's real queries and keys for one head. With the mask the upper triangle of the weight matrix is exactly 0. Without it, the bars show how much weight each position places on the next token, the one it would be trained to predict, and a note reports the average share of attention that leaks to the future. The toy model was trained with the mask, so its outputs are only meaningful with the mask on; the unmasked view recomputes softmax on the same scores to show what would be exposed.

Where this is used

Every decoder-only language model applies a causal mask in every self-attention layer. A standard encoder–decoder uses fully visible masking in the encoder and in encoder–decoder attention, and causal masking in the decoder . In PyTorch, passing is_causal=True to scaled_dot_product_attention applies a lower-triangular mask when the mask is square , and its reference implementation builds that mask with tril` and fills blocked positions with −∞, exactly the paper's approach .

The connection to autoregressive training

Causal language modelling trains a model to predict the next token from the tokens before it; the model can only attend to tokens on its left, so it never sees future tokens . In Hugging Face Transformers the labels can simply be the input IDs, because the one-position shift happens inside the model . The target at each position is the next token; the last logit has no next token and is dropped . Without the mask, this setup would leak each label into the input of the position that predicts it.

The mask also has an inference consequence. Because of it, a processed token's representation never changes when later tokens arrive, which is why past keys and values can be cached safely . Chapters 33 and 35 build on that.

Common misconceptions

• "The mask is only needed during generation." During generation the future tokens do not exist yet, so they could not be read anyway. The mask matters most when many tokens are processed together: in training, and when a whole prompt is processed in one pass.
• "Zeroing weights after softmax is the same." It is not. Setting w_ij = 0 after the softmax leaves the row summing to less than 1 unless you renormalise. Adding −∞ before the softmax removes those positions and renormalises in one step. In the example, row A would sum to 0.24826 instead of 1.
• "The mask hides the future from the model's knowledge." It hides future positions in this sequence. Whatever the model learned in training about how sentences tend to continue is in its weights and is not masked.
• "Masked attention is approximate." The mask changes which positions are allowed; the remaining computation is exact. FlashAttention, used for exactly this computation, is an exact algorithm, not an approximation .
• "Causal" means the same in every architecture. An encoder uses a fully visible mask; T5 also describes a prefix-causal mask that is fully visible over a prefix only . Sliding-window attention is a further pattern that limits each token to at most W previous tokens .

Implementation details vary

The paper describes the mask as −∞ added before softmax ; libraries differ in how they express it.

Detail — What the PyTorch documentation says; Built-in causal flag — is_causal=True gives a lower-triangular mask for square inputs; for a non-square mask it uses an upper-left aligned causal bias; Explicit mask — attn_mask may be boolean (True means the element takes part in attention) or a float mask added to the scores; Both at once — an error is raised if both attn_mask and is_causal are set; Opposite convention — in nn.MultiheadAttention's key_padding_mask, True means "mask this out", so code moving between the two must invert the mask; Kernels — scaled_dot_product_attention has three implementations: FlashAttention-2, memory-efficient attention and a C++ math implementation, and may pick optimised kernels on CUDA

The "non-square" row matters in practice. During generation with a cache, the query block can be one new token while the keys cover the whole history, so the score matrix is 1 × n rather than n × n. Which way a non-square causal mask is aligned decides whether that new token can see the history. Read the documentation of the exact function you call.

Masks and efficient attention kernels

FlashAttention is IO-aware: it uses tiling to cut reads and writes between GPU high-bandwidth memory and on-chip SRAM , and it never writes the full N × N attention matrix to slow memory, processing K and V in blocks . A mask therefore cannot be a materialised n × n tensor inside such a kernel; it is applied per tile, which is why kernels prefer a causal flag to an arbitrary mask tensor. The authors report up to 7.6× faster attention on GPT-2 and memory linear in sequence length . LLaMA's paper mentions using an efficient causal multi-head attention implementation to reduce memory use and runtime .

Padding masks

When sequences of different lengths are batched, shorter ones are padded, and padding positions must not be attended to. Tokenizers in Transformers return an attention mask alongside the input IDs for this purpose . A decoder's effective mask is then the combination of the causal triangle and the padding mask.

An edge case: a fully masked row

If a row has every entry masked, softmax computes 0 / 0, which in floating point is NaN, and NaN then spreads through every later computation that touches it. Causal masks never do this, because the diagonal is always allowed, but combining a causal mask with a padding mask can, for example for a padding position at the start of a left-padded sequence. Implementations guard against it explicitly.

23. Multi-head attention

One attention computation produces one set of weights per position: one way of deciding what to read. A sentence offers many relations at once, such as the previous token, the subject of a verb, or a repeated name. Multi-head attention runs several attention computations side by side, each with its own projections, and combines their results.

The problem: one average blurs things together

A single head's output is a weighted average of value vectors. If one position needs information from two different places for two different reasons, a single average has to compromise between them. The paper's stated reason for multiple heads is exactly this: they let the model attend jointly to information from different representation subspaces at different positions, and with a single head, averaging inhibits this .

The mechanism

Multi-head attention runs several attention layers, called heads, in parallel, each on its own learned projection of Q, K and V . Each head produces d_v-dimensional outputs; these are concatenated and projected again :

``text head_i = Attention(X·W_i^Q, X·W_i^K, X·W_i^V) for i = 1 … h MultiHead(X) = Concat(head_1, …, head_h) · W^O `

Here h is the number of heads, W_i^Q and W_i^K are d_model × d_k, W_i^V is d_model × d_v, and the output matrix W^O is h·d_v × d_model .

The tensor dimensions

Tensor — Shape — Base Transformer — This page's toy; Input X — n × d_model — n × 512 — n × 48; Q, K, V for one head — n × d_k — n × 64 — n × 12; Scores / weights for one head — n × n — n × n — n × n; One head's output — n × d_v — n × 64 — n × 12; Concatenation of h heads — n × h·d_v — n × (8 × 64 = 512) — n × (4 × 12 = 48); After W^O — n × d_model — n × 512 — n × 48

The base model uses h = 8 heads, each with d_k = d_v = d_model / h = 64 . Because each head works in a reduced dimension, the total cost of multi-head attention is similar to single-head attention at full width . Splitting the width does not add parameters: the projections for all heads together are d_model × (h·d_k) = d_model × d_model when h·d_k = d_model.

With a batch dimension B, implementations usually hold Q as a four-dimensional tensor of shape B × h × n × d_k, so all heads of all sequences are computed by one batched matrix multiplication.

A small worked example: two heads, one position

Hand-picked weights, continuing chapter 21's example (tokens A, B, C with X = [[1, 0], [0, 1], [1, 1]]). Only the last position C is shown.

Head 1 is chapter 21's head. Its causal weights for C were 0.04539, 0.18669, 0.76792 and its output was [0.81331, 0.95461].

Head 2 uses different projections: W_2^Q = [[0, 1], [1, 0]] (swap the two components), W_2^K = identity, W_2^V = [[0, 1], [1, 0]]. Then q_C = [1, 1]·W_2^Q = [1, 1], the keys are the rows of X, and the values are X with its columns swapped: [0, 1], [1, 0], [1, 1].

`text scores: q_C·k_A = 1, q_C·k_B = 1, q_C·k_C = 2 scaled: 0.70711, 0.70711, 1.41421 exp: 2.02811, 2.02811, 4.11325 sum 8.16948 weights: 0.24826, 0.24826, 0.50349 output: 0.24826·[0,1] + 0.24826·[1,0] + 0.50349·[1,1] = [0.75174, 0.75174] `

The two heads read the same three tokens differently: head 1 puts 77% of C's attention on C, head 2 only 50%. Measured as entropy, head 1's row has 0.95 bits and head 2's 1.50 bits (a uniform row over three positions would have log₂ 3 = 1.58 bits).

Concatenate and project. C's combined vector is [0.81331, 0.95461, 0.75174, 0.75174], of length h·d_v = 2 × 2 = 4, and W^O (4 × d_model) maps it back to the model width. In this toy example h·d_v is larger than d_model = 2; real models usually choose d_k = d_model / h so the concatenation already has width d_model.

Try it

The lab below shows the four heads of one layer of the trained toy model on the same sequence (the default is a retrieval prompt). Switch between heads 1 to 4 and layers 1 and 2 to compare their weight matrices. The table lists, for every head, the mean entropy of its rows (low means concentrated on few positions), the mean distance it looks back, and the average weight a position puts on itself. No head is given a name or a job: what each one does in this toy is whatever training produced, and the lab only measures it.

Where this is used

Multi-head attention is the attention in essentially every model in this module. PyTorch's nn.MultiheadAttention describes itself as letting the model jointly attend to different representation subspaces and implements the original paper's architecture . Two practical details catch people out: it defaults to batch_first=False, so tensors are (sequence, batch, feature) unless set otherwise , and it returns attention weights by default, while need_weights=False` lets it use the faster fused path .

Common misconceptions

• "Each head has a job, such as a grammar head or a reasoning head." No head is designed for anything; every head starts from random weights and becomes whatever training makes it. Some heads in some models do show consistent, interpretable patterns: Voita et al. found that the important heads in their translation models played positional, syntactic and rare-word roles . The original paper also reported that individual heads appear to learn different tasks, many related to syntax and semantics . These are measurements of particular heads in particular models, not universal categories, and plenty of heads have no clean description.
• "More heads is always better." In the paper's head-count experiments, a single head was 0.9 BLEU worse than the best setting, and quality also fell off with too many heads .
• "Every head is necessary." Michel et al. found that a large share of heads can be removed at test time without much loss, and some layers can be cut to a single head . Voita et al. pruned 38 of 48 encoder heads for a drop of only 0.15 BLEU, and the specialised heads were the last to go .
• "Heads are independent models." They share the input X and their outputs are mixed by W^O, so information from all heads is combined into every output dimension. Studying one head alone ignores how its output is used downstream.

Why multiple heads help, more carefully

Michel et al. frame multiple heads as letting each head potentially focus on different parts of the input, which lets attention express functions beyond a simple weighted average . They also analyse which parts of a model rely more on having multiple heads, and give preliminary evidence that training dynamics play a role in the benefit . One reading is that many heads make training easier even if the trained model ends up not needing all of them, which fits the pruning results. Their paper also studies greedy pruning algorithms and the speed, memory and accuracy gains they bring .

Parameter count

Per layer, the attention sub-layer has four d_model × d_model matrices when h·d_k = d_model: the combined W^Q, W^K and W^V and the output W^O, so 4·d_model² weights plus biases if the model uses them.

Model — d_model — 4·d_model²; Transformer base — 512 — 1,048,576; This page's toy — 48 — 9,216 (plus 192 biases: 9,408)

The count does not depend on h. Changing from 8 heads of 64 to 16 heads of 32 changes how the width is split, not how many weights there are.

Sharing keys and values: MQA and GQA

During generation, every head's keys and values for earlier tokens are cached (chapter 35), so the number of key/value heads drives memory. Multi-query attention (MQA) shares one set of keys and values across all heads, which greatly shrinks those tensors and the bandwidth needed for decoding at a minor cost in quality . Grouped-query attention (GQA) uses more than one but fewer key/value heads than query heads, and gets close to multi-head quality at near-MQA speed . GQA with one group is MQA, and GQA with as many groups as heads is standard multi-head attention . Mistral 7B, for example, uses 32 query heads and 8 key/value heads . In PyTorch's scaled_dot_product_attention this is the experimental enable_gqa option .

Variant — Query heads — Key/value heads — KV cache relative to MHA; Multi-head (MHA) — h — h — 1; Grouped-query, g groups — h — g — g / h; Multi-query (MQA) — h — 1 — 1 / h

For Mistral's 32 and 8, the cache is 8 / 32 = 1/4 of what full multi-head attention with the same head size would need.

Engineering details

• The heads of one layer are computed together, not one after another: the projections for all heads are one matrix multiplication and the attention is one batched operation over the head dimension.
• The output projection W^O is where heads interact. Removing a head is equivalent to zeroing its slice of the concatenation, which is how pruning experiments are run.
• Attention statistics such as the lab's entropy table describe behaviour on particular inputs. A head that looks "positional" on one prompt can behave differently on another, so measurements need many inputs before they mean much.

24. Feed-forward networks, residuals and normalisation

Attention lets tokens exchange information. It is only half of a transformer block. The other half is a small neural network applied to every position on its own, and around both halves are two pieces of plumbing, residual connections and normalisation, without which deep stacks are hard to train.

The problem: attention alone is not enough

Attention's output is a weighted average of value vectors, which is a linear combination. Stacking only attention layers would leave the model with little capacity to transform what it gathers. A deep stack also has to be trainable: gradients must reach the first layer, and activations must not grow or shrink uncontrollably from layer to layer. The feed-forward network supplies the nonlinear transformation; residuals and normalisation keep the stack trainable.

The canonical block (2017, post-LN)

In the original Transformer, each sub-layer is wrapped in a residual connection followed by layer normalisation :

``text x₁ = LayerNorm(x + SelfAttention(x)) x₂ = LayerNorm(x₁ + FFN(x₁)) `

where Sublayer(x) is whatever the sub-layer computes, attention or the feed-forward network . The feed-forward network is applied to each position separately and identically: two linear layers with a ReLU between :

`text FFN(x) = max(0, x·W₁ + b₁)·W₂ + b₂ `

In the base model x has 512 values and the inner layer is 2,048 wide, four times d_model . Its weights are shared across positions within a layer but differ from layer to layer . Dropout of 0.1 is applied to each sub-layer's output before it is added and normalised .

The modern arrangement (pre-LN)

Later models, including T5 and LLaMA (below), move the normalisation inside the residual branch, before each sub-layer:

`text x₁ = x + SelfAttention(LN(x)) ← attention path x₂ = x₁ + FFN(LN(x₁)) ← MLP path `

`text x ─────────────────────┐ x₁ ────────────────────┐ │ │ │ │ LN → self-attention → (+)── x₁ ──► LN → FFN (MLP) ──────(+)── x₂ `

In this Pre-LN arrangement, Xiong et al. show that gradients are well-behaved at initialisation, and Pre-LN models trained without learning-rate warm-up reached comparable results with less training time . Pre-LN also adds one final layer normalisation just before the prediction layer . T5 normalises the input of each sub-block too , and LLaMA does the same for training stability, using RMSNorm .

The mathematics of the two normalisations

Layer normalisation computes the mean and variance over all the values of one position's vector, for a single example, not over a mini-batch as batch normalisation does . For a vector x with d components:

`text μ = (1/d) Σ xᵢ σ² = (1/d) Σ (xᵢ − μ)² LN(x)ᵢ = γᵢ · (xᵢ − μ) / √(σ² + ε) + βᵢ `

γ (gain) and β (bias) are learned per component ; ε is a small constant that prevents division by zero. RMSNorm drops the mean subtraction and divides by the root mean square only :

`text RMS(x) = √( (1/d) Σ xᵢ² ) RMSNorm(x)ᵢ = gᵢ · xᵢ / RMS(x) `

A worked example

Take x = [1, 2, 3, 6] (d = 4), with γ = g = 1, β = 0 and ε ignored.

`text LayerNorm: μ = (1 + 2 + 3 + 6) / 4 = 3 σ² = (4 + 1 + 0 + 9) / 4 = 3.5 σ = 1.87083 LN(x) = [−2, −1, 0, 3] / 1.87083 = [−1.0690, −0.5345, 0.0000, 1.6036]

RMSNorm: RMS = √((1 + 4 + 9 + 36) / 4) = √12.5 = 3.53553 RMSNorm(x) = [1, 2, 3, 6] / 3.53553 = [0.2828, 0.5657, 0.8485, 1.6971] ``

LayerNorm's output has mean 0 and variance 1. RMSNorm's output has root mean square 1 but keeps the vector's offset: its mean is 0.8485, not 0. And the residual: if the attention path produced Δ = [0.5, −0.5, 0, 1] for this position, the block would pass on x + Δ = [1.5, 1.5, 3, 7], an adjustment to x, not a replacement of it.

Try it

The lab below follows one position of the trained toy model through a pre-LN block: LN₁, attention (4 heads of 12, concatenated and projected), the first residual add, LN₂, the expansion W₁ from 48 to 192 values, GELU, the projection W₂ back to 48, and the second residual add. Each row shows the tensor's shape and the vector's length (its norm) for the selected position, so you can see how much each step changes the residual stream. Choose block 1 or 2 and any position.

Where this is used

Every block of every model in this module has this shape: an attention path and an MLP path, each added back to the residual stream. The variations are in the details: where the normalisation sits, LayerNorm or RMSNorm, and which activation or gate the MLP uses. The toy model follows the pre-LN pattern with LayerNorm and a GELU feed-forward network of width 192.

Why skip connections and normalisation help optimisation

A residual block computes y = x + f(x). Its derivative with respect to x is I + ∂f/∂x: the identity plus a correction. However small ∂f/∂x becomes, the identity term carries the gradient straight through, so in a deep stack the gradient reaching layer 1 does not have to pass through every layer's transformation multiplicatively. The same identity term means a freshly initialised block, whose f(x) is small, starts close to doing nothing, which is a stable place to begin.

Normalisation controls scale. Each sub-layer sees an input with a predictable mean and spread, whatever the layers before it did. The placement matters: Xiong et al. show that the Post-LN Transformer has large expected gradients near the output layer at initialisation, which makes training with a large learning rate unstable , which is why Post-LN models usually need a warm-up stage that slows optimisation and adds tuning work . Pre-LN keeps an unnormalised identity path from input to output, and its gradients are well-behaved at initialisation .

Common misconceptions

• "LayerNorm works like batch norm." It normalises over the features of one position of one example . It does exactly the same computation at training and test time and works even with a batch of one .
• "The residual stream is overwritten by each layer." Each sub-layer adds to it. The block's output is the input plus two corrections, as the lab's norms show.
• "The FFN mixes tokens." It is applied to each position separately ; it is equivalent to two convolutions with kernel size 1 . Only attention moves information between positions.
• "Pre-LN and post-LN are the same model with a different diagram." They compute different functions and train differently . A checkpoint trained one way cannot be loaded into the other arrangement.

RMSNorm

RMSNorm's authors hypothesise that LayerNorm's re-centring is not needed . Skipping the mean makes it computationally simpler, and they report comparable quality with 7% to 64% less running time across their models . T5 uses a similar simplified layer norm that only rescales, with no additive bias ; LLaMA uses RMSNorm before each sub-layer .

Gated MLP variants

A Gated Linear Unit multiplies two linear projections element by element, one of them first passed through a sigmoid; variants swap the sigmoid for other functions: ReGLU, GEGLU, and SwiGLU, which uses Swish . Shazeer uses them in place of the first linear layer and activation of the FFN, for example

``text FFN_SwiGLU(x) = (Swish(x·W) ⊗ x·V)·W₂ `

where ⊗ is element-wise multiplication, and finds that some variants improve quality over ReLU or GELU . Because these FFNs have three weight matrices instead of two, the hidden size is reduced by a factor of 2/3 to keep parameters and compute equal . LLaMA replaces ReLU with SwiGLU and uses a hidden size of (2/3)·4d instead of 4d . Shazeer offers no explanation for why the variants work, attributing their success, tongue in cheek, to "divine benevolence" ; the improvement is empirical.

The parameter arithmetic checks out. A standard FFN with hidden size 4d has two matrices of d × 4d: 8d² weights. A gated FFN with hidden size (2/3)·4d = (8/3)d has three matrices of d × (8/3)d: 3 × (8/3)d² = 8d².

Architecture differences

Model — Norm placement — Norm — FFN activation — Positions; Transformer (2017) — after each residual add — LayerNorm — ReLU — added to the input embeddings; T5 — before each sub-block — scale-only layer norm — (not covered here) — (not covered here); LLaMA — before each sub-layer — RMSNorm — SwiGLU — RoPE at each layer; This page's toy — before each sub-layer — LayerNorm — GELU (tanh approximation) — learned, added to embeddings

Engineering details

• GELU. GELU(x) = x·Φ(x), where Φ is the standard Gaussian cumulative distribution function . PyTorch defaults to the exact form; approximate='tanh'` uses 0.5·x·(1 + tanh(√(2/π)·(x + 0.044715·x³))) , which is the form the toy model uses.
• Where the compute goes. For a d_ff = 4d MLP, the two matrices have 8d² weights against 4d² for attention's projections, so in a standard block about two thirds of the weights are in the MLP. In the toy: 18,432 of 27,648 matrix weights per block.
• Memory-bound steps. Normalisation, activation functions, dropout and residual additions are element-wise or reductions; such operations are limited by memory access rather than arithmetic . Kernels often fuse them into the neighbouring matrix multiplication for that reason.
• One extra LayerNorm. In a pre-LN model, the last block's output is unnormalised, so the final LayerNorm before the output layer is required, not optional. Chapter 26 starts there.

25. Stacking transformer blocks

One block lets each token read from the others once and then transform what it read. A useful model repeats that many times. Because every block takes an n × d_model matrix and returns another n × d_model matrix, blocks can be chained like functions of the same type: the output of block 1 is the input of block 2, and so on to the top.

The problem: one round of mixing is shallow

After one attention layer, a token's vector contains information gathered directly from other tokens. Many useful features are built from features: whether a pronoun refers to a noun depends on what that noun is, which itself depends on its context. Each extra block can build on what the previous ones gathered. Depth also lets information travel in hops: position 9 can read position 5 in layer 2 after position 5 has read position 1 in layer 1.

The mechanism

A Transformer is built by stacking identical blocks: the original encoder and decoder each stack N = 6 identical layers, and all sub-layers and embeddings produce vectors of the same size, so one layer's output can feed the next . "Identical" means the same architecture, not the same weights; each block has its own parameters .

``text token IDs ──► token embedding + position embedding ──► h₀ (n × d) h₀ ──► block 1: h₀ + Attn(LN(h₀)) → + FFN(LN(·)) ──► h₁ (n × d) h₁ ──► block 2: same structure, different weights ──► h₂ (n × d) ⋮ h_{L−1} ──► block L ──────────────────────────────────► h_L (n × d) h_L ──► final LayerNorm ──► output layer ──► logits (n × V) `

The sequence of vectors that flows between blocks, h₀, h₁, …, h_L, is often called the residual stream: each block reads it, computes a correction, and adds the correction back (chapter 24). The final hidden representation h_L, one vector per position, is what chapter 26 turns into probabilities.

Width, depth and parameters

Two numbers describe most of a transformer's size: the width d_model (how many numbers represent each token) and the depth n_layer (how many blocks). Kaplan et al. approximate the non-embedding parameter count as

`text N ≈ 12 · n_layer · d_model² ``

The 12 comes from one block's large matrices when the feed-forward layer is 4·d_model wide: 4·d² in attention's four projections plus 8·d² in the feed-forward network's two matrices. Embedding tables are excluded; the vocabulary size multiplies the width there instead.

A worked example: the toy model, exactly

This page's toy has d = 48, a feed-forward width of 192, 2 blocks, a 233-entry vocabulary and a 56-token context window. Counting every parameter:

Part — Calculation — Parameters; Token embedding — 233 × 48 — 11,184; Position embedding — 56 × 48 — 2,688; Attention, per block — Q, K, V: 48 × 144 + 144; output: 48 × 48 + 48 — 9,408; Feed-forward, per block — 48 × 192 + 192 + 192 × 48 + 48 — 18,672; Two LayerNorms, per block — 2 × (48 gains + 48 biases) — 192; One block — 9,408 + 18,672 + 192 — 28,272; Two blocks — 2 × 28,272 — 56,544; Final LayerNorm — 48 + 48 — 96; Output layer — tied to the token embedding — 0; Total — 11,184 + 2,688 + 56,544 + 96 — 70,512

Kaplan's approximation gives 12 × 2 × 48² = 55,296. The exact non-embedding count is 56,544 + 96 = 56,640; the difference of 1,344 is the biases and LayerNorm parameters that the approximation ignores. For a small model they are a few per cent; for a large model they are negligible.

The same formula applied to published configurations:

Model — n_layer — d_model — 12 · n_layer · d² — Reported; GPT-2 defaults in Transformers — 12 — 768 — 84.9 million — (not stated in these sources); LLaMA 7B — 32 — 4,096 — 6.44 billion — 6.7B; LLaMA 65B — 80 — 8,192 — 64.4 billion — 65.2B

The approximation lands close to the reported totals and below them, because it leaves out embedding parameters by design. For GPT-2's defaults, the token embedding table alone is 50,257 × 768 , about 38.6 million parameters.

Try it

The layer explorer below lists the trained toy model's stages: the embeddings, block 1, block 2, and the final LayerNorm plus tied output layer. Select a stage to see its output shape and exact parameter count; the total at the bottom is read from the model itself. The last column passes each stage's hidden state at the final position through the model's own final LayerNorm and output layer, to show which next token that stage would favour. That readout is exploratory: intermediate states were never trained to be read this way.

Where this is used

Every large language model is a stack of this kind with different widths and depths. LLaMA, for example, is a family from 7B to 65B parameters , with 32 layers at width 4,096 for the smallest and 80 layers at width 8,192 for the largest . GPT-3, according to its paper, is an autoregressive language model with 175 billion parameters . For closed production models, exact layer counts, widths and parameter counts are not stated in the primary sources checked here .

Scale: what the scaling-law papers found

In the original paper's ablations, bigger models were better: the big configuration (d_model 1,024, d_ff 4,096, 16 heads) gave the best English–German result . Kaplan et al. went much further. They found that language-model cross-entropy loss follows a power law in model size, dataset size and training compute, with trends spanning more than seven orders of magnitude , and that performance depends strongly on scale but only very weakly, within reasonable limits, on shape such as depth versus width . They also reported that larger models are more sample-efficient and concluded that compute-efficient training uses very large models on relatively modest data, stopped well before convergence .

The Chinchilla paper revised the data part. Training over 400 models, Hoffmann et al. found that model size and training tokens should be scaled equally, doubling the tokens for every doubling of model size , and concluded that the large models of the time were significantly undertrained . Their 70B-parameter Chinchilla, using the same compute as the 280B Gopher but about 4× more data, outperformed Gopher, GPT-3, Jurassic-1 and Megatron-Turing NLG on many tasks . The LLaMA paper similarly reports that LLaMA-13B outperforms GPT-3 on most benchmarks despite being 10× smaller ; its larger models trained on 1.4T tokens and the smaller ones on 1.0T .

The two scaling papers disagree on how to split a compute budget. The later result changed practice, but both are empirical fits over the ranges they tested, not laws of nature.

Common misconceptions

• "Each layer has one job: layer 3 does grammar, layer 20 does facts." Tenney et al. probed BERT layer by layer and found linguistic steps appear in the expected order through the stack: part-of-speech tagging, parsing, named entities, semantic roles, then coreference . The same paper warns that the order holds only in aggregate, and that the model often adjusts the pipeline dynamically, revising lower-level decisions using higher-level information . Their method measures how well information can be extracted from a layer, which is not the same as showing the model uses it there .
• "More layers is always better." Within reasonable limits, shape matters much less than total scale . Depth trades against width at a fixed parameter budget.
• "Parameters store facts like rows in a table." Parameters are shared by every input; no single weight or layer is "the capital of France". What a model can reproduce is spread across many shared weights rather than stored as separate records, so a model's weights are not a document database.
• "Intermediate readouts show what the model is thinking at that layer." Reading an intermediate state through the final output layer, as the lab does, applies a decoder trained only on the final state. It can be informative but is a heuristic view, not a measurement of intent.

Width and depth trade-offs in practice

Change — Parameters — Compute per token — Sequential steps per forward pass; Double d_model — about 4× (d² terms) — about 4× — unchanged; Double n_layer — about 2× — about 2× — doubled; Double heads at fixed d_model — unchanged — about the same — unchanged

Depth adds sequential work: block 2 cannot start until block 1 has finished for a given token. Width adds work that parallelises within each matrix multiplication. This is one reason hardware considerations influence shape even though loss depends only weakly on it.

Engineering details

• Activation memory scales with depth. Training must keep each block's intermediate activations for the backward pass, so memory grows with n_layer × sequence length × width. The memory chapter (31) quantifies the weights and optimiser side.
• Per-layer caches. At inference, attention is computed in every layer, so the key-value cache is kept per layer ; its size grows linearly with depth.
• Sharding by layer or within a layer. Large models are split across devices either by assigning blocks to different devices or by splitting each block's matrices. Tensor parallelism shards each layer's parameters across GPUs and is the most common way to serve a large model on one node .
• Same shape in, same shape out. The fixed n × d_model interface is what lets researchers add or remove blocks, and what makes the logit-lens readout in the lab possible at all.

26. From hidden states to vocabulary probabilities

At the top of the stack, each position holds a vector of d_model numbers. Text needs something else: a choice of one token ID out of the whole vocabulary. This chapter covers the last three steps of the forward pass, normalise, project and softmax, and the step after it, choosing a token, which is where transformer mathematics turns into generated text.

The problem

A hidden vector is not a word. To predict the next token, the model needs one score for every entry in its vocabulary, and those scores must become a probability distribution that a decoding rule can choose from. GPT-2's vocabulary has 50,257 tokens , so at every step its output layer has to produce 50,257 numbers per position.

The mechanism

• Final normalisation. A Pre-LN Transformer applies a final layer normalisation before the prediction layer .
• Output projection (the language-modelling head). A learned linear layer maps the normalised vector to one raw score per vocabulary entry. These raw outputs are the logits : unbounded real numbers .
• Softmax. Softmax turns the logits into probabilities . Each final position then gives a distribution over the vocabulary .
• Token selection. A decoding rule picks one ID, which is appended to the sequence, and the model runs again.

The mathematics

With final hidden vector h (length d), output matrix W (d × V) and vocabulary size V:

``text z = LN_f(h) · W logits, one per vocabulary entry (length V) p_i = exp(z_i) / Σ_j exp(z_j) probability of token i `

Softmax rescales a vector so every element lies in [0, 1] and they sum to 1 . Two properties follow from the formula. Because exp is always positive, every token gets a probability strictly greater than zero; none is ever impossible, only unlikely. And because only differences between logits matter (adding a constant to every z_i cancels), a logit is meaningful only relative to the others.

Tied embeddings

The original Transformer shares one weight matrix between the two embedding layers and the pre-softmax linear layer , multiplying it by √d_model in the embedding layers . Press and Wolf showed that the topmost weight matrix of a language model is itself a valid word embedding and recommended tying the input embedding to it ; they report that tying lowers perplexity and can cut translation models to under half their size without hurting performance . GPT-2's configuration in Transformers ties them by default , and so does this page's toy model.

With tying, the logit for token i is simply the dot product of the final hidden vector with token i's own embedding row: the model scores each candidate by how well the final vector lines up with that token's embedding.

A worked example with three candidate tokens

A two-dimensional final vector and three embedding rows, chosen by hand. These are not probabilities measured from any real model.

Final normalised vector h = [2, 1]. Tied embedding rows:

Token — Embedding row — Logit z = h · e; blue — [1.00, 0.00] — 2·1.00 + 1·0.00 = 2.0; sky — [0.25, 0.50] — 2·0.25 + 1·0.50 = 1.0; green — [0.00, 0.10] — 2·0.00 + 1·0.10 = 0.1

`text exp(2.0) = 7.3891 exp(1.0) = 2.7183 exp(0.1) = 1.1052 sum = 11.2125

p(blue) = 7.3891 / 11.2125 = 0.6590 p(sky) = 2.7183 / 11.2125 = 0.2424 p(green) = 1.1052 / 11.2125 = 0.0986 total = 1.0000 ``

Even "green", whose logit is far below the others, keeps almost 10%. In a real vocabulary of tens of thousands of tokens, thousands of low-probability entries together can hold a noticeable share of the mass, which is why how the next token is chosen matters (chapter 34).

From probabilities to text

The distribution does not choose anything by itself. Greedy search, the default in Transformers, takes the single most likely token at each step ; here that is "blue". Multinomial sampling draws a token at random according to the distribution, so any token with non-zero probability can be chosen ; here "green" would appear about one time in ten. The chosen ID is appended, the model runs again, and decoder models generate text exactly this way, one token at a time . Finally the IDs are turned back into text by detokenisation .

Try it

The lab below takes the trained toy model's final hidden vector at the last position of a sequence, multiplies it by the tied output matrix to get one logit for each of the toy's vocabulary entries, and applies softmax. It shows the top 10 logits and their probabilities, checks that all probabilities sum to 1, and reports the distribution's entropy next to the entropy of a uniform distribution. These probabilities are real for the toy model; they are not the probabilities of any production LLM.

Where this is used

Every autoregressive language model ends this way. In GPT-2's configuration, the token table is 50,257 × 768 , so with tying, the output layer reuses those 38.6 million parameters instead of adding another copy, and each position's projection costs 768 × 50,257 = 38,597,376 multiply-adds.

Common misconceptions

• "The model outputs words." It outputs a distribution over token IDs, and tokens are often pieces of words. GPT-2 encodes "Hello world" as [15496, 995] but " Hello world" with a leading space as [18435, 995] , so even a space changes which ID is predicted. Decoding produces tokens, not necessarily words.
• "A probability of 0.9 means the answer is 90% likely to be true." It is the model's probability for the next token given the sequence so far. Whether that token begins a true statement is a separate question, and chapter 41 returns to it.
• "Softmax picks the answer." Softmax only produces the distribution. The choice is made by the decoding rule: greedy, sampling, or one of the variants in chapter 34.
• "Logits are probabilities before normalisation, so they must be positive." They are unbounded raw scores ; negative logits are normal. PyTorch's CrossEntropyLoss takes raw logits, which do not need to be positive or sum to 1 .
• "Tokens with very low probability cannot be generated." Under sampling, any token with non-zero probability can be chosen , and softmax never gives exactly zero.

Why many outputs keep probability

Training pushes the logit of the observed next token up and the others down, but the loss −ln p of the correct token only approaches zero as that probability approaches 1. Many continuations of real text are genuinely plausible, so a well-trained model should spread probability across them. The original Transformer even trained with label smoothing of 0.1, which the authors say hurts perplexity because the model learns to be more unsure, but improves accuracy and BLEU .

Temperature, briefly

Decoding libraries often divide the logits by a temperature T before the softmax: p_l = exp(u_l / T) / Σ exp(u_l' / T) . For the example above with T = 0.5, the logits become 4.0, 2.0 and 0.2, and the probabilities become 0.8638, 0.1169 and 0.0193: sharper, as expected for a temperature between 0 and 1 . Chapter 34 covers temperature, top-k and top-p properly.

Softmax is everywhere in the model

The same function sits inside attention: it turns the scaled query–key scores into weights on the values, and masked (−∞) scores become zero weight . In attention it normalises over positions; at the output it normalises over the vocabulary. In PyTorch, the dim argument chooses which axis is normalised, and every slice along it sums to 1 , so getting dim wrong silently normalises the wrong axis.

Engineering details

• Log-probabilities for training. PyTorch notes that nn.Softmax does not work directly with NLLLoss, which expects log-probabilities; LogSoftmax should be used instead . In practice training code passes raw logits to CrossEntropyLoss, which normalises internally and is more stable than taking the log of a probability that may have rounded to zero.
• Only the last position is needed when generating. In training, every position's logits are scored against the next token . During generation only the last position's distribution is used to pick the next token, so implementations can compute the output projection for that position only.
• Entropy as a summary. The entropy H = −Σ p_i log₂ p_i measures how spread out the distribution is. The three-token example has H = 1.22 bits; a uniform distribution over the toy's 233 entries would have log₂ 233 = 7.86 bits. The lab shows the toy model's real value next to that maximum.
• Vocabulary size is a cost. The output layer's work and parameters grow linearly with V. That is one reason vocabulary size is a design choice (chapter 15), and why tying the output layer to the embedding table is attractive.

Architecture differences

Model type — What the head predicts; Decoder-only language model — the next token at every position; Encoder-only, masked LM (BERT) — the original IDs of masked positions; Encoder–decoder — the next target token, given the source and earlier target tokens; Encoder used as a classifier — a small task-specific head over a few classes, not the vocabulary

In each case the last step is the same: a linear layer to scores, then softmax to probabilities.

27. Where training data comes from

A language model learns only from the token sequences it is shown. Before the first gradient is computed, people have decided which text is in the training set, which is thrown away, how often each part is repeated and how it is cut into examples. Those decisions shape the model about as much as its architecture does.

The problem

The largest source of text is the public web, and the web is noisy. Raw Common Crawl text varies widely in quality, so using it needs careful extraction and filtering . The FineWeb authors found that Common Crawl's pre-extracted WET text kept too much boilerplate and menu text, so they extracted text from the raw WARC files with the trafilatura library instead . Repetition is common: Lee et al. found many near-duplicate examples and long repeated substrings in standard language-modelling datasets , including a single 61-word English sentence repeated more than 60,000 times in C4 . The web also contains personal information, spam and copies of evaluation benchmarks.

The pipeline

Published pipelines differ in detail, but most follow the same broad shape:

`` Sources ──► Collection ──► Filtering ──► Deduplication ──► Preparation ──► Tokenization ──► Training examples (web, code, (crawl, text (URL, language, (exact and (mixture weights, (token IDs) (fixed-length books, ...) extraction) quality) near-duplicate) decontamination) blocks) `

Sources. Public web text dominates most published mixtures. FineWeb, for example, is a 15-trillion-token dataset built from 96 Common Crawl snapshots . Curated collections add other kinds of writing: the Pile is an 825 GiB English corpus assembled from 22 subsets, many of them academic or professional , on the premise that more diverse data improves cross-domain knowledge and generalisation . LLaMA's published mixture shows a typical spread :

Source — Sampling proportion; CommonCrawl — 67.0% ████████████████████████████████; C4 — 15.0% ███████; GitHub (code) — 4.5% ██; Wikipedia — 4.5% ██; Books — 4.5% ██; ArXiv — 2.5% █; StackExchange — 2.0% █

For code, LLaMA kept only GitHub projects under the Apache, BSD and MIT licences and removed exact duplicate files . Datasets can also be licensed from their owners, but the sources used for this page do not describe the contents of any such agreement, so nothing more is claimed about them. Human-written examples matter most in later stages: InstructGPT's supervised stage used demonstrations written by labelers . Synthetic data, text or labels produced by a model, appears too: FineWeb-Edu's educational-quality labels were generated by Llama-3-70B-Instruct , and Constitutional AI fine-tunes on responses the model has critiqued and revised itself .

Filtering. FineWeb's base filtering removes adult content with a URL blocklist, keeps only English text with a fastText language score of at least 0.65, and applies quality and repetition filters taken from MassiveText . Other pipelines use learned quality judgements: the Pile's Pile-CC subset uses a classifier trained against OpenWebText2 , and LLaMA's CommonCrawl data was filtered with an n-gram language model after line-level deduplication and fastText language identification .

Deduplication. Lee et al. combine two methods: exact-substring removal with a suffix array, and approximate near-duplicate detection with MinHash ; in the exact method a repeated substring must be at least 50 tokens long to be removed .

Preparation. Sources are reweighted rather than simply concatenated; the Pile raises the weight of components it considers high quality so that more than one epoch passes over them . Evaluation data is removed: the Pile drops any training element that appears verbatim in its held-out data .

Tokenization and examples. The tokenizer from chapter 15 turns the text into token IDs, and for pretraining the documents are concatenated and cut into fixed-length blocks that fit the model's maximum input length and the GPU's memory, rather than padding each document .

The mathematics of near-duplicate detection

Represent each document by its set of shingles: every run of n consecutive words. The Jaccard similarity of two sets A and B is

` J(A, B) = |A ∩ B| / |A ∪ B| `

where |S| is the number of elements in set S. J is 1 for identical sets and 0 for sets with nothing in common. Comparing every pair directly is impossible at web scale, so pipelines use MinHash: hash every shingle with a random hash function and keep the minimum. For one random hash, the probability that two documents have the same minimum equals J(A, B). FineWeb computes 112 such hashes over word 5-grams, split into 14 buckets of 8, targeting documents at least 75% similar , and treats two documents as duplicates when all 8 hashes in any one bucket match . With b buckets of r hashes, that happens with probability

` P(flagged) = 1 − (1 − J^r)^b `

A worked example

Take two short documents and their word 3-grams:

• "the sky looks blue today" → {the sky looks, sky looks blue, looks blue today}
• "the sky looks blue now" → {the sky looks, sky looks blue, looks blue now}

They share 2 shingles and have 4 distinct shingles between them, so J = 2/4 = 0.5. Under a threshold of 0.8 both are kept.

With FineWeb's numbers (r = 8, b = 14): for J = 0.75, 0.75⁸ = 0.1001, so P = 1 − 0.8999¹⁴ ≈ 0.772; for J = 0.5, P ≈ 0.053; for J = 0.9, P ≈ 0.9996. The scheme is a soft threshold: pairs well above 75% similarity are almost always caught, pairs well below it almost never.

Try it

The lab below sends twelve invented web pages through six simplified stages: a URL blocklist, language identification, quality heuristics (fewer than 5 words, a unique-word ratio below 0.4 or a symbol ratio above 0.1), exact deduplication, near-deduplication (word 3-gram Jaccard above 0.8), and PII redaction. With every stage on, 6 of the 12 pages survive (127 words). Document 4 is an exact copy of document 1 and goes at the exact stage; document 9 differs from document 3 only by a final "!" instead of "." and so passes exact deduplication, but its word 3-grams are identical (Jaccard 1.00) and near-deduplication removes it. Turn off the URL blocklist and both spam pages are still removed by the quality heuristics (unique-word ratios 0.33 and 0.17), so the filters overlap. The PII stage keeps the contact page but replaces its email address with [EMAIL]`. With every stage off, all 12 pages (222 words) go through. The pages and thresholds are invented and each rule is a simple stand-in for the trained classifiers and MinHash systems real pipelines use; only the Jaccard computation is real.

Where this shows up

Open datasets such as FineWeb and the Pile publish their pipelines. LLaMA showed that state-of-the-art models could be trained on publicly available data alone; LLaMA-13B outperformed GPT-3 (175B) on most benchmarks . Open weights do not mean open data, though: the FineWeb authors note that the pretraining datasets of leading open models such as Llama 3 and Mixtral are not public and little is known about how they were built .

Common misconceptions

"More data is always better." Quantity matters, but how data is deduplicated and filtered can matter as much. FineWeb found that deduplicating across all snapshots at once did not improve results, while deduplicating each snapshot separately did ; the per-snapshot version produced 20 trillion tokens, and a model trained on a sample of it matched RefinedWeb . Filtering by quality can help sharply: models trained on the 1.3-trillion-token FineWeb-Edu subset do much better on knowledge- and reasoning-heavy benchmarks such as MMLU and ARC . The same pattern appears in fine-tuning, where a 9k-sample dataset beat a 450k-sample one on chatbot performance .

"Filtering is neutral cleaning." Every filter decides what the model will see. A threshold that removes spam can also remove dialect, informal writing or short but valuable text; trained classifiers make the same kind of choice, less visibly.

"The model stores its training data like a database." It does not keep documents in retrievable records. Its parameters are shared numbers adjusted by gradient descent, and you cannot list what it contains. But it can memorise. Carlini et al. extracted hundreds of verbatim training sequences from GPT-2, including public personal information, IRC chats, code and 128-bit UUIDs , confirming 604 unique memorised examples from 1,800 candidates . Memorisation happened even for text that appeared in only one training document , and larger models were more vulnerable than smaller ones .

Contamination

If benchmark questions are in the training data, a good benchmark score may reflect memory rather than ability. The GPT-3 abstract itself notes datasets where the model "faces methodological issues related to training on large web corpora" . Lee et al. found train–test overlap in more than 4% of the validation sets of standard datasets, and deduplication reduces it . A common fix is n-gram decontamination: the Pile removed evaluation instances using the same 13-gram overlap filter as the GPT-3 paper . The full GPT-3 contamination analysis is not quoted here because its full text could not be retrieved . Even careful pipelines leave some leakage; the Pile's authors say documents may still be duplicated across their splits . Contamination can also be a deliberate choice: FineWeb skipped decontamination in one perplexity study precisely to measure how well each dataset covers different domains .

Privacy

Deduplication helps privacy as well as quality: models trained on deduplicated data emitted memorised text ten times less often and needed fewer training steps for the same or better accuracy . Redacting identifiers with a pattern, as the lab does, catches only the easiest cases. Differentially private training can prevent memorisation in theory, but it lengthens training and usually lowers model quality , which is why it is not the default for large language models.

Copyright and licensing

Licences are a filter like any other. LLaMA's licence-based selection of GitHub projects is one published example . Whether training on a given work is lawful depends on jurisdiction, licence terms and facts that this page cannot settle; it is a legal question, not a property of the pipeline.

Multilingual representation, bias and coverage

Many published pipelines are English-only by construction: the Pile is an English corpus , FineWeb keeps English text above a language-score threshold , and LLaMA's CommonCrawl pipeline drops non-English pages . The mixture decides which languages, topics and communities are well represented. Biases present in the data can survive later training stages: InstructGPT, for example, showed no significant improvement on the Winogender and CrowSPairs bias benchmarks over GPT-3 .

Synthetic data and model-made labels

FineWeb-Edu's classifier was trained on scores Llama-3-70B-Instruct gave to 460,000 sampled pages on a 0-to-5 educational-quality scale . Model-made labels are cheap, but they carry the labelling model's judgements and mistakes into the next model's data.

Engineering details

• Epochs differ by source. In LLaMA most tokens were seen only once, but Wikipedia and Books got about two epochs . With roughly 1.4T tokens in total and a 4.5% sampling proportion for Books, about 0.045 × 1.4T ≈ 63B training tokens were drawn from books.
• Extraction tools matter. The Pile used jusText and FineWeb trafilatura ; the same crawl yields different text with each.
• Exact and fuzzy deduplication are complementary. Exact methods catch boilerplate repeated inside different documents; MinHash catches documents that differ slightly, like documents 3 and 9 in the lab.

28. Self-supervised pretraining

Give a model a sentence with its last word hidden, ask it to guess the word, then show it the answer and nudge its weights so the right word becomes a little more likely. Repeat this at every position of every sequence, billions of times, and the model's weights end up encoding a great deal about how text tends to continue. That is pretraining.

The problem

Supervised learning (chapter 3) needs labelled examples, and labelling is slow and expensive. A language model needs a training signal for trillions of tokens. The answer is to let the text label itself. Language-model pretraining is self-supervised: the target is computed automatically from the input, so no human labelling is needed . Pretraining starts from randomly initialised weights with no prior knowledge, needs a very large corpus and can take weeks .

The mechanism: causal next-token prediction

Causal language modelling trains a model to predict the next token from the tokens before it; the model can attend only to tokens on its left and never sees future tokens. GPT-2 is an example . It is called causal because each prediction depends only on past and present inputs . The causal mask (chapter 22) enforces this.

Take the training text "the sky appears blue ." as five tokens. The targets are simply the same tokens shifted left by one:

Position — Input the model sees — Target; 1 — the — sky; 2 — the sky — appears; 3 — the sky appears — blue; 4 — the sky appears blue — .

The row in bold is the spec's example: input "the sky appears", target "blue". The first token is never a target, and the logit at the last position has no following token to compare with, so it is dropped . In Hugging Face Transformers the labels can simply be the input IDs, because the one-position shift happens inside the model . The documentation is not perfectly consistent in wording: the causal-LM task guide describes the collator as using the inputs as labels "shifted to the right by one element", while the model docs and the course say the shift happens inside the model; either way the target at each position is the following token .

One training step then runs:

• Training tokens. A batch of fixed-length blocks of token IDs.
• Shifted targets. Each position's target is the next ID.
• Forward pass. The whole block goes through the network once. Because of the causal mask, the output at position t depends only on positions 1 to t, so a single pass produces a prediction for every position at the same time.
• Logits. At each position the output layer produces one score per vocabulary entry (chapter 26).
• Cross-entropy loss. Each position's loss is −ln of the probability given to its target, averaged over positions. The course code shifts labels and logits by one and averages a per-token cross-entropy over all positions ; PyTorch's CrossEntropyLoss defaults to reduction = 'mean' and ignore_index = -100 .
• Backward pass. Backpropagation (chapter 12) computes the gradient of that mean loss with respect to every weight.
• Parameter update. The optimiser moves each weight a small step against its gradient. LLaMA used AdamW with β₁ = 0.9, β₂ = 0.95 and a cosine learning-rate schedule .

This is teacher forcing: at every position the model is conditioned on the real preceding tokens from the data, never on its own earlier guesses . The Hugging Face pages consulted describe this mechanism without using the term .

The mathematics

For a sequence of token IDs x₁ … x_{T+1}, the loss is

`` L(θ) = −(1/T) · Σ_{t=1..T} ln p_θ(x_{t+1} | x_1, …, x_t) `

where θ is every weight in the model, T is the number of predicted positions, and p_θ(x_{t+1} | x_1 … x_t) is the softmax probability the model gives the true next token at position t. For one position, with logits z and probabilities p = softmax(z), the gradient of −ln p_target with respect to the logits is

` ∂L/∂z_k = p_k − y_k (y_k = 1 for the target token, 0 otherwise) `

so the target's logit is pushed up by (1 − p_target) and every other logit is pushed down by its own probability. With averaging over T positions, each of these is divided by T. Perplexity is the exponential of the mean loss; the Transformers guide computes it as math.exp(eval_loss) .

A worked example

The probabilities below are invented to make the arithmetic readable. They are not measurements from any real model.

Position — Target — p(target) — Loss −ln p; 1 — sky — 0.10 — 2.3026; 2 — appears — 0.05 — 2.9957; 3 — blue — 0.60 — 0.5108; 4 — . — 0.50 — 0.6931

The sum is 6.5023, so the mean loss is 6.5023 / 4 = 1.6256 nats and the perplexity is e^1.6256 ≈ 5.08: on average the model is as uncertain as a fair choice among about five tokens. At position 3 the gradient on the "blue" logit is (0.60 − 1) / 4 = −0.10, and a competing token with probability 0.15 gets +0.15 / 4 = +0.0375. Gradient descent subtracts the gradient, so "blue" rises and its competitor falls. Note that position 2 has the largest loss: "the sky" could be followed by many words, and no amount of training can make an unpredictable continuation predictable.

Try it

The lab runs this page's toy transformer (chapter 25), a two-block model trained in exactly this way on a small corpus written for the page. Pick an example sequence or type your own (the toy vocabulary is lower-case words, with punctuation and digits as separate tokens; unknown words become <unk>`), and the lab runs one forward pass over up to the first 24 tokens. Each row shows the position, the last few input tokens the model can see, the target (the next token), the probability the model gave it, the loss −ln P (highlighted above 1) and the model's own top guess, with the mean loss over all positions at the bottom. The computation is real; the model is tiny and has memorised much of its corpus, so low losses on corpus text do not indicate generalisation.

Where this shows up

Every decoder-only LLM is pretrained with this objective, at a vastly larger scale. GPT-3 is an autoregressive language model with 175 billion parameters . All LLaMA models used batches of 4M tokens; the 33B and 65B models trained on 1.4T tokens and the smaller ones on 1.0T . That is 1.4 × 10¹² / 4 × 10⁶ = 350,000 optimiser steps for the larger models and 250,000 for the smaller, each one the seven-step loop above. The toy model ran 16,000 steps of 16 sequences of at most 56 tokens, which is at most 14.34 million target positions.

Training and use are separate phases. GPT-3 was evaluated with no gradient updates, with the task given only as text in the prompt . The contrast:

Inputs at each position — the real text (teacher forcing) — the model's own previous outputs; Positions computed per pass — all positions in parallel — prompt in parallel, then one new token per step; Loss and gradients — yes — no; Weights — change every step — normally fixed

Common misconceptions

"The model is trained to answer questions." It is trained to predict the next token of whatever text it is given, including the question, the boilerplate around it and the typos. A model trained only this way develops a statistical understanding of language but is less useful for specific tasks until it is adapted ; chapter 29 covers that adaptation.

"Training generates text one token at a time." Generation does; training does not. Teacher forcing plus the causal mask means one forward pass scores every position . This parallelism is a large part of why transformers train efficiently on GPUs.

"Low loss means the model tells the truth." The objective rewards text that is likely under the training distribution, not text that is true. A false answer that is common in the data scores well; TruthfulQA's authors call such answers imitative falsehoods .

"Pretraining copies documents into the weights." Training adjusts shared weights to lower the average loss over everything; it does not file documents away. Some text is nonetheless memorised, even text seen in only one document , so "it is just statistics" is not a privacy guarantee either.

"All transformers are pretrained this way." Not all. BERT, an encoder-only model, is trained with a masked language-model objective: random input tokens are masked and predicted from context on both sides . Causal next-token prediction is the decoder-only recipe.

Advanced insight: the gap between training and generation

Because training always conditions on real prefixes, the model never practises recovering from its own mistakes during pretraining. At inference time one unlikely token changes every later input. Decoding strategies (chapter 34) and later training stages partly address this, but it is one reason that a model with good average loss can still drift once it starts generating.

The loss is an average over positions, and positions are not equally hard. In the worked example, the first content word after "the" carried most of the loss. In the toy lab, the highest losses usually fall at the start of an answer or on a fact the model can only guess. A falling mean loss can therefore hide that some kinds of prediction are not improving at all.

Engineering details

• Packing, not padding. Documents are concatenated and cut into chunks of block_size tokens that fit the model's maximum input length and GPU memory .
• Context length is a cost choice. The Hugging Face course trains with 128-token contexts, against 1,024 for GPT-2 and 2,048 for GPT-3, and warns that longer contexts need more GPU memory .
• Masking positions out of the loss. Labels set to −100 are ignored, so only real targets count ; ignored targets contribute nothing to the gradient, and with averaging the loss is averaged over the remaining targets only . The toy model uses the same idea: the <result> tokens that the application writes into its tool-call sequences are excluded from the loss, so the model is never trained to invent tool results.
• Schedules and stability. LLaMA used weight decay 0.1, gradient clipping at 1.0 and 2,000 warmup steps, ending its cosine schedule at 10% of the peak learning rate . The course's from-scratch GPT-2 recipe uses cosine decay with warmup and an effective batch of 256 sequences, the per-device batch size times the number of gradient-accumulation steps . The toy model was trained with AdamW, a short warmup and a cosine schedule.

29. Fine-tuning and instruction following

A pretrained model has learned to continue text. Ask it a question and it may answer, or it may continue with three more questions, because a list of questions is a perfectly likely continuation of a question. Fine-tuning is how that general text-continuer is turned into something that behaves the way a particular application needs.

The problem

Large language models can produce output that is untruthful, toxic or simply unhelpful; out of the box they are not aligned with what their users want . The InstructGPT authors note that base GPT-3 can perform instruction tasks but needs more careful prompting and rarely follows instructions in some domains . Training a new model from scratch for every behaviour is far too expensive, so the usual approach is to keep the pretrained weights as a starting point and train further.

Base model versus instruction-tuned model

Both outputs below were written for this page to show the typical difference. Neither was generated by a real model.

Prompt: Explain in one sentence why the sky is blue.

• Base model (continues the text): Explain in one sentence why grass is green. Explain in one sentence why the sea is salty. Explain in one sentence ... The prompt looks like the start of a worksheet, so more worksheet is a likely continuation.
• Instruction-tuned model (follows the instruction): Air molecules scatter short blue wavelengths of sunlight much more strongly than long red ones, so blue light reaches your eyes from all over the sky.

Same architecture, same tokenizer, same kind of next-token prediction. Only the weights differ.

The mechanism

Fine-tuning is further training of an already pretrained model on a dataset specific to the target task . It builds on pretraining because the pretrained model has already seen data similar to the fine-tuning set . Variants differ mainly in the data:

• Supervised fine-tuning (SFT) on instruction–response pairs. InstructGPT's first stage fine-tuned GPT-3 with supervised learning on demonstrations of the desired behaviour written by labelers , using about 13k training prompts . The loss is the same next-token cross-entropy as pretraining, often computed on the response tokens only by giving the prompt positions the ignored label −100 .
• Instruction tuning across many tasks. FLAN fine-tuned a 137B-parameter pretrained model on more than 60 NLP datasets rewritten as natural-language instructions , with ten templates per dataset ; this substantially improved zero-shot performance on unseen tasks , beating zero-shot 175B GPT-3 on 20 of 25 datasets .
• Task adaptation fine-tunes on one task's labelled data, such as classifying support tickets.
• Domain adaptation continues training on text from one field, such as legal or medical writing, so the model's predictions fit that field's vocabulary and style.

Parameter freezing. You do not have to update every weight. A frozen weight is excluded from the update: no gradient is applied to it and its value stays exactly as it was after pretraining. Full fine-tuning updates everything, which becomes impractical at scale; deploying separate full copies of GPT-3 175B, each with 175B parameters, is prohibitively expensive . Parameter-efficient fine-tuning (PEFT) trains only a small number of (possibly extra) parameters, with performance comparable to full fine-tuning .

LoRA

LoRA freezes the pretrained weights and injects trainable rank-decomposition matrices into each Transformer layer . For a frozen weight matrix W₀ of shape d × k, the adapted weight is

`` W′ = W₀ + ΔW = W₀ + (α / r) · B · A `

where B is d × r, A is r × k, the rank r is much smaller than min(d, k), and only A and B are trained . α is a scaling constant; the update is scaled by α/r . A starts with random Gaussian values and B starts at zero, so B · A = 0 at the beginning and training starts exactly at the pretrained model . The method rests on the hypothesis that the weight change needed for adaptation has a low "intrinsic rank" .

` k r k ┌─────────────┐ ┌──┐ ┌─────────────┐ │ │ │ │ │ A │ r (trained) d │ W₀ frozen │ + α/r · │B │ · └─────────────┘ │ d·k values │ d │ │ (trained, starts at 0) │ │ │ │ └─────────────┘ └──┘ x ──► W₀·x ─────────────────────────┬──► W₀·x + (α/r)·B·(A·x) x ──► A·x (r numbers) ──► B·(A·x) ──┘ ``

The trainable count for one d × k matrix falls from d·k to r·(d + k).

A worked example

For a square projection with d = k = 4096 and r = 8:

• full fine-tuning: 4096 × 4096 = 16,777,216 trainable values;
• LoRA: 8 × (4096 + 4096) = 65,536 trainable values, which is 0.39% of the full count, 256 times fewer.

Scale it to a model. LLaMA-7B has width 4096 and 32 layers . Applying r = 8 to just the query and value projections in every layer gives 2 × 32 × 65,536 = 4,194,304 trainable parameters, about 0.063% of its 6.7 billion.

Now a tiny matrix by hand, as in the lab. With α = 16 and r = 8, α/r = 2. If the first row of B is (0.2, 0) and the first column of A is (0.5, 0), then ΔW[1,1] = 2 × (0.2 × 0.5 + 0 × 0) = 0.2. If the second row of B is (0.1, −0.1) and the second column of A is (−0.3, 0.4), then ΔW[2,2] = 2 × (0.1 × −0.3 + (−0.1) × 0.4) = 2 × (−0.07) = −0.14. A 6 × 6 ΔW has 36 entries but is produced by 6 × 2 + 2 × 6 = 24 trainable numbers, and its rank is at most 2.

Try it

The lab has three sliders: the matrix width d (256 to 8,192, default 4,096), the rank r (1 to 64, default 8) and α (1 to 64, default 16). It reports the full and LoRA trainable counts for one d × d matrix, their ratio, and a 6 × 6 worked example showing the frozen W, a 6 × 2 B, a 2 × 6 A and ΔW = (α/r)·B·A. The small example always uses rank-2 matrices so the numbers stay readable, but its scale factor α/r follows your sliders. The B it shows is a hypothetical trained value; in real LoRA, B starts at zero.

Where this shows up

Compared with fine-tuning GPT-3 175B with Adam, LoRA cut trainable parameters by 10,000 times and GPU memory by 3 times , reducing training VRAM from 1.2TB to 350GB; with r = 4 on the query and value projections only, the checkpoint shrank from 350GB to 35MB . LoRA matched or beat full fine-tuning on RoBERTa, DeBERTa, GPT-2 and GPT-3 . One shared frozen base model can serve many tasks by swapping small A and B matrices .

QLoRA pushes this further. It backpropagates through a frozen, 4-bit-quantised base model into LoRA adapters, which made it possible to fine-tune a 65B model on a single 48GB GPU while keeping full 16-bit fine-tuning performance , cutting average memory from more than 780GB to under 48GB . The arithmetic shows why: 65 × 10⁹ weights × 0.5 bytes = 32.5 GB for the 4-bit weights alone, leaving room for the adapters, their optimiser state and activations, whereas the same weights in 16-bit take 130 GB.

Common misconceptions

"An instruction-tuned model is a different kind of model." It is usually the same network with different weights. What changes is behaviour, and often the expected prompt format: Mistral-7B-Instruct and Zephyr-7B come from the same base but use different control tokens, and the wrong ones badly hurt quality .

"Instruction tuning always helps." Not in every regime. In FLAN it helped models near 100B parameters but hurt held-out-task performance for models of 8B and smaller . Its ablations found that the number of fine-tuning datasets, model scale and natural-language instructions were all key .

"Lower validation loss means a better assistant." InstructGPT's SFT models overfit on validation loss after one epoch, yet training for more epochs still improved reward-model scores and human preference ratings ; the final recipe trained for 16 epochs with cosine decay and residual dropout of 0.2 . Next-token loss on demonstrations is only a proxy for what people want.

"More fine-tuning data is better." QLoRA found data quality far more important than size: 9k samples from OASST1 beat 450k subsampled FLAN v2 samples on chatbot performance .

"LoRA slows inference down." Not once merged. The matrices can be merged into the frozen weights, W′ = W₀ + (α/r)·B·A, so inference costs the same as a fully fine-tuned model ; PEFT's merge_and_unload() does exactly that . Keeping adapters unmerged costs a small extra multiplication per adapted layer, in exchange for being able to swap tasks at run time.

"Freezing means the model's behaviour is fixed." The frozen W₀ never changes, but the effective weight W₀ + ΔW does, and so does the output. Freezing is about what the optimiser may touch, not about what the model computes.

Advanced insight: why low rank is enough

ΔW = B·A can have rank at most r, so LoRA can only express weight changes that live in an r-dimensional subspace of each matrix. The surprise is how small r can be: the LoRA paper found r of one or two sufficient even where the full rank d is 12,288 . One reading is that adaptation mostly re-weights directions the pretrained model already represents, rather than building new ones. The α/r factor divides the update by the rank, so raising r adds capacity without automatically making the initial update larger.

Architecture differences

LoRA and PEFT are not specific to decoder-only models: LoRA was evaluated on encoder-only RoBERTa and DeBERTa as well as decoder-only GPT-2 and GPT-3 . In Transformers, LoRA is typically applied to the attention blocks only, for simplicity and parameter efficiency, though any subset of weight matrices can be targeted .

Engineering details

• Main LoRA settings. In PEFT's LoraConfig: r (rank; lower means fewer trainable parameters), target_modules, lora_alpha (scaling) and bias ('none', 'all' or 'lora_only') .
• QLoRA's data types. Weights are stored in a low-precision type, usually 4-bit, and each weight tensor is dequantised to BFloat16 just before a 16-bit matrix multiply . Its 4-bit NormalFloat (NF4) type is designed for normally distributed weights, Double Quantization quantises the quantisation constants, and Paged Optimizers absorb memory spikes ; Double Quantization saves about 0.37 bits per parameter, roughly 3 GB on a 65B model, and Paged Optimizers use NVIDIA unified memory to absorb gradient-checkpointing spikes on long sequences .
• Dataset shape. InstructGPT's data sizes were about 13k SFT prompts, 33k reward-model prompts and 31k PPO prompts ; the latter two belong to the next chapter's preference stage.
• What the loss sees. Masking prompt tokens out of the loss (labels of −100, chapter 28) makes the model learn to produce responses rather than to reproduce prompts.

30. Preference alignment and model behaviour

It is hard to write down, as a formula, what makes one answer better than another. It is much easier for a person to look at two answers and say which they prefer. Preference alignment turns many such judgements into a training signal.

The problem

Supervised fine-tuning (chapter 29) teaches a model to imitate demonstrations, but demonstrations are slow to write and say nothing about which of two plausible answers is better. The idea of learning from comparisons predates LLMs. Christiano et al. learned goals from non-expert human preferences between pairs of behaviour clips instead of from a hand-written reward function , solving Atari and simulated-robot tasks with feedback on less than 1% of the agent's interactions , and teaching novel behaviours such as a backflip with about an hour of human time .

The mechanism: several techniques, not one pipeline

These methods share a goal but differ in what they train and how. Treat them as options that different developers combine differently, not as one fixed recipe.

• Preference data. For InstructGPT, labelers ranked between K = 4 and K = 9 responses per prompt, which yields K-choose-2 comparisons per prompt ; OpenAI hired 40 contractors, chosen by a screening test, to do the labelling .

• Reward modelling. A reward model reads a prompt and a response and outputs one number. InstructGPT's reward model starts from the SFT model with the final unembedding layer removed and outputs a single scalar ; only 6B reward models were used, to save compute and because 175B reward-model training could be unstable . It is trained so that the difference in rewards is the log odds that a labeler prefers one response over the other , the Bradley–Terry model for estimating scores from pairwise comparisons .

• RLHF. Reinforcement learning from human feedback fine-tunes the SFT model against the reward model . InstructGPT used PPO in a bandit setup: a random prompt, one response, one reward, end of episode . A per-token KL penalty from the SFT model stops the policy over-optimising the reward model .

• Direct Preference Optimization (DPO). DPO's authors describe RLHF as complex and often unstable . DPO reparameterises the reward so that the same objective reduces to a simple classification loss on preference pairs, with no separate reward model to sample from and no RL loop ; it needs no sampling from the model during fine-tuning and little hyperparameter tuning .

• Safety-oriented post-training with AI feedback. Constitutional AI trains a harmless assistant without human labels for harmful outputs; the human oversight is a written list of principles . Its supervised phase has the model critique and revise its own responses and fine-tunes on the revisions ; its RL phase trains a preference model on AI judgements and uses it as the reward, called RL from AI Feedback (RLAIF) .

The mathematics

Bradley–Terry reward model. For a prompt x and responses A and B with scalar rewards r(x, A) and r(x, B),

`` P(A preferred to B) = σ(r(x, A) − r(x, B)), σ(z) = 1 / (1 + e^(−z)) loss_RM = −ln σ(r(x, A) − r(x, B)) for a pair where A was preferred `

The shared objective. Both RLHF and DPO target

` maximise E[ r(x, y) ] − β · KL( π_θ(y | x) ‖ π_ref(y | x) ) `

where π_θ is the model being trained (the policy), π_ref is a frozen reference model (normally the SFT model), y is a response, KL measures how far the two distributions are apart, and β controls how far the policy may move from the reference .

DPO's loss for a prompt x with chosen response y_w and rejected response y_l is

` margin = β · [ (ln π_θ(y_w|x) − ln π_ref(y_w|x)) − (ln π_θ(y_l|x) − ln π_ref(y_l|x)) ] loss = −ln σ(margin) ``

Each ln π(y | x) is the sum of the token log-probabilities of the whole response. The gradient of the loss with respect to the margin is −σ(−margin), so σ(−margin) acts as a per-example weight: pairs the policy already gets right push little, pairs it gets wrong push hard. The DPO update raises the log-probability of the preferred response relative to the dispreferred one with exactly such a dynamic per-example weight, which prevents degeneration .

A worked example

All rewards and log-probabilities here are invented numbers chosen to make the arithmetic readable.

Reward model. If r(x, A) = 1.5 and r(x, B) = 0.5, then P(A preferred) = σ(1.0) = 0.7311 and the loss for a pair where people preferred A is −ln 0.7311 = 0.3133. A labeler who ranks K = 4 responses produces 4·3/2 = 6 comparisons; K = 9 produces 9·8/2 = 36.

DPO. Use the lab's defaults: ln π_θ(A) = −12, ln π_ref(A) = −13, ln π_θ(B) = −14, ln π_ref(B) = −13, β = 0.1.

• The policy has raised A by (−12) − (−13) = +1 relative to the reference, and lowered B by (−14) − (−13) = −1.
• margin = 0.1 × (1 − (−1)) = 0.2.
• loss = −ln σ(0.2) = −ln 0.5498 = 0.5981.
• weight = σ(−0.2) = 0.4502.

Move the policy further, to ln π_θ(A) = −10 and ln π_θ(B) = −16: margin = 0.1 × (3 − (−3)) = 0.6, loss = 0.4375, weight = 0.3543. Swap the policy's two values instead (A = −14, B = −12): margin = −0.2, loss = 0.7981, weight = 0.5498. The more the policy prefers the rejected answer relative to the reference, the larger the loss and the stronger the push.

Try it

The lab has four sliders, from −30 to −1, for the policy and reference log-probabilities of the chosen response A and the rejected response B, and a fifth for β (0.01 to 1). It shows the implicit reward margin with the numbers substituted, the loss −log σ(margin) and the gradient weight σ(−margin). Make the reference prefer B and the policy prefer A, then raise β, and watch the weight fall towards zero. The values are hypothetical, not measured from any model.

Where this shows up

InstructGPT's labelers preferred outputs of its 1.3B model to those of the 175B GPT-3, despite the 100× difference in size . On TruthfulQA it gave truthful and informative answers about twice as often as GPT-3 , and on closed-domain tasks such as summarisation it made up information about half as often, 21% against 41% . It produced about 25% fewer toxic outputs when prompted to be respectful, but showed no significant bias improvement on Winogender and CrowSPairs . DPO exceeded PPO-based RLHF at controlling sentiment and matched or improved response quality in summarisation and single-turn dialogue .

Common misconceptions

"Alignment makes a model truthful." It makes the model match stated preferences better; it does not guarantee facts. InstructGPT still hallucinated at a 21% rate on closed-domain tasks . Its authors say their models are neither fully aligned nor fully safe and still make up facts ; given a false premise the model sometimes assumes it is true, and it can hedge too much on simple questions . They also note that their truthfulness metrics capture only a small part of what truthfulness means . A preference label records which answer a person liked, not which answer is correct.

"Aligned with human values." InstructGPT was aligned to the stated preferences of a specific group, mostly its labelers and researchers, rather than to any broader notion of human values .

"Bigger models will fix hallucination on their own." On TruthfulQA the largest models tested were generally the least truthful, the opposite of most NLP tasks , because false answers that are likely under the training distribution, imitative falsehoods, are what the next-token objective rewards . The authors suggest scaling alone is less promising than fine-tuning with objectives other than imitating web text .

"RLHF, DPO and Constitutional AI are the same thing." They are not:

RLHF (InstructGPT) — human rankings — yes — yes, PPO — SFT model, via KL penalty; DPO — preference pairs — no — no — yes, inside the loss; Constitutional AI — AI judgements guided by written principles — yes, a preference model — yes (RLAIF) — not covered by the sources used here

"Higher reward means better behaviour." The reward model is a learned proxy, and optimising a proxy too hard exploits its flaws. That is why RLHF adds a KL penalty . The same happens with AI feedback: over-trained RL-CAI models showed Goodharting, becoming overly harsh or adding boilerplate .

Evaluations and trade-offs

TruthfulQA has 817 questions in 38 categories, including health, law, finance and politics, written so that some humans would answer falsely because of misconceptions . The best model tested was truthful on 58% of questions, against 94% for humans . The benchmark counts "No comment" or "I don't know" as truthful , which exposes a real trade-off: a model can raise its truthfulness score by refusing, at the cost of helpfulness. Constitutional AI's stated aim of a harmless but non-evasive assistant, one that engages with harmful queries by explaining its objections , is an attempt to manage that tension rather than escape it.

Alignment can cost capability. RLHF gave InstructGPT lower scores than GPT-3 on some public datasets such as SQuAD, DROP, HellaSwag and WMT 2015 French–English, which its authors call an "alignment tax" . Mixing in updates that raise log-likelihood on the pretraining distribution (PPO-ptx) greatly reduced those regressions .

Advanced insight: what β does

β appears in both the RLHF objective and the DPO loss . In the lab, β multiplies the log-ratio difference, so a larger β saturates the sigmoid sooner: a pair stops contributing gradient after a smaller move away from the reference. A small β tolerates large moves before the loss flattens. In both cases the reference model anchors the policy: preference training adjusts a model that already exists rather than building a new one.

Engineering details

• Reference log-probabilities. DPO needs ln π_ref(y | x) for each response. Because the reference is frozen, these can be computed once before training or on the fly with a second forward pass.
• Sequence log-probabilities. The log-probability of a response is the sum over its tokens, computed with the teacher-forced forward pass of chapter 28 on the response positions.
• Dataset sizes. InstructGPT used about 33k prompts for reward-model training and 31k for PPO , far fewer examples than pretraining.
• Evaluation is part of the method. Every row in the comparison table above is judged with benchmarks and human ratings; the limits of those measurements are limits on what any alignment claim can show.

31. Hardware and distributed training

Most of the arithmetic in a neural network is matrix multiplication: a layer of neurons is one matrix multiply plus a bias vector . The hardware question is how fast a machine multiplies large matrices, and whether it can hold all the numbers involved.

The problem

Memory is usually the harder part. ZeRO's example: a 1.5B-parameter GPT-2 needs only 3 GB for its 16-bit weights, yet at least 24 GB for its model states during training, and cannot be trained on a single 32GB GPU with plain PyTorch or TensorFlow .

CPUs, GPUs and memory bandwidth

A CPU has a few powerful cores built for varied, branching code. A GPU has many simpler arithmetic units that apply one operation to large blocks of data at once, which suits matrix multiplication, where every output element is an independent dot product. GPUs carry their own memory (VRAM); on data-centre GPUs it is high-bandwidth memory (HBM). The FlashAttention paper's A100 example has 40–80GB of HBM at 1.5–2.0TB/s, and 192KB of much faster on-chip SRAM in each of its 108 streaming multiprocessors, at about 19TB/s .

Compute has grown faster than memory speed, so operations are increasingly limited by reads and writes to HBM rather than by arithmetic . Large matrix multiplies are compute-bound; elementwise operations such as activations and dropout, and reductions such as softmax and layer norm, are memory-bound . FlashAttention computes exact attention with tiling to cut traffic between HBM and SRAM, giving a 3× training speedup on GPT-2 at sequence length 1K .

Number formats and mixed precision

Each stored number costs bytes, and the format decides range and precision:

Format — Bits (sign–exponent–mantissa) — Bytes — Note; FP32 — 1–8–23 — 4 — IEEE single precision; FP16 — 1–5–10 — 2 — narrow range; BF16 — 1–8–7 — 2 — FP32's range, less precision

In FP16 any value smaller in magnitude than 2⁻²⁴ becomes zero; in one example about 5% of weight-gradient values were that small . Mixed-precision training uses half-precision numbers without losing accuracy, nearly halving memory and speeding up arithmetic on recent GPUs . For FP16 it relies on an FP32 master copy of the weights, loss scaling, and FP16 arithmetic that accumulates in FP32 ; the master copy accumulates each update and is rounded to half precision for the forward and backward passes . Loss scaling multiplies the loss before the backward pass so small gradients stay representable instead of flushing to zero . BF16 has FP32's range, so the two 16-bit formats are not interchangeable: PyTorch's BF16 autocast example uses no loss scaling at all .

Where the memory goes

For training, ZeRO counts model states with mixed-precision Adam: 2 bytes for each 16-bit parameter, 2 bytes for each 16-bit gradient, and 12 bytes of FP32 optimiser state (a parameter copy, momentum and variance, 4 + 4 + 4) , so 16 bytes per parameter before activations . Activations, the intermediate values saved during the forward pass for use in the backward pass, are a separate cost that depends on batch size, sequence length and checkpointing, along with temporary buffers and fragmentation .

For inference there are no gradients or optimiser states, but there is the KV cache: each layer stores a key and a value vector per KV head for every token in context (chapter 35).

`` inference ≥ N · b_w + 2 · L · H_kv · d_h · b_kv · T · B training ≥ 16 · N + activations + buffers ``

where N is the parameter count, b_w the bytes per weight, L the number of layers, H_kv the number of key/value heads, d_h the head dimension, b_kv the bytes per cached number (2 for 16-bit), T the tokens per sequence and B the number of concurrent sequences.

A worked example: LLaMA-7B by hand

LLaMA-7B has 6.7 billion parameters, width 4096, 32 heads and 32 layers . Assume standard multi-head attention, where every head has its own keys and values, so H_kv · d_h equals the width, 4096.

Weights.

• FP32: 6.7 × 10⁹ × 4 B = 26.8 GB
• BF16 or FP16: 6.7 × 10⁹ × 2 B = 13.4 GB (the spec's rule of thumb, "7B in 16 bits ≈ 14 GB", is the same calculation)
• INT8: 6.7 GB; 4-bit: 3.35 GB

KV cache, 16-bit. Per token: 2 × 32 layers × 4096 × 2 B = 524,288 B = 512 KiB. For one 2,048-token sequence: 524,288 × 2,048 = 1,073,741,824 B ≈ 1.07 GB. Eight such sequences at once: about 8.6 GB, more than half the size of the 16-bit weights.

Inference lower bound, BF16, one 2,048-token sequence: 13.4 + 1.07 ≈ 14.5 GB.

Training model states: 6.7 × 10⁹ × 16 B = 107.2 GB, made of 13.4 GB of 16-bit weights, 13.4 GB of gradients and 80.4 GB of FP32 Adam state. That already exceeds an 80GB A100 before a single activation is stored.

These are lower bounds from counting bytes. Real usage adds activations, temporary buffers, fragmentation, framework and kernel workspaces, and communication buffers. No estimate of this kind guarantees that a model will or will not fit on particular hardware.

Try it

The memory lab has a mode switch (inference or training), a model preset (Custom, OPT-13B or Mistral 7B with grouped-query attention), a parameter slider (0.1 to 180 billion) and, in inference mode, a weight-precision choice (FP32, BF16, INT8, 4-bit), a context length (256 to 131,072 tokens, default 4,096) and a batch of concurrent sequences (1 to 64). It shows the lower bound and, separately, a +20% rule-of-thumb heuristic. The default (a 7B model shaped like LLaMA-7B, BF16, 4,096 tokens, batch 1) reads 14.0 GB of weights, 512 KiB of KV cache per token, 2.15 GB of cache in total, a 16.1 GB lower bound and a 19.4 GB heuristic. Set parameters to 6.7 and context to 2,048 to reproduce the hand calculation above. The OPT-13B preset's 800 KiB per token matches the vLLM paper's figure of about 800 KB . Mistral 7B uses 8 key-value heads instead of 32 , so its per-token cache is a quarter of the size. Training mode applies 16 bytes per parameter, without activations.

Where this shows up

Training LLaMA-65B processed about 380 tokens per second per GPU on 2,048 A100 80GB GPUs, so its 1.4T-token dataset took about 21 days . Checking: 380 × 2,048 = 778,240 tokens per second, and 1.4 × 10¹² / 778,240 ≈ 1.8 million seconds ≈ 20.8 days.

Data parallelism

Megatron-LM describes two scaling paradigms: data parallelism splits a minibatch across workers, and model parallelism spreads a model's memory and computation across workers . Data parallelism alone has a hard limit: the whole model must fit on one worker .

In PyTorch's DistributedDataParallel (DDP), construction broadcasts the model state from rank 0 so every replica starts identical, and each process creates a Reducer to synchronise gradients during backward . Gradients are grouped into buckets; when a bucket is ready, an asynchronous all-reduce averages it across processes while the rest of backward continues . Starting identical and applying identical averaged gradients keeps the replicas in sync . If processes run the all-reduces in different bucket orders, results can be wrong or backward can hang .

Sharding: ZeRO and FSDP

Plain data parallelism keeps a full copy of the model states on every GPU; ZeRO-DP removes that redundancy, and partitioning the parameters as well cuts memory linearly with the data-parallel degree, at the cost of about 50% more communication . A trillion-parameter model trained with Adam in 16-bit needs about 16 TB of model states, 16GB per GPU across 1,024 GPUs with ZeRO ; by the 16-byte rule, 10¹² × 16 B = 16 TB and 16 TB / 1,024 ≈ 15.6 GB. For LLaMA-7B's 107.2 GB of model states, full sharding across 8 GPUs gives 13.4 GB each, before activations.

PyTorch's FullyShardedDataParallel (FSDP) shards module parameters across data-parallel workers and is inspired by ZeRO Stage 3 . With FULL_SHARD, parameters, gradients and optimiser states are all sharded: parameters are all-gathered before forward and backward and resharded afterwards, gradients are reduce-scattered, and each rank updates its own optimiser shard . NO_SHARD replicates everything like DDP .

Tensor parallelism

Tensor (intra-layer) parallelism splits each transformer layer's matrices across GPUs and complements pipeline parallelism . In the MLP block, Megatron splits the first matrix multiply by columns so GeLU can run locally on each shard, and the second by rows, then reduces the outputs across GPUs . A tensor-parallel layer needs only two all-reduces in the forward pass and two in the backward pass . Megatron-LM trained models of up to 8.3B parameters on 512 GPUs, sustaining 15.1 PetaFLOPs at 76% scaling efficiency against a single-GPU baseline of 39 TeraFLOPs . Per GPU that is 15,100 / 512 ≈ 29.5 TeraFLOPs, and 29.5 / 39 ≈ 0.76.

Pipeline parallelism

GPipe places consecutive groups of layers on different accelerators, splits each mini-batch into micro-batches and pipelines them through . Gradients are accumulated over all micro-batches and applied at the end of the mini-batch, so the update does not depend on the number of partitions . The cost is idle time, the "bubble", of order (K − 1) / (M + K − 1) for K partitions and M micro-batches; GPipe found it negligible when M ≥ 4K . For K = 4, the formula gives 3/7 ≈ 43% with M = 4, 3/19 ≈ 16% with M = 16 and 3/67 ≈ 4.5% with M = 64.

Gradient accumulation

Gradient accumulation trains with a larger effective batch than fits in memory: gradients from several batches are summed and the optimiser steps only after a set number of them . The Hugging Face course's GPT-2 recipe defines its effective batch of 256 this way, as per-device batch size times accumulation steps . Done naively in a distributed job it can be much slower; Accelerate's accumulate() handles it more efficiently .

Checkpointing, in two senses

Activation checkpointing trades compute for memory: activations inside a checkpointed function are not kept for backward but recomputed by running it again . Chen et al. showed an n-layer network can be trained with O(√n) activation memory for the cost of one extra forward pass per mini-batch , cutting a 1,000-layer residual network from 48G to 7G on ImageNet . LLaMA used a selective form, saving expensive activations such as linear-layer outputs and recomputing the rest , and GPipe stores only activations at partition boundaries .

Training checkpoints are snapshots that let a long run resume after a failure. Resuming exactly needs the weights, the optimiser state and the position in the data, so a full checkpoint is about the size of the model states: around 107 GB for LLaMA-7B. A LoRA checkpoint holds only adapters; the LoRA paper's GPT-3 example shrank from 350GB to 35MB .

Distributed collectives

These schemes are built from a few collectives, operations in which every process takes part. all_reduce combines a tensor across all processes so that every process ends with the identical result; the default operation is a sum and it works in place . all_gather gives every process every shard; reduce_scatter combines and then leaves each process one shard of the result, which is how FSDP handles gradients . PyTorch's rule of thumb is the NCCL backend for distributed training on CUDA GPUs and Gloo for CPU; NCCL is the only backend that supports InfiniBand with GPUDirect .

Common misconceptions

"If the weights fit, training fits." Model states take 16 bytes per parameter against 2 for 16-bit weights , eight times as much, before activations.

"Twice the GPUs, twice the speed." Communication and idle time intervene: ZeRO's partitioning adds about 50% communication , pipelines have bubbles , and even Megatron-LM's well-tuned run reached 76% scaling efficiency .

"A faster GPU means faster everything." Memory-bound operations are limited by bandwidth, not arithmetic .

"The estimator says it fits, so it will run." A lower bound is a floor, and the +20% heuristic is only a rule of thumb.

32. What happens when you send a prompt?

Typing "Why is the sky blue?" into a chat window feels like asking a question. Underneath, an application packs your text and some other material into one sequence of token IDs, a model computes scores for what token might come next, a decoding policy picks one, and the loop repeats until something says stop. Every causal language model, chat-tuned or not, only continues a sequence of tokens, so a chat is still a token sequence underneath .

The problem: a conversation is not a model input

A model's input is a list of integers. A conversation is a structured object: who said what, in which order, under which instructions. Something has to turn the second into the first, in exactly the form the model was trained on. Then the model has to be run repeatedly, because it produces one token per pass, and the result has to be turned back into text. The model weights do none of this bookkeeping. The application and the serving system do.

The mechanism, stage by stage

• The request enters the application. Your app holds the conversation as role-tagged messages. The common roles are user (the person), assistant (the model) and system (directives on how the model should behave, usually placed at the start) . Hosted APIs take role-based messages too: OpenAI's API, for instance, has developer, user and assistant roles, and developer instructions are placed ahead of user messages in priority .
• The model input is constructed. The list of messages is converted into one token sequence, often with control tokens that mark who is speaking . Open models ship a chat template describing how they expect chats to be formatted; in Hugging Face Transformers it is applied with apply_chat_template . Formats are not standard: models fine-tuned from the same base can use different control tokens . Mistral-7B-Instruct wraps user turns in [INST] … [/INST], while Zephyr-7B, built on the same base, uses <|user|> and <|assistant|> tokens, and using the wrong ones badly hurts quality . The template usually ends with tokens that open an assistant reply (add_generation_prompt in Transformers), so the model answers instead of continuing your message .
• Text is tokenized. The formatted string becomes integer IDs, because models only process numbers .
• Token IDs are embedded. Each ID selects a row of a learned lookup table, giving one vector per token .
• The vectors pass through the transformer layers, each block refining every position's vector using attention over earlier positions and a feed-forward network.
• The final hidden vector produces logits. A learned linear layer maps it to one raw score per vocabulary entry, and softmax turns those into probabilities .
• A decoding policy selects the next token: the most likely one, or a random draw shaped by temperature and truncation (chapter 34).
• The new token is appended to the sequence.
• The model runs again, now one token longer. It generates the next token from the prompt plus its own previous outputs .
• Generation stops at an end-of-sequence token or a length limit .
• Tokens are decoded for display. In Transformers, generate() returns token IDs and batch_decode() turns them back into text ; output can also be streamed token by token as it is produced .

Where other material enters

The model sees only the one sequence, so everything that should influence the answer has to be placed in it. System or developer instructions usually go first. Conversation history is re-sent with every request. Retrieved documents are pasted in as text (chapter 38). Tool results come back from the application as further messages: the application runs the tool and sends the output back, and the model then continues . The exact layout differs per model and per API, and hosted APIs do not publish the token-level template they apply on their servers .

A worked example

Take the user text alone and GPT-2's real tokenizer. "Why is the sky blue?" becomes six IDs: [5195, 318, 262, 6766, 4171, 30]. GPT-2's default configuration has an embedding width of 768 , so after the lookup the input is a 6 × 768 matrix: 4,608 numbers. At the last position the output layer produces one logit for each of GPT-2's 50,257 vocabulary entries , so every generation step ends with a vector of 50,257 scores from which one token is chosen.

If the reply is 94 tokens long, the model runs 94 forward passes: the first pass over all six prompt tokens yields the first reply token, and each of the remaining 93 passes adds one more. If the template wraps the six tokens in system text and role markers, the prompt might be 26 tokens instead, but the pass count for the reply does not change. Only the length of the sequence each pass looks at does.

Try it

The lab follows one hosted chat request through ten stages, from the JSON your application sends, past a gateway and a chat template, through prefill and the decode loop, to streamed text and a finish reason. Step through it or press play. The gateway, scheduler and batch numbers and the template markers are illustrative, because providers do not publish their stacks; the token IDs are GPT-2's real IDs for the user's sentence.

Real-world usage

Most bugs in this path are formatting bugs, not model bugs. Transformers warns that chat templates already contain the special tokens they need, and that adding them again (for example a second beginning-of-sequence token when tokenizing the formatted text) is often wrong and hurts output . Decoder-only models return the prompt tokens together with the new ones, so code must slice the output . And generate() returns at most 20 new tokens unless the model's configuration says otherwise, which is why the docs recommend always setting max_new_tokens .

Common misconceptions

• "The model receives my message." It receives one long token sequence that includes your message, wrapped in role markers, preceded by instructions and history, and ending with tokens that open the assistant's turn .
• "The model remembers the conversation." The application re-sends the history each turn. Between requests nothing in the weights changes; what the model "knows" about this chat is exactly what is in the current input (chapter 35).
• "All chat models use the same format." Two models from the same base can need different control tokens, and the wrong ones degrade output . Some models, Llama being the Transformers docs' example, have no special tokens before the assistant reply at all, so add_generation_prompt does nothing for them .
• "The model produces words." It produces token IDs. A token may be a word, part of a word, punctuation or a byte fragment; text appears only after detokenization .

Generation settings live outside the weights

Everything that decides how the loop runs (maximum length, sampling on or off, temperature, stop conditions) is configuration, not learned behaviour. In Transformers these settings sit in a GenerationConfig, usually loaded from the model's generation_config.json, with a default strategy used when the model ships none . The same weights can give terse deterministic answers or long varied ones depending on that file and on what the request overrides.

Prefilling the reply

Because the input is just tokens, an application can write the first few tokens of the answer itself and let the model continue. Transformers supports this with continue_final_message, which removes end-of-sequence tokens so the model continues the last message, for example from the opening brace of a JSON object . This is a formatting trick, not a separate capability: the model is still predicting the next token after a prefix, as it always does.

Batching many prompts at once

When several prompts of different lengths run together, they are padded to the same length. For generation, padding goes on the left, because a model is not trained to continue from padding tokens . Production servers avoid much of this waste with the scheduling techniques of chapter 36, but the rule shows how literal the model is about its input: whatever sits immediately before the next position shapes what comes next.

Architecture differences

This chapter follows a decoder-only model, where prompt and reply are one sequence and generation simply extends it. An encoder–decoder model splits the work: the encoder reads the input once and the decoder generates the output while attending to it (chapter 19). The outer loop of tokenize, generate one token at a time, stop and detokenize is the same; what changes is how the input is held. Not every AI system is a language model either: an image classifier makes one forward pass and returns a label, with no decoding loop at all.

What the application owns

Concern — Owner; Message list, history, system text — Application; Chat template and special tokens — Model (shipped with it) or provider; Tokenization and detokenization — Model's tokenizer; Next-token scores — Model weights; Choosing the token, stopping — Decoding configuration; Running tools, retrieval, storing the chat — Application

The model itself contributes one thing in this table: a vector of scores per step. Everything else around it is software that engineers write, configure and debug.

33. Prefill vs decode

Generation has two phases with very different costs. First the model reads the whole prompt in one go. Then it writes the answer one token at a time. Reading is like scanning a page in one look; writing is like copying it out letter by letter, where each letter has to wait for the one before.

The problem: one token at a time

Autoregressive generation predicts one token at a time, and each prediction depends on all the earlier context, which makes it slow . A naïve loop would re-run the entire sequence through the model for every new token, recomputing work it had already done. Two observations fix most of that waste and explain why serving systems treat the phases separately.

The mechanism

The key-value cache. In each attention layer, every token produces a query, a key and a value (chapter 21). Because of the causal mask, a token's representation never changes when later tokens arrive . So its keys and values can be stored and reused: a KV cache keeps the key and value vectors computed for earlier tokens instead of recomputing them at every step . Attention is computed separately in each layer, so there is one cache per layer .

Prefill. The vLLM paper splits a request into two phases. The prompt phase, usually called prefill, takes the whole prompt at once and computes the probability of the first new token . All prompt positions go through every layer in parallel, their keys and values fill the cache, and the last position's logits give the first output token.

Decode. Every later step runs the model over one token: the one just generated . Only that token's query is needed; its new key and value are appended to the cache . This phase cannot be parallelised across steps because each step needs the previous one's output, and it uses matrix–vector rather than matrix–matrix multiplication . It continues until an end-of-sequence token or a length limit .

The mathematics

With a cache, decode step t computes q_t, k_t and v_t for the newest token only, then

`` K_cache ← concat(K_past, k_t) V_cache ← concat(V_past, v_t) output_t = softmax(q_t · K_cacheᵀ / √d_k) · V_cache `

where K_past and V_past hold one row per earlier token, d_k is the head width, and concat` appends a row . The step computes one score per cached token, so its attention cost grows linearly with sequence length. Without a cache each step recomputes all positions, which grows quadratically, while the cache's memory grows linearly with the number of tokens .

A worked example

Use the lab's defaults: an 8-token prompt, the page's 2-layer toy model, and one prefill pass followed by 6 decode passes. Count a "K/V vector" as one token's key and value in one layer, and an "attention score" as one query–key dot product.

• Prefill: 8 tokens × 2 layers = 16 K/V vectors. Under the causal mask, position i scores against i keys, so 1 + 2 + … + 8 = 36 scores per layer, 72 in all.
• Decode with cache: each pass computes 1 token × 2 layers = 2 K/V vectors, so 6 passes add 12, giving 28. The newest token scores against the whole context of 9, 10, …, 14 tokens, which sums to 69 per layer, so the total is 2 × (36 + 69) = 210 scores.
• Decode without cache: each pass re-runs the whole prefix. The K/V total is 2 × (8 + 9 + … + 14) = 2 × 77 = 154. The scores per layer are 36 + 45 + 55 + 66 + 78 + 91 + 105 = 476, so 952 in all.

The output tokens are identical either way. Caching removes repeated arithmetic, and the saving widens as the context grows.

Try it

The lab draws the passes as a timeline: one row for prefill, then one row per decode step, with cells for every token position. Change the prompt length (1–24) and the number of generated tokens (1–16), and untick Use KV cache to see every earlier position recomputed on each pass. The cumulative K/V and attention-score counters are exact for the toy model's 2 layers; real models multiply them by their layer and head counts.

Real-world usage: two latency numbers

Because the phases behave differently, serving systems measure them separately. Time to first token (TTFT) runs from request arrival until the first output token, and time between tokens (TBT) is the gap between consecutive output tokens . DistServe calls the decode-side measure time per output token (TPOT) and uses TTFT for prefill and TPOT for decode as separate latency targets . vLLM exports TTFT, inter-token latency and per-request TPOT as Prometheus histograms .

A hypothetical example of how these combine: with a TTFT of 0.4 s and a steady 0.02 s per output token, a 200-token answer finishes after 0.4 + 199 × 0.02 = 4.38 s and streams at 1 / 0.02 = 50 tokens per second for that user. Lowering TTFT makes the answer start sooner; lowering TPOT makes it flow faster. These are different engineering problems.

Common misconceptions

• "The KV cache changes the answer." It stores values the model would otherwise recompute. With or without it, the outputs are the same; only the work differs.
• "The cache is model memory that persists between chats." It belongs to one running sequence and is discarded when the request ends. It is also an inference-only device: Transformers warns that enabling it during training may cause unexpected errors .
• "Generation speed is about raw compute." Decode is usually limited by how fast weights and cached keys and values can be read from memory, not by arithmetic.

Compute-bound prefill, memory-bound decode

Sarathi-Serve states the contrast directly: prefill is compute-bound because it processes all prompt tokens in parallel, while decode is memory-bound because it processes only one token per request at a time . The vLLM authors say the decode phase badly under-uses the GPU, is memory-bound, and accounts for most of a single request's latency . Each decode step reads every weight in the model to do a small amount of arithmetic for one token. Shazeer traces slow incremental decoding to the memory bandwidth needed to reload the large key and value tensors at every step , and notes that accelerator compute can be two orders of magnitude higher than memory bandwidth .

That asymmetry explains the serving strategy of chapter 36. Decodes gain a lot from batching, because one read of the weights can serve many requests' tokens, while prefills, already busy with arithmetic, gain little .

Why context length costs money

Longer contexts cost in three places. Prefill work grows with the prompt: every token goes through every layer, and in standard attention the number of operations grows quadratically with sequence length . The cache grows linearly with tokens, per layer . And every decode step must read the whole cache, so a long conversation makes every later token slower and holds more memory for as long as the request runs. Chapter 35 puts numbers on the cache size.

Scheduling the two phases

When many requests share a GPU, prefills and decodes compete. Prioritising prefills raises throughput but hurts tail TBT; prioritising decodes does the opposite . A long prefill scheduled ahead of running decodes can pause them. Sarathi-Serve calls this a generation stall and measured stalls in vLLM, as it stood in 2024, lasting over several seconds . Its remedy, chunked prefill, splits a prompt's prefill into roughly equal chunks so new requests join a batch without pausing ongoing decodes . In the paper's 2024 benchmarks this gave up to 2.6x higher serving capacity than vLLM for Mistral-7B .

DistServe goes further and puts prefill and decode on different GPUs. Running both on the same GPUs causes interference and ties both phases to one resource and parallelism plan . Separating them removes the interference, but the KV cache must then move between GPUs, so DistServe places the phases according to cluster bandwidth . It optimises goodput: the highest request rate each GPU can serve while meeting a latency target for a set share of requests, such as 90% .

Caches in code

In Transformers each cache layer holds key and value tensors shaped [batch_size, num_heads, seq_len, head_dim], and seq_len grows by one per generated token . The default DynamicCache grows as generation proceeds; use_cache=False turns caching off . A StaticCache pre-allocates a fixed maximum size instead, which allows decoding to be compiled with torch.compile but spends attention work on masked, unused positions .

Measuring carefully

TTFT includes queueing. Under load, scheduling delays push TTFT up even when the model itself is fast . Inter-token latency also depends on how tokens are packaged: vLLM notes that its inter-token latency approximates TPOT only when each streamed output holds exactly one token . When comparing systems, check which definition each one reports.

34. How tokens are selected

At the end of every forward pass the model hands over a probability for each vocabulary entry. It does not say which token to emit. That choice belongs to a decoding policy, and the same model can produce flat, repetitive text or lively, rambling text depending on it. A decoding strategy decides how the next token is picked from the model's output distribution, and the choice has a large effect on output quality .

The problem: maximise or sample?

Two obvious policies both fail on open-ended text. Holtzman et al. found that maximisation-based decoding such as beam search produces degenerate text: bland, incoherent, or stuck in repetitive loops . Pure sampling from the full distribution has the opposite failure: incoherent text almost unrelated to the context . Their explanation is that the distribution has an unreliable low-probability tail that should be cut off, and that maximisation is the wrong objective for open-ended text . Practical decoding sits between the two.

The mechanism

• Greedy decoding takes the single most likely token at each step. It is the default in Transformers . It suits short outputs where variety does not matter, but tends to repeat itself on longer ones .
• Multinomial sampling draws a token at random in proportion to its probability, so any token with non-zero probability can be chosen .
• Temperature reshapes the distribution before the draw .
• Top-k keeps only the k most likely tokens. Fan et al. used it for story generation with k = 10 .
• Top-p (nucleus) sampling keeps the smallest set of most likely tokens whose cumulative probability reaches p, then rescales their probabilities to sum to 1 .
• Repetition penalties lower the scores of tokens already generated (deep dive).
• Stopping happens at an end-of-sequence token or a length limit .

The mathematics

For logits z_1 … z_V (one per vocabulary entry, V entries) and temperature T > 0:

`` p_i = softmax(z / T)_i = exp(z_i / T) / Σ_j exp(z_j / T) `

T = 1 leaves the model's distribution unchanged. 0 < T < 1 sharpens it toward likely tokens and moves mass out of the tail . As T approaches 0 the distribution approaches a greedy choice; as T grows it flattens toward uniform .

Do not plug T = 0 into the formula: that divides by zero. Greedy decoding is a separate rule, next = argmax_i z_i, the limit the formula approaches. Libraries treat it as a different mode. In Transformers, do_sample chooses between sampling and greedy decoding , and temperature has an effect only when sampling is on . This page's lab accepts temperature 0 and switches to argmax.

Truncation then keeps a set S of tokens and renormalises: p'_i = p_i / Σ_{j∈S} p_j for i in S, and 0 otherwise.

A worked example by hand

Five candidate tokens with logits z = [2.0, 1.0, 0.5, 0.0, −1.0] (invented for the example, not taken from any real model).

T = 1. The exponentials are 7.3891, 2.7183, 1.6487, 1.0000 and 0.3679, which sum to 13.1239. Dividing gives

token — z — exp(z) — p at T = 1 — cumulative; 1 — 2.0 — 7.3891 — 0.5630 — 0.5630; 2 — 1.0 — 2.7183 — 0.2071 — 0.7701; 3 — 0.5 — 1.6487 — 0.1256 — 0.8958; 4 — 0.0 — 1.0000 — 0.0762 — 0.9720; 5 — −1.0 — 0.3679 — 0.0280 — 1.0000

T = 0.5. Divide the logits by 0.5 to get [4, 2, 1, 0, −2]. The exponentials are 54.5982, 7.3891, 2.7183, 1.0000 and 0.1353, which sum to 65.8408, so p = [0.8292, 0.1122, 0.0413, 0.0152, 0.0021]. The top token rises from 56% to 83%.

T = 2. The logits become [1, 0.5, 0.25, 0, −0.5], with exponentials summing to 7.2576, so p = [0.3745, 0.2272, 0.1769, 0.1378, 0.0836]. The distribution flattens, and the entropy rises from 1.74 bits at T = 1 to 2.15 bits (0.88 bits at T = 0.5).

Top-k = 3 at T = 1. Keep tokens 1–3, whose mass is 0.5630 + 0.2071 + 0.1256 = 0.8958 (0.89577 before rounding). Renormalise: 0.5630 / 0.8958 = 0.6285, 0.2071 / 0.8958 = 0.2312, 0.1256 / 0.8958 = 0.1402. The three values sum to 0.9999 after rounding to four places, and to 1 exactly before rounding.

Top-p = 0.8 at T = 1. The cumulative column first reaches 0.8 at token 3 (0.8958), so the nucleus is the same three tokens and gives the same renormalised result as top-k = 3.

Top-p = 0.9 at T = 1. After three tokens the total is only 0.8958, below 0.9, so token 4 joins (0.9720). Renormalising by 0.9720 gives [0.5793, 0.2131, 0.1293, 0.0784].

Top-p = 0.9 at T = 0.5. Now 0.8292 + 0.1122 = 0.9415 already exceeds 0.9, so the nucleus has just two tokens: [0.8808, 0.1192]. The same p kept four tokens at T = 1 and two at T = 0.5: the nucleus adapts to how peaked the distribution is, which a fixed k cannot do .

Try it

The lab runs the page's toy transformer on one of four prompts and shows its real next-token distribution. Move temperature (0–2.5, where 0 means greedy), top-k (0 = off, up to 20) and top-p (0.05–1). The left chart shows the top 12 tokens after temperature, with truncated ones dimmed. The right chart shows what survives, renormalised, and the sum of the final probabilities is printed to nine decimal places so you can check it is 1. When top-k and top-p are both set, the lab keeps at most k tokens and stops earlier if their cumulative probability reaches p. The seed fixes 200 draws, and the last button generates the rest of the answer with your settings. The probabilities are measured from the toy model, a 2-block network trained on a tiny corpus, not from a production LLM.

Real-world usage

In Transformers, sampling is turned on with do_sample=True and num_beams=1 . When a model's generation_config.json` does not set them, the defaults are temperature 1.0 , top_k 50 and top_p 1.0, which means no nucleus filtering . The library's own guidance is temperatures above 0.8 for creative tasks and below 0.4 for tasks that need "thinking" . Lowering temperature improves quality but reduces diversity .

Common misconceptions

• "Temperature 0 means T = 0 in the formula." It means a different rule, argmax, because the formula is undefined at 0.
• "Low temperature makes the model more accurate." It makes the model pick its own favourite more often. If the favourite is wrong, greedy decoding is reliably wrong.
• "Top-p 0.9 keeps 90% of the tokens." It keeps the fewest top tokens whose probabilities add up to at least 0.9. That can be one token or many: Holtzman et al. found the nucleus typically ranges from one to a thousand candidates .
• "A fixed seed guarantees the same output." Only in an identical environment (see below).

Repetition penalties

CTRL's penalised sampling discourages repetition by discounting the scores of tokens already generated, and it applies only at generation time, not during training . CTRL reports that greedy decoding with a penalty of about 1.2 balances truthful generation against repetition, and a penalty of 1 means no change . The authors add that the method works only if the model has learned a sufficiently reliable distribution . In Transformers, repetition_penalty=1.0 means no penalty , and the docs suggest raising it above 1.0 if the model repeats itself . A blunter tool, no_repeat_ngram_size, bans any n-gram of that size from appearing twice .

As an illustrative sketch of a discount (not any library's exact code), divide the positive logit 2.0 of an already-used token 1 by 1.2. It becomes 1.667, and at T = 1 that token's probability falls from 0.5630 to 0.4800, with the other four tokens sharing the difference.

Beam search and min-p

Beam search keeps several candidate sequences at each step and finally picks the one with the highest overall probability, so it can choose a sequence whose first tokens were less likely . It works best for input-grounded tasks such as captioning or speech recognition, and num_beams=1 is the same as greedy search . Min-p is another truncation rule: it keeps tokens whose probability is at least min_p times that of the most likely token, and the Transformers docs describe typical values of 0.01–0.2 as about as selective as top_p of 0.99–0.8 .

Stopping

A sequence ends when the model emits its end-of-sequence token or a length limit is reached . Because generate() defaults to at most 20 new tokens when the model does not configure more, forgetting max_new_tokens is a classic cause of truncated answers . Applications often add their own stop rules on top, such as checking the decoded text for a marker string. Stopping is part of the decoding policy, not something the weights enforce.

Seeds and reproducibility

A seeded pseudo-random generator makes draws repeatable, which is how the lab's 200 draws come out the same every time. Real frameworks promise less. PyTorch says completely reproducible results are not guaranteed across releases, commits or platforms, and CPU and GPU runs may differ even with the same seeds . A seed gives the same sequence in the same environment only when all other sources of nondeterminism are removed , and deterministic algorithms are often slower . On a hosted service you control neither the hardware nor the serving software, and providers do not publish their serving stacks . A tiny numeric difference that flips one sampled token can change everything generated after it.

Production policies are more than three knobs

Real deployments may combine several of these rules, add penalties, constrain output to a grammar or schema, or run several samples and pick among them (chapter 37). This page applies temperature first, then truncation, then renormalisation; other pipelines may order these differently. Read a library's documentation rather than assuming its pipeline matches this one.

35. Context windows and the KV cache

A model's context is everything it can see when it computes the next token: instructions, conversation, documents, and the answer so far. It works like a desk of fixed size. Whatever is on the desk can be used. Whatever falls off is gone for this request, however important it was.

The problem: a fixed budget with growing costs

Each model has a maximum context length set by its architecture and training. Mistral 7B, for example, lists a context length of 8,192 tokens and an attention window of 4,096 . Prompt and response share that budget, because the reply's tokens are appended to the same sequence (chapter 32). Using the budget is not free either. In standard attention the number of operations grows quadratically with sequence length and memory grows linearly with the number of tokens, which raises latency and lowers throughput .

The mechanism

During generation, attention at each new position compares its query with the keys of all earlier positions in the window and averages their values. The KV cache holds those keys and values per layer so they are not recomputed (chapter 33) . The cache is position-specific: the same token's keys and values differ depending on where it sits and what came before it . That is why caches cannot simply be shared between unrelated prompts; chapter 36 covers the shared-prefix case where they can.

The mathematics: how big is the cache?

For a model with L layers, H_kv key–value heads of width d_head, stored at b bytes per number:

`` bytes per token = 2 × L × H_kv × d_head × b bytes per request = bytes per token × tokens in the sequence total = bytes per request × requests in the batch `

The 2 counts one key and one value. In standard multi-head attention H_kv × d_head equals the model width d. The formula matches the per-layer tensor shape [batch_size, num_heads, seq_len, head_dim] used by Transformers, with one such tensor for keys and one for values .

A worked example: OPT-13B

The vLLM paper works this out for the 13-billion-parameter OPT model: model width 5,120, 40 layers, FP16 (2 bytes) .

` 2 × 40 × 5,120 × 2 B = 819,200 B per token (800 KiB; the paper rounds to "800 KB") `

OPT can generate sequences up to 2,048 tokens, so one full-length request needs

` 819,200 B × 2,048 = 1,677,721,600 B ≈ 1.6 GB ``

which is the paper's 1.6 GB per request . Now place the model on a 40 GB A100. In the paper's example about 65% of memory holds the fixed weights and close to 30% holds per-request state, mostly the KV cache . Thirty per cent of 40 GB is about 12 GB, or 12 × 10⁹ / 819,200 ≈ 14,600 tokens: about seven full-length requests at once. Even a GPU given over entirely to KV cache would fit only a few tens of such requests . Memory, not arithmetic, limits how many users one GPU serves.

Two smaller checks. GPT-2's default configuration has 12 layers and width 768 , so at 2 bytes it needs 2 × 12 × 768 × 2 = 36,864 B (36 KiB) per token, or 36 MiB across its 1,024-position window . The page's toy model has 2 blocks, 4 heads of width 12, and stores numbers as 4-byte floats: 2 × 2 × 4 × 12 × 4 = 768 B per token, and 43,008 B for its full 56-token window.

Try it

The lab gives a deliberately tiny window (128–1,024 tokens, default 256) and a scripted conversation about the sky: user and assistant turns, a retrieved document, follow-up questions. Press Add next message and watch the budget bar fill with the system prompt, each message and the space reserved for the answer. Tokens are counted with GPT-2's real tokenizer plus 4 per message for template markers, which is a stand-in, because each model counts differently. When the window is full, choose Drop oldest (the oldest messages disappear and are struck through) or Reject request (the request is refused). Either way, nothing dropped is kept anywhere else.

Context is not learning

The lab's last note matters most. GPT-3 performed tasks from instructions and examples given only as text in the prompt, with no gradient updates or fine-tuning . Whatever a model picks up from its context lasts for that request. The parameters, which hold what it learned in training, do not change when you chat with it. Clear the context and the "knowledge" from it is gone.

Common misconceptions

• "A bigger window means the model uses everything in it." Liu et al. found a U-shaped curve: models use relevant information best when it sits at the very start or end of the context , and performance drops significantly when it is in the middle . In one setting GPT-3.5-Turbo did worse with the answer document in the middle than with no documents at all , and models with longer windows were not necessarily better at using their context .
• "The model remembers earlier chats." Only if the application puts them back into the context. Weights do not change during inference .
• "The context window is the same as the KV cache." The window is a limit on sequence length. The cache is the memory a running request actually uses, which can be smaller than the window (sliding-window layers) and which grows with the sequence and the batch.
• "We know how closed models manage long context." The primary sources checked publish no architecture, context-handling or KV-cache details for closed hosted models .

Fewer key–value heads: MQA and GQA

The cache formula has H_kv in it, and architectures attack exactly that term. Multi-query attention (MQA) shares one set of keys and values across all heads, which greatly shrinks the cache and the bandwidth needed for decoding at a minor cost in quality , though it can reduce quality and make training unstable . Grouped-query attention (GQA) uses more than one but fewer key–value heads than query heads, and gets close to multi-head quality at near-MQA speed . GQA with one group is MQA; GQA with as many groups as heads is standard multi-head attention . Existing multi-head checkpoints can be uptrained to GQA with 5% of the original pre-training compute .

Mistral 7B uses 32 query heads but 8 key–value heads . Compared with a full multi-head version of the same layer shape, its cache per token is 8 / 32 = one quarter the size. The motivation is bandwidth: decoder inference is bottlenecked by loading the weights and all cached keys and values at every step .

Sliding-window attention

Sliding-window attention (SWA) lets each token attend to at most W tokens from the previous layer. Tokens outside the window still affect the prediction indirectly, through earlier layers . Information moves forward up to W tokens per layer, so after k layers it can reach k × W tokens back; Mistral reports a theoretical span of about 131K tokens with W = 4,096 . A fixed span allows a rolling buffer cache of size W: the keys and values for position i go in slot i mod W, overwriting old entries, so the cache stops growing . On a 32K-token sequence that is 32,768 / 4,096 = 8 times less cache memory, and the Mistral paper reports no loss of quality . Mistral 7B combines GQA and SWA . In Transformers the default cache stops growing on sliding-window layers once it reaches the window size . Longformer takes a related route, combining local windowed attention with task-specific global attention so the cost scales linearly with sequence length .

Truncation and compression

When a conversation outgrows the window, something has to give. Applications drop the oldest turns (as in the lab), keep the system prompt and the most recent turns, summarise older turns into a shorter message, retrieve only the passages relevant to the current question (chapter 38), or refuse the request. Each is a form of prompt compression: fewer tokens carrying, the developer hopes, the information that matters. Each also loses something, and the model cannot tell you what it no longer sees. Combined with the lost-in-the-middle effect above, a large window does not remove the need to choose carefully what goes in it.

When the cache runs out

On a server the cache budget is shared by every running request. When vLLM runs out of KV cache space it preempts requests to free memory and recomputes them later; its docs suggest raising gpu_memory_utilization or tensor_parallel_size to get more KV cache memory . Chapter 36 describes how the cache is laid out in blocks so that little of it is wasted.

36. Production inference infrastructure

Running the toy model in your browser is one forward pass at a time, for one reader, with nothing else competing. A production service runs a model that may be thousands of times larger, for many users at once, on expensive accelerators that must stay busy. The model maths is the same. Almost everything around it is different.

The problem: many users, one expensive machine

Generative models run once per output token, so one request needs many model iterations . The obvious way to share a GPU, batching whole requests, fits badly: requests that finish early cannot return, and new arrivals wait until the whole batch is done . Decode is memory-bound and gains from batching , so the goal is to keep as many useful tokens in each iteration as memory allows, without letting any single user's latency explode.

The mechanism: the request path

`` client → API gateway → authentication / rate limits → request scheduler → model workers → streamed response `

The gateway terminates connections, authenticates and enforces quotas. The scheduler decides which requests run in the next model iteration. The workers hold the weights and the KV cache on accelerators and run prefill and decode. The response streams back token by token. Hosted providers do not publish their serving stacks, so this is the general shape, not a description of any particular provider .

Continuous batching. Orca introduced iteration-level scheduling, now usually called continuous batching: the scheduler runs one model iteration at a time over the batch, so the batch can change between iterations . Finished requests leave after each iteration and new ones join .

Paged KV memory. vLLM's PagedAttention borrows paging from operating systems. The KV cache is split into blocks, each holding the keys and values of a fixed number of tokens, and the blocks need not be contiguous . A block table maps each request's logical blocks to physical ones, as a page table does for a process . The paper's default block was 16 tokens . Earlier systems reserved contiguous memory for each request's maximum length , and the vLLM authors measured only 20.4–38.2% of their KV memory holding real token state .

The mathematics: utilisation and latency

For a schedule of N requests over T iterations with s slots:

` utilisation = (slot-iterations doing prefill or decode) / (T × s) time in system_r = finish_r − arrival_r + 1 (in iterations) `

Packing more requests into each iteration raises throughput (tokens per second), but each iteration then takes longer, so per-user TPOT can rise.

A worked example: the lab's eight requests

Eight requests, A to H, arrive at iterations 0, 0, 1, 2, 3, 4, 6 and 7 and need 3, 9, 4, 2, 6, 3, 5 and 2 output tokens. Each needs one prefill iteration plus one iteration per token: 34 + 8 = 42 slot-iterations of real work. With 3 slots:

• Static batching. A, B and C would start together, but C has not arrived at iteration 0, so the first batch is A and B. B needs 10 iterations, so the batch runs until iteration 9. A finished at iteration 3, and its slot idles for six iterations while C, D and E queue. Everything completes after 23 iterations: utilisation 42 / (23 × 3) = 60.9%, mean time in system 12.5 iterations.
• Continuous batching. C takes the free slot at iteration 1 and D enters as soon as A leaves. Everything completes after 16 iterations: utilisation 42 / (16 × 3) = 87.5%, mean time in system 7.25 iterations.

Same work, same hardware: about 30% fewer iterations and 42% less average waiting.

Try it

The lab draws one row per request and one column per iteration: prefill, decode, waiting in the queue, idle. Switch between Static and Continuous, and change the number of batch slots (1–6), which stands in for KV-cache memory. The counters show iterations to finish, mean time in system and slot utilisation, with the other policy's figures beside them. With one slot the policies are identical (42 iterations); with 6 slots continuous batching needs 12 iterations against static's 17.

Real-world usage

The vLLM paper reports 2–4x higher throughput than FasterTransformer and Orca at the same latency in its 2023 benchmarks, with larger gains for longer sequences, larger models and complex decoding . In vLLM V1, chunked prefill is on by default where possible. The scheduler batches all pending decodes first, then fills the remaining max_num_batched_tokens` budget with prefills, chunking any that do not fit . Mixing compute-bound prefills with memory-bound decodes uses the GPU better, and putting decodes first protects inter-token latency .

Common misconceptions

• "Everyone uses the same engine." Engines differ in target hardware and design, and hosted providers do not say what they run .
• "Optimisations change the answers." Some do and some do not. Prefix caching does not change outputs, according to vLLM's docs , and speculative decoding is designed to keep the output distribution . Quantization changes the numbers the model computes with, and its quality impact has to be measured.

Prefix caching

Many requests start with the same tokens, such as a long system prompt. Prefix caching keeps the KV blocks of processed requests and reuses them when a new request starts with the same prefix, skipping redundant prompt computation . vLLM identifies each cached block by a hash of the parent block's hash, the block's own tokens and any extra keys, and only full blocks are cached . Since v0.11 the default hash is SHA-256, which addresses collision risks in earlier versions . The parent-hash chain encodes the position-dependence of chapter 35: the same tokens after a different prefix produce a different hash and different keys and values . Blocks can also be shared between one prompt's parallel samples, with copy-on-write at block granularity .

Memory pressure, preemption and cancellation

When memory runs out, something must wait. vLLM, as of the 2023 paper, schedules first-come-first-served and preempts the newest requests first . A preempted request can be recovered by swapping its blocks to CPU memory or by recomputation, which is cheaper than it sounds because the generated tokens can be rerun with the prompt as a single prefill pass . Cancellation follows the same accounting from the other side: when a client disconnects, the server should stop decoding and free that request's blocks at once, or it keeps spending iterations on tokens nobody will read.

Quantization

Weights stored in fewer bits take less memory and less bandwidth. At 2 bytes per parameter a 7-billion-parameter model's weights take 14 GB; at 4 bits they take 3.5 GB, before scale factors and other overhead. LLM.int8() runs feed-forward and attention projections with 8-bit matrix multiplication, halving inference memory while keeping full-precision performance . It isolates rare large outlier dimensions into 16-bit multiplication, while more than 99.9% of values are still multiplied in 8-bit . GPTQ is a one-shot post-training method that quantizes 175B-parameter models to 3 or 4 bits per weight in about four GPU hours with negligible accuracy loss , which in 2022 made it possible to run such a model on a single GPU for the first time .

Why this speeds up decoding: generation is memory-bound and dominated by reading weights, and 4-bit weights raise arithmetic intensity to about 4 FLOPs per byte .

Parallelism and the cluster

A model too large for one GPU is split. Tensor parallelism shards each layer's parameters across GPUs. It is the most common way to serve a large model on one node, and it frees per-GPU memory for KV cache . In the 14 GB example, two-way tensor parallelism leaves about 7 GB of weights per GPU, and the rest goes to cache. Above a single node, model servers are ordinary long-running services that need placement on GPU machines, health checks, rolling upgrades and autoscaling. A cluster scheduler such as Kubernetes can treat each replica as a workload to place on a GPU node, health-check and restart. A new replica must load many gigabytes of weights before it is ready.

Faster decoding without changing outputs

Speculative decoding samples from a large model faster, without changing its outputs, by computing several tokens in parallel . A small draft model proposes several tokens, and the large target model checks them all in one pass, accepting or rejecting them without changing the output distribution . Each target run yields at least one token, so it never needs more serial target runs than ordinary decoding . The authors report a 2X–3X speed-up on T5-XXL with identical outputs . FlashAttention cuts memory traffic inside attention itself: it is exact, and tiles the computation to reduce reads and writes between GPU high-bandwidth memory and on-chip SRAM .

The engines

Engine — Role; vLLM — Serving engine with PagedAttention, continuous batching, chunked prefill, prefix caching, many quantization formats, speculative decoding and disaggregated prefill/decode ; streaming and an OpenAI-compatible server; TensorRT-LLM — NVIDIA's open-source library for optimising inference on NVIDIA GPUs, with custom kernels and runtime optimisations such as prefill–decode disaggregation and speculative decoding; llama.cpp — Plain C/C++ inference with no dependencies, for a wide range of hardware, locally and in the cloud ; 1.5- to 8-bit integer quantization and CPU+GPU hybrid inference for models larger than VRAM ; an OpenAI-compatible server

llama.cpp loads models in GGUF, a binary format that stores the tensors together with standard metadata . None of these engines is "the" way models are served: a laptop, a single-GPU startup and a large provider make different trade-offs.

Observability, failures and cost

The numbers worth watching are the ones from chapter 33: TTFT, inter-token latency and TPOT, exported by vLLM as Prometheus histograms , plus queue depth, KV-cache usage, preemptions and tokens per second per GPU. A worker crash loses every running request's cache, so those requests restart from prefill, and naïve automatic retries during an overload add load at the worst moment. Cost per token falls as batches grow, until latency targets break. DistServe frames this as goodput: the highest request rate per GPU that still meets the latency target .

37. What does AI reasoning mean?

"Reasoning" is one of the most loaded words in AI. In this chapter it means something observable and testable: a system's ability to solve problems that need several dependent steps, such as a maths word problem, and to get the final answer right more often. It does not mean a claim about private inner experience. The authors of the paper that popularised chain-of-thought prompting say this directly: producing text that resembles human reasoning does not answer whether the network is actually "reasoning", and they leave that as an open question .

The problem: a fixed budget per token

A decoder-only language model produces one token per forward pass, and every forward pass does the same amount of arithmetic whatever the question. Asked for an answer immediately, the model must squeeze a multi-step calculation into the few tokens of the answer itself. Writing the intermediate steps out changes that: each generated step is another forward pass, so a problem that needs more steps gets more computation . The intermediate text also becomes context that later tokens can attend to, so the result of step 1 is available, as tokens, when step 2 is computed.

Mechanisms

Technique — What happens — Where the extra work lives; Direct answer — The answer is decoded straight after the question — Nowhere: one pass per answer token; Chain of thought — The model writes intermediate steps before its final answer — More generated tokens; Self-consistency — Sample several reasoning paths and keep the most common final answer — Several full generations plus a vote; Verification and search — Generate many candidates; a trained verifier ranks them and the best is returned — Candidates plus a second model; Sequential revision — Each new attempt is conditioned on the previous ones — A chain of generations; Tool-assisted computation — The model writes a program or calculator call; an interpreter computes the result — Outside the model

Chain of thought can be elicited without any fine-tuning, simply by putting worked examples that include reasoning steps into a few-shot prompt ; the original experiments used eight hand-written exemplars . That paper also found that the gains appeared only in very large models (around 100B parameters), while smaller models wrote fluent but illogical chains and did worse .

All of the rows below the first spend more computation at inference time rather than at training time. Snell et al. study two broad ways to scale this test-time compute: searching against process-based verifier reward models, and letting the model revise its own response adaptively . Which works best depends on how hard the prompt is .

Reasoning-oriented training is a separate lever: the weights are trained so that useful intermediate steps become more likely. Published examples in this chapter's sources include training verifiers to judge solutions and training models to emit calculator annotations . None of these papers say how specific commercial "reasoning" models are trained or how they generate or hide their internal traces, and this article makes no claim about that .

The mathematics of voting

Self-consistency's simplest aggregation is a majority vote. If path i ends in answer a_i, the returned answer is

`` â = argmax_a Σ_{i=1..n} 1[a_i = a] `

where n is the number of sampled paths and 1[·] is 1 when its condition holds and 0 otherwise . To see when this helps, assume (unrealistically) that every path is independently correct with probability p, and that a wrong path picks one of k wrong answers uniformly at random. With only one possible wrong answer (k = 1), the vote is right when more than half of the paths are right:

` P(majority correct) = Σ_{j > n/2} C(n, j) · p^j · (1 − p)^(n − j) `

with C(n, j) the binomial coefficient.

Worked example

Take n = 3 and p = 0.6 with a single wrong answer. The vote is right when 2 or 3 paths are right:

• P(3 right) = 0.6³ = 0.216
• P(exactly 2 right) = 3 × 0.6² × 0.4 = 3 × 0.36 × 0.4 = 0.432
• Total = 0.648, up from 0.6 for a single path.

Now let wrong paths spread over k = 3 wrong answers, as in the lab. Two or three right paths still win (0.648). With exactly one right path (probability 3 × 0.6 × 0.4² = 0.288), the two wrong paths differ with probability 2/3, giving a three-way tie that a random tie-break resolves correctly one time in three: 0.288 × 2/3 × 1/3 = 0.064. Total: 0.712. Spreading the errors out helped the vote, because no single wrong answer could gather a majority.

The same arithmetic shows the failure mode. With a single wrong answer and p = 0.45, three votes give 3 × 0.45² × 0.55 + 0.45³ = 0.334125 + 0.091125 = 0.42525, which is worse than one path. Voting amplifies whichever answer is most likely, right or wrong.

One word problem, three ways

A bakery bakes 9 trays of 8 rolls. It sells 23 in the morning, and 7 are burnt. How many rolls are left?

• Immediate guess. A quick answer of 49 is the kind of error a direct answer can make: it subtracts the sales and forgets the burnt rolls. Nothing in the output shows where it went wrong.
• Structured decomposition. 9 × 8 = 72; 72 − 23 = 49; 49 − 7 = 42. Each step is small and checkable, but a model that decomposes correctly can still slip on the arithmetic inside a step .
• Trusted tool. The model writes the expression 9 * 8 - 23 - 7` and an interpreter returns 42. This is the program-aided approach: the model handles the translation from words to a program and the runtime does the computation .

Try it

The lab below simulates self-consistency on that problem (correct answer 42, wrong answers 40, 44 and 38). Set the probability that one path is correct and the number of samples, resample the paths, and read the Monte-Carlo accuracy curve for the majority vote. No language model runs here: each path is a random draw under the independence model above, which is exactly why the gains it shows are an upper bound. Drop p below 0.25 and a wrong answer becomes the single most likely outcome, and more votes make things worse.

Where it appears in practice

GSM8K, 8.5K grade-school maths word problems built to diagnose multi-step reasoning failures, is the usual testbed . On it, historically, self-consistency added 17.9 points over chain-of-thought prompting and program-aided Codex beat PaLM-540B with chain-of-thought by 15 points : results for specific 2022 models, not general laws.

Common misconceptions

"The written chain of thought is a record of what the model computed." Not necessarily. Chain-of-thought explanations can systematically misrepresent the real reason for a prediction . In one test, few-shot answer options were reordered so the correct option was always "(A)"; the models were biased by this and systematically failed to mention it in their explanations . When pushed towards wrong answers, models wrote explanations that rationalised them . Reasoning can be post-hoc, written after the conclusion is effectively fixed . Plausible but misleading explanations risk raising trust without improving safety .

"Writing steps guarantees a correct answer." There is no guarantee that a reasoning path is correct; it can lead to right or wrong answers .

"More samples always help." Only when the correct answer is the most likely single outcome, and only as far as samples are independent. Samples from one model on one prompt share weights, prompt and often the same misconception, so the lab's independence assumption is the best case.

"A reasoning model thinks like a person." Better problem-solving scores are an observation about outputs. They do not establish human-like or conscious reasoning .

Testing faithfulness

Lanham et al. test whether a model's answer actually depends on its chain of thought by intervening on it: inserting mistakes, paraphrasing it, truncating it or replacing it with filler, then checking whether the answer changes. Models varied widely by task in how much they relied on the chain . On most tasks they studied, larger and more capable models produced less faithful reasoning . They also found the performance boost did not seem to come from the extra test-time compute alone, or from information hidden in the particular wording . The practical conclusion is to treat generated reasoning as an output to evaluate, not as a window into the network.

Verifiers, search and their limits

A verifier is a model trained to judge whether a candidate solution is correct. In the GSM8K paper it outputs the probability that a solution is right, and at test time 100 completions were sampled and the highest-scoring one returned . Verification gave roughly the boost of a 30× larger model over a fine-tuning baseline . But search has a ceiling: with enough candidates, the search starts finding adversarial solutions that fool the verifier . Any learned judge is itself a model with errors, and optimising hard against it finds them.

Allocation matters too. A compute-optimal per-prompt strategy was more than 4× more efficient than a best-of-N baseline , and in a FLOPs-matched comparison a small model with extra test-time compute could beat a 14× larger model, but only on problems where the small model already had some success . Test-time compute and pretraining compute are not interchangeable one-for-one: on the hardest questions, more pretraining was more effective .

Architecture differences

None of these techniques changes the network. Chain of thought, self-consistency, best-of-N and revision run on the same model; they differ only in what is decoded and how the application combines the outputs, and inference does not update weights . A "reasoning model" may differ in how it was trained, but its outputs are still sampled tokens. Verifier-based search adds a second model, and tool use moves part of the computation out of the network: in the GSM8K work the calculator overrode sampling for annotated spans .

Engineering notes

• Cost. Self-consistency multiplies generation cost by the number of paths; its authors suggest starting with 5 or 10 . It needs no labels, extra models or fine-tuning .
• Answer extraction. "42", "42 rolls" and "forty-two" must be normalised before votes are counted.
• Exact tools for exact work. Hand arithmetic, dates and lookups to code that is correct by construction.
• Do not show hidden traces as fact. Present intermediate text as generated text that may not reflect the computation, not as an audit log.

38. Retrieval-augmented generation

A language model answering from its weights is taking a closed-book exam. Retrieval-augmented generation (RAG) makes it open-book: before the model generates, the application searches a document collection, pastes the most relevant passages into the prompt, and asks the model to answer from them.

The problem

Knowledge held only in a model's weights (parametric memory) is hard to expand or revise, gives little insight into where an answer came from, and can produce hallucinations . Providing provenance and updating world knowledge were the open problems the original RAG paper set out to address . In practice this shows up in four ways:

• Recently updated facts. The weights are fixed after training and do not change at inference ; a model cannot know what happened after its training data was collected.
• Private documents such as contracts or medical records, which were never in the training data and must not be.
• Company-specific data: product manuals, policies, tickets, prices.
• Source-grounded responses, where an answer is only useful if it points to the passage that supports it.

Because the knowledge lives in an index rather than in the weights, it can be updated by changing the index, while a parametric-only model would need more training .

The pipeline

Stage — What it does; Documents — The source collection: web pages, PDFs, wiki pages, database rows; Parsing — Extract clean text and structure (headings, tables) from each format; Chunking — Split documents into pieces small enough to retrieve individually and fit in the context window; Embeddings — For dense retrieval, map each chunk to a vector with an encoder; Indexing — Store vectors (dense) or an inverted index of terms (sparse), with metadata; Query — Turn the user's question into the same representation; Retrieval — Return the top-k chunks by similarity, optionally filtered by metadata; Reranking — Re-score the candidates with a more expensive model; Context assembly — Order, truncate and format the chosen chunks with their sources; Model — Generate an answer conditioned on question plus context; Answer — Return text, ideally with citations to the chunks used

Sparse retrieval (TF-IDF, BM25) matches keywords through an inverted index; question and passage are treated as very high-dimensional sparse vectors with one weighted dimension per term . Dense retrieval maps text to learned vectors, so a synonym or paraphrase with entirely different tokens can still land close to the query . Dense Passage Retrieval (DPR) uses one BERT encoder for questions and another for passages, and scores relevance with a dot product, sim(q, p) = E_Q(q)ᵀ E_P(p) ; passages are encoded once, offline, and only the question is encoded at query time . Vector similarity is usually a dot product or cosine similarity; finding the top-k by inner product is Maximum Inner Product Search (MIPS) .

The two fail differently. BM25 is sensitive to highly selective keywords, while dense retrieval captures meaning but may miss rare, salient phrases such as product codes or names . Hybrid search combines them; the DPR authors took the top 2,000 passages from each, and reranked the union by BM25(q, p) + λ · sim(q, p) with λ = 1.1 .

Metadata filters restrict candidates by attributes stored with each chunk (product, date, access rights). They are exact constraints, not similarity, and the right place to enforce who may see what.

Reranking follows a retrieve-then-rerank pattern: fetch many candidates (for example a thousand) cheaply with something like BM25, then re-score them with a more expensive model . A cross-encoder reranker reads the query and passage together, scores each passage independently and sorts by that score .

The mathematics of BM25

For a query with terms t, the BM25 score of chunk D is

`` score(D, Q) = Σ_{t ∈ Q} IDF(t) · f(t, D) · (k1 + 1) / ( f(t, D) + k1 · (1 − b + b · |D| / avgdl) )

IDF(t) = ln( 1 + (N − n(t) + 0.5) / (n(t) + 0.5) ) ``

• f(t, D): how many times term t occurs in chunk D
• |D|: the chunk's length in terms; avgdl: the average length over all chunks
• N: the number of chunks; n(t): how many chunks contain t
• k1: term-frequency saturation; b: how strongly length normalises the score

Elasticsearch uses BM25 as its default similarity with k1 = 1.2 and b = 0.75 , and the lab uses the same values with the IDF form above, which is never negative. These are defaults, not universal constants: DPR's Lucene baseline tuned them to b = 0.4 and k1 = 0.9 on development sets .

Worked example

The lab's knowledge base has 12 fictional countries with three facts each. With one sentence per chunk there are N = 36 chunks. After stop words are removed, a capital sentence such as "the capital of veloria is mardon ." has 3 terms (capital, veloria, mardon), an export sentence 3, and a river sentence such as "the main river of veloria is vara ." 4 (main, river, veloria, vara). So avgdl = (12 × 3 + 12 × 3 + 12 × 4) / 36 = 120 / 36 ≈ 3.333.

The question "what is the capital of veloria ?" keeps two terms: capital and veloria. Score the chunk "the capital of veloria is mardon .":

• IDF(capital): n = 12, so ln(1 + 24.5 / 12.5) = ln(2.96) ≈ 1.0852.
• IDF(veloria): n = 3, so ln(1 + 33.5 / 3.5) = ln(10.5714) ≈ 2.3582.
• Length factor: 1 − 0.75 + 0.75 × 3 / 3.333 = 0.25 + 0.675 = 0.925, times k1 gives 1.11.
• Each term has f = 1, so its weight is IDF × 2.2 / (1 + 1.11) = IDF × 1.0427.
• capital: 1.0852 × 1.0427 ≈ 1.132; veloria: 2.3582 × 1.0427 ≈ 2.459. Score ≈ 3.590.

"veloria exports salt ." matches only veloria (≈ 2.459); the longer river sentence also matches only veloria but is penalised for length (≈ 2.180); another country's capital sentence matches only capital (≈ 1.132). The rare term carries the ranking, which is what IDF is for. The lab shows these same figures.

Try it

The lab runs the whole pipeline on the toy model: the documents, chunking with a chosen number of sentences per chunk and overlap, the inverted index and its document frequencies, top-k retrieval with each term's contribution to the BM25 score, the assembled prompt with its token count against the toy model's 56-token window, and the generated answer. With the defaults (one sentence per chunk, top-2), the prompt is 21 tokens. Edit a fact in the knowledge base and ask again: no weights change, only the retrieved text. When the toy copies the new value, that is RAG updating knowledge without retraining; when it does not, the lab says so, because a tiny model can fail even after retrieval succeeds. Turn retrieval off and the model must guess about a country it has never seen described.

Real-world usage

The original RAG model combined a pre-trained seq2seq generator with a dense vector index of Wikipedia, split into disjoint 100-word chunks (21M passages) . It hallucinated less and was factually correct more often than the generator alone .

Common misconceptions

"RAG makes answers correct." RAG is only as good as its knowledge source, and any source will probably never be entirely factual or free of bias . The RGB benchmark tests four abilities RAG needs: noise robustness, negative rejection (declining when no document has the answer), information integration across documents, and counterfactual robustness . The models tested showed some noise robustness but struggled significantly with the other three , depended heavily on what they retrieved, and had no safeguard against misinformation in it . Search and RAG reduce hallucination, but they are not a cure .

"More retrieved text is better." Where relevant text sits in the prompt matters: performance is highest at the very start or end and degrades in the middle . In one historical result, GPT-3.5-Turbo with the answer document in the middle of its retrieved set did worse than with no documents at all . Going from 20 to 50 documents improved results only marginally, even though recall kept rising , and extended-context models were not necessarily better at using their context .

"Dense retrieval has replaced keyword search." DPR beat a strong BM25 baseline by 9–19 points of top-20 accuracy on open-domain QA , but the two have different blind spots , which is why hybrids exist .

"The model looked it up." The application retrieves and builds the prompt; the model conditions on whatever text it is given.

Chunk size is a trade-off

Small chunks are precise but can separate a fact from its context; large chunks dilute the match and spend the window.Splitting along a document's structure (HTML, Markdown, JSON) often keeps related text together . LangChain's recursive splitter tries separators in order (paragraph break, line break, space, then the empty string) until chunks are small enough . chunk_overlap repeats text between neighbours to reduce loss at boundaries, and there is no universal recommended chunk size .

The lab makes the effect on BM25 visible. With three sentences per chunk, each chunk is a whole country, so every chunk contains capital: N = 12, n = 12, and IDF(capital) = ln(1 + 0.5/12.5) = ln(1.04) ≈ 0.039. The term that told capital sentences apart from export sentences now carries almost no weight. Chunking changes what retrieval can distinguish.

Context injection, provenance and citations

Context assembly decides order, format and amount. The Lost-in-the-Middle authors suggest reranking so relevant documents come first, or retrieving fewer . Each chunk should carry its document ID, location and version. A common way to generate citations is to number the chunks in the prompt and ask the model to cite the numbers; the application should then check that each cited chunk exists and actually supports the sentence, because a model can cite a source that does not. RAG-Sequence uses one retrieved document for the whole output, while RAG-Token may draw on a different one per token , so attribution is not always one-to-one.

Retrieval errors and prompt injection

A frequent failure is retrieval rather than generation: the right chunk never reaches the prompt. Web content also contains noise and fake news . Evaluate the retriever on its own (is the right chunk in the top k?) before blaming the model.

Retrieved text is also an attack surface. LLM-integrated applications blur the line between data and instructions, and indirect prompt injection plants instructions in content likely to be retrieved, so an attacker needs no direct access to the application . Processing such text can act like arbitrary code execution: it can change the application's behaviour and control which other APIs are called . RAG and fine-tuning do not fully mitigate prompt injection . Treat every retrieved chunk as untrusted data, and enforce access control in the retrieval layer, not in the prompt.

Engineering details

• Apply permission filters at query time; a model cannot be trusted to withhold a chunk it has already been shown.
• Log the query, retrieved chunk IDs and scores, and the final prompt, so a bad answer can be traced to retrieval or generation.
• Re-embed when the encoder changes: vectors from different encoders are not comparable.

39. Tool calling and AI agents

Ask a model "What's the weather in Mumbai?" and the honest answer from its parameters alone is "I don't know": current weather did not exist when the model was trained, and its weights do not change at inference . Language models are weak at things like arithmetic and up-to-date facts, which simple external tools such as search engines, calculators and calendars can supply . Tool calling is how an application connects the two.

The key idea: the model only writes a request

The model never runs anything itself. It emits a structured request; the application (or the provider's servers) runs the operation and returns the result into the conversation . Six things need to be kept apart:

Step — Who acts — What exists; Normal model response — Model — Plain text for the user; Structured tool request — Model — A tool name plus arguments, in a schema the application declared; Tool execution — Application — Ordinary code calls a weather API, a database, a calculator; Tool result — Application — The output (or an error) is added to the conversation; Continued generation — Model — The model reads the result and writes more text or another call; Multi-step workflow — Application loop — Repeat until the model stops asking or a limit is hit

The documented flow is a multi-step conversation: send a request with the tools the model may call, receive a tool call, execute it in the application, send the output back, then receive either a final answer or more tool calls . Every client-side call is a round trip: the model asks, the application executes and reports back, and the model continues .

Schemas and the structured request

A function tool is defined by a JSON schema: it lets the model pass data to the application, and the application's code accesses the data or takes the action . For the weather question an application might declare:

``json { "name": "get_current_weather", "description": "Current conditions for a city", "parameters": { "type": "object", "properties": { "city": { "type": "string" }, "units": { "type": "string", "enum": ["metric", "imperial"] } }, "required": ["city", "units"], "additionalProperties": false } } `

and the model's turn would contain, instead of prose, a request such as {"name": "get_current_weather", "arguments": {"city": "Mumbai", "units": "metric"}}. The exact envelope differs by provider. With Claude client tools, the response has stop reason tool_use and carries tool_use blocks, and the application answers with a tool_result . OpenAI's strict mode makes calls reliably follow the schema rather than on a best-effort basis, and requires additionalProperties: false with every field required . By default the model decides whether to call zero, one or several functions; "required" forces at least one, or a named function can be forced , and it may call several in one turn unless parallel calls are disabled .

The agent loop

An agent is this round trip in a loop. The documented pattern is a while loop on the stop reason: while it is tool_use, run the tools and continue; exit on any other stop reason, such as end of turn or hitting the token limit . ReAct describes the same structure abstractly: at each step the agent receives an observation from the environment and takes an action according to a policy over its context , interleaving reasoning text with actions that gather information . OWASP describes agent systems as making repeated LLM calls in which earlier outputs direct later ones . The loop, the limits and the side effects are all application code.

Worked example: a retry budget

Suppose (hypothetically) a weather API times out on 10% of calls, independently, the application waits 2 seconds before giving up on a call, and it allows 3 attempts. The chance that all three time out is 0.1³ = 0.001, one request in a thousand. The worst-case wait before the user sees an error is 3 × 2 = 6 seconds, plus any back-off between attempts. A fourth attempt would cut failures to 0.0001 but stretch the worst case to 8 seconds. Retries trade latency for reliability, and they only help if failures are independent: if the API is down, every attempt fails.

Try it

In the lab the toy model handles arithmetic questions such as "what is 17 times 23 ?". It was trained to answer by emitting a calculator call between <tool> and </tool>; for the default question it writes <tool> 1 7 2 3 </tool> (the toy writes numbers digit by digit). The application parses the call, validates it, runs the calculator, appends <result> 3 9 1 </result>, and calls the model again to finish the sentence. Switch the simulated tool outcome to timeout (with a maximum number of attempts) or invalid argument* and watch the flow stop at Execute, with an application-written error instead of an invented number. The toy was not trained to read error results, so the application ends the turn itself.

Real-world usage

Toolformer trained a model to decide which APIs to call, when, with what arguments, and how to use the results , with a calculator, a Q&A system, a search engine, a translation system and a calendar . ReAct used a Wikipedia API with three actions, search, lookup and finish , and by looking facts up rather than relying on memory reduced hallucination and error propagation compared with chain-of-thought . The Model Context Protocol (MCP) standardises how servers expose tools to models, each with a unique name and a schema ; clients discover tools with tools/list and invoke one with tools/call` over JSON-RPC 2.0 .

Common misconceptions

"The model called the API." It did not. It produced text in a format the application agreed to parse . Some providers offer server tools, such as web search or code execution, that run on the provider's infrastructure instead of in your application , but even then a program executes the call, not the network's weights.

"An agent is a special kind of model." An agent is a control loop around a model plus tool integrations. The same model can answer a single question or drive a long workflow; what changes is the code around it.

"If the model chose the tool, the action must be allowed." Authorisation belongs in downstream systems, not in the model: every request through a tool should be checked against security policy .

Validation, authorisation and trust boundaries

Treat a tool call as untrusted input from a component that can be wrong or manipulated. Excessive Agency is the vulnerability that lets damaging actions happen in response to unexpected, ambiguous or manipulated model output, whatever the cause ; its root causes are excessive functionality, excessive permissions and excessive autonomy . The countermeasures follow:

• Argument validation. Check every argument against the schema and against business rules (a date range, an amount limit) before execution. MCP requires servers to validate all inputs, enforce access controls, rate-limit invocations and sanitise outputs; clients should confirm sensitive operations, show inputs, validate results, use timeouts and log usage .
• Least privilege. Give the agent only the functions and permissions it needs, and prefer narrow tools over open-ended ones such as "run a shell command" . Actions for a user should run in that user's security context with minimum privileges , and API tokens should stay in application code rather than being given to the model .
• Human approval. Require a person to approve high-impact actions before they happen . The MCP specification says there should always be a human in the loop who can deny tool invocations .
• Tool results are untrusted too. A fetched web page or email can contain instructions; feeding it back into the context is the indirect prompt-injection path . MCP clients must treat tool annotations as untrusted unless they come from trusted servers .

Errors and retries

Distinguish failures the model can fix from failures it cannot. MCP separates protocol errors (an unknown tool, invalid arguments), reported as JSON-RPC errors, from tool execution errors, which are returned in the result with isError: true . A validation error is worth returning to the model so it can correct its arguments; a timeout is usually retried by the application, with a cap and back-off, and only then reported. Retrying a non-idempotent action (sending a payment, posting a message) can duplicate it, so give such calls an idempotency key or do not retry them automatically. Where tool execution is asynchronous or long-running, a message system such as NATS can carry requests and results between the agent loop and workers; the loop still decides what happens next.

Loop limits and observability

Agents can get stuck. A common ReAct failure is looping: the model repeats its previous thoughts and actions and cannot break out . None of the official tool-calling pages checked state a default or recommended maximum number of iterations; the cap is an application decision . Set limits on iterations, total tool calls, tokens and wall-clock time, and end the turn with a clear message when one is hit. Logging and rate limiting do not prevent excessive agency, but they limit damage and give monitoring time to catch undesirable actions . A useful trace records each model call, each tool request with its validated arguments, each result or error, latency and the stop reason.

Architecture differences

Tool use can be learned in different ways. Toolformer built its training data by sampling candidate API calls, executing them and keeping only calls whose results reduced the loss on the following tokens , but it could not chain tools, because each tool's calls were generated independently . ReAct used prompting with a few in-context examples; on ALFWorld and WebShop it beat imitation and reinforcement-learning methods by 34 and 10 points absolute success rate in 2022 . Hosted APIs expose tools as schemas the provider's chat format renders into the prompt. The page's toy model is the simplest case: its corpus taught it a fixed tag format, and its training excluded the result tokens from the loss, so it learned to ask for results, not to produce them.

40. Multimodal artificial intelligence

A transformer consumes a sequence of vectors. It does not care whether those vectors started life as word tokens, image patches or slices of audio. Multimodal AI is largely the engineering of turning each kind of input into vectors a model can use, and of deciding where in the model the different streams meet.

The problem

Text arrives already discrete. Images, audio and video are dense, continuous signals with far more raw numbers than a sentence: a 224 × 224 RGB image is 224 × 224 × 3 = 150,528 values. Each modality needs a front end that reduces it to a manageable sequence, and the model needs a way to relate the sequences to each other.

Modalities and their representations

`` image ──► patches ──► vision encoder ──┐ video ──► frames ──► vision encoder ───┤ audio ──► log-Mel spectrogram ──► audio encoder ──┼──► projection / cross-attention ──► language model ──► text tokens text ──► tokenizer ──► token embeddings ┤ table ──► serialised text or features ──┘ `

The figure shows one common shape, not a universal design.

• Images: patches. The Vision Transformer (ViT) splits an image into fixed-size patches, linearly embeds each, adds position embeddings and feeds the sequence to a standard transformer encoder . For classification it prepends an extra learnable classification token .
• Audio: spectrogram features. Whisper resamples audio to 16,000 Hz and computes an 80-channel log-magnitude Mel spectrogram on 25 ms windows with a 10 ms stride , in 30-second segments .
• Video can be handled as frames fed to a vision encoder; Flamingo's Perceiver Resampler turns a large, variable number of visual features from an image or a video into a small, fixed number of visual tokens .
• Structured data such as a table is usually either serialised into text and tokenised, or encoded as numeric features. Which is better depends on the task; there is no single standard.

Where the streams meet

• A shared embedding space. CLIP trains an image encoder and a text encoder together by predicting which caption goes with which image, on 400 million image–text pairs . In a batch of N pairs it maximises cosine similarity for the N real pairs and minimises it for the N² − N wrong pairings , and it classifies without task-specific training by embedding class names and picking the closest .
• A projection into the language model's token space. LLaVA takes features from a pre-trained CLIP ViT-L/14 encoder and uses a trainable projection matrix W to turn them into "language embedding tokens" with the same width as the LLM's word embeddings: H_v = W · Z_v . The image becomes extra tokens in one unified input stream.
• Cross-attention. Flamingo keeps a vision model and a language model frozen and inserts new cross-attention layers between the language model's layers; keys and values come from the vision features and queries from the language inputs .

The mathematics of patches

For an image of height H, width W and C channels cut into P × P patches, the number of patches is N = H·W / P², and N is the transformer's input sequence length . Each patch flattens to a vector of P·P·C numbers, and a learned matrix E of shape (P²·C) × D maps it to the model width D: z_i = x_i · E + e_pos,i`, where x_i is the flattened patch and e_pos,i its position embedding .

Worked example

ViT with 16 × 16 patches on a 224 × 224 image: N = 224 × 224 / 16² = 50,176 / 256 = 196 patches . Each RGB patch holds 16 × 16 × 3 = 768 numbers. Halving the patch side to 8 gives 50,176 / 64 = 784 patches, four times as many; sequence length grows with the inverse square of the patch size, so smaller patches cost more compute .

Audio in Whisper: a 30-second segment at a 10 ms stride gives 30 / 0.01 = 3,000 spectrogram frames of 80 values. The encoder's convolutional stem has a second layer with stride 2 , which halves that to 1,500 positions.

Try it

The lab works the patch arithmetic on an 8 × 8 RGB image. Choose 2 × 2 patches (16 patches of 2 × 2 × 3 = 12 numbers) or 4 × 4 patches (4 patches of 48 numbers), select a patch, and see its pixels flattened, scaled to 0–1 and multiplied by a projection matrix to give an 8-dimensional embedding. The projection here is random; in a real model it is learned, and a position embedding would be added. The panel also repeats the 224/16 check above.

Real-world usage

Whisper is a standard encoder–decoder transformer trained to predict transcripts on 680,000 hours of multilingual, multitask audio ; special tokens in the decoder's sequence select the task, such as transcription or translation, so one model replaces several pipeline stages . A voice assistant is often a pipeline: speech-to-text, then a text model, then text-to-speech to turn the reply back into audio. This article's sources do not describe a specific text-to-speech architecture, so we do not describe one.

Common misconceptions

"All multimodal models work the same way." Architectures vary widely. Flamingo injects vision through cross-attention into a frozen language model, while LLaVA projects visual features into the language model's input token space; neither paper describes how specific commercial multimodal models are built .

"Image generators write pictures token by token, like a chatbot." Many do not. Diffusion models learn to reverse a fixed forward process that gradually adds Gaussian noise to data until the signal is destroyed . A text model generates left to right; a diffusion model refines a whole image over many denoising steps.

"The model sees the picture." It receives numbers: pixel values cut into patches and projected into vectors, exactly as text arrives as token IDs mapped to vectors.

Diffusion in more detail

DDPM used T = 1000 steps, with forward-process variances increasing linearly from β₁ = 10⁻⁴ to β_T = 0.02 . A network ε_θ is trained to predict the noise ε that was added to produce the noisy image x_t, and sampling uses that prediction to step from x_t to x_(t−1) . DDPM's denoiser is a U-Net with parameters shared across time steps, told the step t through sinusoidal position embeddings .

Pixel-space diffusion is expensive: training can take hundreds of GPU days, and sampling needs many sequential network evaluations . Latent diffusion runs the process in the latent space of a pre-trained autoencoder instead, and decodes the result to an image in one decoder pass . The downsampling factor f matters: pixel-space LDM-1 trained much more slowly than f = 4 to 16, while f = 32 compressed too much and hurt quality . Text prompts are encoded by a separate encoder and injected into the U-Net through cross-attention , the same mechanism Flamingo uses in the other direction.

Training tricks that recur

• Freeze and bridge. Flamingo trains only the new components between two frozen models . Each new layer's output is multiplied by tanh(α), with α a learnable scalar initialised to 0, so at the start the combined model behaves exactly like the original language model .
• Train the adapter first. In LLaVA's first stage both the vision encoder and the LLM are frozen and only the projection is trained, which acts like training a visual tokenizer for the frozen LLM .
• Scale versus inductive bias. Transformers lack CNN biases such as locality and translation equivariance, so ViT generalises poorly on small datasets but excels when pre-trained on 14M to 300M images .
• Resolution changes. At higher resolution ViT keeps the patch size, so the sequence gets longer, and the pre-trained position embeddings are interpolated in 2-D .
• Prompting the text side. For CLIP, the template "A photo of a {label}." was a good default that often beat the bare label .

Failure modes specific to modalities

Each modality brings its own errors. CLIP struggles with abstract, systematic tasks such as counting objects . Like text-only models, LLaVA can generate output not grounded in facts or in the input image . Whisper's failure modes include repeat loops, skipping the first or last words of a segment, and hallucinating whole unrelated transcripts . Multimodal inputs also open new injection channels, such as instructions hidden in an image that accompanies harmless text .

Engineering details

• Token budgets. Visual tokens consume context like text tokens. A 196-patch image costs as many positions as a long paragraph; high-resolution inputs or many video frames can dominate the window, which is the motivation for compressing to a fixed number of visual tokens .
• Preprocessing must match training. Sample rate, spectrogram settings, image size and normalisation are part of the model's contract. Feeding 44.1 kHz audio to a model trained on 16 kHz features, or images normalised with the wrong statistics, silently degrades results.
• Pipelines versus end-to-end. A speech-to-text → text model → text-to-speech pipeline is easy to debug and to swap parts in, but loses tone and timing between stages; an end-to-end model can keep them but is harder to inspect.

41. Limitations, hallucinations and AI reliability

Everything in this article so far explains why language models are useful. This chapter explains how they fail, why, and what can be done about it. The short version: a model generates likely continuations, and "likely" and "true" are different properties. Most failures come from that gap, from the data the model learned from, or from the system built around it.

Hallucination and incorrect claims

In natural-language generation, hallucination usually means generated content that is nonsensical or unfaithful to the provided source . Intrinsic hallucinations contradict the source ; extrinsic ones can be neither supported nor contradicted by it, so they cannot be verified from it . For LLMs, Huang et al. distinguish factuality hallucination, which conflicts with verifiable real-world facts, from faithfulness hallucination, which diverges from the user's input or is internally inconsistent , including ignoring instructions or the given context . An unsupported citation is a special case: a reference that looks real but does not exist or does not say what is claimed.

Why it happens

Kalai et al. argue that models hallucinate because training and evaluation reward guessing over admitting uncertainty, much as exam-takers guess when a blank answer scores zero . In their analysis, hallucinations are errors in a binary classification: if wrong statements cannot be told apart from facts, the statistical pressures of pretraining produce them . For arbitrary facts with no learnable pattern, the base-model hallucination rate should be at least the fraction of such facts that appear only once in training; if 20% of birthday facts are singletons, expect at least 20% hallucination on birthdays . Most benchmarks give no credit for "I don't know", so guessing improves scores .

Some falsehoods are learned directly. Imitative falsehoods are false answers that the training objective encourages because they are likely under the training distribution . On TruthfulQA, whose 817 questions across 38 categories target common misconceptions , the largest models were generally the least truthful ; in 2021 the best model was truthful on 58% of questions against 94% for humans . Scale alone does not fix this, and alignment is not the same as truthfulness: the InstructGPT authors say their models still make up facts and produce biased output .

The mathematics of abstaining

Kalai et al. propose stating a confidence target t in the instructions: answer only if more than t confident, with a correct answer scoring +1, "I don't know" 0, and a wrong answer −t/(1 − t) . If the model's chance of being right is c, the expected score of answering is

`` E[score | answer] = c · 1 − (1 − c) · t / (1 − t) ``

and abstaining scores 0. Setting E = 0 gives c = t, so answering pays exactly when c > t.

Worked example

With t = 0.75 the penalty is 0.75 / 0.25 = 3 points.

• c = 0.8: E = 0.8 − 0.2 × 3 = 0.8 − 0.6 = +0.2, so answer.
• c = 0.7: E = 0.7 − 0.3 × 3 = 0.7 − 0.9 = −0.2, so abstain.
• c = 0.75: E = 0.75 − 0.25 × 3 = 0, the break-even point.

Under ordinary right/wrong grading (penalty 0), E = c is positive for any c > 0, so a guess always beats abstaining. That is the incentive the authors want to remove.

Calibration

A model is calibrated when its stated probabilities match how often it is right: of all answers given with 80% confidence, about 80% should be correct. Kadavath et al. found larger models well calibrated on multiple-choice and true/false questions when formatted appropriately . Models could self-evaluate by proposing an answer and estimating P(True), the probability it is correct . Models trained to predict whether they know an answer did well and partly generalised, but were poorly calibrated on new tasks . RLHF policies looked miscalibrated, but a simple temperature adjustment made them fairly well calibrated on several evaluations . Calibration is measured, task by task; it is not a property a model simply has.

Try it

The lab presents six constructed failures: a fabricated reference list, outdated knowledge, a retrieval failure, a prompt injection, an instruction lost from a long context, and an arithmetic error (48,729 × 316 answered as 15,399,364; the correct product is 15,398,364). For each, pick the primary cause and the lab explains the mechanism and a mitigation. The cases are illustrations of documented failure types, not transcripts from a particular product. Notice that the right fix changes with the cause: a better retriever does nothing for an arithmetic slip, and a calculator does nothing for an injected instruction.

Real-world usage

Risk frameworks exist to organise this work. NIST AI 600-1 (July 2024) is a cross-sectoral generative-AI profile and companion to the AI Risk Management Framework, intended for voluntary use . Its definition of "confabulation" appears only in the PDF, which this article's research did not use, so we do not quote it .

Common misconceptions

"Bigger models will stop hallucinating." Larger models were less truthful on TruthfulQA , more vulnerable to training-data extraction , and produced less faithful reasoning on most tasks in one study . Making models bigger does not by itself make them follow a user's intent better .

"RAG solves hallucination." It reduces hallucination but does not cure it, because grading still rewards guessing when search fails ; models also struggle to reject questions their documents cannot answer .

"There is a fix for prompt injection." Because models are stochastic, it is unclear whether any fool-proof prevention exists; the known measures mitigate impact .

"A high benchmark score means the model is reliable." See benchmark limitations below.

A catalogue of failure sources

Failure — Mechanism; Training-data bias — The model reproduces patterns in its data. InstructGPT cut toxic outputs by about 25% when prompted to be respectful, but showed no significant bias improvement on Winogender and CrowS-Pairs; Distribution shift — Generalisation assumes training, validation, test and real-world data share a distribution; when they do not, held-out performance stops predicting real-world performance; Outdated knowledge — Weights are fixed after training ; the world is not; Prompt injection — Inputs alter the model's behaviour in unintended ways, even inputs imperceptible to humans ; direct injection comes from the user, indirect injection through websites or files; Tool misuse — Damaging actions follow from unexpected, ambiguous or manipulated model output when the system grants too much agency; Retrieval failure — The right document is not retrieved, or a wrong one is trusted; models lean heavily on what they retrieve; Reasoning errors — Correct decomposition with wrong arithmetic in a step ; written reasoning that does not reflect the real cause of the answer; Lost context — Information in the middle of a long context is used less well , and anything truncated from the window is simply gone

Benchmark limitations

A benchmark measures one distribution of questions with one grading rule. Contamination is the worst case: training on a benchmark's test split and then evaluating on it . It inflates scores, and its true extent is unknown because it is hard to measure , especially for closed models with little documentation . The same hygiene applies to your own evaluations: remove test examples that duplicate training examples, or results will look too good . Binary grading also hides abstention behaviour .

Privacy and data provenance

Models can memorise and leak training data. Querying GPT-2, Carlini et al. extracted hundreds of verbatim training sequences, including public personal information such as names, phone numbers and email addresses , even for sequences that appeared in only one training document . Not overfitting does not mean not memorising . Differentially private training is sound in theory but slows training and usually reduces quality; de-duplication helps but cannot stop every attack . Provenance is often unclear: open model weights do not mean open data, and the pretraining sets of leading open models are not public . If you cannot say what a model was trained on, you cannot fully say what it might reproduce.

Security boundaries

The model sits inside a trust boundary it cannot enforce. OWASP advises separating and clearly marking untrusted external content and adversarial testing that treats the model as an untrusted user, to test trust boundaries and access controls . Authorisation belongs in downstream systems . RAG and fine-tuning do not fully mitigate injection .

Mitigations, none universal

Mitigation — What it addresses — What it does not; Better data (filtering, de-duplication) — Memorised and low-quality text — Facts that change after training; Evaluation suites — Measured error rates on known task types — Distributions you did not test; Grounded retrieval — Missing and private knowledge — Retrieval errors, misinformation in sources; Tool verification — Arithmetic, lookups, current data — A wrongly written tool call; Structured validation — Malformed or out-of-policy output; OWASP lists validating output formats with deterministic code — Well-formed but false content; Human review — High-impact actions — Scale; reviewers miss things too; Monitoring and logging — Detecting harm early and limiting it — Preventing the first incident; Defence in depth — Layers that fail independently — Nothing is guaranteed

The pattern across this article holds here too: each mechanism is precise about what it computes and silent about everything else. A reliable AI system is one where the application knows which component is responsible for which property, checks each one, and does not ask the model to guarantee what only code, data or people can.

42. Follow one prompt through an AI system

Every earlier chapter took one piece of the machine apart. This one puts the pieces back together and runs a single question through all of them, in order: "why is the sky blue ?". You will watch it become tokens, then integers, then vectors, then a stack of matrix arithmetic, then one score per vocabulary entry, then a probability distribution, then a chosen token, and then the whole thing again, once per output token, until the answer is finished.

The one idea to hold on to is that a language model only ever does one thing: given a sequence of tokens, it produces a distribution over the next token. A chat model is no exception, because a conversation is turned into one token sequence before the model sees it . Generation is a loop around that single step. The model predicts the next token from the prompt plus everything it has generated so far, and stops at a length limit or an end-of-sequence token . Everything else in an "AI system" (chat formatting, retrieval, tools, streaming) is application code arranged around that loop.

What the toy model is, and is not

The lab runs this page's own toy model in your browser. The computation is real: every number on every stage is calculated from trained weights when you open the stage, and nothing is pre-recorded. The scale is tiny and the knowledge is memorised. Its probabilities describe this toy model only. They are not measurements of any production language model.

What it is. A decoder-only transformer in GPT-2's arrangement, trained for this page: 2 blocks, 4 attention heads of width 12, model width d = 48, a feed-forward width of 192 with GELU, LayerNorm before each sub-layer (pre-LN), learned token and position embeddings, and an output layer tied to the token embedding table. Its context window is 56 tokens, its vocabulary has 233 entries, and it has about 70,500 parameters. It was trained with AdamW on a small corpus written for the page. The corpus has three kinds of text: short questions and answers about light, colour and machine learning (including the sky question), retrieval exercises about fictional countries, and calculator exercises.

What it is not. It is not a general chatbot. It has seen a few hundred sentences, not a web-scale corpus, so it reproduces the answers it memorised and does poorly on anything else. A question that uses words outside its vocabulary turns those words into <unk>, and the lab warns you when that happens. Its tokenizer works on whole words so that every token is readable. Production models use sub-word tokenizers instead, so GPT-2's real tokenizer is shown beside it for comparison. Like every trained model, it does not learn while you use it. Its weights were fixed when training finished.

The twelve stages

1. Prompt

You type a question. Before any model runs, the application wraps it in the format the model was trained on. For the toy model that format is <q> why is the sky blue ? <a>: a question marker, the question, and an answer marker that tells the model its turn has begun. Real chat models do the same with a chat template, which turns role-tagged messages into one sequence with the model's own control tokens . One example is the tokens that mark the start of an assistant reply. Without them, a model may simply continue the user's message . In depth: chapter 32, What happens when you send a prompt?

2. Tokenize

The text is split into tokens from a fixed vocabulary. The toy tokenizer lowercases the text and separates punctuation, so the prompt above becomes 8 tokens: <q>, why, is, the, sky, blue, ?, <a>. The lab shows GPT-2's real byte-level BPE next to it. GPT-2 splits "Why is the sky blue?" into six tokens, and its leading spaces are part of the tokens. A production tokenizer works this way because a sub-word vocabulary can encode any string, while a word-level one cannot . In depth: chapter 14, Why language models need tokens, and chapter 15, Tokenization step by step.

3. Token IDs

Each token becomes an integer, its row number in the vocabulary . The lab shows the toy IDs and, below them, GPT-2's IDs for the same question: [5195, 318, 262, 6766, 4171, 30], from a vocabulary of 50,257 . The two sets of numbers are unrelated. An ID carries no meaning by itself: it is an address, and neighbouring IDs are unrelated tokens. In depth: chapter 15, Tokenization step by step, and chapter 6, How data becomes numbers.

4. Embeddings

Each ID selects one row of a learned table, the token embedding matrix. That row is a vector of 48 numbers for the toy model. This step is a lookup, not a calculation . The 8 IDs become an 8 × 48 matrix. In depth: chapter 16, Token embeddings.

5. Positions

Attention on its own does not see token order, so position information has to be supplied . The toy model, like GPT-2, learns one vector per position and adds it to the token vector . Other models use fixed sinusoids or rotate queries and keys (RoPE) . In depth: chapter 17, Positional information.

6. Attention

Each position builds a query, a key and a value from its vector. Its output is a weighted average of the values of the positions it may see, with weights from a softmax over scaled query–key dot products . A causal mask forbids any position from looking at later ones . The lab shows the weight matrix for each of the four heads in block 1, and which earlier tokens the last position (<a>) attends to most. The weights are real for this model, but they are not an explanation of why it answers as it does (see the misconceptions in the deep dive). In depth: chapter 20, Self-attention intuition, chapter 21, Queries, keys, and values, chapter 22, Causal attention and masking, and chapter 23, Multi-head attention.

7. Feed-forward

Each position then passes, on its own, through a two-layer network: 48 → 192 → 48 with GELU in between . The lab shows the last position's vector at each step. It also counts how many of the 192 hidden units GELU has pushed to near zero. In depth: chapter 24, Feedforward networks, residuals, and normalization, and chapter 8, Activation functions.

8. Stacked blocks

Attention and the feed-forward network form one block. Each sub-layer's output is added back to its input (the residual stream). The toy model has 2 blocks and large models have dozens . The lab's table shows the norm of the residual stream after each block, and what the output layer would predict if you read it at that point. This "logit lens" readout is exploratory: it is a way of looking, not a stage the model runs. In depth: chapter 25, Stacking transformer blocks, and chapter 19, The transformer architecture.

9. Logits

The last position's vector is normalised and multiplied by the transposed token embedding table. This gives one raw score, a logit, for each of the 233 vocabulary entries . Reusing the embedding table as the output layer is called weight tying . In depth: chapter 26, From hidden states to vocabulary probabilities.

10. Probabilities

Softmax turns the logits into numbers between 0 and 1 that sum to 1 . The lab prints that sum to nine decimal places. These are the model's estimates of the next token given this context and nothing else. In depth: chapter 26, From hidden states to vocabulary probabilities.

11. Select a token

A decoding rule picks one token. With temperature 0 the rule is greedy: always take the most probable token . With a higher temperature the logits are divided by T before the softmax, which flattens or sharpens the distribution . Top-p keeps only the smallest set of tokens whose probabilities reach p, then renormalises them . A seeded random number then picks from what is left. In depth: chapter 34, How tokens are selected.

12. Repeat

The chosen token is appended to the context and the whole forward pass runs again. The loop stops when the model emits <end>, when it hits the 50-token limit, or when the context window fills. Every generated token can be clicked, and shows the probability it had when it was chosen. The toy lab recomputes the full forward pass at every step. Real systems run the prompt once (prefill) and then process only the new token at each step, reusing cached keys and values . In depth: chapter 33, Prefill vs decode, and chapter 35, Context windows and KV cache.

Using the controls

• Stage buttons and playback. The numbered buttons jump to any stage. Next and previous move one stage at a time. Play advances on its own, holding each stage for several seconds, and longer if your system asks for reduced motion. Stepping or jumping pauses playback. Pressing play on the last stage starts again from stage 1. The restart button replays from the beginning, and the speed control runs from 0.25× to 4×.
• Keyboard. With the lab focused, ← and → move between stages. Space plays or pauses, and Home and End jump to the first and last stage.
• Stage 1 chooses the path (plain, retrieval or tool) and lets you edit the question. Use words the toy vocabulary knows.
• Stage 6 chooses which of the four heads to display.
• Stage 11 sets temperature (0 to 2; 0 is greedy and is the default), top-p (0.1 to 1, default 1, meaning no truncation) and seed (1 to 50). These settings carry through every later step of generation.
• Stage 12 lets you pick any generation step, by clicking a token or using the slider, to see the context length and probability at that step.
• Show mathematics (on by default) shows or hides each stage's equation and shapes. Technical explanation opens a paragraph on how production systems do the same stage. Inspect this step's state lists the path, the prompt's token count, the hidden-state shape, the decoding settings, the output length and the number of tool calls. Reset restores every default.

Every stage is a view of one deterministic computation: prompt plus settings gives one trace and one generation. Going backwards, replaying or changing speed therefore never changes a number. Only editing the question, the path or the decoding settings does.

Three paths: plain, retrieval and tool use

The plain path is the twelve stages above. The other two show how applications extend the basic pipeline without changing the model's mechanism.

With retrieval (RAG). The default question is "what does orlund export ?". Before the model runs, the application searches a small knowledge base with BM25, a lexical ranking function . The knowledge base is the page's fictional gazetteer: 12 countries, each with a capital, an export and a river, stored as 36 one-sentence chunks. The top 2 chunks are pasted in front of the question after a <doc> marker, and the model was trained to copy its answer from that text, or to say "the document does not say ." when the fact is missing. The search is worth watching in detail. Once stop words are removed, the query's terms are "orlund" and "export". This BM25 does no stemming, so "export" does not match "exports", and only "orlund" scores. The capital sentence and the export sentence tie, and the river sentence ranks below them only because it is longer. The answer comes out right because the model finds the export fact among the retrieved text, not because the retriever understood the question. Retrieval adds knowledge that the weights do not hold, but it does not guarantee correctness: the answer is only as good as what is retrieved .

With a tool. The default question is "what is 4 8 times 1 7 ?". The toy vocabulary splits numbers into digits. The model was trained to answer arithmetic by writing a calculator call, <tool> 4 8 * 1 7 </tool>. When it emits </tool>, the application, not the model, parses the expression and checks that it is digits with one operator. It computes 48 × 17 = 816 and appends <result> 8 1 6 </result> to the context. The model then continues and writes the answer sentence. In stage 12 those result tokens are marked "app", and no forward pass produced them. In training they were excluded from the loss, so the model never learned to write them itself. Production tool calling works the same way: the model emits a structured request, and the application executes it and sends the result back .

Step — Done by; Formatting the prompt (chat template, <q> … <a>) — application; Searching the knowledge base, choosing chunks, pasting them in — application; Tokenizing and detokenizing — application (the tokenizer is fixed code and data, not part of the network); Embeddings, attention, feed-forward, logits — model; Turning logits into a choice (temperature, top-p, seed) — decoding code around the model; Writing <tool> … </tool> — model; Running the calculator, writing <result> … </result> — application; Deciding when to stop — the model emits <end>; the application enforces the length and context limits

The mathematics in one place

For a prompt of n token IDs t_1 … t_n, with model width d = 48, head width d_h = 12 and vocabulary size V = 233:

`` x_i = W_te[t_i] + W_pe[i] (stages 4–5) for each block: x ← x + Attn(LN₁(x)) (stage 6) x ← x + W₂ · GELU(W₁ · LN₂(x) + b₁) + b₂ (stage 7) z = LN_f(x_n) · W_teᵀ (stage 9, z has V entries) p_k = exp(z_k / T) / Σ_j exp(z_j / T) (stages 10–11) `

Here W_te is the 233 × 48 token embedding table, W_pe the 56 × 48 position table, x_i the vector at position i, LN LayerNorm, W₁ (48 × 192) and W₂ (192 × 48) the feed-forward weights with biases b₁, b₂, z_k the logit for vocabulary entry k and T the temperature (stage 10 shows T = 1). Inside each head, Attn computes softmax(Q Kᵀ / √12 + M) V, where Q, K and V are n × 12 projections of LN₁(x) and M is the causal mask: 0 where position j ≤ i and −∞ where j > i . The four head outputs are concatenated back to width 48 and projected once more .

A worked example

Where the 70,500 parameters are. Each block has two LayerNorms (2 × 96), the combined query/key/value projection (48 × 144 + 144 = 7,056), the attention output projection (48 × 48 + 48 = 2,352) and the two feed-forward layers (48 × 192 + 192 = 9,408 and 192 × 48 + 48 = 9,264). That makes 28,272 per block, of which the feed-forward network holds 18,672, about two thirds. Two blocks give 56,544. Add the token table (233 × 48 = 11,184), the position table (56 × 48 = 2,688) and the final LayerNorm (96) for a total of 70,512. The output layer adds nothing, because it is the token table reused.

Shapes for the sky question. n = 8, so the IDs are a vector of 8 integers and the embeddings an 8 × 48 matrix. Each head's Q, K and V are 8 × 12. Its score matrix is 8 × 8, and the mask blocks 8 × 7 / 2 = 28 of those 64 entries, the ones above the diagonal, leaving 36. The feed-forward hidden layer is 8 × 192. Only the last row, 48 numbers, is used to produce the 233 logits.

Decoding with three illustrative logits. These numbers are invented to make the arithmetic visible; the lab shows the model's real values. Suppose three tokens have logits z = (2, 1, 0).

• At T = 1: exp gives 7.389, 2.718 and 1.000, which sum to 11.107, so p = (0.665, 0.245, 0.090).
• At T = 0.5: the logits become (4, 2, 0), exp gives 54.598, 7.389 and 1.000 (sum 62.987), so p = (0.867, 0.117, 0.016). Lower temperature concentrates probability on the top token.
• Top-p = 0.9 at T = 1: sorted cumulative probabilities are 0.665, then 0.910. The second already reaches 0.9, so two tokens survive and are renormalised to 0.665 / 0.910 = 0.731 and 0.269.
• A seeded draw of u = 0.8 walks the cumulative list (0.731, then 1.000). The first entry that exceeds u is the second token, so the second token is chosen. With u = 0.5 the first token would be chosen. Temperature 0 skips the draw and always takes the first.

Cost of the loop. The answer the toy model was trained on for this question is 30 tokens followed by <end>, so reproducing it takes 31 forward passes. Without a cache, those passes process contexts of 8, 9, …, 38 tokens, which is 31 × 8 + (0 + 1 + … + 30) = 248 + 465 = 713 token positions. With a key–value cache, the prompt is processed once (8 positions) and each later step processes only the one new token (30 more), which is 38 in total. The difference grows quadratically with length, which is why every production system caches .

Try it

Start at stage 1 with the plain path and step through with →. At stage 6, switch heads and see that each head spreads its attention differently. At stage 11, raise the temperature to 1 and change the seed. The first token may stay the same, because a memorised answer is very confident, but later tokens can drift. Then go back to stage 1 and choose "With retrieval". Edit the question to ask about another country's capital, and watch the retrieved chunks and the answer change together. Finally choose "With a tool" and find the tokens marked "app" in stage 12.

Real-world usage

The same twelve stages run whenever a hosted chat service, a coding assistant or a local model answers a prompt. Libraries expose them almost one to one. In Hugging Face Transformers, a tokenizer produces input IDs, generate() runs the decode loop and returns IDs, and batch_decode()` turns them back into text . Generated tokens can be streamed out as they are produced . What changes in production is scale and engineering: much bigger tables and stacks, caches, batching across many users, and serving infrastructure. The deep dive goes through each stage at that scale.

Common misconceptions

• "The model looked the answer up." In the plain path nothing is looked up. The answer comes from patterns stored in the weights, and that kind of parametric memory is hard to inspect or revise and can produce hallucinations . In the retrieval path the application does the looking up and pastes text into the prompt. The model only reads what it is given.
• "Asking questions trains it." Inference leaves the weights unchanged. A model can follow instructions and examples given only as text in the prompt, with no gradient updates . The context is input to a forward pass, not a change to the model.
• "The model ran the calculator." It wrote a request. The application parsed and executed it, as in every tool-calling API .
• "The attention heatmap shows why it answered." Attention weights are real intermediate values, but whether they count as explanations is contested. Different attention patterns can give the same prediction .
• "Each step produces a word." Each step produces a token. In the toy model a token happens to be a word. In production models a token is often part of a word, such as "encod" + "ing" .
• "Temperature 0 means identical answers forever." Greedy decoding is deterministic for a given computation. Across library versions, hardware or CPU versus GPU, PyTorch does not guarantee bitwise-identical results even with the same seed .
• "Retrieval makes the answer correct." RAG is only as good as its knowledge source . Search and RAG reduce hallucinations but do not cure them .

What changes at production scale

The stages are the same. What changes is their size, and the engineering needed to run them for many users at once.

Vocabulary — 233 words — 50,257 tokens — not in our sources; Layers (blocks) — 2 — 12 — 80; Model width d — 48 — 768 — 8,192; Attention heads — 4 — not in our sources — 64; Context window — 56 — 1,024 — not in our sources; Parameters — 70,512 — not in our sources — 65.2B

Kaplan et al.'s rule of thumb for non-embedding parameters, N ≈ 12 · n_layer · d_model² , checks out at both ends. For the toy model it gives 12 × 2 × 48² = 55,296, against the actual 56,544 in its blocks; the gap is mostly biases. For LLaMA's largest shape it gives 12 × 80 × 8,192² ≈ 64.4 billion, close to the 65.2B listed. For closed hosted models, no primary source we checked states layer counts, widths or parameter counts .

• Prompt. Chat formats are not standard. Models fine-tuned from the same base can use different control tokens, and using the wrong ones badly hurts quality . Hosted APIs accept role-based messages but do not publish the token-level template they apply on the server .
• Tokenize. Production tokenizers learn sub-word units from data: BPE (Llama, Gemma, Qwen2), Unigram (T5) or WordPiece (BERT-family models) . Byte-level BPE starts from the 256 byte values, so no input ever becomes an unknown token . On average a token covers about 4 bytes of English text, though this depends on the tokenizer and the language .
• Token IDs. The ID space is the vocabulary size, tens of thousands of entries instead of 233. The tokenizer must be the exact one used in pretraining, or the IDs point at the wrong rows .
• Embeddings. The table grows with the vocabulary. GPT-2's defaults imply a 50,257 × 768 token-embedding table , and GPT-2 ties it to the output layer as the toy model does .
• Positions. Many recent models drop learned absolute positions. LLaMA applies rotary embeddings (RoPE) at every layer , under which the query–key dot product depends only on relative offset .
• Attention. Full attention's cost grows quadratically with sequence length . Exact, IO-aware kernels such as FlashAttention cut memory traffic without approximating . Many models also share key/value heads across query heads. Mistral 7B has 32 query heads but only 8 key-value heads (grouped-query attention), which shrinks its cache .
• Feed-forward. LLaMA replaces GELU with SwiGLU and uses a hidden size of (2/3)·4d . It also normalises with RMSNorm before each sub-layer . The toy model's pre-LN arrangement is the same idea.
• Stacked blocks. Tens of layers instead of two. When one GPU cannot hold the weights, tensor parallelism shards each layer across GPUs .
• Logits. One logit per vocabulary entry, so the output multiply has tens of thousands of columns. Weight-only 4-bit quantization speeds up decoding because decoding is dominated by reading weights , and methods such as GPTQ quantize 175B-parameter models to 3 or 4 bits with negligible accuracy loss .
• Probabilities. Softmax is unchanged. The probabilities are a model's estimates, and calibration (whether they match how often the model is right) has to be measured, not assumed .
• Select a token. The policy is configuration. In Transformers, greedy search is the default strategy . When the model's own configuration does not set them, temperature defaults to 1.0, top_k to 50 and top_p to 1.0 . Nucleus sampling adapts to the model's confidence in a way a fixed top-k cannot . Repetition penalties discount tokens already generated .
• Repeat. This is where most of the engineering goes.
• - Prefill and decode. Prefill processes the whole prompt in parallel and is compute-bound. Decode processes one token per request at a time and is memory-bound . Users feel these as time-to-first-token and time-between-tokens .
• - KV cache. For the 13B OPT model the cache needs 2 × 5,120 × 40 × 2 bytes, about 800 KB per token , and one 2,048-token request can reach 1.6 GB . If the toy model kept a cache in 32-bit floats, it would need 2 × 2 layers × 4 heads × 12 × 4 bytes = 768 bytes per token, about a thousand times less. Serving a 13B model on a 40 GB A100, about 65% of memory holds weights and close to 30% holds per-request state, mostly the cache .
• - Batching. Many users share each forward pass. Iteration-level (continuous) batching lets requests join and leave the batch between steps . PagedAttention stores the cache in fixed-size blocks that need not be contiguous , 16 tokens per block by default in vLLM . Prefix caching reuses the cache of a shared prefix, such as a common system prompt, without changing outputs .
• - Fewer serial steps. Speculative decoding lets a small draft model propose several tokens that the large model checks in parallel, without changing the output distribution .
• - Unknowns. Hosted commercial APIs do not publish their batching, quantization or speculative-decoding details , nor their context-handling and cache design .

The two branches scale too. Production RAG usually retrieves with dense vectors as well as, or instead of, BM25. Each fails differently: BM25 is sensitive to selective keywords, while dense retrieval captures paraphrase but can miss rare phrases . Retrieved text is also an attack surface, because instructions planted in it can steer the model (indirect prompt injection) . Agents run the tool step inside a loop that continues while the model keeps requesting tools . The iteration cap is an application decision that no vendor page we checked sets for you .

Glossary

• Activation function — A nonlinear function applied to a neuron's weighted sum before it is passed on. Without one, stacked layers collapse into a single linear map .
• Agentic loop — Application code that repeatedly calls a model, runs any tools it requests and feeds the results back, until the model stops asking for tools .
• Attention — A function that maps a query and a set of key–value pairs to a weighted sum of the values, weighted by how well the query matches each key .
• Attention head — One of several attention computations run in parallel on separate learned projections. Their outputs are concatenated and projected .
• Autoregressive generation — Producing output one token at a time, each prediction conditioned on all earlier tokens .
• Backpropagation — The algorithm that computes the gradient of the loss with respect to every parameter by applying the chain rule backwards through the network .
• BM25 — A sparse, keyword-based ranking function that scores documents by weighted term matches, using an inverted index .
• Byte-level BPE — BPE whose base vocabulary is the 256 byte values, so any text can be encoded without an unknown token. GPT-2 uses it .
• Byte-pair encoding (BPE) — A tokenizer training method that starts from characters and repeatedly merges the most frequent adjacent pair until the vocabulary reaches a target size .
• Causal mask — A mask that sets attention scores for later positions to −∞ before the softmax, so no position can see the future .
• Chain of thought — Intermediate reasoning steps written out before a final answer. It can improve accuracy, but it is not guaranteed to reflect what drove the answer .
• Chat template — The model-specific rules that turn a list of role-tagged messages into one token sequence with control tokens .
• Chunking — Splitting documents into smaller pieces that can be retrieved individually and fit in the context window .
• Context window — The maximum number of tokens a model can take as input, fixed by its architecture and training .
• Continuous batching — Scheduling one model iteration at a time across a batch, so requests can join and leave between iterations. Orca called it iteration-level scheduling .
• Cross-entropy loss — The training loss for next-token prediction: the negative log of the probability the model gave to the correct token .
• Decode phase — The token-by-token part of generation after prefill. It cannot be parallelised across steps and is memory-bound .
• Decoder-only transformer — A transformer in which each position attends only to earlier positions, usually pretrained to predict the next token. Most modern large language models use this layout .
• Embedding — A learned vector stored in a lookup table and selected by an integer index, such as a token ID .
• End-of-sequence token — A special token whose generation ends the output. The toy model's is <end> .
• Feed-forward network — The per-position two-layer network inside each transformer block, applied to every position separately and identically .
• Fine-tuning — Further training of a pretrained model on a dataset for a specific task .
• FlashAttention — An exact attention algorithm that tiles the computation to reduce reads and writes between GPU high-bandwidth memory and on-chip memory .
• GELU — An activation function, GELU(x) = x · Φ(x), where Φ is the standard normal cumulative distribution function. The toy model uses it .
• Gradient descent — Repeatedly updating each parameter by subtracting the learning rate times its gradient .
• Greedy decoding — Always choosing the single most probable next token. It is the default in Transformers .
• Grouped-query attention (GQA) — Attention with fewer key–value heads than query heads, which shrinks the KV cache at close to full multi-head quality .
• Hallucination — Generated content that is nonsensical or unfaithful to the source or to verifiable facts .
• Inference — Using a trained model to make predictions. A language model's weights do not change at inference .
• KV cache — Stored key and value vectors from earlier tokens, kept per layer, so that each decode step computes only the new token's .
• Layer normalization — Normalising one example's activations by their own mean and variance, then applying a learned gain and bias .
• Logits — The raw, unbounded scores the final linear layer produces, one per vocabulary entry, before softmax .
• Nucleus (top-p) sampling — Sampling from the smallest set of tokens whose cumulative probability reaches p, with their probabilities rescaled to sum to 1 .
• PagedAttention — Storing the KV cache in fixed-size blocks that need not be contiguous, mapped by a block table, like virtual-memory pages .
• Parameters — The weights and biases a model learns in training, as opposed to hyperparameters, which the developer sets .
• Positional embedding — Position information added to token vectors, because attention alone ignores order. It can be learned or fixed .
• Pre-LN — Placing layer normalization inside each residual branch, before the sub-layer, with a final normalization before the output layer .
• Prefill — Processing the whole prompt in one parallel, compute-bound pass that also yields the first output token's distribution .
• Prefix caching — Reusing cached key–value blocks when a new request starts with the same tokens as an earlier one .
• Pretraining — Self-supervised training from random weights on a large corpus, where the targets come from the text itself .
• Prompt injection — Input that changes a model's behaviour in unintended ways. In the indirect form it arrives through retrieved data rather than from the user .
• Quantization — Storing weights in fewer bits, such as 8 or 4, to save memory and speed up memory-bound decoding .
• Query, key, value — The three learned linear projections of each position's vector that attention uses: queries are matched against keys, and values are averaged .
• Residual connection — Adding a sub-layer's output to its input, so that each block refines a running representation rather than replacing it .
• Retrieval-augmented generation (RAG) — Combining a model with an external document index whose retrieved text is supplied as context. The index can be updated without retraining .
• RoPE — Rotary position embedding, which rotates query and key dimensions by position so that their dot product depends on relative offset .
• Seed — The starting state of a pseudo-random generator. It reproduces a sampled output only when every other source of nondeterminism is removed .
• Softmax — The function exp(x_i) / Σ_j exp(x_j), which maps a vector to values in [0, 1] that sum to 1 .
• Special token — A vocabulary entry that marks structure rather than text, such as a role marker or end of sequence. Which ones exist differs by model .
• Speculative decoding — Having a small draft model propose several tokens that the large model verifies in parallel, without changing its output distribution .
• Temperature — A divisor applied to logits before softmax. Values below 1 sharpen the distribution, larger values flatten it, and T → 0 approaches greedy .
• Tensor parallelism — Splitting each layer's parameters across several GPUs so that a model too large for one GPU can be served .
• Token — The unit a language model reads and writes: a word, part of a word, a byte sequence or a special marker .
• Tokenizer — The component that turns text into tokens and token IDs, and back again. It must match the one used in pretraining .
• Tool call — A structured request a model emits for the application to execute. The application sends the result back as further input .
• Top-k sampling — Sampling only among the k most probable next tokens .
• TTFT and TBT — Time to first token (from request arrival to the first output token) and time between tokens (the gap between consecutive output tokens) .
• Unknown token — The placeholder, such as <unk>, that replaces text missing from a vocabulary. Information is lost wherever it appears .
• Vocabulary — The fixed set of distinct tokens. Each one has an integer ID from 0 up to the vocabulary size .
• Weight tying — Using one matrix for both the input embedding and the output layer .

Where to go next

• To see why each stage is built the way it is, start with chapter 19, The transformer architecture, then chapter 21, Queries, keys, and values, and chapter 24, Feedforward networks, residuals, and normalization.
• Where the weights came from: chapter 28, Self-supervised pretraining, and chapter 12, Backpropagation and gradient descent. For how behaviour is shaped after pretraining, see chapter 29, Fine-tuning and instruction following, and chapter 30, Preference alignment and model behavior.
• How a single request becomes a service: chapter 33, Prefill vs decode, chapter 35, Context windows and KV cache, and chapter 36, Production inference infrastructure. If you run model servers on a cluster, the Kubernetes deep dive covers that layer.
• The two branches in full: chapter 38, Retrieval-augmented generation, and chapter 39, Tool calling and AI agents.
• What can still go wrong at every stage: chapter 41, Limitations, hallucinations, and AI reliability.

Sources

1. What is Machine Learning? — Google Machine Learning Crash Course, 2026-01-27 (last updated)
2. Thresholds and the confusion matrix — Google Machine Learning Crash Course, undated
3. Linear regression: Loss — Google Machine Learning Crash Course, undated
4. Linear regression: Gradient descent — Google Machine Learning Crash Course, undated
5. Datasets: Dividing the original dataset — Google Machine Learning Crash Course, undated
6. Overfitting — Google Machine Learning Crash Course, undated
7. Production ML systems: Monitoring pipelines — Google Machine Learning Crash Course, undated
8. Categorical data: Vocabulary and one-hot encoding — Google Machine Learning Crash Course, undated
9. Numerical data: Normalization — Google Machine Learning Crash Course, undated
10. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding — arXiv (Devlin et al., 2018), 2018-10
11. Language Models are Few-Shot Learners — arXiv (Brown et al., 2020), 2020-05
12. Supervised Learning — Google Machine Learning Crash Course, undated
13. Training language models to follow instructions with human feedback — arXiv (Ouyang et al., 2022), 2022-03
14. 1.1. Linear Models — scikit-learn documentation (1.9.1), undated (stable docs, 1.9.1)
15. 1.10. Decision Trees — scikit-learn documentation (1.9.1), undated (stable docs, 1.9.1)
16. 1.11. Ensembles: Gradient boosting, random forests, bagging, voting, stacking — scikit-learn documentation (1.9.1), undated (stable docs, 1.9.1)
17. Introduction to Boosted Trees — XGBoost documentation, undated (stable docs)
18. Why do tree-based models still outperform deep learning on tabular data? — arXiv (Grinsztajn et al., 2022), 2022-07
19. Deep Residual Learning for Image Recognition — arXiv (He et al., 2015), 2015-12
20. LSTM — PyTorch documentation (2.14), undated (2.14 docs)
21. Attention Is All You Need — arXiv (Vaswani et al., 2017), 2017-06
22. Denoising Diffusion Probabilistic Models — arXiv (Ho et al., 2020), 2020-06
23. Machine Learning Glossary — Google Machine Learning Crash Course, undated
24. Auto-Encoding Variational Bayes — arXiv (Kingma et al., 2013), 2013-12
25. torch.Tensor — PyTorch documentation (2.14), undated (2.14 docs)
26. Tensor Attributes — PyTorch documentation (2.14), undated (2.14 docs)
27. Type Info — PyTorch documentation (2.14), undated (2.14 docs)
28. ToTensor — PyTorch documentation (torchvision 0.29), undated (torchvision 0.29 docs)
29. Normalize — PyTorch documentation (torchvision 0.29), undated (torchvision 0.29 docs)
30. torchaudio.load — PyTorch documentation (torchaudio 2.11), undated (torchaudio 2.11 docs)
31. Spectrogram — PyTorch documentation (torchaudio 2.11), undated (torchaudio 2.11 docs)
32. MelSpectrogram — PyTorch documentation (torchaudio 2.11), undated (torchaudio 2.11 docs)
33. OneHotEncoder — scikit-learn documentation (1.9.1), undated (stable docs, 1.9.1)
34. AI RMF 1.0 (Overview) — NIST AI Resource Center (NIST AI 100-1), 2023-01
35. The CIFAR-10 dataset — University of Toronto (Krizhevsky), undated
36. Embedding — PyTorch documentation (2.14), undated (2.14 docs)
37. Working with numerical data — Google Machine Learning Crash Course, undated
38. Appendix B: How AI Risks Differ from Traditional Software Risks — NIST AI Resource Center (NIST AI 100-1), 2023-01
39. Updates to the OECD's definition of an AI system explained — OECD.AI, 2023-11
40. Rules of Machine Learning — Google for Developers (Machine Learning guides), undated
41. Linear regression — Google Machine Learning Crash Course, undated
42. Logistic regression: Calculating a probability with the sigmoid function — Google Machine Learning Crash Course, undated
43. Chat templates — Hugging Face documentation, undated (Transformers v5.21.0)
44. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve — arXiv (Agrawal et al., 2024), 2024-03
45. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving — arXiv (Zhong et al., 2024), 2024-01
46. Metrics — vLLM documentation, undated (latest developer preview)
47. The Curious Case of Neural Text Degeneration — arXiv (Holtzman et al., 2019), 2019-04
48. Hierarchical Neural Story Generation — arXiv (Fan et al., 2018), 2018-05
49. CTRL: A Conditional Transformer Language Model for Controllable Generation — arXiv (Keskar et al., 2019), 2019-09
50. Mistral 7B — arXiv (Jiang et al., 2023), 2023-10
51. Longformer: The Long-Document Transformer — arXiv (Beltagy et al., 2020), 2020-04
52. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints — arXiv (Ainslie et al., 2023), 2023-05
53. Fast Transformer Decoding: One Write-Head is All You Need — arXiv (Shazeer et al., 2019), 2019-11
54. Text generation — Hugging Face documentation, undated (Transformers v5.21.0)
55. Lost in the Middle: How Language Models Use Long Contexts — arXiv (Liu et al., 2023), 2023-07
56. Optimization and Tuning — vLLM documentation, undated (latest developer preview)
57. Orca: A Distributed Serving System for Transformer-Based Generative Models — USENIX OSDI '22 (Yu et al., 2022), 2022-07
58. vLLM (documentation home) — vLLM documentation, undated (latest developer preview)
59. Automatic Prefix Caching — vLLM documentation, undated (latest developer preview)
60. TensorRT-LLM README.md — NVIDIA TensorRT-LLM (GitHub, main), undated (main branch)
61. llama.cpp README.md — ggml-org llama.cpp (GitHub, master), undated (master branch)
62. GGUF — Hugging Face documentation, undated
63. LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale — arXiv (Dettmers et al., 2022), 2022-08
64. Generation strategies — Hugging Face documentation, undated (Transformers v5.21.0)
65. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — arXiv (Frantar et al., 2022), 2022-10
66. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration — arXiv (Lin et al., 2023), 2023-06
67. Fast Inference from Transformers via Speculative Decoding — arXiv (Leviathan et al., 2022), 2022-11
68. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — arXiv (Dao et al., 2022), 2022-05
69. Generation (GenerationConfig) — Hugging Face documentation, undated (Transformers v5.21.0)
70. How caching works — Hugging Face documentation, undated (Transformers v5.21.0)
71. Cache strategies — Hugging Face documentation, undated (Transformers v5.21.0)
72. Text generation — OpenAI API documentation, undated
73. Efficient Memory Management for Large Language Model Serving with PagedAttention — arXiv (Kwon et al., 2023), 2023-09
74. Reproducibility — PyTorch documentation, undated (PyTorch 2.14)
75. Tokenizers — Hugging Face LLM Course, undated
76. Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates — arXiv (Kudo et al., 2018), 2018-04
77. GPT-2 — Hugging Face documentation, undated (Transformers main)
78. tiktoken README.md — OpenAI tiktoken (GitHub, main), undated (main)
79. CosineSimilarity — PyTorch documentation, undated (PyTorch 2.14)
80. Efficient Estimation of Word Representations in Vector Space (abstract) — arXiv (Mikolov et al., 2013), 2013-01
81. Efficient Estimation of Word Representations in Vector Space — arXiv (Mikolov et al., 2013), 2013-01
82. GloVe: Global Vectors for Word Representation (README.md) — Stanford NLP GloVe (GitHub, master), undated (master)
83. A New Pair of GloVes — arXiv (Carlson et al., 2025), 2025-07
84. Deep contextualized word representations — arXiv (Peters et al., 2018), 2018-02
85. Putting it all together — Hugging Face LLM Course, undated
86. Self-Attention with Relative Position Representations — arXiv (Shaw et al., 2018), 2018-03
87. RoFormer: Enhanced Transformer with Rotary Position Embedding — arXiv (Su et al., 2021), 2021-04
88. Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation (abstract) — arXiv (Press et al., 2021), 2021-08
89. A Neural Probabilistic Language Model (abstract) — Journal of Machine Learning Research (Bengio et al., 2003), 2003-02
90. GRU — PyTorch documentation, undated (PyTorch 2.14)
91. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation (abstract) — arXiv (Cho et al., 2014), 2014-06
92. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation — arXiv (Cho et al., 2014), 2014-06
93. Sequence to Sequence Learning with Neural Networks — arXiv (Sutskever et al., 2014), 2014-09
94. Normalization and pre-tokenization — Hugging Face LLM Course, undated
95. Neural Machine Translation by Jointly Learning to Align and Translate — arXiv (Bahdanau et al., 2014), 2014-09
96. Neural Machine Translation by Jointly Learning to Align and Translate (abstract) — arXiv (Bahdanau et al., 2014), 2014-09
97. Byte-Pair Encoding tokenization — Hugging Face LLM Course, undated
98. WordPiece tokenization — Hugging Face LLM Course, undated
99. Unigram tokenization — Hugging Face LLM Course, undated
100. Tokenization algorithms (formerly "Summary of the tokenizers") — Hugging Face documentation, undated (Transformers main, v5.21.0 latest release)
101. Neural Machine Translation of Rare Words with Subword Units — arXiv (Sennrich et al., 2015), 2015-08
102. SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing — arXiv (Kudo et al., 2018), 2018-08
103. Neural networks — Google Machine Learning Crash Course, 2025-08-25
104. Overfitting: L2 regularization — Google Machine Learning Crash Course, undated
105. Overfitting: Interpreting loss curves — Google Machine Learning Crash Course, undated
106. Linear — PyTorch documentation, undated (PyTorch 2.14)
107. ReLU — PyTorch documentation, undated (PyTorch 2.14)
108. LeakyReLU — PyTorch documentation, undated (PyTorch 2.14)
109. Sigmoid — PyTorch documentation, undated (PyTorch 2.14)
110. Tanh — PyTorch documentation, undated (PyTorch 2.14)
111. GELU — PyTorch documentation, undated (PyTorch 2.14)
112. Neural networks: Nodes and hidden layers — Google Machine Learning Crash Course, 2025-12-03
113. SiLU — PyTorch documentation, undated (PyTorch 2.14)
114. Softmax — PyTorch documentation, undated (PyTorch 2.14)
115. MSELoss — PyTorch documentation, undated (PyTorch 2.14)
116. L1Loss — PyTorch documentation, undated (PyTorch 2.14)
117. BCELoss — PyTorch documentation, undated (PyTorch 2.14)
118. BCEWithLogitsLoss — PyTorch documentation, undated (PyTorch 2.14)
119. CrossEntropyLoss — PyTorch documentation, undated (PyTorch 2.14)
120. Dropout — PyTorch documentation, undated (PyTorch 2.14)
121. torch.nn.utils.clip_grad_norm_ — PyTorch documentation, undated (PyTorch 2.14)
122. SGD — PyTorch documentation, undated (PyTorch 2.14)
123. Neural networks: Activation functions — Google Machine Learning Crash Course, 2025-08-25
124. Adam — PyTorch documentation, undated (PyTorch 2.14)
125. AdamW — PyTorch documentation, undated (PyTorch 2.14)
126. Autograd mechanics — PyTorch documentation, undated (PyTorch 2.14)
127. Gaussian Error Linear Units (GELUs) — arXiv (Hendrycks et al., 2016), 2016-06
128. Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning — arXiv (Elfwing et al., 2017), 2017-02
129. Searching for Activation Functions — arXiv (Ramachandran et al., 2017), 2017-10
130. Adam: A Method for Stochastic Optimization — arXiv (Kingma et al., 2014), 2014-12
131. On the difficulty of training Recurrent Neural Networks — arXiv (Pascanu et al., 2012), 2012-11
132. Decoupled Weight Decay Regularization — arXiv (Loshchilov et al., 2017), 2017-11
133. Deep Double Descent: Where Bigger Models and More Data Hurt — arXiv (Nakkiran et al., 2019), 2019-12
134. Neural Networks: Training using backpropagation — Google Machine Learning Crash Course, undated
135. Dropout: A Simple Way to Prevent Neural Networks from Overfitting — Journal of Machine Learning Research (Srivastava et al., 2014), 2014
136. 3.1. Cross-validation: evaluating estimator performance — scikit-learn documentation, undated (stable)
137. Build the Neural Network — PyTorch documentation, undated
138. Automatic Differentiation with torch.autograd — PyTorch documentation, undated
139. Optimizing Model Parameters — PyTorch documentation, undated
140. Module — PyTorch documentation, undated (PyTorch 2.14)
141. Neural networks: Multi-class classification — Google Machine Learning Crash Course, undated
142. Linear regression: Hyperparameters — Google Machine Learning Crash Course, undated
143. Logistic regression: Loss and regularization — Google Machine Learning Crash Course, undated
144. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models — arXiv (Wei et al., 2022), 2022-01
145. Similarity settings — Elasticsearch documentation, undated (current docs)
146. Passage Re-ranking with BERT — arXiv (Nogueira et al., 2019), 2019-01
147. Text splitter integrations — LangChain documentation, undated (current docs)
148. Splitting recursively — LangChain documentation, undated (current docs)
149. Lost in the Middle: How Language Models Use Long Contexts — arXiv (Liu et al., 2023), 2023-07
150. Benchmarking Large Language Models in Retrieval-Augmented Generation — arXiv (Chen et al., 2023), 2023-09
151. Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection — arXiv (Greshake et al., 2023), 2023-02
152. Toolformer: Language Models Can Teach Themselves to Use Tools — arXiv (Schick et al., 2023), 2023-02
153. ReAct: Synergizing Reasoning and Acting in Language Models — arXiv (Yao et al., 2022), 2022-10
154. Function calling — OpenAI API documentation, undated (current docs)
155. Self-Consistency Improves Chain of Thought Reasoning in Language Models — arXiv (Wang et al., 2022), 2022-03
156. Tool use with Claude — Claude Platform documentation, undated (current docs)
157. How tool use works — Claude Platform documentation, undated (current docs)
158. Tools (Model Context Protocol specification, version 2025-06-18) — Model Context Protocol specification, 2025-06-18
159. LLM01:2025 Prompt Injection — OWASP Gen AI Security Project, 2025
160. LLM06:2025 Excessive Agency — OWASP Gen AI Security Project, 2025
161. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale — arXiv (Dosovitskiy et al., 2020), 2020-10
162. Learning Transferable Visual Models From Natural Language Supervision — arXiv (Radford et al., 2021), 2021-02
163. Flamingo: a Visual Language Model for Few-Shot Learning — arXiv (Alayrac et al., 2022), 2022-04
164. Visual Instruction Tuning — arXiv (Liu et al., 2023), 2023-04 (v2 2023-12)
165. Robust Speech Recognition via Large-Scale Weak Supervision — arXiv (Radford et al., 2022), 2022-12
166. Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting — arXiv (Turpin et al., 2023), 2023-05
167. Denoising Diffusion Probabilistic Models — arXiv (Ho et al., 2020), 2020-06 (v2 2020-12)
168. High-Resolution Image Synthesis with Latent Diffusion Models — arXiv (Rombach et al., 2021), 2021-12
169. Survey of Hallucination in Natural Language Generation — arXiv (Ji et al., 2022), 2022-02
170. A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions — arXiv (Huang et al., 2023), 2023-11
171. TruthfulQA: Measuring How Models Mimic Human Falsehoods — arXiv (Lin et al., 2021), 2021-09
172. Language Models (Mostly) Know What They Know — arXiv (Kadavath et al., 2022), 2022-07
173. Why Language Models Hallucinate — arXiv (Kalai et al., 2025), 2025-09
174. NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark — arXiv (Sainz et al., 2023), 2023-10
175. Extracting Training Data from Large Language Models — arXiv (Carlini et al., 2020), 2020-12
176. Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1) — NIST, 2024-07
177. Measuring Faithfulness in Chain-of-Thought Reasoning — arXiv (Lanham et al., 2023), 2023-07
178. Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters — arXiv (Snell et al., 2024), 2024-08
179. PAL: Program-aided Language Models — arXiv (Gao et al., 2022), 2022-11
180. Training Verifiers to Solve Math Word Problems — arXiv (Cobbe et al., 2021), 2021-10
181. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — arXiv (Lewis et al., 2020), 2020-05
182. Dense Passage Retrieval for Open-Domain Question Answering — arXiv (Karpukhin et al., 2020), 2020-04
183. The Pile: An 800GB Dataset of Diverse Text for Language Modeling — arXiv (Gao et al., 2020), 2020-12
184. How do Transformers work? — Hugging Face LLM Course, undated
185. Finetuned Language Models Are Zero-Shot Learners — arXiv (Wei et al., 2021), 2021-09
186. Training language models to follow instructions with human feedback — arXiv (Ouyang et al., 2022), 2022-03
187. LoRA: Low-Rank Adaptation of Large Language Models — arXiv (Hu et al., 2021), 2021-06
188. QLoRA: Efficient Finetuning of Quantized LLMs — arXiv (Dettmers et al., 2023), 2023-05
189. PEFT — Hugging Face documentation, undated (PEFT v0.21.0 docs)
190. LoRA (conceptual guide) — Hugging Face documentation, undated (PEFT main docs)
191. Deep Reinforcement Learning from Human Preferences — arXiv (Christiano et al., 2017), 2017-06
192. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale — arXiv (Penedo et al., 2024), 2024-06
193. Direct Preference Optimization: Your Language Model is Secretly a Reward Model — arXiv (Rafailov et al., 2023), 2023-05
194. Constitutional AI: Harmlessness from AI Feedback — arXiv (Bai et al., 2022), 2022-12
195. Mixed Precision Training — arXiv (Micikevicius et al., 2017), 2017-10
196. Automatic Mixed Precision package - torch.amp — PyTorch documentation, undated (PyTorch 2.14 docs)
197. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models — arXiv (Rajbhandari et al., 2019), 2019-10
198. Training Deep Nets with Sublinear Memory Cost — arXiv (Chen et al., 2016), 2016-04
199. torch.utils.checkpoint — PyTorch documentation, undated (PyTorch 2.14 docs)
200. Distributed Data Parallel — PyTorch documentation, undated (PyTorch 2.14 docs)
201. Deduplicating Training Data Makes Language Models Better — arXiv (Lee et al., 2021), 2021-07
202. FullyShardedDataParallel — PyTorch documentation, undated (PyTorch 2.14 docs)
203. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism — arXiv (Shoeybi et al., 2019), 2019-09
204. GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism — arXiv (Huang et al., 2018), 2018-11
205. Performing gradient accumulation with Accelerate — Hugging Face documentation, undated (current docs)
206. Distributed communication package - torch.distributed — PyTorch documentation, undated (PyTorch 2.14 docs)
207. LLaMA: Open and Efficient Foundation Language Models — arXiv (Touvron et al., 2023), 2023-02
208. Causal language modeling — Hugging Face documentation, undated (current docs)
209. Training a causal language model from scratch — Hugging Face LLM Course, undated
210. MultiheadAttention — PyTorch documentation, undated (PyTorch 2.14 docs)
211. Are Sixteen Heads Really Better than One? — arXiv (Michel et al., 2019), 2019-05
212. Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned — arXiv (Voita et al., 2019), 2019-05
213. Layer Normalization — arXiv (Ba et al., 2016), 2016-07
214. Root Mean Square Layer Normalization — arXiv (Zhang et al., 2019), 2019-10
215. On Layer Normalization in the Transformer Architecture — arXiv (Xiong et al., 2020), 2020-02
216. GLU Variants Improve Transformer — arXiv (Shazeer et al., 2020), 2020-02
217. Scaling Laws for Neural Language Models — arXiv (Kaplan et al., 2020), 2020-01
218. Training Compute-Optimal Large Language Models — arXiv (Hoffmann et al., 2022), 2022-03
219. BERT Rediscovers the Classical NLP Pipeline — arXiv (Tenney et al., 2019), 2019-05
220. Using the Output Embedding to Improve Language Models — arXiv (Press et al., 2016), 2016-08
221. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer — arXiv (Raffel et al., 2019), 2019-10
222. Transformer Architectures — Hugging Face LLM Course, undated
223. Attention is not Explanation — arXiv (Jain et al., 2019), 2019-02
224. Attention is not not Explanation — arXiv (Wiegreffe et al., 2019), 2019-08
225. torch.nn.functional.scaled_dot_product_attention — PyTorch documentation, undated (PyTorch 2.14 docs)