The math inside a transformer

From a signal to an answer, one calculation at a time

Chatbots are built from transformers. Here is a real one, trained on a handful of short sentences: small enough to show every number it uses, with the same design as the big ones. Follow a sentence through all 11 steps, from words to numbers to its guess for the next word. It only ever adds and multiplies.

Try this first
  1. Keep the sentence “the dog sat on the” and read down the steps. Follow the last word, “the”, as it becomes numbers and, at the end, a guess: “mat”.
  2. In step 7, see which earlier words “the” listens to most. Then pick the example “the dog ran to the”: the last word is the same, but the guess becomes “park”, because it listened to different words.
  3. In step 11, slide the temperature up to 3 and watch the chances spread out, then down to 0.2 and watch them sharpen.
  4. Press “Add the answer” to put the guess on the end of the sentence and run every step again.

Your sentence

Press a word to follow it through every step. The last word is the one whose guess is the next word.

Guess for the word after “the dog sat on the”: mat

  1. Step 1: Words become token numbers

    A computer cannot read words, only numbers. The model knows a fixed list of 15 words, its vocabulary. Each word is replaced by its place in that list, called its token number.

    token number = the word’s place in the vocabulary (counting from 0)

    • the1
    • dog4
    • sat6
    • on9
    • the1

    Vocabulary: 0 . · 1 the · 2 a · 3 cat · 4 dog · 5 bird · 6 sat · 7 ran · 8 flew · 9 on · 10 to · 11 in · 12 mat · 13 park · 14 sky

  2. Step 2: Each token picks a row of numbers

    The model keeps a table with one row of 8 numbers for every word it knows. The rows were learned in training, so words used in similar ways get similar rows. Looking up a word’s row is called embedding.

    e = row 1 of the embedding table (15 rows × 8 numbers)

    e for “the”

  3. Step 3: Add the position signal

    The next steps mix words together without knowing their order. So each word gets a second row added to it: a pattern of sine and cosine waves that is different at every position, like a timestamp. This signal is how the model tells the first word from the last.

    p = sin(position ÷ 10000^(slot÷8)) in even slots, cos(…) in odd slots · x = e + p

    • slot 1: sine, fastest
    • slot 2: cosine, fastest
    • slot 3: sine, slower
    • slot 4: cosine, slower
    • the word you follow
    e the word
    p position 5 signal
    x = e + p

    Slot 1, for example: sin(4 ÷ 1) = −0.76. The waves count positions from 0, so position 5 uses 4.

  4. Step 4: Even out the numbers

    Before each big step, a word’s 8 numbers are rescaled so they average 0 and spread out by a standard amount. This stops numbers from growing out of control. Two learned rows, a scale and a shift, then adjust them. This is called layer norm.

    n = (x − mean) ÷ √(variance + 0.00001) × scale + shift

    mean = 0.240 · variance = 0.600 · √variance = 0.775

    x input
    (x − mean) ÷ √variance
    n after the learned scale and shift

  5. Step 5: A question, a label and a message

    Each word makes three new rows by multiplying its numbers by three learned tables: a query (what am I looking for?), a key (what do I offer?) and a value (what will I pass on?). Multiplying by a table is just many multiply-and-add steps. One is written out below so you can check it.

    q = n × W_Q · k = n × W_K · v = n × W_V (each W is an 8 × 8 table)

    q query
    k key
    v value

    q’s first number = −1.45×0.33 + −0.50×(−0.20) + 0.18×0.45 + −0.77×0.05 + 0.52×0.37 + 0.52×(−0.41) + −0.12×(−0.06) + 1.95×(−0.21) = −0.76

  6. Step 6: How well does each word match?

    To decide which words matter, the word you follow compares its query with every key: multiply matching numbers and add them up (a dot product). A big result means a good match. It is divided by √8 to keep scores moderate. Words that come later are hidden, because the model must not peek at the future.

    score = (q · k) ÷ √8 · later words get −∞, so they can never be picked

    • “the” position 10.78 ÷ 2.83 = 0.28
    • “dog” position 20.19 ÷ 2.83 = 0.07
    • “sat” position 32.62 ÷ 2.83 = 0.93
    • “on” position 41.02 ÷ 2.83 = 0.36
    • “the” position 50.98 ÷ 2.83 = 0.35

    q · k for “the” = −0.76×(−0.22) + 0.37×(−1.10) + −0.55×(−0.03) + 0.45×0.14 + −0.11×(−0.19) + 0.17×0.18 + 0.43×(−0.36) + −0.93×(−1.13) = 0.78

  7. Step 7: Turn scores into shares

    Softmax turns the scores into percentages that add up to 100%: raise e (about 2.718) to the power of each score, then divide each one by their total. These percentages are the attention: how much the word listens to each earlier word.

    share = e^score ÷ (sum of e^score over the visible words)

    • “the” 1e^0.28 = 1.32 → 17.0%
    • “dog” 2e^0.07 = 1.07 → 13.8%
    • “sat” 3e^0.93 = 2.53 → 32.6%
    • “on” 4e^0.36 = 1.44 → 18.5%
    • “the” 5e^0.35 = 1.42 → 18.2%

    Total of e^score: 7.77. “the” listens most to “sat” at position 3.

  8. Step 8: Mix what it heard, and keep the original

    The word’s new row is the values of the words it listened to, blended by those percentages, then passed through one more learned table. The result is added on top of the word’s original row, so nothing is lost. Adding the input back like this is called a residual connection.

    blend = Σ share × v · o = blend × W_O · new x = x + o

    blend of the values

    blend’s first number = 0.17×(−0.37) + 0.14×1.38 + 0.33×(−0.08) + 0.18×0.44 + 0.18×0.01 = 0.18

    o after W_O
    x before
    new x = x + o

  9. Step 9: Think it over, one word at a time

    The row is evened out again and stretched to 16 numbers by a learned table. Every negative number is then set to 0 (this on/off rule is called ReLU, and it lets the model learn more than straight lines). A last table shrinks it back to 8 numbers, which are added on again.

    h = ReLU(norm(x) × W₁ + b₁) · f = h × W₂ + b₂ · new x = x + f · ReLU(a) = the larger of a and 0

    norm(x)
    before ReLU 16 numbers
    h after ReLU

    ReLU switched off 9 of the 16 numbers (every negative one became 0).

    f back to 8
    new x = x + f

  10. Step 10: A score for every word it knows

    One final evening-out, then one more table turns the 8 numbers into 15 scores, one for each word in the vocabulary. A higher score means the model thinks that word is more likely to come next. These raw scores are called logits.

    logits = norm(x) × U (U is an 8 × 15 table: one column per word)

    norm(x)
    • full stop (.)1.58
    • the−2.94
    • a2.58
    • cat−3.06
    • dog−1.42
    • bird−4.01
    • sat1.55
    • ran0.70
    • flew−0.18
    • on−3.59
    • to1.26
    • in−7.26
    • mat14.56
    • park2.62
    • sky1.12

  11. Step 11: From scores to chances, and the answer

    Softmax again, this time over the 15 scores, gives the chance of each possible next word. Temperature divides the scores first: below 1 makes the top word even more likely, above 1 spreads the chances out. The most likely word is the answer. Add it to the sentence and the whole journey runs again, which is how chatbots write, one word at a time.

    chance = e^(logit ÷ T) ÷ (sum of e^(logit ÷ T) over all 15 words) · T is the temperature

    Low: confident. 1: the model as trained. High: adventurous.

    • mat100%
    • park0%
    • a0%
    • full stop (.)0%
    • sat0%
    • to0%

    The other 9 words share the remaining 0%.

    Answer: the word after “the dog sat on the” is most likely “mat”.

Word list
Token
A word (or piece of a word) turned into a number the model can work with.
Vocabulary
The fixed list of tokens the model knows. This model knows 15 words.
Vector
A row of numbers. Here every word is a row of 8 numbers.
Embedding
The learned row of numbers that stands for a word.
Position signal
Sine and cosine waves added to each word so the model knows where in the sentence it is.
Sine and cosine
Smooth waves that go up and down between −1 and 1. Different speeds give each position a unique pattern.
Layer norm
Rescaling a row so its numbers average 0 and have a standard spread, then applying a learned scale and shift.
Mean
The average: add the numbers up and divide by how many there are.
Variance
How spread out numbers are: the average of each number’s squared distance from the mean.
Matrix
A table of numbers. The model’s knowledge lives in matrices it learned in training.
Multiplying by a matrix
Making each new number by multiplying every input number by a table entry and adding the results.
Dot product
Multiply two rows number by number and add it all up. Large when the rows point the same way.
Query, key, value
Three rows each word makes: what it is looking for, what it offers to others, and what it passes on.
Attention
The step where each word looks at earlier words and takes in a blend of their values.
Softmax
Turns any list of scores into percentages that add up to 100%, keeping bigger scores bigger.
Residual connection
Adding a step’s input back to its output, so information is kept and only adjusted.
Feed-forward
The step that works on each word on its own: stretch, switch off negatives, shrink back.
ReLU
The rule “keep positive numbers, turn negative ones into 0”.
Logits
The raw score the model gives each word in its vocabulary before they become chances.
Probability
A chance between 0% and 100%. All the next-word chances add up to 100%.
Temperature
A number the scores are divided by before softmax. Low means confident, high means adventurous.
Weights
All the learned numbers in the model’s tables. Training is the process of finding them.
What this model learned from

It has 8 numbers per word, 1 attention head and 1 layer, and was trained once on these 7 sentences. Big chatbots use the same steps with thousands of numbers per word, many heads and dozens of layers.

  • the cat sat on the mat .
  • a cat sat on a mat .
  • the dog ran to the park .
  • a dog ran to a park .
  • the dog sat on the mat .
  • the bird flew in the sky .
  • a bird flew in the sky .