Build an LLM from scratch
Train a small language model on your own device and watch it learn to write short stories.
Seven short steps. You first see what a language model does, then peek at the maths and the design inside it, and finally build and train one yourself. No experience needed.
Step 1 See what a language model does
A language model only ever guesses the next piece of text, and seeing that first makes the rest easier to follow.
Open step 1: See what a language model doesStep 2 Follow the maths inside
Every guess comes from plain arithmetic on lists of numbers, and you can follow each calculation by hand.
Open step 2: Follow the maths insideStep 3 Explore the design of a transformer
A transformer is the kind of model inside most of today’s chatbots. Its design decides what the model can notice, so you will see which parts matter and why.
Open step 3: Explore the design of a transformerStep 4 Get the training stories
A model learns only from the text it reads, so you choose and load its reading material first.
Open step 4: Get the training storiesStep 5 Cut the text into tokens
The model reads small pieces called tokens, not whole sentences, so the text must be cut up and numbered.
Open step 5: Cut the text into tokensStep 6 Train your model
Training is the model guessing, checking and improving many thousands of times until its guesses get good. The loss is a number for how wrong its guesses are: lower is better.
Open step 6: Train your modelStep 7 Write a story with it
This is the payoff: your own model, trained on your device, writing text it has never seen.
Open step 7: Write a story with it