Train a model through its life cycle

Take one tiny model through the stages that turn raw text into a helpful assistant, and see what each stage gives and what it costs.

Seven steps, five ready now. You pre-train a small model, mid-train it on sums, fine-tune it to follow a request, teach it people’s taste with comparisons (RLHF), then do the same with DPO. The last two steps are coming soon. A tiny model shows the idea, not real-scale quality. No experience needed.

7 steps · about 57 minutes

2 more steps are coming soon. The minutes and the count above only include the steps you can open now.

1. Step 1 Pre-training: learn from raw text

   Pre-training is where a model first learns language, by guessing the next piece of text in millions of stories. The small size is the one this program uses. You can train it yourself, or skip it: the ready-made base model is fine to start the next steps from.

   About 10 minutes

   Open step 1: Pre-training: learn from raw text
   Tick each one when it is true for you

   I trained the small size in Lab 04.I watched the loss number go down.

2. Step 2 Mid-training: teach a format

   Mid-training keeps training on a new mix of text, here simple sums. You see what it gains and what it forgets.

   About 10 minutes

   Open step 2: Mid-training: teach a format
   Tick each one when it is true for you

   I saw my model answer sums with number words after mid-training.I saw its stories get worse at 100% sums (the story loss on the chart and fluency in the scorecard).I saw its sums are still near guessing.

3. Step 3 Fine-tuning: follow a request

   Supervised fine-tuning (SFT) shows the model many requests with good answers, so it learns to follow the request format.

   About 12 minutes

   Open step 3: Fine-tuning: follow a request
   Tick each one when it is true for you

   I saw the name used much more often after SFT (88% for the ready-made model).I read that the loss is counted on the answer only.I tried the “train the prompt too” switch.

4. Step 4 Human feedback and RLHF

   People compare two answers and say which they like better. A reward model learns their taste, and reinforcement learning (RL) trains the model towards it, on a leash so it does not cheat.

   About 15 minutes

   Open step 4: Human feedback and RLHF
   Tick each one when it is true for you

   I picked the better story in some pairs and saw how often I agreed with the panel.I saw RLHF raise Kindness from 1.72 to 2.16 (the ready-made model).I saw what happens to the stories with no leash (β 0).

5. Step 5 DPO: learn from preferred answers

   Direct preference optimisation (DPO) learns from pairs of better and worse answers in one stage, with no reward model.

   About 10 minutes

   Open step 5: DPO: learn from preferred answers
   Tick each one when it is true for you

   I saw DPO raise Kindness from 1.72 to 2.28 (the ready-made model).I compared DPO with RLHF on Kindness and time.I read my own taste profile and trained on my labels.

6. Step 6 RL with verifiable rewards Coming soon

   Reinforcement learning where a checker, such as a sum, scores each answer. Coming soon.

7. Step 7 The whole life cycle Coming soon

   Put every stage together and compare the model after each one. Coming soon.