Question in use: What can the town do on Wednesdays?. Method: Hybrid (RRF). Top passage: Saturday market, part 1.
Search, answer, and measure
A question-answering tool built on your own documents has two steps: search for the right passages, then let a language model write the answer from them. If the search misses, the answer is wrong, however good the model is. So this lab lets you try the whole chain once, and then measures every search method fairly on 36 questions. Reranking was never behind on these 36 questions, but it needs a second model and only reorders the hybrid top 10; among single searches, none wins every kind of question. These 36 questions are few and were written by hand for this made-up town, so the numbers show a pattern, not a rule for every search. Every number is shown.
Try this first
- Open “Read the 32 passages” below. They describe a made-up island, Fernport, cut into short pieces.
- In The pipeline, press “What can the town do on Wednesdays?” and choose BM25, then Vector search. Compare the 3 passages.
- Press “Can I bring my pet to the harbour?” and compare the same two methods again.
- In Answer, press “Ask the model” (it needs a one-time download) and check the answer against the passages.
- In Measure, open a recall@3 cell and read the arithmetic. Then open the same cell in the keyword-style rows and the meaning-style rows.
- In Choose, read which technique the numbers point to for which kind of question.
Read the 32 passages
Fernport is a made-up island. Its 10 documents are cut into 32 passages of 2 sentences each.
- Fernport Bakery, part 1
- Fernport Bakery stands on Harbour Street. It is run by Mira Okafor.
- Fernport Bakery, part 2
- Mira is famous for her dark rye bread, baked only on Tuesdays. She learned to bake from Tomas Reyes when she was sixteen.
- Fernport Bakery, part 3
- The shop opens at 7 in the morning and closes when the bread is gone.
- Fernport Lighthouse, part 1
- The Fernport Lighthouse was built in 1891 on the north cliff. Its lamp can be seen 30 kilometres out at sea.
- Fernport Lighthouse, part 2
- Guided tours run on Saturdays and Sundays at 10:00 and 14:00. The guide is Tomas Reyes, who has climbed the lighthouse’s 120 stairs every week for 20 years.
- Fernport Lighthouse, part 3
- The lighthouse is closed in January for repairs.
- Ferry timetable, part 1
- The ferry crosses from Millbay on the mainland to Fernport in 40 minutes. A ferry fare costs 6 coins, and children ride free.
- Ferry timetable, part 2
- Bikes travel for 1 coin more. December to February is the quiet season.
- Ferry timetable, part 3
- Then there is only one crossing a day, at 10:00 from Millbay. The rest of the year, the ferry leaves Millbay at 8:00, 12:00 and 17:00.
- Lantern Festival, part 1
- The Lantern Festival lights up Fernport every October. People hang paper lanterns along Harbour Street for a week.
- Lantern Festival, part 2
- On the last night of the festival, everyone walks to the north cliff to watch the lighthouse lamp switch on.
- Visitor rules, part 1
- Visitors may camp only at the campsite behind the school. Fires are allowed on the beach, but never in the woods.
- Visitor rules, part 2
- Dogs must stay on a lead near the harbour.
- Harbour guide, part 1
- The Fernport harbour is the busiest place in town. The harbour has a long stone wall, and the harbour wall keeps the water calm inside.
- Harbour guide, part 2
- Fishers bring their catch to the harbour at dawn, and the harbour market sells it fresh by seven. Every morning the harbour fills with gulls, nets and carts.
- Harbour guide, part 3
- The ferry from the mainland ties up at the harbour, at the far end of the east pier. A small harbour office hands out maps and answers questions.
- Harbour guide, part 4
- The harbour café serves hot soup to anyone who is cold. Walk along the harbour edge at sunset to see the water turn orange.
- Harbour guide, part 5
- At night the harbour is lit by a row of yellow lamps, and the harbour is quiet. The harbour crane lifts crates of apples, wool and salt from the quay onto carts.
- Harbour guide, part 6
- Young children feed the gulls near the harbour steps, while old fishers mend their nets and trade stories about the weather. Two painted sheds beside the harbour store ropes, buoys and lamps.
- Harbour guide, part 7
- A harbour bell rings at noon, and again when the tide turns. Anyone who loves the sea should spend a slow afternoon at the harbour.
- Fernport School, part 1
- Fernport School teaches about forty children, from age five to twelve. Classes start at 9 in the morning and end at 3.
- Fernport School, part 2
- Every child learns to swim in the sea pool, and older children learn to sail. The school library is open to the whole town on Wednesdays.
- Fernport School, part 3
- The school choir sings at the end of every term.
- Saturday market, part 1
- The Saturday market fills the square beside the town hall. Farmers sell apples, honey and goat cheese.
- Saturday market, part 2
- Weavers sell wool scarves and hats. A woman named Ida Lund sells smoked fish from a blue stall.
- Saturday market, part 3
- The market opens at 8 in the morning and packs up at noon.
- Weather and seasons, part 1
- Fernport is mild and damp. Summer days reach about 22 degrees, and the sea is warm enough to swim in July and August.
- Weather and seasons, part 2
- Autumn brings strong winds from the west. Fog often covers the cliffs in the early morning.
- Weather and seasons, part 3
- It snows only once or twice a year, and the snow melts by noon.
- History of Fernport, part 1
- Fernport began as a fishing village about 400 years ago. The first stone houses were built in 1650, from rock taken from the cliffs.
- History of Fernport, part 2
- A great storm in 1702 washed away the old pier, and the town built a new one. In 1950 a bridge was planned, but the islanders voted against it.
- History of Fernport, part 3
- Today about 900 people live in Fernport.
Chapter 1: The pipeline: search first, then answer
Question in use: What can the town do on Wednesdays?
A pipeline is a chain of steps where each step feeds the next. Here there are two. Retrieve: a search picks the 3 passages most likely to hold the answer. Generate: a language model reads those passages and writes the answer. Pick a question and a search method below. The method decides which 3 passages go into the prompt, and the prompt is exactly the text the model will receive. You can also switch MMR on (from Lab 09, lambda 0.5): it re-picks 3 passages from the method’s top 10 so they differ more from each other. Reranking uses scores saved in advance, so this lab works only with the four example questions.
Press “What can the town do on Wednesdays?” and choose BM25, then Vector search. The top 3 change, and so does the prompt. Then press “Can I bring my pet to the harbour?” and compare again.
Retrieve: the top 3 passages
- Saturday market, part 1
The Saturday market fills the square beside the town hall. Farmers sell apples, honey and goat cheese. - Fernport School, part 2
Every child learns to swim in the sea pool, and older children learn to sail. The school library is open to the whole town on Wednesdays. - Ferry timetable, part 3
Then there is only one crossing a day, at 10:00 from Millbay. The rest of the year, the ferry leaves Millbay at 8:00, 12:00 and 17:00.
Right passage: Fernport School, part 2.
The full prompt the model receives
Answer the question using only the passages below. If they do not contain the answer, say you do not know. [1] Saturday market: The Saturday market fills the square beside the town hall. Farmers sell apples, honey and goat cheese. [2] Fernport School: Every child learns to swim in the sea pool, and older children learn to sail. The school library is open to the whole town on Wednesdays. [3] Ferry timetable: Then there is only one crossing a day, at 10:00 from Millbay. The rest of the year, the ferry leaves Millbay at 8:00, 12:00 and 17:00. Question: What can the town do on Wednesdays?
Chapter 2: Answer: ask the model
Question in use: What can the town do on Wednesdays?
Now the generate step. The button sends the prompt from the chapter above to a small language model that runs on your device (the same one as Lab 05). The first time it downloads the model, which can take a while. The prompt tells it to use only the passages and to say it does not know otherwise. A small model can still make mistakes, and it can invent facts that are not in the passages (a hallucination). So always check its answer against the passages.
Ask with BM25 and with the method that put the right passage first. If the right passage is missing from the top 3, the model has nothing to answer from.
Downloads Qwen3 0.6B (about 580 MB) the first time, then answers on your device. It needs huggingface.co. It will answer using the prompt from The pipeline chapter.
Chapter 3: Measure: score every method
Reading one answer tells you little. An evaluation set is a list of questions where the right passages (the ground truth) are known in advance, so every method can be scored the same way. A passage is relevant when it holds the answer. Recall@k is the share of the relevant passages found in the top k. For one question, recall@3 = (relevant passages in the top 3) ÷ (all relevant passages). Reciprocal rank is 1 ÷ (the rank of the first relevant passage), or 0 if none is in the top 10. MRR is the mean of that number over all questions. Each score below is the mean over the questions in its group. Open any cell to see every question’s rank and the sum behind the number. Half the questions reuse the passage’s own words (keyword style) and half are paraphrases (meaning style), so the table also shows each style on its own.
Open the recall@3 cell for BM25 in the “18 keyword-style questions” rows, then the same cell in the “18 meaning-style questions” rows. Compare the two sums.
Recall at 3 on all questions: BM25 0.704, Vector search 0.856, Hybrid (RRF) 0.912, Hybrid, then rerank 0.981.
Press a number to open the sum behind it. Press it again to close.
Scroll sideways to see every column.
| Questions | Method | Recall@1 | Recall@3 | Recall@5 | MRR |
|---|---|---|---|---|---|
| All 36 questions | BM25 | ||||
| All 36 questions | Vector search | ||||
| All 36 questions | Hybrid (RRF) | ||||
| All 36 questions | Hybrid, then rerank | ||||
| 18 keyword-style questions | BM25 | ||||
| 18 keyword-style questions | Vector search | ||||
| 18 keyword-style questions | Hybrid (RRF) | ||||
| 18 keyword-style questions | Hybrid, then rerank | ||||
| 18 meaning-style questions | BM25 | ||||
| 18 meaning-style questions | Vector search | ||||
| 18 meaning-style questions | Hybrid (RRF) | ||||
| 18 meaning-style questions | Hybrid, then rerank |
These 36 questions are few and were written by hand for this made-up town, so the numbers show a pattern, not a rule for every search.
Chapter 4: Choose: which technique when?
This guide follows from the table above. On these 36 questions: for keyword-style questions (exact names, days and numbers) BM25 found every right passage in the top 3 (recall@3 1.000), while vector search found 0.750. For meaning-style questions (paraphrases) it flipped: vector search 0.963, BM25 0.407. Hybrid search (RRF) was between them on both styles (0.972 and 0.852) and had a better MRR overall (0.869) than either alone (BM25 0.711, vector 0.839). Reranking the hybrid top 10 with a second model scored best overall (recall@3 0.981, MRR 0.986), but it costs a second model run for every question. The rules below are what these numbers suggest. The 36 questions are few and hand-made, so they show a pattern, not a rule. Reranking was never behind on these 36 questions, but it needs a second model and only reorders the hybrid top 10; among single searches, none wins every kind of question. Test on your own questions.
Find the rule below that matches the questions you expect, then check it against your own questions.
- Exact names, days and numbers: keyword search (BM25).
- Paraphrases, where the question shares few words with the passage: vector search.
- Unsure, or a mix of both: hybrid search (RRF).
- You can afford a second model: rerank the top 10 of the hybrid list.
Word list
- Pipeline
- A chain of steps where the output of one step is the input of the next. Here: retrieve, then generate.
- Retrieve
- The search step: find the few passages most likely to hold the answer.
- Generate
- The writing step: a language model reads the question and the retrieved passages and writes an answer.
- Evaluation set
- A list of questions whose right passages are known in advance, used to score a search method. Here: 36 questions, 18 keyword style and 18 meaning style.
- Ground truth
- The answer key: the passages a person decided are the right ones for each question.
- Relevant
- A passage that holds the answer to the question, according to the ground truth.
- Recall@k
- For one question: relevant passages in the top k ÷ all relevant passages. The table shows the mean over a group of questions.
- Reciprocal rank
- 1 ÷ the rank of the first relevant passage: 1 for rank 1, 0.5 for rank 2, 0.333 for rank 3. It is 0 if none is in the top 10.
- MRR
- Mean reciprocal rank: the average of the reciprocal rank over all questions in a group.
- Rank
- A position in a list. Rank 1 is the best result.
- Hallucination
- An answer that sounds sure but states something the passages do not say. Small models do this, so check answers against the passages.
- BM25
- A keyword search from Lab 06 that scores the words a passage and the question share, giving more weight to rare words.
- Vector search
- A search from Lab 08 that compares the meaning of the question with the meaning of each passage, both written as lists of numbers.
- Hybrid search (RRF)
- Running BM25 and vector search and merging their lists by rank: each list gives a passage 1 ÷ (60 + rank) points, and the points are added.
- Reranking
- Reordering the top 10 candidates with a slower, more careful model (a cross-encoder) that reads the question and one passage together.
- Lambda
- In MMR, how much to weigh relevance against variety. 1 means only relevance, 0 means only variety, and 0.5 weighs them equally.
- MMR
- Maximal marginal relevance: pick passages one at a time, balancing closeness to the question against repeating a passage already picked.
- Prompt
- The text sent to the language model: the instruction, the passages and the question.