Question in use: How much is a boat ticket?. Top hybrid passage: Ferry timetable, part 1.
How to combine searches and take a second look
Keyword search is exact about words and numbers. Vector search understands meaning. Each one misses things the other finds. Hybrid search runs both and merges their lists. A reranker then takes a closer look at the best few. You will see exactly how the lists are merged, what filters and variety do, and a fair scoreboard on 36 questions. No single search wins every kind of question; reranking was never behind on this small set, but it needs a second model. These 36 questions are few and were written by hand for this made-up town, so the numbers show a pattern, not a rule for every search. Every number is shown.
Try this first
- Open “Read the 32 passages” below. They describe a made-up island, Fernport, cut into short pieces.
- In Why use two searches, press “What can the town do on Wednesdays?” (BM25 wins), then “Can I bring my pet to the harbour?” (vector search wins).
- In Reciprocal rank fusion, press “What time does the market open?” and add the two fractions in one row by hand.
- In Weighted fusion, set alpha to 0, then to 1, and see the list match BM25, then vector search.
- In Filters, press “How much is a boat ticket?” and tick only “event” to see the empty message.
- In Reranking, press “When can I climb the tower by the sea?” and see the order change.
- In Diversity, set lambda to 0.5 on the boat question and see what the variety costs.
- In Compare, read which search wins on which type of question.
Read the 32 passages
Fernport is a made-up island. Its 10 documents are cut into 32 passages of 2 sentences each.
- Fernport Bakery, part 1
- Fernport Bakery stands on Harbour Street. It is run by Mira Okafor.
- Fernport Bakery, part 2
- Mira is famous for her dark rye bread, baked only on Tuesdays. She learned to bake from Tomas Reyes when she was sixteen.
- Fernport Bakery, part 3
- The shop opens at 7 in the morning and closes when the bread is gone.
- Fernport Lighthouse, part 1
- The Fernport Lighthouse was built in 1891 on the north cliff. Its lamp can be seen 30 kilometres out at sea.
- Fernport Lighthouse, part 2
- Guided tours run on Saturdays and Sundays at 10:00 and 14:00. The guide is Tomas Reyes, who has climbed the lighthouse’s 120 stairs every week for 20 years.
- Fernport Lighthouse, part 3
- The lighthouse is closed in January for repairs.
- Ferry timetable, part 1
- The ferry crosses from Millbay on the mainland to Fernport in 40 minutes. A ferry fare costs 6 coins, and children ride free.
- Ferry timetable, part 2
- Bikes travel for 1 coin more. December to February is the quiet season.
- Ferry timetable, part 3
- Then there is only one crossing a day, at 10:00 from Millbay. The rest of the year, the ferry leaves Millbay at 8:00, 12:00 and 17:00.
- Lantern Festival, part 1
- The Lantern Festival lights up Fernport every October. People hang paper lanterns along Harbour Street for a week.
- Lantern Festival, part 2
- On the last night of the festival, everyone walks to the north cliff to watch the lighthouse lamp switch on.
- Visitor rules, part 1
- Visitors may camp only at the campsite behind the school. Fires are allowed on the beach, but never in the woods.
- Visitor rules, part 2
- Dogs must stay on a lead near the harbour.
- Harbour guide, part 1
- The Fernport harbour is the busiest place in town. The harbour has a long stone wall, and the harbour wall keeps the water calm inside.
- Harbour guide, part 2
- Fishers bring their catch to the harbour at dawn, and the harbour market sells it fresh by seven. Every morning the harbour fills with gulls, nets and carts.
- Harbour guide, part 3
- The ferry from the mainland ties up at the harbour, at the far end of the east pier. A small harbour office hands out maps and answers questions.
- Harbour guide, part 4
- The harbour café serves hot soup to anyone who is cold. Walk along the harbour edge at sunset to see the water turn orange.
- Harbour guide, part 5
- At night the harbour is lit by a row of yellow lamps, and the harbour is quiet. The harbour crane lifts crates of apples, wool and salt from the quay onto carts.
- Harbour guide, part 6
- Young children feed the gulls near the harbour steps, while old fishers mend their nets and trade stories about the weather. Two painted sheds beside the harbour store ropes, buoys and lamps.
- Harbour guide, part 7
- A harbour bell rings at noon, and again when the tide turns. Anyone who loves the sea should spend a slow afternoon at the harbour.
- Fernport School, part 1
- Fernport School teaches about forty children, from age five to twelve. Classes start at 9 in the morning and end at 3.
- Fernport School, part 2
- Every child learns to swim in the sea pool, and older children learn to sail. The school library is open to the whole town on Wednesdays.
- Fernport School, part 3
- The school choir sings at the end of every term.
- Saturday market, part 1
- The Saturday market fills the square beside the town hall. Farmers sell apples, honey and goat cheese.
- Saturday market, part 2
- Weavers sell wool scarves and hats. A woman named Ida Lund sells smoked fish from a blue stall.
- Saturday market, part 3
- The market opens at 8 in the morning and packs up at noon.
- Weather and seasons, part 1
- Fernport is mild and damp. Summer days reach about 22 degrees, and the sea is warm enough to swim in July and August.
- Weather and seasons, part 2
- Autumn brings strong winds from the west. Fog often covers the cliffs in the early morning.
- Weather and seasons, part 3
- It snows only once or twice a year, and the snow melts by noon.
- History of Fernport, part 1
- Fernport began as a fishing village about 400 years ago. The first stone houses were built in 1650, from rock taken from the cliffs.
- History of Fernport, part 2
- A great storm in 1702 washed away the old pier, and the town built a new one. In 1950 a bridge was planned, but the islanders voted against it.
- History of Fernport, part 3
- Today about 900 people live in Fernport.
Chapter 1: Why use two searches at once?
Question in use: How much is a boat ticket?
Keyword search (BM25, from Lab 06) and vector search (from Lab 08) fail in different places. BM25 is exact about words, days, names and numbers, but it cannot link “boat ticket” to a ferry fare. Vector search understands meaning, but it can blur small exact details. Hybrid search runs both and merges the two ranked lists. The table shows, for the question in use, the top 3 passages from each search and from the hybrid merge. For the two questions below, the right answer is known, so the table also says where each search ranked it.
Press “What can the town do on Wednesdays?”. The right passage (Fernport School, part 2) is ranked 1 by BM25 but only 14 by vector search. Then press “Can I bring my pet to the harbour?”: vector search ranks the right passage (Visitor rules, part 2) 1, and BM25 ranks it 9. Neither search is always better.
Top passage: BM25 none, vector search Ferry timetable, part 1, hybrid Ferry timetable, part 1.
BM25
Shared words.
No passage uses these words.
Vector search
Closeness of meaning.
- Ferry timetable, part 1
- Ferry timetable, part 3
- Harbour guide, part 2
Hybrid (RRF)
Both lists merged.
- Ferry timetable, part 1
- Ferry timetable, part 3
- Harbour guide, part 2
Right passage: Ferry timetable, part 1. Its best rank in BM25: not in the top 20. In vector search: 1. In the hybrid list: 1.
Chapter 2: Reciprocal rank fusion (RRF)
Question in use: How much is a boat ticket?
The two searches use different scales, so their scores cannot simply be added. Rank fusion ignores the scores and uses only the positions, called ranks (1 is the best). In reciprocal rank fusion, every list that holds a passage gives it 1 ÷ (60 + rank) points, and the points are added. A passage that is missing from a list gets nothing from that list. The number 60 is called k: it stops a rank 1 from being worth far more than a rank 2. Each list here is its top 10.
Press “What time does the market open?” and check one total by hand: add the two small fractions in a row. A passage that both lists rank high wins, even if neither list put it first by a wide margin.
Top passage after fusion: Ferry timetable, part 1, score 0.01639.
Scroll sideways to see every column.
| Passage | BM25 rank | BM25 part: 1 ÷ (60 + rank) | Vector rank | Vector part: 1 ÷ (60 + rank) | Total score |
|---|---|---|---|---|---|
| Ferry timetable, part 1 | not in the top 10 | 0 | 1 | 1 ÷ 61 = 0.01639 | 0.01639 |
| Ferry timetable, part 3 | not in the top 10 | 0 | 2 | 1 ÷ 62 = 0.01613 | 0.01613 |
| Harbour guide, part 2 | not in the top 10 | 0 | 3 | 1 ÷ 63 = 0.01587 | 0.01587 |
| Harbour guide, part 7 | not in the top 10 | 0 | 4 | 1 ÷ 64 = 0.01563 | 0.01563 |
| Harbour guide, part 3 | not in the top 10 | 0 | 5 | 1 ÷ 65 = 0.01538 | 0.01538 |
| History of Fernport, part 2 | not in the top 10 | 0 | 6 | 1 ÷ 66 = 0.01515 | 0.01515 |
| Harbour guide, part 5 | not in the top 10 | 0 | 7 | 1 ÷ 67 = 0.01493 | 0.01493 |
| Visitor rules, part 2 | not in the top 10 | 0 | 8 | 1 ÷ 68 = 0.01471 | 0.01471 |
| Weather and seasons, part 1 | not in the top 10 | 0 | 9 | 1 ÷ 69 = 0.01449 | 0.01449 |
| Fernport Lighthouse, part 1 | not in the top 10 | 0 | 10 | 1 ÷ 70 = 0.01429 | 0.01429 |
Parts are rounded to 5 decimals, so a total can differ from the sum of the rounded parts in the last digit.
Chapter 3: Weighted fusion: you choose the balance
Question in use: How much is a boat ticket?
Another way to merge is to keep the scores. Each search’s scores are first rescaled so its best passage gets 1 and the 20th gets 0 (min-max scaling, also called normalising). Then the merged score is alpha × vector + (1 − alpha) × BM25. Alpha is a slider from 0 (BM25 only) to 1 (vector only). Careful: scaling gives the last passage in a list 0, which is the same as a passage the list does not contain at all. A passage that just made the top 20 counts as if the search had not found it.
Press “What can the town do on Wednesdays?”. Move alpha to 0 and the top results are the BM25 top results. Move it to 1 and they are the vector top results. Somewhere in between, find where the top passage changes.
Alpha 0.5. Top passage: Ferry timetable, part 1, score 0.500.
Scroll sideways to see every column.
| Passage | BM25 scaled (0 to 1) | Vector scaled (0 to 1) | Score = 0.5 × vector + 0.5 × BM25 |
|---|---|---|---|
| Ferry timetable, part 1 | 0.000 | 1.000 | 0.500 |
| Ferry timetable, part 3 | 0.000 | 0.887 | 0.444 |
| Harbour guide, part 2 | 0.000 | 0.674 | 0.337 |
| Harbour guide, part 7 | 0.000 | 0.502 | 0.251 |
| Harbour guide, part 3 | 0.000 | 0.478 | 0.239 |
| History of Fernport, part 2 | 0.000 | 0.445 | 0.222 |
| Harbour guide, part 5 | 0.000 | 0.387 | 0.193 |
| Visitor rules, part 2 | 0.000 | 0.351 | 0.176 |
Check the first row: 0.5 × 1.000 + 0.5 × 0.000 = 0.500.
Chapter 4: Filters: use the labels on documents
Question in use: How much is a boat ticket?
Every document carries labels, called metadata: here its kind (place, service, event, rules or guide). A filter keeps only the passages whose document has a ticked kind, and drops the rest before you see the list. Filters are exact and cheap, but they can remove the right answer. If no kind is ticked, nothing is filtered. If the ticked kinds leave nothing, the lab says so.
Press “How much is a boat ticket?”. Tick only “event” and see the empty message. Then tick “service” and the ferry passages come back.
10 passages kept. First: Ferry timetable, part 1.
- Ferry timetable, part 1 kind: service
- Ferry timetable, part 3 kind: service
- Harbour guide, part 2 kind: guide
- Harbour guide, part 7 kind: guide
- Harbour guide, part 3 kind: guide
Chapter 5: Reranking: a closer second look
Question in use: How much is a boat ticket?
The searches above compare a question and a passage after each was turned into numbers separately (a bi-encoder, which is fast and lets you prepare every passage in advance). A cross-encoder reads the question and one passage together and gives that pair a score, which is slower but usually more accurate. So the usual recipe is to take the top 10 candidates from the hybrid search and rerank only those. Here the cross-encoder Xenova/ms-marco-MiniLM-L-6-v2 was run once in advance on each example question’s 10 candidates. Its scores are raw numbers: higher is better, and they can be negative. Only their order matters.
Press “When can I climb the tower by the sea?”. The hybrid list puts Harbour guide, part 7 first, but the cross-encoder moves Fernport Lighthouse, part 2 (the tour times) to the top.
Hybrid first: Ferry timetable, part 1. After reranking first: Ferry timetable, part 1. The order changed.
Scroll sideways to see every column.
| Hybrid rank | Candidate | Cross-encoder score | New rank |
|---|---|---|---|
| 1 | Ferry timetable, part 1 | -8.671 | 1 |
| 2 | Ferry timetable, part 3 | -11.079 | 2 |
| 3 | Harbour guide, part 2 | -11.219 | 5 |
| 4 | Harbour guide, part 7 | -11.242 | 6 |
| 5 | Harbour guide, part 3 | -11.323 | 10 |
| 6 | History of Fernport, part 2 | -11.295 | 9 |
| 7 | Harbour guide, part 5 | -11.217 | 4 |
| 8 | Visitor rules, part 2 | -11.278 | 8 |
| 9 | Weather and seasons, part 1 | -11.183 | 3 |
| 10 | Fernport Lighthouse, part 1 | -11.269 | 7 |
The cross-encoder changed the order.
New order (top 3)
- Ferry timetable, part 1
- Ferry timetable, part 3
- Weather and seasons, part 1
Chapter 6: Diversity: MMR
Question in use: How much is a boat ticket?
The top results are often near copies of each other. Maximal marginal relevance (MMR) picks passages one at a time. Each pick maximises lambda × relevance − (1 − lambda) × redundancy. Relevance is the cosine similarity with the question. Redundancy is the cosine similarity with the closest passage already picked (0 for the first pick). Lambda is a slider from 1 (relevance only) to 0 (variety only). The candidates are the vector search’s top 10.
Press “How much is a boat ticket?” and set lambda to 0.5. The picks are Ferry timetable, part 1, then Visitor rules, part 2 and History of Fernport, part 2, which are unrelated to the fare. Diversity can cost relevance. Move lambda back to 1 and the list is the plain cosine order again.
Lambda 1.0. Picks: Ferry timetable, part 1, Ferry timetable, part 3, Harbour guide, part 2, Harbour guide, part 7, Harbour guide, part 3.
Scroll sideways to see every column.
| Pick | Passage | Relevance (cosine with the question) | Redundancy (cosine with the closest pick) | Score = 1.0 × relevance − 0.0 × redundancy |
|---|---|---|---|---|
| 1 | Ferry timetable, part 1 | 0.357 | 0.000 | 0.357 |
| 2 | Ferry timetable, part 3 | 0.326 | 0.606 | 0.326 |
| 3 | Harbour guide, part 2 | 0.268 | 0.374 | 0.268 |
| 4 | Harbour guide, part 7 | 0.220 | 0.478 | 0.220 |
| 5 | Harbour guide, part 3 | 0.214 | 0.431 | 0.214 |
Chapter 7: Compare: which search is best?
Question in use: How much is a boat ticket?
The 36 questions below each have known right passages, so each search can be scored. Recall@3 is the share of the right passages found in the top 3. MRR is the average of 1 ÷ (rank of the first right passage), counting 0 when none is in the top 10. Half the questions reuse the passage’s own words (keyword style) and half are paraphrases (meaning style). BM25 wins the keyword style (recall@3 1.00 against 0.75 for vector search). Vector search wins the meaning style (0.96 against 0.41 for BM25). Hybrid RRF is the safe all-rounder: the best recall@3 overall of the three searches (0.91 against 0.86 and 0.70), but not best on either type alone, and vector search has the better recall@1. Reranking the hybrid top 10 was never behind on these 36 questions (0.98 overall), but it needs a second model and only reorders the hybrid top 10. These 36 questions are few and were written by hand for this made-up town, so the numbers show a pattern, not a rule for every search.
Read the table row by row. Among the three single searches, none wins every kind of question: the lesson is to know your questions. The top 3 for the question in use are shown below the table.
Recall at 3 on all questions: BM25 0.704, Vector search 0.856, Hybrid (RRF) 0.912, Hybrid, then rerank 0.981.
Scroll sideways to see every column.
| Questions | Method | Recall@1 | Recall@3 | Recall@5 | MRR |
|---|---|---|---|---|---|
| All 36 questions | BM25 | 0.648 | 0.704 | 0.704 | 0.711 |
| All 36 questions | Vector search | 0.741 | 0.856 | 0.880 | 0.839 |
| All 36 questions | Hybrid (RRF) | 0.713 | 0.912 | 0.977 | 0.869 |
| All 36 questions | Hybrid, then rerank | 0.907 | 0.981 | 1.000 | 0.986 |
| 18 keyword-style questions | BM25 | 0.944 | 1.000 | 1.000 | 1.000 |
| 18 keyword-style questions | Vector search | 0.722 | 0.750 | 0.778 | 0.761 |
| 18 keyword-style questions | Hybrid (RRF) | 0.833 | 0.972 | 0.972 | 0.944 |
| 18 keyword-style questions | Hybrid, then rerank | 0.944 | 1.000 | 1.000 | 1.000 |
| 18 meaning-style questions | BM25 | 0.352 | 0.407 | 0.407 | 0.423 |
| 18 meaning-style questions | Vector search | 0.759 | 0.963 | 0.981 | 0.917 |
| 18 meaning-style questions | Hybrid (RRF) | 0.593 | 0.852 | 0.981 | 0.794 |
| 18 meaning-style questions | Hybrid, then rerank | 0.870 | 0.963 | 1.000 | 0.972 |
These 36 questions are few and were written by hand for this made-up town, so the numbers show a pattern, not a rule for every search.
The question in use
BM25
No passage uses these words.
Vector search
- Ferry timetable, part 1
- Ferry timetable, part 3
- Harbour guide, part 2
Hybrid (RRF)
- Ferry timetable, part 1
- Ferry timetable, part 3
- Harbour guide, part 2
Hybrid, then rerank
- Ferry timetable, part 1
- Ferry timetable, part 3
- Weather and seasons, part 1
Word list
- Hybrid search
- Running two different searches, usually keyword and vector, and merging their results into one list.
- BM25
- A keyword search from Lab 06 that scores the words a passage and the question share, giving more weight to rare words.
- Vector search
- A search from Lab 08 that compares the meaning of the question with the meaning of each passage, both written as lists of numbers.
- Rank
- A position in a list. Rank 1 is the best result.
- Embedding
- A list of numbers that stands for a text’s meaning. An embedding model turns your question into one, so it can be compared with the passages.
- Paraphrase
- The same question in other words, like “boat ticket” for “ferry fare”.
- Keyword style / meaning style
- Keyword-style questions reuse the passage’s own words. Meaning-style questions are paraphrases.
- Rank fusion
- Merging several ranked lists into one using only the ranks, not the scores.
- RRF
- Reciprocal rank fusion: each list gives a passage 1 ÷ (k + rank) points, and the points are added up. A list that does not hold the passage gives nothing.
- k (in RRF)
- A constant, 60 here. It keeps a rank 1 from being worth far more than a rank 2.
- Normalise
- Rescale numbers to a common range. Min-max scaling gives the best score 1 and the lowest 0, and the others in between. Here the lowest is the 20th of a search’s top 20, and a passage that is missing also counts 0.
- Weighted sum
- Add scores after multiplying each by a weight. Here: alpha × vector score + (1 − alpha) × BM25 score.
- Alpha
- The weight on vector search in a weighted sum. 0 means only BM25, 1 means only vector search.
- Metadata
- Labels about a document, such as its kind, that are not part of its text.
- Filter
- A rule that keeps only the passages whose metadata matches, for example only documents of kind “rules”.
- Candidate
- A passage that is still in the running. Reranking looks only at the top 10 candidates from the hybrid search.
- Reranking
- Reordering a short list of candidates with a more careful, slower scorer.
- Bi-encoder
- A model that turns the question and each passage into numbers separately, then compares the numbers. Fast, and passages can be prepared in advance.
- Cross-encoder
- A model that reads the question and one passage together and scores the pair. Slower, usually more accurate. It is used for reranking.
- Diversity
- How different the results are from each other.
- Redundancy
- How much a passage repeats one already picked: its highest cosine similarity with the picked passages.
- MMR
- Maximal marginal relevance: pick one passage at a time, balancing relevance to the question against redundancy with earlier picks.
- Lambda
- The MMR slider. 1 means only relevance, 0 means only variety.
- Cosine similarity
- How close two lists of numbers point: 1 means the same direction, 0 means unrelated.
- Recall@k
- The share of the right passages that appear in the top k results: recall@1 looks at the first result only, recall@3 at the top 3, recall@5 at the top 5.
- MRR
- Mean reciprocal rank: the average over questions of 1 ÷ (rank of the first right passage), or 0 if none is in the top 10.
- Logit
- A raw score from a model. Higher is better, it can be negative, and it is not a percentage.