Find the right information: retrieval from scratch
Learn how search finds the right page for a question, one technique at a time, so a small AI can answer from documents it has never seen.
Seven short steps. You first see that a model only knows its training, then build search in stages: by words, by spelling, by meaning, and by combining them, ending with an AI that answers from the pages it finds. No experience needed.
Step 1 See what a model knows on its own
A model can only answer from what it read in training. Seeing it guess about something new shows why it needs your documents.
Open step 1: See what a model knows on its ownStep 2 Keyword search: find pages by their words
The oldest kind of search lists which pages contain each word, then scores the pages. You build the list and watch the scores decide the order.
Open step 2: Keyword search: find pages by their wordsStep 3 Fuzzy search: forgive typos and spellings
You will find words even with typos, close spellings and sound-alike names.
Open step 3: Fuzzy search: forgive typos and spellingsStep 4 Vector search: find pages by meaning
You will turn text into lists of numbers that capture meaning, so a question can find a page that uses completely different words.
Open step 4: Vector search: find pages by meaningStep 5 Hybrid search: combine and rerank
You will mix word search and meaning search into one ranking, narrow it with filters, and re-order the best results.
Open step 5: Hybrid search: combine and rerankStep 6 Smarter queries and structure
Rewriting the question, asking it several ways or with a made-up answer (HyDE), following links between names and passing whole documents each fix a mistake plain search makes. You compare them on the same questions.
Open step 6: Smarter queries and structureStep 7 Put it together
You will let a small AI answer questions from the pages your search finds, and measure which technique finds the right page most often.
Open step 7: Put it together