Retrieval-augmented generation

40 minIntermediate

Retrieval-augmented generation

The product-question assistant now has a schema (llm-02) and tools for live stock and slots (llm-03). It still cannot answer "Can I pay cash for a Later slot in festival week?", because that answer lives in Mangrove's help pages. This lesson builds the retrieval that finds the right help section among about 600, quickly and fresh, and makes the model cite it or say it does not know.

By the end you can

Decide which questions go to retrieval and which go to tables or tools
Build an ingest job that chunks help pages and products, re-indexes on change and deletes stale chunks
Run hybrid search in Postgres (pgvector plus full-text search) and fuse the lists with reciprocal rank fusion
Rerank a short list, apply filters, and build a prompt whose answer cites source ids or refuses
Measure retrieval on its own with recall@k on a labelled set

The everyday picture

Think of the help desk at a big public library. You ask "Is the new cookbook on the shelf right now?" and the librarian does not open a leaflet. She checks the computer, because that is a live fact. You ask "How do I renew a book from another branch?" and she goes to the leaflets.

She searches two ways at once. The subject index finds leaflets about renewing even when they say "extend your loan". The exact title list finds the one that mentions your branch code. She pulls ten candidates, reads the best few closely, checks the date printed at the bottom, and puts the old version in the bin. Then she answers and points at the paragraph, so you can read it yourself. If nothing fits, she says "we have nothing on that" instead of guessing. Every part of that routine is a part of this lesson.

When retrieval is the right tool

Designing AI features showed the basic shape of RAG: embed the question, fetch the nearest chunks, put them in the prompt. Read its RAG section first if it is new to you. Here you go deeper, starting with a question teams skip. Should this question use retrieval at all?

Retrieval finds text that is probably relevant. That suits policies written as prose: delivery rules, return windows, cash on delivery, late riders. It is the wrong tool for structured facts. If a customer asks whether a cereal contains nuts, the answer must come from the allergen fields in the product table, which llm-02 already returns as an allergen block that can say "unknown". A retrieved paragraph that mentions nuts is a guess. Price and stock change every hour, so they come from tools, as in Tools and the Model Context Protocol.

So the assistant starts with a router. A small, fast model call (or a classifier) labels each question with a type: allergen, price_stock, order_status, policy, product_discovery or other. Allergen questions read the product row. Price, stock and order questions call tools. Policy questions go to help-page retrieval. Discovery questions such as "a gluten-free pasta for kids" use retrieval over the catalogue to find candidates, then read each candidate's row for the facts. Many questions need both: "Can I get the 5 kg basmati with Now delivery?" needs a stock tool call and the Now delivery rules.

Quick check

Which of these questions should NOT be answered from retrieved help-page text?