LLM Apps: Prompts, RAG and Evals

About this course

Calling a language model is easy. Building a feature that is right most of the time, fast enough, cheap enough and safe is the real job. This course teaches the core techniques of LLM engineering, and above all how to measure whether a change made things better.

You build the product-question assistant in the Mangrove app, the fictional grocery chain used across Kolom. It answers customers' questions about products, allergens, substitutes and delivery from Mangrove's own catalogue and help pages, returns structured data the app can render, calls tools to check live stock and delivery slots with the customer's own permissions, and is measured on a golden set of real customer questions before every release. Along the way you work out what each call costs, why allergen questions must never be answered from guesswork, and how a product description can carry a prompt injection.

Every lesson ends with code you can run: a provider-neutral API wrapper with retries, a validator for structured output, a tool dispatcher with permissions, hybrid retrieval with reciprocal rank fusion, an eval runner that fails CI when quality drops, and a cost model. Auto-checked JavaScript exercises run in your browser, and each lesson has a podcast episode for the bus ride.

What you will be able to do

• Explain tokens, context windows, sampling and streaming, and estimate what a feature costs per 1,000 questions
• Write system instructions and JSON schemas that give consistent, structured output, and validate every reply before the app uses it
• Connect a model to real data with tool calling and the Model Context Protocol, with the user's permissions and no risky write tools
• Build retrieval over your own documents with structure-aware chunking, hybrid search, rank fusion and reranking, and measure it with recall@k
• Measure quality with a golden set, code checks and model grading, and block releases in CI when a category gets worse
• Control cost and latency with caching and model routing, and defend against prompt injection and the OWASP risks for LLM apps

Syllabus

6 stops, about 4 h in total. Each stop fills in when you complete the lesson.

1. 1

   How LLM APIs work

   Every feature in this course starts with one HTTP call to a language model. This lesson opens that call up: what goes in, what comes back, why the bill grows with every message, and how to keep the call fast and reliable when thousands of customers ask at once. By the end you will have a small, provider-neutral wrapper that the rest of the course builds on.

   35 minIntermediateEpisode1 exercise

   LLM APIsToken budgetsStreamingCost estimation

2. 2

   Prompts and structured output

   The product-question assistant in the Mangrove app has to return something the app can render and a customer can rely on, including on allergens, where a guess can hurt someone. This lesson shows how to write system instructions that change behaviour, how to use examples, how to get JSON that follows a schema, and how to check every reply in your own code before a customer sees it.

   35 minIntermediateEpisode1 exercise

   Prompt engineeringStructured outputLLM APIs

3. 3

   Tools and the Model Context Protocol

   With tools, the Mangrove app's product assistant can check live stock at the customer's store and list real delivery slots. This lesson shows how the tool-calling loop works, how to keep every call inside the customer's own permissions, and how the Model Context Protocol lets you write a stock tool once and use it from many AI apps.

   40 minIntermediateEpisode1 exercise

   Tool callingMCPLLM APIsStructured output

4. 4

   Retrieval-augmented generation

   The product-question assistant now has a schema (llm-02) and tools for live stock and slots (llm-03). It still cannot answer "Can I pay cash for a Later slot in festival week?", because that answer lives in Mangrove's help pages. This lesson builds the retrieval that finds the right help section among about 600, quickly and fresh, and makes the model cite it or say it does not know.

   40 minIntermediateEpisode2 exercises

   RAGEmbeddingsHybrid searchReranking

5. 5

   Evals: knowing if it got better

   You changed the prompt and three answers you tried look better. That tells you almost nothing. This lesson builds the engineering side of evals for the product-question assistant: eval cases stored as data, checks written in code, a model judge with a written rubric, retrieval scored on its own, honest numbers despite randomness, and a gate in CI that blocks a pull request when a category gets worse.

   40 minIntermediateEpisode1 exercise

   EvalsStructured outputRAG

6. 6

   Cost, latency and safety

   The product-question assistant in the Mangrove app works and passes its evals. Before launch you need three more answers: what 1,000 questions cost, how long a customer waits, and what happens when someone, or some text, tries to make the model misbehave. This lesson gives you the levers for each and a plan you can defend.

   40 minIntermediateEpisode2 exercises

   LLM APIsEvalsLLM cost controlLLM security

7. Capstone: Ship the product-question assistant

   A realistic project with a rubric and a model answer.

8. Final exam and certificate

   10 questions drawn at random, 70% to pass.

Capstone project

Check what you know

Questions