AI Engineering in Production

About this course

Once an AI feature has real users, new problems appear: quality drifts when a provider updates a model, costs climb past the budget, a clever user or a poisoned document finds a way around the prompt, and legal asks how decisions are made and who is accountable. This course is about running AI systems reliably, safely and economically in production.

You take over two Mangrove Market systems three months after launch: the assistant in the Mangrove app, which answers about 15,000 customer conversations a day about products, allergens and delivery, and Ask Ops, the agent that answers store staff from procedures and drafts tasks for managers to approve. The assistant costs about twice its budget, and its allergen answers got worse after the model behind an unpinned alias changed. You add OpenTelemetry tracing to every call, build offline and online evals with a calibrated judge, cut cost with caching and routing between a small and a large model, decide between better prompts, better retrieval and fine-tuning with evidence, red-team both systems, and write the governance pack that names an owner and a risk tier for every AI feature.

Every lesson has auto-checked SQL or JavaScript exercises that run in your browser (cost per feature from traces, an eval gate, a router, an output guard, an inventory check), a podcast episode, flashcards and practice tasks with model answers. Prices in the lessons are illustrative; the methods work with any provider.

What you will be able to do

• Trace LLM and agent calls end to end with OpenTelemetry conventions, and debug a bad answer from its trace
• Run offline and online evals with a calibrated judge, and catch regressions from model updates before users do
• Cut cost and latency with prompt caching, batching, routing and budgets, without a quality drop
• Choose between prompting, retrieval and fine-tuning with a fair bake-off on your own data
• Red-team AI features, defend them in layers, and meet governance and regulatory duties with named owners

Syllabus

6 stops, about 3.5 h in total. Each stop fills in when you complete the lesson.

1. 1

   Observability for LLM systems

   An LLM feature fails quietly: the request returns 200, the answer reads well, and it is wrong, slow or twice as expensive as planned. This lesson shows you how to trace every model call, retrieval step and tool call, which attributes to record, how to keep customers' words out of places they don't belong, and how to read the first dashboards and a bad answer's trace.

   35 minAdvancedEpisode1 exercise

   LLM observabilityOnline evals

2. 2

   Evals that never stop

   The eval suite you built before launch tells you whether a change is safe to merge. After launch, the questions change, the judge can drift, and the provider can change the model under you. This lesson turns evals into a loop that runs every day: a golden set that grows from production, a judge you have checked against people, a sample of real traffic scored daily, user signals as pointers, and a gate that every model, prompt and retrieval change must pass, category by category.

   35 minAdvancedEpisode1 exercise

   Online evalsLLM observability

3. 3

   Cost and latency at scale

   Three months after launch, the app assistant costs about twice its budget and its slowest answers take 6.5 seconds. This lesson shows you how to find where the money and the time go from your traces, fix them with prompt caching, routing and smaller prompts, prove with evals that quality held, and put a budget and a kill switch around every AI feature.

   35 minAdvancedEpisode1 exercise

   Model routingLLM observability

4. 4

   Prompting, retrieval or fine-tuning?

   After launch, every quality complaint arrives with a proposed fix, and the most expensive one is often "let's fine-tune our own model". This lesson gives you a way to diagnose a failure as missing knowledge, wrong behaviour or not enough capacity, what each fix really changes and costs to keep, how fine-tuning works and what data it needs, and a bake-off on your own evals that makes the decision for you.

   35 minAdvancedEpisode1 exercise

   Fine-tuningOnline evalsModel routing

5. 5

   Guardrails and red-teaming

   At launch you built layered defences and a set of injection cases. Three months later the systems read more content, serve far more people and have tools that change real things, so new holes open that no launch test covered. This lesson turns guardrails into something you operate: a threat model for both systems, guards you measure, attack signals you watch, and a red-team exercise whose findings become fixes and eval cases.

   35 minAdvancedEpisode1 exercise

   GuardrailsRed-teamingOnline evals

6. 6

   Governance and accountability

   Sooner or later, leadership, a regulator or a journalist asks three questions about every AI feature: what does it do, with which data, and who answers when it goes wrong. This lesson gives you the tools to answer in ten minutes: an AI inventory, risk tiers with the controls each one needs, the regulation basics engineers meet, an AI incident process, and the one-page governance pack leadership signs.

   35 minAdvancedEpisode1 exercise

   AI governanceGuardrailsLLM observability

7. Capstone: Run the AI platform review

   A realistic project with a rubric and a model answer.

8. Final exam and certificate

   10 questions drawn at random, 70% to pass.

Capstone project

Check what you know

Questions