1. Courses
  2. AI Engineering in Production
  3. Capstone

Capstone project for AI Engineering in Production

Run the AI platform review

Three months after launch, Mangrove's app assistant costs about twice its $10,000 monthly budget and its allergen answers got worse after a model update nobody noticed for nine days. Run the review. Build a small version of the assistant (or use an LLM feature you own) with retrieval over a few help pages and a product table, instrument it with tracing, build a golden set and an online sampling plan with a calibrated judge, find and fix the causes of cost and quality problems, decide between better retrieval and fine-tuning with evidence, red-team it, and present a governance pack to leadership: an inventory entry, a risk tier, data use, owners and an incident process.

What you hand in

  1. A tracing and eval setup, with before and after numbers for quality per category, cost per conversation and p95 latency
  2. A cost plan with caching, routing and budgets, and the evidence that quality held
  3. A red-team report with findings, severities, owners and fixes, and the eval cases added
  4. A one-page governance summary: inventory entry, risk tier, data use, owners and the AI incident process

Check your work

Tick each item when your project meets it. Be strict: this is the rubric the reviewer uses.

  • Every claim is backed by trace or eval data, with before and after numbers per category
  • Tracing records model requested and returned, tokens, cost, latency and prompt version, with content capture handled safely
  • The eval setup has must-pass categories, a judge checked against human labels, online sampling and pinned model versions
  • Cost per conversation is cut without a quality drop on the routed categories, and a budget with alerts is in place
  • The prompt, retrieval or fine-tuning decision is made with a fair comparison on the same eval set
  • Red-team findings have severities, owners and fixes, and defences assume an injection can succeed
  • Accountability is named for each AI feature, with a risk tier, data use and an incident process

0 of 7 checked

Model answer

The model answer opens once your capstone passes the review, so you can compare your work with an outline of a strong submission.

Hand in your capstone

Your links are checked automatically, then the work is scored against the rubric above, item by item, with comments. A pass counts toward the course certificate and shows on your profile.

Loading…