Designing AI features people can trust
An AI feature will be wrong some of the time, so the design decides what happens next: whether people notice, whether they can recover, and whether anyone learns from it. This lesson covers the parts of AI design that ordinary features don't need: showing uncertainty, keeping the right person in the loop, designing failure states, privacy and safety guardrails, and feedback you can actually use.
By the end you can
- Design an answer that shows its sources, its data and its limits, and says "I don't know" when it should
- Place people in the loop where they can really check, and measure whether they do
- Write the failure states of an AI feature before it ships
- Specify privacy and safety guardrails as code around the model, with a test for each
- Capture feedback that turns into fixes and new test cases
The everyday picture
A good sat-nav shows the route and the arrival time. When it loses the signal it says "GPS signal lost" instead of guessing. It lets you take another road and recalculates without complaint. It warns you before a toll road, and it never drives the car. Drivers trust it because they can see what it is doing and overrule it, and they still watch the road.
A bad sat-nav announces "turn left" into a lake in the same calm voice it uses for everything else. Every sat-nav is sometimes wrong. This one gives you no way to tell when. Designing an AI feature is mostly designing the parts around the answer: the signal-lost message, the "recalculating", the toll-road warning and the hands on the wheel.
Design for uncertainty: show your work
In pm-47 Mangrove chose Ask Ops as an assist feature: it answers store managers' rule and status questions, and people decide what to do. Assist only works if managers can tell good answers from bad ones quickly. So every answer shows its work. Leila, Mangrove's designer, and Imran settled on an answer card with five parts:
- The answer, in five lines or fewer, as steps when there are steps.
- Sources: the procedure and the section it came from, such as "RF-2 Refunds, over $50", one tap to open the exact paragraph.
- Data used, with the time it was read: "GRN 8812, checked 09:42". Numbers that came from data are marked as such, so a manager can see which figures the model wrote and which a system supplied.
- Limits: what it could not check. "I couldn't see the supplier's invoice."
- Next steps: Solved, Not right and Ask the ops desk, which sends the question and the answer to a person.
"I don't know" is a feature. When retrieval finds no procedure that matches the question well enough, Ask Ops says "I couldn't find a procedure that covers this" and offers the desk, with the question already filled in. Answering 85% of questions well and handing off 15% beats answering all of them with one in ten wrong, because every confident wrong answer teaches managers to distrust the right ones.
Avoid fake precision. A model asked "how sure are you?" will produce a number, but that number is not a reliable measure of whether it is right. Show categories you can defend instead, derived from checks the system ran: "from procedure RF-2", "checked against live data", "no procedure found". And set expectations at first use: what Ask Ops can answer, what it can't, and how well. The first two of Microsoft's 18 guidelines for human-AI interaction (Amershi and colleagues, 2019) are exactly that: make clear what the system can do, and how well it can do it.
Quick check
Which answer design builds the right amount of trust?
Sources, data and limits let a manager check in seconds and learn when to rely on it. A clean answer with no sources trains blind trust. A model's self-reported percentage is not reliable. Quoting every possible procedure buries the answer and makes checking slower.