Point of view3 min readAI Design · Product & Experience Design · Technology & Intelligence
Designing AI agents people can trust: limits, approvals and a way back
Seven design rules that keep AI agents useful, honest about what they do not know and unable to cause harm on their own.
Xterra EdzeEditorial
Published
An assistant that answers questions can embarrass you. An agent that sends emails, changes records or spends money can hurt someone. The design work that matters most is deciding what the agent may do on its own and what it must ask about. A person must also be able to see, stop and reverse what it did.
One job, the fewest tools
Scope the agent to one job, with only the tools that job needs and the narrowest permissions they allow. The OWASP Top 10 for LLM Applications calls this risk excessive agency and traces it to three causes: excessive functionality, permissions and autonomy.1 An agent that drafts replies does not need to send them; one that reads a CRM does not need to delete from it.
Treat every input as untrusted
Prompt injection is first on the same list.2 Instructions can arrive in what the user types and in what the agent reads: a web page, a PDF, an email it summarises. Design so that no input can widen what the agent may do. Permissions live in the systems it calls, never only in its prompt.
Show the plan before it runs
For anything consequential, the agent proposes and a named person approves. The approval screen states in plain words what will happen, to which records and why, and shows the sources the agent used. Approving should be one decision, made without reading a transcript.
Make uncertainty visible
Research on human–AI interaction has long asked systems to make clear what they can do and how well.3 4 In practice that means citing sources, saying when the answer is not in them and passing the question to a person instead of guessing. A clean refusal is a feature.
Every action has a way back
- Draft first: emails, records and posts stay drafts until a person approves them.
- Undo for a set period after any change the agent makes.
- A dry run that shows the effect without making the change.
- A stop control that ends the run and leaves everything as it was.
Log everything a reviewer would ask for
Inputs, retrieved sources, tool calls, approvals, refusals and errors, each with a time and an actor. NIST’s Generative AI Profile organises its guidance around four considerations: governance, content provenance, pre-deployment testing and incident disclosure.5 None of them works without that record. The log also shows where the agent helps and where it hesitates.
Test before anyone relies on it
Before release, test the agent on an evaluation set: real tasks with known good outcomes, plus cases it must refuse and attacks it must ignore. Release when it passes, and run the set again on every change to a model, prompt or tool.6 We wrote about building those sets in Evaluation sets before prompts.
- Job per agent
- 1
- With the fewest tools that job needs.
- Irreversible actions without a person
- 0
- Sending, paying, deleting.
- Of steps written to the audit log
- 100%
- Inputs, sources, tool calls, approvals.
The design rules we set for agents we build. Rules, not measured results.
The design work that matters most is deciding what the agent may do on its own.
We design agent experiences in AI Application Design and build them, with their evaluation sets and logs, in AI Strategy & Agents.
Key dates this guide tracks.
The changes this guide covers, in date order, each with its source. See every date in the standards timeline
-
NIST publishes its Generative AI Profile (NIST AI 600-1)
A companion to the AI Risk Management Framework, with actions for generative AI.
Source 5National Institute of Standards and Technology
Sources. Where the facts come from.
Numbered as they are cited in the text. Each link opens the original.
-
Guidelines for Human-AI Interaction
Amershi et al., Microsoft Research (CHI 2019) · May 2019
-
National Institute of Standards and Technology · 26 July 2024
