Designing agents people can trust: guardrails, approvals and a way back

An assistant that answers questions can embarrass you. An agent that sends emails, changes records or spends money can hurt someone. The design work that matters most is not the conversation. It is deciding what the agent may do on its own, what it must ask about, and how a person sees, stops and reverses what it did.

One job, the fewest tools

Scope the agent to one job and give it only the tools that job needs, with the narrowest permissions those tools allow. The OWASP Top 10 for LLM Applications names this risk directly as excessive agency, and traces it to three causes: excessive functionality, excessive permissions and excessive autonomy.¹ An agent that drafts replies does not need to send them. One that reads a CRM does not need to delete from it.

Treat every input as untrusted

Prompt injection is first on the same list.² Instructions can arrive in what the user types, and also in what the agent reads: a web page, a PDF, an email it summarises. Design so that no input can widen what the agent is allowed to do. Permissions live in the systems the agent calls, never only in its prompt.

Show the plan before it runs

For anything consequential, the agent proposes and a person approves. The approval screen shows what will happen, to which records and why, in plain words, with the sources the agent used. Approving should be one decision, not a hunt through a transcript.

Make uncertainty visible

Research on human–AI interaction has long asked systems to make clear what they can do and how well they can do it.^{3 4} In practice that means citing sources, saying when the answer is not in them, and handing the question to a person instead of guessing. A clean refusal is a feature.

Every action has a way back

• Draft first: emails, records and posts are created as drafts until a person approves them.
• Undo for a set period after any change the agent makes.
• A dry run that shows the effect without making the change.
• A stop control that ends the run and leaves everything as it was.

Log everything a reviewer would ask for

Inputs, retrieved sources, tool calls, approvals, refusals and errors, each with a time and an actor. NIST’s Generative AI Profile organises its guidance around four primary considerations — governance, content provenance, pre-deployment testing and incident disclosure — and none of them works without that record.⁵ The log is also how you learn where the agent helps and where it hesitates.

Test before anyone relies on it

Before release, run the agent against an eval set: real tasks with known good outcomes, plus the cases it must refuse and the attacks it must ignore. Release when it passes, and run the set again on every change to a model, a prompt or a tool.⁶ We wrote about building those sets in Evals before prompts.

We design agent experiences in AI Application Design and build them, with their evals and logs, in AI Strategy & Agents.