Evaluation sets before prompts: how to test an AI feature before release

Teams building with AI models often write the prompt first and judge it by trying a few questions. That works until someone edits the prompt, the model is upgraded or a new document enters the knowledge base. Then nobody can say whether quality went up or down. An evaluation set turns quality into a test that passes or fails.^{1 2}

Inside an evaluation set

• Real tasks. Questions and requests collected from the people who will use the system.
• The accepted answer. What a correct response contains, and the source it must cite.
• Refusals. Questions the system must decline or hand to a person: out of scope, unsafe or not answerable from its sources.
• Attacks. Inputs that try to change the system’s instructions, including instructions hidden in documents it reads.³
• Edge cases. Long inputs, other languages, typos and ambiguous requests.

{
  "id": "sso-pricing-07",
  "slice": ["pricing", "high-risk"],
  "input": "Does single sign-on cost extra on the Team plan?",
  "must_cite": ["docs/plans#team"],
  "accept": "Says whether SSO is included on Team, citing the plans page, and offers a person for pricing questions.",
  "reject_if": ["quotes a price that is not in the sources", "gives no citation"]
}

How to grade

Use the cheapest grader that is reliable for each check. Code-based checks cover a lot: did it cite a source from the allowed list, did it refuse, did it stay under the length limit. People grading against a rubric cover judgement. Model-based graders extend that rubric to more cases once you have measured how often they agree with human graders on the same cases.¹

What a release report shows

• Pass rate per slice, against the target set before the run.
• New failures since the last release.
• Refusals that should have been answers, and the reverse.
• Injection attempts caught and missed.
• Cost and response time per answer.

Report results by slice

A single score hides the failures that matter, so break results down by topic, risk level, language and customer segment. A release that improves general answers but worsens pricing or security answers is a regression, whatever the average says.

Make it a gate

Run the set automatically on every change: a new model version, a prompt edit, a new tool or new documents in the index. Set a pass threshold for each slice before you see results, and block the release when any slice falls below it. NIST’s AI Risk Management Framework places this under its Measure function: testing, evaluation, verification and validation that continue after deployment.⁴

Keep the set current

• Add every failure from production as a new case, with its correct answer.
• Retire cases that no longer reflect how people use the system.
• Re-check model-based graders against people on a regular schedule.
• Version the set with the code, so every result can be reproduced.

Open-source frameworks such as OpenAI’s Evals show the pattern in code.⁵ We design evaluation sets in AI Strategy & Agents and run them as release gates in AI Product & Automation.