← All perspectives

Perspective / AI solutions

Evaluating production AI agents: release gates, failure modes and operational evidence

An AI agent can produce a convincing answer while leaving the underlying business process wrong. A practical evaluation architecture connects task outcomes, authorisation, repeatability and operational cost before production release.

11 min readReviewed

Optical inspection of a precision component under a blue testing beam.
Conceptual editorial image.

Production AI evaluation should establish whether an agent completes the right business task, within its authority, at an acceptable level of reliability and cost. A fluent answer is only one observation. For an agent connected to a CRM, finance system or service platform, the decisive evidence often lives in the resulting system state. This article develops an evaluation approach for a hypothetical sales-support agent, with explicit release criteria, failure categories and a worked decision model. All figures and scenarios are illustrative.

The executive view

  • Evaluate actual system outcomes and permissions separately from response quality.
  • Use representative tasks, adversarial challenges and repeated trials for different kinds of evidence.
  • Tie release gates to error consequences, human review and operational performance.
01

Define the business outcome before selecting the metric

Start with a task specification that an accountable business owner can approve. For our hypothetical agent, the task is to prepare a renewal opportunity from an existing customer record. Success requires the correct account, an accurate renewal date, a supported commercial summary and a draft opportunity linked to the right contract. Sending an email or changing an agreed price is outside its authority.

Those conditions separate completion, quality and permission. A single average score would conceal the difference between an awkward summary and an unauthorised contract change. Keep hard boundaries as individual pass/fail checks, then assess softer qualities with explicit rubrics.

Anthropic’s agent evaluation guidance makes a useful distinction between the recorded interaction and the final outcome. Apply that distinction directly: inspect the saved opportunity, account links and audit events rather than relying on the agent’s completion message. This creates a testable commercial specification.

Write the specification as a state transition. The input is an authenticated account identifier plus approved source material; the output is a draft opportunity with defined fields. State what must remain unchanged, including prices, consent flags and contract ownership. Add an explicit abstention outcome for insufficient evidence. Otherwise a system can appear to improve completion by fabricating the missing information.

Make the acceptance boundary different for suggestions and actions. A draft that a trained account manager must inspect can tolerate some correctable omissions. A service that updates the renewal forecast needs stricter evidence about identifiers and amounts. This distinction should influence permissions, review design and evaluation sampling before model selection. It also prevents a later interface change from quietly converting an advisory capability into an operational decision-maker.

02

Build a representative suite with a separate challenge set

A test suite should reflect the distribution of work the service will actually receive. Include ordinary renewals, incomplete records, duplicate account names, stale attachments and conflicting dates. Record why each case exists and which business rule it exercises. An account manager should recognise the situations without needing to understand model internals.

Keep a separate challenge set for low-frequency, high-consequence behaviour: another customer’s document, an instruction embedded in an attachment, or a request to bypass approval. A suite containing mostly adversarial cases gives a distorted estimate of everyday performance; a suite containing only ordinary cases provides weak assurance about boundaries.

Reserve cases that developers do not tune against. Otherwise an apparently improving result may reflect adaptation to familiar examples. Each test needs a stable fixture, expected state and grading version. Remove personal data from fixtures or use synthetic records that preserve the structural problem without representing a real individual.

Stratify the suite by consequence as well as frequency. For example, assign separate labels for new accounts, multi-contract customers, missing dates and restricted attachments. A candidate could improve overall performance simply because ordinary cases dominate the denominator while degrading on the accounts with the greatest contract value. Report those slices even if their sample sizes are too small for precise estimates.

The fixture must also control time. A renewal date interpreted correctly today might be overdue next month, so record the reference clock and timezone. Pin external responses or provide a controlled simulation of them. Where a live dependency is part of the test, say so and retain its response. Reproducibility does not require pretending that every external service is deterministic; it requires knowing which uncertainty the test includes.

03

Choose graders according to what can be verified

Use deterministic checks for properties the application can establish precisely. Does the opportunity belong to the expected account? Did the tool call include the correct contract identifier? Was any restricted field modified? These checks should examine authoritative records and tool logs, with the evaluation environment reset between trials.

For the written summary, use a rubric that distinguishes unsupported claims, missing commercial risks and stylistic weaknesses. A model-based grader can help scale review, but calibrate it against a sample assessed by domain experts. Disagreement is information: refine the rubric or escalate ambiguous cases rather than treating the grader as ground truth.

OpenAI’s evaluation best practices recommends task-specific evaluation and continuous refinement. In this architecture, that means scoring factual support and decision usefulness separately. A polished paragraph cannot compensate for an incorrect renewal date, and extra length should not be rewarded automatically.

Assess the evaluator itself before using it as a release gate. Create known-good, known-bad and deliberately ambiguous summaries. Include plausible but unsupported commercial claims and citations that point to the wrong attachment. If the grader cannot distinguish those cases, increasing the number of automated judgments only scales a measurement defect.

A useful expert review sample includes random outputs and disputed outputs. Reviewing only disagreements estimates the difficulty of edge cases, not the typical accuracy of the grader. Blind reviewers to the candidate model where practical and retain their rationale for adjudicated cases. Keep the grader configuration versioned independently from the agent configuration: changing the rubric can alter scores even when the service itself has not changed. A release comparison should disclose both changes rather than presenting a cleaner chart as better agent performance.

04

Measure reliability across repeated trials

Generative systems vary between runs. A task that succeeds once may fail after a minor change in retrieval order, model settings or tool timing. Repeat important cases and retain the full configuration: model version, instructions, retrieved document identifiers, tool definitions and application release.

Distinguish the chance of obtaining one usable answer after several attempts from the chance of succeeding consistently on every attempt. The first may suit a human selecting among draft suggestions. The second matters when a workflow changes operational records without continuous supervision. Neither can be inferred from a single impressive demonstration.

Also avoid counting repeated trials as entirely independent evidence of business coverage. Running the same uncomplicated renewal fifty times does not replace testing fifty different account conditions. Report case coverage and within-case repeatability separately. Confidence should come from the breadth of scenarios, the consistency of execution and the consequence of the remaining failures.

For a numerical illustration, suppose an ordinary task succeeds in 95% of independent trials. Under that simplifying assumption, the chance of five consecutive successes is 0.95 to the power of five, approximately 77%. That calculation explains why a strong single-run result may be insufficient for unattended workflows. Actual trials can be correlated, so the example is a reasoning aid rather than an assurance estimate.

Investigate variance at the level where it originates. If different document order changes the chosen renewal date, stabilising retrieval and exposing source priority may matter more than increasing generation length. If timeouts cause duplicate writes, model quality is not the primary defect. Group failures by mechanism, attach accountable engineering actions and re-run the affected cases without discarding the broader regression suite.

05

Use a worked release decision rather than a headline score

Consider this illustrative candidate release, evaluated across a fixed set of renewal scenarios:

DimensionIllustrative resultRelease interpretation
Correct account and contract linkage99 of 100 tasksInvestigate the wrong-account case before release
Supported commercial summary94 of 100 tasksRoute uncertain summaries for review
Restricted action attempts1 observed attemptCheck that enforcement blocked it; inspect cause
Restricted actions completed0 observedNecessary evidence, not proof of future immunity
Human review timeMedian 3 minutesCompare with the existing workflow

A 97% blended score would obscure the wrong-account failure. The release decision should instead ask whether the failure can expose data or contaminate pipeline reporting, whether enforcement worked, and whether the system can reliably recognise when review is required.

Observed zero failures also has limits. A finite suite cannot establish that an event is impossible. The responsible claim is that no such failure was observed under the tested conditions, supported by controls that operate outside the model.

The wrong-account result should trigger a concrete investigation: was the supplied account identifier ignored, was a name used as a join key, or did the tool allow an out-of-scope identifier? A retrained summary prompt would not repair an authorisation defect. Preserve the failing trace and fixture before making changes so that the correction can be demonstrated.

Set different thresholds for catastrophic boundary failures and tolerable review work. A blocked unauthorised attempt can still reveal an unsafe plan, while a correctly enforced refusal may be the desired result. Report attempted and completed actions separately. For each exception, identify the exposure, compensating control and person accepting it. The release record should allow a future operator to reconstruct why this particular configuration was considered suitable for this particular workflow.

06

Make cost and latency part of the quality envelope

The business does not consume correctness in isolation. A service that produces a reliable renewal brief after twenty minutes may miss an account manager’s working context. Another that finishes quickly but sends most outputs for extensive review may create less capacity than expected.

Measure end-to-end elapsed time, tool calls, retries, infrastructure consumption and review effort alongside quality. Keep distributions, including slow-tail performance, rather than only averages. Segment by task complexity so that a few difficult accounts do not hide a common operational bottleneck.

Use these measurements to compare specific changes. A larger model might reduce review effort enough to justify higher inference cost. A smaller retrieval context might save tokens while increasing omissions. The relevant unit is the successfully completed business task within its required controls. Evaluate optimisation candidates against that whole envelope rather than declaring the cheapest model or fastest response the winner.

Calculate cost using successful outcomes rather than requests. In an illustrative batch, 1,000 tasks consume £80 of inference and tool usage; 150 require four minutes of review and 30 require ten minutes of repair. At an assumed £30 per hour, review costs £300 and repair costs £150, making the measured operating total £530 before hosting and support. Inference represents about 15% of that total, so a token-only optimisation would address a minority of the observed cost.

Record whether review and repair groups overlap, and count each activity once. Include abandoned attempts and escalation work in the numerator. Compare configurations at the same task mix, permission level and completion standard; otherwise the apparently cheaper option may simply be refusing difficult work or transferring it to staff.

07

Connect evaluation gates to controlled operation

An evaluation suite becomes useful governance when it changes release behaviour. Define who can accept residual errors, which checks block deployment and which require a documented exception. Preserve the last accepted baseline so that a model, prompt or retrieval change can be compared with a known configuration.

Release into a bounded workflow with monitoring for account mismatches, blocked actions, retrieval failures and human overrides. Feed reviewed production failures back into the suite, while retaining a representative holdout. An expanding list of historical incidents should complement the normal workload rather than eventually replacing it.

The decision for a commercial leader is therefore concrete: does the evidence support this level of autonomy for this class of task? Start with a restricted action surface and clear escalation path. Broader permissions should follow demonstrated reliability and operational need, with a fresh evaluation of the additional consequences.

Treat production monitoring as a second evidence stream, not a replacement for evaluation. Monitor blocked tool calls, unsupported-field corrections, repeated retries and account-manager reversals. Sample successful-looking tasks as well: users may accept a result without noticing an error, and a low complaint rate is not a direct accuracy measure.

A change record should connect the candidate configuration, evaluation results, operational owner and reversal procedure. If the previous model is no longer available, rolling back application code may not restore previous behaviour; retain an alternative restricted workflow, such as human preparation of renewal briefs. The practical release gate is therefore a combined decision about measured capability and operational readiness. The team needs a way to identify deterioration, limit further commitments and complete affected customer work.

08

A practical AI evaluation evidence pack

For a renewal agent, assemble four linked artefacts: the task and authority specification, the versioned fixture suite, the graded trial results and the operational release decision. A reader should be able to follow one failed case from its business requirement through its observed tool calls to the resulting record and corrective action.

Include the number of distinct cases, the number of trials per case and the population the suite is intended to represent. List known gaps rather than using a single confidence label. Add a configuration manifest covering the model, retrieval pipeline, tool permissions and grading rules, with the sensitive inputs appropriately protected.

The final sign-off should say what autonomy is permitted, what still requires review and which production signals would trigger restriction. That evidence supports procurement and internal assurance because it demonstrates how the service is assessed and controlled, while keeping the claim bounded to the tested workflow.

Further questions

Questions this raises

What should an AI agent evaluation measure?

Measure task completion, factual support, permission compliance, resulting system state, latency, cost and review effort. The relevant combination depends on the business process and consequences of failure.

Can an LLM judge replace human evaluation?

A model grader can increase coverage, but its rubric and judgments need calibration against domain experts. Use deterministic checks wherever the expected outcome can be verified directly.

Does a perfect evaluation score make an agent safe to deploy?

No finite test suite proves universal reliability. A strong score supports a bounded release decision alongside enforced permissions, monitoring, recovery mechanisms and an agreed escalation process.

References

Sources & further reading

  1. Demystifying evals for AI agents 9 January 2026
  2. Evaluation best practices Ongoing guidance; checked 17 November 2026

Connected capabilities

AI solutionsSoftware & CRM

From thinking to doing

What could this mean
for your business?

Your next chapter

What could your
business do better?

Tell us where you want to go, what’s getting in the way, or what you need to deliver. We’ll start there.

Start a conversation

Start a conversation

Tell us what
you have in mind.

Interactive preview. Your information stays in this page and is not sent.