Turn an agent failure into a regression check

When an agent makes the same mistake twice, save more than the corrective prompt. Build a small test case that reproduces the conditions and checks the resulting work. That gives the next prompt, model, or tool change something concrete to pass.

This guide is for a team that already has an agent workflow. It proposes a compact regression pack, not a benchmark for ranking every agent on the market.

Choose one failure you can reproduce

Suppose a report-writing agent used an outdated fixture record after its search tool returned no results. The report read well, but its central number was wrong.

Create a sanitized fixture with a current record, an older conflicting record, and a controlled search response. Define the required behavior: use the current record when available; report unavailable evidence when it cannot be retrieved. Do not allow the agent to manufacture a value to complete the report.

Keep real credentials and customer data out of the fixture. Record the tool and environment configuration needed to reproduce the case.

At this stage, an unresolved failure is a case for improving capability. Once the corrected workflow passes consistently from reset fixtures, add it to the regression suite to detect later backsliding. Keep known failing cases separate from the behavior you already rely on.

Grade the resulting state

Anthropic's discussion of agent evaluations distinguishes a transcript from the outcome in the environment. That is a useful distinction for a regression check: a confident completion message is one piece of output, not the acceptance criterion.

For the report case, inspect the saved artifact. Does the number match the current fixture? Does the source link identify that record? When evidence is unavailable, does the artifact say so? Did the agent leave unrelated records unchanged?

Use deterministic checks for exact IDs, counts, and values. Have a reviewer assess whether the explanation accurately communicates uncertainty. If you introduce a model-based judge, compare its decisions with human judgments before trusting its score.

Add neighboring cases

Once the original failure is covered, add a few variations that exercise the same decision:

These are proposed fixtures. Their value comes from matching your workflow's actual risks. Avoid expanding the pack with cases that do not influence a decision your agent makes.

Compare changes under the same conditions

Reset the fixture before each run. Record the prompt version, model identifier where available, tool configuration, and artifact. Run both the current and proposed versions against the same cases.

For variable behavior, repeat trials and report the number of runs. A pass on a small pack is evidence about that pack; it does not establish a general reliability percentage. Investigate inconsistent results instead of hiding them behind a single average.

Record human repair separately. A report corrected by a reviewer can be accepted for use while the unassisted agent attempt still fails its check.

Keep the pack useful

Add a case when a real failure exposes a missing condition. Remove or revise cases whose assumptions no longer match the product, retaining the reason in the test history.

For a first product trial, start with A coding-agent trial you can actually score. A regression pack serves a later purpose: preserving behavior your workflow already depends on.

The fieldbook's Find out whether an agent actually helps introduces evaluation and learning from failures. Use this guide's fixture table and reset procedure to implement that idea for one recurring workflow.