Why AI agents drift on the same task—and how to investigate it

An AI agent can receive the same request and produce a different result because the request is only part of the working conditions. Context, tool responses, saved state, model behavior, and the path taken through the task can differ. Investigate those differences before treating every failure as a prompt problem.

Here, drift means a departure from the task’s agreed requirements across or within runs. It is a practical description, not a claim that all such failures share one technical cause. Different wording can be acceptable. Omitting a required source, changing a calculation rule, or taking an unauthorized action is a different matter.

Check whether it was actually the same task

A weekly research prompt may stay unchanged while a competitor changes its prices. A form may gain a required field. A tool may return partial data, or a conversation may include a correction absent from the previous run. A different answer can be the right response to new facts.

Compare the full set of inputs: source material, instructions and skill revision, relevant conversation, model configuration, tool versions, permissions, and starting records. You do not need a giant logging system to begin. A small, controlled test case is enough to make the first comparison meaningful.

Evaluate the outcome, then inspect the path

Anthropic’s agent evaluation guide distinguishes a trial’s interaction record from the final state of its environment, and recommends multiple trials because outputs vary. An agent saying it saved a record and the record actually existing are separate observations.

The 2024 τ-bench paper also examines repeated task reliability. Its pass^k metric asks whether an agent succeeds across repeated trials, rather than merely finding one successful attempt. The benchmark uses specific simulated service settings; its historical results are not a failure-rate estimate for every current agent.

A small investigation you can run

1. Define success. For a research brief: required companies covered, factual claims supported, missing data disclosed, and no unsupported price comparison.
2. Freeze a representative case. Save a permitted source set and the exact instructions and tool configuration. Reset the test environment between attempts.
3. Repeat within a fixed budget. Inspect each outcome. Several trials can reveal a failure pattern, but a small sample cannot establish a production reliability rate.
4. Locate the first consequential difference. Did retrieval fail? Did extraction drop a condition? Did the draft ignore a correct intermediate result?
5. Change one cause and rerun. Keep the failing example as a regression case. Check that the fix still handles a case that previously worked.

Use objective checks for things such as required fields and arithmetic. Use a clear rubric and human review for interpretation. Requiring an identical paragraph would reject useful variation without proving that the work is better.

Preserve decisions that should not change each run

If a pricing comparison must always disclose annual billing, make that an explicit field and a check. If a task may draft but not submit, enforce that at the action boundary. These are proposed design responses to diagnosed failures, not a promise to eliminate all model variation.

PCI treats the repeatable work system as the design object: the outcome, known process, necessary judgment, authority, and evidence. Its aim is acceptable performance under declared conditions. Creative work can still take different paths while keeping a stable standard of review.