The Retry Policy Is Part of the Score

One “pass” can describe two very different runs: a correct answer on the first attempt, or a failure followed by several retries, tool calls, and a person repairing the result. Both may end in a correct state; they do not offer the same reliability, speed, or operating cost.

OpenAI’s evaluation guidance recommends task-specific tests, complete logging, and human judgment alongside numerical measures. Its agent-evaluation documentation describes traces that preserve model calls, tool calls, guardrails, and handoffs for a run. NIST’s August 2026 initial public draft of the TEVV-Athlon framework likewise treats evaluation as a planned assessment: define the goal, construct measures and tools, apply them, then interrogate the evidence. The protocol below extends those ideas to one question that scorecards often hide: what did the retry policy contribute? OpenAI evaluation best practices OpenAI agent evaluation guide NIST TEVV-Athlon initial public draft

Set the one-attempt baseline

Choose a task with an observable pass condition, such as answering a question from a fixed packet and returning a structured result that satisfies an agreed schema. For a tool-using workflow, define the expected final state as well as the correct text. A success message is not evidence that a reservation, update, or lookup actually occurred; the test harness must read back the mock system state.

Freeze the model or service release, instructions, application code, retrieval corpus, tool schema, permissions, settings, and case fixture. Use a fixed set of representative and difficult cases, with the expected answer or state written before running. Decide whether an external environment outage is a system failure or a documented invalid run; do not make that decision after seeing which candidate lost. If the harness itself fails, preserve the evidence and report coverage rather than quietly removing affected cases.

The first-attempt success rate is:

Cases that pass on attempt one ÷ all eligible cases.

Attempt one includes every expected step up to the first result: model response, required tool call, and state readback. A rejected tool call, timeout, wrong answer, unsafe extra action, or missing result counts as a first-attempt failure under the declared task rules. Also report how many cases actually ran. A quality score calculated only on completed answers can hide whether the workflow often fails to produce one.

If the only change under study is retry behavior, the initial attempt in the retry-enabled run can serve as this baseline. If the model, prompt, tool permissions, or fixture also changes, label the result a new workflow comparison; the retry policy no longer explains the difference by itself.

Specify what may be retried

Write the retry policy before execution. Define the maximum extra attempts, the error classes that permit each retry, the delay or backoff, whether state is reset, and who may intervene. Name each attempt type separately:

• Transport retry: a client resends after a declared transient connection or service error.
• Tool retry: the workflow repeats a tool call after the tool returns an error or incomplete response.
• Model retry: the system asks the model to generate or repair an answer again.
• Human escalation: a reviewer checks, corrects, or completes the task.

These are different interventions. A model regeneration is not a free repeat of the original attempt: it adds time, compute, and another chance to change the result. A person’s correction may produce a useful final deliverable, but it is a human-assisted outcome, not an automated recovery.

For a reproducible comparison, choose a fixed cap—for example, one initial attempt and at most two automatic retries for predeclared transient errors. The number is a protocol choice, not a universal best setting. Keep the error classes and cap identical across candidates when comparing them. If a retry may repeat a state-changing action, first check the mock system’s readback or idempotency record; when the state is ambiguous, stop and escalate instead of issuing a duplicate action.

Preserve the attempt trail

Store one immutable row per attempt, connected to a case identifier and a parent run. Include the build and fixture versions; attempt number and type; start and end times; request and response identifiers; tool name, arguments, and response; error class; retry decision and reason; resulting state; and the scorer’s decision. Preserve failed, timed-out, and cancelled attempts. The case summary is a view over the trail, not a replacement for it.

Redact personal or secret values before storing traces, and set access and retention rules for the evidence. Keep a record of redactions and missing fields, because a trace that no longer contains the decisive tool response may not support a later audit. OpenAI’s trace model is a useful platform example of preserving connected calls and handoffs; NIST’s draft framework emphasizes that measurement tools and methods should be chosen for the system, context, and objective rather than treated as one universal score.

Publish a score vector, not a retry-shaped pass rate

Use separate measures with their denominators visible:

• First-attempt success: cases passing on attempt one divided by eligible cases.
• Within-budget automated success: cases that pass after zero or more permitted retries divided by eligible cases.
• Recovery rate: first-attempt failures that pass within the retry cap divided by all first-attempt failures. Keep this beside first-attempt success; do not use it to overwrite that result.
• Failure shape: counts and rates by wrong answer, rejected tool call, tool error, timeout, missing state, duplicate action, policy violation, or other preregistered class.
• Latency: elapsed time from initial request to verified final state, including retry delays. Report the median and a tail measure such as the 95th percentile; list timeouts separately and measure human review time separately.
• Cost: total model, tool, infrastructure, and review cost over all eligible cases, including failures. Report cost per case and, where useful, cost per verified automated pass. State the rate card or internal cost method and the date used.
• Human review: cases referred, review time, overrides, and final outcomes. Mark any human-assisted pass separately from an automated pass.

Do not collapse these into one weighted score unless the decision-maker has documented why each tradeoff and weight reflects the task’s consequences. Averages also deserve a check: a retry policy can improve the mean while creating a long tail of slow or unresolved cases.

Read a synthetic ledger and set a stop rule

The following invented example uses ten mock cases and arbitrary cost units. It is arithmetic for illustration, not a test result or vendor price.

Workflow A is faster and lower cost in this toy ledger, but one case still needs human review. Workflow B has a higher first-attempt rate and completes all ten within its retry cap, at greater latency and cost. The numbers cannot tell a team which tradeoff is acceptable; the preregistered threshold and impact of an error must do that. In a real evaluation, keep the case-level results and uncertainty alongside the summary, and use enough representative cases to support the decision being made.

Stop the run if a retry exceeds its cap, the build or fixture changes, a tool action escapes the mock environment, or the final state cannot be read back. Treat an unrecovered error as a failure, not as missing evidence to be dropped. Before signing the report, verify that every eligible case has a terminal classification, every attempt is present, all costs include failed work, and the summary recomputes from the event log.

This protocol measures a particular system and policy under declared conditions. It cannot establish behavior in every future environment, and a repeatable score can still reward a poor proxy. Revisit the cases, error categories, and human-review rubric when the workflow or its context changes.

Sources and limitations

• NIST, “The TEVV-Athlon Framework for Evaluating AI Systems” (NIST AI 200-2 initial public draft) — checked October 21, 2026; provides a proposed four-stage structure for customized AI assessments and cautions that benchmark scores may be proxies for real-world outcomes. Limitation: this remains an initial public draft; its comment period closed October 6, 2026, and it is not a binding standard.
• OpenAI, “Evaluation best practices” — checked October 21, 2026; supports task-specific datasets, logging, human judgment, and explicit evaluation design. Limitation: the page describes an Evals product workflow; OpenAI’s current deprecation notice says existing Evals become read-only October 31, 2026 and the platform is scheduled to shut down November 30, 2026.
• OpenAI, “Evaluate agent workflows” — checked October 21, 2026; describes trace records for model calls, tools, guardrails, and handoffs, plus repeatable dataset-based evaluation. Limitation: product surfaces may change during the documented Evals transition; this article’s event ledger is a platform-neutral protocol adaptation.