Tool use is a chain, not a single event. OpenAI describes a request with tools, a model-generated call, application-side execution, a second request carrying the result, and a final response or another call (OpenAI’s function-calling guide). An outer request can therefore return successfully, a runner can reach a final turn, and the task can still be missing evidence.

The gap appears when a task requires several results. If an agent needs an order, policy, and region record but only two arrive, the answer may still sound complete. A green HTTP status says that transport finished. It does not prove that every required call returned, that each result has the fields needed for the decision, or that the final sentence stayed inside the evidence.

This is a proposed, unexecuted test. It uses synthetic read-only tools and an authored oracle to make partial failure visible before a team trusts a production answer. No measurement or provider-wide conclusion is claimed.

Define the fixture before calling a model

Create ten synthetic cases with no real customer, employee, account, or contact data. Each case has a user request, versioned tool manifest, expected call set, required output schema, authoritative answer key, and allowed safe-stop message.

Use a harmless decision such as: “For synthetic order DEMO-1042, is the return request inside the published window? Use the order, policy, and region tools. Report the decision, evidence identifiers, and uncertainty. Do not infer a missing value.” The tools might be fetch_order, fetch_policy, and fetch_region, all read-only fixtures.

The clean case returns one valid result for each call. Fault cases remove a result, omit policy_window_days, set a field to null or the wrong type, mark an execution error, duplicate or mismatch an identifier, reorder results, or deliver one after the deadline. Keep tool values and the requested decision constant across arms; change only the injected fault and handling policy.

Identifiers are the spine of the fixture. OpenAI’s function-calling guide places a call_id on each function call and sends the application’s output back with that reference. Anthropic’s tool-use contract similarly uses an id on tool_use and a matching tool_use_id on tool_result, with ordering requirements (Anthropic’s tool-call handling guide). The test should therefore grade identity and completeness, not merely whether some text was returned.

Run a permissive arm and a guarded arm

Arm A is the status-only baseline. It sends the fixed request, executes the synthetic tools, forwards whatever results the adapter collected, and accepts a final answer when the outer request or model turn reports completion. It records provider rejections but adds no application-level completeness gate. This is the shortcut of treating “the call finished” as “the work finished.”

Arm B adds a contract-aware gate while keeping the prompt, model configuration, manifest, fixture values, and fault schedule the same. After calls are emitted, the harness maps results by identifier. Before finalization, it requires one result per expected call, rejects unknown or duplicate identifiers, parses each envelope, surfaces execution errors, and checks required fields for type and non-null status.

If a result or field is missing, Arm B returns incomplete, names the missing call or field, and stops short of a recommendation. A retry or human investigation is a separate next action. It must not substitute an empty object, cached value, or model guess. Passing envelopes still receive a semantic check against the oracle.

Strict tool schemas help with generated arguments, but do not establish that the application returned every result. OpenAI documents strict mode as function-call schema adherence, while MCP defines structuredContent and isError on a result and recommends validation, timeouts, and logging (MCP tools and error handling). Input validity and output completeness are separate gates.

Make the oracle the source of truth

Do not let the model define required evidence after the fact. Store the expected call set and fields in the case file. The oracle should state:

  • the expected call identifiers or logical tool names;
  • the required fields and their types for each result;
  • the fixture values that support each permitted conclusion;
  • which final claims are acceptable; and
  • the exact conditions for a safe stop.

Evaluate in layers: compare call sets, validate envelopes and fields, then grade whether claims follow from the facts. Parallel delivery should not fail merely because order changed; a wrong or duplicate identifier should fail even if values look plausible.

Use four labels: complete-correct for complete evidence and an oracle-agreeing answer; safe-incomplete for a stop that names missing evidence; unsafe-complete for a decision despite incomplete evidence; and complete-wrong for a complete envelope with a contradictory answer. An execution error and a missing result are different causes, but both block completion. MCP distinguishes protocol from tool-execution errors, while Anthropic documents is_error; preserve that distinction instead of writing only “API failed.”

Log the boundary, not just the answer

Every run needs a durable record. Capture run and case IDs, arm, provider/model version, prompt and manifest hashes, fixture version, expected calls, emitted names and identifiers, transport status/request ID, per-tool timings, timeout/retry, error flags, field checks, validator version, final state, and answer hash. Store raw payloads only when permitted; redact fixtures if they gain realistic structure.

A trace can connect model turns to tool spans, but it is not an oracle. OpenAI’s Agents SDK documents traces for generations, tool calls, handoffs, guardrails, and custom events, with spans carrying timing and parent relationships. It warns that generation and function spans may capture sensitive data and provides a control to disable capture (Agents SDK tracing). Use that or an equivalent redaction policy. A missing log entry is an observability failure, not an empty result.

Compare pairs, then read back the persisted state

Run every case once in each arm as a pair. Randomize arm order, hold the model snapshot and tool responses constant, and avoid an unplanned first-pass retry. If the provider is nondeterministic, record the supported seed or treat repeats as new pairs. Report injected-fault detection, false-completion rate, call/field localization, safe-stop clarity, recovery attempts, latency, and review burden separately. These are measurements to collect, not results to assume.

Read back the saved run record, not the terminal screen. Recompute set equality, inspect required fields, compare the outcome with the oracle, and verify that a serializer, dashboard, or retry worker did not turn incomplete into complete. For read-only fixtures, confirm that retries do not duplicate records; add side-effect checks before adapting the test to a real workflow.

The pass condition is narrow: under this fixture, the harness never presents an unsupported decision as complete, and every clean case remains usable. That does not prove universal reliability. It proves the system can see the difference between a successful request and a complete evidence chain. A partial tool-call failure then becomes a recoverable state instead of a polished surprise.

Sources and limitations

  • OpenAI, “Function calling” — checked September 10, 2026; supports the multi-step tool-calling flow, call identifiers, parallel-call behavior, and the distinction between strict function-call arguments and application handling. Limitation: OpenAI’s guide describes its APIs and does not define every provider adapter or runner.
  • Anthropic, “Handle tool calls” — checked September 10, 2026; supports tool_use identifiers, matching tool_result blocks, ordering requirements, empty results, and is_error handling. Limitation: these rules describe Anthropic’s Messages API and are not a universal interchange contract.
  • Model Context Protocol, “Tools” — checked September 10, 2026; supports CallToolResult, optional structuredContent, isError, the protocol/tool-error distinction, and recommendations to validate, time out, and log tool calls. Limitation: MCP fields are partly optional, so a host still needs an application-specific completeness contract.
  • OpenAI Agents SDK, “Tracing” — checked September 10, 2026; supports trace/span coverage for generations and tool calls, timing and parent relationships, and sensitive-data capture controls. Limitation: telemetry helps reconstruct a run but cannot decide whether a case’s required evidence is complete.