A second AI request can feel better because it starts sooner and reports fewer uncached tokens. That may show input reuse; it does not show a revised instruction is clearer, an output is correct, or a total call is cheaper after writes, misses, retries, and output tokens.
This proposed, unexecuted lab note uses synthetic classification cases and a fixed oracle. An oracle is the expected-answer reference used to grade an output. It makes no measured performance claim: a cache read earns a systems observation, while the answer earns its own evidence.
Cache the prefix, not the conclusion
Prompt caching is provider-managed reuse for a matching prompt prefix; later input is still processed. OpenAI prompt caching
Anthropic documents exact matching up to a cache breakpoint, a five-minute default lifetime, and an optional one-hour lifetime. It says caching does not change output-token generation. Anthropic prompt caching
These are provider behaviors, not a universal contract. A cache hit means reusable input state was found under that provider’s rules. It is not a correctness or stability grade, or proof that the entire request was identical.
TTL means time to live: the cache-lifetime setting in this protocol. Treat it as an eligibility window, not proof that an entry was deleted when the clock expires.
For this protocol, put the rubric and examples before the boundary and the case or timestamp after it; check the stored prefix hash.
Freeze the contract before the first write
Write the test contract before warming the cache. Give every synthetic case a stable ID, input, and human-authored expected output. Choose the format and pass rule in advance; a format error and wrong label are separate failures.
Record the serialized input, prefix hash, prompt version, model snapshot, API surface, tools, output schema, reasoning or thinking settings, sampling controls, and maximum output. Also record cache mode, breakpoint, key, TTL, request time, response ID, and read/write/miss/no-data status.
Do not pair a new prompt sentence with a new model or evaluator. If caching cannot be disabled or cleared, say so in the contract. A new key is not an isolation boundary.
OpenAI says prompt_cache_key influences routing but does not pin a request or guarantee a cache read; use it as a routing field, not as proof of a cold or newly written state. OpenAI cache keys
Run paired cache and prompt arms
Use one synthetic fixture pack and five predeclared arms. The labels below describe the design, not results:
| Arm | Prompt version | Cache state | Question answered |
|---|---|---|---|
| A | v1 | Documented no-cache mode, or unknown | What is the baseline without saved prefix work? |
| B | v1 | First-write candidate | What does the write path cost and take? |
| C | v1 | Same prefix reused within its TTL | What changes on a cache read? |
| D | v2, one planned rubric change | First-write candidate for a changed prefix | What changes when the prompt changes? |
| E | v2 | Same v2 prefix reused within its TTL | Do the v2 read and output remain attributable? |
Run the same cases in each arm, randomizing order from a recorded seed and putting the case after the cache boundary. Predeclare the repeat count; five is a pilot, not a confidence guarantee. Arm A is a baseline only when the provider documents a disabling configuration. “No breakpoint” is not a general no-cache condition where implicit or automatic caching remains enabled. Without such a configuration, label the baseline unavailable or unknown.
For B and D, accept a first-write observation only when telemetry shows zero cached tokens and the intended write tokens, or the documented equivalent. Classify each request as cold, write, read, or unknown from telemetry plus prefix identity. Namespace, TTL, and elapsed time alone cannot establish that classification; TTL can be a minimum eligibility period, not a deletion deadline.
The cache comparison changes only cache condition. The prompt comparison changes one named instruction while holding cache treatment, cases, and settings constant. Even after controls and prefix hashes pass, stochastic variation remains: compare repeated paired score distributions. A single changed answer is an observation, not causal proof that v2 caused it.
Score reliability on its own ledger
Inspect raw output against the oracle, not the cache field. Record exact-label correctness, format validity, refusal or incomplete status, and repeat agreement. Agreement measures consistency, not correctness; a stable wrong label is still wrong.
Compare v1 and v2 on the same case IDs. Preserve every output, including failures, so a retry cannot erase the first attempt. If the task returns an explanation, grade the decision separately from optional prose. A cache hit earns no reliability points; it is an explanatory variable.
OpenAI’s evaluation guide describes specifying criteria, running test inputs, then analyzing and iterating; it currently marks hosted Evals for deprecation. Keep the fixture, oracle, and verdict ledger outside a vendor service. Working with evals
Measure time and money from raw fields
Capture time to first token, total response time, queue time where available, retries, and errors. Report cold and warm distributions separately, with a predeclared median and tail measure. One fast warm response is not a latency win; keep provider timing and lifetime semantics in the record.
Build cost from billed categories rather than a single “input tokens” column:
total cost = uncached input × base rate
+ cache reads × read rate
+ cache writes × write rate
+ output tokens × output rate
+ retries and other billed work
OpenAI reports cached_tokens and cache_write_tokens in usage details. OpenAI usage
Anthropic separates cache_read_input_tokens and cache_creation_input_tokens from post-breakpoint input_tokens. Anthropic usage
Use those provider fields with the model and date’s rates; do not copy a multiplier between providers.
Rates vary by provider. Recalculate with the model and date used, including writes, misses, expiry, retries, and output. A discounted read can coexist with a more expensive experiment when reuse is rare.
Call a result only when the proof holds
Issue separate verdicts: cache_observed for matching telemetry and prefix identity; latency_observed for comparable timing distributions; reliability_observed for the same oracle and output rules; and cost_observed for complete billed categories and current rates. “Improved” requires the relevant verdict, not a cache badge.
Hold when a prefix hash, model, tool definition, output schema, sampling setting, evaluator, or case changes unexpectedly, or when a cold arm was already warm, TTL timing is unknown, usage fields are absent, or a retry replaced the first output. If v2 is less reliable or more expensive, report that tradeoff alongside any cache observation.
This five-arm paired ledger makes saved input work, prompt version, latency, output reliability, and total cost independently auditable. Use it for one scoped task and provider setup, not a general benefit or model ranking.
Sources and limitations
- OpenAI, “Prompt caching” — checked September 4, 2026. Supports matching prefixes, model-dependent minimums, routing keys, lifetime settings, usage fields, and cost-accounting guidance. Limitation: a key does not guarantee a hit or isolate a write; availability and retention vary.
- Anthropic, “Prompt caching” — checked September 4, 2026. Supports exact breakpoints, documented TTL options, usage fields, and output-generation scope. Limitation: controls, fields, minimums, and pricing are Anthropic-specific and vary by model/platform.
- OpenAI, “Working with evals” — checked September 4, 2026. Supports criteria, test inputs, analysis, and iteration; it also records the current Evals deprecation timeline. Limitation: hosted Evals is scheduled to become read-only on October 31, 2026 and shut down on November 30, 2026.