The reader question is simple: if one sentence in a controlled document changes, does the answer change with it, or does retrieval serve yesterday's chunk under a confident explanation?

This is a proposed, unexecuted lab protocol using harmless synthetic text in an isolated non-production test environment. Test-only index mutations are the subject; no external side effect is allowed. It separates two assertions:

  1. Freshness: the answer reflects the changed fact.
  2. Lineage: the retrieved context and citation identify the changed revision.

Both are required. “Five days” without a new source may be coincidence; a genuine new citation with the old number is a value-freshness failure.

The system under test

Freeze the path, not just the prompt: source, ingestion job, index, retrieval settings, context assembler, answer model, cache, and citation renderer. Record model snapshot, instructions, tools, filters, result limit, sampling, and run time. Any change can explain a different answer.

The current OpenAI documentation is a concrete reference: file search is a hosted Responses API tool using semantic and keyword search, and vector stores contain files that are chunked, embedded, and indexed. File Search and Retrieval are provider-specific, but the test logic is portable.

Do not treat a similarity score as a truth score. The vector-store search reference returns a score, filename, file ID, and content chunks. That makes retrieval inspectable, not necessarily current.

Fixture pack: paired, synthetic, and intentionally boring

Create two UTF-8 text files whose non-changing lines are byte-for-byte identical. Use distinct filenames so the readback cannot hide an object swap.

File: shipping-policy--v1.txt
Fixture: SYN-RET-001
Document key: shipping-policy
Revision: 1
Effective: 2026-08-01
Revision marker: GLASS-ONE
Standard dispatch: 2 business days.
Express dispatch: 1 business day.
Support hours: 08:00-18:00 ET.

File: shipping-policy--v2.txt
Fixture: SYN-RET-001
Document key: shipping-policy
Revision: 2
Effective: 2026-09-02
Revision marker: GLASS-TWO
Standard dispatch: 5 business days.
Express dispatch: 1 business day.
Support hours: 08:00-18:00 ET.

These are synthetic fixtures, not a real policy or secrets. Hash both before upload. Standard dispatch is the changed fact, the marker is a readback check, and express dispatch plus support hours are invariants. Add PDF, scan, or localization variants only after this small test is understood.

Attach queryable attributes such as document_key, revision, fixture_id, and active. Keep a manifest mapping each file ID to filename and SHA-256. Retrieval attributes narrow search but do not replace content readback.

Baseline: prove version one is actually in play

Create an isolated test index and load only v1. If ingestion is asynchronous, wait for the documented ready state and record transitions. OpenAI says a completed vector-store-file status indicates readiness; use that receipt rather than an upload response alone. Vector-store file operations

Run this exact query with a fresh conversation or request context:

Using only the indexed document with document key shipping-policy, answer:
What are the standard dispatch days and support hours? Return the revision,
the two values, the source file ID, and the source filename. If the source is
absent, say unknown. Do not infer a value.

Keep result limit, ranking, filter, and instructions fixed. Capture raw request and response, file IDs, filenames, scores, chunks, answer, and citations. The fixture oracle expects v1, two business days, 08:00-18:00 ET, and GLASS-ONE in context. It is not a reported model result. If the run differs, invalidate the baseline and stop; otherwise freshness cannot be separated from a bad harness.

If generation is nondeterministic, repeat three times and retain every trace. Do not average answers; one prohibited old citation remains a finding.

Apply the change as a controlled replacement

Add v2 with the same document key and a higher revision, then remove or deactivate v1 using the same replacement logic production uses, reproduced only in this isolated test index. Record old and new object IDs, or the attach-new and remove-old operations. An attribute update is not proof that the parsed body changed.

OpenAI warns that file operations can be asynchronous and vector-store removal eventually consistent, so search may briefly return removed content. Retrieval file operations Define a bounded freshness window, log the clock, and poll until v2 is ready and the active set is reflected. If no provider bound exists, choose a local timeout and classify timeout as inconclusive.

Do not add “version two” to the query. Run the original query in a new request context, then exact-replay it if the application has a cache. The pair distinguishes index from cache staleness.

Read back the source, retrieval, and answer

Perform three readbacks, in order:

  1. Object readback. List active files, retrieve status and attributes, confirm v2 is ready, and confirm v1 is absent or inactive after the window. Preserve the response.
  2. Content readback. Fetch parsed v2 content and compare the changed sentence, marker, invariants, and normalized hash to the manifest. Retrieve vector-store file content
  3. Answer readback. Inspect chunks and answer together. The pass requires GLASS-TWO, revision 2, five days, unchanged support hours, and a citation mapped to v2's file ID. Duplicate filenames without stable mapping are a lineage gap.

Add cache state to every row: cold, warm, bypassed, or unknown. If cold sees v2 but warm sees v1, the index may be fresh while the cache is stale. Report layers separately. An answer change without changed context or citation is unsupported coincidence.

Failure rule and stop condition

After the window, fail if:

  • v1 or GLASS-ONE appears in retrieved context or citations;
  • the answer still says two business days;
  • v2 is ready but is never retrieved for the fixed query;
  • the answer says five days while citing v1;
  • both revisions are returned without an explicit, manifest-backed selection of revision 2;
  • the source content readback does not match the v2 fixture;
  • the response has no machine-readable source identity when the system promises citations.

Stop on an external side effect, permission crossing, or fixture contamination. Never place the synthetic source in production.

Mark inconclusive when the baseline is invalid, ingestion never becomes ready, deletion remains inside an undocumented window, cache state is unknown, or retrieval evidence is insufficient. “The answer looked right” cannot replace telemetry.

Use this result sheet:

case_id | phase | query_hash | active_file_ids | retrieved_file_ids |
v1_marker_seen | v2_marker_seen | answer_value | cited_file_id |
cache_state | object_readback | content_readback | verdict | next_decision

The original contribution is a two-axis freshness assertion: grade changed value and lineage independently, then add cache state as a diagnostic layer. A pass covers this source, update path, query, and readback contract, not retrieval generally. Re-run after changing the indexer, chunking, ranking, model, cache, or citation layer, then expand for OCR, long or multilingual documents, duplicate names, and concurrent updates.

Sources and limitations

  1. OpenAI, “Retrieval” — checked September 2, 2026. Supports vector-store chunking, embedding, indexing, asynchronous file operations, eventual-consistency warning for removal, and file attributes. Limitation: OpenAI-specific behavior; it does not provide a universal freshness SLA.
  2. OpenAI, “File search” — checked September 2, 2026. Supports the hosted file-search surface, semantic and keyword retrieval, result limits, and metadata filtering. Limitation: a provider guide does not establish that a particular application cache or citation renderer is current.
  3. OpenAI API Reference, “Search vector store” — checked September 2, 2026. Supports recording file ID, filename, score, and returned content for retrieval readback. Limitation: scores are retrieval signals, not correctness or recency guarantees.
  4. OpenAI API Reference, “Retrieve vector-store file content” — checked September 2, 2026. Supports fetching parsed content by vector-store file ID. Limitation: parsed output may differ in whitespace or extraction details from the original bytes, so the protocol compares both raw fixture hashes and meaningful content markers.