Specification numbers are easy to remember: 200,000 tokens, 1 million tokens, or another large limit. They answer a capacity question, not the operating question: if relevant detail is buried in a long packet, will the model find it, cite the right evidence, and stay within a usable time and memory budget?

This is timely because current vendor documentation presents million-token context as an available capability. Google’s Gemini documentation describes many Gemini models as having windows of 1 million or more tokens. Anthropic’s context-window documentation describes windows up to 1 million tokens, while warning that more context is not automatically better and accuracy and recall can degrade as token count grows. Those are product facts and vendor claims, not a pass on a team’s task.

The proposed five-card review tests that gap with answer-bearing evidence near the beginning, middle, and end. It reports extraction, citation, latency, memory, and review burden separately. No benchmark is claimed; this is a protocol for producing one.

Card 1 — Turn the context limit into a claim you can check

Start with one job and one answer contract. “Read this archive” is too broad. Choose a bounded task such as identifying a renewal date in synthetic agreements or extracting fields from a fictional product brief. Define acceptable variants, required source IDs, and the response when the packet lacks the answer.

Record the advertised context limit separately from the intended operating point. A model may accept a million tokens while the workflow normally sends 40,000. Test a representative and a stress size, but do not call the latter normal. Pin model ID, revision, tokenizer, runtime, prompt, output limit, sampling, and date; any change opens a new comparison.

The paper “Lost in the Middle: How Language Models Use Long Contexts” explains why placement must vary: controlled question-answering and key-value experiments found performance changed with information location, degrading in the middle. That supports a test design, not a score transferable to a current model.

Card 2 — Build a packet with three positions and honest distractors

Create a synthetic or de-identified packet whose answer can be checked without asking a model for the key. Give each document a stable ID and each answer-bearing statement an exact span. Add distractors such as a different date, a superseded value, an irrelevant clause, and an explicit unknown. A test that is too easy measures pattern recognition, not retrieval.

For each case, make three position variants while holding content constant. Put the decisive span near the beginning, middle, and end of the same token budget. Randomize document order so a model cannot rely on filenames or headings. Include single-hop extraction and a few two-span questions requiring separately placed facts. Preserve the fixture and hashes for later replay.

Ask not only “Did it get the value?” but “Did it use permitted evidence?” Require source IDs and exact supporting spans. A missing answer stays “not found” or “unknown.” A fluent guess fails even if it happens to match an unsupplied fact.

Card 3 — Run paired arms across length and placement

Run the candidate small, open, or local model on the same cases at several lengths. Include a stronger reference only as a comparison, not an oracle. Use short, representative, and stress packets, each with all three placements. If a size is rejected, record the error and stop that arm; trimming changes the question.

Keep prompt and output format fixed. Predeclare repeats for nondeterministic generation and use the same cases in every arm. Add a baseline where a deterministic selector supplies only the answer-bearing span. It shows what long-context overhead buys; it does not prove a model is unnecessary.

Measure input tokens that reached the model, not source-file characters. Anthropic notes that instructions, messages, tools, documents, and generated output count toward its window and responses expose usage fields. Use the analogous record for the runtime under test. Keep cached-input accounting visible: lower billing does not mean fewer tokens occupy context.

Card 4 — Keep five scoreboards separate

First, report extraction: exact-answer rate, field completeness, and position-specific results. Show the best-to-worst placement gap; an average can hide a middle failure.

Second, report citation: whether the source ID and span entail the claim. Unsupported, stale, or mismatched citations fail even when the value is right. Track “unknown” and refusal separately.

Third, report latency: time to first token, end-to-end time, errors, and input/output counts. Label cold and warm runs. A vendor speed label is not a measurement for your machine.

Fourth, report memory: peak process and accelerator memory, load time, and headroom at each length and concurrency. Capture local OS/runtime readings; for a hosted API, label hardware memory unavailable rather than guessing. Capacity is not a concurrency guarantee.

Fifth, report review burden: verification minutes, reconstruction rate, escalations, and difficulty spotting plausible errors. Grade a blinded sample with one rubric. A fast answer that takes longer to prove may be slower overall.

Card 5 — Set the gate before you see the chart

Write the decision rule before running. An illustrative policy could require no critical-field false accepts, minimum extraction and citation floors in every position, latency and memory within budget, and review minutes within capacity. Passing the ends but failing the middle is not arbitrary-placement reliability. A special wrapper is a different contract.

Use three outcomes: pilot the named configuration, revise and retest, or hold it for this workflow. Keep thresholds task-specific. Do not create a universal leaderboard or report a result until fixture, manifest, outputs, errors, usage, and reviewer sheet are reconstructable.

Replay the holdout after a model, context, tokenizer, retrieval, prompt, runtime, or concurrency change. If compaction, caching, or tool use changes what enters context, create a new arm. The claim is then precise: this revision, packet, placement distribution, and review budget met or failed a stated rule.

The conclusion is modest. A context window is an admission limit. Reliability is an observed relationship between evidence position, answer use, source support, time, memory, and human checking. Test those relationships before a large number becomes a deployment assumption.

Sources and limitations

  • Google AI for Developers, “Long context” — checked September 10, 2026; supports the attributed availability and use-case description for Gemini context windows of 1 million or more tokens. Limitation: Google’s documentation is a vendor capability description and does not establish reliability on another model, packet, or workflow.
  • Anthropic, “Context windows” — checked September 10, 2026; supports the attributed explanation of context capacity, context accounting, usage fields, context rot, and overflow behavior. Limitation: these rules are specific to the Claude API and do not provide the proposed test’s results.
  • Liu et al., “Lost in the Middle: How Language Models Use Long Contexts” — checked September 10, 2026; supports the original controlled positional-evaluation method and its finding that relevant information can be harder to use in the middle of long inputs. Limitation: the paper’s models, tasks, and dates are not a benchmark of current systems; no model was run for this article.