Small-model-first routing sounds like a cost decision. It is really an acceptance decision. When the small model answers and the router accepts, a routing error can become a user-facing error. When the router escalates a response that was safe to accept, quality may be intact but the system has spent extra compute or review time. A useful evaluation must show both kinds of mistake.
The reader question is: can a cascade handle routine work cheaply without quietly accepting the wrong answers? Research gives us the shape of the idea. FrugalGPT describes a sequential cascade with a generation-scoring function and a router. RouteLLM frames dynamic selection between stronger and weaker models as a quality-cost tradeoff. Neither paper supplies a threshold that can be imported into your task. The five cards below are a proposed, reproducible evaluation for choosing one.
Card 1 — Define an accepted result before you route anything
“Accepted” must mean more than “the small model returned text.” Write the answer contract first. For each case, specify the required fields, allowed values, evidence that must be present, whether an explicit unknown is valid, and the action a reviewer would take. Use three gold actions: accept, escalate, and reject. A case with no answer in the supplied evidence can have unknown as the correct answer; it should not be forced into a guess.
An illustrative synthetic fixture has 80 cases, split evenly across four strata: routine extraction, ambiguous wording, long or noisy context, and boundary cases where the source is incomplete. The count is a proposed starting point, not a measured sample-size claim. Each JSONL row carries:
{
"case_id": "fixture-001",
"input": "synthetic task text",
"allowed_evidence": ["source-snippet-1"],
"gold_answer": {"field": "known value"},
"gold_action": "accept",
"risk_tier": "routine",
"acceptable_variants": ["known value"],
"expected_human_action": "no change"
}
Label difficult cases deliberately. Include near-duplicates that differ by one missing qualifier, a null value, or a changed date. Keep the fixture synthetic or de-identified and do not let the model generate its own gold labels. Stop if a case has no independent reference, no action label, or no rule for missing evidence. A cascade cannot be judged against a definition of success that moves after the output arrives.
Card 2 — Freeze three lanes against the same cases
Run three lanes over the same fixture: the approved stronger-model baseline, the small model by itself, and the small-model-first cascade. The baseline answers every case. The small-only lane shows what the first model can do without routing. The cascade records whether it accepts, escalates, or rejects and what final answer reaches the user. These lanes turn “cheaper” into a comparison rather than a slogan.
Before execution, pin model, prompt, schema, runtime, template, context, sampling, retrieval policy, and date in run-manifest.json. Use identical input order and evidence packages across lanes. Predeclare repeats for nondeterministic generation; do not tune one lane after seeing its holdout results.
OpenAI's current eval guidance describes an eval as test data plus testing criteria, with a schema for each item and human-provided ground truth available for comparison. As of September 2, 2026, that guide also lists a scheduled read-only date and shutdown date for its legacy Evals platform. Keep the fixture as portable JSONL or CSV so the experiment survives a platform change. This article reports no results: the lanes and manifests are a proposed protocol, not a performed benchmark.
Card 3 — Keep the runtime gate separate from the answer key
The deployed router must never read gold_answer, gold_action, acceptable_variants, or offline facts_ok. Keep those fields behind the evaluation boundary. Otherwise the cascade appears to know the correct answer before deciding whether to trust its output.
Define a gate from observable runtime signals. One proposed extraction gate checks schema validity, permitted source IDs, exact supporting spans, missing-source flags, and task-policy rules. A versioned scoring model may add a threshold fitted only on training data. These signals establish format, source binding, or a scored estimate, not guaranteed factual correctness. Self-reported confidence is not truth.
Declare which failures trigger escalation and which require rejection. An escalated stronger-model answer must pass the declared final gate too; escalation is not automatic acceptance or human review. Record any required human gate separately. A lane using gold answers to accept outputs is an upper-bound diagnostic, not the deployable cascade.
After routing, grade three independent offline passes: parser validity, factual/evidence agreement with held-out gold, and the appropriateness of the routing action. Keep runtime signals and offline verdicts in separate ledger columns.
Preserve a schema-valid invented-date case. A missing supporting span may trigger escalation at runtime. A misleading but real quotation might instead fool the gate; the independent oracle must catch that false accept. Conversely, escalating a correct, permitted answer may waste a model call without creating a human-review task. Neither error disappears because the other lane has a green score.
Card 4 — Count accepted-result cost, false accepts, false rejects, and review burden
Keep a ledger at case level, then summarize by risk stratum. Define the metrics before the run:
- False accept: a runtime release that fails independent factual, evidence, or policy grading, or should have escalated or been rejected under the gold action.
- False reject: escalation of a correct small-model result whose gold action is
accept. - Accepted results: final outputs independently meeting the same contract in every lane, not the router's initial accept count. Count successful escalated or reworked outputs once. Count
unknownonly where the contract permits it and gold confirms it. Exclude failures and unresolved cases; a zero denominator yields undefined cost, never zero cost. - Accepted-result cost: total model-call, runtime scoring/validation, human-gate, and rework cost across all attempted cases, divided by accepted results. Include failed attempts. Keep compute units and minutes separate until approved rates convert them; report offline evaluation cost separately as experiment overhead.
- Review burden: human-gate cases, review/rework minutes, and queue delay, alongside model-only escalation counts. Record scoring/validation latency and end-to-end latency, not just generation time. Report totals and rates per 100 cases.
For a local runtime, capture measured seconds and token counts; for a hosted lane, capture the provider's billed units when available. The current llama.cpp server documentation describes response timing fields such as prompt and predicted token counts and exposes context and parallel-slot settings. Those fields make a useful runtime ledger, but they do not measure answer quality.
Do not hide a high-risk failure inside an average. A two-column summary can be enough: gold_action × router_action, with a second table for parser, factual, and evidence outcomes. The reviewer should be able to open one row and see the input, evidence, answer, route reason, model calls, and correction. If a false accept is found, preserve the exact artifact and prompt that produced it. If the ledger cannot reconstruct the path, classify the case as unknown rather than silently successful.
Card 5 — Set the gate, then replay it after every change
The decision is a policy choice informed by the ledger. Write it before looking at results. An illustrative gate might read:
| Decision | Required evidence |
|---|---|
| Pilot the cascade | No high-risk false accepts, accepted-result cost below the stronger-only baseline by the owner's preset margin, review burden within capacity, and complete route logs |
| Tune the router | High-risk false accepts are absent, but false rejects, latency, or review minutes exceed the owner's limit |
| Hold or reject | Any material high-risk false accept, missing gold evidence, unlogged fallback, or unreproducible model/runtime state |
These are decision rules, not universal safety thresholds. A zero false-accept target may be appropriate for one task and too weak or too strict for another. Make the risk owner sign off on the boundary, and keep a separate rule for cases that must always escalate.
Reserve a holdout set that the router never sees during tuning. Re-run it when the model, quantization, runtime, prompt, schema, retrieval source, or risk policy changes. Compare route distribution as well as answer accuracy: a system that keeps quality by escalating nearly everything has changed its operating economics. Monitor a small canary after release and sample accepted cases for independent review. Route logs should be redacted and retained only as long as the task requires.
The decision packet joins quality and economics without giving the runtime an answer key. A cheap route is useful only when independent grading, complete costs, and review capacity support the same decision.
Sources and limitations
All sources below were checked September 2, 2026.
- Working with evals, OpenAI API documentation — supports the use of a test-data schema, human ground truth, testing criteria or graders, and the current legacy Evals transition dates. Limitation: this is a vendor guide, and the proposed protocol is intentionally portable beyond that platform.
- FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance — supports the sequential-cascade, generation-scoring, and router concepts, plus the paper's warning that labeled examples should match the test distribution. Limitation: its reported experiments used its own models, tasks, and cost assumptions; no result is transferred here.
- RouteLLM: Learning to Route LLMs with Preference Data — supports the quality-cost framing for dynamically selecting stronger and weaker models. Limitation: its preference-data training and evaluation setting do not establish a threshold for another workflow.
- llama.cpp server documentation — supports the runtime timing, token-usage, context, and parallel-slot fields proposed for instrumentation. Limitation: those fields describe server behavior and resource use, not factual correctness or review quality.
No model, router, or runtime was run for this article. All fixture sizes, gates, formulas, and examples are proposed or illustrative. A real decision still needs approved test data, an independent reference, an owner-set risk policy, and a readback of every model call.