The reader question is not “Did the model return JSON?” It is “Does each value earn its place from the supplied evidence?”
Current Structured model outputs documentation promises adherence to a supplied JSON Schema, while the JSON Schema specification defines validity through structural constraints. A document can therefore be type-correct and wrong relative to a source. That boundary belongs to the application, not the parser.
This proposed, unexecuted protocol uses synthetic incidents, an authored oracle, and two passes: safe consumption, then factual and evidence agreement. No model result is claimed.
Set the contract before asking for JSON
Write field meanings and failure behavior before the prompt. Here, high severity routes, absent or conflicting temperature is unknown and goes on hold, and an owner is allowed only when named by the fixture. “Unknown” is not permission to guess.
Use a pinned validator and provider-compatible subset. The current OpenAI guide lists object, array, enum, and anyOf as supported Structured Outputs types and warns that unsupported features can error. Supported schemas Every key below is required; owner is nullable rather than omitted:
{
"type": "object",
"additionalProperties": false,
"properties": {
"case_id": { "type": "string" },
"severity": {
"type": "string",
"enum": ["low", "medium", "high", "unknown"]
},
"decision": {
"type": "string",
"enum": ["route", "hold", "no_action", "unknown"]
},
"owner": {
"anyOf": [
{ "type": "string" },
{ "type": "null" }
]
},
"evidence": {
"type": "array",
"items": {
"type": "object",
"additionalProperties": false,
"properties": {
"source": { "type": "string" },
"quote": { "type": "string" }
},
"required": ["source", "quote"]
}
},
"unknowns": {
"type": "array",
"items": { "type": "string" }
}
},
"required": [
"case_id",
"severity",
"decision",
"owner",
"evidence",
"unknowns"
]
}
Ask for only this object, exact fixture source IDs and supporting quotes, and JSON null for an unnamed owner. Record schema hash, prompt, model snapshot, sampling settings, and validator version.
Fixture pack and gold oracle
All text below is synthetic and plain enough to inspect without a hidden database.
| Case | Synthetic source text | Authored oracle |
|---|---|---|
| INC-A | incident-A: “Battery sensor reports 82 C at 09:10. No injury.” runbook-A: “High severity is 80 C or above. Incident owner is not listed.” | severity high; decision route; owner null; unknowns ["owner"] |
| INC-B | incident-B: “Battery sensor reading unavailable at 09:10. No injury.” runbook-B: “A missing reading cannot establish high severity. Route only at 80 C or above. Incident owner is not listed.” | severity unknown; decision hold; owner null; unknowns ["sensor_temperature", "owner"] |
| INC-C | incident-C: “Logger A reports 74 C and logger B reports 84 C at the same time.” runbook-C: “Route at 80 C or above after resolving conflicting readings. Incident owner is not listed.” | severity unknown; decision hold; owner null; unknowns ["temperature_conflict", "owner"] |
The evidence oracle holds exact source IDs and short supporting spans. INC-A uses the reading and threshold; INC-B uses the unavailable-reading fact and missing-reading rule; INC-C uses both conflicting readings and the route rule. No oracle contains a person’s name.
Add three controls: a hand-authored positive matching each oracle, a parser-negative with a missing key or extra property, and a schema-valid semantic-negative such as INC-B with high, route, and owner “Mara.” This is an illustrative fixture, not a model response. If it passes the factual gate, the evaluator is not testing meaning.
Pass one: parser and schema
Preserve the raw response. Branch first on refusal, truncation, or incomplete status. The Structured Outputs guide identifies refusals and token-limit incompleteness as cases where output may not match schema. Structured Outputs edge cases Record “shape_fail” or “inconclusive”; never turn refusal into unknown.
Parse raw bytes without repairs or coercion. Validate the pinned schema and record required keys, enum/type checks, nested evidence properties, extra properties, and machine-readable errors. This yields “shape_pass” or “shape_fail.” Do not compare facts here: both “high” and “unknown” are valid enum members.
Pass two: facts, evidence, and unknowns
Run a separate evaluator against the oracle and original fixture. Use deterministic checks:
- Match case_id to the requested fixture; a valid object for the wrong case fails binding.
- Compare severity and decision exactly to the oracle. Do not award partial credit for a neighboring enum.
- Accept an owner string only when the exact name appears in allowed evidence. Accept JSON null only when the oracle says unsupported. Reject empty strings, guessed names, and literal “unknown” unless allowed.
- Check each evidence source against the fixture allowlist and each quote against a canonicalized exact substring. A real external URL is still unsupported here.
- Compare unknowns as a set. Missing, conflict, and absent owner are distinct; catch omitted unknowns and made-up resolutions.
- Enforce a case-specific evidence-coverage predicate. Reject
evidence: [], empty source or quote fields, and any output missing a decision-relevant span. INC-A must cover both the82 Creading and the80 C or abovethreshold; INC-B must cover the unavailable-reading fact and missing-reading rule; INC-C must cover both logger readings and the conflict-resolution route rule. Allowlist membership and substring matching are necessary filters, not proof of relevance.
The tri-state rule is the center of the protocol:
| Output state | Structural meaning | Semantic rule |
|---|---|---|
| Key omitted | Requiredness is violated | Fail the parser pass |
| Key present with null | An explicit null value | Pass only when the oracle says the value is unsupported or unknown |
| Key present with a string | A concrete assertion | Pass only when the fixture supplies that value and evidence |
JSON Schema guidance makes the boundary concrete: properties are optional unless required, and null is not absence. JSON Schema object reference The factual and evidence layer here is our proposed application design. The specification's format-annotation section discusses format semantics and application-level validation, not this source-grounded fact-checking protocol. JSON Schema validation, format annotations
Baseline, negative controls, and failure rule
Run positive controls through both passes first. The oracle expects shape and fact passes. The parser-negative should shape-fail even if its facts are right. The semantic-negative should shape-pass then fail facts or provenance. These are oracle expectations, not model outcomes.
Fail if any shape-valid output contains an unsupported value, wrong case, unapproved source or quote, concrete value for absence, unsupported conflict resolution, missing unknown, or a passing semantic-negative. Fail the parser layer separately for malformed JSON, missing keys, wrong types, extra properties, refusal, or incomplete output. Do not collapse layers.
Keep external grading secondary. The OpenAI graders guide describes exact, similarity, score-model, and Python graders and exposes model output JSON. Similarity can flag paraphrases, but exact fields, source membership, and null/unknown rules stay deterministic. The Evals guide frames tests around specified content criteria and notes Evals deprecation, so keep fixture, oracle, and verdict outside the service.
Readback and interpretation
Save raw response, parsed object, schema errors, factual ledger, evidence matches, fixture/schema/prompt hashes, model/settings, validator version, and verdict. Reopen raw response, repeat parsing, and compare with the ledger. A badge or copied answer is not persisted evidence.
Use one row per case:
case_id | shape_pass | refusal_or_incomplete | fact_pass | evidence_pass |
unknowns_pass | semantic_negative_caught | verdict | next_decision
Interpret by layer: “shape_pass + fact_pass” is usable for this contract; “shape_pass + fact_fail” is consumable but rejected or repaired; “shape_fail” is transport, schema, refusal, or generation trouble; “inconclusive” is not a pass.
The original contribution is a two-ledger evaluator: a shape ledger asks whether software can consume the object, while an evidence ledger asks whether a reviewer can defend each value. Explicit null/unknown rules make absence testable, and the semantic-negative proves a green parser is not a green fact check. This is not a benchmark. Add multilingual, OCR, multiple-owner, contradictory-date, empty-evidence, and domain-specific cases before release evidence. Refresh the contract and sources when provider schemas, refusal states, or evaluator APIs change.
Sources and limitations
- OpenAI, “Structured model outputs” — checked September 2, 2026. Supports schema adherence, supported schema types, strict-output configuration, and refusal/incomplete handling. Limitation: provider behavior is not a semantic truth oracle, and only a schema subset is supported.
- JSON Schema, “JSON Schema Validation” — checked September 2, 2026. Supports structural validation and the separate treatment of format annotations. Limitation: the source does not prescribe our factual/evidence evaluator; implementations may support dialects or vocabularies differently.
- JSON Schema, “Object” — checked September 2, 2026. Supports required-property behavior, additional properties, and the difference between absent and null values. Limitation: examples describe JSON Schema semantics, not the behavior of a particular model or application.
- OpenAI, “Graders” — checked September 2, 2026. Supports the documented grader families and access to output JSON for evaluation records. Limitation: a grader score remains an evaluation signal; deterministic fixture oracles are still required for exact factual and provenance rules.
- OpenAI, “Working with evals” — checked September 2, 2026. Supports test-input, reference, and content-criteria framing for evaluations, and records the current deprecation timeline. Limitation: the service timeline may change, so the protocol keeps its source pack and oracle provider-independent.