A model’s per-request compute cost is easy to put in a comparison table. The rest of the system is harder: preparing inputs, handling uncertain outputs, keeping data inside an approved boundary, reviewing edge cases, monitoring drift, and recovering from failures. Those steps can decide whether a small model is affordable for a real workflow.

MLCommons’ inference work offers a useful discipline for measuring performance under a defined model, system, and scenario. NIST’s AI Risk Management Framework adds a lifecycle view: document context and costs, measure the system, and decide how to manage its risks. Neither source gives a universal total-cost formula. The five cards below provide one local method without declaring a universal winner.

Card 1 — Price one accepted unit of work

Choose a concrete task and define “accepted” before running candidates. For example, a small model might classify an incoming document for a staff queue. An accepted unit could mean a correct category, a preserved source reference, and a reviewer’s approval within the team’s normal time limit. A generated answer alone is not an accepted unit if someone must reconstruct its evidence or correct it before use.

Compare at least three paths when relevant: the current manual or rules-based process, the existing model workflow, and the small-model candidate. Give all paths the same input fixture, operating constraints, and quality threshold. Freeze the test cases, expected answers, hardware, software versions, concurrency, latency target, and reviewer instructions. Include ordinary cases and known difficult cases such as ambiguous inputs, missing fields, and out-of-scope requests. Record every attempt, including timeouts, retries, tool failures, and abstentions.

The denominator matters. Count accepted work units, not prompts sent or tokens produced. If a model uses fewer compute resources but creates more rework, its cost per accepted result will show that. Keep the unit and denominator visible in the final sheet so another team does not mistake a per-request estimate for a per-decision cost.

Card 2 — Measure compute and the work before inference

For each path, record the hardware, model and runtime versions, batch size, workload, warm-up, concurrency, and request pattern. Measure cold-start and warm latency, p50 and p95 response time, throughput, peak memory, accelerator or CPU utilization, and energy where it can be measured reliably. Include retries and failed attempts in the workload. Report both the per-item result and the whole batch or shift result that the operator experiences.

MLPerf Inference is designed to measure system performance under defined benchmark scenarios; its process includes configuring the system, running required scenarios, and checking accuracy. Borrow that discipline: do not compare one model’s warm single request with another system’s cold batch, or a high-throughput offline run with a latency-sensitive queue. MLPerf results can help describe a measured system on the benchmark’s terms, but they are not a complete price quote for a different organization’s workflow.

Count the work that happens before and after inference: file conversion, text extraction, image resizing, filtering, retrieval, feature construction, data transfer, storage, and output validation. Track setup and deployment effort too. If a pipeline needs a specialized processor, a second service, or a fragile adapter just to accept the small model’s output, record its labor and resource use instead of assigning it a zero cost.

Card 3 — Put quality, failures, and human review in the same sheet

Set the quality threshold first, then score each path against the same reference cases. Use task-appropriate measures, such as per-class precision and recall for routing, and list the failure types that matter to the receiving team. Keep subgroup or input-condition results when they may change the decision. Averages can conceal that a model handles common clean records well but fails on the few cases that require the most attention.

For each attempt, log whether it passed automatically, was reviewed, was corrected, was escalated, or was rejected. Track reviewer minutes, correction type, escalation delay, and the fraction of cases sent to a person. Include the cost of investigating an unsupported answer, repeating a failed operation, or restoring the previous workflow. A model that abstains safely may be useful; a model that fails quietly may create a more expensive review burden even if both have the same headline accuracy.

NIST’s AI RMF explicitly calls for examining potential costs, including non-monetary costs resulting from errors, and documenting human-oversight processes. A team can translate that into local measures: time spent checking, impact of an error, who can override, and what conditions force a handoff. This is a proposed accounting method, not a claim that the model has been tested or that any single metric captures all effects.

Card 4 — Price the privacy and residency controls

Draw the full data path from source to deletion: collection, preprocessing, model input, output, logs, monitoring, backup, and any human review tool. For each step, note the location, operator, retention period, access roles, and whether content leaves the intended boundary. A model that runs on a local machine can still send data elsewhere through a hosted preprocessing service, telemetry, crash reports, or an external review queue.

Assign the actual work required to maintain the approved path: security review, permission setup, network restrictions, retention configuration, access audits, incident readiness, and recurring vendor or dependency checks. Include the cost of using synthetic or de-identified data to test integrations where production data cannot be used. Keep license and third-party component review in the inventory; a model file and its surrounding software may have different terms.

NIST’s framework treats governance as a lifecycle activity and includes privacy, third-party components, monitoring, and human oversight among the areas organizations can map and measure. It does not certify a system’s location or provide legal advice. The reviewer should validate the actual architecture and applicable requirements with the people responsible for privacy and security rather than infer residency from a model label.

Card 5 — Add operations, then calculate the result

Include the monthly work of deploying and maintaining the system: version pinning, updates, compatibility checks, monitoring, quality sampling, alert response, staff training, documentation, and rollback drills. Track how often a change triggers re-evaluation and how long the system takes to recover from a failed update. Include the baseline process that remains in place when the model is unavailable; a safety fallback has an operating cost too.

Use a transparent local calculation:

Cost per accepted unit = (compute + preprocessing + storage and transfer + review labor + escalation and rework + maintenance and monitoring + incident recovery) ÷ accepted units that meet the quality threshold.

Keep setup or migration costs separate, then show an amortized view over a stated volume and period. Show the assumptions for labor rates, hardware life, utilization, and failed attempts. If a path misses the quality or privacy gate, mark its cost as not comparable rather than celebrating a low number based on unusable output.

Assign one owner to review the sheet and one owner to stop or roll back the pilot. A decision record should state the chosen workload, accepted-unit definition, thresholds, result slices, uncertainty, and review date. NIST describes its AI RMF as a voluntary framework, not a fixed checklist; use the relevant parts for the organization’s risk tolerance and revisit the accounting when workload, model, data path, or staffing changes. The result belongs to that context. It does not crown a model for every team.

Sources and limitations

  • MLCommons, MLPerf Inference Submission Guide — checked October 21, 2026; describes system types, deployment scenarios, load generation, latency tracking, accuracy validation, and result preparation. Limitation: MLPerf measures performance for defined benchmark workloads and submissions; it is not a full organizational cost model or a direct guarantee for another workload.
  • MLCommons, Power Working Group — checked October 21, 2026; describes methods for measuring and reporting energy and power alongside benchmark performance. Limitation: published measurement approaches apply to defined systems and benchmark runs; they do not capture preprocessing, human time, privacy controls, or every deployment’s utility costs.
  • NIST, AI Risk Management Framework Core — checked October 21, 2026; the Core covers governance, context, costs, evaluation, privacy, human oversight, and lifecycle monitoring. Limitation: NIST identifies the framework as voluntary and says AI RMF 1.0 is being updated; it is not a prescribed cost formula, certification, or legal conclusion.