Open Weights Are Not a Local Deployment Plan

An open-weight model can be downloadable and still be a poor local choice. A weight file tells you what was released. It does not tell you whether the selected quantization fits with the context and concurrency you need, whether the runtime matches the artifact, where prompts and logs travel, or whether your product can satisfy the model's terms.

The reader question is simple: can a team name the exact artifact, prove that it fits, trace every data path, and accept the terms before a real prompt touches it? The current Meta Llama 3.2 model card lists 1B and 3B text-only variants, calls the release a static model, and documents several inference routes. That breadth is an invitation to test the whole stack, not a deployment certificate.

Card 1 — Identify the exact thing you are deploying

Start with an artifact record, not a model nickname. Record the publisher, repository, revision or digest, parameter size, precision, file format, tokenizer, chat template, supported languages, license, access state, and intended use. The example in the Meta card is not merely “a three-billion-parameter model”: its page identifies a 3B BF16 checkpoint, a custom Llama 3.2 license, supported languages, a knowledge cutoff, and separate quantized variants. Those details change what can be run and what can be promised.

The first evaluation asset is a model-inventory.yaml file:

model_id: exact-repository/name
revision: commit-or-content-digest
artifact: filename-and-sha256
format: safetensors | gguf | other
tokenizer_and_template: ids-and-digests
license_and_policy: links-and-acceptance-owner
intended_use: narrow-task-description
access: public | gated | credentialed

Treat a third-party quantization as a new artifact. Tie it to the source checkpoint and keep its own hash and license record. Stop when the revision is missing, the artifact cannot be traced, or nobody owns the license decision. “It is the same model” is not enough evidence.

Card 2 — Turn memory into a budget

The file size is only one line in the budget. Use this planning equation:

required memory = weights + runtime overhead + KV cache(context, output, concurrency) + headroom

This is a planning model, not a vendor measurement. Hugging Face's quantization documentation explains that 8-bit and 4-bit representations reduce memory and computational cost, while the llama.cpp server documentation exposes context size, batch size, parallel slots, and GPU-fitting controls. The savings therefore do not erase the memory used by context, concurrent sequences, kernels, the operating system, or a safety margin.

Create memory-budget.csv with one row for every artifact and operating point. Include the artifact hash, precision, file bytes, context tokens, maximum output tokens, concurrency, batch setting, peak RAM, peak VRAM, load time, steady-state latency, and measured headroom. Test cold start and warm generation at the longest normal prompt, then add a deliberately larger prompt to see how failure presents. Do not fill the result cells from a marketing table. They remain untested until someone runs this exact build on the intended machine.

Consider a hypothetical 3B quantized file that loads comfortably while one person is testing it. Four simultaneous requests with long context may push the KV cache into system memory, trigger swapping, or fail the request. The file did not lie; the operating point was underspecified. Choose a headroom rule in advance, record it as a team policy, and stop if the measured configuration misses it. A 20% margin can be an illustrative policy, not a universal guarantee.

Card 3 — Lock the runtime and backend

A model card can show several valid routes without saying which route your team should operate. Transformers may load the original checkpoint with PyTorch and device_map; a quantized Transformers path has its own library and hardware conditions. llama.cpp serves F16 and quantized models on CPU and GPU, expects GGUF files for its server, and supports CPU-plus-GPU placement. The same server exposes context, batch, parallelism, template, grammar, and GPU-layer settings. These are runtime choices, not properties you get for free from the weights.

The second evaluation asset is a runtime-matrix.md containing one row per exact combination:

Include the process owner, endpoint binding, authentication, startup behavior, and rollback version. A successful load is not a successful integration: a wrong chat template can produce malformed turns, and a context setting can silently exclude the input your task depends on. Stop if the exact format, tokenizer, backend, or server version is untested. No benchmark is reported here because no permitted run has been performed.

Card 4 — Draw the data path, including the escape hatch

“Local” describes where inference is intended to happen. It does not by itself describe downloads, updates, caches, logs, crash reports, network bindings, or a fallback provider. Ollama's current FAQ says that it does not see prompts or data when its local mode is used, while cloud-hosted models process prompts and responses to provide the service. The same FAQ documents a local-only setting that disables cloud features, and says the server binds to 127.0.0.1 by default but can be exposed through OLLAMA_HOST. Those are company statements about that product, not independent traffic verification.

Draw the route before approving a prompt:

prompt -> application -> local runtime -> model cache -> local logs/backups
                         |
                         +-> reviewed cloud fallback? -> provider boundary
model download/update -> registry -> cache and package records

The third evaluation asset is a one-page data-path record. Name every endpoint, DNS or proxy hop, model cache, log location, backup, retention rule, administrator, and fallback condition. Mark each item verified, assumed, or unknown. A realistic edge case is a local server being unavailable and an application silently using its cloud default. The safe decision is to disable that fallback until its data, contract, and user notice have been reviewed. Stop when the path cannot be independently read back from configuration and network controls.

Card 5 — Make the license and operating owner visible

An open-weight label is not a universal license. The Llama 3.2 Community License grants a limited license and sets conditions for redistribution, attribution, use policy compliance, and certain very large commercial services. It also states that the materials and outputs are provided without warranties unless applicable law requires otherwise. This is a contract to review, not a conclusion that every use is approved. The runtime, tokenizer, adapter, quantization recipe, and training or retrieval data can carry separate terms.

The final asset is release-record.md: exact hashes; copied license and policy links; approved use; model, runtime, tokenizer, and adapter versions; data boundary; memory budget; owner; rollback artifact; update trigger; and next review date. Keep a short decision line:

The useful output is not “Model X runs locally.” It is a five-card evidence packet with a hash, a budget, a runtime, a data boundary, and an owner. That packet makes the next change reviewable. If a new quantization, server build, driver, or cloud fallback appears, open a new record rather than silently inheriting the old approval.

Sources and limitations

All sources below were checked September 2, 2026.

• Llama 3.2 3B Instruct model card, Meta via Hugging Face — supports the model-family, format, license, intended-use, supported-language, static-release, and inference-route claims. Limitation: it is the publisher's model documentation and cannot establish your hardware result or deployment safety.
• Llama 3.2 Community License, Meta — supports the license scope, redistribution and attribution conditions, commercial threshold, use-policy incorporation, and warranty language. Limitation: this is not legal advice and does not review a particular product or jurisdiction.
• Quantization documentation, Hugging Face Transformers — supports the distinction between lower-precision representations and memory or compute savings. Limitation: the documentation does not predict the complete memory footprint of a chosen runtime.
• llama.cpp server documentation — supports GGUF model sources, CPU/GPU inference, context, batch, parallel, fitting, endpoint, and timing controls. Limitation: flags and behavior can change; no local run was performed for this review.
• FAQ, Ollama — supports the attributed local-versus-cloud data statement, local-only setting, default binding, and multi-GPU loading description. Limitation: it is a vendor statement, not an independent audit of traffic or retention.