Short answer. An air-gapped LLM deployment has no route to the public internet. Models, packages and updates arrive on controlled media, are verified by hash and signature, and are promoted through an offline evaluation gate. Size GPUs for weights plus KV cache plus headroom, and write the update and support model into the contract. Plugsky documents an air-gapped bundle with a monthly model refresh by courier; verify the exact scope and terms for your deployment.
Reference architecture
| Zone | Components | Control |
|---|---|---|
| Transfer | Download station, scanner, signing tools, removable media | No dual-homed host; two-person custody |
| Artifact store | Internal registry, model weights, container images, SBOM | Signed bundles; immutable tags |
| Inference | GPU nodes, vLLM or equivalent, model router | No egress; mTLS; per-team quotas |
| Identity | Internal IdP, LDAP/AD, OIDC, service accounts | Short-lived credentials; no local admin |
| Data | Prompt/response store, vector index, object storage | Encryption at rest; retention policy |
| Observability | Logs, metrics, traces, SIEM | Local only; signed time source |
| Evaluation | Golden set, rubric, human review station | Versioned datasets; offline judge |
Keep the transfer zone physically separate from the inference zone. A laptop that connects to the internet and then to the internal network defeats the air gap. The NIST control family is a useful checklist: boundary protection, transmission integrity, audit events, configuration management and least privilege.
Model and weight transfer
Weights are large, opaque binary artifacts. Treat them like a software supply chain, not a file copy.
- Acquire outside. Download from the vendor or model publisher on a controlled station.
- Verify. Check the published SHA-256 and the publisher signature. Record the model card and licence.
- Scan. Run malware and archive scanning; reject unexpected files or scripts.
- Bundle. Create a signed tarball with weights, tokenizer, config, licence and SBOM.
- Transfer. Write to encrypted removable media; log serial, hash and handler.
- Import. Verify the hash and signature inside; copy to the internal registry; never run from the media.
- Promote. Deploy to a staging namespace, run the offline eval, then promote the same digest.
# Outside: sign and hash the bundle
sha256sum model-bundle.tar.zst > model-bundle.sha256
cosign sign-blob --key cosign.key model-bundle.tar.zst > model-bundle.sig
# Inside: verify before import
sha256sum -c model-bundle.sha256
cosign verify-blob --key cosign.pub \
--signature model-bundle.sig model-bundle.tar.zst
Record the model digest in every request log. If the digest changes, the evaluation result is stale. A signed bundle with no promotion gate is just a slower download.
Licensing and update cadence
| Licence family | Typical obligation | Check before deployment |
|---|---|---|
| Apache-2.0 / MIT | Attribution, licence text, patent terms | Commercial use and redistribution |
| Llama community licence | Acceptable-use policy, attribution, naming | User thresholds and redistribution terms |
| Custom / research-only | Scope, field of use, no redistribution | Whether production and government use are allowed |
| Vendor bundle | Support, updates, indemnity | What is licensed: weights, runtime or both |
Counsel should review the model, runtime and vendor agreements together. Export-control and sanctions rules may also apply.
| Cadence | Content | Downtime | Rollback |
|---|---|---|---|
| Monthly | Model refresh, tokenizer, eval set | Blue/green namespace | Previous signed bundle |
| Quarterly | OS, runtime and security patches | Rolling node drain | Snapshot and image tag |
| Emergency | Critical CVE or model safety issue | Out-of-band, approved | Documented exception |
A monthly refresh is a baseline, not a rule. Match it to your certification and support terms, and rehearse it.
GPU sizing
Start with weights: parameters × bytes per parameter. FP16 is 2 bytes, INT8 is 1, INT4 is 0.5. Then add KV cache, activations and 10–20% overhead. The KV cache grows with context length and concurrent sequences, so a model that fits at 4K context may not fit at 128K.
weights_gb = params_b * bytes_per_param
kv_gb = 2 * layers * kv_heads * head_dim * seq_len * batch * bytes_per_kv / 1e9
total_gb = (weights_gb + kv_gb) * 1.2
| Model | FP16 weights | Minimum practical GPUs | Note |
|---|---|---|---|
| 7B | ~14 GB | 1 × 24–48 GB | Good for extraction and routing |
| 13B | ~26 GB | 1 × 48–80 GB | Add KV cache headroom |
| 70B | ~140 GB | 2 × 80 GB or 1 × 192 GB | Tensor parallel; NVLink preferred |
| 120B | ~240 GB | 3 × 80 GB or 2 × 141 GB | Check interconnect bandwidth |
| 405B | ~810 GB | 11 × 80 GB or 5 × 192 GB | Capacity and power dominate |
Weight estimates only. Quantisation reduces memory but can change quality; evaluate it. H100, H200 and MI300X are common; confirm power. GPU rental can benchmark before buying.
Offline evaluation and audit
You cannot call an external judge or a hosted eval service inside the air gap. Build a local harness with a golden set, deterministic checks and human review. Store the dataset hash, prompt hash, model digest and reviewer decision with every run.
| Check | Offline method | Evidence |
|---|---|---|
| Task quality | Golden set with expected fields | Score per case, reviewer sign-off |
| Safety | Red-team prompts and refusal rubric | Category, decision, reviewer |
| Regression | Diff against the previous model digest | Changed cases and severity |
| Integrity | Hash the weights, runtime and eval set | Signed manifest |
| Audit | Append-only local log, export to SIEM | Who changed what, when |
Use a deterministic rubric where possible, and reserve human review for ambiguity. An offline eval that only checks fluency will pass a model that invents invoice totals. Include extraction, arithmetic, refusal and retrieval cases from your own workload.
Support and cost
Support is part of the architecture. Ask how the vendor handles a failed GPU, a corrupted bundle and a security patch without remote access. Get the answer in the contract.
| Cost driver | What to estimate | Why it surprises teams |
|---|---|---|
| GPUs | Nodes, spares, interconnect | Lead times and minimum order quantities |
| Power and cooling | Rack density, kW, PUE | A 70B cluster is a facilities project |
| Storage | Weights, snapshots, eval data | Multiple model versions add up |
| Staff | Platform, security, on-call | Air gap increases manual work |
| Update logistics | Media, courier, custody, testing | Monthly refresh is a process, not a script |
| Support | On-site, spares, escalation | Remote support may be prohibited |
Compare the total cost with a private VPC before committing; if the requirement is residency, a VPC with no egress may be enough. For the decision, read what air-gapped AI means and air-gapped LLMs for government. Contract and deployment scope should be verified with the vendor and counsel.
FAQ
What makes a deployment air-gapped?
There is no route to the public internet in either direction. Models, packages, updates and telemetry cross the boundary on controlled media or through a one-way transfer process. A private VPC with egress is not air-gapped.
How are model weights transferred into the air gap?
Download and verify the weights outside, sign and hash the bundle, scan it, write it to encrypted removable media, and verify the hash again inside. Never connect the transfer station to both networks at once.
How much GPU memory does an LLM need?
Weights need roughly parameters times bytes per parameter: 14 GB for a 7B model at FP16, 140 GB for 70B, and about 810 GB for 405B. Add KV cache, activations and 10–20% overhead, then size for peak concurrency and context length.
How often should an air-gapped model be updated?
A practical baseline is a monthly model refresh, quarterly security patches and an emergency path for critical CVEs. The right cadence depends on the threat model, the certification and the vendor's support terms; verify the contractual cadence.
Sources
NIST SP 800-53 Rev. 5 security controls, checked 20 November 2026: csrc.nist.gov. vLLM documentation, checked 20 November 2026: docs.vllm.ai. NVIDIA H100 data sheet, checked 20 November 2026: nvidia.com. Sigstore Cosign verification, checked 20 November 2026: docs.sigstore.dev.
Reviewed 20 November 2026. Air-gapped availability, refresh cadence, support and licensing are contract-specific. Verify the current scope with Plugsky and counsel before committing to a deployment. GPU sizing figures are planning estimates, not benchmarks.