Guide · air-gapped deployment

Air-gapped LLM deployment

Run a model with no internet path, no telemetry call-home and no silent weight updates. The hard parts are transfer, licensing, updates and evaluation — not the inference server.

Short answer. An air-gapped LLM deployment has no route to the public internet. Models, packages and updates arrive on controlled media, are verified by hash and signature, and are promoted through an offline evaluation gate. Size GPUs for weights plus KV cache plus headroom, and write the update and support model into the contract. Plugsky documents an air-gapped bundle with a monthly model refresh by courier; verify the exact scope and terms for your deployment.

Reference architecture

ZoneComponentsControl
TransferDownload station, scanner, signing tools, removable mediaNo dual-homed host; two-person custody
Artifact storeInternal registry, model weights, container images, SBOMSigned bundles; immutable tags
InferenceGPU nodes, vLLM or equivalent, model routerNo egress; mTLS; per-team quotas
IdentityInternal IdP, LDAP/AD, OIDC, service accountsShort-lived credentials; no local admin
DataPrompt/response store, vector index, object storageEncryption at rest; retention policy
ObservabilityLogs, metrics, traces, SIEMLocal only; signed time source
EvaluationGolden set, rubric, human review stationVersioned datasets; offline judge

Keep the transfer zone physically separate from the inference zone. A laptop that connects to the internet and then to the internal network defeats the air gap. The NIST control family is a useful checklist: boundary protection, transmission integrity, audit events, configuration management and least privilege.

Model and weight transfer

Weights are large, opaque binary artifacts. Treat them like a software supply chain, not a file copy.

  1. Acquire outside. Download from the vendor or model publisher on a controlled station.
  2. Verify. Check the published SHA-256 and the publisher signature. Record the model card and licence.
  3. Scan. Run malware and archive scanning; reject unexpected files or scripts.
  4. Bundle. Create a signed tarball with weights, tokenizer, config, licence and SBOM.
  5. Transfer. Write to encrypted removable media; log serial, hash and handler.
  6. Import. Verify the hash and signature inside; copy to the internal registry; never run from the media.
  7. Promote. Deploy to a staging namespace, run the offline eval, then promote the same digest.
# Outside: sign and hash the bundle
sha256sum model-bundle.tar.zst > model-bundle.sha256
cosign sign-blob --key cosign.key model-bundle.tar.zst > model-bundle.sig

# Inside: verify before import
sha256sum -c model-bundle.sha256
cosign verify-blob --key cosign.pub \
  --signature model-bundle.sig model-bundle.tar.zst

Record the model digest in every request log. If the digest changes, the evaluation result is stale. A signed bundle with no promotion gate is just a slower download.

Licensing and update cadence

Licence familyTypical obligationCheck before deployment
Apache-2.0 / MITAttribution, licence text, patent termsCommercial use and redistribution
Llama community licenceAcceptable-use policy, attribution, namingUser thresholds and redistribution terms
Custom / research-onlyScope, field of use, no redistributionWhether production and government use are allowed
Vendor bundleSupport, updates, indemnityWhat is licensed: weights, runtime or both

Counsel should review the model, runtime and vendor agreements together. Export-control and sanctions rules may also apply.

CadenceContentDowntimeRollback
MonthlyModel refresh, tokenizer, eval setBlue/green namespacePrevious signed bundle
QuarterlyOS, runtime and security patchesRolling node drainSnapshot and image tag
EmergencyCritical CVE or model safety issueOut-of-band, approvedDocumented exception

A monthly refresh is a baseline, not a rule. Match it to your certification and support terms, and rehearse it.

GPU sizing

Start with weights: parameters × bytes per parameter. FP16 is 2 bytes, INT8 is 1, INT4 is 0.5. Then add KV cache, activations and 10–20% overhead. The KV cache grows with context length and concurrent sequences, so a model that fits at 4K context may not fit at 128K.

weights_gb = params_b * bytes_per_param
kv_gb = 2 * layers * kv_heads * head_dim * seq_len * batch * bytes_per_kv / 1e9
total_gb = (weights_gb + kv_gb) * 1.2
ModelFP16 weightsMinimum practical GPUsNote
7B~14 GB1 × 24–48 GBGood for extraction and routing
13B~26 GB1 × 48–80 GBAdd KV cache headroom
70B~140 GB2 × 80 GB or 1 × 192 GBTensor parallel; NVLink preferred
120B~240 GB3 × 80 GB or 2 × 141 GBCheck interconnect bandwidth
405B~810 GB11 × 80 GB or 5 × 192 GBCapacity and power dominate

Weight estimates only. Quantisation reduces memory but can change quality; evaluate it. H100, H200 and MI300X are common; confirm power. GPU rental can benchmark before buying.

Offline evaluation and audit

You cannot call an external judge or a hosted eval service inside the air gap. Build a local harness with a golden set, deterministic checks and human review. Store the dataset hash, prompt hash, model digest and reviewer decision with every run.

CheckOffline methodEvidence
Task qualityGolden set with expected fieldsScore per case, reviewer sign-off
SafetyRed-team prompts and refusal rubricCategory, decision, reviewer
RegressionDiff against the previous model digestChanged cases and severity
IntegrityHash the weights, runtime and eval setSigned manifest
AuditAppend-only local log, export to SIEMWho changed what, when

Use a deterministic rubric where possible, and reserve human review for ambiguity. An offline eval that only checks fluency will pass a model that invents invoice totals. Include extraction, arithmetic, refusal and retrieval cases from your own workload.

Support and cost

Support is part of the architecture. Ask how the vendor handles a failed GPU, a corrupted bundle and a security patch without remote access. Get the answer in the contract.

Cost driverWhat to estimateWhy it surprises teams
GPUsNodes, spares, interconnectLead times and minimum order quantities
Power and coolingRack density, kW, PUEA 70B cluster is a facilities project
StorageWeights, snapshots, eval dataMultiple model versions add up
StaffPlatform, security, on-callAir gap increases manual work
Update logisticsMedia, courier, custody, testingMonthly refresh is a process, not a script
SupportOn-site, spares, escalationRemote support may be prohibited

Compare the total cost with a private VPC before committing; if the requirement is residency, a VPC with no egress may be enough. For the decision, read what air-gapped AI means and air-gapped LLMs for government. Contract and deployment scope should be verified with the vendor and counsel.

FAQ

What makes a deployment air-gapped?

There is no route to the public internet in either direction. Models, packages, updates and telemetry cross the boundary on controlled media or through a one-way transfer process. A private VPC with egress is not air-gapped.

How are model weights transferred into the air gap?

Download and verify the weights outside, sign and hash the bundle, scan it, write it to encrypted removable media, and verify the hash again inside. Never connect the transfer station to both networks at once.

How much GPU memory does an LLM need?

Weights need roughly parameters times bytes per parameter: 14 GB for a 7B model at FP16, 140 GB for 70B, and about 810 GB for 405B. Add KV cache, activations and 10–20% overhead, then size for peak concurrency and context length.

How often should an air-gapped model be updated?

A practical baseline is a monthly model refresh, quarterly security patches and an emergency path for critical CVEs. The right cadence depends on the threat model, the certification and the vendor's support terms; verify the contractual cadence.

Sources

NIST SP 800-53 Rev. 5 security controls, checked 20 November 2026: csrc.nist.gov. vLLM documentation, checked 20 November 2026: docs.vllm.ai. NVIDIA H100 data sheet, checked 20 November 2026: nvidia.com. Sigstore Cosign verification, checked 20 November 2026: docs.sigstore.dev.

Reviewed 20 November 2026. Air-gapped availability, refresh cadence, support and licensing are contract-specific. Verify the current scope with Plugsky and counsel before committing to a deployment. GPU sizing figures are planning estimates, not benchmarks.