Key facts
| Deployment form | Helm chart or air-gapped installer for your data centre |
| Accelerators | H100, H200, MI300X pools or CPU-only configurations |
| API contract | OpenAI-compatible /v1 endpoints, same as cloud |
| Model set | Curated catalogue; updates delivered through controlled releases |
| Air-gap option | Bundle ships on physical media with periodic model refresh |
| Security | Network isolation, BYOK/HSM, scoped keys, audit logging |
| Support | Enterprise support terms documented at /legal/sla |
| Compliance posture | SOC 2 Type II and ISO 27001 readiness in progress (not yet certified) |
TL;DR
- Size for peak concurrency and context, not for average tokens per day.
- Separate the serving gateway from model instances so routing and upgrades stay controlled.
- Plan the update path before purchase — especially for air-gapped sites.
- On-prem removes external dependencies and adds operational load; staff for it.
- Keep the OpenAI-compatible contract so workloads stay portable across tiers.
How it works, step by step
- Define workloads, concurrency targets and context lengths for the first two years.
- Choose accelerator pools — GPU classes or CPU-only — and size for peak, not average.
- Design the serving layer: gateway, model routing, queues, timeouts and health checks.
- Plan networking and storage: isolated VLANs, artifact repositories and vector store placement.
- Define key custody and audit before go-live, including HSM options for regulated data.
- Build the update and rollback process, including model refresh for air-gapped sites.
- Run a pilot on production-like hardware and measure latency at target concurrency.
Original data
Try it yourself
Open the private LLM cost estimator →
Capacity: what actually drives hardware choice
Three variables determine the bill: concurrency, context length and quality tier. Long contexts consume memory and time even when token counts look modest, and peak concurrency — not daily volume — sets the failure point. Start from a workload table: requests per minute at peak, p95 context length, required quality level and acceptable latency. Then map each workload to a model class. Teams routinely over-buy frontier-class GPUs for extraction workloads that a small model handles, and under-buy for the one workload that genuinely needs reasoning. CPU-only configurations remain viable for small models and low-concurrency internal tools, and are often the right first deployment for a pilot.
Serving architecture on your own network
- Gateway: a single OpenAI-compatible endpoint that handles authentication, routing, quotas and logging.
- Model instances: scaled separately from the gateway so a model swap does not need a gateway release.
- Queues and timeouts: absorb bursts and return clear 429/503 semantics instead of unbounded waits.
- Health and failover: detect a dead model instance and route to a healthy one, with capacity reserved for failover.
- State: vector stores and prompt logs placed deliberately — they carry residency and deletion obligations.
Plugsky's on-prem option ships this stack as a Helm chart with the same API contract as the cloud service, so client code and evals transfer unchanged.
Operations: the part that decides success
On-prem moves operational responsibility to you. Plan for model and platform updates with rollback, capacity headroom for failover, monitoring that covers GPU utilisation as well as request latency, and a patch cadence for the underlying Kubernetes and OS layers. Air-gapped sites need a physical update process, so define who carries media, how integrity is verified, and how often models refresh. Budget for the runbook work — incident response, key rotation, backup restore tests — because those tasks are the difference between a pilot and a service.
Cost, control and the honest trade-off
On-prem makes sense when data cannot leave the building, when network policy forbids external inference, or when long-run utilisation is high enough to beat managed pricing. It is rarely the cheapest option at low utilisation and always the slowest to start. A pragmatic sequence is cloud or region-pinned deployment for the first workloads, VPC when the account boundary matters, and on-prem when the physical boundary does. Because all tiers speak the same OpenAI-compatible API, workloads can move without rewrites. Confirm support and service terms at /legal/sla and the commercial frame at /legal/terms.
Honest comparison
| Decision | On-prem | VPC deployment | Managed cloud |
|---|---|---|---|
| Data path | Fully inside your data centre | Inside your cloud account | Provider regions with pinning |
| Time to first workload | Weeks to months | Days to weeks | Hours |
| Capacity cost model | Capital plus operations | Cloud consumption plus operations | Flat platform plans |
| Hardware choice | H100/H200/MI300X or CPU-only pools | Your cloud GPU quota | Provider-managed |
| Air-gap support | Yes, physical media updates | No | No |
| Ops burden | High, and yours | Moderate | Low |
Frequently asked questions
What hardware does an on-prem LLM need?
Plugsky's on-prem option supports H100, H200 and MI300X GPU pools as well as CPU-only configurations. Sizing depends on peak concurrency, context length and quality tier, so pilot on production-like hardware before committing.
Is on-prem available as Kubernetes?
Yes. The documented deployment form is a Helm chart for your data centre, with an air-gap installer for environments without internet access.
How do air-gapped updates work?
The bundle ships on physical media and models refresh periodically by the same offline process. Define custody, verification and rollback procedures before go-live.
Does application code change on-prem?
No. The deployment exposes the same OpenAI-compatible /v1 endpoints, so SDKs, prompts and evals work across cloud, VPC and on-prem.
When is on-prem the wrong choice?
When utilisation is low, when time to market matters more than physical control, or when you cannot staff operations. Region-pinned cloud or VPC usually satisfies residency needs at lower cost.
How do we secure an on-prem deployment?
Isolate the network, use scoped keys and RBAC, hold encryption keys with BYOK or an HSM, log every request and admin action, and keep the OS and Kubernetes patch cadence under change control.
How long does an on-prem rollout take?
Expect weeks to months depending on hardware lead times, network change approval and security review. Start with a single workload on a pilot cluster rather than a full-platform launch.
What support exists for on-prem?
On-prem and air-gapped deployments are enterprise-scoped, with support and service terms defined at /legal/sla. Confirm the operating model and escalation path during contracting.