Key facts
| Developer fit | Same SDK and requests locally; only the base URL changes |
| Deployment | Same OpenAI-compatible API in Plugsky cloud, your VPC, on-prem or air-gapped |
| Data path | Prompts, embeddings and logs stay inside your network |
| Live endpoints | Chat, streaming, function calling, JSON mode and embeddings |
| Capacity | Size GPUs on peak concurrency and context length; keep upgrade headroom |
| Operations | You own upgrades, monitoring and on-call in on-prem mode |
| Pricing | Flat monthly self-serve plans; enterprise deployment scoped on the pricing page |
| Free plan | plugsky-micro and plugsky-lite, no card; 14-day full-access trial |
TL;DR
- Your code does not change: point the SDK at the internal endpoint and keep streaming and tools working.
- Run the same test suite against cloud and on-prem to catch configuration drift.
- Size on peak concurrency and context length, not average token volume.
- Keep a cloud path for experiments and burst so you are not blocked by local capacity.
- Budget for the work people forget: upgrades, monitoring and on-call.
How it works, step by step
- Prototype against the cloud endpoint first and freeze your prompts, schemas and evaluations before moving anything on-prem.
- Stand up the on-prem deployment in a test network zone and point the same SDK at the internal base URL.
- Run your evaluation suite and load tests to confirm output parity and throughput under realistic concurrency.
- Size GPU capacity for peak concurrency and worst-case context length, and reserve headroom for model upgrades.
- Wire key management into your existing secrets store and keep environment separation intact.
- Add health checks, token and latency metrics, and alerting around queue depth and error rates.
- Keep a cloud or VPC fallback configured so experiments and traffic spikes are not constrained by local capacity.
Try it yourself
Open the LLM GPU capacity calculator →
What on-prem changes for your code
Almost nothing, if you designed around the OpenAI-compatible API. The base URL moves to an internal address, the model name stays configuration, and streaming, JSON mode and function calling behave the same way. Your retry logic, prompt templates and evaluation harness keep working, which is the main reason to avoid vendor-specific clients.
What does change is the environment around the code: secrets come from your own store, logs stay in your own observability stack, and the endpoint you call is one you now operate. Treat it like any other internal service with an owner, SLOs and a runbook.
Sizing and testing before you commit
GPU sizing is where optimistic plans fail. Model memory scales with parameters and precision, and throughput depends on concurrency, context length and batching. Size for your worst realistic prompt mix, not the demo: long documents and tool loops consume far more than short chats.
- Load profile: requests per minute, p95 context length, concurrent streams.
- Headroom: leave capacity for one generation of model upgrade.
- Parity tests: run the same eval set against cloud and on-prem builds.
- Fallback: keep a cloud route configured for spikes and new experiments.
The operational reality
Self-hosting trades token bills for infrastructure and time. Upgrades are yours to schedule, monitoring is yours to build, and incidents are yours to answer. If your team is two engineers shipping product, that trade is usually bad at low volume; it becomes sensible when data governance or latency requires it, or when steady-state utilisation is high enough to beat flat monthly plans.
Use the cloud plan for development and small services — the free tier includes plugsky-micro and plugsky-lite with no card — and keep on-prem for the workloads that must live inside your network. Compare the numbers on the live pricing page, and remember endpoint status is the same in every deployment: chat, streaming, JSON mode, function calling and embeddings are live, while audio, images and files are coming soon.
Honest comparison
| Concern | Plugsky on-prem | Hosted API only | Roll your own serving |
|---|---|---|---|
| Code changes | Base URL and model name only | None | Custom server and client code |
| Data path | Inside your network | Leaves your network | Inside, with your engineering |
| Dev experience | Same SDK, streaming and tools | Same SDK | Whatever you build |
| Ops load | Platform deploy plus your monitoring | Minimal | Full MLOps ownership |
| Cost shape | License plus infrastructure | Per-token | GPUs plus headcount |
Frequently asked questions
Do I need to change my SDK?
No. The on-prem deployment exposes the same OpenAI-compatible API, so you change the base URL and model name and keep the rest of your code.
Can I test on-prem without buying hardware?
Use the cloud or VPC deployment to build and validate first — the free plan includes two models with no card — then move the workload on-prem when required.
How do I size GPUs?
Start from peak concurrency and worst-case context length, not average volume, and leave headroom for the next model generation. Use the capacity calculator for a first estimate.
Who handles upgrades?
You do. On-prem deployments are operated by your team, including patching, model bundle updates under the licensing agreement, and monitoring.
Is streaming supported on-prem?
Yes. Streaming, function calling, JSON mode and embeddings behave the same as in the cloud deployment.
What about air-gapped environments?
The same API runs in air-gapped mode with model bundles licensed for the deployment, so code written against the cloud endpoint still works.
When does on-prem make financial sense?
When steady-state utilisation is high, when residency or latency rules require it, or when a contract mandates it. Below that, flat monthly plans are usually cheaper once you count engineering time.