Key facts
| Bank fit | In-perimeter inference with existing network segmentation and DR sites |
| Deployment | Same OpenAI-compatible API in Plugsky cloud, your VPC, on-prem or air-gapped |
| Data path | Prompts, embeddings and logs stay inside the bank's perimeter |
| Models | 30+ models; on-prem bundles confirmed under the licensing agreement |
| Live endpoints | Chat, streaming, function calling, JSON mode and embeddings |
| Operations | Bank owns capacity, patching windows, upgrades and monitoring |
| Audit | Authentication, key and admin events exportable to SIEM |
| Pricing | Flat monthly self-serve plans; enterprise deployment scoped on the pricing page |
TL;DR
- Run the model API inside the bank's network so no prompt or log leaves the perimeter.
- Keep the OpenAI-compatible surface, so applications and tests do not change.
- Use scoped keys per system plus SSO with SCIM for operator access.
- Plan capacity and dual-site DR before committing a regulated workload.
- Budget for ownership: upgrades, monitoring and GPU operations are the bank's.
How it works, step by step
- Choose a pilot workload that is valuable but not customer-facing — policy search, alert summarisation, correspondence drafting.
- Design the network zone: where the platform runs, which subnets reach it, and how it replicates to the DR site.
- Size capacity against peak concurrency and context length rather than average load, and reserve headroom for upgrades.
- Integrate identity: scoped keys per application and environment from the bank's secret store, SSO with SCIM for operator accounts, RBAC for policy changes.
- Wire log and audit export into the SIEM, including authentication, key lifecycle and inference metadata.
- Run the evaluation suite and a security review against the in-perimeter deployment, then pilot with a named owner.
- Agree the release and patch calendar with the vendor before scaling to further workloads.
Try it yourself
Open the private LLM deployment estimator →
Why banks put inference on-prem
Three forces push banks to self-host. Data governance: some workloads cannot send prompts or documents to shared infrastructure, whatever the contract says. Architecture: the model becomes another internal service behind the same segmentation, egress rules and change control as core systems. Predictability: in-perimeter serving removes a network dependency from the latency budget and keeps behaviour stable when the bank's network is under stress.
On-prem also aligns with how banks already run critical software: defined environments, dual sites, change windows and evidence for internal audit. The same OpenAI-compatible API means the application layer treats the on-prem deployment exactly like the cloud one.
Reference deployment
The shape is a gateway inside a controlled zone, reachable only from approved application subnets. Applications authenticate with scoped keys from the bank's secret manager and call the OpenAI-compatible endpoint with the SDK they already use. Retrieval uses the live embeddings API against a bank-hosted vector store, and function calling invokes internal APIs through the same gateway policy.
- Identity: scoped keys for services, SSO with SCIM and RBAC for operators.
- Evidence: authentication, key lifecycle, admin and inference events to the SIEM.
- Resilience: dual-site deployment with tested failover and restore procedures.
- Capacity: sized on peak concurrency, not averages, with upgrade headroom.
What changes operationally
On-prem is a commitment, not a checkbox. The bank owns GPU capacity, patching, monitoring and release management, and must plan how model bundles are updated under the licensing agreement. That work is real, and it should be compared honestly against the flat monthly self-serve plans and VPC options available on the live pricing page.
Two practical notes for architects. First, endpoint coverage is the same in every deployment: chat, streaming, JSON mode, function calling and embeddings are live, while audio, images, files, batch, moderation, fine-tuning and assistants are labelled coming soon. Second, on-prem does not remove the need for data minimisation — redact what the workload does not require, and set retention deliberately.
Honest comparison
| Concern | Plugsky on-prem | Public API only | DIY model serving |
|---|---|---|---|
| Data path | Prompts and logs stay in the bank | Prompts leave the network | Inside, but you build everything |
| Application change | None: same OpenAI-compatible API | None, but no control | Custom serving and client code |
| Identity and audit | Scoped keys, SSO, SIEM export | Vendor-side controls | Bank builds the whole stack |
| Effort | Deploy platform, keep your app | Fast, low control | Months of MLOps |
| Cost shape | License plus bank infrastructure and ops | Per-token | GPUs plus engineering headcount |
Frequently asked questions
Does on-prem mean the same API as the cloud version?
Yes. The deployment exposes the same OpenAI-compatible endpoint, so applications, SDKs, prompts and tests do not change when you move between cloud, VPC, on-prem or air-gapped.
Which capabilities run on-prem?
Chat completions with streaming, JSON mode and function calling, plus embeddings, are live. Audio, images, files, batch, moderation, fine-tuning and assistants are labelled coming soon.
How are models licensed for on-prem?
Model bundles are confirmed under the licensing agreement for the deployment; the catalogue lists 30+ models and enterprise scoping covers which ones are included.
Who operates the platform?
The bank does, on its own infrastructure: capacity, patching, upgrades and monitoring. The vendor provides the platform releases and support terms.
How do we satisfy internal audit?
Scoped keys per system, SSO and RBAC for operators, and SIEM export of authentication, key lifecycle, admin and inference events give one reconstruction path.
What about disaster recovery?
Design a dual-site deployment with tested failover before production, and treat capacity and upgrade headroom as part of the DR budget.
How do we compare on-prem with the cloud plan?
Model total cost — infrastructure, operations and licensing — against the flat monthly plans on the live pricing page, and use the deployment estimator for an early view.