Private AI for enterprise — on-premise LLM deployment guide

Reference architecture

A production-grade on-premise LLM deployment typically includes these components:

┌─────────────────────────────────────────────────────────┐
│                    Application Layer                      │
│  Internal tools │ Customer apps │ Chat interfaces         │
├─────────────────────────────────────────────────────────┤
│                    API Gateway + Auth                      │
│  API keys │ RBAC │ Rate limiting │ Request logging        │
│  Audit trail │ Budget controls │ IP whitelisting          │
├─────────────────────────────────────────────────────────┤
│                    Model Router                             │
│  Model selection │ Load balancing │ Fallback routing      │
│  A/B testing │ canary deployments                         │
├─────────────────┬───────────────────┬─────────────────────┤
│  vLLM cluster   │  SGLang cluster   │  TensorRT-LLM pool  │
│  (high throughput) │ (structured output) │ (max perf)      │
├─────────────────┴───────────────────┴─────────────────────┤
│                    Inference Storage                        │
│  Model weights (NVMe) │ KV cache (RAM/VRAM) │ Logs (SSD)  │
├─────────────────────────────────────────────────────────┤
│                    Observability                            │
│  Prometheus metrics │ Grafana dashboards │ Loki logs       │
│  Model health checks │ Cost attribution │ Token tracking   │
└─────────────────────────────────────────────────────────┘

Component details

vLLM / SGLang clusters

The inference layer typically uses vLLM for general-purpose high-throughput serving or SGLang for structured output and advanced scheduling. Both support tensor parallelism (splitting a model across multiple GPUs) and continuous batching. A production deployment runs at least 2 nodes for redundancy, with each node hosting 1-4 GPUs depending on model size.

API gateway

An API gateway (Kong, Envoy, or custom) sits in front of the inference cluster and handles authentication, rate limiting, request routing, and audit logging. Every request is logged with timestamp, user identity, model used, token count, and response time. This is critical for SOC 2 and internal compliance.

Model router

The model router selects which model serves each request based on the requested model ID, user tier, and fallback policies. For example, internal HR queries route to an 8B model while legal document analysis routes to a 70B model. The router also handles failover — if one node is overloaded, the request goes to another.

Storage

Model weights are stored on NVMe SSDs for fast loading (a 70B Q4 model ~40 GB should load in under 30 seconds). KV cache is allocated from GPU VRAM during inference. Request logs and usage metrics go to a separate SSD array with 30-90 day retention for compliance audits.

When to self-host vs use Plugsky Enterprise

Factor	Self-host on-prem	Plugsky Enterprise
Data sovereignty	Complete — data stays on premises	VPC / dedicated region / on-prem appliance
Upfront cost	$50K-$500K+ (hardware + setup)	$0 (subscription)
Monthly cost	$2K-$20K+ (power, cooling, ops staff)	$249-$2,500+ (all-inclusive)
Time to deploy	3-12 months	Days to weeks
Team required	ML engineers, SREs, IT ops	None (fully managed)
Model access	Open-weight models only	Open-weight + proprietary frontier models
Model updates	Manual (download, validate, deploy)	Automatic (new models added continuously)
Scalability	Limited by procured hardware	Elastic — scale up/down on demand
Uptime SLA	Depends on your ops	99.9%+ (multi-region available)
Compliance	Your responsibility	SOC 2, GDPR, ISO 27001, GCC ready

Self-host is the right choice when:

• Your compliance framework requires data to never leave your physical premises (air-gapped / classified environments)
• You already have the GPU hardware and operations team in place
• You need total control over model selection, versioning, and inference parameters
• Your workload is predictable and stable enough that elastic scaling isn't a major advantage

Plugsky Enterprise is the right choice when:

• You want to deploy AI in weeks, not months, without procuring GPUs or hiring ML ops staff
• You need access to both open-weight and proprietary models through a single API
• Your workload has variable demand that benefits from elastic scaling
• You want SOC 2 / GDPR / GCC compliance without building the compliance infrastructure yourself
• Your total cost of ownership favors subscription over capital expenditure

Hybrid approach: on-prem + Plugsky

Many enterprise customers use a hybrid architecture that combines on-premise inference for sensitive workloads with Plugsky's managed API for everything else:

┌─────────────────────────────────────────────────────┐
│                   Your Applications                   │
├─────────────────────────────────────────────────────┤
│                  Unified API Gateway                   │
│         Route by data sensitivity / model need       │
├─────────────────────┬───────────────────────────────┤
│   On-prem cluster   │      Plugsky Enterprise        │
│   (vLLM/SGLang)     │      (cloud/VPC/dedicated)     │
│                     │                                │
│   Sensitive data    │  General workloads             │
│   Classified docs   │  Customer support              │
│   PII/PHI data      │  Content generation            │
│   Sovereign data    │  Prototyping / dev             │
│   Air-gapped only   │  Burst capacity                │
└─────────────────────┴───────────────────────────────┘

The unified gateway enforces routing policies: requests tagged with sensitive data never leave the on-prem cluster. General requests route to Plugsky's managed API. A single audit log captures all requests regardless of which backend served them.

Example hybrid deployment timeline

1. Month 1: Start with Plugsky Enterprise (cloud or VPC). Deploy to production in weeks. Users access 30+ models immediately.
2. Months 2-4: Procure and deploy on-prem GPU cluster. Validate model performance parity.
3. Month 5: Implement hybrid gateway. Route sensitive workloads to on-prem. All other traffic stays on Plugsky.
4. Month 6+: Continuously optimize routing. On-prem handles steady-state sensitive load. Plugsky handles everything else, including burst capacity.

Compliance (SOC 2, GDPR, GCC)

Enterprise AI deployments must meet regulatory requirements. Here is how each scenario maps:

SOC 2 Type II

Required by most US enterprises. Covers security, availability, and confidentiality. Both self-hosted and Plugsky Enterprise can achieve SOC 2 compliance. Plugsky maintains SOC 2 Type II certification for the managed platform, which simplifies vendor risk assessment.

GDPR (EU data residency)

GDPR requires personal data to be processed within the EU or in jurisdictions with adequate protection. Plugsky offers data residency in EU regions. Self-hosted deployments within the EU satisfy GDPR by definition — data never leaves your infrastructure.

GCC data residency (KSA, UAE, Qatar, Oman, Bahrain, Kuwait)

Gulf Cooperation Council countries have strict data sovereignty requirements. Financial and government entities often require data to remain within national borders. Plugsky provides dedicated endpoint deployment inside GCC countries on the Enterprise tier. Self-hosted deployments inside the country are also compliant.

Other compliance frameworks

• ISO 27001 — Information security management. Plugsky is ISO 27001 certified.
• PDPL (Saudi Arabia) — Personal Data Protection Law. Data must remain in KSA. On-prem or Plugsky KSA region.
• NDMO (UAE) — National Data Management Office standards for UAE government entities.
• HIPAA — US healthcare. Both self-hosted and Plugsky BAA-supported deployments can achieve HIPAA compliance.

Procurement and deployment timeline

Self-hosted deployment

Phase	Duration	Activities
Hardware procurement	4-12 weeks	GPU vendor selection, PO approval, order, shipping, rack installation
Infrastructure setup	2-4 weeks	Network, storage, power, cooling, OS installation, driver configuration
Software deployment	2-6 weeks	vLLM/SGLang setup, model downloads, API gateway configuration, monitoring
Testing & validation	2-4 weeks	Load testing, model quality validation, failover testing, security review
Go-live	1-2 weeks	Cutover, user onboarding, documentation, runbook handoff

Total: 3-7 months. Estimated cost: $50K-$500K for hardware plus staffing.

Plugsky Enterprise deployment

Phase	Duration	Activities
Contract & onboarding	1-2 weeks	Contract signing, workspace creation, API key generation, documentation
Integration	1-4 weeks	Integrate OpenAI SDK, configure models, set up RBAC and budgets, test
Compliance review	1-2 weeks	Share SOC 2 reports, review data processing agreement, security questionnaire
Go-live	1 week	Cutover, user onboarding, disable old system

Total: 2-9 weeks. Estimated cost: $249-$2,500/month with no capital expenditure.