Key facts
| Definition | A proxy that centralizes access, policy and observability for model traffic |
| Core features | Key management, routing, rate limits, caching, logging and usage tracking |
| Why it matters | Applications stop hard-coding providers and gain one place to enforce policy |
| Plugsky as gateway | One OpenAI-compatible endpoint in front of 30+ models |
| Governance | Scoped API keys, RBAC, SSO and audit logs on the platform side |
| Routing | Model selection per request enables cost and capability routing |
| Deployment | Cloud, VPC, on-prem and air-gapped options |
| Status | Chat, streaming, function calling and embeddings are live behind the gateway |
TL;DR
- A gateway centralizes what would otherwise be scattered in every service.
- Keys, routing, limits and logs belong in one control plane.
- OpenAI compatibility keeps clients simple and portable.
- Gateways add a hop — plan capacity and health checks.
- Plugsky provides this layer in front of 30+ models.
How it works, step by step
- Inventory the model providers and credentials currently used across services.
- Decide which policies to centralize: keys, routing, limits, caching and logging.
- Point applications at the gateway endpoint rather than provider endpoints directly.
- Issue scoped keys per service and environment through the gateway.
- Configure routing rules and fallbacks for model availability.
- Monitor usage, errors and latency from the gateway's observability data.
Try it yourself
Open the AI workload router simulator →
What an AI gateway does
The gateway is the single door to model providers. Applications authenticate to it with scoped keys; it authenticates onward to providers with managed credentials. Along the way it can route to different models, enforce rate limits, cache identical requests, retry failures and record usage. The benefit is uniform policy and visibility rather than each team reimplementing authentication, retries and logging.
Features that matter
- Key management: issue, scope and revoke keys without touching provider accounts.
- Routing: choose models by task, cost or availability, with fallbacks.
- Limits: per-key and per-team quotas that protect shared capacity.
- Caching: serve repeated requests without a model call where appropriate.
- Observability: usage, latency and error data in one place.
- Governance: RBAC, SSO and audit logs for regulated environments.
Common mistakes
- Letting the gateway become a single point of failure without health checks or redundancy.
- Centralizing keys but leaving retry and timeout logic in every client.
- Caching responses that depend on user-specific context, leaking data across requests.
- Routing without evaluation, so cost savings arrive with quality regressions.
- Ignoring the extra network hop when latency budgets are tight.
Plugsky as an AI gateway
Plugsky already presents 30+ models behind one OpenAI-compatible endpoint, which is the core gateway function: applications hold one base URL and one SDK, and model choice is a parameter. Platform controls add scoped keys, RBAC, SSO and audit logs, while routing across models supports cost management. Private deployment options move the gateway inside your network when public egress is not acceptable. For teams that need provider-agnostic caching or multi-cloud provider fan-out beyond a single vendor, the same endpoint still serves as the standard integration point.
Honest comparison
| Concern | With gateway | Direct provider calls | Custom proxy built in-house |
|---|---|---|---|
| Key management | Central and scoped | Scattered per service | You build it |
| Provider switching | Configuration change | Code change everywhere | Your abstraction |
| Observability | One usage view | Per-service logs | You instrument it |
| Governance | RBAC, SSO, audit | Per-provider controls | You implement |
| Failure risk | One hop to harden | Many integrations | You operate it |
Frequently asked questions
What is an AI gateway?
A proxy that sits between applications and model providers to centralize authentication, routing, rate limits, caching, logging and cost tracking.
Is an AI gateway the same as an LLM gateway?
Yes, the terms are used interchangeably. Both describe a control plane for model traffic, as opposed to a single provider SDK.
Does a gateway add latency?
It adds one network hop. Well-run gateways keep that overhead small, and caching or connection reuse can compensate, but budget for it in latency-sensitive paths.
Is Plugsky an AI gateway?
Plugsky provides the gateway function — one OpenAI-compatible endpoint for 30+ models — plus key management, RBAC, audit logs and deployment options.
Can a gateway do cost optimization?
Yes, through model routing and usage visibility. Routing decisions should be designed and evaluated so savings do not come with quality regressions.
Do I still need provider accounts with a gateway?
If the gateway is your only vendor, no. If it aggregates multiple providers, the gateway holds those credentials and your applications still integrate once.