Local AI API — build locally with any framework, deploy with Plugsky when you need scale

Why local AI first

Starting with a local inference engine gives you five concrete advantages before you commit to any cloud provider:

1. Privacy — no data leaves your machine. Every prompt, completion, and embedding stays on your hardware. Useful for prototyping with sensitive data.
2. Cost — free experimentation. Download a model, run it on your laptop, burn through as many tokens as you want. No API bills, no surprise charges.
3. Speed — zero network latency on local GPU. Responses start streaming in milliseconds. Ideal for rapid prompt iteration and UI testing.
4. Iteration — try system prompts, temperature sweeps, tool definitions, and RAG pipelines without worrying about per-token cost. Iterate fast, break things, fix them.
5. Learning — understand model behaviour, quantization trade-offs, context windows, and inference tuning before tying yourself to any provider's API.

The bridge: OpenAI-compatible API

Every major local inference engine — Ollama, vLLM, llama.cpp, LM Studio, LocalAI, SGLang — exposes an OpenAI-compatible HTTP endpoint. Code written against the OpenAI SDK works against any of them by changing two values: the base_url and the model.

This means you can develop and test against a local engine today, then point the same code at Plugsky's production endpoint tomorrow — without a single line of application logic changing.

# Works against Ollama (localhost)
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
resp = client.chat.completions.create(model="llama3.2", messages=[...])

# Works against vLLM (localhost)
client = OpenAI(base_url="http://localhost:8000/v1", api_key="vllm")
resp = client.chat.completions.create(model="meta-llama/Llama-3.2-3B", messages=[...])

# Works against Plugsky (production)
client = OpenAI(base_url="https://api.plugsky.com/v1", api_key="sk-live-...")
resp = client.chat.completions.create(model="plugsky-lite", messages=[...])

The same pattern holds for every OpenAI SDK: Python, Node.js, Go, Rust, Ruby, PHP, and cURL. The request shape, the response shape, streaming format, error codes — all identical.

Local inference engines

Architecture patterns

Depending on where you are in your AI journey, one of these four architectures will match your setup:

a) Laptop / developer

App → OpenAI SDK → Ollama / LM Studio (localhost) → local model

Single user, single machine. The app talks to a local inference engine running on localhost. No network calls, no auth, no persistence. Ideal for prompt engineering and SDK integration testing.

b) Workstation / team

Apps → local gateway → Ollama / llama.cpp / vLLM → GPU workstation
         ├─ auth (API keys)
         ├─ budgets (per-user token caps)
         └─ logs (request/response audit)

A shared GPU workstation running vLLM or Ollama behind a lightweight API gateway. Multiple team members connect over the LAN. The gateway adds API key auth, per-user rate limits, and request logging.

c) Production private cloud

Applications → private API gateway → identity / RBAC + quotas → model router
         → vLLM / SGLang / TensorRT-LLM pools → private storage → observability

Multi-node GPU cluster with a model router, load balancing, persistent logging, and monitoring. This is the architecture you end up building yourself — or you let Plugsky run for you.

d) Hybrid sovereign

Sensitive prompts → in-country / private model
General prompts  → approved managed model
All traffic      → one policy / routing / audit control plane

Sensitive prompts route to a local engine inside the customer's perimeter. Non-sensitive prompts go to a managed provider. A single gateway enforces the routing policy and logs every request. This pattern is common for GCC data residency requirements.

When to stay local

Local AI is the right choice when:

• Single user — you are the only person using the API. No team, no sharing, no concurrent requests.
• Experimentation — you are prototyping, evaluating models, or learning. Free and fast iteration matters more than uptime.
• Sensitive data that cannot leave premises — classified, PII, PHI, or regulated data that must stay on your hardware at all times.
• No uptime requirements — if the laptop goes to sleep, the API goes down. That is acceptable for development.
• Learning and development — understanding GGUF quantization, context sizing, inference parameters, and prompt engineering on your own hardware builds deep knowledge.

When to move to Plugsky

Local AI hits its limits when you need any of these:

• Team access — multiple developers, internal tools, or customer-facing apps need the same endpoint with API key management and per-user budgets.
• Uptime / SLA — you need the API to be available 24/7 without depending on a laptop, workstation, or single GPU.
• Scaling beyond single GPU — your workload exceeds what one GPU or one machine can handle. Multi-node inference, request queuing, and auto-scaling become necessary.
• Multi-model access without managing infrastructure — you want to switch between 18+ models (including frontier models) without downloading weights, managing disk space, or running multiple engines.
• Audit and governance — you need per-request logs, RBAC, data retention policies, and SOC 2 / ISO 27001 compliance.
• GCC data residency — you need the API endpoint inside a specific Gulf Cooperation Council country with data never leaving that jurisdiction.

Migration path

Moving from local to production with Plugsky is a three-step process — and step 3 is optional:

1. Develop with any local engine + the OpenAI SDK. Iterate on prompts, tuning, and tools without cost or latency concerns.
2. Test with Plugsky's free tier. Change the base_url to https://api.plugsky.com/v1 and pick a free model like plugsky-lite. Same code, same SDK, same responses.
3. Graduate to a Plugsky paid plan when you need team access, uptime SLAs, higher rate limits, or frontier models. No code changes required — just update the API key.

Here is the migration in concrete terms:

# === Phase 1: Local development ===
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")

# === Phase 2: Plugsky free tier (same code, different base_url) ===
client = OpenAI(base_url="https://api.plugsky.com/v1", api_key="sk-live-...")

# === Phase 3: Plugsky production (same code, same base_url, upgraded key) ===
client = OpenAI(base_url="https://api.plugsky.com/v1", api_key="sk-live-PROD-...")

# The rest of your code never changes:
resp = client.chat.completions.create(
    model="plugsky-pro",
    messages=[{"role": "user", "content": "Hello"}],
    tools=[...],
    stream=True,
)

Local AI tools

FROM llama3.2
PARAMETER temperature 0.7
PARAMETER num_ctx 8192
SYSTEM "You are a helpful assistant."

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.2-3B-Instruct \
  --tensor-parallel-size 2 \
  --max-model-len 8192

Frequently asked questions