Self-hosted OpenAI-compatible API — develop locally, deploy to Plugsky without code changes

Why self-host an OpenAI-compatible API

Running your own OpenAI-compatible endpoint gives you control that managed APIs cannot match:

• Privacy — data never leaves your network. Every prompt, completion, and embedding stays on hardware you control. No third party ever sees your inputs or outputs.
• Cost — no per-token fees during development. Iterate on prompts, test chains, and build features without watching a meter run. Your GPU is a sunk cost; tokens are free.
• Control — choose your model, quantization level, context length, and inference parameters. Switch between Llama, Mistral, Qwen, or DeepSeek with a command, not a contract change.
• Offline capability — no internet required once the model is downloaded. Run your AI pipeline in air-gapped environments, on planes, or during outages.
• Development speed — zero-latency feedback during iteration. No API call overhead, no rate limits, no queueing. Your edit-inference loop stays under one second.

Options for self-hosting

Several open-source engines expose an OpenAI-compatible API. Each has different strengths:

Quick start with Ollama

Ollama is the fastest way to get an OpenAI-compatible API running on any machine.

# Install Ollama
curl -fsSL https://ollama.ai/install.sh | sh

# Pull a model
ollama pull llama3.2

# The server runs on localhost:11434 by default
# Test the API
curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3.2",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Now use any OpenAI SDK against your local endpoint:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama",  # Ollama ignores the key but it must be present
)

resp = client.chat.completions.create(
    model="llama3.2",
    messages=[{"role": "user", "content": "Write a haiku about local AI."}],
)
print(resp.choices[0].message.content)

Ollama also supports vision models, embeddings, and concurrent requests. For production workloads with higher throughput, switch to vLLM.

Production with vLLM

vLLM is built for throughput. It uses PagedAttention to manage KV cache efficiently and supports continuous batching for high concurrency.

# Install vLLM
pip install vllm

# Download a model and start the server
python -m vllm.entrypoints.openai.api_server \
  --model mistralai/Mistral-7B-Instruct-v0.3 \
  --host 0.0.0.0 \
  --port 8000 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90 \
  --enforce-eager

# Test
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mistralai/Mistral-7B-Instruct-v0.3",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="token-abc123",  # vLLM requires a placeholder or --api-key
)

resp = client.chat.completions.create(
    model="mistralai/Mistral-7B-Instruct-v0.3",
    messages=[{"role": "user", "content": "Explain PagedAttention in one sentence."}],
    max_tokens=200,
)
print(resp.choices[0].message.content)

For multi-GPU setups, add --tensor-parallel-size 2 (or 4, 8). vLLM can serve hundreds of concurrent requests on a single node with proper GPU memory.

The bridge to Plugsky

The key insight: every engine above exposes an OpenAI-compatible API. Code written against any of them works against Plugsky with just a base URL change. The request and response shapes are identical — same chat completions, same streaming, same function calling, same embeddings.

# Local development — Ollama, vLLM, llama.cpp, etc.
client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama",
)

# Production — Plugsky, zero code changes beyond config
client = OpenAI(
    base_url="https://api.plugsky.com/v1",
    api_key="sk-live-PLUGSKY...",
)

# The rest of your code stays exactly the same
resp = client.chat.completions.create(
    model="plugsky-pro",
    messages=[{"role": "user", "content": "Hello!"}],
)

This pattern works with every SDK and every framework:

• Python: openai ≥ 1.0 — change base_url and api_key
• Node.js / TypeScript: openai, vercel/ai-sdk — change baseURL
• Go: sashabaranov/go-openai — change BaseURL
• Rust: async-openai — change base_url
• curl: change the host and add an auth header

Your local development environment, CI tests, staging server, and production all share the same code. Only the configuration file differs.

Choosing between self-hosted and managed

Neither approach is universally better. The right choice depends on your workload, team size, and operational capacity.

Hybrid approach

You do not have to choose one. The most pragmatic setup uses both — self-hosted for what it is good at and Plugsky for what it is good at.

A typical hybrid architecture:

• Local Ollama instance — development, testing, prompt engineering, sensitive data workloads (PII, legal, internal docs)
• Plugsky API — production user-facing traffic, team collaboration, high-availability endpoints, models your GPU cannot run

import os
from openai import OpenAI

def get_client():
    """Route based on environment or data sensitivity."""
    env = os.getenv("APP_ENV", "development")

    if env == "development":
        return OpenAI(
            base_url="http://localhost:11434/v1",
            api_key="ollama",
        )

    # Production — use Plugsky
    return OpenAI(
        base_url="https://api.plugsky.com/v1",
        api_key=os.getenv("PLUGSKY_API_KEY"),
    )

def route_by_sensitivity(prompt: str):
    """Route sensitive prompts to local, others to Plugsky."""
    sensitive_keywords = ["ssn", "salary", "health", "password"]
    if any(kw in prompt.lower() for kw in sensitive_keywords):
        return OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
    return OpenAI(base_url="https://api.plugsky.com/v1", api_key=os.getenv("PLUGSKY_API_KEY"))

This hybrid pattern gives you the best of both worlds: privacy and zero-cost iteration locally, scale and reliability in production.

Frequently asked questions