Local AI

Local AI API — run OpenAI-compatible inference on your own hardware

Set up a local endpoint with Ollama, vLLM, LM Studio, llama.cpp, or LocalAI. Same OpenAI SDK, same code, zero cloud dependency. When you need team access, scaling, and governance — migrate the same application to Plugsky without rewriting a single line.

Start Free → Local AI overview →

Ollama

Ollama is the most beginner-friendly local inference engine. It installs in one command, works on CPU and GPU, and exposes an OpenAI-compatible endpoint automatically. It has a large built-in model library and a simple CLI for pulling and running models.

Installation

bash 
# macOS
brew install ollama

# Linux
curl -fsSL https://ollama.com/install.sh | sh

# Start the server
ollama serve

Pull and run a model

bash 
# Pull a model
ollama pull llama3.2

# Run interactively
ollama run llama3.2

# The OpenAI-compatible endpoint is at:
# http://localhost:11434/v1

Test with curl

bash 
curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3.2",
    "messages": [{"role": "user", "content": "Hello from Ollama"}],
    "stream": false
  }'

vLLM

vLLM is the fastest inference engine for GPU servers. It uses PagedAttention for efficient memory management, continuous batching for high throughput, and supports tensor parallelism across multiple GPUs. Best for production-like local deployments.

Installation

bash 
# Install via pip (requires CUDA)
pip install vllm

# Start the server with a model
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.2-3B-Instruct \
  --tensor-parallel-size 1 \
  --max-model-len 8192 \
  --host 0.0.0.0 \
  --port 8000

Test with curl

bash 
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.2-3B-Instruct",
    "messages": [{"role": "user", "content": "Hello from vLLM"}]
  }'

LM Studio

LM Studio provides a graphical interface for downloading and running GGUF models. It is the best option for users who prefer a GUI over the command line. Built-in chat interface and local API server with one click.

Setup

  1. Download LM Studio from lmstudio.ai
  2. Search and download a model from the built-in model browser
  3. Go to the Server tab and click Start Server
  4. The endpoint is at http://localhost:1234/v1

Test with curl

bash 
curl http://localhost:1234/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "your-model-name",
    "messages": [{"role": "user", "content": "Hello from LM Studio"}]
  }'

llama.cpp

llama.cpp is a C++ implementation of LLM inference optimized for CPU and Apple Silicon. It supports Metal acceleration on macOS, offers the best CPU inference performance, and gives fine-grained control over quantization, context size, and batching.

Installation

bash 
# Clone and build
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make

# Or with Metal support (macOS)
LLAMA_METAL=1 make

# Start the server
./server -m models/llama-3.2-3b-instruct.Q4_K_M.gguf \
  --host 0.0.0.0 \
  --port 8080 \
  --ctx-size 8192

Test with curl

bash 
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3.2-3b-instruct",
    "messages": [{"role": "user", "content": "Hello from llama.cpp"}]
  }'

LocalAI

LocalAI is a Docker-first local inference engine that emulates the full OpenAI API surface — including embeddings, image generation, audio transcription, and text-to-speech — in addition to chat completions. It wraps multiple backends (llama.cpp, transformers, diffusers) under one API.

Installation (Docker)

bash 
docker run -p 8080:8080 \
  -v $PWD/models:/models \
  localai/localai:latest-gpu-nvidia-cuda-12 \
  /bin/bash -c "local-ai run"

Test with curl

bash 
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4",
    "messages": [{"role": "user", "content": "Hello from LocalAI"}]
  }'

Engine comparison

EngineBest forGPU requiredSetup difficultyOpenAI compatibility
OllamaBeginners, fast prototypingNo (CPU ok)Very easyFull
vLLMHigh throughput, production-like testingYesAdvancedFull
LM StudioGUI users, chat experimentationRecommendedVery easyFull
llama.cppCPU/Apple Silicon optimizationNoModerateFull
LocalAIFull OpenAI emulation (vision, audio, embeddings)OptionalModerate (Docker)Extended
SGLangStructured output, advanced schedulingYesAdvancedFull

Testing your API

Once your engine is running, verify the endpoint with a simple curl command. All engines expose the same /v1/chat/completions endpoint:

bash 
# Universal test — works with any engine
curl $BASE_URL/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "$MODEL_NAME",
    "messages": [{"role": "user", "content": "Say hello in 5 words"}]
  }'

Expected response: a JSON object with choices[0].message.content containing the model's reply. Streaming works by adding "stream": true — the response becomes a series of SSE events.

Python example

Here is the same code running against four different local engines — and Plugsky. Only the base_url and model change:

python 
from openai import OpenAI

# Ollama
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
r = client.chat.completions.create(model="llama3.2", messages=[{"role":"user","content":"Hi"}])

# vLLM
client = OpenAI(base_url="http://localhost:8000/v1", api_key="vllm")
r = client.chat.completions.create(model="meta-llama/Llama-3.2-3B-Instruct", messages=[{"role":"user","content":"Hi"}])

# LM Studio
client = OpenAI(base_url="http://localhost:1234/v1", api_key="lm-studio")
r = client.chat.completions.create(model="local-model", messages=[{"role":"user","content":"Hi"}])

# llama.cpp
client = OpenAI(base_url="http://localhost:8080/v1", api_key="llamacpp")
r = client.chat.completions.create(model="llama-3.2-3b-instruct", messages=[{"role":"user","content":"Hi"}])

# Plugsky (production)
client = OpenAI(base_url="https://api.plugsky.com/v1", api_key="sk-live-...")
r = client.chat.completions.create(model="plugsky-pro", messages=[{"role":"user","content":"Hi"}])

print(r.choices[0].message.content)

Migration to Plugsky

When your local setup hits limits — team access, uptime, multi-model access, audit logging, or scaling beyond a single GPU — the migration path is trivial:

  1. Same code. Your application uses the OpenAI SDK. No framework changes, no new SDK, no rewrites.
  2. Change two values: base_url to https://api.plugsky.com/v1 and model to a Plugsky model like plugsky-pro.
  3. Test on free tier. Plugsky's free tier gives you unlimited requests to plugsky-micro and plugsky-lite — no credit card, no commitment.
  4. Graduate. Switch to a paid plan when you need team access, higher rate limits, frontier models, SLAs, or data residency.

Your code stays exactly the same. The same request body, the same response shape, the same streaming format, the same error codes. Migration is a configuration change, not an engineering project.

Build locally, deploy globally

Develop against any local engine with the OpenAI SDK. Deploy to Plugsky with a two-line config change. Same code, same tools, zero rework.

Start Free → Back to Local AI →

Related

Local AI agents

Build private AI agents that run entirely on your hardware.

Best local LLMs

Tested models with quantization, VRAM usage, and token speed benchmarks.

OpenAI-compatible API

Plugsky's managed endpoint with 30+ models, one base_url.