OpenAI-Compatible API Compatibility Report — July 2026

Executive Summary

The OpenAI API has become the de-facto standard for large language model inference. Its /v1/chat/completions request and response shapes are replicated by dozens of providers. But "OpenAI-compatible" is not a regulated term. The degree of compatibility varies widely — from drop-in replacement (change one line of code) to partial support requiring workarounds for every non-trivial feature.

We tested 12 AI providers against six categories: chat completions, streaming, tool/function calling, JSON mode / structured output, embeddings, and error format. Our tests were run against each provider's official API endpoint using the OpenAI Python SDK (v1.30+) with base_url changed to each provider's endpoint. No provider-specific SDKs were used — we tested only the OpenAI-compatible surface.

Key findings: Streaming works everywhere. Basic chat completions pass universally. Tool calling has subtle differences in parallel execution and tool_choice behavior. JSON mode varies significantly — only OpenAI, Plugsky, and DeepSeek fully support strict schema enforcement. Embeddings are missing from most providers. Error format is the most inconsistent category, with some providers returning 400 with OpenAI-shaped errors and others returning 500 with entirely custom shapes. Plugsky is the only provider besides OpenAI to score 100% across all categories.

Methodology

Test dates: July 10–11, 2026. All tests were run from a single AWS eu-west-1 instance to minimize geographic bias.

Test framework: Custom Python test suite using the openai Python SDK v1.30+ in library mode (no API proxy). Each test sends a well-formed request and validates the response against the OpenAI specification. Tests are run three times per provider to account for transient failures.

Categories tested:

• Chat completions: Basic request with messages, model, temperature, max_tokens. Response must contain choices[0].message.content.
• Streaming: Same request with stream: true. Each chunk must contain choices[0].delta with content or tool_calls. Final chunk must have finish_reason.
• Tool/Function calling: Request with tools array containing at least one function definition. Must support tool_choice: "auto" and tool_choice: {"type": "function", "function": {"name": "..."}}.
• JSON mode / Structured output: Request with response_format: {"type": "json_object"}. Response content must be valid JSON. Where supported, tests response_format: {"type": "json_schema", "json_schema": {...}}.
• Embeddings: Request to /v1/embeddings with model and input. Response must contain data[0].embedding as a float array. Tests dimensions parameter.
• Error format: Malformed requests (missing model, invalid auth, oversize input). Response must have HTTP 4xx status and JSON body with an error object containing message and type or code fields.

Authentication: Each provider was tested with a valid API key obtained for that provider. Rate limits were respected; tests were throttled to 10 requests per minute per provider.

Models tested: Each provider was tested with their fastest available model. Plugsky was tested with plugsky-micro and plugsky-pro. OpenAI was tested with gpt-4o-mini. Anthropic with claude-sonnet-4-20250514. Full model list available in the downloadable results.

Full test code is available on request. Contact research@plugsky.com for the complete test suite including edge cases, rate limit tests, and response timing data.

Compatibility Score Table

Scores are calculated as the percentage of tests passed in each category. (✅ = full support, ⚠️ = partial/limited support, ❌ = not supported)

Updated July 2026. Percentage = tests passed / total tests. Baseline is OpenAI's own API behavior.

Detailed Findings: Chat Completions

Result: All providers pass.

Basic chat completions — a POST to /v1/chat/completions with model, messages, and a max_tokens — work across every provider tested. This is the lowest common denominator of OpenAI compatibility. Every provider in our test set correctly returns a choices array with at least one completion choice containing a message object with content.

Differences emerge in parameter handling. OpenAI supports a large parameter surface (temperature, top_p, frequency_penalty, presence_penalty, seed, logprobs, top_logprobs, stop, etc.). Most providers silently ignore unsupported parameters rather than rejecting them, which is the most forgiving behavior for migration. Cerebras and Groq reject certain parameters with a 400 error.

Detailed Findings: Streaming

Result: All providers pass, but SSE event format varies.

Server-Sent Events (SSE) streaming with stream: true works across all 12 providers. Every provider sends data: {"choices": [{"delta": {"content": "..."}}]} chunks followed by a data: [DONE] terminal event.

However, the exact event format varies. Anthropic's OpenAI-compatible endpoint emits additional non-standard fields (anthropic_index, anthropic_type) alongside the OpenAI-standard fields. These are harmless for most SDKs — they just deserialize into the known OpenAI response type — but consumers that validate strict response shapes may reject them. Plugsky and DeepSeek follow the OpenAI event format exactly, emitting only the documented fields.

Rate of chunk emission also varies. Cerebras and Groq emit at <1ms per token (hardware-accelerated inference). OpenAI and Plugsky emit at 5–15ms per token depending on model size.

Detailed Findings: Tool / Function Calling

Result: Most support basics; few support advanced features.

Basic tool calling (sending a tools array and receiving tool_calls in the response) is supported by 10 of 12 providers. Cerebras and Groq do not support tool calling at all — Cerebras returns a parse error, Groq returns a 400 with "tool_use not supported".

More nuanced differences surface with parallel_tool_calls (introduced by OpenAI in late 2024). OpenAI, Plugsky, DeepSeek, and Mistral support emitting multiple tool calls in a single response turn. Together AI and Fireworks AI support it inconsistently — sometimes returning parallel calls, sometimes serializing them. Anthropic and Google require tool_choice to be explicitly set, and Google does not respect tool_choice: "none" (it still calls tools).

tool_choice behavior differs across providers:

• OpenAI, Plugsky, Mistral: Full tool_choice support — "auto", "none", "required", and {"type": "function", "function": {"name": "..."}}.
• Anthropic: OpenAI-compatible endpoint respects tool_choice but "auto" behavior is more aggressive (calls tools more frequently than OpenAI).
• Google Gemini: Does not respect "none". Will still call tools when tool_choice: "none" is set.
• Together AI, Fireworks AI: tool_choice works but "required" is not supported.
• OpenRouter: Behavior depends on the underlying model — OpenRouter proxies to the selected model's native behavior.

Detailed Findings: JSON Mode / Structured Output

Result: Varies significantly — only 3 providers fully support structured JSON output.

OpenAI introduced response_format: {"type": "json_object"} in late 2023 and the stricter {"type": "json_schema", "json_schema": {"schema": {...}}} in late 2024. This is the most inconsistent feature across providers.

Full support (json_object + json_schema): OpenAI, Plugsky, DeepSeek. All three enforce valid JSON output at the API level — if the model produces invalid JSON, the API returns an error.

Partial support (json_object only): Anthropic (via OpenAI endpoint), Google Gemini, Mistral, Together AI, Fireworks AI. These support json_object but do not support schema enforcement. The model is instructed to output JSON but can produce invalid structures. Mistral and Together AI also require specific models for JSON mode to work.

Not supported: xAI/Grok, Groq, Cerebras. These return a 400 error when response_format: {"type": "json_object"} is sent. xAI returns an informative message; Groq and Cerebras return generic errors.

Detailed Findings: Embeddings

Result: Widely missing — only 5 of 12 providers support the OpenAI embeddings format.

The /v1/embeddings endpoint is the least commonly implemented OpenAI API surface. Only OpenAI, Plugsky, Mistral, Anthropic (via a separate OpenAI-compatible embeddings endpoint), and Google Gemini support it. Anthropic's embeddings coverage is limited — it does not support the dimensions parameter (output is always the model's native dimensionality). Google Gemini supports dimensions.

Of the providers that do support embeddings, response shapes vary. OpenAI and Plugsky return data[0].embedding as a standard float array. Mistral returns the same shape but with non-standard metadata in the response object. All others (xAI, DeepSeek, Together AI, Fireworks AI, Groq, Cerebras, OpenRouter) return 404 or 400 — the /v1/embeddings endpoint does not exist.

Detailed Findings: Error Format

Result: Varies widely — the most inconsistent category.

OpenAI returns errors in a consistent shape: HTTP status 4xx with {"error": {"message": "...", "type": "...", "code": "..."}}. Only 4 providers — Plugsky, Mistral, Together AI, and Anthropic — return this exact format. Plugsky matches the OpenAI error shape exactly, down to error codes like invalid_api_key, insufficient_quota, and model_not_found.

Other providers deviate in various ways:

• Google Gemini: Returns HTTP 500 for some invalid requests (should be 400). Error body uses a different structure ({"error": {"code": ..., "message": "...", "status": "..."}}).
• xAI/Grok: OpenAI-shaped errors but returns HTTP 500 for rate limits instead of 429.
• DeepSeek: OpenAI-shaped errors but some return HTTP 502 instead of 400 (upstream proxy error).
• Groq: Custom error format with {"detail": "..."} instead of {"error": {"message": "..."}}.
• Fireworks AI: OpenAI-shaped for validation errors; returns HTTP 502 with custom body for upstream failures.
• Cerebras: FastAPI default error format ({"detail": [...]}).
• OpenRouter: Varies by underlying model — transparently proxies the upstream provider's error format.

What This Means for You

If you are using basic chat completions (send messages, get text back), any provider works. The decision comes down to pricing, latency, and model quality.

If you need streaming, all providers support it. The minor differences in SSE event format are invisible when using the OpenAI SDK.

If you need tool calling + JSON mode + embeddings, your choices narrow significantly. Of the 12 providers we tested, only OpenAI and Plugsky support all three with full fidelity. DeepSeek supports tools and JSON mode but not embeddings. Mistral supports tools and embeddings but JSON mode is limited to json_object (no schema enforcement).

If you are migrating an existing codebase that uses multiple OpenAI features — chat, streaming, tools, JSON mode, embeddings, and expects consistent error handling — Plugsky is the only provider besides OpenAI that scores 100% across all categories. This means zero code changes beyond the base_url swap:

# Before (OpenAI)
client = OpenAI(api_key="sk-...")

# After (Plugsky)
client = OpenAI(
    api_key="sk-live-PLUGSKY...",
    base_url="https://api.plugsky.com/v1",
)

Download Full Results

The complete test output is available as a CSV file with per-test results (pass/fail, response shape, error message, latency) for all 12 providers across all 6 categories. This file is suitable for importing into your own analysis or sharing with your team.

Limitations

This report reflects API behavior as of July 10–11, 2026. AI providers update their APIs frequently — sometimes weekly. A feature marked as ⚠️ (partial) or ❌ (not supported) in this report may have been added after our test date. We recommend testing your specific use case against each provider before making a migration decision.

Our tests use the OpenAI Python SDK v1.30+. Provider-specific SDKs may offer features that the OpenAI SDK does not expose. This report only evaluates the OpenAI-compatible API surface, not native SDK support.

We tested each provider with a single representative model. Provider performance may vary across models within the same provider. For example, Anthropic's Claude Opus may handle tools differently than Claude Sonnet.

Network conditions, rate limits, and geographic routing can affect latency and error rates. All tests were run from a single AWS region (eu-west-1). Results may differ from other geographic locations.

Changelog

Frequently Asked Questions