Integration · FastAPI

FastAPI streaming with an OpenAI API

Server-Sent Events are the simplest way to stream tokens from an OpenAI-compatible endpoint through FastAPI. The hard parts are proxy buffering, disconnects and errors after the headers are sent.

Short answer. Use the OpenAI SDK with stream=True, iterate the chunks inside a generator, format each delta as data: {...}\n\n, and return StreamingResponse(..., media_type="text/event-stream"). Set Cache-Control: no-cache and X-Accel-Buffering: no. Use an async client in production so a client disconnect cancels the upstream request.

Minimal streaming endpoint

import json
from fastapi import FastAPI
from fastapi.responses import StreamingResponse
from openai import AsyncOpenAI

app = FastAPI()

client = AsyncOpenAI(
    api_key="YOUR_PLUGSKY_KEY",
    base_url="https://api.plugsky.com/v1",
)

@app.post("/chat/stream")
async def chat_stream(payload: dict):
    async def events():
        stream = await client.chat.completions.create(
            model=payload.get("model", "plugsky-plus"),
            messages=payload["messages"],
            stream=True,
        )
        async for chunk in stream:
            delta = chunk.choices[0].delta.content
            if delta:
                yield f"data: {json.dumps({'delta': delta})}\n\n"
        yield "data: [DONE]\n\n"

    return StreamingResponse(
        events(),
        media_type="text/event-stream",
        headers={
            "Cache-Control": "no-cache",
            "Connection": "keep-alive",
            "X-Accel-Buffering": "no",
        },
    )

Client-side reader

const res = await fetch("/chat/stream", {
  method: "POST",
  headers: {"Content-Type": "application/json"},
  body: JSON.stringify({messages: [{role: "user", content: "Hello"}]}),
});

const reader = res.body.getReader();
const decoder = new TextDecoder();

while (true) {
  const {value, done} = await reader.read();
  if (done) break;
  process.stdout.write(decoder.decode(value));
}

EventSource only supports GET requests. For POST with a JSON body, read the response body with fetch and parse the SSE frames yourself, or use a small SSE parser.

Disconnects and cancellation

When the browser disconnects, FastAPI cancels the request task. With AsyncOpenAI, the upstream request is cancelled with it. With the synchronous client in a thread, cancellation is less predictable and the model may keep generating (and billing) after the user leaves.

  • Use AsyncOpenAI and async for.
  • Catch asyncio.CancelledError and close the stream cleanly.
  • Do not write to a closed connection in a background task.

Proxy buffering

If tokens arrive in one burst, something is buffering. Check:

  • nginx: proxy_buffering off; and X-Accel-Buffering: no.
  • Cloudflare or another CDN: confirm it does not buffer text/event-stream.
  • Compression middleware: gzip can hold the stream open; disable it for SSE.
  • Application middleware that reads the whole response body.

Errors after the first byte

Once the response headers are sent, you cannot change the HTTP status. Send an in-band error event and close:

try:
    async for chunk in stream:
        ...
except Exception as exc:
    yield f"data: {json.dumps({'error': 'stream_failed'})}\n\n"
    yield "data: [DONE]\n\n"

The client must treat an error event as a failure, not as text. Log the request ID and provider error for support.

Production checklist

  • Authenticate the caller before opening the upstream stream.
  • Cap max_tokens and the request timeout.
  • Record token usage from the final chunk or a follow-up usage call.
  • Retry only before the first token; after that, fail in-band.
  • Test with a slow client and a proxy in front of the app.

FAQ

How do I stream from FastAPI?

Call the provider with stream=True, yield SSE frames from a generator, and return a StreamingResponse with text/event-stream.

Why does the stream arrive all at once?

A proxy or middleware is buffering. Disable nginx buffering, set X-Accel-Buffering: no, and avoid compression for SSE.

SSE or WebSockets?

SSE for one-way token streaming; WebSockets for bidirectional or voice/collaboration workloads.

What happens when the client disconnects?

The generator should stop and the upstream request should be cancelled. Use an async client.

How do I handle errors mid-stream?

Send an in-band error event and close; the HTTP status can no longer change.

Reviewed 20 November 2026. FastAPI, OpenAI SDK and proxy behaviour vary by version and deployment; test streaming through your actual edge before launch.