Short answer. Use the OpenAI SDK with
stream=True, iterate the chunks inside a generator, format each delta asdata: {...}\n\n, and returnStreamingResponse(..., media_type="text/event-stream"). SetCache-Control: no-cacheandX-Accel-Buffering: no. Use an async client in production so a client disconnect cancels the upstream request.
Minimal streaming endpoint
import json
from fastapi import FastAPI
from fastapi.responses import StreamingResponse
from openai import AsyncOpenAI
app = FastAPI()
client = AsyncOpenAI(
api_key="YOUR_PLUGSKY_KEY",
base_url="https://api.plugsky.com/v1",
)
@app.post("/chat/stream")
async def chat_stream(payload: dict):
async def events():
stream = await client.chat.completions.create(
model=payload.get("model", "plugsky-plus"),
messages=payload["messages"],
stream=True,
)
async for chunk in stream:
delta = chunk.choices[0].delta.content
if delta:
yield f"data: {json.dumps({'delta': delta})}\n\n"
yield "data: [DONE]\n\n"
return StreamingResponse(
events(),
media_type="text/event-stream",
headers={
"Cache-Control": "no-cache",
"Connection": "keep-alive",
"X-Accel-Buffering": "no",
},
)
Client-side reader
const res = await fetch("/chat/stream", {
method: "POST",
headers: {"Content-Type": "application/json"},
body: JSON.stringify({messages: [{role: "user", content: "Hello"}]}),
});
const reader = res.body.getReader();
const decoder = new TextDecoder();
while (true) {
const {value, done} = await reader.read();
if (done) break;
process.stdout.write(decoder.decode(value));
}
EventSource only supports GET requests. For POST with a JSON body, read the response body with fetch and parse the SSE frames yourself, or use a small SSE parser.
Disconnects and cancellation
When the browser disconnects, FastAPI cancels the request task. With AsyncOpenAI, the upstream request is cancelled with it. With the synchronous client in a thread, cancellation is less predictable and the model may keep generating (and billing) after the user leaves.
- Use
AsyncOpenAIandasync for. - Catch
asyncio.CancelledErrorand close the stream cleanly. - Do not write to a closed connection in a background task.
Proxy buffering
If tokens arrive in one burst, something is buffering. Check:
- nginx:
proxy_buffering off;andX-Accel-Buffering: no. - Cloudflare or another CDN: confirm it does not buffer
text/event-stream. - Compression middleware: gzip can hold the stream open; disable it for SSE.
- Application middleware that reads the whole response body.
Errors after the first byte
Once the response headers are sent, you cannot change the HTTP status. Send an in-band error event and close:
try:
async for chunk in stream:
...
except Exception as exc:
yield f"data: {json.dumps({'error': 'stream_failed'})}\n\n"
yield "data: [DONE]\n\n"
The client must treat an error event as a failure, not as text. Log the request ID and provider error for support.
Production checklist
- Authenticate the caller before opening the upstream stream.
- Cap
max_tokensand the request timeout. - Record token usage from the final chunk or a follow-up usage call.
- Retry only before the first token; after that, fail in-band.
- Test with a slow client and a proxy in front of the app.
FAQ
How do I stream from FastAPI?
Call the provider with stream=True, yield SSE frames from a generator, and return a StreamingResponse with text/event-stream.
Why does the stream arrive all at once?
A proxy or middleware is buffering. Disable nginx buffering, set X-Accel-Buffering: no, and avoid compression for SSE.
SSE or WebSockets?
SSE for one-way token streaming; WebSockets for bidirectional or voice/collaboration workloads.
What happens when the client disconnects?
The generator should stop and the upstream request should be cancelled. Use an async client.
How do I handle errors mid-stream?
Send an in-band error event and close; the HTTP status can no longer change.
Reviewed 20 November 2026. FastAPI, OpenAI SDK and proxy behaviour vary by version and deployment; test streaming through your actual edge before launch.