How does Plugsky streaming with server-sent events work?

Key facts

Endpoint	POST https://api.plugsky.com/v1/chat/completions with stream=true
Wire format	Server-sent events: data: {json} lines ending with data: [DONE]
Chunk shape	choices[0].delta.content carries the text fragment
Final metadata	The last chunk includes usage for observability
SDK support	Python, Node, Go, Java, Rust and cURL examples in the docs
Back-pressure	Iterate incrementally; the docs show a streamed example with final usage
Tool calls	Tool-call deltas can stream inside the same event stream
Product status	Live

TL;DR

• One flag changes the transport: stream=true turns the response into SSE.
• Append delta.content fragments; do not re-parse the full message each time.
• The stream ends with [DONE]; final usage arrives in the last chunk.
• Cancel on client disconnect and set server timeouts long enough for slow models.
• Disable proxy buffering or streaming will look broken in production.

How it works, step by step

1. Set stream=true on the chat completions request and read the response as an event stream.
2. Iterate chunks and append delta.content to your output buffer.
3. Capture the final chunk for usage and the finish reason.
4. Handle [DONE] as normal termination, not an error.
5. Cancel the upstream request when the client disconnects.
6. Configure reverse proxies with buffering disabled and generous read timeouts.
7. Log time-to-first-token and total latency per stream to track quality of experience.

Try it yourself

Open the OpenAI-compatible API tester →

The streaming contract

Streaming changes only the response encoding. The request body is a normal chat completion with stream: true, and the wire format is server-sent events:

data: {"choices":[{"delta":{"content":"Hello"}}]}

data: {"choices":[{"delta":{"content":" there"}}]}

data: [DONE]

With the SDKs you rarely see that format — Python yields chunks and the docs' back-pressure example prints stream.text_stream before reading the final completion and its usage. Under the hood, each chunk is a partial message, and only the accumulated concatenation forms the full answer.

Back-pressure, cancellation and errors

• Write as you read: flush each fragment to the client so users see progress immediately; batch only when your downstream cannot handle small writes.
• Cancel upstream: when a user navigates away or aborts, close the HTTP request so the model stops generating. Leaked streams consume capacity and confuse rate-limit math.
• Mid-stream errors: an error can arrive after chunks. Surface a clear failure in the UI instead of leaving a half-sentence on screen.
• Retries: do not silently retry a stream that already emitted text; duplicate output is worse than a visible retry button.
• Idempotency: for create-type POSTs, send Idempotency-Key so a retry does not duplicate server-side work.

Production proxy and platform details

Most "streaming is broken" reports trace to infrastructure. Disable response buffering in Nginx (see proxy_buffering off), avoid compressing SSE, and raise read timeouts above your slowest model's time-to-last-token. Serverless platforms often cap response duration — verify your runtime allows long-lived streams, or proxy through a service that does. On the client side, set an inactivity timeout rather than a total timeout so a long but progressing answer is not killed at an arbitrary limit.

What to measure

Track time-to-first-token, inter-token latency and total stream duration per model and per endpoint. Time-to-first-token dominates perceived responsiveness, while inter-token latency mostly affects reading comfort; both change with model choice, so include them in routing decisions. The final chunk's usage plus per-request logs (model, latency, status, request ID) give you the observability needed to explain regressions. Streaming is live on the chat completions endpoint; tool-call deltas arrive in the same stream, so agent UIs can render actions as they happen.

Honest comparison

Capability	Plugsky streaming	Non-streamed request	WebSocket protocol
Transport	Server-sent events over HTTP	Single JSON response	Bidirectional socket
Protocol change	stream=true flag only	Default	Different contract and client
Perceived latency	First token arrives early	Wait for full answer	Early, but more moving parts
Usage data	Final chunk includes usage	Usage in the response	Custom plumbing
Cancellation	Close the request	No partial work to cancel	Explicit protocol frames
Infra notes	Disable proxy buffering	None	Sticky sessions and heartbeats

Frequently asked questions

How do I enable streaming?

Send stream=true in the body of POST /v1/chat/completions. The response becomes a server-sent event stream; SDKs expose it as an iterator so you can append text fragments.

What format does the stream use?

Server-sent events: data: lines each carrying a JSON chunk, terminated by data: [DONE]. Each chunk has a delta object with the incremental content.

How do I get token usage from a stream?

The final chunk of the stream includes usage. In the SDKs, read the final completion object after iterating the stream, as shown in the docs' back-pressure example.

Why does streaming appear to hang in production?

Usually proxy buffering or an aggressive timeout. Disable response buffering, avoid compression on SSE, and raise read timeouts above your slowest model's last-token time.

Can tool calls stream?

Yes. Tool-call deltas arrive in the same event stream, so UIs can show actions as they are chosen. Accumulate argument fragments by index before executing.

Should I retry a stream that failed mid-way?

No. Retrying after text was emitted duplicates output. Surface the failure and let the user retry explicitly; use Idempotency-Key if a server-side action may have executed.

Does streaming cost more?

No. Self-serve plans are flat monthly with fair-use usage rather than per-token billing, and streaming uses the same endpoint and pricing model as non-streamed calls.

How do I cancel a stream?

Close the underlying HTTP request when the user aborts or navigates away. That stops generation upstream and prevents leaked work from consuming capacity.