Streaming

Set "stream": true on POST /v1/chat/completions to receive text/event-stream instead of one JSON body. The response uses server-sent events with one data: line per event and a blank line between events.

Request

curl -N https://<your-domain>/v1/chat/completions \
  -H "Authorization: Bearer $EASYAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "lfm3",
    "messages": [
      { "role": "user", "content": "Count from one to five." }
    ],
    "stream": true
  }'

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://<your-domain>/v1",
    api_key=os.environ["EASYAI_API_KEY"],
)

stream = client.chat.completions.create(
    model="lfm3",
    messages=[{"role": "user", "content": "Count from one to five."}],
    stream=True,
)

for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="")

const response = await fetch("https://<your-domain>/v1/chat/completions", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.EASYAI_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    model: "lfm3",
    messages: [{ role: "user", content: "Count from one to five." }],
    stream: true,
  }),
});

const reader = response.body.getReader();
const decoder = new TextDecoder();
while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  process.stdout.write(decoder.decode(value, { stream: true }));
}

Event format

The gateway writes one chunk per generated token:

data: {"id":"chatcmpl-<job-id>","object":"chat.completion.chunk","created":1762560000,"model":"lfm3","choices":[{"index":0,"delta":{"content":"one"}}]}

data: {"id":"chatcmpl-<job-id>","object":"chat.completion.chunk","created":1762560000,"model":"lfm3","choices":[{"index":0,"delta":{"content":" two"}}]}

data: [DONE]

Chunks contain a single choice at index 0 and a delta.content string. They do not carry a per-chunk usage object or a finish_reason. The stream ends with the exact sentinel data: [DONE] followed by a blank line.

Response headers include Content-Type: text/event-stream, Cache-Control: no-cache, and X-RateLimit-Remaining.

Failures after the stream starts

Validation, authentication, rate-limit, context, credit, and capacity failures that occur before the first event use a normal HTTP error with the { "error": { "code": "..." } } body.

Once headers are sent, a fatal failure is written as one event and the connection closes:

data: {"error":{"code":"DEADLINE_EXCEEDED","message":"job deadline exceeded"}}

A stream that ends without data: [DONE] is incomplete. Treat the partial output as failed rather than as a finished answer.

Cancellation

Closing the client connection requests cancellation of the job. The gateway stops generating at the next cancellation check and releases the reservation. Cancellation is best-effort; a job that has already completed is returned or settled normally.

Rate limit

Streaming requests consume the same per-key window as non-streaming requests. A request rejected before the stream starts returns 429 RATE_LIMITED as JSON. See Errors.