Skip to main content

Streaming responses

Chat completions can return output incrementally using server-sent events (SSE). Set stream: true on the same chat endpoint; the project key and tier permissions are unchanged.

Streaming changes how you receive the response, not what the model is allowed to do. Function tools still need to be executed by your application.

Make a streaming request​

Using the Quickstart variables:

curl --no-buffer --include --request POST \
--header "Authorization: ${INTELLECTIBLE_PROJECT_API_KEY}" \
--header "Content-Type: application/json" \
--data '{"model":"medium","messages":[{"role":"user","content":"Write a short welcome message."}],"stream":true,"max_tokens":2048}' \
"${INTELLECTIBLE_PUBLIC_API_URL}/projects/${INTELLECTIBLE_PROJECT_ID}/chat/completions"

A successful stream uses Content-Type: text/event-stream. curl --no-buffer displays incoming data promptly. The request remains a POST with a JSON body and an authentication header, so use a client that supports those alongside a streamed response body.

Read events, not network chunks​

SSE events are separated by a blank line. Network reads do not necessarily align with event boundaries: one read can contain several events or only part of one.

Use an SSE-capable parser that buffers partial data, handles line endings and multiple data: lines, and ignores comments. Do not call JSON.parse on each raw network chunk.

A simplified text stream might contain:

data: {"object":"chat.completion.chunk","model":"medium","choices":[{"index":0,"delta":{"content":"Welcome"},"finish_reason":null}]}

data: {"object":"chat.completion.chunk","model":"medium","choices":[{"index":0,"delta":{"content":" to your project."},"finish_reason":null}]}

data: {"object":"chat.completion.chunk","model":"medium","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: [DONE]

Actual chunks may include IDs, timestamps, usage, and other fields. Assemble delta.content for each choice in order. A delta may contain only a role, reasoning-related fields, tool calls, or no text; do not assume every event contributes to the final answer.

Check for [DONE] before attempting to parse an event's data as JSON.

Finish reasons and usage​

Inspect the final finish_reason:

  • stop: the model finished its response normally.
  • length: the generation limit was reached; the answer or JSON may be incomplete.
  • tool_calls: the application needs to handle the assembled function calls.

Do not treat an unfamiliar finish reason as unconditional success. Process usage fields when supplied, but do not depend on every chunk—or every stream—containing token counts. Do not sum repeated cumulative totals as though they were independent usage.

A stream should reach its completion marker without an error. A closed connection by itself is not enough to establish success.

Assemble tool calls​

Tool requests arrive under choices[].delta.tool_calls. Each call is identified by its index within the choice. For example:

data: {"choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"id":"call_example","type":"function","function":{"name":"get_order_status","arguments":"{\"orderId\":"}}]},"finish_reason":null}]}

data: {"choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"\"A123\"}"}}]},"finish_reason":null}]}

data: {"choices":[{"index":0,"delta":{},"finish_reason":"tool_calls"}]}

data: [DONE]

Keep a separate buffer for each choice and tool-call index. Retain the call ID and function name when they arrive; concatenate the argument fragments in order. A function name can arrive after the initial call fragment.

Do not parse or execute incomplete arguments. Once the tool calls are complete and the stream has ended successfully, parse the assembled JSON, validate it, check authorisation, and run the allowed functions. Continue the conversation as described in Function tools.

For structured output, the same rule applies to text: collect the complete content before parsing and validating its JSON.

Handle failures​

Before the stream opens, a failed request can return an ordinary HTTP error and a non-SSE body. Check the status and content type first.

After HTTP 200 has started, a failure can arrive as:

event: error
data: {"error":"chat completion failed"}

Treat an error event as failure even if a [DONE] marker follows. A connection ending without [DONE] is an incomplete response unless your client has another explicit failure signal. Do not present partial text as a confirmed complete answer or execute partial tool calls.

There is no documented resume cursor for these streams. Retrying starts another generation and can produce different output. A client disconnect does not guarantee cancellation of server-side work or prevent usage. Set a deadline, avoid unbounded retries, and never repeat already-executed tool actions blindly.

See Errors, limits and retries.