Streaming

By default, a chat completion returns the whole reply when the model is done. With streaming, the reply arrives in pieces while the model writes it, so your app can show the start of a long reply right away.

Tuna streams with server-sent events (SSE), in the same format as the OpenAI API. The OpenAI SDKs and the Vercel AI SDK read it for you.

Turn On Streaming

Set stream to true:

const stream = await client.chat.completions.create({
  model: "tuna-us/example-lab/example-model-id",
  messages: [{
    role: "user",
    content: "Is the tuna one of the fastest sustained swimming fish species? And does it have a streamlined body?",
  }],
  stream: true,
});
for await (const chunk of stream) {
  process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
}

stream = client.chat.completions.create(
    model="tuna-us/example-lab/example-model-id",
    messages=[{
        "role": "user",
        "content": "Is the tuna one of the fastest sustained swimming fish species? And does it have a streamlined body?",
    }],
    stream=True,
)
for chunk in stream:
    if chunk.choices:
        print(chunk.choices[0].delta.content or "", end="")

curl https://api-us.tuna.ai/v1/chat/completions \
  -H "Authorization: Bearer tuna_yourApiKeyHereWithTunaUnderscore" \
  -H "Content-Type: application/json" \
  -N \
  -d '{"model": "tuna-us/example-lab/example-model-id", "stream": true, "messages": [{"role": "user", "content": "Is the tuna one of the fastest sustained swimming fish species? And does it have a streamlined body?"}]}'

The Event Stream

The response has the content type text/event-stream. Each event is a data: line with one JSON chunk, followed by a blank line. The last event is data: [DONE].

data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1790123456,"model":"tuna-us/example-lab/example-model-id","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1790123456,"model":"tuna-us/example-lab/example-model-id","choices":[{"index":0,"delta":{"content":"Yes. Tuna are among "},"finish_reason":null}]}

data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1790123456,"model":"tuna-us/example-lab/example-model-id","choices":[{"index":0,"delta":{"content":"the fastest sustained swimmers."},"finish_reason":null}]}

data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1790123456,"model":"tuna-us/example-lab/example-model-id","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: [DONE]

Each chunk has the same shape as a normal response, except that choices[0].delta holds only the new part of the reply:

The first piece of text is sent as soon as the model writes it. After that, text is sent in small batches instead of one event per token.

Token Usage

A stream doesn't include token usage unless you ask for it. Set stream_options:

{
  "model": "tuna-us/example-lab/example-model-id",
  "messages": [
    {"role": "user", "content": "Is the tuna one of the fastest sustained swimming fish species? And does it have a streamlined body?"}
  ],
  "stream": true,
  "stream_options": {"include_usage": true}
}

Then one more chunk arrives just before data: [DONE]. Its choices list is empty, and its usage field has the token counts for the whole reply, in the same format as a normal response. Code that reads choices[0] has to skip this chunk, like the Python example above does.

Errors During a Stream

If a request fails before the stream starts, you get a normal HTTP error. See Errors.

Once the stream has started, the HTTP status 200 has already been sent. So an error arrives as a last chunk instead. It has an error field with the same code and message as an error response, and a finish_reason of error. The stream ends after it.

{
  "id": "chatcmpl-123",
  "object": "chat.completion.chunk",
  "created": 1790123456,
  "model": "tuna-us/example-lab/example-model-id",
  "error": {
    "code": 402,
    "message": "The account linked with the API key has reached its spending limit/run out of credits."
  },
  "choices": [{"index": 0, "delta": {"content": ""}, "finish_reason": "error"}]
}

For example, if your organization's balance runs out partway through a reply, the stream ends with a 402 error chunk like this one. The tokens generated before the error are charged. See Billing.

The OpenAI SDKs raise an error when they read an error chunk. If you parse the stream yourself, check every chunk for an error field.

Stream Cancellation: Stop a Stream Early

To stop a reply partway through, close the connection. For example, do this when a user presses a stop button, or when the reply already has what you need. Tuna stops generating, and you are charged for the tokens generated until then.

for await (const chunk of stream) {
  if (shouldStop()) {
    stream.controller.abort();
    break;
  }
}

for chunk in stream:
    if should_stop():
        stream.close()
        break

Parse the Stream Yourself

Without an SDK, read the response body line by line:

• A line that starts with data: holds one event. The rest of the line is a JSON chunk, or [DONE] at the end.
• A line that starts with : is a comment. Comments can be sent to keep an idle connection open. Ignore them.
• A blank line ends an event. Ignore any other lines.
• A network read can end in the middle of a line. Keep the unfinished part and join it with the next read.

const response = await fetch("https://api-us.tuna.ai/v1/chat/completions", {
  method: "POST",
  headers: {
    "Authorization": "Bearer tuna_yourApiKeyHereWithTunaUnderscore",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    model: "tuna-us/example-lab/example-model-id",
    stream: true,
    messages: [{
      role: "user",
      content: "Is the tuna one of the fastest sustained swimming fish species? And does it have a streamlined body?",
    }],
  }),
});
if (!response.ok) throw new Error((await response.json()).error.message);

const decoder = new TextDecoder();
let buffer = "";
for await (const bytes of response.body) {
  buffer += decoder.decode(bytes, { stream: true });
  const lines = buffer.split(/\r?\n/);
  buffer = lines.pop();
  for (const line of lines) {
    if (!line.startsWith("data:")) continue;
    const data = line.slice(5).trim();
    if (data === "[DONE]") break;
    const chunk = JSON.parse(data);
    if (chunk.error) throw new Error(chunk.error.message);
    process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
  }
}

A parser library such as eventsource-parser handles these rules for you.