Retries

Some errors are temporary. The service is busy, or a model server had a problem. The same request usually works if you wait a little and send it again. Other errors happen again until something is fixed, so it is better to fix the cause than to retry.

Which Errors to Retry

See Errors for what each status means.

Retry-After

A 429 or 503 response can include a Retry-After header. It is the number of seconds to wait before you try again. When it is there, wait at least that long.

HTTP/1.1 429 Too Many Requests
Retry-After: 30

Exponential Backoff

When there is no Retry-After, wait longer after each failed attempt: 1 second, then 2, then 4, and so on. Add a small random amount to each wait, called jitter, so that many clients that failed at the same moment don't all retry at the same moment too. Stop after a few attempts, such as 3 to 5, and give up once the total time is more than your app can wait.

import random
import time

RETRY_STATUSES = {408, 429, 500, 502, 503}

def post_with_retries(send, max_attempts=5):
    for attempt in range(max_attempts):
        response = send()
        if response.status_code not in RETRY_STATUSES or attempt == max_attempts - 1:
            return response
        retry_after = response.headers.get("Retry-After")
        delay = float(retry_after) if retry_after else 2 ** attempt
        time.sleep(delay + random.random())

The SDKs Retry for You

The OpenAI SDKs retry connection errors and statuses 408, 409, 429, and 500 or higher. By default they retry twice, with exponential backoff, and they follow Retry-After. The Vercel AI SDK also retries twice by default. You can change the number of retries:

const client = new OpenAI({
  baseURL: "https://api-us.tuna.ai/v1",
  apiKey: "tuna_yourApiKeyHereWithTunaUnderscore",
  maxRetries: 5,
});

client = OpenAI(
    base_url="https://api-us.tuna.ai/v1",
    api_key="tuna_yourApiKeyHereWithTunaUnderscore",
    max_retries=5,
)

const { text } = await generateText({
  model: tuna("tuna-us/example-lab/example-model-id"),
  prompt,
  maxRetries: 5,
});

Set the number to 0 to turn retries off and handle them yourself.

Timeouts and Dropped Connections

A timeout or a dropped connection doesn't tell you whether Tuna ran the request. It may have finished, and been charged, before the reply was lost. Retrying runs the request again and is charged again.

For short requests this rarely matters. For long ones:

• Use streaming. Text arrives as it is written, so a dropped connection leaves you with the part you already received. Waiting for a long reply without streaming can also hit your HTTP client's timeout. The OpenAI SDKs time out after 10 minutes by default.
• Don't repeat actions. If your app already acted on part of a reply, like running a tool call, make sure a retry doesn't run it twice.

Errors During a Stream

A stream can fail after it starts, even though the HTTP status was 200. The error arrives in the stream itself. See Errors during a stream.