> This is a page from the Tuna docs. The docs index is at https://tuna.ai/llms.txt, and all pages in one file are at https://tuna.ai/llms-full.txt. Add `.md` to any docs URL to get its Markdown.
>
> Tuna's serverless API is OpenAI compatible. Use an OpenAI SDK with the base URL `https://api-us.tuna.ai/v1` and a Tuna API key.
---
# Introduction to Tuna
Tuna is an AI inference provider. We serve open-weight AI models and make them accessible to anyone who wants to use them. Our focus is on Ultrafast inference, which is 5-10x the speed at 6x the cost compared to the competition on OpenRouter. This is a premium product that shines on use cases where the extremely fast and accurate AI is needed.
## What's Special About Tuna?
- **Quality-first commitment.** Many optimization techniques make AI dumber. Tuna's optimizations preserve intelligence while improving speed. Tuna doesn't quantize models to lower precision.
- **Ultrafast, powered by Bluefin.** Bluefin is a self-improving autoresearch AI system that maximizes per-user speed. Tuna pairs it with traditional engineering to speed up the entire stack.
- **Heterogeneous compute.** Bluefin operates on diverse accelerators, so Tuna can pair models with the fastest hardware. Tuna is building a heterogeneous cloud to further increase speeds.
- **US routing & ZDR.** Everyone should have the freedom to access powerful AI. All requests are processed in the United States. Zero data retention is the default for all users.
Make a first request with curl, Python, or JavaScript.
Create an API key and send it with each request.
See the endpoints, model ids, and what the API can do.
Use the official OpenAI SDKs for Python and JavaScript.
## How Tuna Works
Tuna serves inference through an API. Tuna hosts and runs the models, and your code sends requests to the API with an [API key](https://tuna.ai/docs/authentication.md) and the ID of a model. Each request is charged per token, at the [rates of the model](https://tuna.ai/docs/models.md) you called, from your organization's balance of prepaid credits. See [Billing](https://tuna.ai/docs/account/billing.md).
Tuna is self-serve. Create an [organization](https://tuna.ai/docs/account/organizations.md), add prepaid credit, create a project with an API key inside it, and start sending requests. Pay-as-you-go and self-serve Team organizations cost $0/month. Both include Ultrafast inference, Zero Data Retention (ZDR), US regional routing, [dynamic rate limits](https://tuna.ai/docs/api/rate-limits.md), and budgets and spend control. A pay-as-you-go organization can expand to a Team at any time, add up to 20 seats, and use per-model whitelisting. Companies that need invoicing options, SSO/SAML, higher rate limits, or custom agreements can [contact us](https://tuna.ai/company#contact) about an [enterprise plan](https://tuna.ai/docs/account/enterprise.md).
New users can get a $10 promotional credit after entering valid billing details and a valid payment method. No purchase is required. You will not be charged unless you choose to purchase credits or enable automatic top-ups yourself. Tuna has systems to limit one signup bonus per user. The promotional credit amount is subject to change any time.
Source: https://tuna.ai/docs.md
---
# Models & Pricing
| | Model | Input types | Input /M | Cache read /M | Output /M | SLA speed | Parameter size | Intelligence |
| --- | --- | --- | ---: | ---: | ---: | ---: | ---: | ---: |
| | [Kimi K3 Ultrafast](#kimi-k3-ultrafast) | | $18.00 | $1.80 | $90.00 | >250 tok/s | 2.8 trillion | |
| | [DeepSeek V4.1 Flash Ultrafast](#deepseek-v4.1-flash-ultrafast) | | $1.80 | $0.18 | $7.20 | >350 tok/s | 552 billion | |
| | [GLM-5.3 Flash Ultrafast](#glm-5.3-flash-ultrafast) | | $0.90 | $0.18 | $3.00 | >450 tok/s | 320 billion | |
| Automatically receive up to 50% off if the output speed does not meet SLA. |||||||||
Tuna currently only serves open-weight models, with [US regional routing](https://tuna.ai/docs/us-regional-routing.md) and [Zero Data Retention](https://tuna.ai/docs/privacy-and-zdr.md).
Prices shown are denominated in US dollars, per 1 million tokens. [Precision of billing](https://tuna.ai/docs/account/billing.md) is per 1 single token.
Use the specific model ID to pin your requests to that exact model. If that model is deprecated, requests to its ID will fail. The auto-updating model ID routes to the latest model in the same family, so you can use the newest model without changing your code. Prices can change between model generations in a family. When using an auto-updating model ID, billing is based on whichever model you are routed to.
Cache read is the price for [cached input tokens](https://tuna.ai/docs/api/input-caching.md). Input is implicitly a cache write, and Tuna bills input and cache writes at the same rate. See [Billing](https://tuna.ai/docs/account/billing.md) for how token usage is charged.
Source: https://tuna.ai/docs/models.md
---
# Quickstart
This page takes you from an API key to a first response from a serverless model. See the [Inference Overview](https://tuna.ai/docs/api/overview.md) for what else the API can do.
[Sign up](https://tuna.ai/signup) or [log in](https://tuna.ai/login) to your account. Save your API key, and enter your billing info to get some free credits.
You can have your existing AI agent integrate Tuna for you. Give it web access and the [https://tuna.ai/llms.txt](https://tuna.ai/llms.txt) link so it can read these documentation pages and handle the integration.
```text Agent wrap links
Tuna.ai is a fast LLM inference provider with Zero Data Retention enabled by default. Its API is compatible with OpenAI Chat Completions, at the base URL https://api-us.tuna.ai/v1. Billing is pay-as-you-go from prepaid credits, with optional auto top-up.
Read the documentation for Tuna.ai at https://tuna.ai/llms.txt
More details about models & pricing are available at https://tuna.ai/docs/models or https://tuna.ai/docs/models.md
- Kimi K3 Ultrafast `tuna-us/moonshotai/kimi-latest-ultrafast` `tuna-us/moonshotai/kimi-k3-ultrafast`
- DeepSeek V4.1 Flash Ultrafast `tuna-us/deepseek/deepseek-flash-latest-ultrafast` `tuna-us/deepseek/deepseek-v4.1-flash-ultrafast`
- GLM-5.3 Flash Ultrafast `tuna-us/z-ai/glm-flash-latest-ultrafast` `tuna-us/z-ai/glm-5.3-flash-ultrafast`
Tuna.ai recommends auto-updating model (-latest) ID because it will automatically switch to the latest model within the family. This helps avoid being stuck on a deprecated model. Note that auto-updating may point to a model that responds different and may have different pricing.
The Tuna API key looks like tuna_yourApiKeyHereWithTunaUnderscore.
```
Pick a model from the [models list](https://tuna.ai/docs/api/models.md), then send a [chat completion](https://tuna.ai/docs/api/chat-completions.md). The examples use `tuna-us/example-lab/example-model-id`, which is a placeholder in the same format as a real id. Replace it with the id you picked, and replace `tuna_yourApiKeyHereWithTunaUnderscore` with your key.
```js JavaScript {5,9}
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api-us.tuna.ai/v1",
apiKey: "tuna_yourApiKeyHereWithTunaUnderscore",
});
const response = await client.chat.completions.create({
model: "tuna-us/example-lab/example-model-id",
messages: [{
role: "user",
content: "Is the tuna one of the fastest sustained swimming fish species? And does it have a streamlined body?",
}],
});
console.log(response.choices[0].message.content);
```
```bash curl {2,5}
curl https://api-us.tuna.ai/v1/chat/completions \
-H "Authorization: Bearer tuna_yourApiKeyHereWithTunaUnderscore" \
-H "Content-Type: application/json" \
-d '{
"model": "tuna-us/example-lab/example-model-id",
"messages": [
{"role": "user", "content": "Is the tuna one of the fastest sustained swimming fish species? And does it have a streamlined body?"}
]
}'
```
```python Python {5,9}
from openai import OpenAI
client = OpenAI(
base_url="https://api-us.tuna.ai/v1",
api_key="tuna_yourApiKeyHereWithTunaUnderscore",
)
response = client.chat.completions.create(
model="tuna-us/example-lab/example-model-id",
messages=[{
"role": "user",
"content": "Is the tuna one of the fastest sustained swimming fish species? And does it have a streamlined body?",
}],
)
print(response.choices[0].message.content)
```
The Python and JavaScript examples use the official [OpenAI SDKs](https://tuna.ai/docs/sdks.md). Install them with `pip install openai` or `npm install openai`.
To show text as the model writes it, turn on streaming. See [Streaming](https://tuna.ai/docs/api/streaming.md).
Source: https://tuna.ai/docs/quickstart.md
---
# Authentication
The Tuna API requires an API key, except when listing models.
## API Keys
API keys belong to a project, and projects belong to an organization. A request made with a key is charged to that key's organization. See [Organizations and projects](https://tuna.ai/docs/account/organizations.md).
Create keys in a project on the Tuna dashboard. API keys start with `tuna_`. Treat each key like a password:
- Keep keys on your server. Code in a web page or a mobile app runs on your users' devices, so anyone can read a key that is in it.
- Follow general security best practices for storing and handling API keys.
- Use a separate key for each app and each environment, like production and staging, so you can replace one without touching the others.
## Sending the Key
Send the key in the `Authorization` header as a Bearer token.
```http {3}
POST /v1/chat/completions HTTP/1.1
Host: api-us.tuna.ai
Authorization: Bearer tuna_yourApiKeyHereWithTunaUnderscore
Content-Type: application/json
```
The [OpenAI SDKs](https://tuna.ai/docs/sdks.md) do this for you when you pass `api_key` in Python or `apiKey` in JavaScript.
## Errors
| Status | Message | What to do |
| --- | --- | --- |
| 401 | You must provide an API key. | Add the `Authorization` header. |
| 401 | The API key you provided is invalid. | Check the key for typos, or create a new one. |
See [Errors](https://tuna.ai/docs/api/errors.md) for the other errors the API can return.
Anyone with your key can spend your organization's [balance](https://tuna.ai/docs/account/billing.md). If a key is exposed, create a new key, move your apps to it, and then delete the old key. Requests made with a deleted key fail with status 401.
Source: https://tuna.ai/docs/authentication.md
---
# Tuna Docs for Agents
These docs are written for people and for AI agents. There are four ways to hand them to an agent.
## Markdown Pages
Add `.md` to the end of any docs URL to get the page as Markdown. For example, see the [Quickstart as Markdown](https://tuna.ai/docs/quickstart.md). Links inside these pages point to other `.md` pages, so an agent can follow them.
```bash
curl https://tuna.ai/docs/quickstart.md
```
## Copy Page
The **Copy page** button next to each page's title copies the same Markdown, ready to paste into a chat.
## llms.txt
[https://tuna.ai/llms.txt](https://tuna.ai/llms.txt) is an index of the docs, with a short description and Markdown link for each page. It follows the [llms.txt](https://llmstxt.org) format.
## llms-full.txt
[llms-full.txt](https://tuna.ai/llms-full.txt) contains the complete docs in one file. Use it when an agent should read all of the docs at once. It is much longer than a single page, so it uses more of the agent's context.
Source: https://tuna.ai/docs/ai-agents.md
---
# Privacy and ZDR
Inference requests have Zero Data Retention (ZDR) by default, including [chat completions](https://tuna.ai/docs/api/chat-completions.md).
ZDR is the default for Tuna's products. The exceptions are products that need to store data to work. Batch completions (not yet implemented), for example, need to keep requests until the batch runs. Those exceptions will be described on the product's docs page.
## Zero Data Retention (ZDR) For All
All users are eligible, without a separate request or an [enterprise plan](https://tuna.ai/docs/account/enterprise.md).
Tuna does not store the content of your requests. Prompts and model replies are held only in volatile memory while they are processed, and the text isn't used to train models.
Tuna only keeps what it needs to [bill you](https://tuna.ai/docs/account/billing.md#usage-tracking-accounting): which [organization](https://tuna.ai/docs/account/organizations.md) and key made a request, which model it used, when it ran, and its token counts. None of that includes the model inputs or outputs (text, image, video, audio, etc.).
### KV Cache
When a model reads a prompt, it builds a KV cache: its internal state after reading that text. If the next request starts with the same text, such as the same system prompt, the model can reuse the cache instead of reading it all again. This makes responses faster and is why [cached input](https://tuna.ai/docs/api/input-caching.md) costs less.
The KV cache is held in volatile memory. Depending on how a model is configured, it may use [LMCache](https://lmcache.ai), an open-source KV cache layer for vLLM, and it may use LMCache to offload the KV cache to SSD storage.
When a model uses LMCache with KV cache SSD offload, Tuna always configures LMCache's [encryption](https://blog.lmcache.ai/en/2026/08/19/encrypt-kv-cache-at-rest-lmcache/) this way:
- Cache data is encrypted with AES-256 before it is written to the SSD.
- The key is generated at random when the cache server starts and is kept only in memory. It is never written to the SSD or to any other storage.
- The SSD's filesystem is also encrypted at rest.
As a result, the SSD alone cannot be used to read the cache, and the cache cannot be read after a restart. According to expert cryptographers, AES-256 is quantum safe for use in a post-quantum world. See [NIST's assessment of AES and quantum computing](https://csrc.nist.gov/projects/post-quantum-cryptography/faqs).
For more on why this approach to KV caching is zero data retention (ZDR), see OpenRouter's post, [Is Implicit Caching Prompt Retention?](https://openrouter.ai/blog/insights/is-implicit-caching-prompt-retention/).
## US Regional Routing
See [US Regional Routing](https://tuna.ai/docs/us-regional-routing.md) for the `tuna-us/` location guarantee for GPU inference and KV caches. The page also covers edge processing and internet transit.
## Privacy Policy and Terms of Service
These docs are for informational purposes only. Tuna's [Privacy Policy](https://tuna.ai/privacy-policy) and [Terms of Service](https://tuna.ai/terms-of-service) take precedence over these docs if there is a conflict.
Source: https://tuna.ai/docs/privacy-and-zdr.md
---
# US Regional Routing
Regional routing refers to the physical or geographic location where data is processed. This page explains Tuna's approach to US regional routing.
## US Regional Routing at No Additional Charge
US regional routing is included in Tuna's [model prices](https://tuna.ai/docs/account/billing.md#pricing). It is available to all users without a separate request or an [enterprise plan](https://tuna.ai/docs/account/enterprise.md).
Some providers charge a premium for US-only or regional inference processing. This survey of some providers compares their published prices at the time of writing in 2026:
| Provider | Published regional processing price |
| --- | --- |
| Tuna | No additional charge (0%) US regional routing is the default for all models.[source] |
| OpenAI | +10% for regional processing on eligible models released on or after March 5, 2026. Requires contacting sales.[source] |
| Anthropic | +10% for US-only inference (`inference_geo: "us"`) on Claude 4.6 and later.[source] |
| xAI (Grok) | +10% for US-only inference through `us.api.x.ai` on Grok 4.7 and Grok 4.6.[source] |
| Amazon Bedrock | +10% for regional inference (including US) on Claude 4.5 and later.[source] |
| Google Cloud | +10% for US multi-region inference on Gemini 3.8 Flash and Claude 4.5 and later.[source][source] |
| Fireworks | +10% to +50% for US-only models (10% through OpenRouter, 50% directly, with exceptions).[source] |
## Why US Regional Routing?
For some customers (such as financial institutions, healthcare organizations, and enterprises handling confidential information), US regional routing is necessary. Tuna was built for those customers from the start.
Tuna uses GPU servers in US data centers for AI inference. Serving inference from the United States is part of the company's long-term business strategy to maintain security and reduce unforeseen geopolitical risks. US hosting costs more, but the security and geopolitical benefits are worth the additional cost.
Tuna Inference, Inc. is a US company registered in the state of Delaware, with majority natural person ownership held by US citizens.
US regional routing and US ownership do not override applicable laws. The laws of your own country may still apply to your use of Tuna and to the data you send. Tuna serves non-US customers and may also be subject to non-US laws when applicable.
## The `tuna-us/` Model Prefix Guarantee
For a model id beginning with `tuna-us/`, GPU inference is limited to data centers on land physically located in the United States. Tuna uses US ground-based data centers and currently has no plans to use satellite data centers.
The guarantee covers inference content, including prompts, uploaded images, model replies, and the associated KV caches. It applies to HTTPS termination, Tuna's inference backend services, and GPU inference. The inference backend receives customer requests, routes them to GPU servers, and returns the responses.
| Data | Location |
| --- | --- |
| Inference HTTPS termination and backend processing | Google Cloud, US region |
| Prompts, uploaded images, and replies during GPU inference | US data centers |
| Temporary KV caches, including encrypted SSD offload | US data centers |
| Prompts and replies retained after processing | Not retained ([ZDR](https://tuna.ai/docs/privacy-and-zdr.md)) |
| Management API and account, usage, and billing records | Not covered by US regional routing (see below) |
### Using US Regional Routing
Send inference requests to `api-us.tuna.ai` with a `tuna-us/` model id, such as `tuna-us/moonshotai/kimi-k3-ultrafast`. No project-region setting or sales approval is needed. All of Tuna's current model endpoints use this prefix.
The code examples in these docs use `tuna-us/example-lab/example-model-id` as a placeholder. Choose an available id from [List Model Endpoints](https://tuna.ai/docs/api/models.md) when making a request.
## Data Residency
Under [Zero Data Retention (ZDR)](https://tuna.ai/docs/privacy-and-zdr.md), Tuna does not retain your prompts or model replies. Temporary KV caches reside on GPU servers in US data centers, including any encrypted SSD offload. See [KV Cache](https://tuna.ai/docs/privacy-and-zdr.md#kv-cache) for how these caches are handled.
## Management API and Account Data
Tuna's management API handles user accounts and routes through the edge without US regional routing. Its HTTPS connections may terminate outside the United States. This API does not carry inference prompts. Inference requests use the separate `api-us.tuna.ai` endpoint described above.
Account, usage, and billing records are outside the inference regional routing guarantee. Tuna uses Stripe for payments, Metronome for usage billing, and [WorkOS](https://workos.com) for organization sign-in. These services have their own data-handling policies and residency arrangements. Usage billing records do not contain prompts or model replies. See [Billing](https://tuna.ai/docs/account/billing.md#usage-tracking-accounting) and [Organizations and projects](https://tuna.ai/docs/account/organizations.md) for details.
## HTTPS Termination
HTTPS connections to `api-us.tuna.ai` terminate on Google Cloud servers in the United States. Requests remain encrypted in transit until they reach that US endpoint, where TLS decryption takes place. Traffic between Tuna's inference backend services and GPU servers is also encrypted.
Requests to `api-us.tuna.ai` do not route through Cloudflare.
## Infrastructure Providers
Tuna runs its inference backend services on Google Cloud in a US region. Tuna runs inference on GPU servers hosted in US data centers. Both the inference backend and GPU processing are covered by the regional routing guarantee.
## When Encrypted Traffic May Travel Outside the US
US regional routing determines where Tuna processes inference content. The internet path between your app and Tuna can still cross borders, even when both ends of a connection are in the United States. HTTPS encryption prevents network operators in those countries from reading your prompts and replies. Requests are decrypted only when they reach the Google Cloud endpoint in the United States.
Illustrative examples include (but are not limited to):
- **A request from outside the United States.** The request travels over HTTPS from the customer's country to Google Cloud servers in the United States, where the connection terminates and the request is decrypted.
- **A connection from Alaska.** Depending on the internet provider's route, traffic between Alaska and the contiguous United States may pass through Canada.
- **Satellite internet connection (such as through airplane Wi-Fi or Starlink).** Packets travel to a satellite and back to a ground station. A connection can start and end in the United States while part of its path is in orbit.
- **An image sent as a URL.** When an inference request instructs Tuna to download publicly available content from a URL, such as an [image URL](https://tuna.ai/docs/api/image-input.md), Tuna retrieves that content from the server hosting it. The server hosting that publicly available content may be outside the United States.
## Privacy Policy and Terms of Service
These docs are for informational purposes only. Tuna's [Privacy Policy](https://tuna.ai/privacy-policy) and [Terms of Service](https://tuna.ai/terms-of-service) take precedence over these docs if there is a conflict.
Source: https://tuna.ai/docs/us-regional-routing.md
---
# Inference Overview
Tuna's Inference API provides access to models hosted and run by Tuna. This type of service is commonly called serverless inference: you send requests without having to deploy models or manage servers. Usage is billed per token.
You can use the [OpenAI SDKs](https://tuna.ai/docs/sdks.md) with Tuna's supported endpoints. See [OpenAI compatibility](https://tuna.ai/docs/api/openai-compatibility.md) for client setup and supported features.
## Base URL
API requests use this base URL:
```text
https://api-us.tuna.ai/v1
```
Requests require an [API key](https://tuna.ai/docs/authentication.md) in the `Authorization` header, except when listing models.
## Endpoints
| Endpoint | What it does |
| --- | --- |
| [`POST /v1/chat/completions`](https://tuna.ai/docs/api/chat-completions.md) | Sends a conversation to a model and returns the model's reply. |
| [`GET /v1/models`](https://tuna.ai/docs/api/models.md) | Lists the models Tuna serves, with their ids and prices. |
## Model IDs
Tuna's model ids start with `tuna-us/`, like `tuna-us/moonshotai/kimi-k3-ultrafast`. The prefix provides a [regional routing guarantee](https://tuna.ai/docs/us-regional-routing.md) for GPU inference and KV caches in US data centers. Code examples in these docs use the placeholder `tuna-us/example-lab/example-model-id` in place of a real id. See [List Model Endpoints](https://tuna.ai/docs/api/models.md).
## What the API Can Do
| Page | What it covers |
| --- | --- |
| [Streaming](https://tuna.ai/docs/api/streaming.md) | Get the reply in pieces while the model writes it. |
| [Tool calling](https://tuna.ai/docs/api/tool-calling.md) | Let the model ask your code to run functions. |
| [Reasoning effort](https://tuna.ai/docs/api/reasoning.md) | Choose a reasoning effort level to balance response quality, latency, and token usage. |
| [Image input](https://tuna.ai/docs/api/image-input.md) | Send images to models that support image input. |
| [Input caching](https://tuna.ai/docs/api/input-caching.md) | Pay the lower cached input price for a repeated prompt start. |
| [Advanced model control](https://tuna.ai/docs/api/model-control.md) | Set `max_tokens`, `temperature`, `top_p`, `stop`, and the penalties. |
| [Structured outputs](https://tuna.ai/docs/api/structured-outputs.md) | Make replies match a JSON Schema. Coming soon. |
| [Assistant prefill](https://tuna.ai/docs/api/assistant-prefill.md) | Write the start of the model's reply. Coming soon. |
## Pricing
Serverless requests are billed per token, at the model's input, cached input, and output prices. Charges are paid from your organization's balance of credits. See [Billing](https://tuna.ai/docs/account/billing.md).
## Privacy
The Inference API has Zero Data Retention (ZDR). Tuna does not store prompts or replies. See [Privacy and ZDR](https://tuna.ai/docs/privacy-and-zdr.md).
## Errors and Limits
All the API keys in an [organization](https://tuna.ai/docs/account/organizations.md) share one [rate limit](https://tuna.ai/docs/api/rate-limits.md). A failed request returns an HTTP status and an error message. See [Errors](https://tuna.ai/docs/api/errors.md) for what each status means, and [Retries](https://tuna.ai/docs/api/retries.md) for which ones to retry.
Source: https://tuna.ai/docs/api/overview.md
---
# Recommended SDKs
Tuna supports [OpenAI-compatible endpoints and features](https://tuna.ai/docs/api/openai-compatibility.md), and is tested with these two SDKs:
| SDK | npm packages |
| --- | --- |
| [OpenAI JavaScript SDK](https://www.npmjs.com/package/openai) | `openai` |
| [Vercel AI SDK](https://ai-sdk.dev/providers/openai-compatible-providers) | `ai` and `@ai-sdk/openai-compatible` |
Other OpenAI-compatible SDKs can be used with Tuna when they support a custom base URL and the endpoints your application needs. The Python examples in these docs use the official [OpenAI Python SDK](https://pypi.org/project/openai/).
## OpenAI JavaScript SDK
Install it with `npm install openai`, then point it at Tuna:
```ts {4-5}
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api-us.tuna.ai/v1",
apiKey: "tuna_yourApiKeyHereWithTunaUnderscore",
});
const response = await client.chat.completions.create({
model: "tuna-us/example-lab/example-model-id",
messages: [{
role: "user",
content: "Is the tuna one of the fastest sustained swimming fish species? And does it have a streamlined body?",
}],
});
console.log(response.choices[0].message.content);
```
## Vercel AI SDK
Install it with `npm install ai @ai-sdk/openai-compatible`, then create a Tuna provider:
```ts {8}
import { createOpenAICompatible } from "@ai-sdk/openai-compatible";
import { generateText } from "ai";
const tuna = createOpenAICompatible({
name: "tuna",
baseURL: "https://api-us.tuna.ai/v1",
apiKey: "tuna_yourApiKeyHereWithTunaUnderscore",
includeUsage: true,
});
const { text } = await generateText({
model: tuna("tuna-us/example-lab/example-model-id"),
prompt: "Is the tuna one of the fastest sustained swimming fish species? And does it have a streamlined body?",
});
console.log(text);
```
`includeUsage: true` makes streamed replies report their token usage too. See [Streaming](https://tuna.ai/docs/api/streaming.md#token-usage).
## Without an SDK
You don't need an SDK. The API is plain HTTPS with JSON bodies, so any HTTP client works:
```js
const response = await fetch("https://api-us.tuna.ai/v1/chat/completions", {
method: "POST",
headers: {
"Authorization": "Bearer tuna_yourApiKeyHereWithTunaUnderscore",
"Content-Type": "application/json",
},
body: JSON.stringify({
model: "tuna-us/example-lab/example-model-id",
messages: [{
role: "user",
content: "Is the tuna one of the fastest sustained swimming fish species? And does it have a streamlined body?",
}],
}),
});
const data = await response.json();
if (!response.ok) throw new Error(data.error.message);
console.log(data.choices[0].message.content);
```
SDKs retry failed requests for you. Without one, add your own retries. See [Retries](https://tuna.ai/docs/api/retries.md).
## Compress Requests with gzip
Long conversations, documents, and [images](https://tuna.ai/docs/api/image-input.md) make request bodies large. You can compress the body with gzip and send the `Content-Encoding: gzip` header. JSON usually shrinks to a small part of its size, so your app uploads less and large requests reach Tuna sooner.
```js JavaScript {8,10}
import { gzipSync } from "node:zlib";
const response = await fetch("https://api-us.tuna.ai/v1/chat/completions", {
method: "POST",
headers: {
"Authorization": "Bearer tuna_yourApiKeyHereWithTunaUnderscore",
"Content-Type": "application/json",
"Content-Encoding": "gzip",
},
body: gzipSync(JSON.stringify({ model: "tuna-us/example-lab/example-model-id", messages })),
});
```
```bash curl {4-5}
gzip -c request.json | curl https://api-us.tuna.ai/v1/chat/completions \
-H "Authorization: Bearer tuna_yourApiKeyHereWithTunaUnderscore" \
-H "Content-Type: application/json" \
-H "Content-Encoding: gzip" \
--data-binary @-
```
```python Python {11,13}
import gzip
import json
import requests
response = requests.post(
"https://api-us.tuna.ai/v1/chat/completions",
headers={
"Authorization": "Bearer tuna_yourApiKeyHereWithTunaUnderscore",
"Content-Type": "application/json",
"Content-Encoding": "gzip",
},
data=gzip.compress(json.dumps({
"model": "tuna-us/example-lab/example-model-id",
"messages": messages,
}).encode()),
)
```
Compressing takes a little CPU time, so it helps most with large bodies. Small requests can skip it.
Source: https://tuna.ai/docs/sdks.md
---
# Chat Completions
`POST https://api-us.tuna.ai/v1/chat/completions` sends a conversation to a [serverless](https://tuna.ai/docs/api/overview.md) model and returns the model's next message. It needs an [API key](https://tuna.ai/docs/authentication.md). A request that fails returns an [error](https://tuna.ai/docs/api/errors.md).
## Request
The request body is JSON. These are the parameters most apps use:
| Parameter | Type | Description |
| --- | --- | --- |
| `model` | string | Required. The id of a model from [`GET /v1/models`](https://tuna.ai/docs/api/models.md). |
| `messages` | array | Required. The conversation so far. Each message has a `role` (`system`, `user`, or `assistant`) and `content`. |
| [`max_tokens`](https://tuna.ai/docs/api/model-control.md#maxtokens) | integer | The most tokens the reply can have. Tuna lowers it to the model's own limit if you ask for more. |
| [`temperature`](https://tuna.ai/docs/api/model-control.md#temperature) | number | How random the reply is. Lower values give more predictable text. |
| [`top_p`](https://tuna.ai/docs/api/model-control.md#topp) | number | Samples only from the most likely tokens whose probabilities add up to this value. |
| [`stop`](https://tuna.ai/docs/api/model-control.md#stop) | string or array | Text that ends the reply when the model writes it. |
| [`stream`](https://tuna.ai/docs/api/streaming.md) | boolean | Send the reply in pieces as it is generated. |
| [`stream_options`](https://tuna.ai/docs/api/streaming.md#token-usage) | object | Set `{"include_usage": true}` to get token usage at the end of a stream. |
| [`reasoning_effort`](https://tuna.ai/docs/api/reasoning.md) | string | The model's reasoning effort level. Supported levels depend on the model. |
| [`tools`](https://tuna.ai/docs/api/tool-calling.md) | array | Functions the model can ask your code to run. |
| [`tool_choice`](https://tuna.ai/docs/api/tool-calling.md#choosing-when-tools-are-used) | string or object | Whether the model must, may, or must not call a tool. |
More settings, like `presence_penalty` and `frequency_penalty`, are in [Advanced model control](https://tuna.ai/docs/api/model-control.md). A message's `content` can also be a list of text and image parts. See [Image input](https://tuna.ai/docs/api/image-input.md).
```json title="Request body"
{
"model": "tuna-us/example-lab/example-model-id",
"messages": [
{"role": "system", "content": "You answer in one short sentence."},
{"role": "user", "content": "Is the tuna one of the fastest sustained swimming fish species? And does it have a streamlined body?"}
],
"max_tokens": 200
}
```
## Response
The response has the same shape as [OpenAI's](https://tuna.ai/docs/api/openai-compatibility.md). The reply is in `choices[0].message`, and `usage` counts the tokens you are [charged for](https://tuna.ai/docs/account/billing.md).
```json title="Response" {9-11}
{
"id": "chatcmpl-123",
"object": "chat.completion",
"created": 1790000000,
"model": "tuna-us/example-lab/example-model-id",
"choices": [
{
"index": 0,
"message": {"role": "assistant", "content": "Yes. Tuna are among the fastest sustained swimmers, and their bodies are streamlined."},
"finish_reason": "stop"
}
],
"usage": {"prompt_tokens": 38, "completion_tokens": 17, "total_tokens": 55, "prompt_tokens_details": {"cached_tokens": 0}}
}
```
`usage.prompt_tokens_details.cached_tokens` is how many of the input tokens were billed as cached input. See [Input caching](https://tuna.ai/docs/api/input-caching.md).
`finish_reason` says why the reply ended:
| Value | Meaning |
| --- | --- |
| `stop` | The model finished on its own, or wrote one of the `stop` strings. |
| `length` | The reply reached `max_tokens`. |
| `tool_calls` | The model is asking for [tool calls](https://tuna.ai/docs/api/tool-calling.md). |
| `error` | A stream failed after it started. See [Errors during a stream](https://tuna.ai/docs/api/streaming.md#errors-during-a-stream). |
## Streaming
Set `stream` to `true` to get the reply as it is generated. The response is a stream of server-sent events, in the same format as OpenAI's. See [Streaming](https://tuna.ai/docs/api/streaming.md) for the chunk format, token usage, and errors during a stream.
Source: https://tuna.ai/docs/api/chat-completions.md
---
# List Model Endpoints
`GET https://api-us.tuna.ai/v1/models` lists the [serverless](https://tuna.ai/docs/api/overview.md) models currently available on Tuna. It is public, so it works without an [API key](https://tuna.ai/docs/authentication.md).
```bash
curl https://api-us.tuna.ai/v1/models
```
The response is JSON with a `data` list. Each entry describes one model, including the `id` you pass as `model` in a [chat completion](https://tuna.ai/docs/api/chat-completions.md), and its [prices](https://tuna.ai/docs/account/billing.md#pricing).
With the [OpenAI SDKs](https://tuna.ai/docs/sdks.md), use the models list method:
```js JavaScript
for await (const model of client.models.list()) {
console.log(model.id);
}
```
```python Python
for model in client.models.list():
print(model.id)
```
## Model IDs
Tuna's model ids start with `tuna-us/`, followed by the model's maker and name. For example, `tuna-us/moonshotai/kimi-k3-ultrafast`. The `tuna-us/` prefix provides a [regional routing guarantee](https://tuna.ai/docs/us-regional-routing.md) for GPU inference and KV caches in US data centers.
The code examples in these docs use `tuna-us/example-lab/example-model-id`. It is a placeholder in the same format and does not refer to a real model. Replace it with an id from the list.
## Choosing a Model
The list only includes models that are running, so any id in it works in a request. Models differ in speed, quality, the longest reply they allow, and price. Each model has its own [`max_tokens`](https://tuna.ai/docs/api/model-control.md#maxtokens) limit, documented on the [Models & Pricing](https://tuna.ai/docs/models.md) page. Tuna lowers larger requests to that limit.
Model ids can change when a model is added or deprecated. Read the list from the API instead of hard-coding a copy of it.
## Model Deprecation
A model could be deprecated, meaning it is no longer available through the API. A deprecated model is removed from the models list, and requests that use its id fail with status 400. See [Errors](https://tuna.ai/docs/api/errors.md).
If your app depends on one model, check the list regularly and have a second model you can switch to.
Source: https://tuna.ai/docs/api/models.md
---
# Streaming
By default, a [chat completion](https://tuna.ai/docs/api/chat-completions.md) returns the whole reply when the model is done. With streaming, the reply arrives in pieces while the model writes it, so your app can show the start of a long reply right away.
Tuna streams with server-sent events (SSE), in the same format as the OpenAI API. The [OpenAI SDKs and the Vercel AI SDK](https://tuna.ai/docs/sdks.md) read it for you.
## Turn On Streaming
Set `stream` to `true`:
```js JavaScript {7}
const stream = await client.chat.completions.create({
model: "tuna-us/example-lab/example-model-id",
messages: [{
role: "user",
content: "Is the tuna one of the fastest sustained swimming fish species? And does it have a streamlined body?",
}],
stream: true,
});
for await (const chunk of stream) {
process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
}
```
```python Python {7}
stream = client.chat.completions.create(
model="tuna-us/example-lab/example-model-id",
messages=[{
"role": "user",
"content": "Is the tuna one of the fastest sustained swimming fish species? And does it have a streamlined body?",
}],
stream=True,
)
for chunk in stream:
if chunk.choices:
print(chunk.choices[0].delta.content or "", end="")
```
```bash curl {4-5}
curl https://api-us.tuna.ai/v1/chat/completions \
-H "Authorization: Bearer tuna_yourApiKeyHereWithTunaUnderscore" \
-H "Content-Type: application/json" \
-N \
-d '{"model": "tuna-us/example-lab/example-model-id", "stream": true, "messages": [{"role": "user", "content": "Is the tuna one of the fastest sustained swimming fish species? And does it have a streamlined body?"}]}'
```
## The Event Stream
The response has the content type `text/event-stream`. Each event is a `data:` line with one JSON chunk, followed by a blank line. The last event is `data: [DONE]`.
```sse title="Stream"
data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1790123456,"model":"tuna-us/example-lab/example-model-id","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}
data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1790123456,"model":"tuna-us/example-lab/example-model-id","choices":[{"index":0,"delta":{"content":"Yes. Tuna are among "},"finish_reason":null}]}
data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1790123456,"model":"tuna-us/example-lab/example-model-id","choices":[{"index":0,"delta":{"content":"the fastest sustained swimmers."},"finish_reason":null}]}
data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1790123456,"model":"tuna-us/example-lab/example-model-id","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}
data: [DONE]
```
Each chunk has the same shape as a normal response, except that `choices[0].delta` holds only the new part of the reply:
| Field | What it holds |
| --- | --- |
| `id` | Shared by the chunks of one reply. |
| `choices[0].delta.role` | `assistant`. It is only in the first chunk. |
| `choices[0].delta.content` | The next piece of text. Join the pieces in order to get the whole reply. |
| `choices[0].delta.reasoning` | The next piece of the model's thinking, for [reasoning models](https://tuna.ai/docs/api/reasoning.md). |
| `choices[0].delta.tool_calls` | Pieces of a [tool call](https://tuna.ai/docs/api/tool-calling.md#streaming-tool-calls). |
| `choices[0].finish_reason` | `null` until the last chunk. Then it says why the reply ended, such as `stop` or `length`. |
The first piece of text is sent as soon as the model writes it. After that, text is sent in small batches instead of one event per token.
## Token Usage
A stream doesn't include token usage unless you ask for it. Set `stream_options`:
```json {7}
{
"model": "tuna-us/example-lab/example-model-id",
"messages": [
{"role": "user", "content": "Is the tuna one of the fastest sustained swimming fish species? And does it have a streamlined body?"}
],
"stream": true,
"stream_options": {"include_usage": true}
}
```
Then one more chunk arrives just before `data: [DONE]`. Its `choices` list is empty, and its `usage` field has the token counts for the whole reply, in the same format as a [normal response](https://tuna.ai/docs/api/chat-completions.md#response). Code that reads `choices[0]` has to skip this chunk, like the Python example above does.
## Errors During a Stream
If a request fails before the stream starts, you get a normal HTTP error. See [Errors](https://tuna.ai/docs/api/errors.md).
Once the stream has started, the HTTP status `200` has already been sent. So an error arrives as a last chunk instead. It has an `error` field with the same `code` and `message` as an [error response](https://tuna.ai/docs/api/errors.md), and a `finish_reason` of `error`. The stream ends after it.
```json title="Error chunk" {6-10}
{
"id": "chatcmpl-123",
"object": "chat.completion.chunk",
"created": 1790123456,
"model": "tuna-us/example-lab/example-model-id",
"error": {
"code": 402,
"message": "The account linked with the API key has reached its spending limit/run out of credits."
},
"choices": [{"index": 0, "delta": {"content": ""}, "finish_reason": "error"}]
}
```
For example, if your [organization's](https://tuna.ai/docs/account/organizations.md) balance runs out partway through a reply, the stream ends with a 402 error chunk like this one. The tokens generated before the error are charged. See [Billing](https://tuna.ai/docs/account/billing.md#when-the-balance-runs-out).
The OpenAI SDKs raise an error when they read an error chunk. If you parse the stream yourself, check every chunk for an `error` field.
## Stream Cancellation: Stop a Stream Early {#stop-a-stream-early}
To stop a reply partway through, close the connection. For example, do this when a user presses a stop button, or when the reply already has what you need. Tuna stops generating, and you are [charged](https://tuna.ai/docs/account/billing.md) for the tokens generated until then.
```js JavaScript {3}
for await (const chunk of stream) {
if (shouldStop()) {
stream.controller.abort();
break;
}
}
```
```python Python {3}
for chunk in stream:
if should_stop():
stream.close()
break
```
## Parse the Stream Yourself
Without an SDK, read the response body line by line:
- A line that starts with `data:` holds one event. The rest of the line is a JSON chunk, or `[DONE]` at the end.
- A line that starts with `:` is a comment. Comments can be sent to keep an idle connection open. Ignore them.
- A blank line ends an event. Ignore any other lines.
- A network read can end in the middle of a line. Keep the unfinished part and join it with the next read.
```js title="JavaScript"
const response = await fetch("https://api-us.tuna.ai/v1/chat/completions", {
method: "POST",
headers: {
"Authorization": "Bearer tuna_yourApiKeyHereWithTunaUnderscore",
"Content-Type": "application/json",
},
body: JSON.stringify({
model: "tuna-us/example-lab/example-model-id",
stream: true,
messages: [{
role: "user",
content: "Is the tuna one of the fastest sustained swimming fish species? And does it have a streamlined body?",
}],
}),
});
if (!response.ok) throw new Error((await response.json()).error.message);
const decoder = new TextDecoder();
let buffer = "";
for await (const bytes of response.body) {
buffer += decoder.decode(bytes, { stream: true });
const lines = buffer.split(/\r?\n/);
buffer = lines.pop();
for (const line of lines) {
if (!line.startsWith("data:")) continue;
const data = line.slice(5).trim();
if (data === "[DONE]") break;
const chunk = JSON.parse(data);
if (chunk.error) throw new Error(chunk.error.message);
process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
}
}
```
A parser library such as [eventsource-parser](https://github.com/rexxars/eventsource-parser) handles these rules for you.
Source: https://tuna.ai/docs/api/streaming.md
---
# Tool Calling
Tool calling lets a model request that an application run a function. You describe the available functions in the request. The model returns a function name and arguments. The application validates the request, runs the function when appropriate, and sends the result back to the model.
## Compatible with OpenAI Tool Calling
Tuna uses the industry-standard tool calling format from the OpenAI Chat Completions API. The `tools` list, `tool_choice`, the `tool_calls` in replies, and the `tool` messages you send back all have the same fields as OpenAI's. Code, SDKs, and agent frameworks written for OpenAI tool calling work with Tuna after you set Tuna's base URL and a Tuna model ID. See [OpenAI Compatibility](https://tuna.ai/docs/api/openai-compatibility.md) for the rest of the API.
## How It Works
Send a `tools` list with the request. Each tool has a `name`, a `description` of what it does and when to use it, and its `parameters` as a [JSON Schema](https://json-schema.org).
```python {20}
tools = [{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather in a city.",
"parameters": {
"type": "object",
"properties": {
"city": {"type": "string", "description": "The city name, like Boston"},
},
"required": ["city"],
},
},
}]
messages = [{"role": "user", "content": "What's the weather in Boston?"}]
response = client.chat.completions.create(
model="tuna-us/example-lab/example-model-id",
messages=messages,
tools=tools,
)
```
When the model wants a tool, the reply has `tool_calls` instead of text, and `finish_reason` is `tool_calls`. The `arguments` are a JSON string.
```json title="Response message" {8}
{
"role": "assistant",
"content": null,
"tool_calls": [
{
"id": "call_abc123",
"type": "function",
"function": {"name": "get_weather", "arguments": "{\"city\": \"Boston\"}"}
}
]
}
```
Add the model's message to the conversation. Then run each tool and add one `tool` message per call, with the call's `id` in `tool_call_id`.
```python {4,8}
import json
message = response.choices[0].message
messages.append(message)
for call in message.tool_calls or []:
args = json.loads(call.function.arguments)
result = get_weather(**args) # Your own function
messages.append({"role": "tool", "tool_call_id": call.id, "content": json.dumps(result)})
```
Send the conversation again, with the same `tools`. The model reads the results and answers, or asks for more tool calls. An agent repeats steps 2 to 4 until `finish_reason` is `stop`.
```python {4}
response = client.chat.completions.create(
model="tuna-us/example-lab/example-model-id",
messages=messages,
tools=tools,
)
print(response.choices[0].message.content)
```
## Choosing When Tools Are Used
`tool_choice` controls whether the model calls a tool:
| Value | What happens |
| --- | --- |
| `"auto"` | The model decides. This is the default when you send tools. |
| `"none"` | The model answers with text and calls no tools. |
| `"required"` | The model must call at least one tool. |
| `{"type": "function", "function": {"name": "get_weather"}}` | The model must call this tool. |
A model can also ask for several tool calls in one reply, for example to check the weather in two cities. Each call has its own `id`. Run all of them and send one `tool` message for each.
## Streaming Tool Calls
When [streaming](https://tuna.ai/docs/api/streaming.md), tool calls arrive in pieces in `delta.tool_calls`. The first piece of each call has its `id` and `function.name`. Later pieces add to `function.arguments`. Each piece has an `index` that says which call it belongs to, so join the pieces by `index`. The SDKs' stream helpers do this for you.
## Tips
- **Write clear descriptions.** The model picks tools and fills in arguments from the names and descriptions. Say what each tool does, when to use it, and what each parameter means.
- **Check the arguments.** The model writes `arguments` itself. They can be invalid JSON, miss a required field, or have a value you don't expect. Validate them before you run anything.
- **Limit what tools can do.** Treat a tool call like input from a user. For example, a tool that reads files should only reach the files it needs, and a tool that spends money or deletes data should ask a person first.
- **Only send the tools a request needs.** A shorter list makes it easier for the model to pick the right tool, and uses fewer input tokens.
- **Tools cost input tokens.** Tool definitions are added to the prompt, so they are [billed](https://tuna.ai/docs/account/billing.md) as input tokens on every request that includes them. They stay the same between requests, so they are usually [cached input](https://tuna.ai/docs/api/input-caching.md).
Tool calling works on models that were trained for it, which includes most recent open-weight models.
Source: https://tuna.ai/docs/api/tool-calling.md
---
# Reasoning Effort
Reasoning models generate reasoning text before the final answer. `reasoning_effort` selects a model-specific effort level. Higher effort can improve results on complex tasks, but usually increases response time and the number of billed output tokens.
```json title="Request body" {4}
{
"model": "tuna-us/example-lab/example-model-id",
"messages": [{"role": "user", "content": "How many prime numbers are there below 100?"}],
"reasoning_effort": "low"
}
```
## Levels
Each model accepts its own set of levels. To see a model's levels, open **Details & Examples** on its card in [Models & Pricing](https://tuna.ai/docs/models.md). The models on Tuna accept these:
| Level | Meaning |
| --- | --- |
| `low` | Lower reasoning effort, generally with fewer reasoning tokens and a shorter wait. |
| `high` | Higher reasoning effort for tasks that benefit from more reasoning. |
| `max` | The highest reasoning effort level supported by the selected model. |
Use lower levels when limiting latency and token usage matters. Evaluate higher levels on tasks where the additional reasoning improves results.
Some models accept more levels, such as `none` to skip thinking. If you leave `reasoning_effort` out, the model uses its own default level. Models that don't reason have no thinking to control, so `reasoning_effort` only applies to reasoning models.
## Read the Thinking
The thinking comes back in the `reasoning` field of the message, separate from the reply in `content`:
```json title="Response" {5}
"choices": [
{
"message": {
"role": "assistant",
"reasoning": "Count the primes in each group of ten: 2, 3, 5, 7 ...",
"content": "There are 25 prime numbers below 100."
},
"finish_reason": "stop"
}
]
```
```js JavaScript {7}
const response = await client.chat.completions.create({
model: "tuna-us/example-lab/example-model-id",
messages: [{ role: "user", content: "How many prime numbers are there below 100?" }],
reasoning_effort: "low",
});
const message = response.choices[0].message;
console.log(message.reasoning);
console.log(message.content);
```
```python Python {7}
response = client.chat.completions.create(
model="tuna-us/example-lab/example-model-id",
messages=[{"role": "user", "content": "How many prime numbers are there below 100?"}],
reasoning_effort="low",
)
message = response.choices[0].message
print(message.reasoning)
print(message.content)
```
When [streaming](https://tuna.ai/docs/api/streaming.md), the thinking arrives first, in `delta.reasoning`, and then the reply, in `delta.content`.
Like prompts and replies, the thinking is never stored. See [Privacy and ZDR](https://tuna.ai/docs/privacy-and-zdr.md).
## Tokens and Billing
Thinking is written by the model, so its tokens are output tokens. They are counted in [`usage.completion_tokens`](https://tuna.ai/docs/api/chat-completions.md#response) and billed at the model's output price. See [Billing](https://tuna.ai/docs/account/billing.md).
The thinking also counts toward [`max_tokens`](https://tuna.ai/docs/api/model-control.md). If `max_tokens` is too low, the model can use it all up while thinking. The reply is then cut off, or even empty, and `finish_reason` is `length`. A hard problem can use thousands of thinking tokens before the reply starts, so give reasoning models plenty of room, or leave `max_tokens` out.
Source: https://tuna.ai/docs/api/reasoning.md
---
# Image Input
Models with image input can read images as well as text. You can send screenshots, photos, charts, scanned documents, and other images in a [chat completion](https://tuna.ai/docs/api/chat-completions.md), and ask about them in the same message.
## Send an Image
Make the message's `content` a list of parts instead of a string. A text part has `type: "text"`, and an image part has `type: "image_url"`:
```json title="Request body" {8}
{
"model": "tuna-us/example-lab/example-model-id",
"messages": [
{
"role": "user",
"content": [
{"type": "text", "text": "What does this chart show?"},
{"type": "image_url", "image_url": {"url": "https://example.com/chart.png"}}
]
}
]
}
```
The `url` can be:
- **A public `https` URL.** Tuna downloads the image while it handles the request, so the URL has to work without signing in.
- **A data URL** with the image in base64, like `data:image/png;base64,iVBORw0KG...`. Use this for private images, or for files on your own machine. Base64 makes the image about a third larger, so large images can be worth [compressing with gzip](https://tuna.ai/docs/sdks.md#compress-requests-with-gzip).
```js JavaScript {3,11}
import { readFileSync } from "node:fs";
const image = readFileSync("chart.png").toString("base64");
const response = await client.chat.completions.create({
model: "tuna-us/example-lab/example-model-id",
messages: [{
role: "user",
content: [
{ type: "text", text: "What does this chart show?" },
{ type: "image_url", image_url: { url: `data:image/png;base64,${image}` } },
],
}],
});
console.log(response.choices[0].message.content);
```
```python Python {4,12}
import base64
with open("chart.png", "rb") as chart:
image = base64.b64encode(chart.read()).decode()
response = client.chat.completions.create(
model="tuna-us/example-lab/example-model-id",
messages=[{
"role": "user",
"content": [
{"type": "text", "text": "What does this chart show?"},
{"type": "image_url", "image_url": {"url": f"data:image/png;base64,{image}"}},
],
}],
)
print(response.choices[0].message.content)
```
Use common image formats, such as PNG, JPEG, and WebP.
## Limits
A request can have up to 30 images, counting every message in the conversation. A request with more fails with status 400. See [Errors](https://tuna.ai/docs/api/errors.md).
In a long conversation, earlier images count too. Drop images from old messages when the model no longer needs them.
Check that the model you pick supports image input before you send an image. Sending an image to a text-only model fails.
## Billing
The model turns each image into tokens, and those are billed as input tokens at the model's [input price](https://tuna.ai/docs/api/models.md). A larger image uses more tokens, and the exact number depends on the model. To pay less, resize images to the smallest size that still shows what the model needs to see.
The `usage` field of the response includes the image tokens in `prompt_tokens`. See [Billing](https://tuna.ai/docs/account/billing.md).
## Privacy
Images are part of your prompt, so they get the same Zero Data Retention (ZDR) as text. See [Privacy and ZDR](https://tuna.ai/docs/privacy-and-zdr.md).
Source: https://tuna.ai/docs/api/image-input.md
---
# Input Caching
Many apps reuse the same text at the start of their requests: a long system prompt, tool definitions, a document to ask questions about, or the earlier turns of a conversation. The model's state after reading that shared start is kept and reused. Those reused tokens are billed as cached input, which costs less than normal input.
Caching is automatic. There is nothing to turn on.
## How It Works
When a model reads a prompt, it builds a KV cache: its internal state after reading that text. Recent KV caches are kept for a while. When a new request starts with exactly the same tokens as an earlier one, the cache for that shared start, called the prefix, is reused. Only the tokens after it are processed.
This saves money and time. Cached input costs less, and a long prompt with a cached prefix gets its first token back sooner.
## Cache Writes Cost Nothing Extra
Models have separate prices for normal and cached input:
| Token type | Price |
| --- | --- |
| Input | The model's input price. |
| Cached input | The model's cached input price, which is lower. |
Writing to the cache costs nothing extra. The first request with a new prefix pays the normal input price for it, the same as if there were no cache. Some providers, including [OpenAI](https://developers.openai.com/api/docs/pricing) and [Anthropic](https://platform.claude.com/docs/en/build-with-claude/prompt-caching), charge more than the normal input price for tokens written to the cache. Tuna applies the normal input rate to cache writes and cache misses. Only input tokens reused from the cache receive the lower cached-input rate.
For example, an agent sends the same 20,000-token system prompt and tool list with every request, plus about 500 new tokens. After the first request, almost all of the 20,500 input tokens in each request can be billed as cached input.
See the prices for each model in the [models list](https://tuna.ai/docs/api/models.md), and how charges work in [Billing](https://tuna.ai/docs/account/billing.md).
## Get More Cache Hits
Only the shared prefix can be reused. Everything after the first token that differs is processed again. So:
- **Put what stays the same first.** Start with the system prompt, tool definitions, reference documents, and examples. Put what changes, like the user's latest message, at the end.
- **Keep the prefix exactly the same.** A timestamp, request id, or random value near the start ends the shared prefix there, so tokens after it can't reuse the cache. Move values like these to the end.
- **Add to conversations instead of rewriting them.** Append new messages at the end. Editing or reordering earlier messages changes the prefix.
- **Keep tools in the same order.** [Tool definitions](https://tuna.ai/docs/api/tool-calling.md) are part of the prompt, so reordering them changes the prefix too.
- **Send related requests close together.** A cache is more likely to still be there a few seconds later than an hour later.
Caches are kept for each model separately, so requests to different models never share one. The cache works in blocks of tokens, so the last few tokens of a prefix may be billed as normal input.
## Check Your Cache Hits
`usage.prompt_tokens_details.cached_tokens` shows how many input tokens were cached:
```json title="Response" {5}
"usage": {
"prompt_tokens": 20512,
"completion_tokens": 120,
"total_tokens": 20632,
"prompt_tokens_details": {"cached_tokens": 20480}
}
```
`prompt_tokens` counts all input tokens, cached or not. Here, 20,480 tokens are billed at the cached input price, and the other 32 at the normal input price.
## How Long Caches Last
Caches aren't kept for a set amount of time. How long one lasts depends on load, demand, and available capacity. When space is needed, the caches used least recently are removed first.
A cache miss is never an error. The request works normally, and the prefix is billed at the normal input price.
## Privacy and Encryption
See [Privacy and ZDR](https://tuna.ai/docs/privacy-and-zdr.md#kv-cache).
Source: https://tuna.ai/docs/api/input-caching.md
---
# Advanced Model Control
A few request settings change how the model writes its reply. They are all optional. If you leave one out, the model's default is used.
```json title="Request body" {4-6}
{
"model": "tuna-us/example-lab/example-model-id",
"messages": [{"role": "user", "content": "Suggest a name for an ocean research vessel."}],
"max_tokens": 100,
"temperature": 1.2,
"stop": ["\n\n"]
}
```
| Setting | Type | What it does |
| --- | --- | --- |
| `max_tokens` | integer | The most tokens the reply can have. |
| `temperature` | number, 0 to 2 | How random the reply is. |
| `top_p` | number, 0 to 1 | Limits the model to its most likely tokens. |
| `stop` | string or array | Text that ends the reply. |
| `presence_penalty` | number, -2 to 2 | Pushes the model toward new topics. |
| `frequency_penalty` | number, -2 to 2 | Makes the model repeat itself less. |
## How the Model Picks Each Token
A model writes its reply one token at a time. It assigns probabilities to possible next tokens, then picks one at random using those probabilities. `temperature`, `top_p`, and the penalties change that last step. They only affect how a token is chosen from those probabilities, and they can't add knowledge the model doesn't already have.
## max_tokens
`max_tokens` caps the length of the reply. If the reply reaches it, the reply is cut off, and `finish_reason` is `length` instead of `stop`.
- Each model's limit is documented on the [Models & Pricing](https://tuna.ai/docs/models.md) page. If you ask for more, Tuna lowers `max_tokens` to that limit.
- If you leave it out, `max_tokens` defaults to the model's maximum output. The setting can only reduce that limit, which can help control costs.
- For [reasoning models](https://tuna.ai/docs/api/reasoning.md), the model's thinking counts toward `max_tokens` too.
- `max_completion_tokens` is another name for the same setting, from newer versions of the OpenAI API. Some tools send it instead.
A low `max_tokens` doesn't make the model write shorter answers. It only cuts them off. To get short answers, ask for them in the prompt.
## temperature
`temperature` controls how random the pick is.
- Low values, like `0.2`, make the model almost always pick its most likely token. Replies are focused and predictable. This suits code, data extraction, and answers with one right answer.
- High values, like `1.2`, spread the pick across more tokens. Replies are more varied. This suits brainstorming and creative writing.
- At `0`, the model picks the most likely next token. Replies to the same request are then usually the same, although an exact match is not guaranteed.
Very high values, above about `1.5`, often make replies wander or stop making sense.
## top_p
`top_p` limits the pick to the most likely tokens whose probabilities add up to `top_p`. At `0.9`, the least likely tokens, which make up the last 10% of the probability, are never picked.
Values like `0.9` or `0.95` are common.
`top_p` and `temperature` both control randomness. Change one of them and leave the other at its default.
## stop
`stop` is a string, or a list of strings, that ends the reply as soon as the model writes one of them. The stop text itself isn't included in the reply, and `finish_reason` is `stop`.
For example, `"stop": ["\n\n"]` ends the reply at the first blank line.
## presence_penalty and frequency_penalty
Both penalties lower the chance of tokens that are already in the reply. Positive values push away from repeats, and negative values make repeats more likely.
- `presence_penalty` lowers the chance of any token that has appeared at least once. This nudges the model toward new words and topics.
- `frequency_penalty` lowers the chance of a token more each time it appears. This helps when a model repeats the same phrase over and over.
Small values, like `0.1` to `0.5`, are usually enough. Large values can make the model avoid words it needs.
Source: https://tuna.ai/docs/api/model-control.md
---
# Structured Outputs (Coming Soon)
Structured outputs aren't available yet. If you need them soon, contact us and we'll prioritize it.
Structured outputs will constrain a model's response to the JSON Schema you provide. Check that `finish_reason` is `stop` before parsing the response, and validate the returned values for your application.
## Why They Are Useful
Models are good at following format instructions, but they still make mistakes. A reply might wrap the JSON in extra text, leave out a field, or add a trailing comma. Code that parses replies then needs checks and retries.
Common uses include:
- **Extracting data.** Pull names, dates, and amounts from emails or documents into fixed fields.
- **Classifying.** Make the model pick one label from a fixed list, with `enum` in the schema.
- **Passing data between steps.** An agent or pipeline can hand the reply straight to the next piece of code.
- **Filling in a UI.** Render the reply as a form, table, or card without guessing at its shape.
## How They Will Work
Structured outputs will use the `response_format` parameter from the [OpenAI API](https://tuna.ai/docs/api/openai-compatibility.md):
```json title="Request body"
{
"model": "tuna-us/example-lab/example-model-id",
"messages": [{"role": "user", "content": "Jane Doe, 34, lives in Boston."}],
"response_format": {
"type": "json_schema",
"json_schema": {
"name": "person",
"strict": true,
"schema": {
"type": "object",
"properties": {
"name": {"type": "string"},
"age": {"type": "integer"},
"city": {"type": "string"}
},
"required": ["name", "age", "city"],
"additionalProperties": false
}
}
}
}
```
The reply's `content` would then be JSON like `{"name": "Jane Doe", "age": 34, "city": "Boston"}`.
This works by constrained decoding. As the model writes each token, any token that would break the schema is ruled out, so the reply can only follow the schema. A reply can still be cut off by [`max_tokens`](https://tuna.ai/docs/api/model-control.md#maxtokens), so check that `finish_reason` is `stop`.
A simpler JSON mode, `{"type": "json_object"}`, will make the reply valid JSON of any shape.
## Until Then
- **Ask for JSON in the prompt.** Describe the fields you want, show an example, and parse the reply. Validate it and retry if it doesn't match.
- **Use [tool calling](https://tuna.ai/docs/api/tool-calling.md).** Describe a tool whose `parameters` are your schema, and set `tool_choice` to that tool. The model's `arguments` then follow your schema closely, though you should still validate them.
Source: https://tuna.ai/docs/api/structured-outputs.md
---
# Assistant Prefill (Coming Soon)
Assistant prefill isn't available yet. If you need it soon, contact us and we'll prioritize it.
With assistant prefill, you write the first part of the model's reply, and the model continues from where your text ends. It is a simple way to steer a reply, because the model treats your text as something it already wrote.
## How It Will Work
End the `messages` list with an `assistant` message. Its `content` is the start of the reply:
```json title="Request body" {5}
{
"model": "tuna-us/example-lab/example-model-id",
"messages": [
{"role": "user", "content": "List three fish that live in the Atlantic, as JSON."},
{"role": "assistant", "content": "{\"fish\": ["}
]
}
```
The reply will hold only the new text that continues the prefill, such as `"Atlantic cod", "Bluefin tuna", "Haddock"]}`. To get the full reply, join your prefill and the new text.
## What It Is Useful For
- **Setting the format.** Start the reply with `{`, a code fence, or a heading, and the model continues in that format.
- **Skipping the preamble.** Start the reply with the answer itself, so the model doesn't open with "Sure, here is...".
- **Beginning a plan.** Supply the first step of a plan as an assistant prefill. The model will generate text that continues from that prefix.
- **Continuing a partial reply.** In a new request, include the original conversation and end it with the partial assistant reply. The response will contain only the continuation. Resent input is billed at the normal or [cached-input rate](https://tuna.ai/docs/api/input-caching.md), depending on cache reuse. Newly generated tokens are billed as output.
Source: https://tuna.ai/docs/api/assistant-prefill.md
---
# OpenAI Compatibility
Tuna supports OpenAI's Chat Completions request format for the features listed below. To use an OpenAI-compatible client:
- Set the base URL to `https://api-us.tuna.ai/v1`.
- Use a [Tuna API key](https://tuna.ai/docs/authentication.md).
- Select a model ID from [Tuna's models](https://tuna.ai/docs/models.md).
Check the compatibility table below for the endpoints and features your application uses.
## Point an OpenAI Client at Tuna
```js JavaScript {4-5,9}
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api-us.tuna.ai/v1",
apiKey: "tuna_yourApiKeyHereWithTunaUnderscore",
});
const response = await client.chat.completions.create({
model: "tuna-us/example-lab/example-model-id",
messages: [{
role: "user",
content: "Is the tuna one of the fastest sustained swimming fish species? And does it have a streamlined body?",
}],
});
console.log(response.choices[0].message.content);
```
```python Python {4-5,9}
from openai import OpenAI
client = OpenAI(
base_url="https://api-us.tuna.ai/v1",
api_key="tuna_yourApiKeyHereWithTunaUnderscore",
)
response = client.chat.completions.create(
model="tuna-us/example-lab/example-model-id",
messages=[{
"role": "user",
"content": "Is the tuna one of the fastest sustained swimming fish species? And does it have a streamlined body?",
}],
)
print(response.choices[0].message.content)
```
```bash curl
curl https://api-us.tuna.ai/v1/chat/completions \
-H "Authorization: Bearer tuna_yourApiKeyHereWithTunaUnderscore" \
-H "Content-Type: application/json" \
-d '{"model": "tuna-us/example-lab/example-model-id", "messages": [{"role": "user", "content": "Is the tuna one of the fastest sustained swimming fish species? And does it have a streamlined body?"}]}'
```
The OpenAI SDKs also read the base URL and key from environment variables, and so do many tools built on them:
```bash
export OPENAI_BASE_URL="https://api-us.tuna.ai/v1"
export OPENAI_API_KEY="tuna_yourApiKeyHereWithTunaUnderscore"
```
## Endpoint Compatibility Comparison
This table lists OpenAI SDK calls, and what happens when the client's base URL is Tuna's. The `chat.completions.create` calls go to [`POST /v1/chat/completions`](https://tuna.ai/docs/api/chat-completions.md), and `models.list` goes to [`GET /v1/models`](https://tuna.ai/docs/api/models.md).
| OpenAI SDK call | Supported | Docs |
| --- | --- | --- |
| `chat.completions.create` | Yes | [Chat completions](https://tuna.ai/docs/api/chat-completions.md), [Streaming](https://tuna.ai/docs/api/streaming.md) |
| `chat.completions.create` with `tools` | Yes | [Tool calling](https://tuna.ai/docs/api/tool-calling.md) |
| `chat.completions.create` with image parts | Yes | [Image input](https://tuna.ai/docs/api/image-input.md) |
| `chat.completions.create` with `reasoning_effort` | Yes | [Reasoning effort](https://tuna.ai/docs/api/reasoning.md) |
| `chat.completions.create` with `response_format` | Coming soon | [Structured outputs](https://tuna.ai/docs/api/structured-outputs.md) |
| `models.list` | Yes | [List Model Endpoints](https://tuna.ai/docs/api/models.md) |
| `responses.create` | No | Use [chat completions](https://tuna.ai/docs/api/chat-completions.md). |
| `completions.create` | No | Use [chat completions](https://tuna.ai/docs/api/chat-completions.md). |
| `assistants.*`, `threads.*` | No | Run the agent loop in your code with [tool calling](https://tuna.ai/docs/api/tool-calling.md). |
| `embeddings.create` | No | |
| `images.*`, `audio.*` | No | |
| `moderations.create` | No | |
| `batches.*`, `files.*`, `fine_tuning.*` | No | |
A call that isn't supported returns an [error](https://tuna.ai/docs/api/errors.md).
Supported features need no extra setup. Input caching is automatic, and cached tokens are counted in `usage.prompt_tokens_details.cached_tokens`, the same field OpenAI uses. See [Input caching](https://tuna.ai/docs/api/input-caching.md).
## Differences from OpenAI
### Model IDs
OpenAI model names like `gpt-4o` aren't available, because Tuna serves open-weight models. Use a Tuna id instead. Tuna's ids start with `tuna-us/`, then the model's maker and name, like `tuna-us/moonshotai/kimi-k3-ultrafast`. Get the current list from [`GET /v1/models`](https://tuna.ai/docs/api/models.md).
### Parameters
- If [`max_tokens`](https://tuna.ai/docs/api/model-control.md#maxtokens) is higher than the model's own limit, Tuna lowers it to that limit instead of returning an error.
- `max_completion_tokens` is treated the same as `max_tokens`.
- `reasoning_effort` only has an effect on reasoning models. Some models accept more levels than `low`, `medium`, and `high`. See [Reasoning effort](https://tuna.ai/docs/api/reasoning.md).
### Responses
- Reasoning models put their thinking in a `reasoning` field on the message, next to `content`. In a stream, it arrives in `delta.reasoning`. See [Reasoning effort](https://tuna.ai/docs/api/reasoning.md).
- If a stream fails after it started, the last chunk has an `error` object and `finish_reason` is `error`. See [Errors during a stream](https://tuna.ai/docs/api/streaming.md#errors-during-a-stream).
### Errors
An error body has the form `{"error": {"code": 401, "message": "..."}}`. The `code` is the HTTP status as a number, and the message text is Tuna's own. Base error handling in your code on the HTTP status. The message text can change. See [Errors](https://tuna.ai/docs/api/errors.md).
### Pricing
Writing to the input cache costs the same as normal input. OpenAI charges more than its input price for cache writes on some models. See [Input caching](https://tuna.ai/docs/api/input-caching.md#cache-writes-cost-nothing-extra).
## Other Libraries
Most libraries and tools that let you set an OpenAI base URL work with Tuna too. See [Recommended SDKs](https://tuna.ai/docs/sdks.md) for the ones Tuna tests with.
Source: https://tuna.ai/docs/api/openai-compatibility.md
---
# Errors
When a request fails, the API responds with an HTTP error status and a JSON body that describes the error:
```json title="Error response"
{
"error": {
"code": 401,
"message": "The API key you provided is invalid."
}
}
```
`code` is the same as the HTTP status, and `message` explains the error. Some errors also have a `metadata` object with more detail.
## Status Codes
| Status | Meaning | Retry? |
| --- | --- | --- |
| 400 | The request is invalid. For example, the body isn't valid JSON, a required parameter is missing, the model id doesn't exist, or there are [too many images](https://tuna.ai/docs/api/image-input.md#limits). | No |
| 401 | The API key is missing or invalid. | No |
| 402 | The organization's [balance](https://tuna.ai/docs/account/billing.md) is too low for the request. | No |
| 403 | The API key isn't allowed to make this request. | No |
| 408 | The request took too long. | Yes |
| 429 | The organization sent too many requests. See [Rate limits](https://tuna.ai/docs/api/rate-limits.md). | Yes |
| 500 | Tuna had an internal error. It is usually temporary. | Yes |
| 502 | The model failed or returned an invalid reply. | Yes |
| 503 | Tuna doesn't have capacity for this model right now. Try again shortly, or use another model. | Yes |
See [Retries](https://tuna.ai/docs/api/retries.md) for how long to wait before trying again.
## 400: Invalid Request
The `message` says what is wrong. Fix the request before sending it again, since the same request will fail the same way. If the model id is the problem, get a current id from the [models list](https://tuna.ai/docs/api/models.md).
## 401: Authentication
| Message | Cause |
| --- | --- |
| You must provide an API key. | The request has no `Authorization` header. |
| The API key you provided is invalid. | The key is malformed or does not exist. |
Check that the request sends `Authorization: Bearer tuna_yourApiKeyHereWithTunaUnderscore`, and that the key was copied in full. See [Authentication](https://tuna.ai/docs/authentication.md).
## 402: Balance
The message is "The account linked with the API key has reached its spending limit/run out of credits."
The [organization's](https://tuna.ai/docs/account/organizations.md) balance is checked before a request runs, using an estimate of the request's size. If the balance is too low, the request fails with 402 before any tokens are charged. Add credit to fix it. See [Billing](https://tuna.ai/docs/account/billing.md).
Retrying a 402 won't help, because it happens again until credit is added. Show the error to whoever manages billing for your organization instead. That is usually the organization's [Owner](https://tuna.ai/docs/account/organizations.md#roles).
## 429, 500, 502, and 503: Temporary Errors
These errors usually go away on their own. Wait and retry, and follow the `Retry-After` header when a response has one. The OpenAI SDKs and the Vercel AI SDK retry them for you. See [Retries](https://tuna.ai/docs/api/retries.md).
## Errors During a Stream
If a [stream](https://tuna.ai/docs/api/streaming.md) fails after it has started, the HTTP status is already `200`. The error comes as a last chunk with an `error` field and a `finish_reason` of `error`. See [Errors during a stream](https://tuna.ai/docs/api/streaming.md#errors-during-a-stream).
## Handling Errors in Code
The [OpenAI SDKs](https://tuna.ai/docs/sdks.md) turn error responses into exceptions that carry the status and message:
```js JavaScript {8}
try {
const response = await client.chat.completions.create({
model: "tuna-us/example-lab/example-model-id",
messages,
});
} catch (error) {
if (!(error instanceof OpenAI.APIError)) throw error;
console.error(error.status, error.message);
}
```
```python Python {9}
import openai
try:
response = client.chat.completions.create(
model="tuna-us/example-lab/example-model-id",
messages=messages,
)
except openai.APIStatusError as error:
print(error.status_code, error.message)
except openai.APIConnectionError:
print("Could not reach Tuna")
```
By the time your code sees a 429 or 5xx error, the SDK has already retried it, twice by default. See [Retries](https://tuna.ai/docs/api/retries.md#the-sdks-retry-for-you).
Source: https://tuna.ai/docs/api/errors.md
---
# Retries
Some errors are temporary. The service is busy, or a model server had a problem. The same request usually works if you wait a little and send it again. Other errors happen again until something is fixed, so it is better to fix the cause than to retry.
## Which Errors to Retry
| Status | Retry? |
| --- | --- |
| 400, 401, 403 | No. Fix the request or the [API key](https://tuna.ai/docs/authentication.md) first. |
| 402 | No. Add credit first. See [Billing](https://tuna.ai/docs/account/billing.md). |
| 408 | Yes. |
| 429 | Yes, after the time in `Retry-After`. See [Rate limits](https://tuna.ai/docs/api/rate-limits.md). |
| 500, 502, 503 | Yes. |
| Timeout or dropped connection | Yes, but read Timeouts and dropped connections first. |
See [Errors](https://tuna.ai/docs/api/errors.md) for what each status means.
## Retry-After
A 429 or 503 response can include a `Retry-After` header. It is the number of seconds to wait before you try again. When it is there, wait at least that long.
```http {2}
HTTP/1.1 429 Too Many Requests
Retry-After: 30
```
## Exponential Backoff
When there is no `Retry-After`, wait longer after each failed attempt: 1 second, then 2, then 4, and so on. Add a small random amount to each wait, called jitter, so that many clients that failed at the same moment don't all retry at the same moment too. Stop after a few attempts, such as 3 to 5, and give up once the total time is more than your app can wait.
```python title="Python"
import random
import time
RETRY_STATUSES = {408, 429, 500, 502, 503}
def post_with_retries(send, max_attempts=5):
for attempt in range(max_attempts):
response = send()
if response.status_code not in RETRY_STATUSES or attempt == max_attempts - 1:
return response
retry_after = response.headers.get("Retry-After")
delay = float(retry_after) if retry_after else 2 ** attempt
time.sleep(delay + random.random())
```
## The SDKs Retry for You
The [OpenAI SDKs](https://tuna.ai/docs/sdks.md#openai-javascript-sdk) retry connection errors and statuses 408, 409, 429, and 500 or higher. By default they retry twice, with exponential backoff, and they follow `Retry-After`. The [Vercel AI SDK](https://tuna.ai/docs/sdks.md#vercel-ai-sdk) also retries twice by default. You can change the number of retries:
```js JavaScript {4}
const client = new OpenAI({
baseURL: "https://api-us.tuna.ai/v1",
apiKey: "tuna_yourApiKeyHereWithTunaUnderscore",
maxRetries: 5,
});
```
```python Python {4}
client = OpenAI(
base_url="https://api-us.tuna.ai/v1",
api_key="tuna_yourApiKeyHereWithTunaUnderscore",
max_retries=5,
)
```
```ts Vercel AI SDK {4}
const { text } = await generateText({
model: tuna("tuna-us/example-lab/example-model-id"),
prompt,
maxRetries: 5,
});
```
Set the number to `0` to turn retries off and handle them yourself.
## Timeouts and Dropped Connections
A timeout or a dropped connection doesn't tell you whether Tuna ran the request. It may have finished, and been charged, before the reply was lost. Retrying runs the request again and is charged again.
For short requests this rarely matters. For long ones:
- **Use [streaming](https://tuna.ai/docs/api/streaming.md).** Text arrives as it is written, so a dropped connection leaves you with the part you already received. Waiting for a long reply without streaming can also hit your HTTP client's timeout. The OpenAI SDKs time out after 10 minutes by default.
- **Don't repeat actions.** If your app already acted on part of a reply, like running a [tool call](https://tuna.ai/docs/api/tool-calling.md), make sure a retry doesn't run it twice.
## Errors During a Stream
A stream can fail after it starts, even though the HTTP status was `200`. The error arrives in the stream itself. See [Errors during a stream](https://tuna.ai/docs/api/streaming.md#errors-during-a-stream).
Source: https://tuna.ai/docs/api/retries.md
---
# Rate Limits
Rate limits apply to the combined request traffic of an organization. All projects and API keys share the limit. Requests above it receive HTTP 429.
## Limits Are Shared by the Organization
An organization has one rate limit. Requests from every project and [API key](https://tuna.ai/docs/authentication.md) in the organization count toward that limit, so creating more keys doesn't raise it. See [Organizations and projects](https://tuna.ai/docs/account/organizations.md).
## Limits Grow with Your Balance
Your organization's rate limit depends on its balance. A larger balance gives a higher limit. If you often hit the limit, adding credit raises it. See [Billing](https://tuna.ai/docs/account/billing.md).
For limits beyond that, see [Enterprise](https://tuna.ai/docs/account/enterprise.md).
## When You Hit the Limit
A request over the limit fails with status 429. This doesn't mean anything is wrong with your account. It means the organization is sending requests faster than its limit allows right now. The response has a `Retry-After` header with the number of seconds to wait. After that, send the request again. The [OpenAI SDKs and the Vercel AI SDK](https://tuna.ai/docs/sdks.md) do this for you. See [Retries](https://tuna.ai/docs/api/retries.md).
To hit the limit less often:
- **Limit how many requests run at once.** For batch jobs, use a queue with a fixed number of workers instead of sending the whole batch at once.
- **Spread requests out.** A steady flow of requests is less likely to hit the limit than bursts.
- **Combine small tasks.** Many small tasks, like labeling 20 short texts, can often be done in one request instead of 20.
A [503 error](https://tuna.ai/docs/api/errors.md) is different. It means Tuna doesn't have capacity for that model right now, and it isn't about your organization's limit. It is also safe to retry.
Source: https://tuna.ai/docs/api/rate-limits.md
---
# Organizations and Projects
Accounts have three levels:
- An **organization** is your company or team. It has members and one prepaid [balance](https://tuna.ai/docs/account/billing.md).
- A **project** is one app or environment inside an organization, like "Production" or "Staging".
- An **API key** belongs to one project. Requests made with it are charged to the organization.
An organization can have any number of projects, and a project can have any number of [keys](https://tuna.ai/docs/authentication.md). All the projects in an organization spend from the same balance and share one [rate limit](https://tuna.ai/docs/api/rate-limits.md).
Tuna uses [WorkOS](https://workos.com) to manage sign-in, organizations, and API keys.
## Team Sign-In
Invite teammates to your organization so each person can sign in with their own account and access the organization's shared projects and balance. Teams can use these features without an enterprise plan or a company single sign-on (SSO) setup.
## Roles
Organization members have one of four roles: Owner, Admin, Member, or View-only.
- Each organization has exactly one **Owner**. The person who creates the organization becomes its Owner, and the Owner is responsible for the organization's billing.
- An organization can have any number of members with the **Admin**, **Member**, or **View-only** role.
| Permission | Owner | Admin | Member | View-only |
| --- | :---: | :---: | :---: | :---: |
| View dashboard | Allowed | Allowed | Allowed | Allowed |
| Use web chat | Allowed | Allowed | Allowed | Not allowed |
| View truncated API key | Allowed | Allowed | Allowed | Allowed |
| Create & delete API keys | Allowed | Allowed | Allowed | Not allowed |
| View billing page | Allowed | Allowed | Allowed | Allowed |
| Manage payment methods | Allowed | Allowed | Not allowed | Not allowed |
| Buy credits & top-up | Allowed | Allowed | Not allowed | Not allowed |
| Edit spend limits | Allowed | Allowed | Not allowed | Not allowed |
| Add/remove members | Allowed | Allowed | Not allowed | Not allowed |
| Add/remove admins | Allowed | Allowed | Not allowed | Not allowed |
| Remove the owner | Not allowed | Not allowed | Not allowed | Not allowed |
## Using Projects
Projects keep keys for different apps apart. A common setup is one project per app, or one per environment. If you need to replace a key, only the app that uses it has to change.
## Single Sign-On
Single sign-on (SSO) lets members sign in through their company's account system, called an identity provider. Enterprise customers can connect that system to Tuna using SAML, a standard for sending signed confirmations that a member has signed in. This is optional for teams. See [Enterprise](https://tuna.ai/docs/account/enterprise.md#single-sign-on) for an explanation and setup information.
Source: https://tuna.ai/docs/account/organizations.md
---
# Billing
Tuna is prepaid. Your [organization](https://tuna.ai/docs/account/organizations.md) keeps a balance of credits, and each [serverless](https://tuna.ai/docs/api/overview.md) request is paid from it. You pay per token, and there is no charge for idle time or for keeping a model running.
## Pricing
Token prices vary by model and token type:
| Token type | What it is |
| --- | --- |
| Input | The tokens in your messages, including [images](https://tuna.ai/docs/api/image-input.md#billing) and [tool definitions](https://tuna.ai/docs/api/tool-calling.md#tips). |
| Cached input | Input tokens the model processed recently, such as a long system prompt reused across requests. They are billed at the model's cached input price. See [Input caching](https://tuna.ai/docs/api/input-caching.md). |
| Output | The tokens the model writes, including the thinking of [reasoning models](https://tuna.ai/docs/api/reasoning.md#tokens-and-billing). |
Prices are quoted per million tokens, but billing is based on the exact number of tokens used. Usage is never rounded up to a full million. Charges are deducted from your organization's balance of credits.
The `usage` field of each response shows how many tokens the request used. See [Models & pricing](https://tuna.ai/docs/models.md) for each model's prices.
For example, suppose a model costs $1.00 per million input tokens and $4.00 per million output tokens. A request with 2,000 input tokens and 500 output tokens then costs $0.002 for input and $0.002 for output, which is $0.004 in total. The amount is paid from the organization's balance of credits. These prices are only an example.
## Precision
Charges are calculated and internal amounts are kept with $0.000000001 precision.
Amounts shown in the UI are rounded to the nearest penny ($0.01). This rounding is only for display and does not change the internal amounts.
Monetary amounts returned by Tuna's APIs may also be rounded. Charges are calculated from the full-precision internal amounts.
## Usage Tracking & Accounting
Tuna uses [Metronome](https://metronome.com), a usage billing service, to track usage and balances. After each request, Tuna sends Metronome the model, the token counts, the cost, and the time. It never sends the text of your prompts or replies.
With an [enterprise plan](https://tuna.ai/docs/account/enterprise.md), you can see the cost of each request as recorded in Metronome.
## Adding Credit
Credit is added to the organization's balance, which is shared by its projects. The organization's [Owner](https://tuna.ai/docs/account/organizations.md#roles) is responsible for billing.
Credits are denominated in US dollars. You can pay in US dollars or in your local currency, and the payment is converted into credits. Credits have no cash value and are non-refundable, unless the law requires otherwise.
## Payment Methods
Payments are processed by [Stripe](https://stripe.com). You can pay with all major credit cards. Stripe handles your card details, and Tuna doesn't store your card number.
Paying with [crypto](https://tuna.ai/docs/account/crypto.md) and with [x402](https://tuna.ai/docs/account/x402.md) is coming soon.
## When the Balance Runs Out
They return status 402 until someone adds credit. After credit is added, new requests work again without any other change. See [Errors](https://tuna.ai/docs/api/errors.md#402-balance).
A streaming reply can stop early if the balance runs out partway through. Tokens are counted as they stream, and the request stops once the balance is used up. The stream then ends with a 402 error chunk. See [Errors during a stream](https://tuna.ai/docs/api/streaming.md#errors-during-a-stream).
Before a request runs, its cost is estimated from the size of the input. If the estimate is more than the balance, the request fails with 402 without being charged. Adding credit fixes it, and so does sending a shorter input.
Source: https://tuna.ai/docs/account/billing.md
---
# x402 Agentic Payments
Tuna doesn't accept x402 payments yet. If you need them soon, contact us and we'll prioritize it.
## What x402 Is
[x402](https://www.x402.org) is an open standard for paying for things over HTTP. Coinbase created it in 2025, and it is now run by the x402 Foundation at the Linux Foundation. It uses the HTTP status `402 Payment Required`, which has been part of HTTP for decades but was rarely used.
A payment works like this:
1. A client, such as an AI agent, sends a request.
2. The server replies with 402 and says how much to pay and how.
3. The client pays, usually with a stablecoin such as USDC, and sends the request again with proof of the payment. A stablecoin is a cryptocurrency whose value is tied to a regular currency, like the US dollar.
4. The server checks the payment and handles the request.
The client doesn't need an account or a signup first. This lets an AI agent pay for the services it uses on its own.
## x402 on Tuna
Tuna plans to support x402, so that agents can pay for requests over HTTP. Until then, [organizations](https://tuna.ai/docs/account/organizations.md) pay with a [credit card](https://tuna.ai/docs/account/billing.md#payment-methods).
Source: https://tuna.ai/docs/account/x402.md
---
# Pay with Crypto (Coming Soon)
Crypto payments aren't available yet. If you need them soon, contact us and we'll prioritize it. Please include which coins and networks you use.
Tuna plans to accept cryptocurrency payments through a payment partner. Like a card payment, a crypto payment will be converted into credits in your [organization's](https://tuna.ai/docs/account/organizations.md) balance. Credits are denominated in US dollars. See [Billing](https://tuna.ai/docs/account/billing.md).
Until then, you can pay with a [credit card](https://tuna.ai/docs/account/billing.md#payment-methods). For AI agents that pay for each request over HTTP, see [x402](https://tuna.ai/docs/account/x402.md).
Source: https://tuna.ai/docs/account/crypto.md
---
# Enterprise
Tuna works with companies that need more than a self-serve account. Contact us to talk about what your team needs. These are some of the options.
## Single Sign-On
Single sign-on (SSO) lets your team sign in to Tuna with the work accounts managed by your company. The service that manages those accounts is called an identity provider, such as Google Workspace, Microsoft Entra ID, or Okta.
Tuna uses [WorkOS](https://workos.com) for sign-in and supports SSO through SAML (Security Assertion Markup Language). SAML is a standard that lets your company's identity provider send Tuna a signed confirmation that a member has signed in. Contact us to set up SSO for your organization.
Signing in with a Google account is separate from setting up Google Workspace SAML SSO. A SAML connection lets your company's administrator control who can sign in to Tuna through that connection.
Teams can also use Tuna without SSO or SAML. Invite teammates to a shared [organization](https://tuna.ai/docs/account/organizations.md#team-sign-in), and each person signs in with their own account. This does not require an enterprise plan.
## Billing in Arrears
Self-serve organizations are prepaid. They add credit first, and requests are paid from the balance. Enterprise customers can instead be billed in arrears, with an invoice for what they used after each billing period. See [Billing](https://tuna.ai/docs/account/billing.md).
Tuna tracks usage with [Metronome](https://tuna.ai/docs/account/billing.md#usage-tracking-accounting), a usage billing service. On an enterprise plan, you can see the cost of each request as recorded there.
## Higher Rate Limits
Rate limits normally grow with your balance. Enterprise customers can agree on limits that fit their traffic. See [Rate limits](https://tuna.ai/docs/api/rate-limits.md).
## Features for Your Company
Tell us what your company needs, and we'll look at adding it. If your application depends on Anthropic's API format, contact us about compatibility. Include the endpoints and SDK features you use.
## Contact Us
When you contact us, it helps to include:
- About how many requests or tokens you expect each month
- Which models you plan to use
- Your identity provider, if you want SSO
- How you want to be billed
Zero Data Retention (ZDR) is on for everyone by default. You don't need an enterprise plan for it. See [Privacy and ZDR](https://tuna.ai/docs/privacy-and-zdr.md).
Source: https://tuna.ai/docs/account/enterprise.md