Input Caching

Many apps reuse the same text at the start of their requests: a long system prompt, tool definitions, a document to ask questions about, or the earlier turns of a conversation. The model's state after reading that shared start is kept and reused. Those reused tokens are billed as cached input, which costs less than normal input.

Caching is automatic. There is nothing to turn on.

How It Works

When a model reads a prompt, it builds a KV cache: its internal state after reading that text. Recent KV caches are kept for a while. When a new request starts with exactly the same tokens as an earlier one, the cache for that shared start, called the prefix, is reused. Only the tokens after it are processed.

This saves money and time. Cached input costs less, and a long prompt with a cached prefix gets its first token back sooner.

Cache Writes Cost Nothing Extra

Models have separate prices for normal and cached input:

Writing to the cache costs nothing extra. The first request with a new prefix pays the normal input price for it, the same as if there were no cache. Some providers, including OpenAI and Anthropic, charge more than the normal input price for tokens written to the cache. Tuna applies the normal input rate to cache writes and cache misses. Only input tokens reused from the cache receive the lower cached-input rate.

For example, an agent sends the same 20,000-token system prompt and tool list with every request, plus about 500 new tokens. After the first request, almost all of the 20,500 input tokens in each request can be billed as cached input.

See the prices for each model in the models list, and how charges work in Billing.

Get More Cache Hits

Only the shared prefix can be reused. Everything after the first token that differs is processed again. So:

• Put what stays the same first. Start with the system prompt, tool definitions, reference documents, and examples. Put what changes, like the user's latest message, at the end.
• Keep the prefix exactly the same. A timestamp, request id, or random value near the start ends the shared prefix there, so tokens after it can't reuse the cache. Move values like these to the end.
• Add to conversations instead of rewriting them. Append new messages at the end. Editing or reordering earlier messages changes the prefix.
• Keep tools in the same order. Tool definitions are part of the prompt, so reordering them changes the prefix too.
• Send related requests close together. A cache is more likely to still be there a few seconds later than an hour later.

Caches are kept for each model separately, so requests to different models never share one. The cache works in blocks of tokens, so the last few tokens of a prefix may be billed as normal input.

Check Your Cache Hits

usage.prompt_tokens_details.cached_tokens shows how many input tokens were cached:

"usage": {
  "prompt_tokens": 20512,
  "completion_tokens": 120,
  "total_tokens": 20632,
  "prompt_tokens_details": {"cached_tokens": 20480}
}

prompt_tokens counts all input tokens, cached or not. Here, 20,480 tokens are billed at the cached input price, and the other 32 at the normal input price.

How Long Caches Last

Caches aren't kept for a set amount of time. How long one lasts depends on load, demand, and available capacity. When space is needed, the caches used least recently are removed first.

A cache miss is never an error. The request works normally, and the prefix is billed at the normal input price.

Privacy and Encryption

See Privacy and ZDR.