Advanced Model Control

A few request settings change how the model writes its reply. They are all optional. If you leave one out, the model's default is used.

{
  "model": "tuna-us/example-lab/example-model-id",
  "messages": [{"role": "user", "content": "Suggest a name for an ocean research vessel."}],
  "max_tokens": 100,
  "temperature": 1.2,
  "stop": ["\n\n"]
}

How the Model Picks Each Token

A model writes its reply one token at a time. It assigns probabilities to possible next tokens, then picks one at random using those probabilities. temperature, top_p, and the penalties change that last step. They only affect how a token is chosen from those probabilities, and they can't add knowledge the model doesn't already have.

max_tokens

max_tokens caps the length of the reply. If the reply reaches it, the reply is cut off, and finish_reason is length instead of stop.

• Each model's limit is documented on the Models & Pricing page. If you ask for more, Tuna lowers max_tokens to that limit.
• If you leave it out, max_tokens defaults to the model's maximum output. The setting can only reduce that limit, which can help control costs.
• For reasoning models, the model's thinking counts toward max_tokens too.
• max_completion_tokens is another name for the same setting, from newer versions of the OpenAI API. Some tools send it instead.

A low max_tokens doesn't make the model write shorter answers. It only cuts them off. To get short answers, ask for them in the prompt.

temperature

temperature controls how random the pick is.

• Low values, like 0.2, make the model almost always pick its most likely token. Replies are focused and predictable. This suits code, data extraction, and answers with one right answer.
• High values, like 1.2, spread the pick across more tokens. Replies are more varied. This suits brainstorming and creative writing.
• At 0, the model picks the most likely next token. Replies to the same request are then usually the same, although an exact match is not guaranteed.

Very high values, above about 1.5, often make replies wander or stop making sense.

top_p

top_p limits the pick to the most likely tokens whose probabilities add up to top_p. At 0.9, the least likely tokens, which make up the last 10% of the probability, are never picked.

Values like 0.9 or 0.95 are common.

top_p and temperature both control randomness. Change one of them and leave the other at its default.

stop

stop is a string, or a list of strings, that ends the reply as soon as the model writes one of them. The stop text itself isn't included in the reply, and finish_reason is stop.

For example, "stop": ["\n\n"] ends the reply at the first blank line.

presence_penalty and frequency_penalty

Both penalties lower the chance of tokens that are already in the reply. Positive values push away from repeats, and negative values make repeats more likely.

• presence_penalty lowers the chance of any token that has appeared at least once. This nudges the model toward new words and topics.
• frequency_penalty lowers the chance of a token more each time it appears. This helps when a model repeats the same phrase over and over.

Small values, like 0.1 to 0.5, are usually enough. Large values can make the model avoid words it needs.