Prompt Caching with Claude

Prompt caching lets Claude reuse the part of your prompt that repeats from request to request — tool definitions, a long system prompt, reference documents, the conversation so far. Cached input is billed at the cache-read rate, a small fraction of the normal input price, and it is also faster to process. For agents and long conversations it is usually the single largest saving available.

Run BiOS speaks Anthropic's caching syntax: the same cache_control field, the same placement rules and the same usage fields. Code written for Anthropic's API works here after a base-URL change. This page applies to every Claude model in the catalog — the ones whose model page lists Anthropic-compatible cache_control under Pricing → Prompt caching.

How it works

A cache entry is always a prefix of the prompt, read in this order: tools → system → messages. A cache_control breakpoint on a block means "cache everything up to and including this block". The next request that starts with exactly the same bytes reads that prefix back from the cache instead of processing it again.

The exact rates for each model are on its model page (Serverless → Models) and in the pricing block of GET /v1/models (cache_read, cache_write_5m, cache_write_1h). A write pays for itself as soon as the prefix is read back once: 1.25× + 0.1× is less than paying 1× twice.

What happens if you do nothing

On every Claude model (marked Platform-managed on its model page), Run BiOS places cache breakpoints for you when a request carries no cache_control of its own and the prompt is long enough to be worth caching: one at the end of the system prompt (which covers your tool definitions too) and, once a conversation has an assistant turn, one on the latest message. These use the 5-minute TTL, and you are never billed the write premium for a breakpoint the platform placed — you pay the normal input rate on the first request and the cache-read rate on the requests that hit.

• As soon as you place any breakpoint yourself, the platform places none. You are then in full control, and cache writes bill at the write rate for the TTL you chose.
• To switch the platform-managed breakpoints off without placing your own, send "prompt_cache": {"mode": "off"}.
• Place breakpoints yourself when you want a 1-hour cache, when you want a stable part of the prompt cached separately from a changing part, or when your prompts are shorter than the platform threshold but still repeat.

Automatic caching (one field)

The simplest way to control caching yourself: add cache_control at the top level of the request — on /v1/messages exactly as on Anthropic's API, and on /v1/chat/completions in the same shape. The breakpoint is placed on the last cacheable block of the request and moves forward on its own as the conversation grows, so a multi-turn chat re-reads everything it has already sent.

curl https://api.runbios.ai/v1/messages \
  -H "x-api-key: $RUNBIOS_API_KEY" \
  -H "anthropic-version: 2023-06-01" \
  -H "content-type: application/json" \
  -d '{
    "model": "claude-sonnet-5",
    "max_tokens": 1024,
    "cache_control": {"type": "ephemeral"},
    "system": "You are a support agent for Acme. <long policy document>",
    "messages": [{"role": "user", "content": "How do I reset my password?"}]
  }'

If the last block already carries its own cache_control, that one is kept and the top-level field adds nothing. The automatic breakpoint uses one of the four breakpoint slots.

Explicit breakpoints

For exact control, put "cache_control": {"type": "ephemeral"} on the block that ends the part you want cached. It is accepted on:

• Tool definitions — on the last tool, to cache the whole tool list.
• System blocks — send system as an array of text blocks to mark it.
• Message content blocks — text, images, and tool_result blocks, in user or assistant turns.

A typical agent caches the stable part (tools + instructions) once, and the growing conversation separately:

msg = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=1024,
    tools=[
        {"name": "search_docs", "description": "Search the knowledge base.",
         "input_schema": {"type": "object", "properties": {"q": {"type": "string"}}}},
        {"name": "open_ticket", "description": "Open a support ticket.",
         "input_schema": {"type": "object", "properties": {"title": {"type": "string"}}},
         "cache_control": {"type": "ephemeral"}},              # 1: caches all tools
    ],
    system=[
        {"type": "text", "text": "<long instructions and policies>",
         "cache_control": {"type": "ephemeral", "ttl": "1h"}}, # 2: tools + system, kept 1 hour
    ],
    messages=[
        {"role": "user", "content": "My invoice is wrong."},
        {"role": "assistant", "content": "Let me look that up."},
        {"role": "user", "content": [
            {"type": "text", "text": "It's invoice INV-1042.",
             "cache_control": {"type": "ephemeral"}},          # 3: the whole conversation so far
        ]},
    ],
)

• Up to 4 breakpoints per request, counting the automatic one.
• A breakpoint caches the prefix up to its block, so place it on the last block that stays identical between requests — not on content that changes every call.
• Only "type": "ephemeral" exists. Any other type, or a malformed value, is refused with a 400 naming the field.
• thinking blocks cannot carry a breakpoint (a 400); put it on the text block after them. They are still cached as part of the prefix when you pass them back.

Cache lifetime (TTL)

Entries live for 5 minutes by default, and every cache hit refreshes that timer at no extra cost. For traffic with longer gaps — a user who comes back after a coffee, a batch that runs every 20 minutes — ask for a 1-hour entry:

"cache_control": {"type": "ephemeral", "ttl": "1h"}

• ttl accepts "5m" (the default when omitted) or "1h". Anything else is a 400.
• A 1-hour write costs more (typically 2× input against 1.25×); reads cost the same. Use it when a 5-minute entry would expire before it is read again.
• When you mix lifetimes in one request, place 1-hour breakpoints before 5-minute ones.
• The lifetime counts from the start of the request that wrote or read the entry, so a long streamed answer uses up part of it.

Checking that it works

Every response reports what was written and what was read. Send the same prompt twice within the TTL:

// First request — the prefix is written to the cache
"usage": {
  "input_tokens": 8,
  "cache_creation_input_tokens": 13626,
  "cache_creation": {"ephemeral_5m_input_tokens": 13626, "ephemeral_1h_input_tokens": 0},
  "output_tokens": 8
}

// Second request — the same prefix is read back at the cache-read rate
"usage": {
  "input_tokens": 8,
  "cache_read_input_tokens": 13626,
  "output_tokens": 8
}

• The cache fields are omitted when zero (Anthropic's API sends explicit zeros). Official SDKs read them as 0 / None, so read them with a default.
• Your total prompt is input_tokens + cache_creation_input_tokens + cache_read_input_tokens.
• A request that carries cache_control also gets an X-BiOS-Cache-Marker response header: honoured when the model acts on your breakpoints, ignored when it caches on its own and your markers have no effect.
• A write with no later read usually means the prefix changed between requests — see Best practices below.

On the OpenAI-compatible endpoint

/v1/chat/completions accepts the same caching on Claude models. The one-field version is a top-level cache_control:

import os
from openai import OpenAI

client = OpenAI(base_url="https://api.runbios.ai/v1", api_key=os.environ["RUNBIOS_API_KEY"])

resp = client.chat.completions.create(
    model="claude-sonnet-5",
    max_tokens=1024,
    extra_body={"cache_control": {"type": "ephemeral"}},   # automatic caching
    messages=[
        {"role": "system", "content": "<long instructions>"},
        {"role": "user", "content": "My invoice is wrong."},
    ],
)
print(resp.usage.prompt_tokens_details)  # cached_tokens on the next request

For exact placement, put cache_control on a content part, or on a tool object next to function:

{
  "model": "claude-sonnet-5",
  "max_tokens": 1024,
  "tools": [{
    "type": "function",
    "function": {"name": "search_docs", "parameters": {"type": "object"}},
    "cache_control": {"type": "ephemeral"}
  }],
  "messages": [
    {"role": "system", "content": [
      {"type": "text", "text": "<long instructions>", "cache_control": {"type": "ephemeral", "ttl": "1h"}}
    ]},
    {"role": "user", "content": "My invoice is wrong."}
  ]
}

The counts come back in usage.prompt_tokens_details: cached_tokens (reads), plus cache_write_5m_tokens and cache_write_1h_tokens on a request that wrote. Here prompt_tokens already includes the cached tokens.

Claude Code and agent frameworks

Claude Code, the Anthropic SDKs and frameworks built on them already send cache_control breakpoints, so they cache on Run BiOS with no change beyond pointing them at https://api.runbios.ai with your key (for Claude Code: ANTHROPIC_BASE_URL and ANTHROPIC_API_KEY). Check cache_read_input_tokens in the responses to confirm.

Streaming, thinking and tool calls

Claude behaves here as it does on Anthropic's API. Three behaviours matter for latency and for clients with idle timeouts:

• Thinking is on by default on Claude 5 models (effort high; medium on Claude Opus 5.5), and thinking tokens bill as output. Lower it with output_config.effort (or reasoning_effort on chat completions), or turn it off with "thinking": {"type": "disabled"} where the model allows. On /v1/messages the thinking text is omitted by default, as on Anthropic — which also starts the answer sooner; send "thinking": {"type": "adaptive", "display": "summarized"} to receive it. On /v1/chat/completions the summary is returned in reasoning_content.
• Tool-call arguments. On a streaming /v1/chat/completions request the arguments stream as the model writes them, as they do on OpenAI. On /v1/messages Anthropic's default applies: each parameter is buffered and sent when complete, so a tool that writes a whole file can be silent for tens of seconds. Set "eager_input_streaming": true on that tool definition to stream it as it is generated (the JSON is then not validated until the end — parse it after the block closes).
• Keep-alives. While the model is silent, the stream carries a keep-alive every 15 seconds (event: ping on /v1/messages, an SSE comment on /v1/chat/completions). Give agent clients an idle timeout of a few minutes, not 30–60 seconds, so a long tool call is not abandoned.
• A failed or aborted request is not charged. If the stream ends in an error, or your client disconnects before the answer completes, the request is billed at zero, and the model request is cancelled the moment you disconnect.

Best practices

• Stable content first, changing content last. Tools, then instructions, then reference documents, then the conversation. Anything that changes per request — a timestamp, a user name, a request ID — belongs after the last breakpoint, or it breaks the cache for everything behind it.
• Keep the cached prefix byte-identical. Reordering tools, re-serialising JSON with different key order or whitespace, or editing an earlier message all produce a different prefix and a cache miss.
• Keep the model and settings fixed. Each model has its own cache, and changing the thinking settings or adding and removing images changes the prompt and misses the cache.
• Mind the minimum length. Each Claude model has a minimum cacheable prefix. A shorter prefix is served normally and simply not cached — no error and no write charge.
• Long conversations: a breakpoint finds earlier cache entries by looking back about 20 blocks. In a very long transcript, keep a breakpoint on the system prompt too, so the stable part still hits once the conversation outgrows that window.
• Parallel requests: an entry exists only once the first response starts. When fanning out many requests with the same prefix, send one first and the rest after it begins responding.

Errors