Serverless Inference API

Serverless inference is pay-per-token access to frontier models over one OpenAI-compatible API — no GPUs to size, no pods to manage, no idle capacity: you send a request, you are billed for the tokens it used. Base URL https://api.runbios.ai; if your code already talks to an OpenAI-compatible API, the change is the base URL, the key, and a model slug.

Quickstart

1. Create an API key with the serverless scope under Settings → API Keys with your workspace selected (see API Keys). A new key activates on the inference path within about a minute.
2. Pick a model slug from the catalog below.
3. Send a chat completion — pick your client:

curl https://api.runbios.ai/v1/chat/completions \
  -H "Authorization: Bearer bios-your_serverless_key" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "anthropic/claude-sonnet-5",
    "messages": [{"role": "user", "content": "Explain prompt caching in one sentence."}],
    "max_tokens": 256
  }'

The response is standard OpenAI chat-completion JSON, including a usage block with the exact tokens you were billed for:

{
  "id": "chatcmpl-01M0K4XW…",
  "object": "chat.completion",
  "created": 1787349102,
  "model": "anthropic/claude-sonnet-5",
  "choices": [
    {"index": 0, "message": {"role": "assistant", "content": "Prompt caching stores…"}, "logprobs": null, "finish_reason": "stop"}
  ],
  "usage": {"prompt_tokens": 20, "completion_tokens": 89, "total_tokens": 109}
}

Already using the OpenAI SDK? Point it at https://api.runbios.ai with your key and change nothing else. The full endpoint reference — streaming, the Anthropic-compatible dialect, errors — lives in Serverless Inference API.

Models & Pricing

Every model answers on the same endpoint; only the model field changes. Prices are per 1M tokens (input / output), in USD.

Cached read is the per-1M-token rate when a repeated prompt prefix is served from prompt caching — see Prompt Caching. A model with no cached-read rate of its own bills cached tokens at its input rate. Not sure which model to pick? BiOS Adaptive chooses for you.

Authentication

Send any API key that carries the serverless scope as a Bearer token. Keys are created under Settings → API Keys — see API Keys for scopes and rotation. A key without the scope is refused with 401.

Two timing details: the key must belong to a workspace (the console attaches one automatically when a workspace is selected — usage and billing are attributed to it), and a freshly minted key activates on the inference path within about a minute.

Authorization: Bearer bios-your_serverless_key

Chat Completions

Reasoning Effort & max_tokens

Models that reason take reasoning_effort. Each model publishes its own subset of the ladder — low, medium, high, xhigh, max — and GET /v1/models reports which. Two facts decide how you should set it:

Naming a level a model does not publish is not an error. It is served at the nearest level that model does publish, preferring the next one up: on a model whose ladder is [high], a request for medium runs at high; a request for low on a [medium, high] ladder runs at medium. If a model publishes nothing above the level you asked for, you get the highest level it does publish — never the lowest: on a [low, medium] ladder, a request for max runs at medium. Because the substitution can mean more thinking than you asked for — and thinking bills as output — the response says so on X-BiOS-Params-Adjusted: reasoning_effort, and your max_tokens still caps the total spend.

The same holds for turning reasoning off, and for models that cannot reason at all. "reasoning_effort": "none" is honoured wherever a model can be turned off. On a model that always reasons it cannot be, so instead of the model's default level you get the cheapest level it publishes — minimal when it has one, else low, and so on — which is the closest thing to "don't think" that model offers. And a reasoning_effort sent to a model that does not reason is no longer rejected: the parameter is dropped and you get the answer. Both are reported on X-BiOS-Params-Adjusted: reasoning_effort.

So the only value that errors is one outside the vocabulary — the six levels above plus none (with off and "" accepted as spellings of it). We will not guess which level "aggressive" meant. Every documented value is servable on every model.

• Thinking tokens count toward max_tokens and are billed as output tokens. max_tokens is a hard ceiling on total output — the thinking and the visible answer share it.
• low and medium minimise or skip thinking, which is what you want for latency-sensitive work. high, xhigh and max think, so they need a generous max_tokens: at xhigh or max, tens of thousands of tokens is a sensible starting point rather than a few hundred.

The consequence to plan for: a small max_tokens at a thinking effort can be consumed entirely by thinking, and the answer comes back empty. That response is a 200 with finish_reason: "length", content: "", and completion_tokens at your ceiling — measured on anthropic/claude-sonnet-5 at reasoning_effort: "max" with max_tokens: 1200, which returned reasoning and no answer text. Read choices[0].message.content rather than assuming a 2xx carries an answer, and on finish_reason: "length" raise max_tokens or lower the effort. Your max_tokens is never rewritten for you; if the routed model's context window cannot hold the number you asked for, it is reduced to fit and the response says so in X-BiOS-Params-Adjusted: max_tokens.

{
  "model": "anthropic/claude-sonnet-5",
  "messages": [{"role": "user", "content": "Prove this bound, showing your working."}],
  "reasoning_effort": "max",
  "max_tokens": 40960
}

Reasoning can also be turned off: "reasoning_effort": "none" on a model that allows it. With reasoning off no thinking tokens are generated and none are billed, and the whole of max_tokens is available to the answer. A model whose reasoning cannot be disabled reports that in its catalog entry and keeps reasoning regardless.

Streaming

With "stream": true the response is text/event-stream: one data: line per token delta, a final chunk carrying usage, then data: [DONE]. A failure before the first event arrives as a non-2xx JSON error.

data: {"id":"chatcmpl-01M0K4XW…","object":"chat.completion.chunk","model":"anthropic/claude-sonnet-5","choices":[{"index":0,"delta":{"content":"Prompt"},"finish_reason":null}]}

data: {"id":"chatcmpl-01M0K4XW…",…,"choices":[{"index":0,"delta":{"content":" caching"},"finish_reason":null}]}

data: {"id":"chatcmpl-01M0K4XW…",…,"choices":[{"index":0,"delta":{},"finish_reason":"stop"}],"usage":{"prompt_tokens":20,"completion_tokens":89,"total_tokens":109}}

data: [DONE]

Tool Calling

Declare callable functions in tools and the model answers with structured tool_calls when it wants to invoke one. You run the function, then continue the conversation with a role: "tool" message carrying the result.

curl https://api.runbios.ai/v1/chat/completions \
  -H "Authorization: Bearer bios-your_serverless_key" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "anthropic/claude-sonnet-5",
    "messages": [{"role": "user", "content": "What is the weather in Paris?"}],
    "tools": [{
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get the current weather for a city",
        "parameters": {
          "type": "object",
          "properties": {"city": {"type": "string"}},
          "required": ["city"]
        }
      }
    }],
    "tool_choice": "auto"
  }'

When the model decides to call it, the response carries the call — note finish_reason: "tool_calls":

{
  "model": "anthropic/claude-sonnet-5",
  "choices": [{
    "index": 0,
    "message": {
      "role": "assistant",
      "content": null,
      "tool_calls": [{
        "id": "toolu_01X7…",
        "type": "function",
        "function": {"name": "get_weather", "arguments": "{\"city\":\"Paris\"}"}
      }]
    },
    "finish_reason": "tool_calls"
  }],
  "usage": {"prompt_tokens": 214, "completion_tokens": 18, "total_tokens": 232}
}

• Send the result back as {"role": "tool", "tool_call_id": "toolu_01X7…", "content": "…"} to continue the loop.
• parallel_tool_calls defaults to true — set it false to get at most one call per turn.
• Streaming: tool arguments arrive as fragments — concatenate tool_calls[].delta entries by their index; the completed call ends with finish_reason: "tool_calls".
• On the Anthropic dialect (/v1/messages) the same feature is native tools / tool_use blocks.

JSON Output

Add "response_format": {"type": "json_object"} to receive the answer as JSON text. The field constrains the output surface — still instruct the model in your prompt to produce JSON (and the shape you want); enforcement strength varies by model.

Vision (Images)

Models whose /v1/models row has supports_vision: true accept images as content parts, alongside text:

{
  "model": "anthropic/claude-sonnet-5",
  "messages": [{
    "role": "user",
    "content": [
      {"type": "text", "text": "What is in this image?"},
      {"type": "image_url", "image_url": {"url": "data:image/png;base64,iVBORw0K…"}}
    ]
  }],
  "max_tokens": 256
}

Send images as base64 data URIs. Hosted image URLs work only on some models — where unsupported, the request is refused with a 400 naming the limitation.

Anthropic-Compatible Messages

POST /v1/messages is the same catalog in the Anthropic Messages dialect, so Anthropic-SDK clients — including Claude Code — work with a base-URL change:

from anthropic import Anthropic

client = Anthropic(
    base_url="https://api.runbios.ai",
    api_key="bios-your_serverless_key",
)

msg = client.messages.create(
    model="anthropic/claude-sonnet-5",
    max_tokens=256,
    messages=[{"role": "user", "content": "Hello"}],
)
print(msg.content)

Dialect notes: reasoning is steered with the Anthropic thinking / output_config.effort fields here (not reasoning_effort), and parameters that belong to the other dialect are rejected with a 400 rather than silently ignored.

List Models

The catalog is public: no key is needed to list the serverless models or read their prices, so trackers and aggregators can follow what we serve. Every row carries a pricing block in USD per 1M tokens — input, output, and, where a model prices them separately, cache_read, cache_write_5m and cache_write_1h (each a min/max band; an absent cache block means cached tokens bill at the input rate). Send a key and the list also includes your workspace deployments. GET /v1/models/{model} retrieves one row. The same two paths serve the Anthropic SDK: client.models.list() sends x-api-key and anthropic-version and receives Anthropic's page envelope (data, has_more, first_id, last_id); the OpenAI SDK's client.models.list() receives the OpenAI list below.

Prompt Caching

Repeated prompt prefixes are served from cache automatically on supported models — no request changes needed. Cached tokens are billed at the much lower cache-read rate listed in Models & Pricing above, and the usage block breaks them out per request. On Claude models you control caching with Anthropic's cache_control — a top-level field for automatic caching, or breakpoints on tools, system and message blocks with a 5-minute or 1-hour lifetime. The full guide, with examples for both endpoints: Prompt Caching with Claude.

Every turn of a conversation goes to the same cache. Requests are routed by conversation, so turn two reads what turn one cached, while different conversations spread across capacity. A conversation is identified by its system prompt and first message; to group requests explicitly, send the same session_id (body, chat completions) or x-session-id header, or your own prompt_cache_key on OpenAI models.

OpenAI GPT-5.6 and later bill a cache write at 1.25× the input rate on every prompt of 1,024+ tokens, even without opting in. For a prompt you will not send again, turn caching off with prompt_cache_options {"mode": "explicit"} and no breakpoint; to cache only a stable prefix, mark its last block with prompt_cache_breakpoint. A Claude-style cache_control block works too and is sent as a breakpoint. On other models these fields are accepted and have no effect.

{
  "model": "openai/gpt-6-luna",
  "prompt_cache_options": {"mode": "explicit"},
  "messages": [
    {"role": "system", "content": [
      {"type": "text", "text": "<stable instructions>",
       "prompt_cache_breakpoint": {"mode": "explicit"}}
    ]},
    {"role": "user", "content": "<this turn's question>"}
  ]
}

Errors & Rate Limits

Error bodies on this surface use the OpenAI error object on /v1/chat/completions and Anthropic’s on /v1/messages — neither is the platform API’s shape. All three are written out in Overview & Authentication. Tokens are debited from your wallet per request; balance and history live under Billing & Wallet.

In the Console

• Serverless → Models — browse the live catalog with per-model details and pricing.
• Serverless → Playground — try any model (or BiOS Adaptive, the default) with your own prompts; the playground uses your key and bills like the API.
• Serverless → Limits — the per-workspace request and token ceilings your keys run under.
• Analytics — per-model and per-key token usage and spend over time.

Serverless vs Deployments