Serverless Inference API

Serverless inference is pay-per-token access to frontier models over one OpenAI-compatible API — no GPUs to size, no pods to manage, no idle capacity: you send a request, you are billed for the tokens it used. Base URL https://api.runbios.ai; if your code already talks to an OpenAI-compatible API, the change is the base URL, the key, and a model slug.

Quickstart

1. Create an API key with the serverless scope under Settings → API Keys with your workspace selected (see API Keys). A new key activates on the inference path within about a minute.
2. Pick a model slug from the catalog below.
3. Send a chat completion — pick your client:

curl https://api.runbios.ai/v1/chat/completions \
  -H "Authorization: Bearer bios-your_serverless_key" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-5",
    "messages": [{"role": "user", "content": "Explain prompt caching in one sentence."}],
    "max_tokens": 256
  }'

The response is standard OpenAI chat-completion JSON, including a usage block with the exact tokens you were billed for:

{
  "id": "chatcmpl-01M0K4XW…",
  "object": "chat.completion",
  "created": 1787349102,
  "model": "claude-sonnet-5",
  "choices": [
    {"index": 0, "message": {"role": "assistant", "content": "Prompt caching stores…"}, "logprobs": null, "finish_reason": "stop"}
  ],
  "usage": {"prompt_tokens": 20, "completion_tokens": 89, "total_tokens": 109}
}

Already using the OpenAI SDK? Point it at https://api.runbios.ai with your key and change nothing else. The full endpoint reference — streaming, the Anthropic-compatible dialect, errors — lives in Serverless Inference API.

Models & Pricing

Every model answers on the same endpoint; only the model field changes. Prices are per 1M tokens (input / output), in USD.

Cached read is the per-1M-token rate when a repeated prompt prefix is served from prompt caching — see Prompt Caching. A model with no cached-read rate of its own bills cached tokens at its input rate. Not sure which model to pick? BiOS Adaptive chooses for you.

Authentication

Send any API key that carries the serverless scope as a Bearer token. Keys are created under Settings → API Keys — see API Keys for scopes and rotation. A key without the scope is refused with 401.

Two timing details: the key must belong to a workspace (the console attaches one automatically when a workspace is selected — usage and billing are attributed to it), and a freshly minted key activates on the inference path within about a minute.

Authorization: Bearer bios-your_serverless_key

Chat Completions

Reasoning Effort & max_tokens

Models that reason take reasoning_effort. Each model publishes its own subset of the ladder — low, medium, high, xhigh, max — and GET /v1/models reports which. Two facts decide how you should set it:

Naming a level a model does not publish is not an error. It is served at the nearest level that model does publish, preferring the next one up: on a model whose ladder is [high], a request for medium runs at high; a request for low on a [medium, high] ladder runs at medium. Only when nothing deeper is published does it fall the other way. Because the substitution can mean more thinking than you asked for — and thinking bills as output — the response says so on X-BiOS-Params-Adjusted: reasoning_effort, and your max_tokens still caps the total spend. Two things remain errors: a level outside the ladder vocabulary, and any reasoning_effort on a model that does not reason. A request for none is never substituted — off stays off.

• Thinking tokens count toward max_tokens and are billed as output tokens. max_tokens is a hard ceiling on total output — the thinking and the visible answer share it.
• low and medium minimise or skip thinking, which is what you want for latency-sensitive work. high, xhigh and max think, so they need a generous max_tokens: at xhigh or max, tens of thousands of tokens is a sensible starting point rather than a few hundred.

The consequence to plan for: a small max_tokens at a thinking effort can be consumed entirely by thinking, and the answer comes back empty. That response is a 200 with finish_reason: "length", content: "", and completion_tokens at your ceiling — measured on claude-sonnet-5 at reasoning_effort: "max" with max_tokens: 1200, which returned reasoning and no answer text. Read choices[0].message.content rather than assuming a 2xx carries an answer, and on finish_reason: "length" raise max_tokens or lower the effort. Your max_tokens is never rewritten for you; if the routed model's context window cannot hold the number you asked for, it is reduced to fit and the response says so in X-BiOS-Params-Adjusted: max_tokens.

{
  "model": "claude-sonnet-5",
  "messages": [{"role": "user", "content": "Prove this bound, showing your working."}],
  "reasoning_effort": "max",
  "max_tokens": 40960
}

Reasoning can also be turned off: "reasoning_effort": "none" on a model that allows it. With reasoning off no thinking tokens are generated and none are billed, and the whole of max_tokens is available to the answer. A model whose reasoning cannot be disabled reports that in its catalog entry and keeps reasoning regardless.

Streaming

With "stream": true the response is text/event-stream: one data: line per token delta, a final chunk carrying usage, then data: [DONE]. A failure before the first event arrives as a non-2xx JSON error.

data: {"id":"chatcmpl-01M0K4XW…","object":"chat.completion.chunk","model":"claude-sonnet-5","choices":[{"index":0,"delta":{"content":"Prompt"},"finish_reason":null}]}

data: {"id":"chatcmpl-01M0K4XW…",…,"choices":[{"index":0,"delta":{"content":" caching"},"finish_reason":null}]}

data: {"id":"chatcmpl-01M0K4XW…",…,"choices":[{"index":0,"delta":{},"finish_reason":"stop"}],"usage":{"prompt_tokens":20,"completion_tokens":89,"total_tokens":109}}

data: [DONE]

Tool Calling

Declare callable functions in tools and the model answers with structured tool_calls when it wants to invoke one. You run the function, then continue the conversation with a role: "tool" message carrying the result.

curl https://api.runbios.ai/v1/chat/completions \
  -H "Authorization: Bearer bios-your_serverless_key" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-5",
    "messages": [{"role": "user", "content": "What is the weather in Paris?"}],
    "tools": [{
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get the current weather for a city",
        "parameters": {
          "type": "object",
          "properties": {"city": {"type": "string"}},
          "required": ["city"]
        }
      }
    }],
    "tool_choice": "auto"
  }'

When the model decides to call it, the response carries the call — note finish_reason: "tool_calls":

{
  "model": "claude-sonnet-5",
  "choices": [{
    "index": 0,
    "message": {
      "role": "assistant",
      "content": null,
      "tool_calls": [{
        "id": "toolu_01X7…",
        "type": "function",
        "function": {"name": "get_weather", "arguments": "{\"city\":\"Paris\"}"}
      }]
    },
    "finish_reason": "tool_calls"
  }],
  "usage": {"prompt_tokens": 214, "completion_tokens": 18, "total_tokens": 232}
}

• Send the result back as {"role": "tool", "tool_call_id": "toolu_01X7…", "content": "…"} to continue the loop.
• parallel_tool_calls defaults to true — set it false to get at most one call per turn.
• Streaming: tool arguments arrive as fragments — concatenate tool_calls[].delta entries by their index; the completed call ends with finish_reason: "tool_calls".
• On the Anthropic dialect (/v1/messages) the same feature is native tools / tool_use blocks.

JSON Output

Add "response_format": {"type": "json_object"} to receive the answer as JSON text. The field constrains the output surface — still instruct the model in your prompt to produce JSON (and the shape you want); enforcement strength varies by model.

Vision (Images)

Models whose /v1/models row has supports_vision: true accept images as content parts, alongside text:

{
  "model": "claude-sonnet-5",
  "messages": [{
    "role": "user",
    "content": [
      {"type": "text", "text": "What is in this image?"},
      {"type": "image_url", "image_url": {"url": "data:image/png;base64,iVBORw0K…"}}
    ]
  }],
  "max_tokens": 256
}

Send images as base64 data URIs. Hosted image URLs work only on some models — where unsupported, the request is refused with a 400 naming the limitation.

Anthropic-Compatible Messages

POST /v1/messages is the same catalog in the Anthropic Messages dialect, so Anthropic-SDK clients — including Claude Code — work with a base-URL change:

from anthropic import Anthropic

client = Anthropic(
    base_url="https://api.runbios.ai",
    api_key="bios-your_serverless_key",
)

msg = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=256,
    messages=[{"role": "user", "content": "Hello"}],
)
print(msg.content)

Dialect notes: reasoning is steered with the Anthropic thinking / output_config.effort fields here (not reasoning_effort), and parameters that belong to the other dialect are rejected with a 400 rather than silently ignored.

List Models

The catalog is public: no key is needed to list the serverless models or read their prices, so trackers and aggregators can follow what we serve. Every row carries a pricing block in USD per 1M tokens — input, output, and, where a model prices them separately, cache_read, cache_write_5m and cache_write_1h (each a min/max band; an absent cache block means cached tokens bill at the input rate). Send a key and the list also includes your workspace deployments. GET /v1/models/{model} retrieves one row. The same two paths serve the Anthropic SDK: client.models.list() sends x-api-key and anthropic-version and receives Anthropic's page envelope (data, has_more, first_id, last_id); the OpenAI SDK's client.models.list() receives the OpenAI list below.

Prompt Caching

Repeated prompt prefixes are served from cache automatically on supported models — no request changes needed. Cached tokens are billed at the much lower cache-read rate listed in Models & Pricing above, and the usage block breaks them out per request. On Claude models you can additionally pin cache breakpoints explicitly with Anthropic-style cache_control markers.

Errors & Rate Limits

Error bodies on this surface use the OpenAI error object on /v1/chat/completions and Anthropic’s on /v1/messages — neither is the platform API’s shape. All three are written out in Overview & Authentication. Tokens are debited from your wallet per request; balance and history live under Billing & Wallet.

In the Console

• Serverless → Models — browse the live catalog with per-model details and pricing.
• Serverless → Playground — try any model (or BiOS Adaptive, the default) with your own prompts; the playground uses your key and bills like the API.
• Serverless → Limits — the per-workspace request and token ceilings your keys run under.
• Analytics — per-model and per-key token usage and spend over time.

Serverless vs Deployments