Serverless Inference API

Serverless inference is pay-per-token access to frontier models over one OpenAI-compatible API — no GPUs to size, no pods to manage, no idle capacity: you send a request, you are billed for the tokens it used. Base URL https://api.runbios.ai; if your code already talks to an OpenAI-compatible API, the change is the base URL, the key, and a model slug.

Quickstart

1. Create an API key with the serverless scope under Settings → API Keys with your workspace selected (see API Keys). A new key activates on the inference path within about a minute.
2. Pick a model slug from the catalog below.
3. Send a chat completion — pick your client:

curl https://api.runbios.ai/v1/chat/completions \
  -H "Authorization: Bearer bios-your_serverless_key" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-5",
    "messages": [{"role": "user", "content": "Explain prompt caching in one sentence."}],
    "max_tokens": 256
  }'

The response is standard OpenAI chat-completion JSON, including a usage block with the exact tokens you were billed for:

{
  "id": "chatcmpl-01M0K4XW…",
  "object": "chat.completion",
  "created": 1787349102,
  "model": "claude-sonnet-5",
  "choices": [
    {"index": 0, "message": {"role": "assistant", "content": "Prompt caching stores…"}, "logprobs": null, "finish_reason": "stop"}
  ],
  "usage": {"prompt_tokens": 20, "completion_tokens": 89, "total_tokens": 109}
}

Already using the OpenAI SDK? Point it at https://api.runbios.ai with your key and change nothing else. The full endpoint reference — streaming, the Anthropic-compatible dialect, errors — lives in Serverless Inference API.

Models & Pricing

Every model answers on the same endpoint; only the model field changes. Prices are per 1M tokens (input / output), in USD.

Cached read is the per-1M-token rate when a repeated prompt prefix is served from prompt caching — see Prompt Caching. The authoritative live list, with context windows and current availability, is GET /v1/models. Not sure which model to pick? BiOS Adaptive chooses for you.

Authentication

Send any API key that carries the serverless scope as a Bearer token. Keys are created under Settings → API Keys — see API Keys for scopes and rotation. A key without the scope is refused with 401.

Two timing details: the key must belong to a workspace (the console attaches one automatically when a workspace is selected — usage and billing are attributed to it), and a freshly minted key activates on the inference path within about a minute.

Authorization: Bearer bios-your_serverless_key

Chat Completions

Streaming

With "stream": true the response is text/event-stream: one data: line per token delta, a final chunk carrying usage, then data: [DONE]. A failure before the first event arrives as a non-2xx JSON error.

data: {"id":"chatcmpl-01M0K4XW…","object":"chat.completion.chunk","model":"claude-sonnet-5","choices":[{"index":0,"delta":{"content":"Prompt"},"finish_reason":null}]}

data: {"id":"chatcmpl-01M0K4XW…",…,"choices":[{"index":0,"delta":{"content":" caching"},"finish_reason":null}]}

data: {"id":"chatcmpl-01M0K4XW…",…,"choices":[{"index":0,"delta":{},"finish_reason":"stop"}],"usage":{"prompt_tokens":20,"completion_tokens":89,"total_tokens":109}}

data: [DONE]

Tool Calling

Declare callable functions in tools and the model answers with structured tool_calls when it wants to invoke one. You run the function, then continue the conversation with a role: "tool" message carrying the result.

curl https://api.runbios.ai/v1/chat/completions \
  -H "Authorization: Bearer bios-your_serverless_key" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-5",
    "messages": [{"role": "user", "content": "What is the weather in Paris?"}],
    "tools": [{
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get the current weather for a city",
        "parameters": {
          "type": "object",
          "properties": {"city": {"type": "string"}},
          "required": ["city"]
        }
      }
    }],
    "tool_choice": "auto"
  }'

When the model decides to call it, the response carries the call — note finish_reason: "tool_calls":

{
  "model": "claude-sonnet-5",
  "choices": [{
    "index": 0,
    "message": {
      "role": "assistant",
      "content": null,
      "tool_calls": [{
        "id": "toolu_01X7…",
        "type": "function",
        "function": {"name": "get_weather", "arguments": "{\"city\":\"Paris\"}"}
      }]
    },
    "finish_reason": "tool_calls"
  }],
  "usage": {"prompt_tokens": 214, "completion_tokens": 18, "total_tokens": 232}
}

• Send the result back as {"role": "tool", "tool_call_id": "toolu_01X7…", "content": "…"} to continue the loop.
• parallel_tool_calls defaults to true — set it false to get at most one call per turn.
• Streaming: tool arguments arrive as fragments — concatenate tool_calls[].delta entries by their index; the completed call ends with finish_reason: "tool_calls".
• On the Anthropic dialect (/v1/messages) the same feature is native tools / tool_use blocks.

JSON Output

Add "response_format": {"type": "json_object"} to receive the answer as JSON text. The field constrains the output surface — still instruct the model in your prompt to produce JSON (and the shape you want); enforcement strength varies by model.

Vision (Images)

Models whose /v1/models row has supports_vision: true accept images as content parts, alongside text:

{
  "model": "claude-sonnet-5",
  "messages": [{
    "role": "user",
    "content": [
      {"type": "text", "text": "What is in this image?"},
      {"type": "image_url", "image_url": {"url": "data:image/png;base64,iVBORw0K…"}}
    ]
  }],
  "max_tokens": 256
}

Send images as base64 data URIs. Hosted image URLs work only on some models — where unsupported, the request is refused with a 400 naming the limitation.

Anthropic-Compatible Messages

POST /v1/messages is the same catalog in the Anthropic Messages dialect, so Anthropic-SDK clients — including Claude Code — work with a base-URL change:

from anthropic import Anthropic

client = Anthropic(
    base_url="https://api.runbios.ai",
    api_key="bios-your_serverless_key",
)

msg = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=256,
    messages=[{"role": "user", "content": "Hello"}],
)
print(msg.content)

Dialect notes: reasoning is steered with the Anthropic thinking / output_config.effort fields here (not reasoning_effort), and parameters that belong to the other dialect are rejected with a 400 rather than silently ignored.

List Models

Prompt Caching

Repeated prompt prefixes are served from cache automatically on supported models — no request changes needed. Cached tokens are billed at the much lower cache-read rate listed in Models & Pricing above, and the usage block breaks them out per request. On Claude models you can additionally pin cache breakpoints explicitly with Anthropic-style cache_control markers.

Errors & Rate Limits

Error bodies follow the platform-wide JSON error shape — see Overview & Authentication. Tokens are debited from your wallet per request; balance and history live under Billing & Wallet.

In the Console

• Serverless → Models — browse the live catalog with per-model details and pricing.
• Serverless → Playground — try any model (or BiOS Adaptive, the default) with your own prompts; the playground uses your key and bills like the API.
• Serverless → Limits — the per-workspace request and token ceilings your keys run under.
• Analytics — per-model and per-key token usage and spend over time.

Serverless vs Deployments