Conscious Loop

Conscious Loop in 2 minutes

A pipeline keeps one model your app uses getting better, by itself, from your app’s own conversations. The live version is the version your app gets answers from right now. Each time the pipeline has learned enough, it makes a new version, tests it, and puts it live when it does better.

Your app sends its conversations (each one is a sample), and you say which answers were good and which were wrong. With enough new samples the pipeline trains, and the training is tested on questions it never saw. When it does better than the live version, or a member approves it, it becomes the next version, and your app gets its answers from then on, with no change to your code.

Start in three steps.

1. Create a pipeline: New pipeline on the Pipelines page. Give it a name and choose a model; everything else has a default. Saving it lets it train by itself and pay from your wallet for what its trainings and tests use.
2. Add the pipeline ID header to your app’s call: one more header, X-Loop-Pipeline, with the pipeline ID, on the model call your app already makes. The pipeline ID is made from the name (Support replies becomes support-replies) and never changes.
3. Send Good / Wrong / Wrong + right answer: each reply carries a sample ID; send your feedback with it. When 1,100 samples have good feedback (1,000 to learn from and 100 kept aside to test it), the first training starts. When it passes its test it becomes v1: point your app at the model loop-support-replies.

What it costs: each training pays for the machine time to train it and test it, and for the judge, at most $25 a training; the live version’s machine is billed by the hour while it is switched on (Costs, privacy and stopping).

The words

Every loop page uses these words, each for one thing:

The Pipelines list

Conscious Loop in the dashboard sidebar opens your pipelines. Open one to see it and change it.

• The menu: Conscious Loop in the dashboard sidebar takes you to your pipelines. On every Conscious Loop page it stays open and shows two pages: Pipelines (all your pipelines; open one to see and change it) and Datasets. The page you are on is highlighted; the arrow next to it hides or shows the two pages.
• Pipelines: one row each, in four columns: the pipeline and its model; its live version and how much of its own test it got right (v5 · 97% right), or No version yet, with a dot that says whether it answers your app right now (green On; amber Starting or Paused; grey Off when its machine is stopped or was deleted); what it is doing now (Collecting samples: 921 of 1,000 for the next training, Training #7 · 18 min, Testing training #7, Needs you: approve training #7, Paused); and the last thing that changed. The live version is the version your app gets answers from right now. Search them by name, pipeline ID or model, or pick a filter (Training now, Needs you, Serving, Paused); each filter says how many pipelines it holds in the whole workspace. Needs you holds the pipelines with a training waiting for your approval or for funds; a pipeline held for another reason says what to do in its own line, for example Press Train now: training #7 ended without a result. While a pipeline is training, being tested, waiting or going live, the list updates itself every 30 seconds. New pipeline starts one; Every training, under the list, shows every pipeline’s trainings, newest first, and can be narrowed to one pipeline or one result.

A pipeline’s Overview

A pipeline has six tabs: Overview, Versions, Samples, Tests, Connect and Settings. The Overview is five lines and one chart: Live (the version your app gets answers from right now, and whether its machine is switched on), How good (how much of its automatic test it got right, beside the version before it on the same questions, and your main benchmark), how each version scored, So far (every training and version, and what they cost) and Next (when the next training starts, and Train now, which skips the schedule, never the minimum).

Connect your app

Your app sends its conversations to a pipeline by adding one header to the model call it already makes; nothing else has to be switched on. The pipeline’s Connect tab has all of it ready to copy: the pipeline ID, the model name, the API key it needs, and code in Python, TypeScript and cURL, in three steps.

1. Send your app's conversations: add the header X-Loop-Pipeline with the pipeline ID to each call. The reply carries a sample ID in its headers (X-Loop-Sample-Id); keep it to send feedback. The SDKs hand it to you, so nobody reads headers by hand. It works with any model you call through Run BiOS, streamed or not.
2. Tell it how each answer did: with the sample ID, say Good, Wrong, or Wrong + right answer. Feedback sent the instant an answer arrives can reach us before the conversation is saved: wait 2 seconds and send it again (the SDKs do). Changed your mind? Send new feedback; the newest one counts.
3. Use the improved model: once v1 is live, call the model loop-support-replies. Each new version takes over that name automatically, so you change no code when one goes live. Keep the header on these calls too, so they keep teaching it.

# pip install runbios-sdk
from bios import RunBiOS

# Reads RUNBIOS_API_KEY.
client = RunBiOS(base_url="https://api.runbios.ai")

reply = client.inference.chat_completions(
    model="your-current-model",
    messages=[{"role": "user", "content": "How long do refunds take?"}],
    loop={"pipeline": "support-replies"},
)
answer = reply["choices"][0]["message"]["content"]

# The sample ID came back in the reply's headers.
# Keep it to send feedback (step 2).
# None means this call was not saved.
sample_id = reply.loop_sample_id

# sample_id is the one from step 1.
# None: the call was not saved.
if sample_id:
    # The answer was right:
    client.loop.upvote(sample_id)

    # It was wrong:
    # client.loop.downvote(sample_id)

    # Wrong, and this is the right answer:
    # client.loop.correct(sample_id, "Refunds take five working days.")

Send a test message, above the steps, asks the model something and shows the answer, its sample ID and the three feedback buttons, so you can watch one sample arrive before you change any code. It is a real call, billed like any other, and it becomes a sample of the pipeline.

Below the steps, three other ways to send conversations: Import a file of past conversations, one per line; Record every call of a deployment, with no header; and Send a conversation yourself, for a model that runs outside Run BiOS.

Nothing is saved without your say-so: a call is saved only when it carries the pipeline ID header, or when you switched recording on for its model. To keep one call out whatever else it says, add the header X-Loop-Capture: off. The Connect tab says when the last sample arrived, and which calls were not saved, and why.

For developers: recording, feedback and import in full, with every header and refusal

What your feedback does

Each kind of feedback does one thing, the same as the table on the Connect tab says:

That last row is with automatic scoring on, as it is unless you switch it off in Settings (Scoring unreviewed samples); with it off, a sample with no feedback is not learned until someone gives it some. A person’s feedback always wins.

Training that learns from rankings or rewards (like DPO or GRPO) is not available yet; every version learns from your good and corrected answers.

Models that think before they answer (thinking models) answer with their thinking switched off in a pipeline’s versions and in its tests: the pipeline learns from your app’s answers, which carry no thinking. Samples and their feedback are kept 30 days by default, and imported ones 365 days. Right answers are cleaned of emails, long numbers and keys before they are saved.

Samples and datasets

A sample is one conversation your app had (what was asked and what the model answered, tool calls included), plus your feedback. A pipeline learns from its own samples and from any datasets you add to it, and only from the ones marked Good or Wrong + right answer, or picked by automatic scoring. One marked Wrong is kept and shown but teaches nothing.

• The Samples tab counts them (total, Good, Wrong, Wrong + right answer, No feedback, and For the automatic test: the ones kept aside to test each training; where versions build on each other, also learned and not yet learned) and lists them 25 at a time, filtered by All, Good, Wrong, Wrong + right answer or No feedback, by tag when the pipeline has more than one, or by a search of their words (2 to 200 characters). A sample kept in your own storage cannot be searched, and the tab says how many a search left out. A chart shows how many arrived each day over the last 30 days, by feedback.
• One sample opens the whole conversation: give feedback, change its tags, see which pipelines it teaches, or delete it. A sample can teach several pipelines.
• A correction to a sample a version already learned is learned by the next training. The old answer is not unlearned; it fades as the right one is learned.
• Deleting a sample takes it out of every later training, and rebuilds a live version that learned it (Deleting a sample a version learned).

For developers: tags, import formats and what each kind of feedback trains

Datasets

A dataset is a saved set of conditions that picks samples. It is not a copy: a sample that matches later joins it. Every pipeline already learns from its own samples; add a dataset to a pipeline to teach it that dataset’s samples too (Settings → What it trains on). One dataset can feed any number of pipelines. Make one on the Datasets page of the Conscious Loop menu.

• The conditions are all optional, and every one you set must hold: tags, feedback (Good, Wrong, Wrong + right answer or No feedback), where it came from (your app, or imported files), when it was saved, and the model that answered. With none, a dataset picks every sample in the workspace.
• The Datasets page starts with every sample in the workspace: how many, how many have feedback and how many can be learned from. Each dataset has its own page: what it includes, who saved it, which pipelines use it and which trainings learned from it, its samples, and Download. Its counts are taken before repeated and over-long conversations are set aside for training. A sample in one of your benchmarks is counted, and is never learned from.
• A change applies from each pipeline’s next training. Nothing is trained again because of it, and versions already made keep what they learned.
• Deleting a dataset takes it off every pipeline that used it, from their next training. Its samples stay.

Scoring unreviewed samples

When a pipeline does not yet have the samples its next training needs, the judge also scores samples nobody has reviewed, newest first. One it scores 0.8 or more is learned like one marked Good; one below is not. Your feedback always decides: a sample a person reviewed is never scored, and reviewing one it scored replaces its score. Scored samples stay under No feedback on the Samples tab: Picked by the judge, Below the bar and Not scored yet split them. Scoring stops as soon as the next training has what it needs, and what it cost shows on that training as Scoring unreviewed samples. It is on by default; the switch is in Settings.

Versions

Every time a pipeline trains, that training is numbered (Training #1, #2 and on) and tested. A training that goes live becomes the next version; one that does not changes nothing in your app. Version numbers count up and never change, and there is no limit to how many there can be.

When it trains. The first training waits for 1,100 samples with good feedback: 1,000 to learn from and 100 kept aside for the automatic test (1,250 if the pipeline keeps 20% aside). Each next training waits for 1,000 new ones since the last. These minimums are the same for every pipeline, so that every training has enough to be worth testing, and the pipeline says how many are still missing. A schedule (daily or weekly, at an hour you choose) is optional: with one, a training starts at the next scheduled time once the minimum is met. Train now skips the schedule, never the minimum. A pipeline trains one training at a time.

The Versions tab shows one card per version, newest first, for example v5 · Live · 5 Oct · Builds on v4 · Learned 3,160 new samples · 97% right (v4: 85%); a second model’s version also names its model. The live card has Go back to v4, and each card opens that version’s page. Below them, Trainings that did not become a version opens every training, newest first, with its result, what it learned, its share right, GPU time and cost.

Version limit. By default a pipeline keeps making versions with no end. A limit from 1 to 100 (Settings → Advanced) stops it after that many versions; trainings that do not go live do not count. Raise or remove the limit to carry on.

For developers: how trainings and versions work in full

How each new version is made

Each new version builds on the live version. Say v4 is live: the next training starts from v4 and learns the samples v4 has not learned yet, which are the new ones that arrived since v4 and those of trainings that did not become a version. So that it does not forget, it also goes over samples v4 had learned before, picked at random: as many as it learns new, unless you choose fewer in Training settings. Its page says all of this in numbers.

When a model cannot build on the version before it, each training of it learns from the original model on all your samples, and its page says so. Each version is kept as a saved copy of the model while it is live and while you can still go back to it (Storage).

A worked example: v5

From a real pipeline, Northwind support, after 6 trainings and 5 versions. Training #6 became v5, and its page said:

• Builds on v4.
• Learned 3,160 new samples: 2,144 that arrived since v4, and 1,016 from training #5, which did not become a version.
• It also went over 3,160 samples it had learned before, so it does not forget them.
• Right on 97% of its test. v4 got 85% on the same questions.

So training #5’s samples were not lost: training #6 learned them along with the new ones. And because 97% is clearly better than 85% on the same questions, v5 went live automatically.

Second model

A pipeline can also train a second model (Settings → Second model). It must be a different model from the main one. Each time the main model trains, the second model trains on the same samples and is tested the same way, against the live version; whichever does better becomes the next version. Each model keeps its own line of versions: the first time, the second model trains from its original on all samples; after that it builds on its own latest version plus the samples new since that version. For example:

• v1, v2 and v3 come from model 1.
• Training #4: model 1 builds on v3 with the new samples; model 2 trains from its original on all samples. Model 2 does better, so it becomes v4.
• Training #5: model 1 builds on v3 plus every sample new since v3; model 2 builds on v4 plus the samples new since v4.

Each training of either model is billed like any training. When the second model’s training goes live, the two change places: it becomes the main model, and the one it replaced becomes the second model, each still building on its own line. Settings says where the second model stands: waiting for the main model’s next training, training now, or why it did not train. Removing it does not stop a training of it already under way, which can still become the next version.

Going back

For 30 days after a version goes live you can go back to the one before: Go back to v4 on the live version’s card on the Versions tab, or on its page. Your app gets answers from v4 again; version numbers do not change. Only the newest version can be taken back, and going back from v1 leaves no live version until another one goes live. The next training builds on the version you went back to, and learns again what the newer one had learned.

You cannot go back while a training is building on the live version or being tested against it: the page offers to stop that training first (you pay only for what it used, and its samples go to the next training). Nor can you go back to a version that learned from a sample you deleted.

Deleting a sample a version learned

Deleting a sample the live version learned from rebuilds the model without it: the next training learns from the original model on every remaining sample, without waiting for new ones, and Train now starts it at once. It needs 1,000 samples to learn from, and samples deleted close together are rebuilt once.

The rebuild is tested against the live version like any training, and goes live unless it is clearly worse; if it is, a member is asked, and under Only when I approve it waits for you. It is never put live untested. Until it goes live the live version still holds what it learned from that sample; once it does, no version that learned the sample can be gone back to, and their saved copies are deleted. A sample that expires on its own rebuilds nothing.

A training’s page

Each training has its own page, titled v5 when it became a version or Training #5 when it did not. At the top, in plain words with exact numbers: the result, what it learned, its automatic test and each benchmark beside the live version, its cost, and Approve, Reject, Stop or Go back. Below, closed until you open them: During training, Test, question by question, Cost, part by part with its timeline, and Details for experts. Every training lists every pipeline’s trainings the same way. What became of a training:

Comparing two trainings

Compare puts any two trainings or versions of a pipeline side by side. Their automatic tests are shown side by side, never subtracted, because each was tested against what was live on its own day; the questions both answered are listed with both scores, and that share right is the like-for-like number. Benchmarks both took show both scores and the change.

Training settings

How each training learns is chosen for you from how many samples it has and how long they are: Automatic, recommended. Settings → Advanced → Training settings lets an expert set some values themselves (Custom), and leaves the rest automatic. A change applies from the next training, and each training’s page says which it used: Automatic settings or Custom settings. Going over fewer learned samples trains faster and forgets more. Gemma 4 learns longer and keeps its last step, because it learns facts slowly.

Tests

Every training is tested before it can go live: on questions from your own samples it never trained on (the automatic test), and on your benchmarks if you add any. The main model and the second model get the same tests. Nothing untested goes live: a member can approve a training that was not better, but never one that was never tested.

The Tests tab has one row per test: its newest result, a small chart of recent trainings, and See questions. An answer is right when the judge rates it at least 7 out of 10 against the right answer (on an exact-answer benchmark, when it matches).

A training that ended before it could be tested (no machine was free, say) shows Couldn’t be tested and why. When its trained model was kept, Test again on its page tests it now, at the usual cost.

For developers: the exact rules of the tests, scoring and the test machines

Automatic test

Questions from your own samples that this model never trained on. 5% of the pipeline’s samples with good feedback are kept aside for it, at least 100 (Settings → Advanced → Test share raises it to as much as 20%), and are never learned from. Each training answers up to 1,500 of them, and so does the live version (before v1, the original model). The judge compares both answers with the right answer, never told which is live, and the training counts as better only when it wins clearly, more often than chance would explain. The judge costs one to two cents a question, at most $25 a training. The kept-aside set grows with the pipeline, so two trainings seldom answer exactly the same questions; a benchmark is the same questions every time.

Your benchmarks

A benchmark is your fixed set of 20 to 200 questions, each with its right answer, such as facts your app must get right. Add one on the Tests tab, from a file or from samples you pick. From then on every training, and the live version, answers the same questions, so you can see whether each version gets better. The first in the list is your main benchmark, which the growth chart follows. The judge checks each answer by default; Exact answer suits a number, a label or a tool call. A benchmark’s questions are never learned from. It cannot be edited: make a new one. Taking it off a pipeline keeps every score it gave.

Question by question

See questions, or any mark on a test’s small chart on the Tests tab, opens one test on one training: 31 of 40 right; v4 got 28 of 40 on the same questions, then every question with the right answer, this training’s answer and the answer of the version it was tested against (v5’s answer, v4’s answer), and which of the two got it right. Filter by Got worse (this training wrong, that version right), Got better, both right or both wrong, search their words, and open a question for the whole conversation and the judge’s reasons. On a benchmark each question also shows how the pipeline’s last trainings did on it.

During training

A training’s page has a During training section, closed until you open it, showing how it went step by step: its mistakes on the samples it trained on and on practice questions it kept aside, lower being better, and the step whose model was kept (where it did best on the practice questions). A model whose recipe keeps the last step (Gemma 4) says so: its practice mistakes can rise while it is still learning facts, so the test results are what count.

How each version scored

The Overview’s chart has one point per version, v1 to the newest, with no limit. The line is your main benchmark: the same questions every time. Without a benchmark, it is the questions from your samples that every version answered, kept aside before v1. Until those scores exist, each point is that version’s own automatic test, whose questions differ each time, so the points are not joined; add a benchmark on the Tests tab to follow one fixed set. Hovering a point gives its exact number and, on the same questions, the change from the version before; clicking it opens that version’s page. One line sums it up, for example From 55% (v1) to 78% (v5): 23 points better. A version with no score is a gap, never a zero.

Going live

The pipeline’s going-live choice turns the test results into one decision. Change it on the Tests tab (Change beside its one line). Under every choice but Only when I approve, a training that is better goes live by itself, one that is not better is closed, and one that couldn’t decide waits for you.

A training waiting for you is approved (it becomes the next version) or rejected on its own page. While it waits its test machine is stopped, so waiting costs nothing, and after 14 days without a decision it is closed. The one exception to the choices is the rebuild after a deleted sample (above).

Which models you can use

Every model the Training page can train is meant to work in pipelines. A model is offered once the platform has checked that its improved versions can be served and tested here; until then Settings → Model lists it under Coming soon to pipelines, with one plain reason, for example: You can train it on the Training page; its improved versions can’t be tested here yet. We’re fixing it.

Coming soon to pipelines means: you can already train that model on the Training page, but a pipeline cannot use it yet. It moves into the list by itself when the check is done; you do not need to do anything. Not available for pipelines lists the models a pipeline cannot use and that are not promised, for example one larger than the models the platform trains today.

The model you pick is fixed for each training. You can change it in Settings; the next training uses the new model, and version numbers carry on. The second model is chosen from the same list.

Settings

The Settings tab is the form you created the pipeline with: name and purpose, model, second model and when to train in sight, and the rest under Advanced; then automatic scoring, pause and delete. A change applies from the next training. Each setting is explained where it matters:

• Name and purpose: Conscious Loop in 2 minutes; the pipeline ID and model name never change.
• Model: Which models you can use. Second model: Second model.
• When to train and Version limit: Versions. Training settings: Training settings.
• What it trains on: Datasets. Test share: Automatic test. Going live is shown in one line and chosen on the Tests tab: Going live.
• Score samples nobody gave feedback on: Scoring unreviewed samples. Storage: Storage, and Your own storage.
• Pause and Delete: Pausing and deleting.

Costs, privacy and stopping

A pipeline costs what its trainings and tests use, paid from your wallet, and its live version’s machine while it is switched on. You never choose machines or prices, and waiting for a machine is never billed.

What a pipeline has cost

The Overview’s So far line adds up every training of the pipeline: its trainings, its versions, and what was spent on training and testing (the machine time to train, the test machine, the judge, benchmarks, and scoring samples nobody reviewed). Each training’s page has its own cost, and Cost, part by part. An amount is said one of three ways:

• $8.02: final.
• at most $8.40: not final yet. the test machine is counted from the moment it started, and your bill starts slightly later, so the final charge can be lower.
• spent so far: a training is still under way, so the total will grow.

The live version’s machine is not in that total: it is billed on its deployment, like any other.

The live version’s machine

The live version answers from its own deployment: the machine your app reaches through the pipeline’s model name. While it is switched on it is billed by the hour, like any deployment; switch it off or on from its deployment (linked on the Overview). While it is off, your app’s calls to the model name fail, nothing is billed, and Train now waits, because a training could not be tested against it. When a newer version replaces it, the old machine is stopped ten minutes after the switch.

Waiting for funds. A training starts only once your wallet holds enough for all of its training, because training is never stopped part way for money. Until then it shows Not started: waiting for funds, and starts by itself once there is enough (or automatic top-up is on); after 24 hours it ends and the pipeline pauses. If your wallet runs out later in a training, it waits up to 24 hours for a top-up, then stops and the pipeline pauses: add funds and press Switch back on in Settings. When no machine is free it shows Waiting for a machine and tries again for up to a day.

Storage

Settings → Advanced → Storage says what the pipeline keeps. Test results and training data: how much there is, and whether it is billed yet, at the storage rate. Saved model copies for going back: billed as model storage while they are kept. Both count toward your workspace’s storage, and deleting the pipeline deletes them.

Pausing and deleting

Pause in Settings stops new trainings: samples keep arriving and the live version keeps answering. Whoever switches it back on is the member it trains for from then on.

Delete at the bottom of Settings stops a training under way and stops the live version: your app’s model name stops answering. Deleting a pipeline stops its machine, so it is no longer billed, unless another pipeline or model name of yours answers from that same machine. Its versions, trainings and test results leave the console at once; then the conversations in its test results, its saved models and its training data are deleted. Its samples stay, and its name stays reserved.

Privacy

• Nothing is saved without your say-so: only calls that carry the pipeline ID header, or calls to a model you switched recording on for. A call that says X-Loop-Capture: off is never saved.
• Secrets are removed before anything is saved: email addresses, API keys and tokens, private keys, card numbers, US social security numbers and IP addresses are replaced with a marker such as [redacted:email], in the conversation, the answer and every correction.
• How long samples are kept: 30 days by default (each model’s recording setting can keep them from 1 to 3,650 days), and imported ones 365 days. A sample that expires is never learned from again; versions already made keep what they learned.
• Deleting a sample removes it and its feedback, and no later training learns from it.
• Only your workspace can see its samples.
• Tests send questions to your models and to the judge through Run BiOS, on a key named Conscious Loop in your workspace. Those calls are never saved as samples.

Your own storage

A workspace can keep what the loop saves in a bucket or container it owns (Amazon S3, Azure Blob Storage or Google Cloud Storage): connect one under Integrations → Your own storage, run its connection check, and make it the default (how). From then on new conversations and the detail of test results are written there; the loop keeps only an index and your feedback.

• A sample kept there cannot be searched by its words, and while your storage cannot be read the sample says so instead of showing its text.
• Deleting a sample or a pipeline deletes what the loop wrote there, one object at a time; it never touches anything else in your storage.
• What is in your storage is not billed by us as storage.

For developers: privacy, storage and deleting in full

For developers

Everything above works from code too. The Conscious Loop API page has every route with a request and a response, every status code, and the rules in full.

SDKs and MCP

The Python and TypeScript SDKs cover the whole loop with the same names, and the MCP server lets an AI assistant do it for you, asking you before anything that spends from your wallet. See the SDK calls and the MCP tools, and how to install them: TypeScript SDK, Python SDK and MCP server.

API reference

• Recording with one header: the pipeline ID header and the sample ID.
• Samples and feedback: send, list, read and delete samples; feedback; tags.
• Import: every row shape, and retrying.
• Datasets: save, change, download and delete them.
• Models: the models a pipeline can train, and why the others cannot be used yet.
• Pipelines: create, read, change, delete, train now, their trainings and versions.
• Trainings: approve, reject, go back, stop, test again; test results question by question.
• Benchmarks: create, list, read, retire, and every question.
• Live traffic and stats: recording settings, counts, history.
• Rules in full and details by topic.