Conscious Loop

The Conscious Loop turns what your model actually did into what it learns next. It records the conversations a model had, lets you say whether each answer was right, and curates the result into training data for supervised fine-tuning, preference optimisation and reward-based training. Nothing is recorded until you switch it on for a source, and you can delete any conversation at any time.

Start here: one conversation to one training set

Everything below this walkthrough is reference. This part is the whole thing end to end, in the order it actually happens, and it is short on purpose — the loop only has five moves.

1. Decide what gets recorded

Nothing is recorded until you say so, per source, and the row remembers who turned it on. Anything that looks like a key, a card number or a personal detail is stripped before the record is written — never afterwards, because a secret that has reached storage has already escaped.

client.loop.set_config("support-agent", enabled=True, retention_days=30)

Already have data? Import it instead — it arrives as conversations needing review, not as finished training data.

2. Say whether an answer was right

Three ways, and they are not interchangeable. A thumbs down says this was wrong. A correction says what should have been said instead. A gold answer states the reference, without claiming the model was wrong. Each one unlocks different methods, which is why the difference matters.

client.loop.downvote(trace_id, reason="Quotes the policy instead of answering")
client.loop.correct(trace_id, "I have asked payments to trace it and will email you today.")
client.loop.gold(trace_id, "Refunds reach the original card within five working days.")

3. Scale past what you can read

Human review is the most trustworthy feedback and the least available. Two things close the gap, and they answer different questions:

A rule is deterministic and free; a judge costs a model call and can answer the question a rule cannot. Most workspaces end up using both.

4. Carve it into the slices you train on

An average over everything is the least useful thing to train on. Label conversations as they arrive — bare tags, or named dimensions like category and task — and one captured corpus becomes a different training set for every thing you want to fix.

5. Build a set, and read what it says

ds = client.loop.create_dataset(
    name="Billing replies, Spanish", method="sft",
    attributes={"category": "billing", "language": "es"},
    holdout_percent=20,
)
print(ds["item_count"], "rows")
for w in ds["report"]["warnings"]:
    print(w["code"], "-", w["message"])

Then read the report before training. It is the difference between a set that will help and one that will quietly make the model worse.

Is this set worth training on?

A row count is the number that looks fine when everything else is wrong. Every set carries a report of what it is actually made of, and a plain warning for each way it might make the model worse rather than better.

{
  "rows": 240, "train_rows": 192, "holdout_rows": 48,
  "balance": { "desirable": 140, "undesirable": 100 },
  "sources": { "human": 180, "judge": 60 },
  "reviewers": { "you@example.com": 120, "sam@example.com": 60 },
  "models": { "llama-3-8b": 240 },
  "graders": ["Commits to a next step"],
  "judges": ["Support replies"],
  "warnings": []
}

The warnings, and why each one matters:

• Too few rows. Below about fifty, a fine-tune memorises these examples and gets worse at everything else.
• No hold-back. Nothing was kept aside, so there is no way to tell whether training helped.
• One-sided labels. If nearly every row says the answer was good, the model learns to approve of everything — because that is what the data says.
• Every score identical. The rules or the rubric never actually told good answers from bad ones, so the scores add nothing.
• One reviewer. The model learns one person’s preferences, including the ones they would not defend.
• Nobody checked it. Entirely machine-scored is a confident corpus of whatever the rubric got wrong.
• Stale feedback. Verdicts about a model you have already replaced.
• Mixed models. A correction written against one model is not always right for another.

None of these stop you. They are sentences to read, not refusals — a small corpus you know is small is a perfectly reasonable thing to train on while you iterate. A set with nothing wrong reports no warnings at all, so an empty list means checked-and-fine rather than not-checked.

What the loop does

A model that is never corrected never improves. The loop closes that gap in four steps, and you can stop after any of them:

1. Record what the model was asked and what it answered.
2. Judge it, by hand or by rule.
3. Curate the judgements into a training set.
4. Train on the file, and start again with the new model.

Recording conversations

Recording is off by default and is switched on one source at a time, under Conscious Loop → Capture. A source is a deployment you run here, or any name you choose for a model you run somewhere else. The row remembers who turned it on and when.

Conversations from a model served on Run BiOS are recorded for you. For a model you run elsewhere, send each exchange yourself:

from bios import RunBiOS

client = RunBiOS(api_key="bios-...")

# Switch recording on once, for this source.
client.loop.set_config("my-agent", enabled=True, retention_days=30)

result = client.loop.capture(
    deployment_id="my-agent",
    model="gpt-4o",
    messages=[{"role": "user", "content": "What is your refund window?"}],
    completion="You can return anything within 30 days of delivery.",
    request_id="your-own-id-123",   # replaying it will not store a duplicate
)
trace_id = result["trace_id"]       # ours, not the id you sent

If the source is not switched on, the call returns captured: false with a reason rather than an error — recording must never be the thing that breaks your application. Pass tool_calls and tools when the answer invoked a function; without them a tool-using exchange trains the model to reply in prose exactly where it should have called something.

Data you already have

You do not have to start from zero. Bring in an export from another provider, a spreadsheet of past answers, or a set of preference pairs, and it enters the loop as conversations needing review — not as a finished training set.

That distinction is the whole point. A file of what your model said classifies everywhere else as known-good training data, and nothing in it separates “these were checked and are right” from “these are merely what happened”. Train on the second as though it were the first and you teach the model to repeat its own mistakes with more confidence than it had the first time.

result = client.loop.import_rows(
    source="zendesk-2026",            # becomes the source each row is filed under
    labels=["refunds"],
    attributes={"category": "billing"},
    rows=[
        {"prompt": "why was I charged twice",
         "chosen": "I have refunded the duplicate today.",
         "rejected": "Please see our billing policy."},
        {"messages": [{"role": "user", "content": "how long do refunds take"},
                      {"role": "assistant", "content": "About a week."}]},
    ],
)
print(result["imported"], result["reviewed"], result["by_shape"])

Pass each row in whatever shape it already has. The shape is how the server decides whether it already carries a verdict, so reshaping rows into one tidy envelope first would erase exactly the information you are importing:

The field names a row is read from

A row is recognised by which of these it carries. They are the names the training ecosystem already uses, so an export from a fine-tuning toolchain usually needs no reshaping at all:

A spreadsheet with its own column names — question and answer, say — needs renaming to these first. Rows that carry none of them are refused and counted under unrecognised_shape rather than guessed at, so a whole file in the wrong column names shows up as a refusal with a reason instead of an import that quietly did nothing.

A verdict that arrives with the file is kept — discarding somebody’s judgement would be worse — but it is recorded as having come from your own earlier process rather than from a reviewer here, and every conversation records whether it was captured or imported. Neither fact can be reconstructed afterwards, so both are written at the door.

Read by_shape and refused_why back before you trust the result. They are how you find out that a file was silently the wrong shape, rather than merely smaller than you expected. At most 5,000 rows per call: the call is synchronous and somebody is waiting on it.

The four kinds of feedback

Feedback is append-only: recording a second verdict does not replace the first, because two reviewers disagreeing about an answer is information worth keeping. Which verdict you choose decides what can be trained from it.

Thumbs up and thumbs down

The fastest feedback there is, and the least informative. A thumbs up keeps the answer as a training example. A thumbs down is worth understanding: on its own it removes the answer from training rather than teaching anything, because there is no better version to learn from. If you know what it should have said, use Fix it instead — it is worth several times more.

client.loop.upvote(trace_id)
client.loop.downvote(trace_id, reason="Invented a policy we do not have")

Correcting a wrong answer

A correction says the model was wrong and supplies the better answer. One action produces both halves of a preference pair: the model learns your answer and learns to avoid its own. This is the single most valuable thing a reviewer can do.

client.loop.correct(
    trace_id,
    "Refunds are back on your original payment method within 30 days, "
    "normally 3 to 5 business days.",
)

Recording the gold answer

A gold answer is the reference for a question, recorded whatever the model happened to say. That is the difference from a correction, and it matters: a correction asserts the model was wrong, so the original becomes the rejected half of a pair. A gold answer asserts nothing about the model — so if it matches what was said, no preference is invented, and if it differs, a pair falls out anyway.

It is the most reusable feedback you can record. One gold answer trains supervised fine-tuning, forms a preference pair when the model disagreed with it, and stands in as the reward reference for GRPO when you have not supplied a separate ground truth.

client.loop.gold(trace_id, "30 days from delivery, no questions asked.")

# Use it even when the model was already right: the reference is still the
# reference, and it is what the next model will be measured against.

Scoring answers automatically

Reading every answer does not scale. A scoring rule is written once under Conscious Loop → Scoring rules and applied to every answer afterwards. Every rule is deterministic — no model is asked — so the same answer always gets the same verdict, which is why an automatic score is trusted above a model’s opinion when they disagree.

• Says the right things — required and forbidden phrases.
• Matches an exact answer — ignoring case and spacing.
• Gets the number right — compares values, so 12.00 and 12 agree.
• Returns valid JSON — parses, and carries the keys you require.
• Matches a pattern — a regular expression.
• Calls the right tool — invoked the function, not just mentioned it.

Each rule carries a weight, and the score is their weighted average over the rules that actually applied. A rule with nothing to check — a number rule against an answer containing no number — is skipped, not failed: it is left out of the average entirely. Scoring an absent opinion as zero would quietly punish every answer the rule was never written for.

A rule can be aimed at a slice. Outside it the rule is not applied — not applied, rather than failed — so it stays out of the score entirely, and a master group covers everything under it. This matters more than it sounds: a rule like “must name the order number” with no slice also applies to every billing reply and marks each one down for not mentioning a shipment, which makes the whole set of scores look like evidence of a problem that is not there.

Sampling alternatives

Preference training needs a better and a worse answer to the same question, and a person writing every one of those does not scale. Instead, sample your model several times for a recorded question, submit each sample, and score them: the best becomes the preferred answer and the worst the rejected one, with nobody reading anything. Submit answers from a stronger model instead and the same machinery performs distillation.

for sample in my_model.sample(prompt, n=4):
    client.loop.add_candidate(trace_id, completion=sample)

# Score the original AND every sample against your rules, in one call.
report = client.loop.grade(trace_id)
print(report["report"]["score"], report["candidates_graded"])

An unscored alternative takes no part in a pair. A missing score is an absence of information, not a low one, and treating it as low would invent a preference nobody expressed. If every sample scores the same, no pair is built and the report says so — that points at your rules, not at your model.

Labels, groups and choosing a slice

An average over everything is the least useful thing to train on. A model weak at refunds is fixed with refund examples — and you can only select those if you said so when the conversation arrived. Label conversations as they are recorded, then build a set from one slice.

A label may name a parent, which is what makes a master group. Label something refunds with parent billing, and it is selectable as either. Selecting billing later gathers every child — refunds, invoices, whatever else you added — without anybody maintaining a list of them. Labels are lowercased and trimmed, so Refunds and refunds are one label.

client.loop.label(trace_id, ["refunds"], parent="billing")
client.loop.label(other_id, ["invoices"], parent="billing")

# Everything under the master group, newest 20% held back for evaluation.
ds = client.loop.create_dataset(
    name="Billing agent", method="sft",
    label="billing", holdout_percent=20,
)

# Or a random subset, when the corpus is larger than a run needs.
ds = client.loop.create_dataset(
    name="Spot check", method="sft", label="billing", sample=500,
)

You can also narrow by source, by date range, and by taking a random sample. The sample is taken before the hold-back split, so what is held back is still the most recent of what was chosen — a shuffled slice would hold back a random sample instead of a recent one, which is exactly the flattery the time split exists to prevent.

Dimensions: one corpus, many datasets

A bare tag answers “is this about refunds”. It cannot answer “which of these are billing conversations, in Spanish, that escalated” — and that is the question you ask when you want one dataset per task out of one pile of conversations.

So a label can carry a key: the dimension it belongs to. Set as many as you like, on capture or afterwards.

# At capture, while you still know what it was.
client.loop.capture(
    source="support-agent", model="llama-3-8b",
    messages=[...], completion="...",
    labels=["refunds"],
    attributes={"category": "billing", "task": "status-lookup", "language": "es"},
)

# Or afterwards. Correcting one dimension leaves the others alone.
client.loop.set_attributes(trace_id, {"category": "billing"})
client.loop.unlabel(trace_id, "biling", key="category")

# Then slice it however the training run needs.
client.loop.create_dataset(name="Billing, Spanish", method="sft",
                           attributes={"category": "billing", "language": "es"})
client.loop.create_dataset(name="Every status lookup", method="sft",
                           attributes={"task": "status-lookup"})
client.loop.create_dataset(name="Not organised yet", method="sft", unlabelled=True)

Filters on different dimensions narrow together, so five conversations captured once can produce a different training set for every combination you ask for — and nothing is re-labelled to do it.

• A dimension is matched exactly within its key. source=supportdoes not match team=support. Only bare tags reach through master groups, because that is what a master group is.
• Correcting a label needs no delete first. Setting categoryagain overwrites it and leaves task, language and your tags exactly where they were.
• “Not labelled yet” is a filter, not a label. Nothing writes an unknown tag onto your conversations — it would have to be removed the moment a real label arrived, and it would appear in every count as though somebody had chosen it.
• A bad label never loses the conversation. Capture records the exchange and reports the label problem separately: the conversation is the valuable part.

Judges: scoring what a rule cannot check

A scoring rule is deterministic and blind to anything it was not told to look for. It can check that the refund window was quoted. It cannot say whether the reply was helpful. A judge is the other half: instructions in your own words, several things scored separately, applied to one slice.

Score several things rather than one. “Helpful but wrong” is not something a single number can say, and an average that hides it is worse than no score at all.

You run the model, and why

The service that stores your conversations holds no credentials to any model— no provider keys, nothing it could use to send your prompts anywhere. That is most of the reason it is safe to store them there, and it is not a property worth spending for convenience. So a run selects the conversations, freezes the rubric onto them, and hands the work out. You send it to whatever model you like — ours or anyone’s — and post the scores back.

Each item arrives with the prompt already written: the rubric, the conversation, the answer, and the exact reply format. Send it as it comes. Assembling your own is how two callers end up applying the same rubric differently, and two judges given different instructions are not one judge.

judge = client.loop.create_judge(
    name="Billing replies",
    instructions="""Score each reply as a support answer to a paying customer.
An answer is good when it resolves the question actually asked, says only what
is true, and would leave a reasonable person feeling helped. Length is not quality.""",
    dimensions=[
        {"key": "accuracy", "description": "is everything it says true"},
        {"key": "tone", "description": "would a customer feel helped"},
    ],
    selection={"label": "billing", "sample": 50, "only_unscored": True},
)

run = client.loop.start_run(judge["id"])
work = client.loop.take_work(run["id"])

verdicts = []
for item in work["items"]:
    reply = client.chat.completions.create(          # any model you like
        model="llama-3-8b",
        messages=[{"role": "user", "content": item["prompt"]}],
    )
    try:
        scored = json.loads(reply.choices[0].message.content)
        verdicts.append({"trace_id": item["trace_id"], **scored})
    except ValueError as exc:
        # Record it rather than dropping it: a run that quietly shrinks is one
        # whose coverage nobody can state.
        verdicts.append({"trace_id": item["trace_id"], "error": str(exc)})

client.loop.post_verdicts(run["id"], verdicts)

An agent can do all of this itself over MCP: loop_create_judge, loop_start_judge_run, loop_take_judge_work, loop_post_judge_verdicts.

Things worth knowing before you rely on one

• A run freezes its selection and its rubric. A run whose selection stayed a live query would silently grow as conversations arrived, so “we scored the refunds slice” would be a claim about a set that no longer exists. Editing a judge afterwards never changes verdicts already given.
• A score the rubric never asked for is refused, not averaged in. A judge that invents a dimension has not answered the question it was asked, and quietly folding the invention into the score would hide that.
• A human verdict outranks a judge’s when both exist on one conversation. The judge’s score is kept either way — two opinions disagreeing is information, and nothing here is ever overwritten.
• An item you could not score should be sent back with an errorrather than left out. A run that quietly shrinks is a run whose coverage nobody can state.
• Letting a judge write gold answers is distillation. It is off unless you turn it on. Training on an answer a model wrote teaches your model to imitate that model, which can be exactly what you want — it is how a smaller model is taught by a larger one — but it is worth choosing rather than discovering.

A rubric is a reward function

This is the part that matters most, and it is easy to miss. GRPO does not train on answers you already have — it generates its own and needs a way to score them. An exact-match reward needs a ground truth, and most real work has none: there is no string a support reply has to equal.

A rubric does not need one. So a conversation a judge has scored exports with the rubric attached as its reward, and becomes trainable where the same conversation without one is dropped for having nothing to check against. The rubric travels inside the sample, frozen as the run used it — a trainer that had to fetch it later would be scoring against whatever the rubric has since become.

{
  "prompt": [{"role": "user", "content": "where is my refund"}],
  "reward": {
    "kind": "judge",
    "judge": "Billing replies",
    "model": "llama-3-8b",
    "instructions": "Score each reply as a support answer to a paying customer...",
    "dimensions": [{"key": "accuracy"}, {"key": "tone"}]
  }
}

No completion, as always for GRPO — it writes its own during training and scores each one against the rubric above.

Building training sets

The three methods need genuinely different feedback, so a set built for one cannotbe reshaped into another afterwards.

Why kto usually has the most rows

The other three methods all need something beyond a verdict. SFT needs a right answer to learn, DPO needs a second answer to prefer against, GRPO needs a way to score an answer nobody has written yet. A thumbs-down with nothing written satisfies none of them — so the cheapest and most abundant feedback you can collect produced nothing at all.

KTO is unpaired: “this answer was bad” is a complete training row on its own. That makes the feedback people will actually give into feedback a model can learn from. A rewrite becomes two rows, not one — the answer somebody wrote is desirable and the answer they replaced is not.

Scored answers become rows too, above and below a label_threshold that defaults to 0.5. A score sitting exactly on it is refused rather than rounded, and reported as score_on_the_fence: a number that is neither good nor bad is not a quiet no, and forcing it either way invents feedback nobody gave.

Every set reports why rows were left out, by reason. A small set with an explanation is useful; a small set without one is just alarming. The hold-back percentage keeps your most recent work aside rather than a random slice, so an evaluation measures whether the model generalised instead of memorised the same week.

Download a set as JSONL — one training row per line, the format every trainer reads — from the dashboard, or with loop.downloadDataset(id). Pass split=holdoutfor the slice held back.

API, SDK and MCP

Everything on this page is reachable three ways, and they do the same thing. API keys need the loop:read and loop:write scopes, which are not part of the Read Only preset — what they return is your raw prompts and completions, not catalog metadata, so a key carries them only by an explicit grant.

From an agent, the same surface is available as MCP tools — loop_capture_trace, loop_add_signal, loop_add_candidate, loop_create_grader, loop_grade_trace, loop_build_dataset and the reads beside them. Start with loop_set_capture: every other loop tool is inert until recording is switched on.

What is stored, and for how long

• Nothing is recorded until you switch it on, per source. The row records who did and when.
• Secrets are removed before the record is written, never afterwards. Anything shaped like an API key, a card number, an email address or a private key is replaced with a marker — including inside tool-call arguments. The conversation shows you what was removed.
• Recordings expire after the retention window you chose, and are deleted automatically.
• Deleting a conversation removes it from every training set built from it, including sets already exported. Training runs that have already finished are not affected.
• Only your workspace can see it. A conversation belonging to another workspace is indistinguishable from one that does not exist.