Training Jobs

Create fine-tuning jobs across the supported model catalog using 15+ training methods, 6 alignment algorithms, and continued pre-training. All jobs are billed per second of GPU compute.

Creating a Job

The training wizard guides you through an 8-step process:

1. Model: search and select a base model from HuggingFace (e.g., Llama, Mistral, Qwen, Gemma).
2. Method: choose SFT, RLHF, Continued Pre-Training (CPT), or VLM.
3. Adapter: pick an adapter compatible with your model and method (LoRA, QLoRA, full fine-tune).
4. Dataset: select from your uploaded datasets or import one from HuggingFace.
5. GPU: choose a GPU tier based on model size and budget. The platform recommends appropriate hardware.
6. Backups: nominate backup GPUs to fall back to if your chosen GPU is not available when the job starts.
7. Config: set hyperparameters: learning rate, epochs, batch size, LoRA rank, etc.
8. Review: confirm all settings and estimated cost before launching.

Training Methods

Monitoring Jobs

Once a job is running, the training detail page shows:

• Metrics graphs: live charts for training loss, evaluation loss, learning rate schedule, and gradient norms.
• Logs: streaming stdout/stderr from the training process.
• Checkpoints: saved model snapshots at configurable intervals. Each checkpoint can be deployed or deleted; the newest ones (up to your max checkpoints setting) plus the final model are kept in your storage.
• Cost tracking: running cost for the current job in real time.

Sequence Length and Long Samples

Max Sequence Length is how many tokens of one training sample the model sees at a time. The wizard recommends a value that covers 95% of your dataset and warns when the value you picked would cut a meaningful share of it. The recommendation is capped by the model's own context window.

Samples longer than that limit follow the "Samples longer than max length" setting:

• Truncate (default): the sample keeps its last tokens, so the assistant answer — the part the model learns to produce — is always trained, and only the oldest context is cut.
• Drop the sample: the whole sample is skipped. Nothing is cut, but you train on fewer examples.

Either way the job logs report exactly how many samples were affected and how long the longest one was, so raising the limit is always an informed decision. Samples that carry images are dropped rather than truncated when they overflow, because image placeholders must stay aligned with their pixels.

Early Stopping (Best Model)

A model keeps fitting the training data long after it stops getting better at anything else — it starts memorizing. Early stopping watches the validation loss and ends the run once it has not improved for a number of consecutive evaluations. It is on by default (3 evaluations)whenever you set a validation split, and you can choose 3, 4, 5, or turn it off to always train the full schedule.

• Every new best is saved: whenever validation loss reaches a new low, that checkpoint is written immediately and flagged Best, so the most general model is always available to deploy.
• The Best checkpoint is never pruned: the max-checkpoints setting removes older ordinary checkpoints, never the Best one or the final model.
• The run still completes: an early stop is a successful job. The logs name the step whose validation loss was best, and you are billed only for the time actually used.

Stopping and Resuming

You can stop a running job at any time by clicking the Stop button on the training detail page. When you stop a job:

• Checkpoints are preserved: saved checkpoints stay in your storage and can be deployed at any time.
• Billing stops immediately: you are only charged for the compute time used up to the stop point, billed per second.
• Logs and metrics are retained: the training loss graph, logs, and configuration remain accessible on the job detail page.
• Resume from checkpoint: stopped jobs can be resumed from the last saved checkpoint, continuing training where it left off.