Skip to main content

How a run works

Every recipe runs on the same runner, and the runner has a handful of rules that explain most of what you will observe: why a second run is instant, why two configs share a data stage, why a crashed training continues instead of restarting, and why a trained model is never handed on without passing validation.

The short version: a stage's directory is named by a hash of everything that decides its result, so "has this been built?" is a directory lookup. The long version follows.

pipeline:
  recipe: llm_sft                   # which pipeline builds this model
  name: my-model
  stages: [data, train]             # optional; default all. Stages they read are added, train always brings validate
  • Fingerprints. A stage's fingerprint is its own config inputs (plus the content of local files it reads) and the fingerprints of the stages it reads. Its directory is output/_stages/<stage>-<fingerprint>/, so two configs that differ only in training settings share the data stages, and a rerun of the same config builds nothing. Each training method, backbone and head class carries a version; bumping one after changing its code rebuilds the experiments that use that part and leaves every other recipe's cache alone.
  • Reruns. A stage is recorded only when it finishes. A crashed run is resumed by running the same command again; training continues from resume.pt in the train stage directory and ends with the same weights as an uninterrupted run: batches, learning rates and the samples of a sampling method (llm_grpo, llm_decision_cispo) are all derived from the seed and the step. With method.iterations above 1 a snapshot is written only between groups of steps that share samples, so a resume never lands inside such a group.
  • Locks. A stage directory is locked while it is built, so a CLI run and an API job asking for the same stage never build it twice; the second one says that it is waiting (progress note, log) and then reuses the result.
  • Identity. The experiment directory is <name>-<hash>; stages, output_dir and tracker are not part of the hash, because they say how to run, not what to build.
  • Progress. The CLI shows two bars (stages, and steps inside the running stage); API jobs report the same state in progress. Every long phase names itself there: loading the model, checking lengths, a method's preparation (reference margins, reasoning chains), the steps, fitting the temperature, saving.
  • Lengths. Before the first training step every training and held-out row is measured against training.max_length, including the tokens a method adds later (generated completions, reasoning chains). A row that does not fit stops the run there (training.overflow: error, the default) or is left out (overflow: skip); nothing is cut silently and nothing fails hours into training. Embedding texts are the exception: they are truncated, as usual for retrieval.
  • Evaluation while training. training.eval_every: N measures the model on the held-out rows every N steps with the validate stage's own metrics (training.eval_rows caps how many rows), and records them as eval/<metric> next to the loss in training.jsonl and in the tracker. The weights are as they are at that step: for the decision recipes the temperature is not fitted yet, so nll and ece there are uncalibrated.
  • Validation. In the Modeling channel validate follows train and cannot be left out. It measures the trained model on data training never saw and checks validation.min / validation.max; a miss, or a metric that is not a finite number, stops the pipeline. A later stage reads the checkpoint from validate, never from train. Without bounds the metrics are only recorded, and a warning says so.
  • Tracking. With tracker.enabled each training step is sent to wandb as it happens; a run interrupted and resumed continues the same wandb run. tracker is not part of an experiment's identity, so switching it on later replays the already cached run once instead of retraining. A run is sent to a given project only once.
  • Jobs. The API keeps its jobs in a SQLite table (JOBS_DB), so they survive a restart; a job the server was running when it stopped is marked interrupted and resumes when submitted again. DELETE /api/jobs/<id> cancels a pending or running job: it stops at its next progress report, keeps what it built, and continues from there when submitted again. JOB_WORKERS jobs run at once, each worker on its own accelerator: set it to the number of GPUs on a multi-GPU server and keep 1 otherwise. On a server others can reach, set API_TOKEN (every /api call then needs Authorization: Bearer <token>) and API_PATHS (the directories the API may read configs and rows from and write output to; any other path is refused).
  • Typos. Every config section rejects keys it does not know, so validaton: or training.learning_rate is an error instead of a silently ignored setting.

Source: README.md. This page is generated; edit the source and run npm run sync.