Stages and metrics

Every tuning recipe is data then train then validate. The tables here are the ones to keep open while writing a config: which keys your rows need, and which metric names validation.min and validation.max may refer to.

Stage	What it does
data	read every data.sources[*], check each row against the recipe's method, hold out data.validation of them
train	load the backbone, run the training loop with the method's loss; resumes from resume.pt after a crash
validate	the method's metrics on the held-out rows against validation.min/max
rows, score, report	evaluation_custom: read and check the rows, measure the chosen checkpoint with the experiment's method, write report.json
score:<task>..., report	evaluation_benchmark: one lm-evaluation-harness run per benchmark (fingerprint: checkpoint content, task, few-shot, limit, decoding, harness version), then one report
score:<run>:<task>..., rows, rows:<run>..., report	evaluation_compare: the stages above per run, shared with the single-model recipes; the report compares every run with the base
rows, score, report	evaluation_decision: every question through the head without reasoning and after greedy reasoning; items.jsonl per question
score:<task>..., report	evaluation_embedding: one mteb run per task (fingerprint: checkpoint content, task, max_length, mteb version)
clips, traces, report	evaluation_wakeword: recordings and ambient WAVs found; the detector's score every 80 ms on all of them under each condition; the threshold sweep and the operating point
data, train, validate	tts_piper: recordings found and split; piper.train fit from the starting checkpoint, exported to voice.onnx; the voice reads the held-out texts and the fixed replies, Whisper transcribes them
data, features, train, validate	wakeword: recordings of both classes found and split; noisy windows as embeddings; the detector trained and exported; held-out streams scored every 80 ms

Recipe	Row keys	Metrics for validation.min/max
llm_sft	messages	loss, perplexity
llm_instruction	instruction, input?, output	loss, perplexity
llm_dpo	prompt, chosen, rejected	accuracy (chosen more likely than rejected), margin
llm_grpo	prompt + what the reward reads (answer)	reward (greedy completion)
llm_decision_sft	state, questions (Jev records) or state, question, label (one question per row)	accuracy, nll, ece
llm_decision_cispo	same	accuracy, nll, ece, think_accuracy (after reasoning)
embedding_contrastive	query, positive, negative?	accuracy (positive ranked first), mrr
tts_piper	audio (a WAV file), text	wer (Whisper on the voice), reference_wer (Whisper on the recordings), duration_ratio, phrase_wer (the worst-heard fixed reply)
wakeword	audio, text? (a positive's must be the phrase)	recall, false_accepts, scenario (phrase then command: one firing, on the phrase)

By default every weight is trained, in FP32. training.adapter switches any of these recipes to LoRA:

training:
  adapter: {name: lora, r: 16, alpha: 32, dropout: 0.05}   # omit to train every weight

The base weights stay frozen and only the adapter trains, so the resume snapshot holds just the adapter. Which layers are adapted comes from the architecture (transformer, bi_encoder: the attention projections; hybrid: those plus the linear-attention projections) unless targets names others. The adapter is folded into the base weights when the checkpoint is written, so a LoRA checkpoint and a fully trained one load the same way: a plain save_pretrained directory at output/<experiment>/train/checkpoint/ (and, once validation passed, at output/<experiment>/validate/checkpoint).

Source: README.md. This page is generated; edit the source and run npm run sync.