Your own rows: evaluation_custom

The quickest useful evaluation, for any LLM, decision or embedding model Modeling trained: point at a finished experiment, give it rows in the same format it trained on, and get the metrics its validate reports (loss and perplexity for SFT, accuracy and calibration for a decision model, ranking accuracy for an embedding model), now on data of your choosing. The speech recipes have their own measurements: tts_piper's validation and evaluation_wakeword.

model.checkpoint	Measures
validate	the checkpoint that passed validation
train	the last trained weights, also when validation failed
base	the weights the experiment started from: the baseline
a directory	any checkpoint the experiment's recipe can load

Because the experiment's own config says how the model loads and is measured, there is nothing to configure about the model.

Config and details

pipeline: {recipe: evaluation_custom, name: sft-on-my-rows}
model:
  experiment: output/sft-qwen3.5-0.8b-<hash>   # the experiment's config.yaml says how the model loads and is measured
  checkpoint: validate                          # validate | train | base (what it started from) | a checkpoint directory
data:
  sources: [{name: local_jsonl, path: my-rows.jsonl}]   # rows in the experiment's recipe format
evaluation: {batch_size: 8}

Stages: rows (read and check against the recipe) -> score (load the checkpoint, the recipe's evaluate) -> report (report.json: model, settings, rows, scores, contamination). base is the baseline: the weights the experiment started from, measured the same way. Rows over the experiment's training.max_length follow its training.overflow. The score stage's fingerprint is the experiment's config and the checkpoint's content, so a rebuilt experiment is a new evaluation and an unchanged one is cached.

Source: evaluation/README.md. This page is generated; edit the source and run npm run sync.