Comparing checkpoints: evaluation_compare

A single score means little; a difference against the base model, with its uncertainty, is what tells you whether training helped. This recipe takes several runs (the base model is required), scores them on the same measurements, and reports every difference with a verdict.

What a run is scored on	Section	Works for
public benchmarks	benchmarks	generative LLMs
your rows, with the recipe's metrics	rows	LLM, decision and embedding models
decision questions, without and with reasoning	decision	decision models
MTEB tasks	embedding	embedding models

Verdict	Meaning
decided	the difference lies outside the 95% interval of the two measurements
not decided (~)	inside it
open (?)	no interval exists (the recipes' own metrics)

Nothing is measured twice: each run's stages are the ones the single-model recipes use, so results already computed are reused and adding a run scores only that run. The output is a table, a polygon chart over the axes you define, and a Markdown summary you can paste into a report.

Config and details

pipeline: {recipe: evaluation_compare, name: sft-vs-base}
runs:
  - {label: base, experiment: output/sft-qwen3.5-0.8b-<hash>, checkpoint: base}     # the starting weights, or a Hub model: {label: qwen, name: Qwen/Qwen3.5-0.8B, revision: <commit>}
  - {label: final, experiment: output/sft-qwen3.5-0.8b-<hash>, checkpoint: validate}
  - {label: other, experiment: output/sft-lora-qwen3.5-0.8b-<hash>, checkpoint: validate}
benchmarks: {preset: small_general, limit: 100}                   # optional
rows: {sources: [{name: local_jsonl, path: my-rows.jsonl}]}        # optional: the recipes' own metrics
decision: {data: {sources: [{name: jevbench, path: data/jevbench}]}, think: both}   # optional: decision experiments
embedding: {tasks: [STSBenchmark, NFCorpus]}                        # optional: embedding experiments
axes: {knowledge: [mmlu, mmlu_pro], math: [gsm8k], decide: [decision:think:accuracy], sts: [embedding:STSBenchmark]}

Every run gets the benchmark stages of evaluation_benchmark (score:<label>:<task>) and the row stages of evaluation_custom (rows, rows:<label>). Stages carry a scope, so their directories are shared across recipes and names: what a single-model evaluation already measured is reused as it is, and adding a run scores only that run. The report (report.json) holds per benchmark and run the primary metric (the first of exact_match, acc_norm, acc, prompt_level_strict_acc, f1, ...), its standard error and item count; per row metric the value per run; and comparison: the difference of every run to the base, with decided true when it lies outside the 95% interval of the two measurements, false when inside (the Markdown marks it ~: not decided), null when no standard error exists (the recipes' own metrics; marked ?). axes.svg is the polygon chart, summary.md the tables.

Source: evaluation/README.md. This page is generated; edit the source and run npm run sync.