Design and plan

This page is the design the channel was built from, kept because it explains the shape of the recipes and records what is still open. The milestones are marked as built or not.

A channel is a package that registers recipes; it reuses Stage, Pipeline, Progress, Experiment and the API's jobs as they are, and joins by adding "evaluation" to CHANNELS in core/registry.py.

Recipes

Recipe	Input	Stages	Output
evaluation_benchmark	one checkpoint (or a Hub model) and a list of benchmarks	score (lm-evaluation-harness, one run per benchmark, cached by checkpoint content + task + settings + harness version) → report	results/<benchmark>.json with scores, standard errors, settings, samples; report.json
evaluation_custom	one checkpoint, any row source, the training recipe whose evaluate applies (llm_sft, llm_dpo, llm_decision_sft, ...)	rows → score → report	the recipe's own metrics on rows it never trained on
evaluation_decision	a decision checkpoint and JevBench or Jev-format rows	rows → score (with and without reasoning; accuracy, NLL, ECE, latency per item) → report	the Jeeves-style table
evaluation_embedding	an embedding checkpoint and MTEB task names	score (mteb) → report	MTEB results
evaluation_compare	several finished evaluation experiments (or checkpoints to evaluate with one benchmark list), the base model among them	collect → report	one table across checkpoints, a polygon chart per axis, a Markdown summary

Config shape, following the modeling recipes (sections reject unknown keys; identities hash content):

pipeline:
  recipe: evaluation_benchmark
  name: sft-qwen3.5-0.8b-eval
model:
  checkpoint: output/sft-qwen3.5-0.8b-<hash>/validate/checkpoint   # or name + revision for a Hub model
  thinking: false
benchmarks:
  - {name: mmlu_redux, num_fewshot: 5}
  - {name: gsm8k, num_fewshot: 8, limit: 500}
  - {name: ifeval}
  - {name: kmmlu, num_fewshot: 5, limit: 1000}
decoding: {temperature: 0, max_new_tokens: 1024}
contamination: {training_rows: output/sft-qwen3.5-0.8b-<hash>/data/train.jsonl}
axes: {knowledge: [mmlu_redux], math: [gsm8k], instructions: [ifeval], korean: [kmmlu]}

pipeline:
  recipe: evaluation_compare
  name: sft-qwen3.5-0.8b-vs-base
runs:
  - {label: base, checkpoint: Qwen/Qwen3.5-0.8B, revision: <commit>}
  - {label: step-200, checkpoint: output/sft-qwen3.5-0.8b-<hash>/train/checkpoint}
  - {label: final, checkpoint: output/sft-qwen3.5-0.8b-<hash>/validate/checkpoint}
benchmarks: [...]        # the same list for every run; each run's scores come from the benchmark stage cache

Where results live and how they are found

• Every score stage directory is output/_stages/score-<fingerprint>/, shared by every evaluation that asks for the same checkpoint, benchmark and settings: evaluating three checkpoints and then comparing them builds nothing twice, and re-running a comparison after one more checkpoint scores only that one.
• The checkpoint counts by content (directory_signature), as model.init does in training: a rebuilt experiment is a new evaluation.
• report.json per evaluation experiment: {"checkpoint", "settings", "scores": {benchmark: {metric, stderr, n}}, "contamination": {...}}; the compare report adds {"runs": [...], "axes": {axis: {label: value}}, "chart": "axes.svg"}.
• Later: GET /api/evaluations listing reports across output/, and the tracker sending the table to wandb.

Chart

One polygon (radar) per comparison: axes from axes, each axis the mean of its benchmarks' primary metric scaled to 0..1, one polygon per run. Drawn as SVG by the channel itself (no plotting dependency); the Markdown summary embeds it and the table.

Each milestone ends green on CI with tests on the tiny random model, as the modeling channel does.

Milestone	Builds	Done when
1. Skeleton and evaluation_custom (built)	the evaluation package, EvaluationConfig, rows and score stages that call a modeling method's evaluate on a checkpoint, report.json, CHANNELS entry	llmclick run configs/evaluation/custom.yaml scores an SFT checkpoint on a JSONL file; the API lists the recipe; cached rerun builds nothing
2. evaluation_benchmark (built; the GPU run is still to do)	lm-evaluation-harness behind an optional extra (uv sync --extra eval), a benchmarks section with presets (small_general, korean), harness version in the fingerprint, limit for smoke runs, settings recorded in results	the preset runs on Qwen3.5-0.8B on one GPU with limit; results carry stderr and settings; a missing extra gives one clear error
3. evaluation_compare (built)	several checkpoints, the base model required, the table, the polygon SVG, Markdown summary, "not decided" marking from intervals; stage scopes so single-model evaluations and comparisons share their runs	one command compares base, mid-training and final checkpoints of one experiment
4. Decision and embedding (built)	evaluation_decision (JevBench reader, think and no-think rows, ECE, latency), evaluation_embedding (mteb)	the decision pointer pilot reproduces the metrics validate reports, plus the JevBench tiers; the embedding pilot runs one MTEB retrieval task
5. Contamination and surfacing (built)	exact and near-duplicate overlap between an experiment's training rows and the benchmark items, in the report; GET /api/evaluations; tracker table; decision and embedding sections in evaluation_compare	a deliberately contaminated pilot shows the overlap (the tests score an experiment on its own training rows: 64 of 64 exact); the API returns the reports

Order of 4 and 5 can swap depending on which models are trained first.

Decisions to take before milestone 2

• Panel. Which of the general, Korean, code and long-context axes are needed for the models being trained. The default preset is general plus Korean, no code, no long context.

• Dependency weight. lm-evaluation-harness brings many dependencies; it goes behind an extra and is not in the Docker image unless the image is built with it.

• Compute budget. Full MMLU-Pro or KMMLU on a 2B model takes hours on one GPU; limit presets (smoke: 100, standard: 1000, full) are part of the config, and the report says which was used.

• Thinking mode. Whether the Qwen3.5 fine-tunes are evaluated with thinking on, off, or both (both doubles cost).

• Each training method already has an evaluate that returns its metrics on a set of rows (modeling/tuning/method.py); milestone 1 calls it on rows that were never part of training.

• DecisionMethod.collect and measured (modeling/llm/methods/decision/base.py) compute accuracy, NLL and ECE with and without reasoning; milestone 4 adds a JevBench reader and timing around them.

• The removed jeff package carried benchmark builders (a five-benchmark panel and JevBench) for decision models; they are in the git history before the commit "vendored jeff 와 jev 레시피 제거".

• Row sources (local_jsonl, huggingface) live in modeling/tuning/sources.py; the rows stage reuses them.

Source: evaluation/README.md. This page is generated; edit the source and run npm run sync.