Benchmark survey

The short survey behind the presets: which benchmark measures which ability, why those were chosen for small fine-tuned models, which harness runs them, and the pitfalls the design guards against. Sources are at the end.

Assumptions. Models are the catalog's (Qwen3-0.6B/1.7B, Qwen3.5-0.8B/2B, Qwen3-Embedding-0.6B) and their fine-tunes; one GPU or a Mac; the comparison that matters is checkpoint against checkpoint and against the base model, not against frontier models. Survey depth: light (direct searches, no literature review).

Benchmarks by what they tell

Axis	Benchmark	Why this one	Harness task
Knowledge	MMLU-Redux, MMLU-Pro	what the Qwen3.5 small-model cards report, so base-model numbers exist to compare with	mmlu_redux, mmlu_pro
Hard knowledge	GPQA Diamond	graduate-level, Google-proof; low for small models but moves with reasoning training	gpqa
Math	GSM8K, MATH-500	saturated for frontier models, still discriminative for small and fine-tuned ones	gsm8k, minerva_math/math500
Instruction following	IFEval	verifiable constraints, no judge; reported for Qwen3.5-0.8B/2B (52.1 / 61.2 non-thinking)	ifeval
Commonsense, regression panel	HellaSwag, ARC-Challenge, TruthfulQA, Winogrande	cheap log-likelihood tasks; a fine-tune that forgets shows up here first	hellaswag, arc_challenge, truthfulqa_mc2, winogrande
Code	HumanEval+, MBPP+ (EvalPlus)	execution-based; only when the fine-tune targets code	humaneval, mbpp (EvalPlus variants)
Korean	KMMLU, KMMLU-Redux/Pro, HAE-RAE Bench, KoBEST, CLIcK, HRM8K	original Korean exams and culture, not translations; all present in lm-evaluation-harness	kmmlu, haerae, kobest, click, hrm8k
Long context	RULER, LongBench	only when the fine-tune changes context handling; expensive	ruler, longbench
Decision models	JevBench (easy, standard, hard tiers; 231 public items), plus the Jeeves transfer panel: MMLU, MMLU-Pro, PAWS, QNLI, SciQ, TweetEval offensive, Emotion	the reference evaluation of the decision recipes' origins; accuracy and ECE, with and without reasoning, plus latency	own runner (DecisionMethod.evaluate + JevBench format)
Embeddings	MTEB v2: MTEB(eng, v2) retrieval subset; Korean retrieval tasks from MMTEB	the standard; Qwen3-Embedding is reported on it	mteb package

Base-model reference points (Qwen3.5 small-model cards, non-thinking unless noted):

Benchmark	Qwen3.5-0.8B	Qwen3.5-2B
MMLU-Pro	29.7 (thinking 42.3)	55.3 (thinking 66.5)
MMLU-Redux	48.5 (thinking 59.5)	69.2 (thinking 79.6)
IFEval	52.1	61.2
SuperGPQA	16.9	30.4
MMMLU	34.1	56.9

These numbers are the vendor's, under the vendor's prompts and decoding. The plan treats them as orientation only: the channel measures the base model itself, under the same settings as the checkpoints, and that is the baseline.

Harness

• lm-evaluation-harness (EleutherAI) for every static benchmark above: 200+ YAML-defined tasks, the Korean tasks included, local Hugging Face checkpoints via --model hf, log-likelihood and generative tasks, per-task standard errors. It is the default choice of the field for this kind of static scoring.
• mteb for embedding models.
• An own runner for decision models: the recipes' DecisionMethod.evaluate already computes calibrated accuracy, NLL and ECE with and without reasoning; it needs a JevBench reader and a latency measurement.
• Not needed now: Inspect AI (agentic and tool-use evaluation), HELM (multi-metric profiling of many models), LLM-as-judge suites (a judge adds cost and bias that a checkpoint comparison does not need).

Pitfalls the design must handle

• Contamination. A fine-tune on paraphrased benchmark items reaches inflated scores and evades n-gram checks (fine-tuning-phase contamination is the hardest to detect). The channel records, per evaluation, the overlap between the training rows of the experiment and the benchmark items (exact and near-duplicate), and the report shows it. The Data channel's leak filter is the place to prevent it; this channel only measures it.

• Settings decide the number. Few-shot count, prompt format, decoding (greedy vs sampled, temperature, max tokens), thinking on or off, harness version: each changes scores by points. Every result carries all of them, and two results are only compared when they match.

• Small samples, noisy scores. 231 JevBench items or a limit of 200 give confidence intervals of several points. The report shows the harness's standard error (or a bootstrap interval) next to every score, and marks a difference inside the interval as "not decided", not as "no difference".

• Thinking and non-thinking are two models. The decision recipes and Qwen3.5 answer differently with reasoning on. They are reported as separate rows, never averaged.

• Base model first. An evaluation of a checkpoint without the base model under the same settings is not interpretable; the compare stage insists on a baseline row.

• Wake word evaluation: openWakeWord README, performance and evaluation (targets under 5% misses and 0.5 false accepts per hour; DiPCo), its training script (false positives counted per window over 11.3 h); Picovoice wake-word-benchmark (miss rate at 1 false alarm per 10 hours, LibriSpeech test-clean with DEMAND noise); microWakeWord test.py (cutoff sweep in 0.01 steps, accepts with a cooldown, false accepts per hour of ambient audio); Coucke et al., Efficient keyword spotting using dilated convolutions and gating (Hey Snips; miss rate at 0.5 false accepts per hour)

• Qwen3.5 small models and their benchmark table: Artificial Analysis, Qwen/Qwen3.5-0.8B model card

• lm-evaluation-harness task list (Korean tasks kmmlu, haerae, kobest, click, hrm8k; mmlu_pro, gsm8k, ifeval, gpqa, humaneval, ruler, longbench): tasks README

• Harness comparison: LLM evaluation tools comparison (2026), lm-evaluation-harness review, LightEval tutorial

• Korean benchmarks: KMMLU, KMMLU-Redux and KMMLU-Pro, HAE-RAE Bench, Navigating Korean LLM research: evaluation tools

• Contamination and settings: systematic review of contamination detection (ACL 2026), contamination survey, static to dynamic, DICE: contamination in the fine-tuning phase, Questionable practices in machine learning

• Leaderboards in 2026 (Open LLM Leaderboard archived; HELM, LMArena, Artificial Analysis): LLM leaderboard explained, HELM

• Decision models: PostHog Jeeves repository (JevBench tiers, transfer panel, ECE, latency), Jeeves announcement coverage

• Embeddings: MTEB v2, Qwen3 Embedding

Source: evaluation/README.md. This page is generated; edit the source and run npm run sync.