Decision models: evaluation_decision

A decision model is judged on three things: whether it picks the right option, whether its confidence means what it says, and how fast it answers. This recipe measures all three, on your Jev rows or on the public JevBench tiers, and does it twice when asked: without reasoning and after the model reasons first, so you can see what thinking buys and what it costs.

Number	Tells you
accuracy (per tier against chance when rows carry tiers)	whether it picks the right option
nll, ece	whether a 0.9 means right nine times in ten; calibrated with the checkpoint's temperature
latency p50, p95	seconds a question at the batch size used
reasoning tokens, closed chains	what thinking costs (with think: on or both)

Config and details

pipeline: {recipe: evaluation_decision, name: decision-on-jevbench}
model: {experiment: output/decision-pointer-qwen3.5-0.8b-<hash>, checkpoint: validate}
data: {sources: [{name: jevbench, path: data/jevbench}]}   # or local_jsonl rows in the Jev format
think: both            # off | on | both
max_think: 256         # null: the experiment's method.max_think
evaluation: {batch_size: 8}

The jevbench source reads the public tier files (easy.jsonl, original.jsonl, hard.jsonl; the format jevbench.py of the Jeeves repository reads) into Jev records with a tier and family. Without path they are downloaded once from the JevBench repository (fstandhartinger/jevbench, MIT; 48, 72 and 111 items) at the pinned revision into LLMCLICK_CACHE (default ~/.cache/llmclick); tiers and limit per tier are optional. Every question goes through the head without reasoning and, with think, after greedy reasoning. The report holds per mode (nothink, think): accuracy, NLL, ECE (calibrated with the checkpoint's temperature), reasoning tokens, the share of chains that closed themselves, latency per question (p50, p95 at the batch size used) and, when rows carry tiers, per tier the accuracy against the chance level of its option counts and the chance-corrected share. items.jsonl keeps every question's prediction, confidence and timing. Not yet: the JevBench coherence suite on the Hub (JevBench/jevbench, a different benchmark: metamorphic relations between rewordings), and the Jeeves transfer panel (MMLU, PAWS, QNLI, SciQ, TweetEval, Emotion as decision rows), which the Data channel's converters will produce. The sealed judge tier is not public.

Source: evaluation/README.md. This page is generated; edit the source and run npm run sync.