Contamination, tracking and the API

A model that trained on the test set scores well and means nothing. Rather than trusting that this never happened, every evaluation report measures it, and every report can be read by a person, a dashboard or a script.

Feature	What it gives
contamination	exact matches and near-duplicates between the evaluation items and the rows the experiment trained on, next to the scores
tracker	the report's numbers in Weights and Biases, once per report
API	GET /api/evaluations lists every report under an output directory

Contamination is a measurement, not a filter: the Data channel is where leaks are removed before training.

Details

Every report carries contamination: the overlap between the evaluation items (rows, benchmark documents as the harness logged them, decision questions) and the rows the experiment trained on (<experiment>/data/train.jsonl): exact (same text after case and whitespace normalization), near (at least half of an item's word 8-grams occur in the training rows; includes the exact ones), share_mean, and the first flagged item indices. It is null for base checkpoints (those weights never saw the rows) and for MTEB tasks. A comparison reports it per run. It is a measurement, not a filter: the Data channel's leak filter is where contamination is prevented.

With tracker: {enabled: true, project: ...} the report is sent to wandb once per report stage directory: every number in it as a summary metric (scores/loss, benchmarks/hellaswag/value, contamination/exact, ...) and the same numbers as a table. As for training runs, tracker is not part of the identity, and a report already sent to a project is not sent again.

GET /api/evaluations?output_dir=./output&recipe=&experiment= lists every evaluation report under an output directory (headline scores, model, contamination); GET /api/evaluations/{experiment} returns one report in full. output_dir must lie inside API_PATHS.

Source: evaluation/README.md. This page is generated; edit the source and run npm run sync.