Embedding models: evaluation_embedding

Retrieval quality is measured on MTEB tasks, the standard for embedding models, which covers retrieval, sentence similarity, classification, clustering and reranking. This recipe wraps the experiment's backbone as an MTEB encoder, so the pooling and instructions are exactly the ones training used, and caches one stage per task. Any MTEB task runs by name; python -c "import mteb; print(mteb.get_tasks())" lists them.

Config and details

pipeline: {recipe: evaluation_embedding, name: contrastive-on-mteb}
model: {experiment: output/contrastive-qwen3-embedding-0.6b-<hash>, checkpoint: validate}
tasks: [STSBenchmark, NFCorpus, Banking77Classification]
max_length: null       # tokens per text; null: the experiment's training.max_length
evaluation: {batch_size: 32}

Needs uv sync --extra eval (mteb). One stage per task (score:<task>, scope mteb), fingerprinted by the checkpoint's content, the task, max_length and the mteb version. The experiment's backbone is wrapped as an MTEB encoder (texts in, unit vectors out; cosine similarity), so pooling and instructions are the recipe's own. The report holds per task the main score and metric, the task type and every split's scores; results.json keeps mteb's full result. mteb's mock tasks (MockSTSTask, ...) are accepted for smoke tests without downloads.

Source: evaluation/README.md. This page is generated; edit the source and run npm run sync.