Data overview
Most of the work in a good fine-tune is the data. The Data channel makes the rows a Modeling recipe trains on, in the exact format that recipe reads, with a manifest that says what came from where, how much was dropped at each step and what it cost. Today rows are made by a teacher model from seeds; converting open datasets is designed.
What it makes, for which recipe
| Recipe | Makes | For | Shipped configs |
|---|---|---|---|
data_synthetic | conversations, instruction rows, preference pairs, prompts with answers, decision questions | llm_sft, llm_dpo, llm_grpo, llm_decision_sft (select.assemble: sft, dpo, grpo, decision) | data/synthetic.yaml (Korean conversations), data/route.yaml (router decisions) |
data_speech | recordings with their texts, checked by Whisper | tts_piper, wakeword, evaluation_wakeword | data/speech.yaml, data/wake_*.yaml, data/wake_eval_*.yaml |
data_open (designed, not built) | rows converted from open datasets | every recipe |
Teachers
teacher.name | Calls | Cost |
|---|---|---|
anthropic | the Anthropic API (ANTHROPIC_API_KEY) | from a price table or price: [in, out] per million tokens |
openai | any chat-completions server: OpenAI, vLLM, LM Studio, gateways (base_url); reasoning models with reasoning_effort | the same |
ollama | a local Ollama server | free |
local | a checkpoint Modeling trained, or a Hub model | free |
What every data recipe guarantees
- Every teacher call is cached under a key made from the request, so a run that crashes or runs out of budget continues where it stopped, and the same config run twice costs nothing.
- The budget is checked before each uncached call (
budget.max_usd), and the cache is the ledger. - Rows leave in the target recipe's format: the last check is that recipe's own row check.
- Duplicates and evaluation items are removed before the rows are written, so a benchmark score is not inflated by its own items.
A Modeling config reads the result through data.sources:
{name: local_jsonl, path: output/<data experiment>/select/rows.jsonl}.
Produce the rows the Modeling channel trains on, two ways:
- Open data: load public datasets (Hugging Face Hub at a pinned revision, or local files), convert them into the row format of a recipe, filter them, remove what would leak into evaluation, and write JSONL.
- Synthetic data: generate rows with a teacher model (an API model or a local checkpoint) from seeds (personas, topics, documents, existing rows), verify them (schema, verifier, judge), filter, remove leaks, and write JSONL.
Either way the product is a JSONL file in a recipe's row format plus a manifest (counts per step, sources,
revisions, licences, teacher and cost), which Modeling reads through data.sources and never needs to know how it
was built. Leak removal against evaluation sets and benchmarks lives here as the last stage of both recipes; the
Evaluation channel only measures contamination, this channel is where it is prevented.
Source: data/README.md. This page is generated; edit the source and run npm run sync.