Synthetic rows: data_synthetic

data_synthetic makes training rows for the LLM and decision recipes from seeds, through a teacher model. It works as a funnel: the early stages make many candidates cheaply, the later ones throw most of them away for a reason that is recorded. The published recipes the design follows (Magpie, Tulu 3, OpenThoughts, Evol-Instruct, Nemotron) all agree on that shape: generate more than you keep, and let verifiers and judges choose.

What it can make

select.assemble	Row	For	Typical verification
sft	a conversation: the prompt and the best answer	llm_sft	a judge with a rubric
dpo	the prompt, the best and the worst answer by judge score	llm_dpo	a judge; margin sets the minimum gap
grpo	the prompt and its verified answer	llm_grpo	a reward against the answer, majority agreement
decision	a state (earlier turns and the request) and one choice question	llm_decision_sft	the teacher's route equals the seed's option, majority agreement

Generator (prompts.generator)	Makes from a seed
topic_questions	questions about a topic, each rewritten to be more specific (Nemotron)
persona_task	one request a persona would make (Tulu 3)
document_qa	a question the passage answers, with its answer; the passage becomes context
labelled_requests	requests that belong to one option of a decision, with 0 to 2 earlier turns, in several languages
passthrough	the seed is already the instruction

Run one

uv run llmclick validate configs/data/synthetic.yaml   # experiment key and stage plan
uv run llmclick run configs/data/synthetic.yaml        # rows at output/<name>-<hash>/select/rows.jsonl

Before a large run, do a small one (two seeds, two answers) with the teacher you intend to use and read the cache directories: they hold every request and answer, and they show quickly whether the teacher follows the generator's format. A small teacher often does not; the filters will then reject most of its output, which is the pipeline working as designed.

Stages and settings

pipeline: {recipe: data_synthetic, name: topics-ko-sft}
teacher: {name: ollama, model: qwen3.5:0.8b}               # anthropic (claude-opus-5-5), openai (base_url), ollama, local (experiment)
seeds:
  - {kind: topic, values: [tides, yeast]}
  - {kind: persona, source: {name: huggingface, dataset: ..., revision: <commit>, split: train, limit: 200}, field: persona}
prompts: {generator: topic_questions, per_seed: 1, params: {n: 3, language: Korean}, constraints: [...]}
evolve: {rounds: 1, operators: {constraints: 1, deepen: 1, concretize: 1, reasoning: 1, breadth: 0.5}}   # optional
respond: {answers_per_question: 2, max_tokens: 512, temperature: 0.7}
verify: {schema_recipe: llm_sft, majority: null, reward: null, judge: {rubric: ..., min_score: 6}}
select: {assemble: sft, dedup: {near: 0.75, ngram: 5}, leak: {against: [{name: local_jsonl, path: eval.jsonl}]}}
budget: {max_usd: 5}

Six stages, each a Stage with its own fingerprint, so a changed verify section reruns verification on the cached answers and nothing before it:

Stage	Does	Idempotent because
seeds	inline values and source rows become {id, kind, text, row}; the same text is one seed	pure function of the config and the source files; ids are content hashes
prompts	a generator (topic_questions, persona_task, document_qa, labelled_requests, passthrough) makes instructions per seed; constraints appends a verifiable one	every generator job is cached under key(seed, generator, index, params) in cache/; a rerun makes only the missing ones
evolve	Evol-Instruct operators (five in-depth, breadth, WizardCoder's code heuristics, WizardMath's easier) per round with a seeded draw; rules 3 and 4 and an optional judged "no gain" drop failures; originals and every round are kept	the operator draw is seeded by the prompt's key; each rewrite is cached under key(parent, round, operator)
respond	the teacher answers each prompt answers_per_question times	each answer is cached under key(prompt, sample, teacher identity, settings); the budget is checked before every uncached call and the cache is the ledger, so a crash or a budget stop costs nothing twice
verify	empty and "sorry" answers out, the target recipe's check, a REWARDS function against the prompt's answer, majority agreement, a judge score (cached per candidate)	pure rules over the candidates file plus a cached judge
select	exact and n-gram near-duplicates out, leaks against evaluation items out (an item is leaked when a row holds half of its 8-grams), one row per prompt assembled as sft, dpo (judge gap margin), grpo or decision; manifest.json with every stage's counts, the teacher and the cost	pure function of the verified rows and the leak sources

Rows and files are written through a temporary name and renamed, so a reader never sees a half file. Running the same config twice builds nothing the second time; running it after a crash finishes the remaining teacher calls only (the tests kill the fake teacher mid-stage and count the calls). The manifest's stages trail carries each stage's summary downstream, so select can report the whole run from its own inputs.

Decision rows (assemble: decision, for llm_decision_sft): a seed's kind is the option and its text a situation; the labelled_requests generator has the teacher write requests that belong to that option, in each of params.languages, with params.history turns of conversation before them (the turns become the row's context). The teacher then routes every request in respond (respond.system holds the options, the answer is one word), and verify keeps a request when its routes agree with its option (reward: exact_match, majority: exact). select writes {"state": <earlier turns, a blank line, the request>, "questions": {<select.decision.name>: {"type": "choice", "instructions", "criteria", "label"}}}. configs/data/route.yaml builds a router this way: whether an agent runs inner thought on a request.

Teachers (data/teachers.py): anthropic (official SDK, ANTHROPIC_API_KEY, default claude-opus-5-5, cost from a small price table or price: [in, out] per million tokens), openai (chat-completions shape: OpenAI, vLLM, LM Studio, gateways; base_url, api_key_env; reasoning_effort for reasoning models, which then get max_completion_tokens and no temperature), ollama (native /api/chat, keep_alive), local (a checkpoint of a modeling experiment or a Hub model, seeded, free). A teacher's identity (name, model, generation-relevant parameters; never keys or prices) is part of the fingerprints of the stages that call it.

Not yet (next milestone): multi-turn dialogues and reasoning on/off pairs in respond, judge verdicts with swapped order for pairs, the student's pass rate as a difficulty filter, arena-style selection, data_open.

Source: data/README.md. This page is generated; edit the source and run npm run sync.