Design and plan

The design both data recipes follow, the registries you extend (generators, evolvers, teachers, verifiers), what makes their fingerprints, and the milestones with their status.

A channel is a package that registers recipes; it reuses Stage, Pipeline, Progress, Experiment and the API's jobs as they are, and joins by adding "data" to CHANNELS in core/registry.py.

Recipes

Recipe	Input	Stages	Output
data_open	public or local datasets, the target recipe, a converter, filters, leak targets	load -> convert -> filter -> leak -> write	rows.jsonl in the recipe's format, manifest.json
data_synthetic	a teacher, seeds, a prompt and task kind, verification, filters, leak targets	seeds -> generate -> verify -> filter -> leak -> write	the same

pipeline: {recipe: data_open, name: tulu-sft-ko}
sources:
  - {name: huggingface, dataset: allenai/tulu-3-sft-mixture, revision: <commit>, split: train, limit: 50000}
convert: {to: llm_sft, converter: messages}                 # or fields: {messages: conversation}
filters:
  length: {min_chars: 20, max_chars: 8000}
  languages: [en, ko]
  dedup: {method: minhash, ngram: 5, threshold: 0.75}       # exact is always on
leak:
  against: [{name: local_jsonl, path: output/sft-qwen3.5-0.8b-<hash>/data/validation.jsonl}, {benchmark: gsm8k}]
  ngram: 8
  threshold: 0.5

pipeline: {recipe: data_synthetic, name: persona-chat-ko}
teacher: {name: anthropic, model: claude-opus-5-5, batch: true, max_tokens: 2048, temperature: 1.0}
task: {kind: conversation, to: llm_sft, turns: [2, 6]}
seeds:
  - {name: personas, source: {name: huggingface, dataset: proj-persona/PersonaHub, revision: <commit>, split: train, limit: 2000}}
  - {name: topics, values: [customer support, travel, cooking]}
prompt: {template: prompts/persona_chat_ko.jinja, language: ko}
count: 5000
verify:
  schema: true                                               # structured outputs: the row must parse as the recipe's row
  judge: {teacher: {name: anthropic, model: claude-sonnet-5-5}, rubric: prompts/judge_helpfulness.jinja, min_score: 7}
filters: {dedup: {method: minhash, threshold: 0.75}}
leak: {against: [{benchmark: kmmlu}, {benchmark: haerae}]}
budget: {max_usd: 50}

• Converters are a registry (CONVERTERS): messages (a column of chat turns), instruction (instruction, input, output columns), pairs (prompt, chosen, rejected; UltraFeedback and HelpSteer3 shapes), answer (prompt and a verifiable answer; GSM8K's #### 18), triples (query, positive, negatives; MIRACL and MS MARCO shapes), classification (a text and a label set rendered as a Jev choice question with a per-source template, as the Jeeves prep does; NLI as choice, boolean as noul, ratings as score). A converter has a version that joins the fingerprint. fields maps columns when no converter fits.
• Teachers are a registry (TEACHERS) with four entries from the start: anthropic (the official SDK; Message Batches when batch: true; structured outputs for schema verification; prompt caching on the template; the key from ANTHROPIC_API_KEY through settings), openai (the OpenAI chat-completions format: OpenAI itself, vLLM, LM Studio, any gateway with that shape; base_url and OPENAI_API_KEY), ollama (a local Ollama server through its native API, with model pull and keep_alive; its OpenAI-compatible endpoint works through openai too), and local (a checkpoint through the Modeling channel's backbones, seeded, no cost). Every teacher offers the same complete(messages, schema?) and reports tokens and an estimated cost per call into the manifest; budget.max_usd stops a run before it overspends. JSON-schema output goes through each provider's native structured-output feature where it exists and through a parse-and-retry otherwise.
• Verification chain: schema (the row parses as the target recipe's row, through the recipe's own check), verifier (a reward function from REWARDS, for answers and code), judge (a teacher scoring with a rubric; a different model family than the generator by default). Rejected rows are kept in rejected.jsonl with the reason.
• Leak removal reuses evaluation/contamination.py (exact and n-gram overlap) and the jevbench and benchmark readers for items, so the same arithmetic that measures contamination removes it; against takes row sources, benchmark names (through lm-evaluation-harness's documents) and JevBench tiers.
• Fingerprints: dataset revisions and file contents, converter versions, filter settings, the prompt template's content, the teacher's model and generation settings and the seed. generate is resumable: rows are written per seed batch as they arrive, and a rerun continues from the last batch; the cache directory makes a second config with the same seeds and prompt reuse the generation.
• Output: output/<experiment>/write/rows.jsonl and manifest.json (per stage the counts in and out, sources with revisions and licences, teacher, tokens, cost, rejected reasons). Modeling's data.sources points at the rows; later a dataset source can name a data experiment directly.
• Row sources move here: local_jsonl and huggingface in modeling/tuning/sources.py become data/sources.py; Modeling and Evaluation import them from here.

Reports and surfacing

manifest.json is the report. GET /api/datasets lists manifests under an output directory (counts, sources, cost); with tracker.enabled the manifest's numbers go to wandb as the evaluation reports do.

Each milestone ends green on CI with tests on small local files and a fake teacher, as the other channels do.

Milestone	Builds	Done when
1. Skeleton and data_open	the data package, row sources moved here, load, convert (messages, instruction, pairs, answer, triples, fields), filter (length, exact dedup, language), leak (reusing the contamination code), write with the manifest; sample configs for Tulu, UltraFeedback, GSM8K, MIRACL	llmclick run configs/data/open_sft.yaml writes rows that configs/llm/sft.yaml trains on by path; a rerun builds nothing; the API lists the recipe
2. Decision rows and near-duplicates	the classification converter with per-source templates for the Jeeves prep sources (trainable and eval-only panels), MinHash near-duplicate removal, the Jeeves transfer panel as decision evaluation rows for evaluation_decision	a decision SFT pilot trains on converted banking77 rows; the transfer panel scores a decision checkpoint
3. data_synthetic with a local teacher	seeds, generate (resumable, per-batch cache, seeded), verify (schema and verifier), the local teacher, instruction and conversation tasks (Magpie-style and persona-seeded prompts), templates	a tiny local model generates rows that llm_sft trains on; the fake teacher tests cover resume and cost accounting
4. Anthropic teacher, judge, preferences and traces	the anthropic teacher (Batches, structured outputs, caching, budget), the judge verifier, preference pairs by rejection sampling and by judge, reasoning traces with verifier agreement, decision questions over states	a 1k-row Korean persona conversation set and a 1k-pair preference set are produced within the budget and pass the judge; costs are in the manifest
5. Korean presets and surfacing	presets for Korean SFT, DPO and retrieval rows, translation-artefact filters, GET /api/datasets, the tracker table, a dataset row source for Modeling	the presets run end to end on a GPU; the API lists the manifests

Decisions to take before milestone 3

• Default teacher. The Anthropic API (claude-opus-5-5; claude-sonnet-5-5 for judges and bulk) or a local checkpoint. The design supports both; the default decides what the sample configs show and what a run costs.

• Budget per run. budget.max_usd default and whether Batches (half price, hours of latency) is the default.

• First target. Korean conversation rows for llm_sft, decision rows for llm_decision_sft, or preference pairs for llm_dpo; the milestones above assume SFT first, decision second.

• Licence policy. Which licences are acceptable for training rows (commercial use or not); the manifest records them either way.

• Row sources (local_jsonl, huggingface) in modeling/tuning/sources.py, and the jevbench source in evaluation/decision.py.

• Contamination measurement in evaluation/contamination.py (exact and n-gram overlap, numpy-backed), to be reused as the leak filter.

• Reward functions in modeling/llm/methods/grpo.py (exact_match, contains, last_number) as verifiers.

• The vendored jeff package that was removed carried a data pipeline (27 public datasets converted into decision rows, teacher-written synthetic questions, a near-duplicate leak filter). It is in the git history before the commit "vendored jeff 와 jev 레시피 제거" and is the reference for the decision converters and the question generator; the Jeeves repository's prep/ (public sources at pinned revisions, admission rules) is the other.

Source: data/README.md. This page is generated; edit the source and run npm run sync.