Datasets and methods survey

Where to get rows for each recipe when you do not generate them, which synthetic methods exist and what each is good for, and the tools that implement them. Sources are at the end.

Assumptions. Targets are the recipes of this repository (SFT, instruction, DPO, GRPO, decision, contrastive) on 0.6B to 2B models, English and Korean, one GPU or a Mac for local teachers and an API for stronger ones. Survey depth: light.

Open datasets by recipe

Recipe	Dataset	Size	Why	Licence (check before use)
llm_sft	Tulu 3 SFT mixture (allenai)	939k	seven domains, the reference open post-training mix	ODC-BY, per-source terms
llm_sft	SmolTalk / smol-smoltalk / smoltalk2 (HuggingFaceTB)	1M / smaller	built for small models; smol-smoltalk for under 1B; Magpie-generated core	Apache-2.0
llm_sft (reasoning)	OpenThoughts3-1.2M	850k math, 250k code, 100k science	the published recipe for reasoning SFT (question sourcing, teacher traces, filtering)	Apache-2.0
llm_instruction, Korean	KoAlpaca v1.1, KIT-19, open-korean-instructions, Magpie-Pro-MT-300K-ko, GLAN-QnA-KR	21k to 303k	Korean instruction data is mostly translated or aggregated; GLAN-QnA-KR and Magpie-ko are native synthetic	varies
llm_dpo	UltraFeedback (binarized)	64k	the standard preference set; SmolLM2 used it after SFT	MIT
llm_dpo	HelpSteer3-Preference (NVIDIA)	40k	human-annotated, multi-domain and multilingual, commercial use allowed	CC-BY-4.0
llm_dpo, Korean	Ko-UltraFeedback	62k	translated and refined UltraFeedback	check
llm_grpo	GSM8K, MATH	8k, 12k	verifiable answers; still discriminative for small models	MIT
llm_grpo	OpenR1-Math-220k, Open-Reasoner-Zero (129k)	220k, 129k	larger verifiable math sets used by R1 replications	Apache-2.0
llm_grpo, Korean	HRM8K	8k	Korean math with answers	check
embedding_contrastive	MS MARCO / mMARCO, MIRACL (ko: 12.8k train pairs), BGE training collection, RLHN (relabelled hard negatives)	600k+	query-positive pairs; hard negatives mined or relabelled	varies
embedding_contrastive, Korean	ko-triplet-v1.0, korean-embedding-performance-v1 (1M)	1M	Korean triples, mostly synthetic	check
llm_decision_*	the Jeeves prep sources: banking77, boolq, ag_news, mnli, sst5, yelp, trec, dbpedia14, amazon reviews, imdb, wanli (trainable); mmlu, emotion, tweet_eval offensive, qnli, paws, sciq (eval only), at the revisions prep/public.py pins	4.7k accepted per source	classification and NLI datasets rendered as Jev questions, the reference decision-training data	per dataset

Synthetic-data methods

• Self-Instruct / Evol-Instruct: a seed set of instructions, evolved by a teacher into harder or more varied ones; the oldest recipe and still the baseline.
• Magpie: an instruct model given only its own chat prefix writes both the user turn and the answer; 300k filtered Magpie rows beat ShareGPT, Evol-Instruct and OpenHermes for Llama-3-8B, and SmolTalk's core is Magpie.
• Persona Hub: a billion web-derived personas as the diversity source; a persona in the prompt makes the same task yield different rows, the main defence against collapse of synthetic sets onto a few styles.
• Distilled reasoning traces (OpenThoughts): source questions, have a strong teacher write traces, keep by difficulty and verifier agreement; the mix (math, code, science) was tuned by ablation.
• Rejection sampling with verifiers: sample several answers, keep those a verifier accepts (math answers, unit tests); doubles as preference pairs (accepted vs rejected) for DPO.
• Judge-ranked preferences (UltraFeedback): several models answer, a judge scores by aspect, the best and a worse answer form the pair.
• Multi-agent simulation (MATRIX-Gen): personas interact to create realistic multi-turn conversations.
• Document-grounded QA (synthetic-data-kit, Easy Dataset): chunk documents, generate questions, answers and traces per chunk; the route for domain data.
• Decision questions over states (jeff): a teacher writes typed questions (choice, noul, score) with answers about a given state text, producing Jev records from any corpus of states.

Tools

• distilabel (Argilla): pipeline framework with steps for EvolInstruct, UltraFeedback, Magpie and judges; the most complete open implementation of the methods above.
• Bespoke Curator: batch-oriented generation with structured outputs, used for reasoning and function-calling data.
• synthetic-data-kit (Meta): documents in, QA and CoT pairs out, with a CLI.
• Anthropic API as a teacher: claude-opus-5-5 by default, the Message Batches API at half price for bulk generation, structured outputs (output_config.format) to get rows that already match a recipe's schema, prompt caching for the shared instructions and persona lists.
• Local teacher: any checkpoint the Modeling channel can load, through LLMBackbone.generate; free and reproducible (seeded), weaker.

The channel does not adopt distilabel or Curator as a dependency: their pipelines overlap with the Stage and Pipeline runner already here (fingerprints, caching, resume, progress, jobs), and a generation step is a few hundred lines on top of the Anthropic SDK and the existing backbones. Their method implementations are the reference for the prompts and the filters.

Quality, duplication, leakage

• Near-duplicates: MinHash over word 5-grams with LSH (the FineWeb setting: 112 hashes in 14 buckets, 75% similarity) removes paraphrased repeats that exact hashing misses; embedding-based dedup catches semantic repeats.

• Decontamination: remove training rows whose n-gram overlap with an evaluation item exceeds a threshold (50% MinHash similarity, or answer overlap at 60% and passage overlap at 40% in token-level detectors). The Evaluation channel measures exactly this kind of overlap after the fact; this channel removes it before training.

• Judges: a hostile judge from a different model family than the generator, scoring faithfulness, clarity, plausibility and novelty against the task definition, rejects weak rows; learned quality filters beat heuristics for educational value and toxicity.

• Diversity and collapse: filtering and dedup are what keeps the effective distribution of synthetic data broad; seed diversity (personas, documents) matters more than volume. Human spot-checks on a sample stay in the loop.

• Known pitfalls: contamination introduced through instruction tuning is the hardest to detect, and paraphrased benchmark items evade n-gram detectors, so the leak stage must also run on synthetic rows, with the teacher's tendency to reproduce benchmark items in mind. Translated Korean corpora carry translation artefacts; native synthetic generation with Korean personas is the better route for Korean SFT. Licences differ per source and must be recorded per row set.

• Synthetic data methods and tools: Distilabel, Bespoke Curator and synthesis landscape (2026 guide), Meta synthetic-data-kit, Scaling synthetic data creation with 1B personas, MATRIX-Gen: multi-agent simulation, Easy Dataset

• SFT and reasoning sets: Tulu 3 SFT mixture, SmolTalk, smol-smoltalk, SmolLM2 paper, OpenThoughts

• Preference sets: UltraFeedback and a systematic study of DPO datasets, HelpSteer3-Preference, curated list of post-training datasets

• Verifiable reasoning sets: 100 days after DeepSeek-R1 (replication survey), training small R1-like models

• Korean: KIT-19, GLAN-QnA-KR, open-korean-instructions, KoAlpaca

• Embedding training data: MIRACL and multilingual retrieval data engineering, korean-embedding-performance-v1, embedding models 2026 guide

• Quality, dedup, decontamination: FineWeb-Edu, LSHBloom, NVIDIA text data processing, contamination detection review (ACL 2026)

• Decision data: PostHog Jeeves repository (prep/, data/manifest.json)

• Nemotron and WizardLM lineages: Nemotron-4 340B report, synthetic data section, Llama-Nemotron report, Nemotron post-training dataset, WizardLM / Evol-Instruct (ICLR 2024) and its prompts in the repository, WizardCoder, WizardMath (RLEIF), Auto Evol-Instruct, Arena Learning

• Reference practices: Magpie (ICLR 2025) and its filtered 300k sets, OpenThoughts findings, Tulu 3 report, Phi-4 technical report, Nemotron-4 340B report, GLAN, Deita and the data-selection survey, Superfiltering (IFD), random selection at scale, LLM Data Auditor survey (2026), Best practices and lessons learned on synthetic data, model collapse under accumulation, synthetic data pipelines that don't collapse

Source: data/README.md. This page is generated; edit the source and run npm run sync.