Skip to main content

What makes synthetic data good

If you read one page before generating data, read this one. It condenses what the open recipes actually did, with the numbers they reported, into rules of thumb: seeds decide coverage, few question sources but many answers, filtering is where quality is made, decontaminate before counting, and the only ground truth is a model trained on the result.

What the published recipes actually did, with the numbers they report; the checklist at the end is what data_synthetic builds in. Sources are at the end of this file.

Seeds decide coverage; the prompt decides little.

  • Taxonomy-driven (GLAN): fields, sub-fields, disciplines, then subjects and a syllabus per subject, generated with LLM help and reviewed; instructions come from the syllabus, no seed data. GLAN-QnA-KR reproduces it in Korean over 1,084 disciplines with a 100 to 900 difficulty scale and a two-layer contamination audit against Korean benchmarks.
  • Persona-driven (Tulu 3, Persona Hub): about 250k personas from Persona Hub combined with per-skill prompts (precise instruction following, math, code) so one task yields many perspectives; Persona Hub holds a billion web-derived personas.
  • Document-driven (Phi-4): high-quality seed passages from many domains rewritten into exercises, discussions and reasoning tasks by multi-step prompting; instruction reversal (from an answer, write the instruction) adds instruction data from any text; about 50 synthetic dataset types, 400B tokens.
  • From nothing (Magpie): an instruct model given only its chat prefix writes the user turn and then the answer; cheap and diverse, but everything rests on the filtering below.

Questions: few good sources, many answers. OpenThoughts (27 code, 21 math, 14 science sources ablated): mixing at most two high-quality question sources beat mixing sixteen by 5 points on average; sampling 16 answers per question beat adding questions (answer diversity inside a problem is worth more than question diversity across problems); filtering answers by majority consensus or length did not beat keeping every sample; the best benchmark model was not the best teacher (QwQ-32B taught better than DeepSeek-R1). The teacher and the sources are chosen by training a student and measuring, not by the teacher's own scores.

Filtering is where the quality is made.

  • Magpie's filters on 4M raw rows down to 300k: input quality tagged by a judge (keep at least average or good), input difficulty (at least medium), a reward model score and the reward difference against the base model's answer (at least 0), removal of repetition and incomplete instructions, minimum neighbour distance in embedding space (all-mpnet-base-v2 with FAISS) against near-duplicates, then the longest responses up to the budget.
  • SmolTalk's core (Smol-Magpie-Ultra, 400k) was generated with Llama-3.1-405B-Instruct and filtered with smaller Llama judges plus the ArmoRM reward model; for small models the mix adds task-specific sets (constraints 36k, rewrite 50k, summarize 100k) and a reduced smol-smoltalk for models under 1B.
  • Nemotron-4 340B gated 98% synthetic post-training data with a five-attribute reward model (helpfulness, correctness, coherence, complexity, verbosity) for both filtering and preference ranking, iterated weak to strong.
  • Phi-4 added self-revision: the model critiques and improves its own answers against rubrics on reasoning and factual accuracy, then rejection sampling for post-training.
  • Tulu 3 made instruction following verifiable: 25 constraint types from IFEval, about 30k examples checked by heuristics, no judge needed.

Selection scorers are cheap filters, not oracles. Deita (complexity and quality scorers plus a diversity constraint), IFD (instruction-following difficulty from a small model's perplexity with and without the instruction; 5% of Alpaca selected by IFD beat the full set), AlpaGasus (judge scores on helpfulness and accuracy), InsTag (tag diversity). At scale, random selection is hard to beat with these alone, so a scorer is used to drop the worst rows and the rest is decided by training a student.

Decontamination and duplicates. Tulu 3 matched evaluation and training by shared 8-grams per token and dropped the overlaps; GLAN-QnA-KR audits against the Korean benchmarks; MinHash with LSH (5-grams, 112 hashes, 75%) removes paraphrased repeats; embedding clustering shows whether a large set is really a few clusters. Synthetic rows need this as much as scraped ones: teachers reproduce benchmark items from memory, and paraphrases evade n-gram checks, so the leak stage runs on every synthetic set and the Evaluation channel reports what remains.

Collapse is avoided by accumulation and diversity, not by volume. Training on synthetic data that replaces real data collapses (as little as one synthetic row in a thousand under replacement, per ICLR 2025 work); accumulating synthetic rows on top of real ones bounds the error (Stanford SALT, 2024). The tail still erodes inside the synthetic share, so diversity is measured (Vendi score, number of embedding clusters, the minimum neighbour distance distribution) and generation stops when it falls, rather than when a row count is reached.

The only ground truth is a trained student. SmolLM2, Tulu 3 and OpenThoughts all chose data by training a model on it and evaluating; a 5 to 10% judge audit of a new set (the LLM Data Auditor framing: diversity, correctness, instruction adherence, safety) decides whether to filter row by row before that.

Korean. Translated corpora carry artefacts (KIT-19's finding, and the reason GLAN-QnA-KR and Magpie-ko generate natively); the recipe is native generation from Korean personas or a Korean-labelled taxonomy, a Korean rubric for the judge, and decontamination against KMMLU, HAE-RAE, KoBEST and CLIcK.

  • Seeds as first-class input: taxonomy, personas, documents, existing rows; a run records which seeds produced which rows, so coverage can be read off the manifest.
  • Questions from at most a few sources per run, and answers_per_question as a setting (default above 1 for reasoning and preference tasks); the teacher is a config choice to be compared by a student's evaluation.
  • The Magpie filter chain as named filters with thresholds in the config: schema, completeness and repetition, judge tags (quality, difficulty), reward or judge score with a margin against a base answer, minimum neighbour distance, length policy. Rejections are kept with reasons.
  • Verifiers before judges: heuristics for constraints, REWARDS for answers and code, a judge from another model family last. Self-revision as an optional second pass with a rubric.
  • Leak removal against the evaluation sets and benchmarks the model will be measured on, in every run.
  • Diversity numbers in the manifest (clusters, neighbour distances, Vendi score) and a stop rule on them.
  • The student loop: a sample config that trains the pilot recipe on the new rows and runs evaluation_compare against the base model, so a data run ends with a measured verdict rather than a row count.
  • An evolve step (WizardLM lineage) as a registry of operators with their prompts: the five in-depth and the in-breadth operators, the code heuristics, the math upward and downward ones; rounds and the operator draw per round in the config; the four elimination rules as named filters; an optional Auto-Evol pass that lets the teacher rewrite the operator against a development set's failure rate.
  • Prompt factories (Nemotron lineage): topic and keyword lists as seeds with per-task-type templates and a refinement call; three-turn role-played dialogues with user-style variants and politeness stripping; verifiable constraints appended to prompts; reasoning on and off responses paired on the same prompt.
  • Candidate-then-choose everywhere: answers_per_question, equivalence judging and majority vote as ground truth, judge verdicts accepted only when consistent under swapped order, difficulty by pass rate of the student (drop prompts it already solves, order the rest easy to hard).
  • Arena-style selection for preference and SFT rows: the student answers next to stronger models, a judge scores both orders, losses become SFT targets and win-loss pairs become preference rows.

Source: data/README.md. This page is generated; edit the source and run npm run sync.