Skip to main content

Nemotron and WizardLM, in detail

Two families of recipes shaped the pipeline more than the others. Nemotron is a staged factory with a reward model at every gate; WizardLM makes instructions harder and proves they got harder. This page records what each actually did, operator by operator, so the choices in data_synthetic can be traced to their source.

Nemotron (NVIDIA): a staged factory with a reward model at every gate.

  • Nemotron-4 340B (2024): over 98% of alignment data synthetic, 200k SFT rows plus 800k code rows and 160k preference triplets, from about 20k human rows (10k SFT, 10k HelpSteer2 for the reward model).
    • Prompts: 3,000 topics (LLM-generated macro-topics and subtopics plus hand-collected ones) and keyword lists (12k Python, 17k math) mined from pretraining data and Wikipedia; per task type a template, for example open Q&A "Can you generate {n} questions or requests related to {topic}?", then a refinement call that makes the question "more detailed and specific"; writing prompts name a document type; closed Q&A prompts are generated over C4 passages; instruction-following prompts concatenate a base prompt with a verifiable constraint ("Your response should have three paragraphs"); two-turn prompts start from ShareGPT first turns; LMSYS-Chat-1M real prompts are mixed in (unsafe ones kept only for preference data).
    • Dialogues: three turns by iterative role-play, the generator alternating user and assistant; three user styles (normal, "complex and diverse", "critical, concise, real-life tone"), greedy decoding, politeness ("Thank you for ...") stripped; the reward model scores every dialogue and drops those under a threshold.
    • Preferences: several intermediate models answer each prompt; the judge is the ground truth where one exists (GSM8K, MATH, verifiable constraints), an LLM judge with both orders tried and only consistent verdicts kept, or the reward model (0.87 vs 0.54 accuracy against the LLM judge on Chat-Hard).
    • Weak-to-strong: Mixtral-8x7B generates data for an intermediate 340B checkpoint, which then generates the next round's data; the loop is the point, not a single pass.
  • Llama-Nemotron (2025, 33M rows released): math from AoPS forums (middle school excluded), problems extracted and classified by an LLM, question variations prompted from Qwen2.5, 16 DeepSeek-R1 reasoning traces and 64 Qwen2.5-Math non-reasoning solutions per problem, answers checked for equivalence by a Qwen2.5-32B judge, majority vote as ground truth where none is extractable; code from 28.9k deduplicated competitive-programming problems (TACO, APPS, CodeContests, CodeForces), R1 solutions at temperature 0.6 and top-p 0.95, syntax checked, code stripped out of reasoning sections, under 0.3% benchmark overlap by cosine similarity plus two LLM judges, and the finding that harder problems (CodeContests) gave the largest gains and the scaling curve did not plateau at 736k; STEM MCQs generated by topic and difficulty with variations, decontaminated against GPQA, MMLU and MMLU-Pro; chat from 20k ShareGPT and WildChat first turns, responses by a Nemotron-70B instruct model through a feedback-edit-select loop, filtered by the Nemotron-70B reward model; every prompt paired with a "detailed thinking on" and a "detailed thinking off" response so the toggle is learnt; RL with GRPO, an LLM accuracy judge and a format reward for the think tags, prompts with a pass rate of 0.75 or higher (8 generations of the previous model) discarded, and a curriculum from easy to hard by pass rate.

WizardLM (Microsoft): make the instruction harder, then prove it got harder.

  • Evol-Instruct (2023): from 52k Alpaca instructions, four rounds, one of six operators drawn at random per instruction per round, 250k instructions; 70k sampled for training beat Vicuna's 70k human rows. The in-depth operators share one rewriter prompt ("rewrite a given prompt into a more complex version ... can only add 10 to 20 words ... '#The Given Prompt#', '#Rewritten Prompt#', 'given prompt' and 'rewritten prompt' are not allowed to appear") with one method each: add constraints ("add one more constraints/requirements"), deepen ("the depth and breadth of the inquiry can be increased"), concretize ("replace general concepts with more specific concepts"), reasoning ("rewrite it to explicitly request multiple-step reasoning"), complicate input (a table, code or data added to the prompt); the in-breadth operator is a creator prompt ("draw inspiration from the #Given Prompt# to create a brand new prompt ... same domain ... even more rare ... LENGTH and complexity similar"). Elimination drops an evolution when (1) a judge finds no information gain over the original, (2) the response contains "sorry" and is under 80 words, (3) the response is only punctuation and stop words, (4) the instruction copies prompt scaffolding such as "given prompt" or "#Rewritten Prompt#". Rows from every round are merged and shuffled so difficulties are spread evenly.
  • WizardCoder (2023): Code Alpaca 20k to 78k with one prompt ("increase the difficulty of the given programming test question a bit ... using, but not limited to, the following methods: {method}") and five code heuristics: add requirements (about 10 words), swap a common requirement for a rare one, add reasoning steps, give erroneous code as misdirection, demand higher time or space complexity (sparingly); rounds continue until a development set stops improving.
  • WizardMath (2023, RLEIF): 15k GSM8K and MATH seeds, five rounds (two downward evolutions that make the question easier or move it to an easier topic, three upward with constraints, concretizing and more reasoning), six variants per round, 448k rows, 17k duplicates and 30k contaminated rows removed, 418k kept. Two reward models: an instruction reward model trained on GPT-4 rankings (1 to 6) of evolved instructions against the original for difficulty and clarity, and a process reward model trained on GPT-4 step labels; PPO's reward is the product of instruction quality and the minimum step correctness.
  • Auto Evol-Instruct (2024): the operator is written by an optimizer LLM rather than a person: start from a universal method (list ways to complicate, plan, rewrite, review and fix), analyse a trajectory of evolutions for three failure types (stagnant complexity: the response starts "Understood" and ends with "?"; insufficient qualification: starts "Sure", ends "?"; loss of key information), rewrite the method, sample five candidates per step, keep the one with the lowest failure rate on a 50-example development set, 10 to 12 steps; then evolve the whole set once. 10k ShareGPT rows: MT-Bench +0.44, AlpacaEval +3.4; 7k GSM8K rows: +11.9 on GSM8K.
  • Arena Learning (2024, the WizardLM-2 flywheel): 276k deduplicated instructions, the target model battles stronger models, a judge (Llama-3-70B) scores both answers 1 to 10 with explanations and position swapped, the losses become SFT rows with the winner's answer as target, win-loss pairs become DPO data, judge scores feed PPO; three iterations over nine data shards, each iteration SFT then DPO then PPO; the offline test set (WizardArena: 500 K-means clusters times two for diversity, the 1,000 hardest of 10,000 by GPT-4 for difficulty) agrees 98.79% with the human arena; a third of the data (33.7k of 90k) sufficed for the final SFT, with difficulty rising 4.7 to 7.4 across iterations.

What the two lineages agree on: generate many candidates and let a verifier, reward model or judge choose; make difficulty an explicit, measured axis (tags, pass rates, instruction reward, elimination rules); iterate with the model being trained inside the loop; decontaminate before counting rows.

Source: data/README.md. This page is generated; edit the source and run npm run sync.