llm_dpo: preference pairs

DPO teaches a model which of two answers to prefer, without a reward model and without sampling. Use it after SFT, when you can say for a prompt which answer is better (human ratings, a judge, a verifier that rejected one of them). The base model is the reference: its log-likelihood margins are computed once before training, and the policy is pushed to prefer chosen over rejected more than the reference does.

Rows

{"prompt": "What is 9 plus 2? Reply with the number only.", "chosen": "11", "rejected": "9"}

prompt is a string or a list of chat messages.

Config

configs/llm/dpo.yaml

# Direct Preference Optimization: prefer `chosen` over `rejected` more than the base model does.
pipeline:
  recipe: llm_dpo
  name: dpo-qwen3.5-0.8b
  seed: 20260920
  output_dir: ./output

model:
  architecture: hybrid              # Qwen3.5: linear-attention + full-attention layers, vision tower frozen
  name: Qwen/Qwen3.5-0.8B           # or Qwen/Qwen3.5-2B @ 15852e8c16360a2fea060d615a32b45270f8a8fc
  revision: 2fc06364715b967f1860aea9cf38778875588b17

data:
  sources:
    - name: local_jsonl
      path: samples/llm_dpo.jsonl   # rows: {"prompt": str | messages, "chosen": ..., "rejected": ...}
  validation: 0.1

method:
  beta: 0.1                         # smaller lets the policy move further from the base model

training: {epochs: 1, lr: 5.0e-6, batch_size: 4, accumulation: 2, max_length: 512, max_steps: 4}

validation:
  min: {accuracy: 0.5}            # share of held-out pairs where chosen is more likely than rejected

Use

uv run llmclick run configs/llm/dpo.yaml

• Gate: accuracy, the share of held-out pairs where chosen is more likely than rejected; margin is the mean log-likelihood gap.
• Settings: method.beta bounds how far the policy moves from the reference; smaller lets it move further. Start from an SFT checkpoint with model.init: output/<sft experiment>/validate/checkpoint.
• Output: output/dpo-qwen3.5-0.8b-<hash>/validate/checkpoint. The Data channel makes pairs with select.assemble: dpo (a judge's score gap decides which answer is chosen).

Source: configs/llm/dpo.yaml. This page is generated; edit the source and run npm run sync.