llm_grpo: rewards

GRPO improves a model on tasks whose answers can be checked: arithmetic, an exact label, valid JSON, a rule. For every prompt the model samples a group of completions, a reward function scores each, and the completions above their group's mean are reinforced. No reward model and no value network: the group is the baseline.

Rows

{"prompt": "What is 9 plus 2? Reply with the number only.", "answer": "11"}

prompt is a string or chat messages; the other keys are what the reward reads. Shipped rewards: exact_match, contains, last_number (the last number in the completion against answer, for GSM8K-style problems).

Config

configs/llm/grpo.yaml

# Group Relative Policy Optimization: sample completions, reward them, reinforce the ones above their group's mean.
pipeline:
  recipe: llm_grpo
  name: grpo-qwen3.5-0.8b
  seed: 20260920
  output_dir: ./output

model:
  architecture: hybrid              # Qwen3.5: linear-attention + full-attention layers, vision tower frozen
  name: Qwen/Qwen3.5-0.8B           # or Qwen/Qwen3.5-2B @ 15852e8c16360a2fea060d615a32b45270f8a8fc
  revision: 2fc06364715b967f1860aea9cf38778875588b17

data:
  sources:
    - name: local_jsonl
      path: samples/llm_grpo.jsonl  # rows: {"prompt": str | messages, "answer": ...}
  validation: 0.1

method:
  group_size: 4                     # completions sampled per prompt
  max_new_tokens: 8
  temperature: 1.0
  reward: {name: exact_match}     # or contains; add your own in modeling/llm/methods/grpo.py

training: {epochs: 1, lr: 2.0e-6, batch_size: 2, accumulation: 2, max_length: 256, max_steps: 4}

validation:
  min: {reward: 0.0}              # mean reward of the greedy completion on held-out prompts

configs/llm/grpo_gsm8k.yaml trains on GSM8K word problems from the Hub with the last_number reward.

Use

uv run llmclick run configs/llm/grpo.yaml

• Gate: reward, the mean reward of the greedy completion on held-out prompts.
• Cost: sampling dominates. method.group_size completions per prompt, each up to method.max_new_tokens; training.max_length must leave room for them, and the length check counts them before the first step.
• Output: output/grpo-qwen3.5-0.8b-<hash>/validate/checkpoint.

Reusing sampled groups, and writing a reward

Sampling is the expensive part of both methods. method.iterations takes that many optimizer steps on each sampled group before sampling again. From the second step on the policy has moved away from the one that produced the samples, and method.clip bounds how far: GRPO clips the policy ratio to 1 +- clip (PPO's surrogate), CISPO caps the importance weight of each reasoning token at 1 + clip. llm_grpo also takes method.beta, a KL penalty towards the base model; it needs training.adapter, because the base model is then the same weights with the adapter switched off and no second model is loaded.

method: {group_size: 4, iterations: 2, clip: 0.2, beta: 0.05}
training: {adapter: {name: lora, r: 16, alpha: 32}}

@REWARDS.register("json_valid")                 # modeling/llm/methods/grpo.py
def json_valid(completion: str, row: dict, params: dict) -> float: ...

Source: README.md, configs/llm/grpo.yaml. This page is generated; edit the source and run npm run sync.