Skip to main content

wakeword: the agent's name

A voice agent listens all the time, but it should act only after hearing its name, such as "Jarvis". This recipe trains the small detector that listens for that name; the command that follows it ("wake up", "search the web") is heard by speech recognition afterwards. It uses openWakeWord's frozen speech features and trains only a small scoring head on top of them, so a few hundred synthetic recordings are enough. The exported model loads in openWakeWord's runtime.

Validation plays the agent's first moment back to the detector: the user says the name and then something else, with noise underneath, and the detector must fire exactly once, on the name and not on what follows.

Rows​

Two sets: positive recordings say the name and nothing else; negative recordings never say it (near misses such as "Travis", the commands said alone, ordinary speech). Rows are audio and an optional text.

Config​

configs/wakeword/jarvis.yaml
# A detector for the agent's name, "Jarvis": it fires as the name ends, and what follows ("wake up", "search the web
# for ...") is the command, heard by speech recognition. Trained on openWakeWord's frozen speech features; the
# checkpoint directory holds the model, the two feature models and wakeword.json, and loads in openWakeWord's runtime:
#   openwakeword.Model(wakeword_models=["<checkpoint>/jarvis.onnx"], inference_framework="onnx",
#                      melspec_model_path="<checkpoint>/melspectrogram.onnx", embedding_model_path="<checkpoint>/embedding_model.onnx")
# Replace <hash> with the data experiments' (`llmclick validate configs/data/wake_phrase.yaml`, wake_negatives, wake_nearmiss).
pipeline:
  recipe: wakeword
  name: jarvis-wake
  seed: 20260920
  output_dir: ./output

phrase: Jarvis                      # the wake word; data/wake_phrase.yaml must say it, data/wake_negatives.yaml must not

data:
  positive:
    - {name: local_jsonl, path: output/wake-phrase-<hash>/select/rows.jsonl}
    # - {name: delimited, path: recordings/wake/metadata.csv, audio_dir: recordings/wake/wavs}   # your own voice
  negative:
    - {name: local_jsonl, path: output/wake-negatives-<hash>/select/rows.jsonl}
    - {name: local_jsonl, path: output/wake-nearmiss-<hash>/select/rows.jsonl}           # data/wake_nearmiss.yaml
  background: []                    # directories of noise WAVs (room tone, music, a fan); empty: Gaussian noise
  validation: 0.2

# followed: "Jarvis" with a command after it, as negatives, so the detector fires as the name ends and not again
augment: {rounds: 4, followed: 2, noise: 200, snr: [5, 25], gain: [-12, 0], tail: 0.3}

# negative_weight with the cosine schedule: over six seeds false accepts stayed at or under 4.7% (constant rate and equal
# weights: 3.7% to 8.8%, depending on the seed); recall stayed 100%
training: {steps: 3000, batch_size: 256, learning_rate: 1.0e-3, layer_dim: 32, blocks: 1, negative_weight: 100, schedule: cosine}

validation:
  threshold: 0.5
  snr: 20
  min: {recall: 0.9, scenario: 0.9}  # scenario: "Jarvis", then another utterance; it must fire once, on the name
  max: {false_accepts: 0.05}

Use​

uv run llmclick run configs/data/wake_phrase.yaml      # the name, read by 300 speakers
uv run llmclick run configs/data/wake_negatives.yaml   # near misses, commands, ordinary lines
uv run llmclick run configs/data/wake_nearmiss.yaml    # more near misses, more speakers
uv run llmclick run configs/wakeword/jarvis.yaml       # replace each <hash> first

Measure a detector on voices and hours of audio it never heard with evaluation_wakeword.

How it works​

wakeword trains the detector that wakes a voice agent: a small model that listens to the microphone all the time and fires when it hears one phrase. It follows openWakeWord (Apache-2.0): audio becomes a melspectrogram and then a 96-number speech embedding every 80 ms through two frozen ONNX models (pinned by hash, downloaded once), and a small dnn head scores the last 16 embeddings (2 s). Only the head is trained, so a few hundred synthetic recordings are enough, and the exported file loads in openWakeWord's runtime.

phrase: Jarvis                     # the wake word: the agent's name; what follows it is the command
data:
  positive: [{name: local_jsonl, path: output/wake-phrase-<hash>/select/rows.jsonl}]     # data/wake_phrase.yaml
  negative: [{name: local_jsonl, path: output/wake-negatives-<hash>/select/rows.jsonl},  # data/wake_negatives.yaml
             {name: local_jsonl, path: output/wake-nearmiss-<hash>/select/rows.jsonl}]   # data/wake_nearmiss.yaml
  background: []                   # directories of noise WAVs; empty: Gaussian noise
augment: {rounds: 4, followed: 2, noise: 200, snr: [5, 25]}            # followed: "Jarvis" + a command, as negatives
validation: {threshold: 0.5, min: {recall: 0.9, scenario: 0.9}, max: {false_accepts: 0.05}}
  • Phrase. The wake word is the agent's name; the command after it ("Jarvis, wake up", "Jarvis, search the web for ...") is not part of it, and speech recognition hears it once the detector has fired. Positives say the phrase and nothing else (a positive row with a text must say it); negatives must never contain it, and the data stage stops on one that does. The hardest negatives are near misses ("Travis", "Davis", "Karen"), and the commands said alone. The model file is named after the phrase: jarvis.onnx.
  • Recordings. data_speech with a multi-speaker voice (voice.speakers) and select.per_text: null makes them: the name read by hundreds of speakers, each checked by Whisper. Recordings of the people who will use the agent (delimited or local_jsonl) belong in positive too: synthetic voices are a start, not the target.
  • Windows. Every training recording becomes augment.rounds 2-second windows over noise at a random SNR and gain, ending up to augment.tail seconds after the speech. Negatives are also cut short at random, so half a phrase is a negative; augment.followed windows per phrase put other speech after it, so the detector fires as the name ends and not again while the command that follows is spoken. All randomness comes from the seed and the recording.
  • Training. Balanced batches (half positive), BCE, Adam, training.steps steps on the CPU, seeded: a rerun after a crash trains the same detector again.
  • Validation. Held-out recordings in noise (validation.snr), scored every 80 ms as the runtime scores them, a firing counted at most once a second: recall and false_accepts per recording, and scenario, the agent's first moment: "Jarvis", a pause, then another held-out utterance (a command such as "Please search to the paper reference on the web", a near miss, or ordinary speech). It passes when the detector fires exactly once, while or just after the name is said.
  • Checkpoint. <experiment>/validate/checkpoint/ holds jarvis.onnx, the two feature models and wakeword.json (the phrase, the files, the feature models' hashes):
from openwakeword.model import Model      # pip install openwakeword
detector = Model(wakeword_models=["checkpoint/jarvis.onnx"], inference_framework="onnx",
                 melspec_model_path="checkpoint/melspectrogram.onnx", embedding_model_path="checkpoint/embedding_model.onnx")
score = detector.predict(chunk)["jarvis"]   # chunk: 1280 samples, 16 kHz, int16; wake at >= the threshold

false_accepts is measured on the negatives you give, not on hours of real audio; openWakeWord's own models add a large corpus of everyday speech and music for that. Add real background recordings and your own speech before trusting the rate. openWakeWord's code is Apache-2.0 and its README puts its pre-trained models under CC BY-NC-SA 4.0; the two feature models are derived from Google's Apache-2.0 speech_embedding, so treat a detector as non-commercial until you have checked their terms.

Source: README.md, configs/wakeword/jarvis.yaml. This page is generated; edit the source and run npm run sync.