Wake word detectors: `evaluation_wakeword`

A wake word detector fails two ways: it misses the name, or it wakes on something else. This recipe measures both on what the detector never trained on, the way wake word detectors are compared elsewhere.

Number	Measured on	Target
miss rate (FRR), per noise condition	the name said by voices the detector never heard, clean and at 10 and 5 dB	under 5% (openWakeWord)
false accepts per hour	hours of speech that never says the name, as one long stream	0.5 an hour (openWakeWord), 0.1 (Picovoice)
negatives fired	near misses and commands said by unseen voices	as low as the sweep allows
the sweep	all three at every threshold from 0.01 to 0.99	the miss rate at each target

wakeword's own validate says whether a detector passed on a held-out split of its training voice; this recipe says how it will do in the room.

Config and details

pipeline: {recipe: evaluation_wakeword, name: jarvis-wake-eval}
model: {experiment: output/jarvis-wake-<hash>, checkpoint: validate}   # validate | train | a bundle directory
data:
  positive: [{name: local_jsonl, path: output/wake-eval-phrase-<hash>/select/rows.jsonl}]     # the name, other voices
  negative: [{name: local_jsonl, path: output/wake-eval-negatives-<hash>/select/rows.jsonl}]  # near misses, commands
  ambient: [corpora/LibriSpeech/test-clean]   # WAVs that never say the name: false accepts per hour
  noise: []                                    # noise WAVs for the noisy conditions; empty: Gaussian
conditions: [null, 10, 5]   # clean, then SNR in dB
threshold: 0.5
refractory: 1.0
targets: [0.5, 0.1]         # false accepts per hour at which the miss rate is read

validate of the wakeword recipe says whether a detector passed on a held-out split of its own training voice. This recipe says how it does on what it never heard, the way wake word detectors are compared elsewhere:

Miss rate (FRR) per recording, under each condition: the recording in a stream of its own, 2 s of noise before it, as validate builds them. It counts as heard if the detector fires between the start of the name and 1 s after its end; the report keeps the median time from the end of the name to that firing.
False accepts per hour on the ambient audio, read as one long stream (as an always-on microphone hears it), in 60 s chunks that overlap by one window, so the scores equal those of the whole stream. A false accept is an event: a firing, then nothing counted for refractory seconds. openWakeWord's training script counts every window over the threshold instead, which counts one long firing several times.
Negatives fired: the share of short recordings that must not wake it, per condition, and per text at the operating point, so the near misses that remain are named.
The sweep: all three at every threshold from 0.01 to 0.99, and the miss rate at the most sensitive threshold whose false accepts per hour stay within each target: 0.5 per hour is openWakeWord's target for its models (with under 5% misses), 1 per 10 hours is the operating point of Picovoice's benchmark. Intervals: Wilson for shares, Poisson for rates; with 5 hours of ambient audio, zero events still allows up to 0.7 per hour.

Stages: clips (find the recordings, list the ambient and noise WAVs with their length) -> traces (the detector's score every 80 ms on everything, scope wakeword-traces) -> report. Thresholds, refractory and targets are read only by the report, so a new operating point rereads the scores instead of listening again. Each recording's noise is drawn from the seed and the recording's content.

Data: data/wake_eval_phrase.yaml and data/wake_eval_negatives.yaml have another Piper voice (VCTK, 109 speakers of British English accents) read the name, near misses and commands; training used LibriTTS-R, so neither the voice model nor any speaker is shared. Ambient audio is WAV: the config shows how to fetch LibriSpeech test-clean (5.4 h, CC BY 4.0) and convert it with ffmpeg. Not yet: room reverberation (RIRs), real far-field audio such as DiPCo, and interpolation between swept thresholds.

Source: evaluation/README.md. This page is generated; edit the source and run npm run sync.

Config and details​

Config and details