Skip to main content

tts_piper: a voice

A Piper voice is a small VITS model that reads phonemes and runs anywhere as one ONNX file. This recipe starts from an existing Piper checkpoint, trains it on recordings with their texts, exports the result, and then has Whisper listen to the new voice reading texts it never trained on. The voice passes when Whisper understands it about as well as it understands the recordings themselves.

The training itself is Piper's: the recipe runs Piper's own trainer as a separate program, in an environment of its own (Piper is GPL), and adds what the rest of this repository gives every model: fingerprinted stages, resuming after a crash, and a validation gate before the voice is passed on.

Rows​

audio (a WAV file) and text, from rows.jsonl that data_speech writes or a Piper metadata.csv (<file>|<text> lines) of your own recordings.

Config​

configs/tts/piper_jarvis.yaml
# Fine-tune a Piper voice from the jgkawell/jarvis medium checkpoint on the set configs/data/speech.yaml writes.
# Needs the Piper environment (environment/piper/setup.sh) and, for a real run, a GPU: on a CPU an epoch over a few
# hundred clips takes minutes. Replace <hash> with the data experiment's (`llmclick validate configs/data/speech.yaml`).
pipeline:
  recipe: tts_piper
  name: jarvis-medium-ft
  seed: 20260920
  output_dir: ./output

model:
  quality: medium                   # the starting checkpoint's size
  espeak_voice: en-gb-x-rp          # the reference voice's phonemizer (its .onnx.json)
  sample_rate: 22050
  init:
    name: jgkawell/jarvis
    revision: 37f8763122312665f091d1fc760abaf1f79b02cc
    file: en/en_GB/jarvis/medium/checkpoints/epoch=8338-step=2012650.ckpt

data:
  sources:
    - name: local_jsonl
      path: output/jarvis-lines-<hash>/select/rows.jsonl   # audio paths are relative to this file
    # - name: delimited                                    # recordings of your own: `<file>|<text>` lines
    #   path: recordings/metadata.csv
    #   audio_dir: recordings/wavs
  validation: 0.05

training:
  epochs: 100
  batch_size: 16
  # args: {model.learning_rate: 1.0e-4}                   # any other `piper.train fit` option

validation:
  asr: {language: en}
  rows: 50
  phrases:                          # the agent's read-backs, kept in <experiment>/validate/phrases/<name>.wav; the
                                    # agent plays one when it recognises that command after the wake word (wakeword/jarvis.yaml)
    wake_up: At your service, sir.
    web_search: Searching Reference.
    horizontal_task: Processing Long Task.
    remember_pattern: Checking the Pattern.
    analysis: Target. Analysis.     # the full stop makes the voice pause; as "Target Analysis." no take was heard right
    focus_mode_enter: Activate Focus mode.
    focus_mode_quit: Deactivate Focus mode.
    persona_karen: Change to KAREN.
    persona_jarvis: Change to JARVIS.
  phrase_takes: 5                   # takes per read-back; the one Whisper hears best is kept
  max: {wer: 0.15, phrase_wer: 0.0}  # compare wer with reference_wer: Whisper's error on the recordings themselves

Use​

environment/piper/setup.sh                        # once: Piper's locked environment
uv run llmclick run configs/data/speech.yaml      # the training set
uv run llmclick run configs/tts/piper_jarvis.yaml # replace <hash> first

The voice is output/jarvis-medium-ft-<hash>/validate/checkpoint/voice.onnx (piper -m voice.onnx), and each fixed reply is validate/phrases/<name>.wav.

How it works​

tts_piper fine-tunes a Piper voice: a VITS model that reads espeak-ng phonemes and runs anywhere as one ONNX file. Piper is GPL-3.0-or-later and pins its own torch, so it is not a dependency of this package: it lives in environment/piper/ (its own pyproject.toml and uv.lock), and both speech recipes run it as a separate process through PIPER_PYTHON. Build it once, and again after pulling: the script is safe to repeat.

environment/piper/setup.sh       # uv sync of the locked environment, plus the monotonic_align extension the wheel lacks
model:
  quality: medium                  # must match the starting checkpoint; high sets Piper's larger decoder
  espeak_voice: en-gb-x-rp
  init:                            # a Piper .ckpt on the Hub (pinned), a local .ckpt, or <experiment>/validate/checkpoint
    name: jgkawell/jarvis
    revision: 37f8763122312665f091d1fc760abaf1f79b02cc
    file: en/en_GB/jarvis/medium/checkpoints/epoch=8338-step=2012650.ckpt
data:
  sources:
    - {name: delimited, path: recordings/metadata.csv, audio_dir: recordings/wavs}   # `<file>|<text>` lines
training: {epochs: 100, batch_size: 16, args: {model.learning_rate: 1.0e-4}}         # args: any other `piper.train fit` option
validation: {asr: {language: en}, max: {wer: 0.15}}
  • Rows. audio and text. A relative audio path starts at the source's audio_dir, else at the directory of the source file, and an LJSpeech id without .wav finds its file. delimited reads a headerless | file such as an LJSpeech or Piper metadata.csv; local_jsonl reads the rows.jsonl that data_speech writes. The audio directories count in the data fingerprint, so re-recorded files rebuild the experiment.
  • Training. The starting weights are copied in (model.warmstart_ckpt: every tensor whose shape matches, a fresh optimizer), so an old Piper checkpoint loads into the current trainer and training.epochs counts from zero. Piper's own validation split is off; the validate stage measures on this recipe's held-out rows instead. A rerun after a crash continues from Piper's last epoch snapshot (run/lightning_logs/version_<n>/checkpoints/last.ckpt); Piper turns cuDNN determinism off, so a GPU run is not bit-identical to an earlier one. fit.log holds Piper's output; a cancelled API job stops when the fit ends.
  • Checkpoint. <experiment>/validate/checkpoint/ holds voice.onnx and voice.onnx.json (a Piper voice: piper -m voice.onnx, Home Assistant's /share/piper) and model.ckpt, the start of a further fine-tune.
  • Validation. The voice reads up to validation.rows held-out texts; Whisper (validation.asr, pinned) transcribes them and the original recordings. reference_wer is the floor wer can reach with that recognizer; numbers written as digits in one text and as words in the other count as errors.
  • Fixed replies. validation.phrases (name: text) are the lines the agent plays as they are, such as the read-back before a task (web_search: Searching Reference.). Piper's noise makes every take of a text different, so each is spoken validation.phrase_takes times (default 3) and the take Whisper hears best becomes <experiment>/validate/phrases/<name>.wav; phrases.json lists every take and what was heard, and phrase_wer is the worst kept take, so a voice that garbles one reply fails validation. Changing a reply reruns only validate; a rerun keeps the takes it has, so the agent plays the file that passed.

Before training on a voice, read its terms. jgkawell/jarvis is MIT on its weights per its model card, but it imitates a film character played by a real actor; MIT does not cover the voice, so treat it as personal and non-commercial.

Source: README.md, configs/tts/piper_jarvis.yaml. This page is generated; edit the source and run npm run sync.