Behrens/ttslibre-poc-es-en-de

ttslibre 017: trilingual es+en+de PoC

19.8M-param flow-matching TTS checkpoint, warm-started from the 016 bilingual es+en checkpoint and continued on an interleaved Spanish/English/German mix (4h per language: OpenSLR-61 Argentinian Spanish, LibriTTS-R English, Thorsten-Voice German).

Question. Does German fit into the shared 19.8M model without degrading Spanish/English?

Answer (this run). Not stably. Across ~8h of training (4h original budget + a 4.5h extension after German hadn't converged), held-out German WER oscillated between roughly 0.48 and 1.24 and never held under the 0.3 success threshold for 3 consecutive evaluation rounds. Spanish crossed under 0.3 repeatedly (best 0.16) and English improved from its 0.89 baseline into a 0.6–0.8 range without forgetting. So the other two languages held up fine under the added German data; German itself didn't converge to the target WER in this budget.

Checkpoints

File	Step	Wall time	Notes
ttl_best_step33000.pt	33,000	~2h18m	Best mean held-out WER checkpoint (0.50) from the whole run
ttl_4h_step60000.pt	60,000	~4h11m	Checkpoint at the original 4h training budget boundary
ttl_8h_final_step103828.pt	103,828	8h00m	Final checkpoint at the end of the extended (8h total) run
ttl_8h_final_step103828_resumable.pt	103,828	8h00m	Same final checkpoint, including optimizer state

ttl_best_step33000.pt, ttl_4h_step60000.pt, and ttl_8h_final_step103828.pt are {"model": state_dict, "step": int, "vocab": [...], "elapsed": float} — model weights only, smallest to download for inference/synthesis. ttl_8h_final_step103828_resumable.pt is the same final weights plus "opt": optimizer_state_dict, for resuming training directly (~3x the file size).

Supporting files

• config.json — full experiment config (architecture, training hyperparameters, data sources)
• vocab.json — character-level tokenizer vocabulary (Spanish base + German chars/tags appended, ids unchanged from 016)
• ae.pt — the shared mel↔latent ConvNeXt autoencoder (same across all ttslibre experiments)
• latent_stats.pt — per-channel latent mean/std for the autoencoder

WER trajectory (held-out probes, per checkpoint)

step     de     en     es
0        1.91   0.89   0.62   (016 baseline, before any German training)
3000     1.15   0.83   0.50
6000     0.54   0.85   0.31
9000     0.59   0.73   0.31
33000    0.48   0.61   0.39   (best mean WER: 0.50)
54000    0.73   0.81   0.16
60000    0.55   0.72   0.52   (4h budget boundary)
75000    0.68   0.66   0.26
103828   1.05   0.69   0.38   (final)

German never stabilizes below 0.3 for 3 consecutive rounds; Spanish and English are healthy throughout.

Architecture

Character tokenizer (no G2P), mel↔latent ConvNeXt autoencoder (4× compression), text-to-latent flow-matching vector field with reference-encoder style tokens, utterance-level duration predictor, Euler sampler with CFG, Vocos as external mel vocoder. Same architecture family as Supertonic.

Part of ttslibre — training code MIT, aiming for a fully open (CC0-track) sub-100M TTS.