Depth Through Recurrence: Looped Transformers for Flow-Matching TTS
1 Shenzhen Loop Area Institute, Shenzhen, China
2 The Chinese University of Hong Kong, Shenzhen
Weight reuse, executed depth, and sampling budget in flow-matching TTS
Abstract
We study how to organize Transformer depth through recurrence in flow-matching text-to-speech, varying the amount, order, and placement of weight reuse. Seven layouts perform 18 block calls per network evaluation under a common training objective and sampler. On Seed-TTS and LibriSpeech-PC, SEQUENCE applies each of nine blocks twice consecutively, retaining competitive intelligibility, speaker similarity, and predicted speech quality at 32 sampling steps with 47.1% fewer parameters than the unshared baseline. Cycling six blocks three times further reduces model size but raises 32-step word error rates relative to cycling nine blocks twice. At matched parameter counts and executed depth, reuse order and sharing position produce different quality trade-offs. These comparisons depend on sampling budget: Prefix and Suffix have similar 32-step word error rates, but Suffix is worse by 3.44 and 5.97 percentage points at four steps on the two datasets, respectively. Only Middle ranks first or second in mean word error rate at 32 and four steps on both datasets. These results show that the organization of recurrent computation affects synthesis quality, and that reuse layouts should be selected for both the sampling budget and the quality dimensions of interest.
Seven layouts, the same executed depth
Full loops vary how much is shared. CYCLE and SEQUENCE vary the order of reuse. Partial loops vary where sharing occurs, at the same parameter count.
Loop placement and speech quality
Compare all seven models at 32 and 4 sampling steps. Baseline, CYCLE 9 × 2, SEQUENCE 9 × 2, and Loop 6 × 3 provide the full-stack comparison; Prefix, Middle, and Suffix each have 108.4M parameters.
At four steps, Suffix has 3.44 / 5.97 percentage points higher WER than Prefix on Seed-TTS / LibriSpeech-PC. Among the partial loops, Middle has the lowest 32-step WER on both datasets.
WER across 4, 8, 16, and 32 steps · All seven models, fixed inference seeds.
Audio samples
Compare the Baseline with CYCLE 9 × 2 and Loop 6 × 3 to hear the effect of sharing more weights. Compare CYCLE and SEQUENCE to hear the effect of reuse order. Compare Prefix, Middle, and Suffix to hear the effect of sharing position, then listen across the two sampling budgets.
Loading the selected sentence…
Transcript
| Model | 32 steps | 4 steps |
|---|
Six examples selected by a fixed rule across short, medium, and long utterances, before inspecting model scores. Individual clips can differ from the aggregate ranking above. All models use the same text, reference voice, and inference seed.
Audio settings and sample selection
Seed-TTS test-en; seed 0; 500k-update EMA checkpoints; Euler sampling, CFG 2, sway −1. Two examples per target-duration group, with distinct reference prompts. Original WAV files are served without trimming or loudness adjustment.
See the sample manifest for the fixed selection rule, transcripts, and per-clip objective metrics, and the audio checksums for file verification. These examples are not a human listening-test result.
All seven models
Compare intelligibility, speaker similarity, and predicted naturalness alongside parameter count and memory relative to the Baseline. The metric selection is shared with the figure above.
| Model | Params (M) | Seed-TTS WER ↓ | LibriSpeech-PC WER ↓ | Allocated memory (% of Baseline) ↓ | |||
|---|---|---|---|---|---|---|---|
| 32 steps | 4 steps | 32 steps | 4 steps | Inference | Training* | ||
Mean utterance WER (%): means over four inference seeds on Seed-TTS and three on LibriSpeech-PC. Bold: lowest mean in each WER column. All quality results use 500k-update EMA checkpoints.
Memory: peak allocated memory relative to Baseline (100%).
*Training: recorded cumulative allocation peaks.