The third generation of Xiaohongshu's FireRedTTS line, and the first to unify generation with editing. Continuous autoregressive TTS models work directly on continuous speech representations, which keeps acoustic detail and lets them inherit a text LLM's instruction following — but they accumulate error as they generate, and the usual fixes bolt on semantic modules, multi-stage tokenizer pipelines or more elaborate autoregressive architectures. FireRedTTS3 attacks the problem at the representation level instead: a frozen Audio Encoder trained on diverse speech understanding tasks acts as a semantic teacher regularizing the audio feature space, which improves text-speech alignment and steadies generation while the surrounding system stays simple.

Two variants ship: FireRedTTS3-Base for multilingual and multi-dialect zero-shot voice cloning, and FireRedTTS3-Instruct for voice cloning, instruction-controlled voice design and speech editing in one model. Base records the best average speech intelligibility and speaker similarity among the compared systems on Seed-TTS-Eval and MiniMax-MLS-Test; Instruct beats competing systems on InstructTTSEval and Ming-Freeform-Audio-Edit. Apache 2.0 code and weights, with about 1,760 GitHub stars. The same team's FireRedAudio followed nine days later and reuses the frozen-encoder idea on both sides of a single audio-language model.

Model Details

License Apache 2.0

Variants

Name Parameters Notes
FireRedTTS3-Base — Multilingual and multi-dialect zero-shot voice cloning
FireRedTTS3-Instruct — Voice cloning plus instruction-controlled voice design and speech editing

Paper

Authors: Feiyu Shen · Kun Xie · Yichen Wu · Ziqi Dai · Yichen Han · Junjie Li · Xuelong Geng · Fenglong Xie
speechttsaudiomultilingualopen-weight

Related