FireRedAudio
modelYour notes
A general-purpose audio language model from Xiaohongshu's FireRed team, built around a single shared 9B-parameter LLM. Its argument is about representation: understanding wants compact features that survive long-context modelling, while speech generation wants reconstructible features that keep fine acoustic detail, and the two pull in opposite directions. FireRedAudio keeps both — the team says it is the first publicly disclosed unified audio-language model with separate continuous input representations for understanding and generation inside one trainable autoregressive LLM. Audio to be recognized or analysed goes through a dedicated Audio Encoder; speech destined for generation takes a RedAE-based path. The LLM then either emits text directly or conditions a flow-matching DiT that produces continuous acoustic latents.
Progressive multitask training gives it ASR and audio understanding over recordings of up to an hour, zero-shot and instruction-guided TTS, and both semantic and acoustic speech editing, with second-level timestamp accuracy when it structures long-form audio. The paper reports competitive or leading results on audio understanding and multilingual ASR, strong content accuracy and speaker preservation in zero-shot TTS, leading instruction following on Instruct TTS, and substantial gains over Ming-UniAudio-Edit on both kinds of speech editing — the case being that decoupled continuous input representations can unify understanding and continuous-latent generation at moderate scale. Apache 2.0 weights and code; about 2,000 GitHub stars within six weeks, the FireRed team's most-starred release.