A replacement for auxiliary multi-token prediction (MTP) from Baidu's ERNIE team. Standard MTP trains extra modules attached to the backbone, and each added horizon costs another module and another forward pass. ECHO instead has the backbone's own final layer predict every horizon, so the multi-horizon supervision trains the same parameters that produce the next token. Dynamic Depth Sampling keeps deep horizons affordable by sampling the training depth with probabilities that decay with depth.

The authors report two effects. In pretraining, extra depth turns into downstream accuracy where auxiliary modules saturate, mostly on reasoning and code, at half the overhead of training MTP at every depth. At inference, the pretrained model drafts for itself, drafting beyond the depth it was trained for. It keeps its draft-target alignment on open-ended, high-entropy generation, where post-hoc drafters lose theirs. The advantage holds with a doubled token budget, a repeated pretraining run and supervised fine-tuning. The abstract gives no model scale. The method ships in ERNIE 5.5 Flash Base (280B-A14B). There it replaced the MTP module, was trained with single-step ECHO, and its reused final layer serves as the speculative-decoding draft.

Paper

Authors: Yuxuan Zhou · Junyuan Shang · Tianyu Wang · Chunrui Zheng · Zichang Liu · Wenbin Chang · Shuohuan Wang · Dianhai Yu
architectureefficiencyinferenceresearch

Related