Towards Looped Models Done Right, Part II: Rethinking at Fixed Points
paperYour notes
Looped language models reuse a block of layers several times per token, buying effective depth without new parameters. The bill arrives everywhere else: "every recurrence of a looped language model adds cost in training, decoding, prefill, and reinforcement learning." This paper's organizing claim is that as recurrent states approach fixed points, the path to them stops mattering, so the endpoint can stand in for the whole trajectory. That is an empirical property of a particular model, not a universal one, and the paper leads with the counterexample: reusing terminal KV caches across loops barely changes Huginn's zero-shot GSM8K but collapses that of Ouro at four loops — because Huginn's states converge and Ouro's do not. Four shortcuts follow from tolerant fixed points: truncated backpropagation in pretraining, terminal-KV sharing in decoding, a distilled student for prefill, and RL updates computed from saved rollout states.
So the method targets the two training components that shape those fixed points. First, the depth prior: fixed-depth training breaks KV sharing outright, while Huginn's broad Poisson-lognormal prior supports sharing but dilutes supervision at the target depth more than sharing actually requires. They replace it with a categorical distribution over depths learned from prediction feedback, plus an entropy term that keeps it broad and a budget term. Second, input injection: Huginn's and Parcae's schemes leave the state's component along the injected input unconstrained, so depending on sign it amplifies or cancels the injection; orthogonal injection (OrthoInj) projects that component out so every recurrence receives the input at the same strength.
Trained at three scales — S (100M, 21.5B tokens), M (400M, 85.9B) and L (1.6B, 343.6B) — the learned prior lowers validation perplexity by 0.7–1.8% relative to the fixed PLN-5 prior for at most 1.6% more training FLOPs, and at 1.6B with a 3× smaller KV cache it matches the full-cache downstream average of fixed-depth training. The entropy term is load-bearing: without it, GSM8K at L drops from 47.92 to 36.47. The contrast case is stark — fixed-depth training with full backpropagation through time, then terminal sharing, sends GSM8K from 50.6 to 21.2, while entropy-regularized learned-prior models stay within 0.4 points of their full-cache downstream average. OrthoInj beats Parcae's injection, the strongest baseline, by 0.3–1.4% validation perplexity across all three scales.
The two new shortcuts are honest about their costs. Distilled prefill, trained on a quarter of the teacher's pretraining tokens, is 1.5–1.8× faster on 8K prompts (up to 1.79×) but trails the teacher's downstream average by 1.0 (S), 2.3 (M) and 4.5 (L) points — a gap that grows with scale; at matched latency it is only 0.5–0.9 points above simply stopping the teacher at two recurrences. Reusing rollout states for RL cuts scoring and backward time 1.99× on GSM8K and 2.02× on MBPP+ versus replaying the loops with full BPTT, and stays within 1.6 points of that baseline's GSM8K pass@1 (all four variants move GSM8K pass@1 from 40.6% to 60–63% and MBPP+ from 30.9% to 37–41%). The authors' own limitations: models up to 1.6B with one seed per configuration, ablations that "do not exhaust the design space," and larger scales left untested. Part I — topology, input injection and recurrent-state organization, 17 dense and MoE recipes — reached arXiv as 2610.23036 on 2026-10-16. Training from scratch at matched parameters, logical depth and tokens, it favours the Huginn-style prelude–recurrent–coda design: at 730M parameters and 336B tokens full Huginn beats Ouro on all ten benchmarks, and the advantage carries over to an 8B-A0.8B MoE (32B-A3B unrolled) trained on 500B tokens. Input injection mainly helps knowledge retrieval and context use; random state initialisation and separate H/L recurrent states give mixed or negative returns. The xllm-loop release (~39 stars, Apache 2.0) adds looped-Transformer and Huginn architectures with depth-controlled training to MBZUAI-IFM's xLLM, and publishes 49 checkpoints across both parts (25 for Part I, 24 for Part II), including ouro-topology recipes. Same MBZUAI-IFM and USC-ISI pairing as EMO; for the MoE side of looped models at matched compute, see SMELT.