Cola DLM
paperYour notes
"Continuous Latent Diffusion Language Model." A hierarchical latent diffusion LM that replaces token-level autoregression with continuous-latent prior transport: a Text-VAE encodes tokens into continuous latents, then a block-causal DiT prior under flow matching generates new latents that the VAE decodes back to text. ~2B params; scales to ~2000 EFLOPs of pretraining.
Another diffusion-LM bet from a frontier lab, complementing Seed Diffusion's discrete-state approach with a continuous-latent formulation. Open code + weights + checkpoints.
Cola2 (arXiv 2610.31305, 22 October 2026) is the sequel from the same first author, Hongcan Guo (HKU, with ByteDance Seed), billed as a Seedance Team research project. It replaces the fixed latent layout with dynamic semantic spans, about two tokens per latent and at most eight, and controls the compression rate with an auxiliary-loss-free negative-feedback loop. The decoder becomes a block-causal masked-diffusion decoder trained on clean, noisy and dropped latents, with classifier-free guidance setting the balance between reconstruction and free generation. Token-prediction gradients flow back into the encoder, while the flow-matching prior is stop-gradiented off it. Controlled ablations at about 210B training tokens show scaling that has not saturated. A 3B budget is best split as a 2B DiT plus a 1B decoder. The headline checkpoint is not a from-scratch model. Its decoder and encoder embedding are initialised from Qwen3-4B-Base and trained with an ~4B latent DiT on OLMo 2 data over 346B source tokens. That unconverged checkpoint averages 60.10 over eight tasks, against Llama-3-8B's 66.25, and beats Llama-3-8B on HellaSwag (41.75 vs 39.21) and StoryCloze (80.42 vs 79.49). It still trails Qwen3-8B-Base (75.87) and LLaDA-8B-Base (72.11) by a wide margin, and the comparison is not compute-matched. The 346B-token checkpoint (about 8.8B parameters across VAE, DiT and decoder) and inference code followed on 2026-11-06 under ByteDance-Seed/Cola2; training code is not included.