Text diffusion: A new paradigm for LLMs

Text diffusion is a new paradigm for LLMs. As opposed to mainstream auto-regressive models like GPT, Claude or Gemini (which predict one token at a time), diffusion-based LLMs draft an entire response and refine it progressively. This leads to 10x faster inference. Models like Gemini Diffusion, Mercury Coder from Inception Labs and Seed Diffusion from ByteDance are already competitive on coding benchmarks. Inspired by physical diffusion, such models make use of Markov chains to model data generation as a particle hopping through discrete states. We'll walk through the D3PM and LLaDA papers as case studies. 📖 Papers: Full reading list: https://www.patreon.com/posts/papers-diffusion-140452266 D3PM: https://arxiv.org/abs/2107.03006 LLaDA: https://arxiv.org/abs/2502.09992 Scaling up Masked Diffusion Models on Text: https://arxiv.org/abs/2410.18514 ▶️ The physics behind diffusion models: https://youtu.be/R0uMcXsfo2o?si=OqdGg4TPefSNTK3t 00:00 Intro 01:04 Auto-regressive vs diffusion LLMs 02:06 Why bother with diffusion for text? 06:30 The probability landscape 07:57 Diffusion in latent embedding space 11:00 Diffusion in token embedding space 12:13 Diffusion in text token space 13:49 Markov chains 16:46 Paper study: D3PM 19:42 Paper study: LLaDA 22:30 Evaluation