Diffusion Language Models vs Autoregressive Language Models

There is a new Google Gemini model which uses Diffusion to generate text instead of the more tried-and-true autoregressive token generation approach that GPT models use. In this video, we are breaking down what exactly text diffusion is. How these models are trained, and why this could be a big deal? We are also comparing them with standard GPT's across a variety of axes - underlying algorithms, ideologies, training methods, inference speeds, interpretability, and how controllable these two approaches are. We are also looking at the new LLaDA diffusion paper that also explores diffusion based large language models! #googlegemini #ai #diffusion #llm Follow on Twitter: https://x.com/neural_avb Buy me a coffee at https://ko-fi.com/neuralavb ! To join our Patreon, visit: https://www.patreon.com/NeuralBreakdownwithAVB Members get access to EVERYTHING behind-the-scenes that go into producing my videos. Plus, it supports the channel in a big way and helps to pay my bills. Videos to watch next: Full attention to LLM playlist: https://www.youtube.com/playlist?list=PLGXWtN1HUjPfq0MSqD5dX8V7Gx5ow4QYW Image Diffusion Models from scratch: https://youtu.be/w8YQcEd77_o Video Diffusion Models: https://youtu.be/KRTEOkYftUY References for further reading: Google Post: https://deepmind.google/models/gemini-diffusion/ Large Language Diffusion Models (LLaDA paper) - https://arxiv.org/pdf/2502.09992 Block Diffusion paper: https://arxiv.org/abs/2503.09573 DIffusion LM: https://arxiv.org/abs/2205.14217 Timestamps: 0:00 - Intro 2:24 - Ideological differences between Autoregressive and Diffusion 5:50 - Diffusion for Image Generation 6:55 - Diffusion for Text - LLaDA paper 8:42 - LLaDA Diffusion LLM training 9:53 - LLaDA Diffusion LLM Inferencing 11:51 - ARMs vs Diffusion - Speed, Scalability, Controllability 14:20 - Why Diffusion LMs can be HUGE!