Diffusion: AI Mental Model #7

Hello everyone. Welcome to lecture number seven of the great mental models of artificial intelligence. Today I'm going to talk about a mental model which is extremely non-intuitive. I would have never ever thought of this uh mental model, but now it it turns out that this mental model is showing up everywhere around us. So, when um the AI boom started, especially with ChatGPT, there was this startup called Midjourney and they produced AI-generated images. So, if you look at images like these, these are all AI-generated beautiful images. They had a Discord server which became extremely popular and people were just blown away by the kind of images which AI could generate. So, if you uh if some alien came into this world and if you showed them this image and uh you tell them that, "Hey, AI has generated this image." How would you go about doing this if you were tasked with generating such an image using AI? What that alien, a smart alien, would probably say is that I will teach the AI the process of drawing. I will teach the AI the process of painting. I will teach AI uh how pixels, when they are arranged together, how they look good together. I'll probably give the AI millions and billions of images and ask the AI to learn from it so that it can recreate or construct new images. That's how a normal person would have thought of it, right? But, the today's technique which we are going to look at it is completely different. What this technique says is that you start from complete noise. You start from chaos and you slowly construct a purer image from the chaos. Which means uh you teach the AI to denoise. Rather than thinking about building an image from first principles, you first break the image into noise, and then rebuild it. Rebuild the image from the noise. Let me clarify what I mean in today's lecture. So, today's lecture is titled reverse the corruption. So, the idea philosophically is that to learn how to create something, first we must learn how to destroy it. And in the destruction process, we just run the destruction process backward. It's like going back reverse in time. This is the the strangest but beautiful trick in artificial intelligence, right? And it shows it as we are going to see today, this trick shows up in image generation, in language generation, and in molecules like drug discovery also. We'll come to that a bit later. So, let's start from scratch. Suppose I ask you to invent a way for a machine to paint a brand new picture from nothing. What would you do? You'd probably teach the machine to draw, right? To lay down strokes, build up shapes, add detail. That's a normal way of thinking. No one would probably say that let's teach it to take a finished painting and slowly smear it into meaningless static. Uh this is exactly what this reverse the corruption process is going to do. Uh this backward-sounding idea of destroying things first and then creating it is exactly how nearly every AI image, video, and increasingly even AI text is now created. This idea is called as diffusion. And it actually feels like a magic trick. The algorithm behind diffusion is take real things, destroy them gently and gradually into noise, and then you learn to undo that destruction or undo that noise one tiny step at a time. And once you learn these tiny steps of undoing the destruction, what you can do is that you can start from pure noise. And then reconstruct the whole image backwards, whole image or audio or video or text backwards from it. So, the whole idea is to create, you first destroy. The path to to creation runs through destruction. So, with this mental model series, I don't go into technical details too much. The goal is not to frighten you or intimidate you. The goal is to show that the world around you, the artificial intelligence world around you, actually runs on a fixed number of mental models, which are incredible to think about. And diffusion is one of the strongest mental models which I see around me right now. So, diffusion has multiple steps. The step number one, which is of course called as the noising process or it's called as the forward process, is simple. What you do is that you take an image and you corrupt it. Which means you take the original image, you add a little noise again and again and again and again until the image becomes pure noise or it's pure static. Okay? But remember this, at every single step, you know exactly what noise you added. It's like that Hansel and Gretel story when they walk into a forest, they leave breadcrumbs to trace their path back, right? Here, you know that you're going towards chaos, but at every single step, you are actually noting down exactly what noise you have added to go from a clean image to a chaotic image. Keep that in mind. You have to know exactly what noise you have added. That's an important thing. The step two is the denoising process, and this is the non-intuitive part. What you do in this step is that you show a network, a slightly noisy image, and you ask the network to predict what noise was just added. So, the idea with in the step two is basic. So, step one is the noising process. Step two is basically you Let's say this image is noise. You take a random noisy image, and you ask the machine that, "Hey, how much How much noise has been added? Can you predict the noise which has been added?" If the machine predicts the noise which has been added, you can essentially take a noisy image, remove the noise, and get the clean image. So, what we train the machine to do is that we learn to undo one step. Which means that you take a noisy image, it might be at any step, and then you teach a machine, or a neural network, or any kind of an AI system, to You teach the machine how much noise has been added to get this noisy image. And then you predict the noise. So, this is a noise prediction machine. You create a noise prediction machine. What this noise prediction machine does is that given a noisy image, it actually predicts how much noise has been added to that image. This is the most important step, and this is the challenge of designing this machine. So, this machine is not as simple as what I've shown over here. It might be something like a U-Net. Uh We have not learned about U-Net, but uh if anyone is interested, just take a look at this paper. It has almost as many citations as the attention is all you need. It's a beautiful paper, but anyways, the model which is used to denoise can be something complex. We are not going into that right now. We are just looking at the mental picture. Okay? You take a noisy image at any time step, and you teach an AI to denoise it. So, you train a model to denoise one step, and then comes step number four, which is or the next step, which is you start with a screen of pure random static or noise that was never a picture of anything, and ask the model the only question it knows, "What noise should I remove?" at that step. You take the model's answer, subtract it, and you have something very slightly less random, and then you ask this question again and again and again and again. So, you denoise at every step, because now the model has learned essentially how to denoise, right? You know how to go from step T to step T minus one. The model knows how to remove a little bit of noise at each step. Essentially, in this step we are we're training the model to undo one time step to make it little less noisy. So, the idea is the model takes the noisy image at any time point and predicts what less noise was there at T minus one. So, it goes from a noisy image at T to cleaner image at step T minus one. Here, what we are doing is that we start from pure noise, and then the model is trained to go one step backwards, right? We keep applying this model again and again and again and again, and we go right back to the start. So, take a look at this. This is pure noise, and we have learned the model to undo the noise one step at a time. So, this is one step, it becomes uh less noisy. The second step, it becomes even less noisy. Third step, even less noisy. Fourth step, fifth step, sixth step, seventh step, eighth step, ninth step, 10th step. See? Now, a picture appears in front of you. A picture which never existed has walked out from the noise. This is incredible, right? You have created a destruction, but you have noted down how much you have destroyed at each time point. You have noted down how much you have destroyed, so you can train a model to undo the destruction at each time point. Now that a model has been trained to undo the destruction at each time point, you can essentially take any blurred image or noisy image and completely generate a clean image out of it. Over multiple steps. This is extremely bold. Uh There is a paper which is an incredibly bold and beautiful paper which kind of shows that diffusion models beat the previous trick for image generation was something which is called uh GANs or uh generative adversarial network. So, this paper essentially showed for the first time that diffusion models outperform GANs on the task of image generation. I'll show this paper when it loads. But, it's an extremely bold and a beautiful paper of which clearly illustrates this technique. Uh So, you generate a picture that never existed right from the noise, right? You start from pure noise, and then you start denoising slowly. As you start denoising, first there is partial denoising, then a structure starts to emerge. And as the structure emerges, then we refine the details, and finally we get an image directly from the pure pure noise. If you ask me, this kind of mental models which AI researchers come up is not done in any other field. That's why I love this field so much. Who would have thought that to create an image you destroy something first, and then you build it back from scratch? So, the idea is taking a thousand tiny easy steps. Generating a beautiful image from nothing is not easy, right? But, removing a little noise is easy. I cannot generate a beautiful picture directly from this image, but I can remove a little noise because I have trained my model to do that for me. So, it's again taking one smaller step at a time. You start from noise, you take one small step at a time, and you build a clean image out of it. This paper has now loaded. These are some of the images which uh were made using diffusion models. Incredible, right? What diffusion models can do. Uh starting from pure noise, we can get images such as these. So, just in the spirit of the series, which is a bit philosophical, I know, but it's to inspire all of you. There is There is an old line attributed to Michelangelo. The statue was already inside the marble. Which means the statue was already inside this marble. He just removed everything that wasn't. So, the statue is already inside. We just remove all the other things and the statue pops out. That's the signal. This is the noisy block. We chip away all the noise and then we get a pure signal out of it. That's how I visualize the denoising process. You You have the image hiding here, but you slowly have to denoise using your model to recover the clean image. So, when this came out, people thought that this is only for images, right? Uh but it turned out to have a huge number of applications. First of all, a large number of products came out of it. New technique, stable diffusion is now a technique which powers a lot of image generation around us. Midjourney is a product. Uh it's a very famous startup. Every one of these tools, what they do is they take in a text and they generate noise out of it, right? All of these tools which take text and generate, sorry, they generate image out of it. Each of these Each of these tools like Midjourney, DALL-E, Stable Diffusion, Imagen, all of these tools take a text prompt and then generate an image out of it. Under the hood, they are actually doing the same thing. They are starting from noise and they're denoising, but they are just anchoring it on the prompt now. It's called a text conditioning. But the underlying process remains exactly the same. This is how many of you might get inspired to learn more about the diffusion process, the stable diffusion process, etc. Now, yeah, as I mentioned when when this technique came out, people thought that it's only for images, right? If it's only for images, then I would not have thought this to be a model uh mental model of artificial intelligence, but now it's showing up everywhere. For example, there is a company called Mercury which has this LLM diffusion model. Yeah. It's uh it's an LLM diffusion model. Essentially language models like ChatGPT, if you interact with it, right? Give me or you can say give me a travel plan. It generates one text at a time, but diffusion is something different. In diffusion, all the text appears at once. So, if I ask something here see, text has already appeared. It does not appear one text at a time. It's incredibly fast, see? If you see carefully what's going on here uh the text is appearing all at once here on the right-hand side. So if I were to show you you all take a look at this. Yeah, this is how the diffusion process actually happens for language model generation, right? Text can be generated anywhere. It's again the same thing. You start from noise and then you uh slowly uncover the noise. You denoise. It's the exact same process. But the idea is that in normal LLM inference, which is on the left-hand side, which what ChatGPT, Gemini, Claude do, they do one token at a time. But diffusion is much faster because it predicts the tokens all at once on the screen. I think there is a GIF associated with this. Yeah, this one. Yeah. If you take a look at the left-hand side, that is a normal language model and on the right-hand side, it's diffusion. See, diffusion is so much faster because it comes from everywhere whereas on the right whereas on the left-hand side, this is much slower. Take a look again. The left-hand side is much slower. The right-hand side is much faster. The right-hand side is diffusion language model. And the left-hand side is an autoregressive language model where one token is predicted at a time. The autoregressive LLM takes 75 iteration, but the diffusion takes only 14. Incredible. So, this kind of a technique even um shows up for language models. So, as I mentioned, there are two ways to construct a language model. There is an autoregressive model which we are used to seeing around us like ChatGPT, Claude, Gemini, etc. But, new companies are emerging who are working on these diffusion models. I think now Gemini also has this diffusion LLM. I think so or they might come up with it. There's this diffusion Gamaya. It is 4x faster text generation and it works on the principle of again diffusion where the whole idea is that you start with uh you start with noise. You start with noise and then you predict tokens at a time. So, you're not constricted to predicting one token which is based on the previous tokens. The tokens can show up all at once. It's called also masking and demasking. But, the idea is that diffusion language models is a promising new field. Uh it's a great way to enter into research because not many people know about this right now. All of us are used to autoregressive models, but as Andrej Karpathy has mentioned in this post, most of the LLMs are autoregressive. Diffusion is different. It doesn't go left to right, but all at once. You start with noise and gradually denoise into a spoken stream. Most of the image video generation actually use diffusion, but this is the first time language has done it. That's what he says. So, what he says is that the model like diffusion model has the potential to be different and possibly showcase new unique psychology or new strengths and weaknesses. So, of course, it's a new technique, but it fits in our mental model. Now, once you have this diffusion trick for images and languages, it does not stop there. It spreads everywhere. Because almost everything can be corrupted and uncorrupted, right? It shows up for video, so you can noise and denoise across space and time. And again, a whole clip emerges from space-time static. Like here, you you noise and denoise across space-time and then a video emerges out of it. This is how uh um video makers like Sora um not this. Yeah, this is how video makers like Sora actually work. They are work They work on the principle of diffusion. Uh it's incredible, actually, how far along this technique has come. So, this video is made out of diffusion. Start with noise and then you slowly go on denoising it and you you aggregate images which are denoised and outcomes a video. Incredible, right? All of these videos are also So, the prompt is historical footage of California during Gold Rush. And you get something like this. It's incredible what uh diffusion can do for you. But people think diffusion is only for images. It's not like that. It's only for tech It's also for text. It's for videos. It's for audios, also. You can actually uh generate speech out of a diffusion model. You start with noise. You denoise the signal. And then you get clear and melodic speech out of it. You can even do this for medicines, actually. Molecules, you scatter atoms at random and denoise them into a valid structured protein or drug candidate. So, diffusion is now a serious tool in designing new medicines. You denoise random atoms into a real molecule. This is also an incredible field to get into if anyone of you is interested. It's drug design. And diffusion has started to show up in drug design for making new medicines, also. So, this is one trick. The reason I call this as a mental model is that it shows up in images, it shows up in language, it shows up in video, it shows up in audio, and it has now started to show up in medicine, also. That's incredible coverage across all the fields. That's why it's one of the most important uh most important mental model for all of us to learn. Right? Now, let us quickly summarize that this is actually very strange and non-intuitive. The most natural assumption in the world is that creating and destroying are opposites. That to build something, you do the reverse of tearing it down. Diffusion is very non-intuitive because it realizes is that the only way the machine learns to create is by first learning how to destroy one step at a time. And then running that destruction backwards. So, the idea is that you cannot teach a machine to create from nothing, but you can teach it to undo a destruction which you fully understand. That's what that model learns. It learns to undo one step at a time. So, you destroy on purpose. Carefully, you destroy very slowly. That's important. You don't destroy very fast. You destroy very, very slowly. Uh I hope you have seen that video of diffusion, right? So, chemical diffusion video. So, what it says um um there is a theorem in diffusion that if you actually rotate something a bit slowly and then you bring it back, it actually returns back to its original state. Uh reversible fluid mechanics diffusion Yeah, this. Look at this video. You see uh This is a chemical, right? And it will soon be rotated. So, this is adding noise and you'll see that So, now the person will rotate it. Yeah, see this is adding noise. Okay. And the noise needs to be added slowly, otherwise this will not work. This is the same thing as the noising process in an image. So, this is adding noise. And now this is pure noise, right? See? Now, what this person will do is that he'll slowly start rotating it in the reverse direction. So, see now it's pure noise. We don't know what it started. Now, he starts rotating in the reverse direction. This is the denoising process. Uh now the fluid is actually going back, similar to how we could construct the clean image from pure noise. This will only work if the noising process is slow, otherwise it will not work. So, now you can see this is quite beautiful, right? Because as you already might see we are now actually getting back the same configuration which we started with. This is truly beautiful, isn't it? We have now got back the exact same configuration which we started with. That's the idea of noising and denoising. And that's where the name diffusion actually comes from. Here the chemicals are diffused. Which is why it's called denoising cuz the chemicals are diffused. And then when you bring it back, it's actually reverse diffusion. That's where the name actually comes from. Anyways, so the whole takeaway is that when you face Now, the idea of this mental model series is that so that you can apply this to the real world when you are stuck or when you're facing problems in AI, right? So when you face a generation problem that feels impossibly hard to solve like make an image, write a passage, fold a protein, resist the urge to build it directly. Ask instead that can I define a gentle reversible way to destroy the finished thing? Because if you can corrupt it step by step, you can train a model to walk those steps backward. And when the model walks those step backward, that same thing which you wanted to generate might just fall out from that denoising process. So when creating something is too hard, define a gradual reversible corruption of it instead. Then train a model to undo a single step and run that undo from pure noise. Images, language, video, molecules, creation again and again turns out to be destruction played backwards. This model which is mentioned over here, that's the hard part to do. But once you get into this field, uh all of this will be simple to understand since you would be interested and passionate about The goal of this series is for you to equip you with tools in the real world to solve problems, but also for those of you who are new to this field to understand what all is there in artificial intelligence uh and to be passionate about something. So if diffusion or this idea of reversing the corruption, destruction, generation fascinates you, diffusion is uh your area to get into and do incredible research, build amazing products. Maybe you can be the one who builds the first incredible diffusion language model which every consumer uses. Currently, there is no language diffusion language model which is so popular like chat GPT for diffusion, right? You might be the first one to do that. Thanks everyone and I look forward to seeing you in the next lecture.