Food for Diffusion

Key Takeaways

okay great uh all right to introduce our next speaker Patrick Essa is a principal research scientist at Runway focused on machine learning for Creative applications so he was also a developer of stable diffusion and vqan and part of that same conference group at University of Heidelberg uh Sir Patrick so great to have you here and I'm really looking forward to seeing some of the applications that you're working on and also yeah just all your other thoughts on uh what you've been what you've been doing with these incredible models yeah great thank you um can everybody hear me yep it's perfect and I think I can go through the slides in here right yep nice um yeah thanks for helping me really glad to be here um my talk is called food for diffusion um basically I just want to talk well I guess like the the one point is uh to talk about to give a bit of food for thought also what one can can work with uh on the fusion models um but the other part of the title is basically that I will talk a bit about the inputs to to the model and how that affects the output so essentially what I want to start with is that composition is something that matters a lot for Aesthetics um so here this is a nice example I guess from Um moonworth's Kingdom by um yeah which which always has like really nice compositions and this is in a way where it really becomes kind of clear that there's something going on something specific about the composition of the frames um while still being beautiful so I think this will be generally a quite a hard goal to reach that kind of uh quality in in the Aesthetics how how scenes are composited but um yeah I think we are it's definitely something we should strive for and while those are uh not generated they've been made by a human master um at least like he has something like the first slide um that is generated yeah so basically for composition I guess um I'm not really an expert in the topic but I guess like you could on a high level say that the relative position of different objects in a scene that make up the the image um it matters right so on a very very basic level you have to think about like um positions in images um yeah I think one should also say like even though even though we train on huge databases like Lion um which come from everywhere and contain all kinds of different content um it should one should still uh realized that it's like we don't take even those pictures are not random right like um even people like me who are not aware of uh good compositions or anything even if I take a picture I usually take a picture of some object so we can generally expect is that um images will be object-centric for example already puts like a weak bias into the data set and the questions then really kind of like okay how can the model actually um figure out where to put objects be that for matching the object-centric prior or be that for um coming up with good compositions so kind of like as a thought experiment if you want um is the idea like imagine you're a convolution like usually or for example uh for for energy diffusion models we the backbone of the diffusion model is often a unit which mainly consists of convolutions but completely true there's also we also use attention most of the time um nevertheless um yeah if you if you look at the basic operation of a convolution it's like you use a fixed set of weights and slide that over the window right so you apply the same um functions though to say uh through every point in the image you slide it across so now with the diffusion model if you sample eventually like your first step will start from a Pure Noise image as shown on the top right here right so if if you're a convolution kind of and you get your input which really consists only of noise and you see two different noise patterns it's basically impossible for you to tell where you are in the image like two patches in a year they just look absolutely the same um so basic questions exactly like how well we know that it works right like we do get object-centric samples out of the model so um the question is actually like where does that come from and it's really nice work about this by Islam at all uh on in a bit of a con different context not for challenge with models but for discriminative models um which analyzes first of all like how much position information [Music] um convolutional networks encode and also analyze the the mechanisms of uh of how that works and yeah I guess like one of the on a high level the the high level finding or one of the conclusions is that um one way that cnns do get Precision or do encode processional information is through a combination of having a large receptive field and the fact that we often use zero padding for in combination with those for convolutions right so here's a visualization of of that principle basically um so if compared to the to the previous slide we now have a model that has a larger receptor field and then if you're at some point in your in your image you want to apply your convolutional crown there but the problem becomes like you will it will it might be bigger than the image right and so the simplest solution is to just add zeros in that position and so if you do that as visualized here with the um the black borders basically you see that now those two like what a what a conclusion sees in those two different locations actually it does develop differ like now it sees oh on the left there's a black border so I'm probably somewhere on the left side of the image and similarly for the right patch you you can see that there is a black bar on the bottom so I know I'm somewhere close to the bottom and of course like since all of this is um learned um it might it might come up with very sophisticated ways how that encodes the position um yeah so with this kind of like uh idea of how how actually like uh a CNN and thereby also probably a diffusion model actually orients itself or like things about uh compositions in samples and images uh I want to digress a little bit uh and go towards textures right so sections I would say are a bit different from our typical images in the sense that here it's you wouldn't say there's no like real object prior in the in the images in fact I think one of the things that is like a helpful description is kind of like that they look pretty much the same everywhere right like you can look at any patch and they kind of look very similar at least like often a statistical sense and yeah so in that case it might actually like Clash a bit with uh with the idea of having um position encoded because you want to look at the same and also yeah one one also makes use of that uh idea that textures kind of look the same everywhere if you wanna generate large textures right often this will be like a a large area that you want to paint without another 3D um asset or whatever you you need to create like potentially arbitrarily large textures so there are a lot of words on or like the most basic idea is that since it looks everywhere the same you can just take like a small patch and then you just tile it arrange it next to each other to get um a larger texture and so one could try to do that essentially with um with samples like a nice thing that does work very well but like um text to diffusion models you might say like uh as a prompt you just ask for a texture patch of the word right here shown on the right is the the result that you get out and the curve makes sense it might make for a nice uh nice texture if you can increase the increase the size but if you do that now you just dial it as shown on the left side you can clearly see that they're all like yeah you can clearly see that it is titled right so that's not very desirable it doesn't really look good there is still basically um yeah there I would say there are two issues one is um that like the spatially it's not homogeneous enough so there is still quite a bit of structure uh in the image which might come from something like a zero padding General model nose and the other one is the blending around the around the edges right and yeah like I said this has been I mean it's has been extensively analyzed and there are many algorithms for that um for doing proper texture tiling and one approaches uh um was described in his work by Heights uh on high performance by example noise using a histogram preserving blending operator which I think is really popular for texture tiling and it performs often very well um I think it's just that it makes like again very rather strong assumptions that what you that the image actually can be described well by its uh statistic and yeah if you so if we apply something like this now shown on the left um the result is definitely improved it looks uh much much more natural the boundaries but generally speaking one can still see like this structure of a two by three grit right just because there's so much um this little structure left so yeah if one takes a look at this like padding behavior um and if one thinks about that this might be one of the core things how the model thinks about position and we want to get a more homogeneous um texture out of it uh very simple idea is to just replace you have an existing model um you just replace the padding mechanism in all of your conclusional layers to do like a circular padding and what do you get out of that is then basically your sample becomes something like here shown on the bottom right and without any post-processing so the image on the left doesn't use any blending you just tile them together and you see that they actually fit which is yeah which is a really nice property because now you can directly generate those um textures and have them directly tileable right without any post processing and also without any any real changes to the model yeah so I think I really like this example because it's I would say well that's simple like both in terms of you you can take an existing model and just apply the change and all of a sudden you get um you'll get samples with completely different properties just by thinking a bit about how the model internally works and yeah maybe the other thing that really surprised me for this example is that that it actually works right like often I would say you have some intuition about how the model Works regarding like the padding um but now in this example here we go in and directly manipulate the way how that is handled right and that changes like everything internally all the internal representations of the model um and just the fact that um that this works is quite surprising because I think you can there are so many examples where you can um really mess up the uh model performance really quickly just by um by changing slight details like I I had issues where like I would train a model with a very specific um resampling algorithms for resizing training examples and at test time if you would use any other it would like completely fail so very often there is this fact that models can can overfit to specifics very strongly but in this case it uh it really performs nicely and produces the desired result okay um that yeah the training data kind of brings me to my second point that I wanted to talk about um which now goes again back to what we started with and which is kind of the opposite of textures um that usually there is compositional structure in images so what we see here is basically the first page that you find if you go to the website of the rely on Aesthetics data set for relatively High Aesthetics threshold um and you see it in their in their original form right like all the images have different aspect ratios they also have different solutions they're just recessed for visualization but I guess the the important thing is like um yeah they usually have very different aspect ratios and the other thing to note is that actually more often than not they are not square right um I guess like the two most popular formats are basically like either like a um landscape format something like this which is like a white screen aspect ratio or also like a portrait mode which is especially popular for portraits right but also for other images enemy I would say um that a lot of those images come from artists who actually created them and um I mean they usually put thought into like how they arrange objects in an image right that that is part of what the image should Express so I think that's actually quite important data but if you then look at what we actually do quite often when we when we train those models it's like I would say basically a simple technical artifact that we wanna stack all our training examples into a single tensor to have a good memory layout that we can feed it through the network fast right we're gonna already takes long enough so you have to make sure the performance is good so to make that possible this stacking of images what we usually do we just extract Square images resize them to a common size and we get a batch that for this example basically it looks like that um I mean the images still look good from a first perspective but um yeah like I said I think like in this example it's not even that bad like but we can often see like here already indicated that hats get a little bit cut off here and uh there are no more hands visible and yeah here the composition completely changes where we where we did see something of the background behind the um ours here so yeah I think it really changes the the whole um meaning or definitely destroys the composition of the of the intended original image right and okay as a consequence of what happens if we do that right um so we do this fixed size training and then what we see like as a first first observation just um regarding the the generalization is that okay so we train on topically here this this model is also trained on 512 by 5.12. uh and this resolution um we get reasonably good images like I did not excessively Cherry Picked those here this one's definitely not perfect but you can see like it overall gets the chest and if you produce like a quite a few more samples uh you usually find pretty good good ones as well but then you um try to do something else right during sampling time it's in general it's one of the nice things of a convolutional model that you can just like [Music] um change the change the shape during during inference and produce samples at different resolutions um yeah and I think this is relatively interesting what happens here um you can see often if you go to like higher resolutions um you very often see this repeated objects floating around somewhere in the image and once again I think like if you yeah if you think back to the fact that the mall usually does encode the boundary it was trained on like it knows like there is a 512 by 5512 that's what I'm training at and most of the time I do expect an object like a person or something in the middle of that image right and that's basically what it's doing here right like this is double the size so I guess like the the most extreme case would be like if you see like kind of four um four reasonable images um even though their composition doesn't make sense right like their relationship is just floating around because um yeah uh 10 24 by 2024 image is not a random combination of four five hundred twelve by 5 12 inches so there we um yeah we really quickly run into trouble in higher resolutions and something else that we um which is maybe even a bit more surprising I find is um that the same happens if we go to lower resolutions like if for example this model at 256 by 256 here in the top left then it actually really um doesn't make any sense anymore so um it was also interesting like hard to tell what exactly happens it's like all of a sudden now borders are overlapping which the model never saw so yeah I think that's that's one of the points where I mean like um yeah don't don't expect to get magical generalization to all the cases right we see it doesn't um yeah and then yeah one thing we we explored quite a bit now is basically that we train on Dynamic sizes so that means both that we take [Music] um we keep the original aspect ratios which is more about preserving the composition of the scenes but we also keep the all the original resolutions up through a threshold so we train somewhere between 256 and 2048 um and yeah what we see is definitely that things improve quite a bit so for 512 by 5.12 we still get um nice results in this case also maybe an interesting composition might be related to the to the different aspect ratios as well um for 2024 by 2024 it also produces good results um for 256 by 256. I guess you can see that it's like a bit more well I mean it is a lower resolution image um also all looks a bit more washy but that is also um actually like a yeah fact off that this is also like a two-stage model and uh the first stage also um introduces reconstruction Euro set at lower resolutions um yeah generally speaking I don't think you want to go like to something 256 by 256 but uh it is actually important that you can at least do like one size uh smaller in case you want to do pretty extreme aspect ratios right you want to have might have like a very long side combined with a very small side um yeah and that brings me also to the to the other point which I think is maybe even more interesting is like this um this fact about yeah composing images at different aspect ratios here uh as an example batch for um yeah landscape um mode on the top this is the one that was trained uh on Dynamic sizes and dynamic aspect ratios you can see that it produces a quite nice results often and it also seems to have some I mean I don't want to judge that I'm not like I said I'm not an expert but I think it's uh can also be used to to explore nice compositions of like often like most of the time you're interested in probably something like widescreen or different formats and I think that was something this you can it provides much better tool to not only explore like the specific content um that you want to put somewhere but also the the composition right you can you can get inspired by that as well um yeah whereas the the same model basically with the static size training at 5.512 again we already saw that um doesn't make much uh sense in this case we again see all those repetitions of um objects in the center yeah that one's just uh basically just the same example but for portrait mode um I think that's something that also that I hear a lot um people are also a bit annoyed with um that often have you if you do a port remote and you're interested in in sampling like a full body person um all kinds of things uh funny things happen like this one is very common that you that it just cuts off the head right which is related to what I was talking about in the in the training batch example they're just often cut off and we don't see it so kind of not too useful anymore and the other one is this duplication of bodies which sometimes really provide funny looking outputs but uh in general it's also not really what you want right whereas we can see again I think especially for this for support remote uh with characters since we train on their original size I think there are quite a few of those in the training data set um we get much more coherent um generations of characters yes um time okay I think I'm um yeah maybe I will stop here and take questions if there are any awesome thank you so much Patrick that was a really you know deep dive into you know sort of ways to make uh diffusion models work which is pretty nice to see because you know a lot of the time you just see the output right like uh check out my fancy image on Twitter and um It's actually kind of revealing to see that there's a fair amount of like careful um you know intuition in engineering as needed to get this out um we have a few questions from the audience so one of the ones here um from Evan Jones is uh based on the way convolutions learn to orient themselves in the image do you suspect composition is better at the edges but weaker towards the center where the guideposts are fewer yeah that's really good question actually um I think there are two things that won't have to differentiate with the composition one is really this fact that because you crop you never see the original composition right so it's in that case if you do the center cropping it's much much less common that you actually see like a in a widescreen image that would be a white screen image that you see like a long stretch of nothing and then maybe at like one third or something of the of the scene you have like a portrait which like might be like a common composition so since you never see that I I think like that is one problem why the model would never really if you ask for a product where it would never really make a long stretch of nothing before it puts a person um yeah so from that I don't think the conclusions are too much involved but definitely from the from the perspective of yeah orienting itself um I think actually that's one can go quite a bit deeper into that question and really ask a question like why are we actually not providing it a better way of orienting yourself right you know you are there is the idea of concatenating coordinates or uh some some representations of the coordinates um to the to the data input and that kind of often like depending on the task I think it can really improve results um one reason why often white is not done by default I think is because you again even you you lose generalization to other resolutions even more right because now the model really starts to use the the precise spacing between points the the resolution that you drain it if you change that then during during sampling time it's like really confused so I think that's why you often don't do it but I think with if we go to Dynamic size training um this will be very interesting to explore to see because then yeah if the model sees all the different resolution words and different sizes then we might get the generalization nevertheless thanks yeah I it feels a little bit to me like um in the space of diffusion models we're kind of trying to ReDiscover what are like the techniques from computer vision that we've used for decades and now what are the equivalent ones right like you know we have all these hacks for data augmentation which work really well and I think it's pretty much too soon right to know exactly you know how you do this in the diffusion process yeah yeah I agree I think that's super interesting topics and also on a very high level I think that's uh I'm always excited about working with a new technology while reading maybe some um older paper and trying to see how those ideas match or can help each other and about football maybe we can take one tongue-in-cheek question uh so you know maybe this is the question of dynamic training or resizing and stuff you know will this fix hands and maybe a question like part of that for me has always been like why what is it about you know stable diffusion that gives you these like you know nightmarish like interlocking hands what's the intuition there like what is it about hands that are like such a difficult thing to do I mean I used to be a painter and I used to struggle with like you know feet and hands I hated them so my own images were always like you know hands hidden but like is it the same for diffusion models yeah um I mean uh actually I think we we already made a bit of uh of progress there because I mean it used to be that that it was really freaky looking at uh faces that were generated and often they still are but uh but uh but we're getting pretty good at them um I mean yeah also there I think we also talk a lot about that we definitely think it's there's this human part where it seems that we're really overly sensitive to human anatomy so we just noticed that so much more than anything else um but then yeah I think at least what we saw also a lot when we we trained a lot of this these first stages which try to compress images through a smaller representation and then reconstruct it from there and I would actually say that that already there we very often observe that like that faces look very strange and uh especially hands and I really think it is also related to the to the size of those features um I mean it works better if you if you ask it for a full front or close-up portrait right because there's like essentially if you look at it relatively there are now more codes that that work to represent the face then if you have like a face that's very far away it becomes much smaller and maybe there's like only a high level if you look at the receptive field of that code it might be only a single representation so yeah so there's more you have to compress it more and that's where you lose information and things get worse um so I think that's a big part and the hands are definitely more even more difficult due to their size right but we're good yeah I mean I think already in stable diffusion 2 you can see great improvements so this is like it's pretty exciting you know that the great mystery of hands will be sold the biggest problem of all times you know some people are you know doing protein modeling and you know climate change and we're worried about hands yeah and consists of proteins so yeah um with that I'd like to thank you again Patrick for joining us it's been a real pleasure and um yeah we'll see you at the next event or yeah around that's really nice to meet you hi foreign