OpenAI's DALL-E explained. How GPT-3 creates images from descriptions.

Key Takeaways

[Music] hello did you hear about dali no not wall-e dal-e no not a human painter the transformer-based ai painter twitter exploded again because open ai's new image generation model called dal-e was announced so here we are trying to figure out how cool it is and how this could happen image generation was already spectacular with dedicated models used for rendering people there were also models for rendering animals from textual input but the difficulty was to generate these kinds of images from text when the text combines concepts and the image has to produce plausible combinations of these concepts so open ai looked down upon the world decided to stop resting after publishing the great 3 and quickly solve the problem of image generation from text 2. yes just like that dal e was born and open ai rested the next day oh no they should not rest because they still have a paper to write about doll e and of course we do not have access to the model so we only have openai's blog posts to look at but the situation is not new they did the same with gpt3 perhaps again a multi-dollar company will buy exclusive rights to the model and we will never see it again okay miss coffee bean we were cynical enough so here it is dull e creates images that combine unrelated concepts in plausible ways check it can also generate storefront text i have never seen anything like this did you let us know in the comments if you did dull e can do also style transfer by a text prompt so it can modify the style of an image following a textual description of the desired output wow how crazy is this ok enough showing off what is this model and how can it work dull e is i cite a 12 billion parameter version of gpt 3 trained to generate images from textual descriptions oh that explains everything no it does not gpt 3 looked at text not at images what do images and text have in common miss coffee bean has a variety of videos on this trying to explain that images and text are not that different kinds of input for a transformer that usually works on sets as long as both images and text are regarded somehow as sequences or elements in a set everything works again there have been some kinds of image sequentialization approaches in the literature but it is safe to assume that open ai stuck to their approach from image gpt where they sequentialize the image row by row into a 1d vector so now that we know how images can be sequences we are ready to know how dal e is basically gpt3 also looking at images dal e is also a language model that receives both text tokens and sequentialized images as input up to 1280 tokens why is the sequence length limited to this number because attention scales quadratically and increasing the input size can really blow up computational time then like in language modeling the goal is to predict one token after the next text tokens are words or sub words and image tokens are image pixel values this also means that dal e does not only generate images from text but can also continue images if the beginning of the image is already in the prompt following the text so that was it for the technical details okay but why does it work why did the image generation problems need a specific gun or vae architecture to solve the problem only partially and now a transformer based model can do it all well because of the formidable amount of training data how much data was used exactly we don't know but it must have been the whole internet like open ai also did with gpt 3 and things that were not possible with academic size data sets have now become feasible with a model that has seen it all this does not make the dull e results less impressive but the creativity that we see with doll e has something to do with the variance of the data it has seen take this example where they visualize perspective and three dimensionality yes perspective and 3d looks very good in general but impressive form is coffee bean is how the model knows to make a capybara from voxers it becomes clear that something like this is possible only because of all the minecraft and other pixel art images that the model had access to we would never expect from a style gun to make something sensible out of the word voxel if it has never seen the voxelized images before in the gaming or art area of the internet and has been limited to a benchmark data set of celebrity faces only with enormous data and variance comes great power let's look at more showing off of this power openai invested a lot of time into this blog post did you ever have trouble imagining a cube made out of porcupine well wait no more here it is okay so a cube with the texture of keyboard okay one can also draw multiple boxes and really tell the model how to render these wow this is awesome oh these plates are so cute and so were you making a youtube video but did not have exactly that emoji you were imagining wait no more dal e is here to paint it for you and this is very interesting from a multi-modal perspective that one description can have so many possible outcomes while this is amazing i can play with it the whole day remember the image completion example well one can even change the pose of this bust okay here are some failure cases interesting to see especially because we don't know if open ai was cherry-picking in this blog post we don't know because we don't have access to the model and of course the applications are limitless do you want to design a new t-shirt just tell daley how to draw it and then you have many versions of said t-shirt on a mannequin because you introduce the mannequin head with an image prompt and well you have here your designed fashion this has huge industrial applications and yes the combination of unrelated concepts wow i mean this is is this art one could ask it made a snail into a harp or you know it made a heart made of snail and this can be creativity one might think so it is exactly how they put it it is a 3d rendering engine via natural language you just say what you want daley to paint and it paints it and you can expect the model to fill in the blanks which means that you underspecified your rendering instructions and the model comes up with something and we cannot tell for sure but it might just be that this something that it fills in the blanks with is just the most probable example that it has seen in the training set which already opens the pandora box of bias in the blog they promise to investigate the societal impacts and check for biases hopefully sooner than later until then or until they release the model one can just scroll through this blog indefinitely but we really cannot deny it these are impressive results from dal-e bridging the gap between vision and language stepping into the multi-modal realm multi-modality is my dearest topic in machine learning miss coffee bean what's your favorite topic oh okay so you don't want to tell us fine but back to dal e however excited i might be about this i have to say i am also a little frustrated about all this because throwing the whole data of the internet that the problem solves the problem rather than clever tricks and gimmicks presented so far in the multi-modality literature you know by ordinary mortals unlike open ai especially as a phd student one has to take these results you know a little diluted to still feel a little relevant you know in this whole systems but you know everything is not done yet future research will tell us about the real capabilities and problems that these models bring with them also i really hope that they will keep their promise and investigate societal impacts and biases until then i will keep scrolling this blog in all miss coffee bean will get back to making another video and you know what about you let us know in the comments what you will do and hope to see you in the next video okay bye you