What is Google Deep Dream? (Basic Theory) | Deep Dream Series #1

Key Takeaways

i'm starting a new video series with this video about the algorithm called uh deep dream and in this video we're going to cover the basic theory behind the algorithm and in the next one we'll go through the code i just open sourced a couple of days ago as you can see it here on the screen and it will give you some really psychedelic looking uh imagery uh but that's not the the point obviously of of this uh of this series uh it's gonna teach you you're gonna learn a lot of more about deep learning itself about like uh computational graphs in pi torch uh and also how to create some really nice artwork using uh the uh this this code but first let's jump to some basic theory about the deep dream algorithm itself so what's the idea behind this algorithm so given an input image the the thing that happens is that the neural network uh see pre-trained on some like classification test or something like on imagenet or mit places 365 basically sees some something in that image certain features and whatever the network sees we just uh tell it to just amplify those exact features and by doing that you get the image on the right that you can see on the screen i found the story how it all started really interesting so mort vince of the guy that actually created the deep dream algorithm the initial uh idea uh actually woke up like around two am and had a nightmare he just sat down on his computer coded this up in maybe 15 minutes and instantaneously he got some really interesting results uh what's even more uncommon maybe is that they did not publish a paper on this they just wrote the blog and subsequently they open sourced uh a simple implementation of the deep dream algorithm and you can see on the screen they started getting some really weird creatures and results uh titled like pig snail or the dogfish it just sparked a lot of imagination from the very beginning and those creatures came from this uh cloud image you can see in the top left and also the the interesting thing is that depending on what you give to the network what you input as the as the as the input for example if you if you give it some scene it will start populating it with some towers or buildings and if you give it some kind of maybe a leaf or something it will start populating it with insects and animals so it also depends on the on the data that the network itself was trained on not only on the input image but i'll get into a bit more details a bit later but deep dream was not only about like creating this psychedelic crazy looking images it was actually a part of the effort to better understand neural networks uh a field called interpretability of neural networks and you can see in the in the bottom left those examples so what they actually told the network was hey uh tell me what do you see what do you think dumbbell is and going in reverse and reconstructing the image they got these four images in the bottom left and you can see that the network actually learned to always expect to see some muscular hands of a weightlifter next to the dumbbell or a bodybuilder whatever and did not manage to actually extract the concept itself of a dumbbell how we humans actually think about the dumbbell itself as a independent object not dependent on the context where we usually see it and let's get a bit deeper into how the thing actually works so what you do you have a pre-trained neural network as i already said uh pre-trained on some classification data set like imagenet or mit places 365 and you pass in the input image and you feed it through to a certain layer and when you come to a certain layer whatever activations you have there you want to amplify them and you do that by doing the gradient ascent and not descent for example you took a simple l2 or msc loss on those activations and you wanted to do a classical procedure that's a minimization uh what you would end up with would be the that the input image would either become black or more probably just some random noise image on the other hand if you tell it to maximize instead of minimize those activations you get the deep dream results that we're familiar with as the image on the right here now the interesting thing is that depending on which exact layer in the network you're trying to optimize maximize in this case you get different interesting results so if you take some shallower layers from like some of the first layers in the neural network and you try and do this uh procedure i just described you'll get this low-level app patterns really interesting patterns and colors edges and corners the stuff that we know the usual low-level uh vision um stuff but on the other hand if you took some deeper layer in the network you would start getting some more increasingly complex features such as like maybe animal eyes or snout or something i'm assuming of course here that the network was trained on imagenet which contained a lot of like images of dogs and cats and animals in general but if we had some other data set then those high level features would be some other things and not eyes or snouts now i already mentioned imagenet and mit places 365 but those are not the only data sets out there obviously you can use some other ones but just that i use these two and these two are really common in the deep dream community so you can see on the left here the image was produced by a network that was trained on a lot of imagery that contained animals on the right you can see uh the network that was trained on the mit places which contains some scenery human-built objects uh pubs cafes and stuff like that and that's the reason it has these uh human-like built uh things uh ingrained into the image itself going a bit more deeper into the uh theory uh so i already mentioned gradient ascent so basically the only difference is you just change the sign uh when you do the update for a single pixel you don't do the minus where you use the learning rate and the the gradients you just switch it to plus but there are also some more advanced stuff like uh image pyramid so the uh the idea there is so first what what's image pyramid so it's basically you have your your base image and you just down sample it by some ratio let's say 1.5 and then you do it again and again again and you just stack those images and that's the image pyramid so now why do we do that now the thing is there is something called receptive field of network and when you feed all of those images from the image pyramid with different sizes to the network you will see different features in all of them some will be more fine-grained some will be like coarser features and combining all of those across the layers you'll get a really nice looking deep dream image so why they work is that uh when you have a smaller image in the input a single neuron somewhere in some deeper layer will be able to cover most of the pixels in that input image and they will be able to manipulate and tweak those pixels uh whereas when you have bigger image that pixel will only can will only be able to change and modify just a small area of the input image so the algorithm proceeds like this you just start with the smallest resolution you do the deep dream then you just up sample the image then again you do the deep dream and then again again until you get to the base image of the pyramid the biggest resolution and you just stop there are two more important things that you need to implement in order to have some nice images uh one is uh the gradient smoothing and normalization so basically if you only took gradients that you get from this optimization and just blindly apply them the image will quickly explode out of the the bounds that we are keeping it in so what you have to do is to just treat the gradients as a as a as a simple image and take a gaussian kernel and just uh filter out the through the image your convolution with the with the with the kernel and that will just smooth now the gradients and additionally you just want to take the maybe standard deviation of the of the whole thing and just scale all of those gradients with that standard deviation and that will just keep the things uh scoped aside from that it's quite uh usual to see to to just uh after applying the gradient of sand step you just want to clip the image so that you keep it in the designated boundary that you want to keep it in and those are usually image net normalized boundaries that should be enough for you to understand the next coding video where we'll go step through the code and just try and run this ourselves and get some really awesome looking images yeah and by the way don't forget to check out the description where i've linked the github code uh feel free to just play around with it before i create the actual video of explaining how to use it i guess most of you are coders or experienced machine learning engineers already so go ahead and play and as always if you found this video useful and only if you found it useful uh please consider subscribing and sharing the video that would mean a lot to me and yeah see you in the next video