How to Do Style Transfer with Tensorflow (LIVE)

Key Takeaways

hello world it's and today we're g to build style transfer using just tensorflow nothing else so let's answer some questions five minutes five minute Q&A and then we're gonna get started mute myself all right who who's in this chat room right now all right oh we got Dan schiffman in here from the coding train we got Fernando we got some cool people in here right now we got some celebrities guys we got some celebrities in here salahudin AE Jack hi everybody wow Dan was actually the first one in here I'm honored Dan to see you here uh we we watch each other's live streams occasionally get get inspiration hi from Germany we got a bunch of people here so let me uh answer some questions hello from Moscow we are going to do the mass behind style transfer guys it's going to be awesome and we're going to really understand exact what's happening under the hood no caros it's gonna be awesome so check out the link in the description to follow along with my notes usually I wouldn't you know show that but I'm gonna show it right now hello from the Netherlands parvesh arvind I mean I'll say three more names and then we're good to go lunchtime is perfect timing isn't it uh G as and then one more name Bulgaria uh buya from Stockholm Sweden and of course all these other people okay so hit me up with those questions guys because I am so excited to actually build this thing because I've been looking at my notes and trying to figure out exactly how to explain it so it's going to be awesome uh we got Granada okay questions Norway okay hit me up with your best questions as they say as I say right now H hello from Israel is tensorflow a good library for a deep learning beginner yes because it has the most documentation it's got the most Headway in terms of all the other deep learning libraries there's so much content created created round tensorflow not even just me like other people that yes I think it's the best way to get started and it's it's right now right now it's the industry standard we'll see how that what happens in the future what is the meaning of style transfer style transfer means it's we frame adding a filter it's you know it's like filters right Instagram filters but we frame this as an optimization problem okay in machine learning we try to frame everything as an optimization problem where we minimize a loss function so style transfer is an optimization problem where we have two images we have a styled image which say say a Dan go painting and then an image of say whatever we want like yourself and we're trying to minimize the difference between these two im images and that's the transfer we're transferring the style of this van go painting all the textures and the winess and the colors onto this uh host painting that or this host image and we're going to figure out how to do this mathematically it's not magic we know how to do it all of it okay what is zero shot learning we're going to talk about zero shot one shot learning at the end of this course it's at the bleeding edge of everything I can't wait to get started with that but it means but it means uh machine learning with very little data and that I believe is the future of all machine learning by the way there is so much cool coming out right now out of Deep Mind uh my homie orol Bal they they had this new paper that basically beat deep Q uh which is their own algorithm that like cause Google to buy the out with this new model called hierarchical reinforcement learning so we're starting to see a convergence no pun intended between different machine learning ideologies like this idea of abstractions between layers and uh the idea of reinforcement and then combining them together and then this just they they they were able to uh beat pong in like 10 minutes it was awesome okay uh let me answer three more questions before we get started how transfer learning can be done using TF tensor flow we're going to use transfer learning right now so I'll answer your question as we code uh would this also work with WR Styles I actually have a video on that called build an AI Rider yes absolutely and I'm going to make another video on that literally next week uh can you can we make a program that extract Knowledge from story absolutely um text summarization I will have a video on that later on in this course soaj answer my question if the network is possible to use it in trading we're not going to use a convolutional net in trading we're going to use a recurrent net in trading and we're going to and I made a video on that um not this one this one won't be used where's your girlfriend no girlfriend I'm single and happy being single I have too much to do to dat I'm happy being single I am a beginner go more detailed I will go so detailed to darts don't even don't even trip SJ can you recommend ml Forum please uh the comment section of all my videos we are essentially a community guys we we essentially we are a community of 75,000 strong Engineers everyone from middle school students who are just getting started learning algebra to phds at Google pushing real research so we are a very large community and the the culture we set up is such that if you ask a question you will get it answered okay can we build on Google inception for this task yes you can we're going to use a different model called a vgg16 model okay so one more question and then we're gonna get started with this what is the best area to focus in AI uh so deep learning but specifically uh Focus I mean you know Focus can be taken lot of different ways Focus for research Focus for production Focus for a beginner uh Focus means just uh focus on uh I mean I mean there's so many things you could focus on but I would say it's such a generic question uh generative adversarial networks because those are the coolest things right now to me uh all right so that's it for the questions uh and so now we're going to get started with this so we are going to build today a a com we're going to build uh style transfer and it's not that we're going to build a convolutional net from scratch we already have that we're going to chop off the top layer and then we're going to add our own layer on top of that to do the style transfer so let's just get started with this I will explain things as we go and remember the code is in the description follow along with me it's going to be awesome Focus here we go all right okay so let's see let's get started with this okay so what I'm going to do is I'm not going to code uh the parts the parts that are uh not machine learning I'm only going to code the machine learning parts so what we're going to do today is we are going to perform style transfer so let's look at this uh image for a second let's go to this link this is just a paper of a bunch of style transfer uh images and it's just a way of describing what we're doing here so the first paper that did this I talked about this in the weekly video was oh let me show myself as well was the uh gtis paper right a neural algorithm for artistic style that was much love Anton that was the first paper that did this okay and we and since then there have been a quite a few papers that have built on top of that right so the first one just transfer the style which is what we're going to do today we're going to trans for the style today but there were subsequent papers that built on that idea they did it for video for example and what they found when doing it for video was that there was actually a lot of choppiness because think of it a video is just a collection of images and we want to what happened was there was a lot of choppiness between frames so what they did in the next paper it wasn't the same group it was a different group but for video style transfer was they added another loss function called Optical flow-based loss so they minimize not just three or two loss functions they minimize three or four loss functions so they're just chaining these loss functions together and what happened was this beautiful video that we can see here you know like it's it's very you you can just think about all the potential this just happened last year so just think about all the potential this has for artists for all sorts of creative things and Snapchat now has realtime filters so this doesn't even take training you could just do this in real time okay so we've seen a lot of ress starting from that initial paper and so what we're going to do is we're going to uh build style transfer the original way the way that the original paper was done and then we'll talk about uh all the ways of improving that so let's just go down to this image and take a look at this image right because these are these are the two images that we are using right now and we let's talk about the process of what we're doing we're going to slice off that top layer because originally okay so the network we originally used was called vgg16 right this was a an image classification convolutional Network right so at each layer it would apply a series of uh operations to our input image so we can think of our input image as a matrix of values it's a matrix of values right a pixel values and we apply a series of operations to that image because at each layer at each convolutional layer we have what are called a stack of filters okay and these filters are learned over time and so for vgg16 they trained it on thousands of images so it's already got a bunch of filters at each layer and what's happening out of each of these filters is it is multiplying those filter values by the input image okay and video is glitching a lot skipping a lot of frames okay thanks for that let me just remove myself that's going to help all right and so greetings from Hong Kong okay so hopefully that helps okay and let me know if this is still glitching all right let me close out this other frame just try to keep it minimal okay so uh we're going to uh so that's what we're going to do and so this is the vgg-16 layer okay and so the idea here is that let's let's let's also look at uh this this image right here that's helping right okay so let's take a look at this so let let me let me explain what's happening here for a second so this is a great great image right here so we want the contour lines from the content image we want all those lines from the content image and we want the textures from the style image okay those are the two different things we want and the way we're going to do that is we're going to at each layer there are a collection of filters so what are what are these filters thanks Alan what are these filters these filters are 3D uh vectors so why are they 3D well they are 2D they're a collection of matrices right but they 3D because the third uh the third dimension is a it's it's that RGB and it's a collection of them so we've got like say however many filters we want to define the more filters the more accurate uh if we were going to do classification is going to be and these are the filters that we're going to use and so what happens is at each convolutional filter we're going to perform a a matrix multiplication and then a summation operation and if there if it detects if it detects the feature then the result is going to be a very large number but if it doesn't detect the feature it's going to be zero and so we can think of these filters as feature identifiers okay they're identifying certain features and so what we're going to do for for this for this is we are going to perform um we're going to minimize the loss between the labeled image and the feature uh so what ha no actually so what vgg was initially set out to do was it was initially set out to minimize a loss between a labeled image and a feature map output so at each of these layers after all of these uh operations it's going to out output what's called a feature map or an activation map it's the same thing which is just a huge Matrix of values okay and we want to minimize the difference between what that feature map is and then what our content is okay so that's what we're going to do here and um so that those are our two images right we have a Content image that's going to be whatever we want to modify and then a style image which is going to be whatever style that we think is cool that we want to apply to this content image lastly we have this mixed image which is going to be just an empty random noise initialized Matrix that we're going to add to over time okay we're going to use we're going to compute two different style uh loss functions we're going to do the style loss and we're going to do the content loss and then we're going to combine those loss functions and then we're going to use a gradient value which we're going to talk about to update this mixed image okay so that's the high level overview of what this what's going to happen and after 100 to say a thousand iterations this image is going to be mixed okay so that's what the high level is of what this is okay so we're going to start off by importing our four dependencies map plot live is going to help us plot out what we see tensorflow is our machine learning library numpy is going to help us calculate What's called the gram Matrix that we're going to do I'm going to talk about later on on and then pill. image is just going to show the image that we want to show okay I am now using tensorflow version uh one and everybody else should as well they have up updated it and the other thing is uh uh I'm using Python 3 so everybody should update to Python 3 yes I finally updated to Python 3 okay and it's really easy to update just you know Google update Python 3 jupyter notebook it's just um you know you could do it in two or three lines command lines with h Anaconda so the first thing we're going to do is import vgg16 this is a 16 layer convolutional neural network okay this was pre-trained okay so this network already has all of those filters it already has all of those filters and we're going to use these filters as tools to help us uh perform style transfer okay so that's the first step is importing vgg16 and then we're going to download it so I've already downloaded it it's a big file half a gig and it's probably going to take you know depending on your bandwidth anywhere from 10 minutes to an hour to download this thing but once you download it you're good to go because everything else is going to be local okay so now the next step is to Define our image helper functions so we've got three of these and you know they're they're very you know standard load load the image save the image and plot the image okay and it's using numpy to do these three uh functions so these are just standard functions and I'll talk about these later on but right now just know that they are to load an image save an image and plot an image okay so we haven't done any machine learning yet uh these are just you know standard image helper functions so now let's get to the machine learning okay so what we're going to do right now is we are going to Define our uh loss functions okay so before we Define our loss functions we want to define the uh mean squared error so let me just type this out and then I'm going to talk about what is the mean squared error okay so the mean squared error is a function given two input tensors so our input tensors in this case are going to be called A and B and I'll talk about what these tensors are but inside of this it's just one line of code and the one line is going to be tensor flows reduce mean function and then it's going to we're going to take the Square value of the difference between both of these tensors that's it okay so and then I'll I'll compile that okay so what is happening here so the mean squared error is the average of two uh is the is the it's the difference between it's the average of the square of the difference between both the output feature map and the image that we're going to the art input image so let's look at what what's happening here okay so it was called um vgg16 because there's 16 layers and uh you know more layers mean you know generally better predictions but less layers mean less computation time so it's a trade-off so for mean squared error we're taking two input tensors so one of these tensors is going to be our image okay uh either our content image or our style image and we're we're going to use it for both and the other tensor is going to be the uh the feature map okay the result of the uh whatever layer that we choose and so what we're going to do is we're going to subtract the difference so these are two matrices okay and we're going to subtract the difference the square of the okay and then we're going to square that and then we're going to do reduced mean so what is reduced mean okay so right so this just this value right here tf. square of the difference between these two uh tensors is going to be a matrix value and then we want to reduce the mean so what does that mean so we if we have a matrix to reduce the mean so it would just be like this we're going to find the average of all of those values inside of that Matrix so for this uh two-dimensional uh this Matrix right here 1122 okay it's going to be uh 1.5 so that's the average value of this whole Matrix so this is going to Output a scaler which is a single value okay what mean squared error is doing okay and it's a way and and and we the reason we Square it so that it's positive okay we want a positive value okay so that's what we're going to use as a as a helper function inside of our content loss and our style loss okay so the first thing we're going to do is write out a function for our content loss okay so we're going to call it create content loss and inside of it we're going to use that uh mean squared error function all right thanks ageras uh so our parameters for this are going to be session okay so our tensorflow session the model we're going to use I need to relax okay I'm trying to talk slower in these live sessions so let me do that because at one point I'm hyped and the other point I want to be as clear and communicative as possible so I will slow down I'm really trying to improve the speed of my speech I've gotten a lot of great feedback on that and I'm very open to more feedback so the parameters for this content loss are going to be the tensorflow session the model which is vgg16 the content image and then the layer IDs which are the indices for which layers we want to use to get those output feature Maps these are the the the layers that we we have decided to use for optimizing for Content loss and these are going to be the higher level layers okay because those higher level layers in a convolutional net are akin to the content right these are the big highlevel objects that are inside of an image like you know for a van go it would be the moon and the you know houses things like that and the lower levels are the texture or the Contours like the lines Quantum AI actually increases the speed of my videos interesting that's be okay well you're in the minority okay but maybe I don't even know it' be cool to get some data maybe I could do a poll back to this so the first thing we're going to do is we're going to define a feed dictionary now a feed dictionary is a python dictionary object that's generated with placeholders as keys so this essentially is going to generate a set of of placeholder values okay and and the placeholders are going to be this image right so that's it so what this did is it generated a key value pair where the key was the image and the value was the content image so we do this every time we want to feed anything into our computation graph we feed it in as a dictionary okay so that was the first step and then the next step we're going to do is we're going to Define what our layers are now we have our model that we Define as our parameter and then we're going to get those tensors uh by defining those layer IDs okay so we can play around with what indices we want to use for our layer IDs and the researchers in this paper did as well you're going to get different results for different layers that you use okay and this is not to say that these are the best indices but they are they are a set of indices that do give us good results and you know abstracting this uh question even further what defines Beauty can we can we frame Beauty in and of itself as an optimization problem this is something to think about in the future how can we minimize a loss for beauty beauty is so subjective but can can it be objective so something to blow your mind a little bit uh okay so those are layers so the next thing we're going to do is we're going to calculate our values so these are the output values when we run our session given the layer that we've just calculated and we feed it in our feed dictionary okay okay so what this is going to do is is going to get the uh output values for those layers for the for the content image okay so the next step is we're going to say okay so we're going to run the model graph now okay so for the model graph we're going to say as default all right see how many okay so model graph as default and uh okay so now what we're going to do is we're going to initialize our losses so this is setting so we can so we're going to add computational noes to it and uh that's what we're going to do right now so let's first Define our losses because we're actually going to calculate a collection of losses so we'll initialize an empty list okay we'll call it layer uh losses that's that's the name of our list and the next step is we're going to let me move down a little bit we're going to say okay so for each layer that we have for for value for each layer that we have we're going to iterate through those layers that we defined in those indices okay values layers okay best way to start contributing to AI is to just start pushing code to GitHub that you that you find fun and documenting it really well and then you know marketing like posting it to people and sharing it with people we're all sharing you know the line between research and production is is quite thin everything is really a discovery in and a in in in any way even a simple hyperparameter change uh can can can result in an incredible Discovery uh Krishna the link is in the description okay so what we're doing is for every uh layer we are iterating through every single layer right now we're going to get that um value okay so what is that value we're going to Define it as a TF constant which means it cannot change and this is going to uh be the U content images value okay so then what we're going to do now is we're going to calculate that loss okay that loss is and now this is where we use that mean squared error that we defined earlier and we're going to use it to the two input tenses are going to be the value at a layer and then the constant value so this is the the both the content image and then the the the set of features at a layer and we're going to minimize that loss the best book for deep learning is um the Deep learning book by Ian Goodfellow okay uh no the one by yosua Benjo search yosua Benjo deeping such a great book especially for the math loss of mean squared error okay so now we're going to say okay so these are all of our losses that we calculated and we're going to append it to that list that empty list going to do right more games I'm about a fullon game AI course next so we'll see how that goes I think it it'll be very very popular so once we have all of those loss functions calculate the uh uh the average value using this reduced mean function okay so these are our layers losses and that's after we're done with everything okay and then uh we're going to say okay so at the end of this we're going to return the total loss so what this is going to do is this is going to give us a total loss value and let's talk about what we just did we said okay uh we're going to get the mean squared error of the feature activations in the given layers in the model so between the content image and the mixed image and so when this is minimized it's going to when we minimize this it's going to make the mixed image that much more stylized okay so this this defines our content loss so ideally we could do this with a style loss as well right we just take both uh feature the feature maps from a layer and then the uh you know the the image uh what whether the the style image or the content image and we just uh calculate the mean squared error between them and we minimize that but we cannot uh previous line from the last I have a typo thank you but we cannot do that okay so we cannot do that because uh for whatever reason wow that actually compiled without any errors because okay so let's let's actually answer this why we can't uh talk about right total L thank you alen so so we don't do that for style for style we add a another step called the gram Matrix and let's let's really talk about what is happening here uh so the reason we use a gram Matrix so let's let let me show an image of this this this this will help so okay so this is so this is kind of scary looking but don't worry about it a grand Matrix all it is is we are taking the uh measuring the correlation between our feature CH like vectors after flattening the feat filter image into vectors so all this is doing is it's taking it's it's it's the matrix product between our initial Matrix and then it's transposed which is just uh which is just flipping at 90 degrees essentially we're just multiplying a matrix by its transpose that's it multiplying a matrix by its transpose that is the gram Matrix okay and so that's what we're doing for that layer okay and for our style so for every uh style layer that we choose we're going to calculate a gram Matrix and that gram Matrix is what we're going to use to minimize via the mean squared error so let's just start building and then we'll start start talking about what exactly is happening here so for our gram Matrix function we're going to have our input tensor okay so it's a vector of the dot products for vectors of the feature activations of a style layer uh yeah I did it was the equal sign right for the total total loss okay so the gr Matrix and remember guys be following along in the GitHub link that I've linked to in the description it's got all my notes I'm basically talking through it and and coding at the same time so this is going to be a 4D tensor okay this is going to be a 4D tensor uh at a given layer because we have the we have right so it's going to be the collection of pixels and then the RGB which makes it 3D and it's 4D because we have a collection of those so we have several of those so we have a collection of 3D tensors which means it's a 4D tensor and that's what this shape is going to be and what we want to do for our style layers is we're going to calculate our uh uh gr Matrix hello Abdullah so now we're going to say let's get those channels okay so channels are just uh each of those so if you think of a the outputs of of each layer so an output of each layer is going to be let's see where our history was see these like stacks of layers each of these Stacks is a matrix and we call those our channels each of these is is one of our channels okay so it's and the word activation and feature it's it's used similarly so feature map activation map same thing okay so back to our number of channels what we want to do is we want to Define these channels and we're going to get the number of feat feature channels for the input tensor which is assumed to be a convol layer with four diens iions okay and then we're going to say let's get the Matrix which is going to be the let's talk about what this Matrix is going to be this Matrix is going to be we have the shape it's going to be negative one and then number channel so what do we just do here so uh negative one means whatever number makes this data fit that's why we put negative one in here but what we're doing is we're reshaping the tensor so it is a two-dimensional Matrix so it basically just flattens the contents of each of the featur channels that's what it's flattening the contents of each of the feature channels so that we can multiply it uh in a second through tf's matrix multiplication function we are we are making it's it's akin to normalization okay it's we're taking both of these values between our style uh and our mixed image and we are normalizing it using this reshaped function such so that we can multiply it in a second okay so let's get that gram Matrix now okay so we're going to use tensorflow's matrix multiplication function to do this and it's got this great transpose function so we're going to remember it's a gram Matrix is just a matrix multiplied by its transpose so the same Matrix will multiply by its transpose and then okay so the transpose by The Matrix and remove this comma because we don't need it and then that's going to be our gr Matrix and then we can return that okay uh so sty transfer is an extension of uh convolutional Nets has uh Gans are a whole different ball game they're a whole different ball game and we're going to talk about that later but before we understand how Gans work we have to understand how uh transfer learning works and how convolutional Nets really work and that's the goal of this okay so that's it for our gram Matrix so that is the extra step we add for style loss okay so let's let's actually compute style losss all right so what we're going to do is we're going to compute style losss now given that gram Matrix function that we used earlier so the inputs to this are going to be the same uh you know basic format that like we did for the content laws except we're using a solid image as our input not a Content image right and we're adding this extra step which is this gram Matrix step so we'll talk about what this looks like when we get to it but we're feeding in remember the dictionary and this creates placeholders for our uh fict and we're going to and this is going to create a placeholder value for our style image and then the next step is to get the layers so our layers are going to be whatever we got from our layer ID so whatever we defined and we defined these earlier or we will Define these layer IDs right these are essentially against to hyperparameters for this for this particular optimization problem because as we tune these the results will be different right so those are our layers so now we're going to Define our model okay so it's you know very similar we're going to Define our model as default and so we're going to Define our gram layer so this is where our this is the line that is different comparatively compared to the content uh loss the style loss means we have to take the gram matrices that is the difference here we're taking the gram matrices of each layer we're not just saying here's the raw layer value which is just you know that Matrix we're saying get the gram Matrix of that layer so and also you know to to give some you know backup to this gtis the guy who initially made this when asked why the gram Matrix in a talk he said the gram Matrix encodes second order statistics of the set of filters I'll talk about D noising in a second Javier but basically it it's looking at it from a higher level so it sort of mushes all of the features at a given layer tossing spatial information aside in favor of a measure of how different features are correlated so by so Grand matrices essentially are like they toss away everything that's unne necessary just to focus on the style and and this works for style but not for Content um it's lower layer and I'm sure there's more theory around why we're using the gr Matrix but uh that's the best I could get out of gtis um and you know it's better than just saying it just works right which is a lot of machine learning uh right so so that's the gram uh layers and then we have where were we again so I went up for a second let me go back down okay so where were we we were right here so we said that we're taking our gram layers and we we're feeding in our dictionary so we're feeding it that dictionary that we just defined feed dict and we're saying that uh we're going to Define our layers losses so and then okay so now we're defining that empty list that we're going to add all of our layer losses to right so so that's that those are our two lines before we get started with our iteration period so for our iteration period what we're going to do is we're going to say okay so for all of those brand matrices at each layer we're going to compute the uh loss function so that we can minimize it and so we have we have two um inputs here the value which is of our image and then the gram layer which is the gram layer okay so the value is going to be constant because it's a tf. con because we don't want it to change right we it's not a variable because we don't want it to change it's going to stay the same and uh now we're going to calculate the loss so loss is going to be the mean squared error between and now this is the important part the important part is we're calculating the mean squared error of the gram layer and then the value constants so this is between the gram Matrix and the value of a gram M Matrix when inputting a style image Okay so that's between each of the layers and the input style image layer okay so then once we have all of those losses we're going to add them all to our list style list so layer equals layer the losses at each layer so you know we're we're combining these losses and we later combine the combination of those losses between right so it's like combining and then combining again we'll we'll talk about the second step of combining but this is the first step of combining we're combining losses for uh style and for Content okay so we're appending whatever we calculated at each iteration here and then now we're going to calculate the total loss so the total loss the total loss is going to be the we're going to reduce the mean between layers so that we're going to take the average of all of those losses remember this just takes the average of every single number that we have in our Matrix and then we're going to return that value okay and that's our style loss and we can and this is a scalar value this a single number okay the scalar value so that's our style loss and then what do we got here we got some invalid syntax my favorite my favorite okay all right comma duh okay so that's what we did for that and uh right so feed dict model. create okay model. create feed dict image style image uh what is it saying here oh gram Matrix so for gram Matrix gram layers we have feick where was I so many things to keep cack up here oh right that was what it was okay so um let's see oh right yeah that great Ram Matrix layers and then fict what were we here R Matrix layers four layers oh right let's see hold on I have a lot of code here that I'm looking at in hold on a second so gram layers are going to be the gram Matrix layers for layers in layers oh so what actually there's an extra line here that I forgot about so for got about a certain line here hold on there's a certain line that I'm missing here ah okay let me just remove these lines right here great so now so there's actually one more loss I lied there's one more loss but I lied because it's very very trivial and we used it was used in the paper but it's it's not you know it's it's very trivial and it's it just serves to confuse if we if we talk about it at the start but we're going to talk about it now so this is just this is the den noising loss and so what happened was when when they when they didn't do this loss when they just computed the style and the content loss the image was okay but when they added this the noising loss which we can also call the total variation loss it's also called that what happened was the results improved so what it does is it calculates a difference between the so it shifts the input image by one pixel on the X and Y AIS and then it calculates a difference between the shifted image and the original image and so the absolute and we and we use the absolute value to make it positive okay so let's talk about this so this is the one more loss function that we're GNA add very trivial loss function it's going to be just you know two lines of code but uhu you know it's necessary to just you know slightly improve our result and you know there could be even more loss functions that we can think about and um right so there's 16 layers here okay so we're going to use the absolute value um for this and uh so in fact I could just you know I'm just gonna paste this in here because we're going to get right to the meat of the code I'll just talk about this since it's just two lines and it's got a lot of uh grammatical things here so we are we what we're doing here is we are uh taking the absolute value to make this uh a positive value right and then we're going to calculate the sum of the pixels in these images and basically it helps suppress noise in the mixed images that we generating okay so we're calculating the sum of the pixel values between the shifted image and our original image and this the variation and the variation is referring to that shifted image and the original image is it removes the noise and the noise is like the blurriness it's not that clear and by shifting it and by you know minimizing it we're kind of like recreating this uh blurriness and we're saying you know we could even shift it more than one pixel and and try that they didn't try that you know shifting it you know we could do other um you know artificial noising step like what can we do to like you know mess up this image in whatever way okay and then we want to optimize for for minimizing that so that our our actual value is smaller okay so so now what we're going to do is um how's the how's the frame rate by the way guys uh definitely tell me in the comments how you feel about the frame rates um I'm trying to improve my live live streams okay so now we're going to get to the good stuff this is the meat of the code this is where it all comes together the style transfer algorithm this is where we build our model and we're going to use the loss functions that we calculated okay so let's talk about what these parameters are going to be so the parameters are going to be uh a the text is blurred pretty cool frame rate thanks ni okay frame rate seems good all right all right cool so how much so how much do we want to weigh these loss functions is a good question okay so you know like this this can be played with right so what they did so you can see here that the weight style the weight for the style is going to be 10 so it's almost you know it's it's like eight times more than the weight for the content and we can modify these weights so that we can we got 144p over there okay um bury the text let make this bigger got 720p by sham okay interesting so we've got all over the place um okay so so we're weighing the style way more than we're wearing the content and it doesn't mean that these are the best weights and in fact we can have learning algorithms for learning the best weights we could learn the best weights later right so that's that those are the weights uh that we're going to use and we're going to Define these weights uh in a second but right so let's go ahead and just do our first step like I'm assuming that everything else we did was just a helper function for this this is the real deal so we're first going to Define our model okay it's going to be vgg16 we imported it we're good to go right 16 layer convolutional net fully connected layer at the end it's it's thinking like oh I'm I'm GNA be used for classification that's what it's thinking right now okay so now uh we're going but it doesn't know any better it doesn't know any better we're not going to use it for classification we're going to use it for style transfer so we've got our Gra and we're going to feed in our model okay so that's the session remember the session always encapsulates our computation graphs okay so then we've got these print functions here that we're going to use to help us see what's happening and I'll just paste these in because they'll really just help her you know so now now is the the good stuff so here is our loss for our content this is our first loss function and we Define the um we Define the the loss function right so the session is going to we're going to give it the session value we're going to give it the model value and we're going to give it the uh content image and the layer IDs okay so no buffering though interesting okay interesting interesting all good things to consider okay so we're going to give it the content obviously because we are trying to minimize the loss for the content and then we're going to say the layer ID so these are the indices so right we just feed in those layer indices and it's going to uh calculate the difference between the raw activations at a at the given layers and our content image and we think of that as a matrix just a raw activations activations no hidden step and the value the output value is our lost content and remember this is a scalar value okay this is going to give us a scalar value the next step is to uh calc calate our our style loss okay so for our style loss there is that extra step that extra step is going to be the uh brand matrices calculation so it's not that we just give it those raw activations we also want to give it the we we take those raw activations and we calculate the gram matrices from them okay and and the same thing but on the you know in this in this uh you know function we're not we're not talking about the grand matrices that's all happening internally because we defined that on the outside it looks the same which is good programming right we we we want it to look the same because um there's no reason for us to expose that gr Matrix gr Matrix calculation externally for you know this this function for re reusability okay so that's for our loss and that's for our style and remember we had one more very trivial loss function for d noising right so for denoising we said we just said well take our model and then remember to denoise it so that you know it's just so it's a little less blurry just you know a little touchup at the end okay so those are our three loss functions and now what we're going to do um is we're going to adjust Javier that's a great question why don't we use the grand matrices for Content loss um so uh so it's a second order op it's a second order statistic meaning for for sty so style is actually you know it's it's you can think of style as more abstract than content content is something we can point at and say that is content like for example the what is a famous content like the the the clock that is bending in the dolly picture doll's um uh where's my mouse the doly P you know the with the clock that's bending that that is content that is something we can look at and we can definitively say that is the content so we can take the raw activation and say you know it's just literally there it just it describes that that object but style is much more Nuance style is much more embedded and ingrained throughout every layer it's not just like at one layer we see oh that is the that is where the clock is it's it's embedded in every layer so by taking the gram Matrix we're kind of abstracting the we're taking another layer of abstraction to say um that we want to style is even more abstract than content and so Grand matricies help us create that extra layer of abstraction okay so now we're going to uh create those uh adjustment weights right so for our content and our style so it's basically just the same thing here so we'll just you know copy and paste same thing over and over again the three lines but basically it is creating tensor of variables for adjusting the values of the loss functions Okay so so we want to adjust how uh uh we want to adjust how uh how much we want to weigh each of them okay and so now what we're going to do is we're going to uh initialize them right so we say we're going to say session. run we want to initialize these values so this is the actual run step and the the the next step is we're going to uh adj actually adjust them because we haven't actually ad adusted them we've just defined them now we actually adjust it now we actually adjust them uh style transfer can definitely be applied to audio and wavenet is a great application of that um and there are you know there's a definitely a lot of discoveries to be made around speeding up wavenet okay um but the idea of transferring the style to something can be applied to all forms of media not just images audio video uh you know all sorts of things but these are the updated values and the reason we're using this 1 E10 value is because we it basically it's just taking the reciprocal values of the loss functions and then it's going to uh using this small value 1 E minus 10 is added to avoid the possibility of division by zero so if loss content was equal to zero this would just throw an error so that's why we add this very small we could almost think of it as a bias Vector but for adjusting adjusting these uh these uh uh weight values okay and so now we're going to actually combine those losses so now that we've defined the losses we've defined how much we want to weigh each of them now we finally we combine the losses into one big huge loss function and it's a weighted loss function that we will minimize to generate the mixed images and the reason we're multiplying these loss uh values with the reciprocal adjustment values that we calculated up here is because uh we can use the relative weights for the loss functions that are easier to select the exact choice and the the the content layers of the style and the content layers so this is our combined loss function and give me and so give it five five minutes and we're gon to get right to the um we're GNA get right to the actual output value we're almost there guys sck you guys are very uh patient for sticking with this right there's a lot of stuff we're talking now so now we're going to calculate the gradients so let's so the gradients is a great thing to talk about right so we have our combined loss functions and now we're going to use tensor flows built in gradient function to get it multiple values right it's not just one uh we're using multiple gradient values and we're going to get the combined loss given our model's input value and the combined loss so this is what it's going to do is going to get the gradient of the combined loss function with regard to the input image which is the mixed image so this this wants to minimize difference between the mixed image and remember that mixed image is just a blurry you know uh nothing image right it's just an initialized random noise image and that's what we want to Output as our fully you know output value that the transferred style image and the gradient is going to get the um it's a mathematical function so the gradient value is going to give us a direction and so uh a great uh image of this I have a great um that I'll talk about it's called build an AI writer also I'll do another one uh you know generate Shakespeare paintings or Shakespeare text check out that is not tf. gradient that's Nike and some dude um what's like that gradient at sigmoid basically I'm just trying to visualize what this looks like basically we are taking the none of these are okay so we have a sigmoid function and we the gradient is like if we were to take a line I'm just showing this with my mouse up or down that's it up or down we take numbers and do we want to push these numbers up do we want to push these numbers down and so that's what the gradient is doing so when we take these gradient values okay we take these gradient values and we multiply it by weights the weights in this case are going to be the image it's going to update our image in a direction which is up or down that is going to minimize a loss okay so the gradient is a collection of those gradients for each of those uh feature vectors uh for each of those remember it's a it's a stack of features it's a stack of features okay to take the list of tensors so but the gradient was import super important so I pushing numbers up and down so you didn't get that right so these are a collection of scalar values okay these are a collection of scalar values and when we take these gradients and we uh these are a collection of scalar values and um we're going to use them not here but in a second we're going to actually I'll I'll talk about them when we when we get to them but basically uh I'll talk about them just okay so this is going to give us a list of scalar values okay that's what greatness is doing with regard to the loss function and we'll talk about what it's doing with the scalar values in a second but just think of them as scalar values right now we're about to get to them so now we're going to initialize that mixed image we haven't actually initialized the mixed image yet now we initialize it right so we're using random the numpy's random. Rand function to initialize an empty Matrix of just random values okay and we're going to update this mixed image uh through through iteration okay so so now it's time for our training sta so for every so for the number of iterations that we have and we have quite a bit of iterations to do you know any anywhere from 100 to a thousand this by the way is going to take like an hour on my laptop which is a MacBook 2016 the force touch bar which sucks God I need a better computer anyway so um create so now so the first step remember is going to be to create the feed dictionary all right so we're going to create our feed dictionary and then given that mixed image right so we just initialize the mixed image create that feed dictionary okay now what we're going to do is we're going to calculate the value of the gradient as well as adjusting the value so it's breaking up from time to time it could just be you okay interesting to um note so we got couple minutes left so let what I'm going to do now is I'm going to talk about the gradients and so in order to focus on the gradients that we're going to do past the rest so that we can focus on the gradient values okay so so we have those gradients right so they're a collection of scalar values between the mixed image and our combined loss function so you know think of them as like they are the representation of like the what we want to use to help us you know increase the the style in our the mixed image what how do we stylize that image the gradients are going to are give us tell us how to update our image okay a gradient is a slope value it's a so given a sigmoid um function so if it would just is you remember so for you know feed forward Nets we took the derivative of a sigmoid and it gave us a a line a straight line gradient means slope it's just a straight line and this the line either points up or it points down this is a great image it either points up or it points down okay uh so it's you know it could either be plus two or minus two right and so so when we take these gradients and we multiply it by that image it's going to transfer it it's going to uh update that image in some way and every time we uh every time we minimize our loss the gradient is going to be is going to be better and by better I mean it's going to help us uh it's going to be it's going to uh minimize the difference between the output image and our style image all right it's gonna every time it's going to be minim minimal all right so eventually when it's zero a local minimum uh the the output image is going to be the most stylized that it can be for whatever uh however we frame this problem so this step is going to remove the single dimensional entries from the shape of an array so that our gradient value is going to be reduced such that it's going to be the same size so that we can multiply it by our Matrix okay so then we're going to uh scale the gradient so what are we what are we doing why are we doing this this is the same thing as learning rate so it's the ratio between the weights and the updates so if if if if we don't scale it properly uh then uh it's not going to converge fast but if we if we don't scale it fast enough then it's not going to converge ever if we scale it too SL no sorry if we scale it too slowly then it's never going to converge if we scale it too fast it's going to overfit so so we have to update not just our um gradients but our our weights at each layer so so that it's going to converge properly that's what this step is and then we're going to perform gradient descent and so this is the actual gradient desense part right this is what I'm talking about we take those that gradient which is a scaler and then we multiply it by the scale value that we calculated here and it's going to give us that output mixed image which is going to suck it's going to suck at first it's not going to look like anything but the more we update this gradient the more this image is going to look Doper and Doper okay like more and more stylized okay and so that's going to be our mixed image and then we're going to clip all the values that are not between zero and 255 so everything more we're going to clip it because we want this to be an RGB value and then we're going to print out the number of iterations and we're going to plot them okay so that's what that's going to do and then we're ready to uh show you know what we're going to so that's that's our style transfer algorithm now we can print out load image is not defined I probably didn't oh I need to define the load image function earlier it's all I didn't compile that earlier let's um let me just show this okay so this is actually like if I were to actually run this right now it would take a lot of computation power which would lag the hell out of this live stream and I definitely don't want to lag this live stream further so let me talk about this for two minutes and we're going to do an ending Q&A okay so where were we okay so remember so given the uh style image the content image and then the indices of whatever we want to use like what are the layers we're choosing for this and then how much we want to weigh each of those loss functions it's going to slowly minimize that loss so this this even this first iteration is going to take like like five minutes so even in this first iteration you already see that it's getting better like like already went from nothing to this in one single iteration okay and after the others it's just going to be so it's kind of like an U um a negative exponential curve so uh it's it's super good at first and then every uh subsequent iteration is g to get uh you know the the the um Improvement is going to get smaller and smaller over time okay okay so uh let me do five minute Q&A to end and then we're g to get started hi guys I'm definitely going to improve the lag God I really want to improve these live streams and I really want to and I will I promise I will because I love you guys and I want to make this better for you okay so ask any you know leftover questions you have from this live stream which I'm going to read out very clearly for the people who are watching this later and then I'm going to you know whatever other questions you have as well about life or I mean machine learning soaj what do you think of neuroevolution of augmenting topologies neat as a genetic algorithm and do you think you will do a video on it sometime so genetic algorithms I've done a video on genetic algorithms it's called um genetic algorithms for Learn Python for data science they're really cool the idea of evolution of darwinian evolution it's really cool and it's it's how we've evolved right um but in practice in practice we don't see um you know as good results as we see with you know this with deep learning but you know we might L on you know who knows and I think that there are ideas from gen genetic algorithms that can be applied to deep learning remember what we're seeing and the real discoveries that are happening right now is when we are taking these abstract ideas this this idea of hierarchy from Deep learning this idea of an agent of an of a of reinforcement of trial and error from reinforcement and by combining these ideas we're going to get even better results so I think genetic algorithms have a lot of great ideas to give to us okay and uh I would I will definitely do more videos on genetic algorithms in the future how to communicate with Objective C uh don't use Objective C use Swift how to improve because apple is now maintaining that and actually Swift is great for even serers side code at this point not just mobile apps how to improve CV for Google intership mol I have a video on job interviews that I'm going to release very soon either this week or next week I already have the footage I just need to edit and release that how close are we to humans to solving intelligence my best guess is 5 to 10 years uh unless some kind of cataclysmic you know anything bad happens that prevents the the rate of discovery that's happening right now are there starter videos that you talk slower in uh I'm just gonna keep talking slower and slower okay uh okay so a couple more questions do you think an evolutionary gentic algorithm can create the structure of a Network absolutely I mean absolutely they we've seen them create a bunch of other things as well I don't think I've ever seen anyone try to do that so that's a great project to try to recreate a let's say a simple thre layer feed forward neural network using genetic algorithms that would be a great project five to 10 years is very optimistic but just look at like the paper that was just released Yesterday by Deep Mind where they already beat their deep Q Network that is can be uh generalized to a bunch of different games um what else should we learn in AI after ML and DL after ML and DL that's really the the bleeding edge right now um but I would say if you want to go just even further like the real future one shot learning specifically probabilistic programming okay probabilistic programming or you could call it basy in program learning it's the same thing do you have any examples of using a neural net on random images from the internet uh yeah build a tensorflow image classifier in five minutes okay how do you keep up with all the white papers that are released Andre karpathy has a great website called archiv [Music] ree next year or should I roll on my own with you and Udacity roll on your own Anton roll on your own remember to have your GitHub full of great projects because GitHub is the new resume okay and practice your interviewing skills and I'll have a video on that very soon okay two more okay so actually two more questions additional resources for style transfer I will link those to those in the description also my last video last weekly video got some great uh um resources in there one more question okay what do you think about the new no uh I want to make it a good question what drives you Cham gupa I okay so what drives me to be you know totally honest what drives me number one more than anything is to solve intelligence by teaching you guys you know whatever I whatever I know uh you guys can help us all because it's it's a share Journey we're trying to solve intelligence so that we can solve everything else we have so many problems in the world if you've traveled to developing countries you see just the the scope of problems is so immense that it is just I am very impatient and I want to solve all these in my lifetime I want to see it all solved and the best way to do that is to make better more intelligent software it is the best way to do that okay and uh that's what drives me and also just I've always wanted to do great things and this is a a great way to do that and also I just I want to be out there and I want to I feel like I I feel like I have a responsibility to lead it's like this internal like I I just feel like I I need to dream big and and lead and and and I want to and and see you guys become leaders as well because you are all the real leaders of the future I'm here to help you guys become leaders okay uh so that's what I'm here for and I will I will keep going I will keep going no matter what happens I will just keep going and um I will keep going until my last breath hopefully there's no last breath because we Sol Ai and you know we have the singularity okay is deep learning a fad no it's not a fad it's a it's it's a next step in in a series of discoveries and I'm sure the next thing is going to be even better but Dave uh in a way it's not like it's that's the end goal right there's GNA be more after it but it's a part of this um thing about I'm not working at toio um okay so uh this what I'm doing fulltime all right so that's all for my questions thanks guys for being here I have a video coming out this uh on Friday like always and uh okay so that's it for this live stream let's see anything else I want to do uh no I'll just say I'll save the wrapper next time okay so love you guys remember to uh links are going to be in the description everything's going to be awesome remember to join our slack Channel all right love you guys and uh thanks for watching for now I've got to go drink some coffee and then transfer the sty the coffee into my brain and just minimize a loss between my uh sleepiness and my wakefulness so oh freestyle oh someone said freestyle so I'll do it because you guys know I love to Freestyle um uh freestyle about coffee so I love to drink coffee no no no that's that's I want to do it about machine learning so uh so what's a what we just talked about we talked about uh style transfer so style okay I love to talk about style I take my boat drive down the the River Nile I do it and I spit out bile that's not B it's flow anyway that's the it for this time people laughing cuz I'm trying to Freestyle it doesn't matter because I'm trying to go in the Nile River you got ancient secrets that the Egyptians made but now we got even more man bars bars bars okay W freestyle transfer anyway all right thanks for watching guys okay