C4W4L10 Style Cost Function

Key Takeaways

in the last video you saw how to define the content cost function for neuro style transfer mix let's take a look at the style cost function so what is the style of an image mean let's say you have an input image like this you're used to seeing a confident like that compute features that there is different hidden layers and let's say you've chosen some layer L maybe that layer to define the measure of the style of an image what we're going to do is define the style as the correlation between activations across different channels in this layer L activation so here's what I mean by that let's say you take that layer L activations so this is going to be a niche by NW by NC block of activations and we're going to ask how correlated are the activations across different channels so to explain what I mean by this may be slightly cryptic phrase let's take this block of activations and let me shade the different channels there are different colors so in this lower example we have say five channels and which is why I have five shades of color here in practice of course in your network we usually have a lot more channels than five but using just five makes the drawing easier but to capture the style of an image what you're gonna do is the following let's look at the first two channels let's look at the red Channel and the yellow Channel and say how correlated are activations in these first two channels so for example in the lower right hand corner you have some activation in the first channel and some activation in the second Channel so that gives you a pair of numbers and what you do is look at different positions across this block live active ations and just look at those two pairs of numbers one in the first channel the red channel one in the yellow channel the second channel and you just look at these two pairs of numbers and see when you look across all of these positions all of these and H bar and W positions how correlated are these two numbers so why does this capture style let's look at an example here's one of the visualizations from the earlier video this comes from again the paper by Matthews either and brought ferger's that had reference earlier and let's say for the sake of arguments that the event neuron corresponds to and let's say for the sake of arguments that the red channel corresponds to this neuron says trying to figure out if does this in a little vertical texture or in a particular position in the image and let's say that this second channel this yellow second channel corresponds to this neuron which is you know vaguely looking for orange colored patches so what does it mean for these two channels to be highly correlated well if they're highly correlated what that means is whenever part of the image has this type of a subtle vertical texture that part of the image will probably have this orange tint and what does it mean for them to be uncorrelated well it means that whenever there is this vertical texture it's probably won't have that orange tint and so the correlation tells you which of these high level texture components tend to occur or not occur together in part of an image and it's the degree of correlation that gives you one way of measuring how often these different high level features such as vertical texture or this orange tint or other things as well how often they occur and how often they occur together and don't occur together in different parts of an image and so if we use the degree of correlation between channels as a measure of the style then what you can do is measure the degree to which in your generated image this first channel is correlated or uncorrelated with the second channel and that will tell you in the generated image how often this type of vertical texture occurs or doesn't occur with this orange tint and this gives you a measure of how similar is the style of the generated image to the style of the input style image so let's now formalize this intuition for each of the two images the star image and the generated image you're going to compute a style matrix so more formally let's say that you're using layer L to measure the style let's let any subscript IJ K be the activation at position ijk in that hidden layer L so this indexes into the position height width and this indexes into the different channels so what you're going to do is compute a style matrix for layer L and for the style image and this will be a NC by NC dimensional matrix and you do the same thing for the generated image as well but now let's define this style image so ye define using layer L and on the style image it's going to be a matrix where the height and width of this matrix is the number of channels by number of channels so in this matrix the k k prime element is going to measure how correlated our channels K and K Prime so more formally let me define this as sum over I sum over J of the activation at position I J of channel K times the activation at the same position I J for that channel K Prime and just multiply these two things together so I and J sum over the height and width right at that layer L so you summing over the different positions the X&Y positions who is finally height and width and then just multiplying out the activations at channel K with channel K Prime and so far I've been using the term correlation technically this is the unnormalized cross covariance because we're not subtracting a mean and you're just multiplying all these things so this is going to be the style matrix for the input style image s and then you do the same thing for the generated image so this is going to be really the same thing sum from J equals 1 equals n WL of the same thing IJ K a subscript IJ a subscript IJ K Prime now and if you want to distinguish these activations I guess you could put a superscript round bracket s and G just to distinguish that these are the activations on the sow images and on the generator image G and we denote these matrices using the capital alphabet G because in linear algebra this is sometimes also called the DRAM matrix but I'm just gonna call this the style matrix in this video oh sorry just pick up KK prime there right so these are the formulas for defining the KK prime elements of this NC by NC square matrix so what you're going to do is given an image computes something called a style matrix which will measure all those correlations we talked about on the last slide so more formally let's let a superscript L subscript IJ K denote the activation at position ijk in hidden layer L so I indexes into the height J indexes into the width and K index across the different channels so in the previous slide we had five channels but K will index across those five channels so what the style matrix will do is you're going to compute a matrix we call this G superscript around the square bracket L this is going to be an NC by NC dimensional matrix so be a square matrix remember you have NC channels and so you have an NC by NC dimensional matrix in order to measure how correlated each pair of them is so particular g/l ke k prime will measure how correlated are the activations in channel k prepared to the activations in channel k prime where here k and k prime will range from 1 through and see the number of channels there are in that layer so more formally the way you compute G L and I'm just going to write down the formula for computing 1 elements of the KK prime elements of this this is going to be sum of I sum of a J of the activation in that layer IJ K times the activation at i j k prime so here remember I and J index across the different positions in the block indexes over the height and width so I is the sum from 1 to N H and J is a sum from 1 to N W and K here and K prime index over the channel so K and K Prime range from 1 to the total number of channels in that layer of the neural network so all this is doing is is summing over the different positions of the image over the height and width and just multiplying the activations together of the channels K and K Prime and that's the definition of G cake and you do this for every value of K and K Prime to compute this matrix G also called the style matrix and so notice that if both of these activations tend to be lashed together then G KK prime will be launched where as that they are uncorrelated then G QJ prime might be small and technically I've been using the term correlation to convey intuition but this is actually the unnormalized cross covariance because we're not subtracting out the mean and this is just multiplying how these elements directly so this is how you compute the style of an image and you'd actually do this for both the style image s and for the generated image G so just to distinguish that this is the style image you know maybe let me add a round bracket s there just to denote that this is the style image for the image s and those are the activations on the image s and what you do is then compute the same thing for the generated image so it's really the same thing some of rice some of you J AI j k l AI j k hello um and the summation indices are the same right as follows and you wanna just to note this is for the generated image I'll just put the round practice G there so now you have two matrices they capture whether it's the style of the image s and what is the style of the image G and by the way I we've been using the alphabet to capital G to denote these matrices in linear algebra these are also called the gram matrix all these are called gram matrices but in this video I'm just going to use the term style matrix but this is term gram matrix that motivates using capital to denote these matrices finally the cost function the style cost function if you're doing this on layer L between s and G you can now define that to be just the UM you can now define that to be just the difference between these two matrices GL G squared and these are matrices so I'll just take the previous long so this is just the sum of squares of the element-wise differences between these two matrices and just the right result this is going to be sum over k sum of a k prime of these differences s K prime minus G L g k k prime and then the sum of squares of elements the authors actually use this for normalization constants two times R and H and W that layer and see that layer and then square this you can you know put this up here as well but the normalization constant doesn't matter that much because this cost is multiplied by some hyper parameter B anyway so just to finish up this is the style cost function defined using layer L and as you saw under previous line this is basically the Frobenius norm between two style matrices computed on the image s and on the image G for business on squared and then we read additional normalization constant which isn't that important and finally it turns out that you get more visually pleasing results if you use the style cost function from multiple different layers so the overall style cost function you can define as sum over all the different layers of the style cost function for that layer you should define the book weighted by some set of parameters by some set of additional hyper parameters which you only know as lambda L here so what this does is allows you to use different layers in the neural network both the early ones which measure relatively simpler low-level features like edges as well as some later layers which measure high level features and cause the neural network to take both low level and high level correlations into account when computing style and in the program exercise you gain more intuition about what might be reasonable choices for this hyper parameter lambda as well and so just to wrap this up you can now define the overall cost function as alpha times the content cost between C and G plus beta times the style cost between s and G and then use gradient descent or a more sophisticated optimization algorithm if you want in order to try to find an image G that human eyes that tries to minimize this cost function J or P and if you do that you can generate pretty good-looking neuro artistic and if you do that you'll be able to generate some pretty nice novel artwork so that's it for neural style transfer and I hope you have fun implementing it in this week's for an exercise before wrapping up this week there's just one last thing on my share of you which is how to do convolutions over 1d or 3d data rather than over only 2d images let's go on to the last video