Image Filters using WebGL - Developer Diary #Day4

Key Takeaways

hey everyone time for another edition of the developer diary for my camera one of the things that got me excited about this project in the first place was the image filters the filters that I've made a fairly standard things that you might see in other camera apps or image editing software like changing the brightness contrast and that sort of thing the filters in this app a written using WebGL I'm not going to dive into the details of WebGL in this episode but I recently record an episode of supercharged Reaser mer where I wrote some code that sets up WebGL so you can use it to manipulate an image it's about an hour long so I'm not going to assume that you've watched it but there is a link in the description if you want to go do that now the very brief summary of the WebGL setup is that you create a canvas element and attach some code to it that runs for each pixel this has to output what color that pixel should be should be this code is called a shader the code for the shader is written in a special language that is kind of like C I'm not going to go into the details of the language but instead I want to talk about what my code is actually doing the shader can look at data from the original image only when you're talking about images in WebGL you tend to refer to it as a texture so here we have the code for my shader we have a thin cord of varying up here which is called tex cords this is the coordinates of the pixel within the canvas that can be used to look into the original image the texture hence tex cords and this is a different value for each pixel which is why it's a varying you can also pass in some values from your JavaScript that was shared for every pixel these variables are called uniforms because they are the same for every pixel this is how it can pass through things like what the contrast or brightness levels actually are so here we have uniforms for all of the filter settings saturation warmth etc this texture sampler is how we're going to refer to our original image and this woman she is source size this gives us the width and height in pixels of the original image so how do these filters actually work what are we doing well one of the first things that I I do in my code is something called a convolution filter so a convolution filter it's kind of a complicated name something that's actually relatively simple what we're going to do is we're going to calculate the color of an out pixel based on more than one input pixel so here I want to work out what color a particular pixel should be when I produce my image and the way I do that is I look at the values of the colors for that pixel and some pixels around it so here we have a 3x3 convolution because we're going to look at nine pixels and a 3x3 grid each pixel is given a weight this is how important it is to the output this particular set of weights is going to give us what's known as a blur so what we do is we gather up the colors of all the pixels around it and we multiply all of the color values for each of these pixels by the weight because all these color values are going to be numbers from 0 to 1 and then we add them all together we can earn it with a number between 0 and in this case 16 once we've added them together you know adding with the weights will then divide by 16 to get back to values between 0 & 1 the way that you might think that you're going to do this is you have an array with your weights in and you loop through you know from minus 1 minus 1 to 1 1 and you go through and look up all 9 pixels in a loop and you get each color and you multiply it by its weight and you add them to a total and at the end you divide by the sum of all those weights and that's more or less what you do except that in WebGL shader I'm gonna optimize this slightly by unrolling this loop so there is instead of doing the loop as a loop I'm going to write out each individual statement that's going to get run on its own line so there instead of running a loop nine times I'm going to write nine instructions so you might be wondering why would I do this while we don't all this loop and do have each instruction separate rather than looping over them and it's to do with how graphics cards work so a graphics card contains hundreds of GPU cores which are great at doing mathematical calculations but I think they're not so good at that a ordinary CPU is is branching so that is when things are different when you run the code again so in a GPU it's used to doing the same calculations over and over and over again it sets up a pipeline to just read in data and output the output without doing any like decision-making so when we have our loop many graphics card drivers will actually optimize that out automatically you will say okay so it's a constant number of runs for this code and we can work out at compile time that this is going to run nine times and it's going to use these values and it can produce that unrolled version for you but not every driver does it and particularly things like texture reads we're looking at the color information of the original image that can be very highly optimized and you know pipelining in those those colors and you know based on the fact that we're going left to right and we're always pulling things out in the same order so the graphics card can make that really fast if it knows ahead of time what the pixels are going to be that are going to be looked up and when you introduce that loop some graphics card drivers have that as a branch and they don't optimize for anyone so this could be extremely slow on some graphics cards now most of the time this will be fast most modern devices this will work just fine but not all of them I test it on some devices where this was extremely slow and so that's why I'm doing it so let's have a look at the actual code in my shader for this so I did a 3x3 matrix in the example I've actually used a 5x5 matrix in my code because sampling well values you get a smoother blur effect but it's otherwise exactly the same so here I'm using the texture 2d call of the shader language to look up in my texture sampler some coordinates including the offset that I'm looking at to get the color value for a pixel then down here I'm adding up all of those color values along with their weights and then dividing by the sum of those weights now you may notice the in the app here I don't actually have blur as an option what I actually have is sharpen now here I've got the sharpness turned way up but I can turn it down to nothing and you can see that the sharpness of the image changes when I move the slider so how am I using a blur to make an image sharper well the way that that works is using something called an unsharp mask this is a technique that was actually invented for film back when it really was film this is a pre-digital technique to make things sharper what you do is you take the original image and you take the blurred version and you find out the difference between them so you you're basically saying how does this image change when it gets blurry and then what you can do is you can undo that so if the pixel gets more red and less blue when it's blurry then you can make that pixel less red and more blue to make it sharper which is actually extremely clever way of doing this you do a negative blur almost so whether it's implemented here is that I work out this distance between the texture and the blurred version and I add it to the original texture color and then I actually scale this effect with the sharpness value from the slider so if sharpen here which is just a number if let's say this was zero then this whole thing would be zero as well and though we would be adding nothing this would have no effect but also if sharpen is negative then we actually just add on the blurred version we make the image more blurry and you can see that in the app so here we take the sharpness and we move it down to the negative and the image becomes blurrier than it was so that's applying a convolution filter in this case using a blur to sharpen the image there are convolution filters that do all such things that give you an embossed image or confined edges there are all sorts things you can do with a convolution filter I'm just using it to increase sharpness now there are some simpler operations you can do as well so for brightness what I actually do here is I take each color in the original and I just multiply it by whatever the brightness value is so if you have a brightness of 1 nothing happens whereas if you have a brightness of 2 every pixel gets twice as bright that is the red green and blue values are doubled and similarly a brightness of 1/2 will mean that everything gets darker all the pixel values get to be half what they were want is another interesting one I kind of thought this would be much more complicated but I found a very simple formula that works pretty well so you take your warm with value and you just increase the read of the pixel by that warmth value you add the want value on and you get the blue value and you subtract the warmth from it it's an extremely simple formula but it actually gives a pretty good effect so you can see that what happens is that when you increase the warmth slider the image gets less blue and more red and it gives it a warmer tone and similarly when you decrease the slider the image gets more blue and less red so this is obviously an extremely simple way of achieving this effect which I found pretty surprising and I found that for most of the effects that are used in filters there were kind of two schools of thought for each of them one of them would be an extremely simple mechanical method of achieving the effect so for warmth just change the red and blue values and it's just by a fixed value and that gives a pretty good effect other people are taking detailed models of you know how the brain perceives images and how your eye works to produce models that are a bit more natural but more complicated so I've opted for this simple way in all of these but you may find that if you search around for how to actually implement these that you get some of those complicated versions coming through as well so the next filter I want to talk about is contrast this was another one that was interesting I didn't know how this worked before I start this project and what we're doing is the idea is to make the pixels seem more distinct to make every part of the image a little different from the other parts of the image so the way that we can do this is we can say let's make every pixel make every color get a little bit further away from gray so gray is where the bread is at half green is at half blues at half like it's halfway between white and black and the way that we increase the contrast in this situation is if the red value of our pixel is more than half then we make it bigger and if it's less than half we make it smaller we make all the pixels get further away from gray this increases the contrast now this doesn't work perfectly if your image is let's say it's very dark so most of the pixels will be less than gray this actually will decrease the contrast everything gets closer to black when they're all very dark and same will happen if everything's very light so the other thing that I've got here is the ability to change the gray point so that's what value are using as gray so we can set the the gray point from almost black to almost white and when the gray point is almost white then that means that many of the lighter pixels will still get darker and similarly when we move the gray point down to towards black darker pixels some of them will get lighter so you can choose the gray point suit your image generally speaking you want to set the gray point to be the average of all of the pixels in your image so that more pixels are moving away from each other on average I haven't implemented that automatically though that is a thing that some image applications do instead I've just given a slider and you can choose whatever looks nice so that's how I implemented some of these filters you know things like the vignette it's making the image darker the further away is from the center and things like that there are many other kinds of filters that I could have implemented that you might find in other applications I could have done them in the more complicated way but I hope this is a reasonable example of how these things work so I hope you enjoyed the show and that you'll join me next time when we'll talk about something else thanks for watching my video remember you can subscribe to the channel for more from Chrome developers or follow the links on your screen to watch other episodes of develop attire Chiz