Machine Learning API Tutorial (LIVE)

Key Takeaways

here we go I'm about to start starting the stream all right okay okay okay okay okay here we go here we go here we go all right hello world it's Suraj and welcome to my livestream in this livestream I'm going to build a machine learning API and it's called a neural style transfer API and why talk about it if I can just show you okay so here is the base image okay it's this best laptop for machine learning image and when I feed it into style transfer it's going to become this okay so it's going to do this so what I want to do is just do another demo and just show you really quickly how this works hi everybody it's good to see you I have a little chat window here of everybody here hello we're all blessings from normal way Wow the Vikings the Vikings are alive and well oh man I'm so happy to see you guys I wanted I kind of like last minute decided to do a live stream because you know I set up a live stream studio and it's time so I really want to do these weekly right so so that that's the goal okay so down to business so what we've got here is we've got this little app file and inside of this app file I have my my thumbnail videos that I use for YouTube and so what I want to do is I'm going to take that image and I'm gonna send it to my API so what I did was I created an API and so I can send that image there so here's the here's the console let me open up the console we got people from all over the world here Malaysia India Casablanca that's a movie Saudi Arabia Canada Nantucket Nantucket that's what's up right there I think that's like a jam as well Nantucket jam sounds like it would be though right like a peach jam alright so let me send this image test I remember I called a test three Oh JPEG I'm sending it to the hold on it's not called test three it's called test two JPEG will test to JPEG and okay hold on give me a second give me a second here oh no yes I see it's because it's not a JPEG right it's got to be it's got a beat be a JPEG right now it's an it's a PNG duh image files we got Utah Utah what's up Utah okay so test3 now now let's do this again let's do this again ready ready so remember that how to learn mathematics fast image I just showed you guys now okay here we go here we go machine learning is happening inference inference on a trained neural style transfer model is happening as we speak right now and when it is done we will see that output okay so where is that output where is that output so you saw that right that did not exist before that is some machine learning right there I took my plain little image and I turned it into this style transferred image okay so that's that's the tutorial for today's video okay so I'm gonna show you guys how to do that in the cloud okay so you can build this code yourself I don't care where you are I don't care who you are you will build this code and you will in for command line for commands on command line you will send this to the cloud and create an API just like that that you can run inference on you could curl the image you can send a post request you can build a service that's how this goes okay so we're gonna start off with this okay so also before I start doing this I'm just gonna say ten names just to you know done y'all fitrah polar bear Luis Abhishek Aly David and Ben thank you guys for being here so let's get started okay so mmm-hmm for people who are watching this recording we want to I want to I want to satisfy both to live and my recorded audience so we have three steps to do this the step one is to build a model of course right we've got to build the model and once we build the model then we can train the model on Floyd Huff's I'm gonna talk about a service called Floyd hub it's not the best service okay I just want to say that it's not the best but it is the best for beginners just to get something quick and dirty up and running Floyd up is great alright if you want to start scaling really really scaling to you know hundreds of thousands of users then you want to move to AWS then you want to move to Google Cloud or even Azure and I'll have videos on each of those individual platforms coming out soon either next week or the week after that it's okay so the third step is to serve the model via an API so if you're not familiar with neural stall transfer here's how it goes right you have an image right it could be media key you and then you have a source image and that's the style that you want to transfer to your base image and when you do that boom you've got that you get this filter and this paper came out I think four years ago four years ago now and it was called neural style transfer it was by Gatti's that was the name of the researcher and there had been a lot of variations to that Irish to that original paper but really that original paper was the landmark it's the one that started it all okay we got someone saying make South America great again so we got some really hilarious people clean in this chat room okay so you guys are so hilarious by the way I forgot how funny you guys are okay so how are we gonna do this well it turns out that we all we need to do is use one neural network okay it's it's a convolutional neural network we're gonna build this thing in Python in tensorflow and that's all we really need so what happens when we have a convolutional neural network is what it is is it's a series of matrix operations it's a series of matrix operations we have our input data that is an image and an image is just a matrix right just numbers pixel values between 0 and 255 that say what color every pixel should be and we take that input matrix and we continuously apply operations to it multiplication right the dot product and in linear algebra terms we're just multiplying a matrix by the matrix by the matrices the weights of the network over and over and over and over again and we might add some other little tweaks to that but it's just a series of operations and you might be thinking well that's too hard I don't understand that loss let me stop you there it's not too hard first of all stop telling yourself that it's all about your belief can it's it's like athletes right so if you believe that you can become you know like whoa marine level jacked status you will do that if you believe that you can learn something then you will learn it you got to believe right so again I am NOT a PhD but do I feel very comfortable with this stuff enough to go live in front of a live audience on the internet with my name on the line yes I do so and why is that because I believed that I could learn it even though I'm not a PhD and guess what I did learn it so so take that as an example okay so right so convolutional networks they're just a series of matrix operations apply to input data until we get an output and the output is a prediction right and so right this is used for image classification but what the researchers of neural style transfer thought were they said you know what let's use those intermediary layers and and use those to trance to transfer the style of one image onto another let's not care about the outputs of the network right so just cut out that the the last part of the network and just tear about what's what's happening in the middle right here right so these colored images are actually just matrix values these are matrix values and they have numbers and through optimization through the learning process those Neuros numbers become better and better at classifying the image right through a training process which we don't actually care about right now what we care about are these learned layers because we can use those learned layers and apply what they've learned to our image right so that's what we're gonna focus on we're gonna use these layers to then create our style transferred image and so how do how does that work so here's how it works so there are three parts to this workflow right so we have a Content extractor we have a style extractor and then we have the combination of both like so we put them together so now I'm going to talk about math okay so get ready for this I'm gonna talk about math in a second so that this is the fun part right so so the first part is a content extractor right so we have our image and what we want so so there's different ways that we can think about this problem and so what the researchers did was they said okay we have an image and that image has some content in it that means it has a human you know objects inside of the image it's got a sunset it's got a mountain whatever it's got in terms of objects though that's the content and we want a way to preserve that content and and and model that content and represent that content mathematically so then we can we can later utilize that right so what they did was they used a very famous convolutional network called vgg 19 this was a convolutional net work that was trained on hundreds of thousands of images of all sorts of categories from boats to planes to dogs whatever okay so they took this pre-trained Network okay and this is what it looks like a convolutional network we have a convolution which is a matrix operation a pooling a pooling operation which is another operation you know I have a great I don't I don't want to repeat what I've repeated before if you want to know how a convolutional network in general works search convolutional network Siraj on youtube first link it's an amazing explanation if I do say so myself but so that's how it works but what what we care about or these are these layers okay so this is actually just one layer right so but what a convolutional network is is it's multiple layers so we care about this this layer you know a few of these layers right here and this is pooling so so if we were to visualize and by the way guys the code is going to be in the github in the video description the github code by the way so if so fall along with me here so if we were if we were to visualize what these layers have learned it's gonna look like this right so it's gonna be images I mean it's gonna be edges and then it's gonna get increasingly more abstract over time so edges will become eyes and eyes will become faces and faces will become humans as the layer the network progresses so what they did was they represented content as this formula get ready now you now you got me excited because now I'm talking about math by the way okay so mm-hmm-hmm you guys are so funny oh my god you guys had the most random stuff okay so content right so content how did they represent content like what is actually in the image in terms of objects so what they did was they passed both images through the network right so you have your base image what you want to stylize and you have the the style image right or the the image the orchid let's just call it artistic image and so they passed both through the network and so what they did was they found that they took a particular layer let's call it layer 2 layer 2 of 5 but it could be any layer really and it took layer 2 of 5 and that's that's a that's a matrix of values right is it to learn matrix of values and once we do that we can then that's gonna that's going to give us a scalar value okay that's gonna give us a scalar a single value and what we can do is we can find the difference between those values right so we pass in one image and it's gonna learn a filter it's gonna learn a filter for that well pass in another image it's gonna learn a filter for that we can basically subtract those two filters together and we can get the and then we can square that and we can do that for all of those images will sum them all up and then we're finding their difference and then we're squaring it okay so that's how that works and that's what the Sigma notation means for all of those images okay so we're also multiplying each of the represent representations by the value alpha that's right that's right the value alpha but we have those two features that's what I'm trying to say here we have two learned matrices we're finding the difference between them and that is our content loss so right so loss functions I've got such an amazing loss function video coming out this weekend I cannot wait to show you guys this but loss functions are all about measuring how bad our network is at making a prediction right so what we want is we want a loss function that's not going to measure how bad we are at classifying dogs we want a loss function that measures how bad we are at creating a stylized image and so if you think about it that's a very abstract at first thought like how good a filter is but by representing it mathematically this way we have found a way to Siraj 20/20 I mean guys one thing at a time right I got to take over this YouTube game first and then we'll let's talk about that not I'm not saying anything anyway you guys are so funny oh my god I need to like close this chat window because okay anyway no I like it though I can do both so that's content right we have a look we have the content and now we're the next thing we're gonna do is we're gonna strike the style so so content is just think of it as a single value right so all of this comes down to a single value that represents how how how much content has been transferred to our base image it's like how different our base image is from our content image and what we want to do is we want to minimize that difference and when we minimize that difference our base image will become more and more and more like that style image in terms of content but content is not enough there's also style right so for style for style we're going to do the same thing except we don't want we don't care about the content we care about the the texture of the image right the colors the what what is where right so if we think about like a filter in general there's not really images that are being overlaid there's just like these textures and colors right so there's this kind of gradient not not in the sense of the math word it can be very careful about worse here but um how did they extract the style well what one thing to think would be well let's just directly do the same thing right so input the style image and find the difference right between them but what they instead what they did was they said let's compute what's called a gram matrix and what a gram matrix is it is it is the it is the transpose of a matrix right so a gram matrix is the multiple if you take a matrix and you multiply it by the transpose you flip it of itself that's the gram matrix and so they took those gram matrices for both the base image and the style image the artistic image and they found the difference between them so that's what this is so this calculates the gram matrix then once we do that we can find the difference between them and we multiply by this out by this value beta so there are two like threshold not threshold but static values here we have alpha which is for the content we have beta which is for the style and we can tweak those to see how stylized our end result will be so that gives us our loss for for saying how stylized we want our image to be so what we want to do is we want to combine both of those and the way that we combine both of those is by creating this final equation here so we just we just add them up that's it we add the content loss and we add the style loss and that's our total loss variation loss you can call it so our loss function consists of other loss functions and the reason we added it is because we just want a single we want a single end-to-end optimization approach or we can just optimize the entire thing we compute the error using this loss function and then we use the error to compute the partial derivative with respect to the weights of our network okay and if you want to understand how back propagation works again search back propagation Siraj on YouTube and like a million links will pop up there as well literally anything ai related just search the name of what you're thinking and then the word Suraj and I promise you it's going to show up on YouTube 98% of the time okay so all right so and the way they optimized it was using l-bfgs and this this is a second-order optimization scheme and I've got a great video on that as well just search second-order optimization by Suraj on YouTube and we talked about how instead of just computing the derivative we compute the derivative of the derivative and in some cases it's called Newton's method and another in another case it's called l-bfgs but it is a gradient based optimization ski okay so we're going to build this now we want to make this into an API so let's get to the code shall we so let's let's just build a simple model here okay so what we first want to do is we want to import vgg right so vgg is that convolutional network we also want to implement we will also want to import tensorflow we've got people doing the hundred days of ML code challenge by the way amazing stuff hi I am so proud of the people of the Wizards who are who are doing that it's just it's just amazing mind-blowing so vgg is our network tensorflow is our tensorflow is my city no no tensorflow is our machine learning library and then we're gonna import transform we're gonna import transform to then transform our images as we as we do want to modify them yes I know coding train I love Dan we actually collaborated with Dan and I'll probably will collaborate with Dan again he's a great guy and I promoted him and he's promoted me and you know in a way we're kind of all like the Kardashians of machine learning shift made me three blue one brown this evening a carry carry Hwang is coming over he doesn't know that I'm live-streaming right now but he's coming in a few hours so I mean we all we're all like you know cross promoting each other and you know what I really want to do is I want to make a curriculum that involves all of them and you know we all have our you know we all have our distribution channels or YouTube channels but we're all pushing content towards the same curriculum can you imagine the anime crossover level you know if I'm like sitting here building convolutional Network Daniel Schiffman comes in and he is like adding in his like p5.js JavaScript stuff for like front-end stuff you know Kari comes in he's hilarious andrew traps comes in and he like explains things so beautifully Jibril's comes in of course and he like makes a game out of it drills is off is awesome as well we collabed as well how cool would that be we guys we need to make this happen I mean it's gonna happen so we'll see we'll see so one step at a time basically the Kardashians a visitor okay so let's let's get to this before we run out of time here so from utils import get image okay so now what we really want is a couple of those layers so we're going to specify what those layers are going to be so let's just call these layers by their names which they were named by the vgg authors and once we do that riilu to renew three really foetida really really really by the way is stands for rectified linear units which is the type of activation function that diminishes the the the vanishing gradient problem we live for okay so we've got four layers here that we're gonna bring it and think about when it comes to style and then when it comes to content we just have one layer we're gonna call riilu for two okay and so now that we have that let's let's now say let's optimize so in the optimize function here we want to perform optimization okay so what I'm gonna do is I'm gonna say inside of our TF graph so we want we want to initialize our graph and we want to take vgg an existing network and extract those features from it right so so how do we do that how do we actually extract features from from vgg and so I'm going to show you how we do that as default get the device there's going to be our device right has device yes YouTube is my main job it is my main job that's the way I like it okay okay so with that we're going to compute this style image first and that's going to be TF that placeholder so we want a placeholder for that style image right so that initial let me make this bigger let me make this bigger a bigger bigger bigger is always better not always not always okay style image is going to be we're going to use a placeholder so what the placeholder is and then in the tensor flow computation graph is it is the gateway into the network right so this is where data flows in and so that's why we're creating a placeholder here because we want a way to input this style image right so I'm gonna do that for the style image and we want to do that for the the base image okay so our placeholder is going to be of type float 32 it's a very simple image we have a predefined shape that we want to give it my sound and image are not in sync okay well that we got to fix that really okay hold on let me go check that out really my sound and image are not in sync seriously oh my god see this is what happens when I livestream alone I'm like trying to set up this this whole thing okay okay so what I want to do actually is just okay so next time I'm gonna be better about this but let's find that since citizens thing for you right great let's look at this code so let me just show you this code guys okay so we have our style image okay so and then we have our base image here we have our network vgg net we feed both in as parameters that gives us our network okay so once we have that we take those layers and we compute the gram matrices of each of those layers okay we compute the gram matrices of each of those layers using them and then using the matrix multiply featured right remember the gram matrix is the transpose of a matrix multiplied by itself which is exactly what this line is this is the gram matrix we are feeding all of those we are feeding all and I'll wrap at the end by the way do not leave and I will wrap if you leave you don't get to hear the rap I'll probably make a fool of myself but that's just how it goes right whatever it takes right yeah solve AI or die trying so gram matrix we're storing all of those inside of this array style features okay so that that was us computing the style loss in the form of gram matrices for all of those layers that we defined beforehand right so that was that and so once we do that then then and only then we can say okay inside of our TF graph let's compute the content preacher so we already did this for style and now we're doing the same thing for content so we've already defined a content layer right and remember there was no gram matrix involved in the content so you don't see that here we are directly pulling the learned value for the content just like this whereas for the gram matrix we had to compute that right using TF or NP numpy z-- matrix multiply function and so once we do that then we can combine both of them so this right here is just uh this is like a this is like an exception check right if it's if it's slow then set the image back to what it was before because it's taking too long so we can ignore that part so now for our content loss we already found the value from that specific layer and this is us computing it by multiplying it by the weight and and so if you look at this you're thinking wait a second what you're thinking like wait a second where is that where is that value right so here it is this is this is exactly what we're doing see so this equation is what we are looking at right here where is it where is it it is it is the difference between both of those features divided by the size of the content multiplied by the weight and that gives us the content loss okay so now that we have all of those style layers now that we have those layers we can compute the style loss so we had the style layer values and now we're competing the style loss values until it comes out to this so what is happening here let's talk about we have our the size it's the height by the width by the number of filters so it's a three dimensional tensor size we have our features so what we're doing is we're reshaping what we just computed no not this one this is for each of the layers so these are the features for the layers we have our graham matrices that's our style graham matrices we append to the losses the difference between those graham matrices and that gives us our style losses and lastly we use we use that value multiplied by the weight and that is our style loss now the word variation here just means this total loss so the total variation loss and so in order to compute the total variation loss we add both the style loss and the content loss together and that's going to give us our total loss okay so this right here is this pre-processing step that's not it's not really necessary but they went ahead and did it anyway it's just kind of a it's a nice to have it's a gotcha that's the thing about a lot of this research code I mean this is research code like it's kind of it's kind of messy it's you know it's not good one of the most impactful things you can do by the way if you're interested in AI which I love you if you are which you are so I love you if you're watching this one of the most impactful things you can do is creating clean well-documented readable code all right for the ML community as a whole right this is this is what beginners need better code and care offs would be a great wrapper for this and there is a Kaos wrapper for this but right so this is the training loop train train train etc okay so now let's get to the the part that I've been talking about the API part right so how did I do this so if you look at what I've done here if you look at what I've done here I have made an API using Floyd Hub so let's let's talk about Floyd out for a second okay so Floyd hub is not the best oh my god Floyd all right Floyd humpy is not necessarily the best GPU provider on the web but I'll tell you what it is it's really easy for beginners to get started okay so I've got a couple of jobs here that I've been running and so when it comes down to it for Floyd hub you literally can say here is my here is my repository so I've got a repository here and right so let me open this here's my repository let me let me look distant sublime sublime okay evaluate you've got to have this so in this Floyd requirements that txt you just add all of your dependency is just like list them and it's gonna install all of those dependencies in your virtual environment in the cloud for you so this could be tensor flow we care ostomy whatever you want and it will install that for you in your virtual environment so what you can do is once you once you find your repository you can run the command Floyd in it right so Floyd in it fast style transfer and what that does is it initializes a new project in your current directory and then you can serve well first of all you can even train that model so this is a pre trained model that we can serve directly from our hold on hold on we can run that directly from our command line like that and so what this is now doing is it's taking our code it's putting it on the web and so I've reached your maximum er of jobs I've been running a lot of jobs on this thing and so once we do that then we can simply look at some people okay so then we can simply just run a simple API call using this line of code just like this and then it's going to return back whatever that whatever that output image is like I should like I showed you before okay that is super simple and if we go to the the code that I have for you in the github description if we go to that check it out amazing instructions for you here's how to train it or here's how to train it on floyd up one line of code and your training your model on this cloud service here's how to evaluate it right you trained it okay here's here's how you run inference okay you can do the same thing for video and really this applies to any kind of model and how do you even do this well you've got to make a floyd hub account right let me do this with you okay let's do this together let's make a FOID hub account sign up free okay Suraj Suraj Lilla okay no credit cards needed here and no they didn't sponsor this they did not sponsor this video I just personally like it okay just like that and so I don't know how it's already oh it's already got projects for me so we can just look at those if we wanted to and so inside this work it's even got the instructions like how do we do it well we just initialize it here and so once we initialize it locally we can add some code here and then we can run it on a GPU so Floyd Run GPU and then it's going to run on the GPU in the cloud so I want to say and so once we do that we can serve it as well from from that API we can we can create an API around this so you might be thinking well there must be machine learning API is out there so clarify is one example right so clarify does have does have some models that are trained for you pre training the cloud and they're pretty good but they're doing way too much right so if someone can clearly pick one of these and do them really well using one of the newer models I'm talking about generative adversarial networks I'm talking about I'm talking about variational autoencoders you know one of the more advanced models that you know people don't know about so much and you know clarify is a great example of a of a of a group of people who created a really sustainable business around just AI so think about it just having one single API call as a service you can do that you can make so much money you can make so much money doing that and you're just just yesterday I had a farmer in Belgium who who emailed me a picture of his crop and he was asking if I could help him with classifying that's right so there's just so much potential out there to create really impactful businesses for the world clarifies one example I just want to show you that really quickly okay so that's it for this live stream let me answer some questions at the at the end I'm gonna wrap so guys go ahead and ask some questions and then once you do that I will I will I will end this livestream after I hands her your questions okay so while I look for an instrumental to play rap instrumental any questions guys what are the questions here how to become confident with large code bases great question so the best thing to do is to contact the author of the code base right so they are the architect and they know what parts are best for beginners so don't just try to dive in yourself contact the author look at the documentation ideally ideally it's got documentation but if not contact the author and they will help you they will guide you don't just try to do it yourself you mean you can but it would take longer so you want to go to the fastest way to do this what bit are you thinking of making next I want videos on GPUs specifically GPU providers should I go for AI or ml which one before guys AI is the biggest bubble machine running ins inside and deep onions inside machine learning is a subset of AI do you think JavaScript will overtake Python soon in ml I don't think it's gonna overtake it but I think it will catch up for sure it is catching up is machine learning very impro very important of course it is Manish thank you and two more two more questions is Floyd costly when used to train and test yes so so that's why I'm saying it's only good for beginners right because once you want to start scaling then it's gonna get expensive and I think when it comes to scaling you want them the cheapest solution that is also the most dependable and for that I would choose either AWS cloud or as your but to start off with you know there's there's a very small learning curve for for Floyd Hub that's why you should start with that how to start with deep learning watch my intro to deep learning playlist on YouTube Jerris law is because that's my signature and William last question what ceiling would you recommend for tensorflow I would recommend Python okay so now I'm gonna rap about some topics would someone just say a topic one word go Friday everybody it's Friday this is kind of a hard be there we go all right so I'm about to rap that's a freestyle someone gonna say a topic right now here we go what is it where's the where's the topic guys where's the topic I think these guys are GN ends that's not really like a a topic that's them all a yo I'll read some Isaac asks a mob don't try to look at me don't try to piss me off I try to read sci-fi in my books and my movies in my looks try to dress that way I got hair is so wavy okay I level up I level down an LP man I'm rapping like a clown you guys are laughing in the chat but it's okay cuz I got Bitcoin anyway it's all good hey hey hey all right oh I actually ashleighb woodford all right guys so thanks for tuning in to this live stream I've got more live streams coming up for you I love you guys thank you for watching I've got so much AI machine learning content coming up so if you haven't subscribed hit the subscribe button and yeah thanks guys for watching okay