Python Machine Learning Tutorial #9 - SVM P.2 - How Support Vector Machines Work

Key Takeaways

hey guys and welcome back to another machine learning tutorial with python so in today's video we're going to be talking about svm and how support Vector machines work so spvm obviously stands for support Vector machines and these are going to be used at least in our purposes for classification although we can use them for something known as regression which I talked about in the first few videos uh but I'm not going to be explaining that here so essentially uh really basically how svms work is they attempt to create something known as a hyperplane and a hyper plane is something that can divide your data using uh hopefully something that's straight so like a plane is straight a uh what do you call a line is straight you can have like four-dimensional stuff so anything that's straight right so like a linear way to divide your data um so essentially a hyper plane for the data set that I have up here could look something like this right and you can see that it's dividing our data points uh so green is on this side and red is on this side right now how do we create this hyperplane what are like the for it like what do we need to do to you to do this well the requirements for a hyperplane essentially are that you find or the hyperplane sorry is uh the same distance from the two closest points of each opposite class that's confusing but I'll I'll draw it out and essentially the closest point to the line now is this red point and the closest point from the green side is actually have to draw a new one to make sure that this works well let's do one like here is this screen point that I just drew okay so these are the two closest points to the line from either class this red point is the closest and this green point is the closest okay and the distance between this red point in the line and this green point in the line are the exact same so imagine if we start tilting the line this way then obviously this red Point's going to be closer and this green Point's going to be further away right so just just you can picture that okay so these two points are the closest points to the line and the distance between them and the line is the same that's how we generate a hyper plane so with that information we actually now know that we can generate an infinite amount of hyperplanes for this data set another hyper plane could look something like this where the two closest points are this green point and this red point and their distance to the line is the exact same okay we can draw all kinds of different hyperplanes uh but which ones are the best right so for example I could probably I'm going to try to do one like this uh we could say that like this point and this point are the two closest points at the line and their distance is the same from the line and that again is another hyper plane now how do we pick which hyperplane to generate what is the best one what is going to give us the best result so essentially since I didn't really talk about it um when anything on this side of the hyper plane is going to be green right for when we're predicting and anything on this side is going to be red that's essentially how we're going to use this hyper plane but how do we pick the best one well this one would probably be the best hyper plane to for our data set and the reason is because the two closest points to the line are actually um the furthest possible points away like the distance between them and the line so let's say like this one and maybe uh let's say this one I know this one's a bit closer but just imagine the same distance is the largest distance that we can generate no matter what other points we pick or where we draw this line there's no way that we can find a distance uh d dened by D that is greater than this now why would we want this distance to be so so big like why would we want this distance to be as large as possible well I'm going to just draw two lines here quickly uh let's say like this okay and the distance uh denoted by D and between these two points or like in between all this right so where there's no other points is known as our margin okay and we obviously we want to maximize this margin now why why is that the case why do we want to do that well let's think about it for for a second all right so if I remove all this okay so let's erase I don't want to click that let's erase this line and let's draw another line and let's compare that line that I just drew like that one to the one I'm about to to do okay so let's say we have a point that we're trying to predict its class and the point is right here well what point do you think as like a human this should be you think it should be green or you think it should be red I would hope most of you would say it's green but if we use a uh what do you call it a hyper plane like this it's going to classify this as red because it's on the the left side of the hyperplane so why would we want it to be as large as possible because ones like this if we had that that same point that's somewhere around here it's going to be classified more correctly as green because the larger the distance and the larger the margin the more we can separate the two classes uh and do more accurate predictions so essentially that is the basis behind support vectors machines we're trying to find something like a line or a plane that we can separate our data points by we want to find the largest distance between our what's known as support vectors which is going to be this red one and this green one uh and then we'll generate that hyper plane and we'll use it to predict data based on what side of the hyperplane is on okay that's the most basic understanding now let's talk about the problems that we run into when we do this so right now we have data that is in my opinion quite nice right we look at this and we can tell really quickly um that we're going to have a line like this that's what it's going to look like right but what if we change this up a bit and we go to look at data that is not like this it doesn't linearly kind of correlate like this is really nice data that is not common to find especially in the real world when you're dealing with well real data and stuff that is not going to look exactly like that so let's do some red points I'm not going to do as many because I'm going to do something with these in a second and maybe they look like this okay that's our red points and maybe we have some green points and our green points look something like this well if I asked you can you draw me a hyper plane for this you would say what a hyper plane where where does the hyper plane go does it go like that does it go like that uh does it go like that we honestly have no idea where to draw the hyper plane and even if we can come up with one is it going to be accurate because on here we have no idea uh if it's actually going to be red or green or if that's going to be a correct prediction we're pretty much just guessing at that point so what do we do to fix this problem well we use something called kernels now this is in two Dimensions right so we have X2 and X x one as our features and then red it's green is denoting what the class is going to be okay so what we want to do is we actually want to turn this data into a form where we can draw a hyper plane or like an accurate hyper plane through it that divides our training data so what we're going to do is we're actually going to bring this whole thing up so this whole two-dimensional data we're going to bring this into three dimensions and the way we do that is using something called a kernel now a kernel sounds confusing is a central just a function so you can have have F okay denoting a function that takes inputs in this case is going to have X1 and X2 and spits out an output which is going to be X3 which is a higher Dimension okay I'm going to show this in a drawing in just one second so it's going to take our two features X1 and X2 for every one of our data points and based on that input it's going to give us an output of x3 which is going to represent like the third coordinate for our points cuz we're going to go up a dimension we're going to go to three dimension from two Dimensions so if right now we have X1 and X2 for every one of our points we're going to be adding an X3 and then plotting it on a three-dimensional graph so let's erase this and let's do just that so threedimensional graphs we're going to have our three axes so we'll keep this as X2 this as X1 and we'll draw one out like here which is going to be our X3 okay now I'm just going to draw some red points and some green points so do some red ones and we'll do some green ones now imagine that these are the same points that we had in two Dimensions right so I'll draw a small graph here just so you remember what it looked like in two Dimensions right we had like red points and Green points kind of all just scattered around in the middle so I know this is not the same but essentially when we go from this to this now we look here and well we can draw a hyperplane look at this by generating that third Point we've successfully been able to divide our green points and our red points because of this third dimension right so if you imagine now you squish this back to two dimensions then we're going to come back to looking something like this okay so they're right if we remove that X3 coordinate but now that we have this and we've used this kernel we can actually generate a hyperplane looking something like this that divides our data and this works the exact same as we did for hyperplanes in two-dimensional space where let's say like the distance here to this red point is the same as the distance here to this green point and they're the two closest points to our plane so I know this might be a bit confusing because I'm trying to explain a pretty complex topic like really simply and sometimes that even makes it more confusing but essentially if we have data that looks like this right in two dimensions and we can't classify it in two Dimensions we can't draw a hyper plane we can add a third dimension to it in hopes of getting data that looks like this right in hopes of getting a graph that looks like this now obviously this doesn't always work right so when we use this function we use that kernel function we get X1 and we get x2 uh and that brings us to X3 maybe our data still is is impossible to classify right maybe we can't do a hyper plane through it there's just there's no way um and we still get points over here and here and they're all mixed up in that case we would repeat the process and add a fourth dimension to our data now I want to just talk about kernels a little bit more because I feel like some people might have still have no idea what they are essentially it's just a function that takes our features so in two Dimensions we so here we had X2 and we had X1 and we fed that into the function it did something with it and it returned to us a third dimension and then with that third dimension right we can now plot our data and by using that third dimension we've spread our data out we went from like only being on this plane right in two dimensions and we've just brought it up and down so that we can divide it by hyper plane so I hope that makes sense all kernel is is a function there's all different kinds of kernels um you typically don't create your own you just use an existing kernel and an example of a kernel could be something like X1 2 + X2 2 this is a kernel this is perfectly fine and that would result in the value X3 right so say this is our point and in two space its coordinates is like one and two okay if we want to turn this into a third three coordinate or 3D object or point we take X1 2 plus X2 2 so we get 1+ 4 because 2^ squ right and that gives us the value five and now we have as our uh let's actually let's draw it here I guess we have as our coordinate point one 2 and five and that's what we plot here and we hope that by applying this kernel it's going to make all of our red points kind of either shoot up and all of our green points go down or go to the side or something that we can draw an accurate hyper plane for so that is essentially how a support Vector Machine Works um I'm not going to go any further than this because if I keep going it's just going to get way more confusing and way more math there's a ton of stuff behind how we generate this and how we do this this is a super highle understanding um Now you kind of know enough to be able to implement this and so we can kind of tune some parameters and whatnot uh but yeah that is essentially uh all there is to svm there's one more thing which is like a soft um what do you call it a soft margin and a hard margin which I'll cover really quickly but there's the reason I left it to the end is not that important um essentially like if we have our two-dimensional data like this I'll do really quick because I don't want to waste any more time uh but let's see we have some pink points or purple points all right uh and we have some red points like this okay well we can again we can draw our hyperplane and right now you might say Okay so the hyper plane is going to have to look something like that but that might not be a great hyper plane to draw and what we a hyper plane we might want to draw instead is something that looks like this now you're saying well we we can't do this hyper plane and you're you're correct right now based on what I've told you because well this point is uh what do you call it it's the closest point to the line and uh like we're not using that because we're going to use this and this as our support vectors for this this hyper plane and this red Point you're like well that can't be there that that's incorrect well you're you're correct right now but this is where we introduce something called a soft margin so if we're saying that this is our uh what do you call it support vector and this is our support Vector we'll draw these and we'll say that we're going to use something called a soft margin and we're going to allow for a few points like this to exist in between the margin we're going to say you know what by allowing these kind of outlier points to exist we're actually going to get a better classifier with our uh what do you call it hyper plane here and that's fine so that's another parameter that we can we can tweak we can give something called a soft margin allowing a few points or however many we say to exist inside this margin and not affect the hyper plane so that's another parameter if you using a hard margin essentially all that means is what I taught before you can't have any points like this like this red this is not a valid uh hyperplane but with the soft margin this is fine so there it is that's it for support Vector machines please feel free to uh contact me on Discord ask me any questions leave some questions in the comments I try to respond to everyone if you guys enjoyed the video please make sure you leave a like and in the next video we're going to be getting into implementing this algorithm [Music]