How Support Vector Machines work / How to open a black box

Key Takeaways

you can learn to drive a car by getting the keys having someone show you how the steering wheel the gas and the brakes work try it out in a quiet parking lot and then ease yourself onto the open road and before long you're able to get yourself to the grocery store or into a friend's house and you can get done the basics now contrast that with becoming a race car driver in that case you want to be able to get as much performance as possible out of your car and to do that you need to know how it works down to the nuts and bolts that's the process of looking under the hood or opening the black box if you want to get the most out of your tools then how things work really matters so to show you what I mean by this for most of us beginning machine learning or software engineers this is what a support vector machine looks like here is the black box you import scikit-learn cue up a simple example and run it and you're done here are the keys here's the steering wheel go but to get the most out of it you have to go deeper my goal in this talk is to walk you through the process of going deeper but more importantly to show how any of us can go deep on any subject that we want to get it become that we want to master so as I told support vector machines which were fairly new to me when I started this presentation the first step I took was to read the scikit-learn documentation this is what I came across it's not particularly helpful to me being new to support vector machines it's a nice concise summary of the basic principles of how they work but it doesn't explain it to someone who doesn't already know them well one thing that these scikit-learn documents did provide it was this great diagram showing here two different colors of points being separated by a line that line looks kind of like a road where those two lanes are as wide as they can possibly get before they start touching data points this was helpful my next step was to go find a nice tutorial and read it I found one that was highly recommended and started reading through it and I saw definitions and tables and equations and graphs and diagrams and plots and very soon got very overwhelmed and there were theorems for God's sake and I felt strongly that it was probably important stuff but it was not very accessible to someone who is new to the topic so I took a step back I set that aside and I went to YouTube and I pulled up some of the most popular videos on support vector machines and in the course of doing this saw multiple explanations of what they are what they're used for how to visualize them how the math underneath them works and the principles finally started to click this began becoming clear in my brain following this I went and found some more blog posts again from a variety of posters each with their own way of visualizing how they work of explaining the principles behind it and now the equations and the math and the definitions started to make a little sense enough so that I in the next step started to try to explain it to myself and the way I do this is by drawing pictures and so I started illustrating some of these concepts in a way that would make sense to me in my brain now that the ideas underneath support vector machines at this stage they're their crystals their nuggets but their scaffold but they're not fleshed out so to do this I find it very helpful to choose a toy example it's this is a bit of an art it's trial and error but finding an example that simple enough that you understand it completely and you know exactly how it should work but it's just complicated enough to illustrate the principle so for this one I settled on fruit so imagine that we have fruit it can either be small or large or yellow or purple now any small fruit is a plum if it's yellow it's not yet ripe but if it's purple it's ripe and good to eat any large fruit is a peach if it's yellow it's great to eat but if it's purple it's rotten you don't want to eat it so in this example we have in the world of peaches there's a size axis and the color axis and you can see the good things to eat are at the upper left and the lower right now in this example once you have this example if you can get it to where you can explain it to you know roughly like a twelve-year-old sixth grader which means that you use words and ideas that are common no jargon or if you do use a jargon term you explain it thoroughly so here's my attempt to do that for support vector machines so imagine you have peaches and they can be any color between yellow and purple and you would like to figure out which ones are good to eat you'd like to know in fact if you get a new peach based on its color whether you should eat it or not so what you do is you get a bunch of peaches and you grab one and you try it you get one that happens to be yellow you taste it it's good so you make a green circle and put it at the point that represents its color you grab another one that's pretty purple then you taste it and it's nasty it's rotten so you put a black X at the point that represents its color and you do this again for a few more peaches some yellow ones some purple ones some ones in between and before long you have a data set that looks like this the green circles all show good peaches the black X's show bad peaches now that you have all this data you would like to make a prediction based on the peaches color do you expect it to be good to eat support vector machine allow you to do this and what they do when you have two groups of data is they come and they put what looks like a road in between them there's a dotted center line and then two lanes and it tries to make that road as wide as it can possibly get until the outside of those two lanes bump up against your two data sets the center line is the divider between the two groups anything to the left of that will be assumed to be good to eat anything to the right of that will be assumed to be bad to eat and the lanes on either side are called margins for lack of a better term now imagine though that in this set of peaches this can be trickier what if you have some that don't really fall with the group you get some yellow peaches that just don't taste right or you get a purple Beach that for some reason tastes amazing now the data set there's nowhere you can draw a line that separates the green circles from the black X's but what you can do is still create a dividing line with its margins but any data point that is on the wrong side of its margin gets a penalty based on how far it is over and so you can move the position of this dividing line and the width of its margins to take that penalty and add that penalty in and still make that as small as you can possibly can so you can still use support vector machines in cases where your data isn't completely separate the fancy term for that is linearly separable just means you can't separate it with a line so it can handle non linearly separable data now let's look at a different case instead of just peaches we have peaches and plums the good ones we either want to eat yellow peaches or purple plums a yellow plum isn't ripe and a purple peach is rotten so we can do the same thing in this world we try a bunch of fruit of different sizes and different colors we find that the yellow peaches are delicious we find that the purple plums are delicious but the yellow plums are terrible and the black peaches the purple peaches are terrible so we end up with a data set that looks like this the challenge now is that if we try to draw a line to separate these out it's not just that a few data points are gonna be a little bit off it's there's a whole chunk of our data that we're missing we're not capturing it well so this data is obviously not linearly separable to help us visualize this I went and took it into Python and made a different visualization of it but it's the same thing green circles and black X's we'd like to try to separate them and in this case we can't with a line now support vector machines have an answer to this what you do is you imagine that all of these data points are not on a flat plane but they're on a sheet of rubber and you can pick that sheet of rubber up and you can stretch it and bend it and warp it however you want and you can probably visualize here that if you take that and bend it just right and slice it you can separate out the good to eat fruit from the bad to eat fruit and this is exactly how you would do it if you can now you with a single straight slice you can separate these things out nicely this trick of bending your sheet of paper or warping it it's called the kernel trick and it refers to how it's calculated but in practice all you need to know is you can take this space that your data is in this paper that it's on and twist it and bend it however you want you can take and hold down the middle and pull up all four edges or you can pull up all four edges and pull up the middle and leave a little ring low around the middle or low around the center or you can take it and pull it up and pull it down and make it like an egg great so that you can capture really irregularly spaced data and with a single slice you can separate it all out from each other so the kernel trick is really powerful and in fact there is no limit - what you can do with this space - bend it to illustrate another way that it's powerful let's consider a slightly different problem now we have fruit we don't care about the size but we have five different colors green peaches are unripe they're not yet ripe yellow peaches are ripe so they're good orange fruit is an unripe plum and a purple is a ripe plum and then a black fruit is rotten so the good ones to eat are the yellow peaches and the purple plums any other color is bad to eat now you can see that all of this data it's just on one line but there's no nice way to slice it to separate the green circles from the black X's no we can do and we can use the kernel trick we can take that line essentially and bend it however we want one way to do that is to just make a single bend in it like a smiley face and you can see that great you know we bent it but still those circles and X's are not laid out so that with a single cut we can separate them from each other now the cool part about the support vector machine kernel trick is that you can come back and bend your space again in a different direction so this represents a two dimensional kernel and now it's not too hard to see that with the right slice you can come in there with the plane and separate out those green circles from those black X's now if you look carefully you'll notice that this slice is actually not exactly the one you want it kind of misses but you can imagine where it would go to separate those out so here we did two different warping zwi took our line and we bent it one direction and then we bent it another direction because of the math it's a little bit mind-blowing but you can actually take whatever space your data is in and you can bend it in an infinite number of directions to make it so that you can slice it and separate out your two of data so that is pretty powerful stuff this is what support vector machines do they take and find the best slice that separates out two groups of data and if your space if your data is hard to separate you can warp and twist your space until you find a way to separate okay so that's the explanation now comes the most important part in addition to understanding the strengths of a method you have to understand its weaknesses if you're going to use it well and push it to its limits so with support vector machines issues include if you have data with lots of error so if you notice any time that we're finding a slice between two groups of data there location of that slice depends almost entirely on the very nearest data points the other ones it doesn't matter if they're close to the margin or miles away from the margin its those nearest data points that determine exactly where that margin is going to be and if each of those has a lot of error associated with it then that error gets a really loud vote more so than most of your data so that is an issue another way that it can break is if you choose the wrong kernel if you look back at the original data set where we were bending our paper like a sheet of rubber if we bent it the wrong way we would not be able to separate out our data sets we had to bend it in just the right way that's choosing the right kernel and the act of choosing the right kernel is an art and it's done by trial and error and after a while by experience and then finally large data sets can break support vector machines calculating the kernel some kernels especially can be very expensive take a lot of computing power and so if you're not dealing with just hundreds of data points billions then the amount of time to calculate those is prohibitive so with large data set you have to stick with linearly separable problems so you have to go back and hand engineer features that help your data get separated out help it so you can separate it with a straight straight line single cut so each of these requires a human in the loop to determine when it's a problem and to work around it this is important to know this means that support vector machines are powerful but to get the most out of them you need someone who has used them quite a bit and understands them well this doesn't mean that they can't be used but this is important to know when you're deciding what method to use on your problem now taking a step back that was a quick walk through support vector machines hopefully you understand it now a little bit better than you did before if you were new to it and now going into another project you know what you need to do to make good use of it a comment on the process we went through this together there was no formal education no coursework no textbook no professor no permission granted no special libraries nothing purchased this is all information that's out there this is something that you can do with any tool you want you can take and open the box and see what's inside you can lift up the hood and you can see what the pieces are and how they work together it's not an easy process and sometimes it's quite painful but it is something that you have at your disposal and it's something that in the cases of my key tools I have found it very worth my while so I encourage you when you have something that you think you might need to use heavily take some time open the box figure out how it works so you go from a grocery getter to a really high performing race car thank you