The Function That Changed Everything

Key Takeaways

[Music] I dare you to think of anything moving faster than deep learning but what if I tell you that behind all of it the complexity the hardware the data behind all of that there is a simple piece that took us years to find let me tell you the story about the unreasonable effectiveness of the function that made deep learning possible the thinking machine it's 1958 and the New York time publishes an article about a device that will be able to walk talk see and write then just interviewed the scientist that's about to change the world his name is Frank rosenblatt an American psychologist who published this report one year before in 1957. here rosenblatt proposes the construction of what many consider the first predecessor of the neural networks we use today the perceptron scientists built a few working perceptrons as these artificial brains were called April 1957. this is a receipt from the lab acknowledging the report and here is what it says about the project designing fabricating and evaluating an electronic brain model we've been trying to figure out neural networks for a long time fast forward 50 years mid to late 2000s and we still couldn't train a neural network with more than a couple of hidden layers but to understand what's missing we first need to talk about something we call activation functions regardless of how big neural networks are by default they can only solve linear problems unfortunately most things that matter are more complex than that here is one example imagine you have two classes orange and blue and you want to draw a function that separates them a neural network cannot solve this problem unless we use non-linear activation functions number of layers number of neurons none of that matters here is what Wikipedia says about activation functions only non-linear activation functions allow such networks to compute non-trivial problems we need these activations to create some sort of bump a disturbance that will allow networks to solve all sorts of problems like this one here but we've known about activation functions for a long time but that wasn't enough something was not right with these functions Sigma antennas were by far the two most popular activation functions back then look at the blue lines and don't worry about the red lines for now these functions checked every single box we needed to train neural networks well almost every single box to tackle more complex problems like image recognition text generation and audio translation we needed deeper network but as soon as we try with more than a few layers neural networks wouldn't work at all let me show you this this here is the tensorflow playground anytime I want to play with neural networks I come here because I don't have to write any code and it's really easy for me to try any of my wacky theories that is here this is configured to solve the same problem I showed you before I'm gonna add a few more hidden layers to this network I'm gonna change the activation to Sigma point and then I'm gonna click play I did this before and I let it run for a long time before I stopped it over 5 000 iterations that the network could not solve the problem but wait that's not necessarily an issue right maybe sigmoid is not good enough to solve this particular problem except I run the same experiment but instead of using six hidden layers I used just two and the network solved it this was the thing preventing deep learning from becoming something we couldn't train deeper networks because they wouldn't work my feeling is if you want to understand a really complicated device like a brain you should build one that was Joffrey hinston's voice he played a central role in making deep learning a reality but to appreciate what happened we first need to understand why this activation functions didn't work with deep networks time to look at the red lines now these are the derivatives of the functions we use these gradients during back propagation to update the weights of the network the deeper the network the more iterations we need I'm not gonna get too deep into the math here but if the gradients are smaller than one and you multiply a bunch of them the results will get smaller and smaller look at the gradients of these two functions the maximum possible value of sigmoids gradient is 0.25 that's really small and for 10 inches one but that only happens at this particular point the gradient is very small everywhere else and that right there is the problem the deeper the network the smaller the updates get until they're so small that the network dies we call this the vanishing gradient problem and that's why these functions did not work work we should look at biology and we should try and make systems that work roughly like the brain okay it's 2010 Emperor comes out paper that proposes an idea so simple that it looks ridiculous they show how a function they call Rectify linear unit solves the problem they had with the other activation functions here is what the paper says Rectify linear units preserve information about relative intensities as information travels through multiple layers of feature detectors this was the function that's it this was the crucial missing piece Nair and Hinton wrote the paper and although they made this function popular I found references to it from decades before like in this paper from 1975. Fukushima doesn't give this function a name but that's the rectified linear unit in the context of neural networks but here is the most surprising part this function that works so well doesn't even meet one of the most basic requirements of an activation function this function is not differentiable so how come the simplest function that doesn't even meet the requirements is the one that makes everything works let's look at the laptop I have a very simple notebook here to plot the Rectify linear unit for for short we call it relu if I run this cell we get the chart here is the plot of the relu function the x-axis is the input to the function while the y-axis is the output of it notice how relu returns 0 for any negative input and it doesn't touch positive values at the point where x equals zero we cannot compute the derivative of the function and that's a big problem in theory it turns out that in practice we can return a specific value for that particular point and everything works fine here I'm returning zero and if I plot the derivative is the red line on the chart one final thing for you to notice on this plot the gradient is either zero for any negative values or one for positive values that means that value doesn't suffer from the vanishing gradient issue the signal propagating through the network will never disappear that's a major win and the main reason this simple function works so well by the way the gradient is only one of relu's advantages the function is very simple so it's really fast to compute and it doesn't saturate like sigmoid and 10 H2 it just works exceptionally well and thanks to it we have deep learning today and by the way when I think about this story and how well really works despite the simplicity I can't help but wonder what else are we missing today