PyTorch Tutorial : Backpropagation by auto-differentiation

Want to learn more? Take the full course at https://learn.datacamp.com/courses/introduction-to-deep-learning-with-pytorch at your own pace. More than a video, you'll learn hands-on coding & quickly apply skills to your daily work. --- In this lesson, we are going to introduce the main algorithm in neural networks, the so-called backpropagation algorithm, and see how we can use it on PyTorch. This lesson is a bit more theoretical than most of the lessons in the course, but there is no need to get scared of that. Derivatives are one of the central concepts in calculus. In layman's terms, the derivatives represent the rate of change in a function, so where the function is rapidly changing, the absolute value of derivatives is high, while when the function is not changing, the derivatives are close to 0. They could also be interpreted as describing the steepness of a function. For example, in the function here, points A and C have large derivatives, the line is steep in these positions, while point B has a very small derivative. If you haven't ever heard about derivatives, I would highly recommend taking a look at them, on Khan Academy for example. Some important rules of derivatives are the addition and multiplication rule. The addition (or sum) rule says that for two functions f and g, the derivative of their sum is the sum of their individual derivatives. On the other hand, the multiplication rule says that the derivative of their product, is f times derivative of g plus g times derivative of f. Derivative of a number times a function, is the number, for example, the derivative of 3x is 3. The derivative of a number itself is always 0. The derivative of something with respect to itself is always 1. Another important rule is chain rule which deals with the composition of functions. In the example in the slide, the derivative of f(g(x)) is derivative of f under function g(x) times derivative of g(x). A closely related term with derivatives is the gradient. The gr