The spelled-out intro to neural networks and backpropagation: building micrograd

Andrej Karpathy · Beginner ·📐 ML Fundamentals ·3y ago

Key Takeaways

This video provides a step-by-step introduction to neural networks and backpropagation by building the micrograd library, assuming only basic knowledge of Python and high school calculus.

Original Description

This is the most step-by-step spelled-out explanation of backpropagation and training of neural networks. It only assumes basic knowledge of Python and a vague recollection of calculus from high school. Links: - micrograd on github: https://github.com/karpathy/micrograd - jupyter notebooks I built in this video: https://github.com/karpathy/nn-zero-to-hero/tree/master/lectures/micrograd - my website: https://karpathy.ai - my twitter: https://twitter.com/karpathy - "discussion forum": nvm, use youtube comments below for now :) - (new) Neural Networks: Zero to Hero series Discord channel: https://discord.gg/3zy8kqD9Cp , for people who'd like to chat more and go beyond youtube comments Exercises: you should now be able to complete the following google collab, good luck!: https://colab.research.google.com/drive/1FPTx1RXtBfc4MaTkf7viZZD4U2F9gtKN?usp=sharing Chapters: 00:00:00 intro 00:00:25 micrograd overview 00:08:08 derivative of a simple function with one input 00:14:12 derivative of a function with multiple inputs 00:19:09 starting the core Value object of micrograd and its visualization 00:32:10 manual backpropagation example #1: simple expression 00:51:10 preview of a single optimization step 00:52:52 manual backpropagation example #2: a neuron 01:09:02 implementing the backward function for each operation 01:17:32 implementing the backward function for a whole expression graph 01:22:28 fixing a backprop bug when one node is used multiple times 01:27:05 breaking up a tanh, exercising with more operations 01:39:31 doing the same thing but in PyTorch: comparison 01:43:55 building out a neural net library (multi-layer perceptron) in micrograd 01:51:04 creating a tiny dataset, writing the loss function 01:57:56 collecting all of the parameters of the neural net 02:01:12 doing gradient descent optimization manually, training the network 02:14:03 summary of what we learned, how to go towards modern neural nets 02:16:46 walkthrough of the full code of micrograd on githu
Watch on YouTube ↗ (saves to browser)
Sign in to unlock AI tutor explanation · ⚡30

Playlist

Uploads from Andrej Karpathy · Andrej Karpathy · 3 of 17

← Previous Next →

This video teaches the basics of neural networks and backpropagation by building the micrograd library, providing a hands-on introduction to machine learning fundamentals. It covers topics such as derivatives, optimization, and gradient descent, and provides a comparison with PyTorch.

Key Takeaways
  1. Build a simple neural network using micrograd
  2. Implement backpropagation manually
  3. Train a neural network using gradient descent
  4. Compare micrograd with PyTorch
  5. Build a multi-layer perceptron using micrograd
💡 Backpropagation is a key component of neural networks, and understanding how to implement it manually is crucial for building and training effective models.

Related Reads

📰
Edmonds-Karp Max Flow with Go
Learn to implement the Edmonds-Karp algorithm for max flow problems in Go and understand its application in network flow optimization
Medium · Programming
 📰
Stop Wasting Memory in C#: Why [] is Better Than {}
Learn why using [] is more memory-efficient than {} in C# and how to apply this knowledge to optimize your code
Medium · Programming
 📰
Run forty experiments against one eval set and you will find an improvement that is not there
Running multiple experiments can lead to false positives, learn how to avoid overfitting and ensure reliable results
Medium · Data Science
 📰
Evaluating an intent classifier: what I check beyond accuracy
Learn to evaluate an intent classifier beyond just accuracy for better performance in support agents
Dev.to · Kartik N V J K

Chapters (19)

intro
0:25 micrograd overview
8:08 derivative of a simple function with one input
14:12 derivative of a function with multiple inputs
19:09 starting the core Value object of micrograd and its visualization
32:10 manual backpropagation example #1: simple expression
51:10 preview of a single optimization step
52:52 manual backpropagation example #2: a neuron
1:09:02 implementing the backward function for each operation
1:17:32 implementing the backward function for a whole expression graph
1:22:28 fixing a backprop bug when one node is used multiple times
1:27:05 breaking up a tanh, exercising with more operations
1:39:31 doing the same thing but in PyTorch: comparison
1:43:55 building out a neural net library (multi-layer perceptron) in micrograd
1:51:04 creating a tiny dataset, writing the loss function
1:57:56 collecting all of the parameters of the neural net
2:01:12 doing gradient descent optimization manually, training the network
2:14:03 summary of what we learned, how to go towards modern neural nets
2:16:46 walkthrough of the full code of micrograd on githu
Up next
The Adam Optimizer is Just Momentum + RMSProp
DataMListic
Watch →