Evolution of back propagation: Who invented it?
Let's understand the evolution of back propagation. Who created it?
Last video on multiplayer perceptron: https://youtu.be/poNXEd3jCwI
RESOURCES
[1 ๐] 1847, Cauchy invented gradient descent. Here is the English rendition of this work with comments: http://www.kurims.kyoto-u.ac.jp/EMIS/journals/DMJDMV/vol-ismp/40_lemarechal-claude.pdf
[2 ๐] Hinton explaining the mechanics of back propagation introduced by Rumelhart, Williams and himself: https://www.youtube.com/watch?v=VCT1N0EsGj0
[3 ๐] Proof that gradient vector points in the direction of steepest ascent: https://sootlasten.github.io/2017/gradient-steepest-ascent/
[4 ๐] Online 3D plotter: https://c3d.libretexts.org/CalcPlot3D/index.html
[5 ๐] 1986 paper that is the most famous implementation of back propagation: https://www.semanticscholar.org/paper/Learning-representations-by-back-propagating-errors-Rumelhart-Hinton/052b1d8ce63b07fec3de9dbb583772d860b7c769
[6 ๐] Henry J Kellyโs 1960 paper on determining the Optimal flight path and hence developing their own version of back propagation: https://gwern.net/doc/statistics/decision/1960-kelley.pdf
[7 ๐] Backstory of back propagation: https://yuxi-liu-wired.github.io/essays/posts/backstory-of-backpropagation
[8 ๐] Tourist guide to differential equations by 3blue1brown: https://www.youtube.com/watch?v=p_di4Zn4wz4
[9 ๐] Khan Academy on derivatives, directional derivatives: https://www.khanacademy.org/math/multivariable-calculus/multivariable-derivatives/gradient-and-directional-derivatives/v/gradient
[10 ๐] A nice tutorial on reverse mode automatic differentiation: https://rufflewind.com/2016-12-30/reverse-mode-automatic-differentiation
[11 ๐ ] Everything you need to know about control theory. Amazing resource: https://youtu.be/lBC1nEq0_nk?si=-Jkh7PZl0bjY6LGE
[12 ๐] Thesis by Paul Werbos in 1974 (one of the first successful implementations of a back propagation learning procedure on a multi-layered perceptron): https://gwern.net/doc/ai/nn/1974-werbos.pdf
Watch on YouTube โ
(saves to browser)
Sign in to unlock AI tutor explanation ยท โก30
Playlist
Uploads from CodeEmporium ยท CodeEmporium ยท 0 of 60
โ Previous
Next โ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
Linear Regression and Multiple Regression
CodeEmporium
Logistic Regression - THE MATH YOU SHOULD KNOW!
CodeEmporium
Generative Adversarial Networks - FUTURISTIC & FUN AI !
CodeEmporium
Deep Learning on the Cloud - GPU TO LEARN FASTER
CodeEmporium
Deep Mind's AlphaGo Zero - EXPLAINED
CodeEmporium
Mask Region based Convolution Neural Networks - EXPLAINED!
CodeEmporium
Attention in Neural Networks
CodeEmporium
Depthwise Separable Convolution - A FASTER CONVOLUTION!
CodeEmporium
One Neural network learns EVERYTHING ?!
CodeEmporium
Neural Voice Cloning
CodeEmporium
AI creates Image Classifiersโฆby DRAWING?
CodeEmporium
Unpaired Image-Image Translation using CycleGANs
CodeEmporium
K-Means Clustering - EXPLAINED!
CodeEmporium
Random Forest Classification
CodeEmporium
Data Science in Finance
CodeEmporium
Hypothesis testing with Applications in Data Science
CodeEmporium
A/B Testing - Simply Explained
CodeEmporium
The Kernel Trick - THE MATH YOU SHOULD KNOW!
CodeEmporium
Support Vector Machines - THE MATH YOU SHOULD KNOW
CodeEmporium
Principal Component Analysis (PCA) - THE MATH YOU SHOULD KNOW!
CodeEmporium
History of Calculus - Animated
CodeEmporium
Curiosity in AI
CodeEmporium
DropBlock - A BETTER DROPOUT for Neural Networks
CodeEmporium
Autoencoders - EXPLAINED
CodeEmporium
Recurrent Neural Networks - EXPLAINED!
CodeEmporium
LSTM Networks - EXPLAINED!
CodeEmporium
Building an Image Captioner with Neural Networks
CodeEmporium
10 Machine Learning Questions - ANSWERED!
CodeEmporium
How do neural networks work?
CodeEmporium
Evolution of Face Generation | Evolution of GANs
CodeEmporium
How does Google Translate's AI work?
CodeEmporium
How to keep up with AI research?
CodeEmporium
How does YouTube recommend videos? - AI EXPLAINED!
CodeEmporium
Variational Autoencoders - EXPLAINED!
CodeEmporium
Logistic Regression - VISUALIZED!
CodeEmporium
Gradient Descent - THE MATH YOU SHOULD KNOW
CodeEmporium
Boosting - EXPLAINED!
CodeEmporium
Transformer Neural Networks - EXPLAINED! (Attention is all you need)
CodeEmporium
Loss Functions - EXPLAINED!
CodeEmporium
Optimizers - EXPLAINED!
CodeEmporium
NLP with Neural Networks & Transformers
CodeEmporium
Batch Normalization - EXPLAINED!
CodeEmporium
Activation Functions - EXPLAINED!
CodeEmporium
Data Scientist Answers Interview Questions
CodeEmporium
Why use GPU with Neural Networks?
CodeEmporium
How do GPUs speed up Neural Network training?
CodeEmporium
BERT Neural Network - EXPLAINED!
CodeEmporium
ConvNets Scaled Efficiently
CodeEmporium
Transformer Neural Net makes music! (JukeboxAI)
CodeEmporium
What do filters of Convolution Neural Network learn?
CodeEmporium
We're hosting a Machine Learning Conference!
CodeEmporium
MLconfEU 2020: Machine Learning Conference for Software Engineers
CodeEmporium
Are Neural Networks Intelligent?
CodeEmporium
Time Series Forecasting with Machine Learning
CodeEmporium
Few Shot Learning - EXPLAINED!
CodeEmporium
How does a Data Scientist Fight FRAUD?
CodeEmporium
How would a Data Scientist analyze Customer Churn?
CodeEmporium
Expectations with Machine Learning
CodeEmporium
Why Logistic Regression DOESN'T return probabilities?!
CodeEmporium
How you SHOULD code Machine Learning
CodeEmporium
More on: ML Maths Basics
View skill โRelated AI Lessons
โก
โก
โก
โก
The GenAI Honeymoon is Over: The Brutal Realities of Production AI
Medium ยท Data Science
Quantization From First Principles: Build Your Own INT8 Inference Engine
Medium ยท Machine Learning
Quantization From First Principles: Build Your Own INT8 Inference Engine
Medium ยท Data Science
Quantization From First Principles: Build Your Own INT8 Inference Engine
Medium ยท Python
๐
Tutor Explanation
DeepCamp AI