Active Dendrites avoid catastrophic forgetting - Interview with the Authors

Yannic Kilcher · Advanced ·📄 Research Papers Explained ·4y ago
#multitasklearning #biology #neuralnetworks This is an interview with the paper's authors: Abhiram Iyer, Karan Grewal, and Akash Velu! Paper Review Video: https://youtu.be/O_dJ31T01i8 Check out Zak's course on Graph Neural Networks (discount with this link): https://www.graphneuralnets.com/p/introduction-to-gnns?coupon_code=SUNGLASSES&affcode=999036_lzknae-d Catastrophic forgetting is a big problem in mutli-task and continual learning. Gradients of different objectives tend to conflict, and new tasks tend to override past knowledge. In biological neural networks, each neuron carries a compl…
Watch on YouTube ↗ (saves to browser)

Chapters (17)

Intro
0:55 Sponsor: GNN Course
2:30 How did the idea come to be?
7:05 What roles do the different parts of the method play?
8:50 What was missing in the paper review?
10:35 Are biological concepts viable if we still have backprop?
11:50 How many dendrites are necessary?
14:10 Why is there a plateau in the sparsity plot?
20:50 How does task difficulty play into the algorithm?
24:10 Why are there different setups in the experiments?
30:00 Is there a place for unsupervised pre-training?
32:50 How can we apply the online prototyping to more difficult tasks?
37:00 What did not work out during the project?
41:30 How do you debug a project like this?
47:10 How is this related to other architectures?
51:10 What other things from neuroscience are to be included?
55:50 Don't miss the awesome ending :)

Playlist

Uploads from Yannic Kilcher · Yannic Kilcher · 0 of 60

← Previous Next →
1 Yannic Kilcher Live Stream
Yannic Kilcher Live Stream
Yannic Kilcher
 2 Imagination-Augmented Agents for Deep Reinforcement Learning
Imagination-Augmented Agents for Deep Reinforcement Learning
Yannic Kilcher
 3 Learning model-based planning from scratch
Learning model-based planning from scratch
Yannic Kilcher
 4 Reinforcement Learning with Unsupervised Auxiliary Tasks
Reinforcement Learning with Unsupervised Auxiliary Tasks
Yannic Kilcher
 5 Attention Is All You Need
Attention Is All You Need
Yannic Kilcher
 6 git for research basics: fundamentals, commits, branches, merging
git for research basics: fundamentals, commits, branches, merging
Yannic Kilcher
 7 Curiosity-driven Exploration by Self-supervised Prediction
Curiosity-driven Exploration by Self-supervised Prediction
Yannic Kilcher
 8 World Models
World Models
Yannic Kilcher
 9 Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations
Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations
Yannic Kilcher
 10 Stochastic RNNs without Teacher-Forcing
Stochastic RNNs without Teacher-Forcing
Yannic Kilcher
 11 What’s in a name? The need to nip NIPS
What’s in a name? The need to nip NIPS
Yannic Kilcher
 12 BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Yannic Kilcher
 13 Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Yannic Kilcher
 14 GPT-2: Language Models are Unsupervised Multitask Learners
GPT-2: Language Models are Unsupervised Multitask Learners
Yannic Kilcher
 15 Neural Ordinary Differential Equations
Neural Ordinary Differential Equations
Yannic Kilcher
 16 The Odds are Odd: A Statistical Test for Detecting Adversarial Examples
The Odds are Odd: A Statistical Test for Detecting Adversarial Examples
Yannic Kilcher
 17 Discriminating Systems - Gender, Race, and Power in AI
Discriminating Systems - Gender, Race, and Power in AI
Yannic Kilcher
 18 Blockwise Parallel Decoding for Deep Autoregressive Models
Blockwise Parallel Decoding for Deep Autoregressive Models
Yannic Kilcher
 19 S.H.E. - Search. Human. Equalizer.
S.H.E. - Search. Human. Equalizer.
Yannic Kilcher
 20 Reinforcement Learning, Fast and Slow
Reinforcement Learning, Fast and Slow
Yannic Kilcher
 21 Adversarial Examples Are Not Bugs, They Are Features
Adversarial Examples Are Not Bugs, They Are Features
Yannic Kilcher
 22 I'm at ICML19 :)
I'm at ICML19 :)
Yannic Kilcher
 23 Population-Based Search and Open-Ended Algorithms
Population-Based Search and Open-Ended Algorithms
Yannic Kilcher
 24 Talking to companies at ICML19
Talking to companies at ICML19
Yannic Kilcher
 25 XLNet: Generalized Autoregressive Pretraining for Language Understanding
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Yannic Kilcher
 26 Conversation about Population-Based Methods (Re-upload)
Conversation about Population-Based Methods (Re-upload)
Yannic Kilcher
 27 Reconciling modern machine learning and the bias-variance trade-off
Reconciling modern machine learning and the bias-variance trade-off
Yannic Kilcher
 28 Learning World Graphs to Accelerate Hierarchical Reinforcement Learning
Learning World Graphs to Accelerate Hierarchical Reinforcement Learning
Yannic Kilcher
 29 Manifold Mixup: Better Representations by Interpolating Hidden States
Manifold Mixup: Better Representations by Interpolating Hidden States
Yannic Kilcher
 30 Processing Megapixel Images with Deep Attention-Sampling Models
Processing Megapixel Images with Deep Attention-Sampling Models
Yannic Kilcher
 31 Gauge Equivariant Convolutional Networks and the Icosahedral CNN
Gauge Equivariant Convolutional Networks and the Icosahedral CNN
Yannic Kilcher
 32 Auditing Radicalization Pathways on YouTube
Auditing Radicalization Pathways on YouTube
Yannic Kilcher
 33 RoBERTa: A Robustly Optimized BERT Pretraining Approach
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Yannic Kilcher
 34 Dynamic Routing Between Capsules
Dynamic Routing Between Capsules
Yannic Kilcher
 35 DEEP LEARNING MEME REVIEW - Episode 1
DEEP LEARNING MEME REVIEW - Episode 1
Yannic Kilcher
 36 Accelerating Deep Learning by Focusing on the Biggest Losers
Accelerating Deep Learning by Focusing on the Biggest Losers
Yannic Kilcher
 37 [News] The Siraj Raval Controversy
[News] The Siraj Raval Controversy
Yannic Kilcher
 38 LeDeepChef 👨‍🍳 Deep Reinforcement Learning Agent for Families of Text-Based Games
LeDeepChef 👨‍🍳 Deep Reinforcement Learning Agent for Families of Text-Based Games
Yannic Kilcher
 39 The Visual Task Adaptation Benchmark
The Visual Task Adaptation Benchmark
Yannic Kilcher
 40 IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
Yannic Kilcher
 41 AlphaStar: Grandmaster level in StarCraft II using multi-agent reinforcement learning
AlphaStar: Grandmaster level in StarCraft II using multi-agent reinforcement learning
Yannic Kilcher
 42 SinGAN: Learning a Generative Model from a Single Natural Image
SinGAN: Learning a Generative Model from a Single Natural Image
Yannic Kilcher
 43 A neurally plausible model learns successor representations in partially observable environments
A neurally plausible model learns successor representations in partially observable environments
Yannic Kilcher
 44 MuZero: Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model
MuZero: Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model
Yannic Kilcher
 45 NeurIPS 2019
NeurIPS 2019
Yannic Kilcher
 46 Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actions
Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actions
Yannic Kilcher
 47 NeurIPS 19 Poster Session
NeurIPS 19 Poster Session
Yannic Kilcher
 48 Go-Explore: a New Approach for Hard-Exploration Problems
Go-Explore: a New Approach for Hard-Exploration Problems
Yannic Kilcher
 49 Reformer: The Efficient Transformer
Reformer: The Efficient Transformer
Yannic Kilcher
 50 [Interview] Mark Ledwich - Algorithmic Extremism: Examining YouTube's Rabbit Hole of Radicalization
[Interview] Mark Ledwich - Algorithmic Extremism: Examining YouTube's Rabbit Hole of Radicalization
Yannic Kilcher
 51 Turing-NLG, DeepSpeed and the ZeRO optimizer
Turing-NLG, DeepSpeed and the ZeRO optimizer
Yannic Kilcher
 52 Growing Neural Cellular Automata
Growing Neural Cellular Automata
Yannic Kilcher
 53 NeurIPS 2020 Changes to Paper Submission Process
NeurIPS 2020 Changes to Paper Submission Process
Yannic Kilcher
 54 Deep Learning for Symbolic Mathematics
Deep Learning for Symbolic Mathematics
Yannic Kilcher
 55 Online Education - How I Make My Videos
Online Education - How I Make My Videos
Yannic Kilcher
 56 [Rant] coronavirus
[Rant] coronavirus
Yannic Kilcher
 57 Axial Attention & MetNet: A Neural Weather Model for Precipitation Forecasting
Axial Attention & MetNet: A Neural Weather Model for Precipitation Forecasting
Yannic Kilcher
 58 Agent57: Outperforming the Atari Human Benchmark
Agent57: Outperforming the Atari Human Benchmark
Yannic Kilcher
 59 State-of-Art-Reviewing: A Radical Proposal to Improve Scientific Publication
State-of-Art-Reviewing: A Radical Proposal to Improve Scientific Publication
Yannic Kilcher
 60 Can we Contain Covid-19 without Locking-down the Economy?
Can we Contain Covid-19 without Locking-down the Economy?
Yannic Kilcher
The Secret Spy Tech Inside Every Credit Card
Next Up
The Secret Spy Tech Inside Every Credit Card
Veritasium