How Residual Connections Are Getting an Upgrade [mHC]

Residual connections are adopted in virtually every deep learning model. BUT, can we further improve the residual connections? Hyper-connections are an exciting recent exploration to generalize residual connections. In this video, we will cover the following: 00:00 Training deep models 00:46 Residual connections 02:45 Hyper-connections - Intuition 04:48 Hyper-connections - Math 05:54 Dynamic hyper-connections 06:47 Training instability 07:52 Example of instability 08:44 mHC: Stabilizing training 11:02 mHC: Improving parametrization 12:30 mHC: Efficient infrastructure designs References: …