Moonwalk: Inverse-Forward Differentiation

1. 1Revisit the structure of gradient computation in backpropagation
2. 2Identify the need to store intermediate activations as a limitation
3. 3Apply Inverse-Forward Differentiation to compute gradients without storing activations
4. 4Implement Moonwalk in deep learning frameworks to enable training of deeper networks

ML researchers and engineers on a team can benefit from Moonwalk as it enables training of deeper networks, and software engineers can implement this technique in deep learning frameworks

💡 Inverse-Forward Differentiation can avoid storing intermediate activations during the forward pass

🚀 Moonwalk: Inverse-Forward Differentiation reduces memory usage in deep neural networks