How do GPUs speed up Neural Network training?

Understand both the hardware and software aspect to how GPUs speed up training. Deep Dive into Block Multiplication, CUDA. REFERENCES [1] Why GPU’s work well in deep learning: https://www.quora.com/Why-are-GPUs-well-suited-to-deep-learning [2] Matrix math on a GPU: http://www.ijcee.org/vol9/949-E1621.pdf [3] GPUs are not necessarily faster than CPUs: https://www.quora.com/Why-are-GPUs-more-powerful-than-CPUs [4] Code for difference: https://medium.com/analytics-vidhya/using-pytorch-and-cuda-for-large-computation-in-google-colabs-f1c026c17673 [5] GPU memory architecture: https://medium.com/@ashanpriyadarshana/cuda-gpu-memory-architecture-8c3ac644bd64 [6] Block Matrix Multiplication: https://www.researchgate.net/publication/221391920_Parallel_Processing_of_Matrix_Multiplication_in_a_CPU_and_GPU_Heterogeneous_Environment [7] Matrix Multiplication with CUDA: https://www.fz-juelich.de/SharedDocs/Downloads/IAS/JSC/EN/slides/cuda/05-cuda-mm.pdf?__blob=publicationFile