Why use GPU with Neural Networks?

Key Takeaways

why use GPUs in neural networks there's a hardware and a software aspect to this answer and so this is going to be a two-part series in this video we're going to take a look at what the GPU Hardware provides that makes neural net training much faster and in the next video what software and algorithm changes do GPUs make to speed-up GPU processing what is five times six times seven 210 210 what is eight times three times five 120 120 what is seven times 12 times three to 50 to 250 to the CPU is blazing fast the GPU is almost as fast but slightly trails behind what is matrix a times matrix B this CPU your answer this looks like the CPU took its own sweet time GPUs are used in neural nets because of this ability to perform matrix multiplications so fast but why does this happen why or CPUs great with scalar multiplications and GPUs better with matrix multiplications three main reasons GPUs have a larger memory bandwidth they use parallelization and they have faster and more memory access than CPUs we're gonna delve into these three points in detail and then show some PI torch code in action that really demonstrates the difference in speed let's get started so here's a block diagram we have a system memory this is the main memory which contains the matrices that we need to multiply we have a CPU and a GPU and each of these also have their own memory that I'll just label as memory without the details for now let's make an analogy here the CPU is a Ferrari and the GPU is a truck case one performing scalar multiplication the CPU can get this data with a fetch operation to memory the amount of data that we can fetch though is pretty tiny but it's large enough to fetch floating-point numbers within a couple of trips and these fetches happen blazing fast like a Ferrari but GPUs fetch operation on the other hand can transmit a lot more data but they are not optimized for speed so they're like a truck they can take more time to fetch and process small chunks of data and that's why for scalar multiplications GPUs can lag behind CPUs case two when we multiply two large matrices CPU it has a super fast fetch operation but each fetch can only transmit a tiny amount of data to and from memory so it can take thousands of trips to fetch all the required data well one trip is being fetched the CPU memory is processed and freed up for the next inputs this is where GPUs work much better at a time we can fetch much larger amounts of data from the ram into the GPUs memory so the GPU doesn't need to make that many trips in more technical terms GPUs have a higher memory bandwidth than CPUs memory bandwidth is the amount of data that can be transmitted in a single trip to and from the memory and this is one of the main reasons GPUs have an edge over the CPUs for large matrix multiplication but even if we get a lot of packages or a lot of data the GPU processors remain idle for a while after all they're too fast and the truck is just too slow so instead of having a single truck we can have a fleet of trucks this way GPU processors do not need to wait around and it always has data to work with this notion of using a fleet of trucks is called parallelization coupling large memory bandwidth with parallelization reduces any time a GPU would be waiting so basically we fetch a lot of data and we do it fast but GPUs offer something more faster memory I use the term memory very vaguely before but what I'm really talking about are the caches and the registers GPUs have a similar structure but GPUs l1 and l2 caches are smaller in size than a CPUs l1 and l2 caches smaller size means that it can be accessed much faster and also these streamlined processors of a GPU have a bunch of registers which are super fast and GPUs have upwards of a thousand times more registers to play around with then CPUs so all of these memory enhancements together makes computations just blazing fast and it's a combination of these three points that GPUs are faster than CPUs from matrix multiplication especially for large matrices faster multiplication means that we can perform operations in deep learning also faster cool but how do we actually make use of GPUs as a programmer well this is where CUDA comes in from a programmers view CUDA provides an API that allows us to access the components of a GPU like the stream line processors the caches and the registers CUDA already provides a nice high level abstraction but deep learning frameworks like pi torch make life even easier we don't even need to know about the inner components of a GPU we just treat it as one big abstract unit called a GPU and we're all good to go let's actually see how matrix multiplication with GPUs perform I'm in Google Kolob and it's an environment that allows you to run your code in chunks and see the outputs directly I have some code cells here that we can walk through the first cell is a scalar multiplication with the CPU we're just taking the square of a number PI torch uses tensors as the fundamental building blocks and you can think of tensors as wrappers around your scalars and matrices for any dimension I'm just squaring a 1 cross 1 tensor here we're using a command called timeit to calculate the execution time of this block of code these are known as magic functions if you want to look them up for more reference it looks like the CPU took 5.26 microseconds to do this that's super fast in the second cell we have similar code but this time we multiply a 10,000 by 10,000 matrix with itself and this took eleven point eight seconds it probably seems like it it's taking longer when you actually run it but that's only because it's being executed three times for the rest of the code you need to set up a GPU so for this you got to go to the menu bar and click on run time then on the drop down you go to change run time type and from the hardware accelerator drop-down select GPU this initializes the GPU we now have to tell PI torch to use this GPU and we use torch device to get a reference to this GPU and this dot two device tells PI torch to store and process this variable Z using the GPU let's run the cell you okay so that took thirty five point two microseconds which was slower than the five point two six microseconds that we did using without the GPU so it was slower let's now run the large matrix multiplication with the GPU okay wow that took less than a second way faster than the 12 seconds it took before pretty slick right what I explained right here is only half the reason GPUs are blazing fast with matrix multiplication in the next video we're going to take a look at how we can change the original matrix multiplication algorithm for faster processing but that's all I've got for you now hope y'all enjoy what you saw click one of these cards to see some of my amazing work and I will see you very soon bye you