Kimbo Chen - Dragon in the CUDA Moat NVIDIA Tensor Core Evolution

Key Takeaways

[Music] Is it started recording? >> Yep. >> Okay. Um, so hi everyone. Uh, my name is Kimbo. I work at uh semi analysis as an engineer. Uh, today I'm going to talk about the ele evolution of um, Nvidia Tensor. Uh, uh, I believe it's, uh, people say CUDA mode. Um I believe tensor core are one of the most important part of the CUDA mode. So I called it the dragon in the CUDA mode. Um so why should we care about um the evolution of tensor cores? Uh the reason is that tensor cores are mysterious and important. Um, as we all know, uh, Nvidia GPUs are one of the most important, uh, deep learning hardware accelerators that has driven major innovations in AI. Um, as you can see in the graph on the right, um, throughout the past almost 10 years, um, the Nvidia's GPU's uh, flops throughput has increased by quite a lot. Um for example like the GB300 is uh closing to 15,000 teraflops at FP4. Uh but um as tensor core architecture becomes more and more complex it is difficult for ML practitioners to keep up with the latest changes and the reason behind those architectural changes. So to the practitioners, tensor cores kind of became a mysterious black box um to when using them. But I I believe um understanding tensor cores is important because how critical tensor cores are to performance. Um as we see in the graph on the left um which is the breakdown of the throughput by GPU we see tensor cores uh compute flops are basically uh most of the whole GPU. Uh for example uh for B200s it's over uh 90% are um coming from tensor cores. It's important to understand how they work and uh um to use them properly to squeeze the most performance out of Nvidia GPUs. And uh that's uh the reason why we wrote the article and um today's talk it'll be an excerpt of uh the talk of the post. Um we will be only talking about four things. uh first we'll talk about uh at a very high level how tensor cores perform matrix multiplication and um and later three um design trends and the principles behind them. So first how does uh tensor core uh perform a matrix multiplication? So um uh before we started we talked about this um uh I hope everyone has a very basic understanding or like the ter heard of the terminologies of CUDA programming model. Here is a very uh um summary of the the terminology we'll be using. So um the column on left are the execution unit hierarchies a thread a CT and a grid. The column right are the corresponding memory pools. So for example, a thread can access a registered file. A CTA uh can access shared memory. A grid access global memory. So the first thing we'll talk about is how the first three generations of uh tensor cores volta curing and then the nampier how uh those architectures perform matrix multiplication. So there are four steps. First you load um the data from uh global memory to shared memory and then you load data from the shared memory to register file. Third step you uh execute an MMA operation and then you perform the epilog operations. Usually they refer to uh non-tensor core operations or operations done on CUDA cores. um the most one of the most common ones being uh element wise operations or like um activation functions etc. One thing to note about um those steps are um it is a collective operation meaning that it takes multiple threads to collaboratively complete the operations. Uh it also means that it'll need some sort of synchronization um uh which we uh draw here in the graph. Um the core instruction uh MMA stands for matrix multiply and accumulate. It is uh PTX instruction for tensor cores. um uh which we will talk a bit more about later uh throughout um volta uh to ampere turing is kind of um in the middle between volta and ampier in terms of design. So the major differences between volta and ampier are um the number of threads you need to um launch an MMA. Uh specifically, Lola uses eight threads. Uh Ampier uses a warp of 32 threads. Um the second difference is um Ampier is able to load um data from global memory to shared memory in an asynchronous fashion. Asynchronous execution is a very important trend or very important feature um for Nvidia GPUs that we will see a lot more later and um we'll also explain uh what it does um or why it exists um later in the talk. Okay. So uh before we uh go um further um just want to make sure uh that everyone is following uh cuz this is like a very rough overview. Does anyone have any maybe questions? Um everyone good? Okay. Oh. Uh, someone is uh how do I pronounce your name? Sorry. Go on, please. >> Yeah, you can unmute yourself. Yes, if you have a question. Oh, I'm assuming maybe it's my mistake. Uh, you can go ahead. >> Okay. >> Oh, should I go on then? >> Uh, yeah, probably. >> Okay. I guess >> yeah. >> Yeah. Uh feel free to ask questions later because I I know this might be a very big jump in terms of like loading a lot of um concepts and terminologies into it. Okay. So those are the first three uh architectures. Um the architecture after that is hopper. Hopper comes with a new type of MMA uh instruction called WGMMA. U WG stands for warp group. warp group um is uh basically a set of four warps and uh which is 128 threads. Um by having more threads you could um use uh more registers. Basically the uh another critical feature about WGMMA is that is asynchronous. Um again asynchronous would uh come up again and again um and we would talk about it uh uh later. in the matrix multiplication uh flow. Uh another new component is uh the tensor memory accelerator um also known as TMA. TMA is a dedicated hardware unit for handling uh for accelerating the uh memory loads uh from global memory to shared memory. What it does specifically is that um it handles address generation which is typically done by uh threads and CUDA cores. So by using a dedicated hardware unit to do so um you could free up the threads to do other operations um launches uh epilogs uh etc. Another uh uh feature is that TMAS has single thread semantics, meaning that you only need one thread to launch um an instruction to get TMA to work. This is a departure from the CUDA programming model where you need multiple threads to operate. And um this is also a common theme that we would uh keep seeing um after this Um so after Hopper we did Blackwell um currently the newest architecture. The corresponding new MMA is uh usually called UMA um also known as TCO5 MMA. Again it has single thread semantics like uh TMA. Um it's a departure from a CUDA programming model. And um another um big difference is that it um TC Geno5 MMA can use two SM to uh to perform one single uh MMA. So uh before Blackwell uh each uh tensor coursees um tensor cores exist on one SM and um uh you launch uh an MMA within the SM. But now you could um expand across SMS to perform even larger mat matrix multiplication. Another big difference in the black hole architecture is uh the tensor memory also known as TM. TM is uh um memory component within NSM uh that is dedicated for tensor core operations. it replaces uh register files as an operant location. Um and um one of these features is that it uh has a restricted memory access pattern. Uh specifically uh in order to access the whole TM you need four warps to do so and every warp can only access a specific region of it. Uh by doing so um in terms of hardware you could uh reduce the number of memory ports, you could simplify the memory design uh which um saves uh power and uh chip area um as a result um improving the power efficiency. Another uh important uh feature of T-m is that uh again a departure from the CUDA programming model where um each thread holds a specific set of registers. TM's uh memory can be accessible uh across different warps. meaning that uh I can use one warp to load data into T-M but later use another warp to uh store data from TM. Um by doing uh by having this it we can uh simplify some um advanced featur features in uh black world such as uh warps specialization and um uh by doing so um you don't need um the pingpong scheduling in hopper um and have a simplified workflow. um ping pong scheduling and warp specialization are um quite complicated um uh features and um techniques um for uh I don't think it's even uh covered in the blog post but um for people who are interested interested um I would recommend uh watching um the Blackwell GTC cutless talk um they talk about um the pattern Okay. So, uh the first trend uh I want to talk about is uh the fact that tensor core size is increasing generation by generation and the reason why um and also the the principle behind it. So, uh tensor cores have been uh able to perform increasingly large matrix multiplications. Um this uh uh implies that the tensor core size is increasing. Um the graph on the right shows um the number of flops that um you need to perform um the major MMA shapes for each GPU. So we see uh the y-axis is actually a log uh um in log format meaning that the number of flops is actually increasing exponentially. So the motivation behind um performing like making tensor chords performing larger macmos is because uh matrix multiplication has a linear arithmetic intensity. Specifically uh we all know matrix multiplication is uh cubic in computation but uh it's also quadratic in memory loads. So arithmetic intensity being computation over the number of bytes loaded. Um having a linear arithmetic intensity meaning there will be um uh it scales arithmetic intensity scales linearly with the size of the matrix multiplication and having a higher arithmetic intensity implies that there will be a higher performance ceiling. um because of uh larger MMA shapes um it it is also why we need to increase the thread count to launch the MMA uh at least before Blackwell. So um we see Involta uses eight, MP uses 32 and Hopper uses 128. Um, by using more threads, you could allocate more registers. Um, and um, you could load larger MMA shapes. You could improve the data reuse, increase the data reuse by, um, uh, because you don't need to reload parts of or tiles of the same matrix by just loading everything once. The reason why uh architecture designers are so conscious of memory loads is because um in the blog post we call data movement is a sin because uh so the sin the reason why it's a sin is um because um memory uh speed and memory capacity hasn't been improving um as fast as compute. So making um computation a lot cheaper than data movement. So um that's why tensor um architecture designers are very conscious of how much data movement you need uh to perform um u matrix modification and um to get good performance. The the second trend I want to talk about is the programming model uh going to low occupancy and even single occupancy and the uh the principle principle behind it uh is uh strong scaling. So for non-delearning CUDA programming we usually do a thing called a overs subscription. Uh this means that we uh programmers try to um allocate uh multiple work units and designate them to uh the workers so that each worker has multiple work units to work at the same time. And um while um by context switching between multiple work units, a worker can save time um when one working unit is stalling, it can switch to the other to do work. um this type of concurrency um is um or this kind of parallelism is what GPUs do and programmers usually measure this kind of parallelism uh in terms of the number of uh active workers in um PTX machine model in Nvidia um we measure number of warps um so uh this is also called a occupancy so um in non-earn learning CUDA programming um having a high occupancy is the key to performance. However, in deep learning CUDA programming um NVIDIA GPUs strive for low occupancy and um even single occupancy. A way to understand why this is happening is that um for the same matrix size as we see in the graph here, if we increase the MMA shape like we saw in the first uh trend um we will have less working units because um MMA shapes um are getting larger. Uh so since the less uh there are less working units um Nvidia designed it so that the number of working units um each worker is assigned um one working unit and actually in some cases even one single unit uh would use multiple workers. So the uh why did uh Nvidia try to change to low occupancy and single occupancy? The reason is um Nvidia wants to strong scale matrix multiplication. So uh strong scaling and um the opposite weak scaling um they have uh definitions. Um please refer to the blog post or um uh search up on what they mean. uh but uh what I want to talk about is the effects of strong scaling and weak scaling. Uh the effects of strong scaling is that it improves the performance across all problem sizes. In our case, it'll be improving the performance of all matrix shapes. Um while weak scaling um the uh opposite case would be adding more tensor cores. This way it only guarantees better performance on larger matrix shapes because you need um more working units to saturate more uh workers uh to improve performance. But that doesn't guarantee for the same matrix shape it will run faster. Okay. The last um feature is the asynchronous execution and the principle behind it is AMD's law. So um the reason why we need asynchronous execution is because uh strong scaling uh matrix multiplication almost requires asynchronous execution. Uh one way to see is see this is that if we draw the um um if we draw the matrix multiplication flow as we see here in the graph um LDGSDS the red boxes are uh basically data loading and the green boxes are the the MMA operations. So um if we try to improve our strong scale MMA uh in this case tensor course throughput has been doubling very fast uh the dark green boxes they have uh shrked by quite a lot but as we do so um data loading would become a bottleneck um very quickly and this is quantitatively described by AMD's law um AMD's um uh please look it up. Um there there's like a formulas for it. But the idea is that scaling um strong scaling specific components would make other things the bottleneck and no matter how fast you scale specific components. The way to um avoid um AMD doll's law or work around AMD's law is by introducing asynchronous execution. So you overlap uh the data loading with the MMA operations that we see in the third row. So by uh by being able to overlap them um you could uh avoid the bottleneck of data loading asynchronous execution uh has been a trend in the NVIDIA's architectures. Um every generation come was comes with uh new asynchronous features which um we don't have enough time to go through every of them but uh if you're uh if you're interested uh please read the blog post. Um this is like a a summary of the tables uh a table of the um the main features that are asynchronous. Okay. So that's uh pretty much uh what I uh the main four main things. Um I finally I want to talk about what is semi analysis. So semi analysis is a research company uh that offers insights into the AI industry. Uh starting from data centers all the way down to semiconductors. uh we create benchmarks and analyses like um the the post um that we we were talking about the tensor core evolution posts and in addition to architectures we also do uh across different sectors. Personally my focus is ML systems and um uh my Twitter and Discord handle are both Kimbo Chen. Um feel free to contact me. Um, and finally, I just want to shout out um to uh my favorite cover image um which is the one you see here. Uh it's uh drawn by Simon Go. Okay. Uh thank you everyone for listening and then happy to um answer any questions. If anyone has question, you can just probably unmute and go for it. Does anyone want to go first? We could probably I could uh ask a question. I >> one slightly uh like this is my understanding that from hopper at least uh a lot of the uh a lot of the benefits or this I mean like to use a native term the speed up or the flops right it seems to be coming from architectural changes like the TMA and other like the WG MMA that's on Hopper >> and the other GPUs don't really have it, right? Like so for example in the ADA generation the L40S for example, it just doesn't have this. Uh and then moving forward like a concerning trend is that a lot of the performance is going to be limited to like the topmost line of GPUs. Do you do you also get that sense from your research like or where are the other GPUs going to like are they going to be permanently bottlenecked in some sense like not what is not the B200 what is not the H100 uh say the ADA 6000 or the L40S or does that broadly make sense the question? >> Yeah. Um so my my um to repeat what you said um my understanding is um you're asking about um whether the these features that um we saw in the talk is um whether they exist in um only just uh topofthe-line uh data center GPUs or not. I I'd assume >> uh I'd assume they like they don't exist, right? Like because since the hardware is not there uh on the other GPUs. So uh >> yeah um so I think uh I'm not sure for for every feature like um either they exist or not. uh one thing I'm I'm certain is that the features that I mentioned here all exist in the data center once and um the other thing is that uh for MMA this uh um um MMA has always existed um across different uh architectures um or at least MMA still exists in Hopper and Blackwell um but it's just not um it doesn't make sense to use them given and that they're not as performant. And um MMA uh if I remember correctly exists on consumer GPUs even for um the blackwell ones and that is the one that is used um on the consumer GPUs. Does that make sense? >> Yeah. So we like is uh the curiosity was more like do do you see like some other line of optimization for like the non high-end data center GPUs? I'm just curious because uh this was a bottleneck. So uh uh yeah that that was uh broadly but but I get what you're saying in in terms of like you would just use a different set of operations on uh you'd use the MMA if you don't have access to an access to an 8100 or say a Blackwell or something of this sort. Yeah. >> Yeah. Yeah. Um I I don't know the details of um the architecture differences. I know typically they do um um like even within between chips um the same H100s um they different versions of H100s they disable a number of SM depending on uh the version you get. So I think it's quite common that um chips have uh different feature sets being enabled. Um but but yeah, I don't know the exact um what features exist in um what line of GPUs. >> I think there is a question in chat. I'll just probably read it out unless uh Shrihari, do you want to unmute and speak? Okay, you can. No worries. >> Um okay, so I I see a question. So a sense of timeline of this evolution. >> Yeah. Yeah. Um so Volta started in 2017. Um Turing 2018, Ampier 2019 and um Hopper um yeah um I don't know I don't remember top of my head for the numbers but I think Hopper is around the 2020s either 2020 or 2022 I don't remember exactly. And then it comes blackwell. Blackwell is the late uh the latest at least for now. Um uh B200 the number that we're referring to is one of them and um the upcoming one being I think the GP300's. Um okay, second question. What are some research areas that have the potential to create another generation? Um I'm not sure what you mean by this question. If if you can um uh clarify a bit um that would be great. Um I mean I could try to uh provide some more detail like like um like I saw from what I can understand it seems like asynchronous computing um at the chip level is that that's like one of the uh breakthroughs in enabling the latest generation. Uh, is there something like that uh that's being researched? I'm sure there's like a variety of things, but do we have like a sense of um some promising areas that could prompt yet another revolution? >> I I think I I roughly get what you mean. Um I think in terms of uh I I wouldn't say asynchronous execution being like an um like research area. It's more like a principle that um applied uh that the the designers applied uh so it it has always been there. It's just um a matter of how much or how you apply them. uh but uh one um thing I I I would guess is um another trend is um uh we we talked a lot about in the talk which is uh the the flow of matrix multiplication is uh departing from the CUDA programming model. So uh for example um you right now you basically need only one thread to launch um most instructions and um whereas um in CUDA programming model you typically do um multiple threads and collaboratively uh do things and um there are uh a couple more trends mentioned in the blog post which I took out because of um we don't have enough time um but but Yeah, feel free to refer to read the blog post um for more um other trends um in it. So back uh off the top of my head maybe um the trend is also reducing data types which is very relevant to our efficiency group um from FP32 all the way down to FP4 recently. Um um what other things? Yeah, things like that. Um, yeah, please, um, look, um, read the blog post. Any more questions? Uh, I just generally as a another question, right? like this is more of a uh how do you put it like a speculative question than it is a concrete one. um is uh pro programming complexity >> the the trade-off that's coming in with programming complexity uh across generations >> like like how how we would be is is it uh getting significantly more for to draw more performance out of these tensor cores across these things or is it like almost simplified because I believe you mentioned like there were some restrictions that are coming in in terms of memory access and those kind of things. >> Do you have a broad sense of how this is also trending >> other than from the software side of things? Yeah. >> Yeah. Um so uh as far as I understand um the programming uh complexity um is Volta actually is very difficult to program properly because there the tensor core uh design um like we mentioned it uses eight threads but um we all know um the unit of execution or the um programmers is usually thinking warps when it comes to CUDA programming model and it being using eight threads kind of makes the data layout very complicating. Um for more information um can look uh to the the Volta cutless talk they have very beautiful um uh visualizations of how complex that layout is. Uh but Ampere became because using threads and other designs uh corresponding designs. Amperia is actually very um uh a lot better a lot easier to program compared to Volta and Hopper uh became complicated again because um it's trying to use four warps to cross um um the warp uh concepts of um CUDA programming model and it makes calculations um a lot more complex uh and um Blackwell actually simplifies um hopper. Um I I heard a lot of kernel programmers told tell me um black is so much better designed. Um one uh one thing main thing being the T-M that we mentioned um so the the restriction in the memory access pattern. Um I my guess is that I'm not sure but my guess is that it wouldn't be increasing the programming complexity too much. Um it would be just a matter of how you uh allocate the um declare the warps uh to load what part of the data. Um so does that answer your question? So maybe just in summary just Volta very hard, Ampier easier, Hopper became hard again and black hole became easier. >> So bucking the trend the next one might be hard again and >> by by extrapolation uh I don't know. >> Yeah, >> hopefully not. Hopefully not. >> Yeah, >> I think uh one last question like building on what Shi kind of asked. Uh it seems like Nvidia is betting huge on quantized data types like quantiz training. So they're reporting these magic numbers at every generation like with I think uh with Hopper they did FP8 flops and then with uh with black it seems to be F4. Uh I see I think we've had a lot of work over the past like uh two two months or so of people coming in and talking about like stabilizing training and uh those kind of ideas, right? But these are diminishing returns. I think after FP4 it becomes very dicey. You start going into like like sub4bit is a weird situation to be in. >> I wonder is it is it is it really going to trend towards sub4bit like micro I know we have microscaling and those kind of ideas etc but >> like I for me it's not clear like the question is like where do the next benefits come so easily because we can't just lop off another two bits and just be happy I guess so >> just curious on that part >> yeah yeah very very great great great question so um uh in terms of architecture. Um we we also wrote this in more detail in the blog post is the the data type reduction trend. Um it makes a lot of sense from the architecture perspective to reduce the uh precision. Um so in principle you don't need as many uh as many bits to do machine learning operations and um because of that um reducing the number of bits has uh many performance uh benefits including you use less bandwidth. you um uh use less um you less power to activate the um less power to do the computation. You need uh um less capacity. You can load more parameters at the same time given the same uh amount of capacity. Many more many benefits. Um but um this uh like you said um uh architecture uh wise it makes sense but the reality is that does it impact or how much does it impact um the model capabilities. Um so um as far as I know um right now FB8 is the standard for training um and um deploying FB8 is easy and and intuitive basically training what you get um for the newer generations for uh like Blackwell um we support like MX microscaling formats which um seems to be standardized but um one thing we highlighted in our post is um the difference between um um FP4 formats um which is uh the NVFP4 the one that uh Nvidia used and the the standard MXFP4 and as far as as we we see NVFP4 is uh far superior from MXFP4. Um I think the trend is going definitely going downwards. We're right now um stuck at FP8 and uh FP4 is still at the inference stage. Um but um yeah, who knows? Um maybe maybe someone to figure out how to do training in FP4 and um I would say integer formats um is also a potential. So yeah, hopefully that answers your question. >> Yeah. Yeah, that was good. Yeah, but unfortunately I think they locked off the infert tensor course. So probably they're making that decision for us. You know, Nvidia in some sense. I think in black do not doesn't have it anymore. Yeah. >> Yeah. Hardware lottery. Yeah, basically. Yeah. Yeah. >> Makes sense.