Floating Points are no more, Changes everything for LLMs!!!

Key Takeaways

This changes everything in the LLM world. LLM that stands for large language model are basically deep learning models. Deep learning models are deep neural networks. We have got multiple neural networks that deeply stacked together and that gives you a final output. But how does it give you a final output? These deep neural networks have got weights and these weights are multiplied like a matrix multiplication that happens at highly computational high performance computing environment and that is how you get these deep neural networks working. And this is one of the reason why people are using GPUs, especially Nvidia GPUs, because Nvidia GPUs have got CUDA and that supports this high performance matrix multiplication. But imagine a world where you don't need GPU, where you don't need high performance matrix multiplication. Is that even possible? And the answer to the question is it is possible. In fact, it is possible and also it is going to be as the same as the current large language models in performance and that is what the paper that we're going to discuss today is. The era of one-bit LLMs. All large language models are in 1.58 bits. I mean, the title might be a little confusing. Let me take you what the title is and what the paper is. First of all, if you see typically a large language model looks like this. It is either a 32-bit model or a 16-bit model. What does it mean? That means every single parameter of the model is represented as a 32-bit float or a 16-bit float. If you're not familiar with float, float is nothing but a decimal point value and integer is nothing but like a whole number with a positive or a negative sign. So, +1, -1, 0, um +2, +3, +4, this is all integers. 0.2345, 0.5678, 0.7013, these is all float and different floating points have different precision level. So, we either have a 32-bit floating point or a 16-bit floating point. And like this, we have a different precision. The current deep learning models, the current large language models look something like this. But what if or what they have done here is that they've taken this matrix multiplication or this matrix and transformed it into something like this. So, they have represented everything that you can into -1, 0, 1. And that is exactly what this BitNet B 1.58 model does. So, every single parameter on the model, instead of being represented as a 32-bit floating point or a 16-bit floating point, it represents it as a ternary value, 0, 1, -1. So, +1, 0, -1. And how did they get this idea? They got this idea not like it's not they did this the first another set of researchers from Microsoft. Maybe there is a good overlap between this and then they created something called a one-bit transformer. So, in the current paper that we are seeing that we have everything represented as +1, -1 and 0 instead of floating point values. But the previous paper, the one-bit transformer paper, what it did is that again from Microsoft and University of Chinese Academy of Sciences, they did everything to be either 1, +1 or -1. So, that is why it's called a one-bit transformer model or a one-bit weights. So, every parameter here was represented as one bit and in the new paper that we have got, every parameter is represented as +1, -1, 0. And that is why it is a 1.58 bit, not one bit, because we have added zero and they say that the zero addition of zero gives them more advantage in terms of learning capability. So, what is happening here? Simply, you have got a matrix like this. You have got another matrix like this. Typically in the current deep learning neural networks, what you need, you need matrix multiplication and you need addition. We call it a dot product. So, you multiply, add, multiply, add, multiply, add. So, that is what has been regularly happening. You can see here this one, the weights and the inputs are multiplied and then it is added. This is multiplied and then it is added. So, you can see the equation here. Y is equal to function of W and X. W is here weights, which is already part of the model. Input is what you give inside the model and that is how the calculation is done. So, if you need, I can provide additional study materials of the understand the deep learning is working here. But technically, you can see you've got model weight and then you have got input X. So, this is all here. Now, what is changing here is that instead of having the floating point value, you're going to have +1, -1, 0, the ternary values, which in this case is integers. That means you can just do addition and you can completely get away with multiplication. And the fact that you can get away with multiplication means like you you can like go you don't need multiplication. Maybe I said it wrong in English. You don't need multiplication. So, if you do away with I think I should say do away. If you do away with multiplication, that means you don't need a GPU kind of hardware. In fact, you can completely build a new kind of hardware, something like what Groq is doing. Um that's an example. I don't think Groq is doing exactly like this, but you can have a different kind of hardware that supports that is optimized that is highly optimized for this kind of calculation. See, how did GPU become so popular? GPUs were used by gamers, GPUs were used by miners, but GPU became really popular because of accelerated computing and somehow Jensen Huang actually hit a jackpot and a lottery that Nvidia GPUs that supported CUDA became the highly optimized place for deep learning, which is matrix multiplication. So, what this can provide is it can help you provide create a new hardware that does not require matrix multiplication. That means the innovation in the hardware side is there is a huge potential, there is a huge window of opportunity. Innovation in the software side, the optimization algorithm side is also huge and that is what this paper is proving. This paper is saying that once we have a BitNet architecture, which in this case is B 1.58, every single parameter or the weight in an LLM is defined as ternary, which is -1, 0, 1 and the accuracy, the performance matches the full precision FP16 floating point 16 or B floating point 16 transformer LLM. And with the same model size, like if you take a 7 billion parameter here, if you take a 7 billion parameter, if you take a 700 million parameter, if you take a 700 million parameter, one is BitNet, the other one is transformer, it has the same performance. While the performance is same, the biggest advantage comes in latency, how fast it can give you memory, how much memory it requires from the RAM and throughput, how many tokens it can generate per second and also most importantly the energy consumption, which would further reduce your cost in running this model. So, ultimately, more profoundly, the 1.58 bit LLM defines a completely new scaling law about how you can scale models and the new recipe for training new generations of LLMs that are both high performance and also cost-effective. And this can also give way for new hardware, which is what they're saying. So, I hope you understood this particular point. To quickly show you what they have done, they have taken like I said like they've taken all these things and rounded off their values and they made it like -1, 0, 1 and they have all the benefits of the previous paper that I mentioned, the one-bit transformer. I might cover this separately because I think there is a lot of useful information in this paper. I cannot believe that I missed this paper. This came on 17th October, but this this retains the architecture of that particular paper, like the one-bit BitNet. The only change it does is instead of two values, -1, 1, 1, which is a one-bit value, they've added a zero that makes it 1.58 bit and it gives you two advantages. One, it makes the feature filtering possible and also it makes the quality of the models to learn in terms of the perplexity, the metric used to measure whether the LLM can produce coherent text and also in the downstream task, like for example, question and answering, figuring out something, the end task, it also does a better performance, which we can see quickly. So, if you see, this is the model, the LLaMA LLM, LLaMA-based architecture large language model. They're not doing any transformer model. They're literally comparing it with LLaMA model and in fact, to make things equal, to make things as a fair play, what they have done is they have taken the model completely scratch trained it from scratch on a RedPajama dataset. So, they have tried to use similar amount of dataset, which is like the LLaMA dataset and they have compared it. So, if you compare the LLaMA LLM, the 700 million parameter, 1.3 billion parameter, 3 billion parameter model. So, on three different occasions, you can see the perplexity being almost equivalent. You can see perplexity, the lower the perplexity is better. So, perplexity being almost equivalent and in fact, the largest 3 billion parameter model, the perplexity is lower than LLaMA LLM. And I've always wondered, see, perplexity is good measure for next word prediction and that's not the only measure, because we these days want to use models for a lot of downstream task, like tasks like question and answering, text classification, text summarization and a lot of other things. So, they have also compared this with a bunch of metrics like HEL SWAG, ARC and all these like very popular metrics that people use for downstream task and you can see on an average I mean, you can look at individual numbers, but on an average, if you see the BitNet B 1.58 model, the 3 billion parameter model and the 3.9 billion parameter model, which is in the ballpark of 3 billion parameter model, scores above the equivalent LLaMA transformer model. See, this is still transformer. It's not like it's not a a transformer model. This is still a transformer model. The reason is if you go to the one-bit model, one thing that you would understand is what they have done is they have taken literally the transformer architecture, and in that transformer architecture, they have taken everything, and the only thing is they have used a bit net the bit linear instead of a conventional matrix multiplication. So, wherever you need matrix multiplication, they've used a bit linear there. Where you don't need matrix multiplication, you can just go away with the addition part. So, this is still a transformer. Llama is also still a transformer, but the way you represent the numbers inside the weights completely change and the way you do the calculation for, you know, finding out the deep learning part completely changes, and it scores good in perplexity. It scores good in the downstream tasks like ARC and LSAT. The biggest thing is it's not just the accuracy, the time it takes to produce the latency is improved, and you can see like there's 2.7x improvement, the 2.4 times improvement, and and the throughput, if you see, the throughput is like nine times huge. You can generate a 2,977 tokens per second for a 70 billion parameter model. This is way above what Grok is doing currently in production. And this does not require sophisticated, um, let's say, hardware, I guess. At least that's my understanding because this is you're a very simple, um, you don't need as a hardware like a GPU, but I think this gives a an immense boost in the throughput, the tokens per second thing. People are going bonkers and crazy for the 500, uh, tokens per second that Grok is giving for Mistral Mixtral 8x7 billion parameter model. This can give you 2,977, which is like nine times more than what is available here with this Llama based architecture model. I think this is this is huge. There are a lot more interesting details in this. I don't want to bore you with all the details, but this gives an enormous potential about where you can run the model, what hardware you can run the model like, for example, mobile, edge, uh, different type of hardware like I don't know if you have known seen that the BBC four-bit kit. So, I think the potential is immense, um, the way you can scale up the models, the the LLM scaling laws, the way you can run the model, the throughput. I think the game will completely change, and, uh, if more people build on top of it, that's one of the reason I've made this video. Let me know in the comment section what do you feel about this completely new direction of having one-bit or 1.58-bit LLMs. Just to give you a perception. So, let's say you have got 16-bit model, okay? You've got a 16-bit, uh, precision model LLM, and, if you have got a 4 billion parameter model, all the 4 billion parameters have to be represented in 16-bit floating-point values. That means whatever the 16 times hardware that you need, now you can reduce it a much simpler because you are in a 1.58-bit. So, there is a huge like 15, 14-ish times improvement, or let's say eight-ish time improvements that you are going to get simply by using an architecture like this. I think this is quite exciting to see what people are doing in this particular space, and I hope to cover more of this. See you in another video. Happy prompting.