8-bit Methods for Efficient Deep Learning with Tim Dettmers

Key Takeaways

foreign my name is Ellie I am here on behalf of cohere for AI uh Welcome to our latest edition of technical talks today I'm welcoming Tim detmers here to talk about 8-bit methods for efficient deep learning as we get the slides up and ready to go a little bit about Jim he's a PhD student at the University of Washington he's working on representation learning and neuro-inspired and Hardware optimized deep learning I previously interned at the UCL machine reading group I was working on information retrieval and Link prediction in knowledge graphs and Main research thesis is that computationally efficient methods will accelerate and enable progress in an understanding of deep learning we were just talking before we came on live about how quickly everything is accelerating in AI these days so this feels like a particularly timely topic um a couple of housekeeping notes before we get underway the session is being recorded uh we will be posting this on the cohere for AI YouTube playlists in about a week or so uh so if there's something that you miss as we're going through this don't worry we've got you covered um if you have questions for Tim as well about the presentation please use the Q a button it should be the very bottom of your screen please use that to submit questions we will have time for some q a at the end of the presentation and we'll try to fish out relevant uh questions as we go uh so that's enough talking for me thanks everyone for joining and I'm going to turn it over to you Tim thanks for joining us today thank you so much Ellie um yeah so so today I present on a methods for efficient deep learning um things are moving very very fast so maybe the title should be cable methods because we're going lower than eight bits but um here I present work which main motivation is like making large models more accessible if you look at sort of existing open source uh models and more recently also llama kind of Open Source and they use quite a bit of memory both for inference and fine tuning and so in my work I try to quantize large models to make them more accessible and so that would reduce memory footprint considerably and then we achieve some Milestones where you can run model on Hardware on consumer Hardware that you couldn't do before and so in this talk I will talk about four different uh papers and um that is like a one about a specific data type and one about apid optimizers then there is llm and eight that allows efficient inference in 8-bit for a lot from for large Transformers and then my most recent work that is kbit inference scaling loss where we show that 4-bit is actually really good for inference and so before I jump into all these uh papers um I want to give you a little bit of background so this is mostly about quantization so I want to give you a little bit of background about quantization so what is quantization quantization is if we have a floating point or a real representation we want to quantize into buckets and that's very similar to histogram binning so here you have a histogram of a normal distribution and the histogram has 16 bins that's basically a four bit quantization and if you have linear quantization it means integer quantization uh you can see it as a histogram where the width of each pin is equal you slice up the normal distribution in equal parts each with the same width and then you take the middle value of each bin and you quantize all the values within this bin to the middle value and that's how you get a 4-bit quantization a four bit integer quantization now you can have other data types which are sort of non-linear that means that the width of each pin can be different for each um for each each particular bin so maybe you want to have smaller bins around zero and larger pins at the edges so that you um quantize where there are more values in the middle and you have like a more a different different representations for that and So based on where you choose the bin width to be larger or smaller and that's how you can control where you get basically High precision and where you get errors if you see the bus basically overlapping um sort of sticking out of the normal distribution that's that's a representation of how much error is in this bin and so um yeah there are different ways how you can reduce the error based on how you choose your bins and to sort of formalize it a little bit more and we can look at integer for in four data type and floating point four data type um here I have given you a part of it so and these contain 15 values and I only gave you five but um sort of if we don't want to generalize it into a single mathematical form what we want to do is normalize it into the range minus one one and so then these basically values become comparable and with a single implementation that maps from the index to a particular value in the range of minus one one we can represent any data type and um so usually this mapping is done through um dividing by the absolute maximum value um and so that normalizes us into the range minus one one and if we then want to quantize a certain value we can just find the closest Value that's in this list of quantized values between -11 and then we take the index as the representation as a bit representation for the data type and that might sound a little bit confusing let me let's go through an example just to make it clear and just um to basically highlight that this is very general I have a non-standard two-bit data type so um you see the index 0123 and then the data type from -1 to 1 and this time we have like 0.3 and 0.5 so it's non-symmetric and now we have an input tensor 10 minus three five four and so to quantize we follow the the these four steps first we normalize by the absolute maximum value the absolute maximum value is 10 so if we divide all the elements in the tensor by 10. with that we get the tensor into the minus one one range then we find the closest Value from this new tensor this is from this normalized Tangier to the values that we have in our quantization map which is also called the code book and um there we find uh basically maps to 1 0.3 0.5 and 0.5 and now these values have an Associated index that we can also look up in the dictionary in the Mac and so we find three one two two we store that and that's our quantize stupid representation of the input tensor if we want to dequantize it we load the index three one two two do a lookup which values they represent so we just index the values in the quantization map then it's 1 0.30.54.5 and now we need to denormalize so we multiply by the absolute maximum value to reverse the first operation and so then we basically recover the dequantized value and in this case you see we have a large error so the minus three turns into a three and this basically means our quantization data type was not good for this input um and that is sort of a big consideration if you do quantization what data types do you use how do you construct your data type um just to sort of demonstrate this further and if you look at floating Point data types they're made of three parts one bit is for the sign then you have seven bits and they can distribute it in different ways you can have exponent bits which are basically two times uh the power of an exponent um and the exponent is basically an inch so two to the power of an integer represented by the exponent and then we can have the fraction um which is um the fractional part um and so we multiply the exponent by the fractional part we then get a floating Point number and so um if we have this we can for example take more exponents for more bits for the exponents and that helps you to approximate large numbers well and very small numbers well but because your fraction has relatively few bits um you don't get as high of a Precision for certain numbers you can also allocate more Blitz for the fraction and then you get high Precision for values but now your exponent is very small and you cannot approximate a good range of numbers and so depending on the inputs certain combination of exponents and fractions is best and um what we later see especially in 8-bit optimizers um certain times we need to make certain trade-offs that are more complicated and so um what I designed um is a data type that has a dynamic exponent and it works like this so it still has three parts and the exponent of the fraction and the sign but now it has an extra part an indicate a bit and so how it works is you've assigned it then the first is zero bits indicate the exponent and then the first bit that is a one is an indicator bit and this indicator bit indicates that all following bits are for the fraction and so what you can do with this indicator bit as you slide it from left to right you can determine how much how many bits you use of exponent and how many bits you use for the fraction and so what this data type allows is you can approximate very large and small value as well and you can approximate um very large values of high precision where this data type is not good is sort of intermediate values that have intermediate size and and then the Precision is low and so um this is a data type I developed and that will be later important for input optimizers which actually follows now and so um if you have any questions um so far for the background let me know I can answer them as I go through the next slides um but yeah we've talked now about April optimizers and so the main motivation is if you look at training and we have this setup we have the input gradients which depend on the model size but also other factors like the sequence blank so over batch size but then we have some memory this is just determined by the model size these are the weights the gradients uh the main weights if you have mixed Precision training and then atom Optimizer States uh if you use if you do a language modeling often you have use atom Optimizer the add-on Optimizer has two buffers and each buffer is just as large as the weight and usually an Optimizer sets a 32 bit so um they take up quite some bits of memory and so with apid optimizers we can reduce it by approximately 40 percent so now we reduce its 32 atom buffers to um 8-bit and then they are much smaller and so that saves a lot of memory and so um um if we try this this doesn't quite work and why doesn't it work um we have outliers in our atom buffers and so it looks a little like this so here you have because then again a 16 um 16 value histogram so a two-bit quantization but now you have an outlier -10 and so what it does because we have absolute maximum normalization is basically you have a bin with a single value at -10 and then all the bins up to -3 are empty and then the other pins are sort of filled and so what it basically does is a lot of bits are wasted because they don't have any values only certain bit values have um basically values allocated to them and so this increases the error so so this is basically no longer um a four bit quantization it's more close to a three bit quantization and so you basically lose a bit of representation if your data type isn't well aligned with outliers and so to overcome this problem what we do is we um We Shrunk the atom States into blocks and we treat each block independently and quantize it independently and so if an outlier is in a certain block then it will have an effect on the absolute maximum value and with that on the quantizations but because everything is independent the other blocks will be unaffected so with this we can isolate problems of outliers into particular blocks and so the smaller the block size the more we can isolate outliers with this 8-bit optimizers become much more stable and so um to put everything together um with if you have these blocks and we learned already about quantization and we use this Dynamic quantization data type that introduced so um what you basically do is if you want to update in optimize the state um you have the state you shrunk it into blocks you find the absolute maximum value normalize it by Edge find the closest it bit value exactly like in our example now you find the index and now you can store it in eight bit and if you want to dequantize it you will basically reverse the steps and you take the index values undo the normalization undo the blockwise normalization and then you have the optimizer state so you can update again and so you repeat this Loop over and over with that basically you have an 8-bit Optimizer that is very compact but um that is actually as good as a 32-bit optimizer so this is a very large table lots of results probably very confusing uh the main takeaway here is so we have like um glue we have image snap classification we have nutrients translation we have um emissionet pre-training we have language modeling multiple and we have mass language modeling and for all of these we basically replicate 32-bit performance with paper optimizers so we can reduce the memory footprint on the very right column quite considerably and at the same time get the same performance and it's also a little bit of faster which is sort of a little bit of added bonus um yeah and so um optimizers work um and um if we sort of ablate the components we find that we need everything that we should have introduced the dynamic uh exponent data type we need blockwise quantization then also um stable embedding layers which is sort of more described in the paper um but yeah that we can have a bit Optimizer States and that makes fine-tuning Mass more accessible 8-bit optimizers are widely used for people that have a single GPU particularly if they want to fine-tunes for example stable diffusion I'll maybe just pop in here for a second because we do have a question before we go on to the next one um is the computation efficient on mapping onto hardware for non-standard data types so yes um it is relatively efficient so so um what you want to do is basically if you have April optimizers is um you de-quantize the values in cash and then do the operations and 32-bit or 16-bit in cash and that's relatively fast it only overhead is a decanterization and so the dequantization is basically a lookup so if you have your index um like here so here we have basically the step four that's the main overhead quantization and cash is very fast but if you de-quantize if you load the value and dequantize that it's a little bit slower and so what you want to do is basically a lookup and a lookup if you have parallel Hardware can be quite complicated so basically you do a lookup in a table of 256 values and that's how you do decantization and that's a little bit slow on gpus but as we see um if it optimizes are still faster than a little bit optimizer and so um yeah there's a little bit of overhead but because you have also a smaller memory footprint you actually gain also some speed and so overall there's speed up sale so we have a couple more questions maybe I can quickly go through them and so um the absolute maximum values are stored with the tensor so you need the 8-Bit tensor and the absolute maximum values and so you need to do both to do the decanterization um but the absolute maximum values are relatively small so you have like one value and four if you have a block size of four thousand so every four thousand elements in The Matrix has one absolute maximum value so they're relatively small can be loaded quickly um exactly so um the question is um if you don't lose accuracy um doesn't mean we just don't need 32 bits the answer is yes um we uh as I show in the next paper and we can get away with eight bits for inference um I follow up work where we show you can get also away with 8-bit during training and you can actually also get four bits or lower doing inference and um um work that probably comes us the next couple of weeks we also show you can fine tune and four bits so without losing precision so neural networks don't need super high position at the end of the talk also more information about this um but let me dive into the next project this is llmn8 and this is allows you to do inference on consumer gpus more efficiently so if you have opt 175 billion which is the largest open source model or not also llama 65 billion these are quite big models and it's difficult to sort of fit them on a single machine and so if you need multiple machines that you need to have fast networking that's very expensive eight GPU machines are very expensive consumer cheap machines have usually four gpus and that and so um if we can get away with less memory we can sort of hit these Milestones of having inference on a single machine with eight gpus or even on a single machine with four gpus for example with this method it's easy to do llama inference on four gpus so you can do it on a single consumer machine and so um yeah um as I said um if we compress things we can have with 8-bit we can have the model in on one machine it's just half the size and um if we have sort of a little bit smaller model like llama we can fit it on four gpus which is a consumer machine and so um the first thing that we did in the project is uh we took the best eight method 8-bit method for quantization in the literature and we applied it and so what you here see is a shot of the model size on the x-axis and then zero short accuracy across uh a couple of zero shot tasks and um you see the 16 bit Baseline in green that just increases um so the larger the model the better but now an Apert method um it's the best 8-bit method Works quite well but then at 6.7 billion the performance drops dramatically and then the performance drops further to random performance basically I have from 13 billion parameters and so something is wrong um at this Mark of 6.7 billion parameters and this was the main problem in the project that we tackled our what we tackles was basically find out what is going on there then fix it and then get full 16-bit performance and that's what we do in this paper and so what we found is the main problems are outliers again similar to the 8-Bit Optimizer papers outliers are a big problem but outliers are also important you cannot just remove them and so just to highlight what happens if you have outliers so here we I have um basically you can see it as a hidden Dimension and the weight or in the input space and I put one outlier that I sort of almost doubled with every iteration and so if you quantize this Vector with the same quantization method then if your outline value is too large and you have some small values in your vector at some point they will be quantized to zero and so zero multiplied by the absolute maximum value is always zero to basically lose that information it's just lost it's gone and so the larger the outline the more information is gone so on the right we see how many values are quantized to zero and so um if you have larger outliers more and more the iso contracts to zero that means if we have very large outliers a lot of information is just quantized to zero we lose that information and this information is important and then our model no longer works and if we have too many zeros we get random performance because the important information is just gone and so what we find for large models is that at 6.7 billion the outliers reach a magnitude of 60. and then a lot of values are quantized to zero and this is the main problem and to highlight sort of what it looks like to find um these outliers and um so so the most interesting part about this project is we found outlines that are highly systematic they are sort of emergent they come with scale and at small models they're almost invisible but sort of as you scale up they come more and more and more invisible and at some point they're obvious but um they haven't been detected before because it's not quite easy to sort of find them in the first place but if you know what you're looking for they're very easy to find and so this is a view of what it looks like if you look um at these um outliers so here we have basically an input batch and we have on the x-axis the hidden or feature Dimension and in the y-axis we have the sequence dimension and so usually because we have layer normalization we would expect a normal distribution so you expect values that are approximately normal distributed like in the left and the right you see already some larger values minus three minus six and minus seven and so as we scale up these values become uh larger and larger and the probability increases so in the beginning it was just 1.5 percent of values would be this large with the outlier and the right column and and 98.5 percent it was just fine which is a normal distribution and so as we scale up these outliers become larger and more common and then if we hit 6.7 billion suddenly almost all the mini batches have outliers the outliers are minus 40 to minus 60 and now it's rare to have a normal distribution with where everything is normal so instead we find we have a normal distribution with just one column having very large outliers and um sort of these columns also increase with a model size and so at 6.7 billion there are six column in the Transformer that have outliers and the interesting thing is these happen in the same dimension for every layer and a Transformer so if you find these outliers in first layer you can say you can point to layer 30 and say like the outliers will be there and you find them exactly at that spot and um yeah the interesting thing is you want the phase shift happens the problem the proportions no longer increase like 75 percent of inputs will have outliers the magnitude slightly increases but if you look at the jump from like 6 billion to a 7.6.7 where the phase shift happens it triples and then it stays relatively constant and so um after the spaceshift something happens and it stays there um and so if we look at the overall data and how it is analyzed um here's what we see we have two plots on the left we have a plot with parameter and billions on the x-axis and then on the y-axis we have the percentage of layers um or tokens that are affected and so in blue we have the number of layers affected and an orange we have the number of tokens affected so if you pick a random layer in the Transformer or a random token in the input sequence what is the proportion or the probability that um this will be affected and as you see um if you have small networks in blue few layers are affected but if you hear this phase shift 100 of laser affected all the layers affected all the time and this continues at scale and it continues for very different models we tried lots of different models Fair seat models opt models we looked at Blue models and we see it in all models it's a general thing in Orange we see the number of tokens affected and also very sort of Peak at 75 percent roughly 75 percent and they stay there and on the right and so on the left it looks like this phase something something subtly changed but if we look in terms of perplexity uh which is logarithmic we see actually the trend is a smooth exponential so it's a smoothly exponential increasing function so this property of outliers increases exponentially but at some point the exponential grows so quickly that it looks like a phase shift and that's also one main takeaway you can detect emergence in Transformers by looking at exponential trends and um this is basically how outliers emerge once they are merged they um remain stable and the other thing as I showed before the magnitude increases rapidly at some point and this is actually exactly the point that um causes these large quantization errors and that makes uh the 8-Bit method not work it makes it break down the 8-Bit method cannot deal with outliers and so um what we did after looking at all of this data is we've found like okay there are these outliers but there are very few of them they're highly systematic because it's just columns we just need to find these columns and what we do is we multiply The Columns and 16-bit and all other values in 8-bit and so all the other values in 8-bit are 99.9 and only 0.1 percent is outliers which we multiply in 16 bit so we have one matrix multiplication for the outliers in 16-bit one matrix multiplication for the non-outliers in 8-bit and then we add them together and if we do that um we get our main output from the matrix multiplication and so um this is still very efficient because 99.9 of Weights are still in 8-bit we just need to store a couple of 16-bit weights and with that we have full 8-bit matrix multiplication and so um if we again look at the graph um now we have in blue our message lln8 uh where we do the sort of decomposition and 8-bit and 16-bit and there we see we replicate the full performance now our model is just as good as 16 bit but it has 8-bit weights and so it has half the memory size and so that makes these models much more accessible and that is how llm intake Works um here's a shot of um memory footprints at different kind of Hardware setups and then the largest model that you can fit and so as we can see um if you have the free Cloud um I think they're no longer free I think Kegel is still free but collabs no longer free but um at that point it was still free and you can fit easily a T5 model of the 11 billion parameters on the free cloud um a pet Cloud um 13 billion um yeah and so um it's an academic desktop with the best consumer gpus you can fit opt 66 billion or basically now llama 65 billion so um that that makes llama models extremely accessible you no longer need a server you can have a desktop and do it so um the evaluation so and we have a question about fuel shock performance so this this um oh I see few shots um few shot is uh basically very similar to zero shock we see um so one very interesting finding from this analysis is also we compared zero shot performance and complexity on language modeling we find that the correlation is minus 0.95 which basically means if you evaluate a language model on perplexity you basically get a zero shot performance and it's actually similar location short performance so you don't need to evaluate the language model on zero shots you just need to evaluate on complexity if you want to compare two language models or two approaches and so um k-sharp performance is also basically the same as 16-bit so we have another question what does effective mean in the context of tokens and layers affected so this means basically yeah the Criterium that we have um basically for this is a little bit more complicated um but um the or the motivation is complicated but the Criterion would be used in this paper is uh we set a threshold of six and say um and a dimension has an outlier if it has a value larger than six because we have layer normalized values um if you have a normal distribution and a value of six the probability should be tiny it should be like 0.000 or one percent or something like that and so by chance you wouldn't expect such large values and we say like if we see such a large value we say this Dimension has an outlier and as I said there are very few outliers so if you have like three billion parameters there like two or three outliers if you have 12 billion parameters here seven outliers or seven outlier Dimensions so very few Dimensions highly concentrated highly systematic a percentage of outliers um above 6.7 yeah I guess I answered it just now so if you have larger models see a little bit more but it doesn't grow quickly um it's um a couple more outliers and in some models you see people report oh I found on my model there are lots of outliers for example an opt-66 billion is actually very unstable and it's standard deviation um increases the depth which is very unusual and so if you find more outliers if you have this Criterion on threshold six but in the follow-up work we have actually a better Criterion which again finds just a handful of outliers in these models and um I can talk a bit more about that a little bit later so the inference speech question the implementations that I have for this paper were designed for training for 8-bit training not in 8-bit inference so this is a 8-bit training project that turned into an Apert inference project and that's why the implementations are a little bit slow but if you implement it right you get a speed up of basically two for inference [Music] um insights why outliers are clustered in a single column that seems counterintuitive yes so um one important thing is that if you want to attend to sort of a single thing because attention comes basically from the same inputs so you take an input you separate it to Key query and value and now you do attention and if you want to to pay attention to a single value now you basically need to separate everything you need to have the sparse output from a linear projection that's very difficult to achieve in a context dependent manner and so what the language model needs to learn in order to do this efficiently is to have some values that are context independent and this is what kind of happens with the outliers so if you look more closely at the outliers um most outliers are in the beginning of sequence token because it's independent of all inputs because all input sequences have the beginning of sequence token and they are aligned in a way that it cancels out exactly those features that are not needed so it basically helps with spice attention and I think I have also no I don't have it here um in the paper we also talk about how much performance you lose and if you remove these outliers in attention in particular and so these outliers say make up most of the probability mass and attention so really important for attention and that's why you see them sort of in single columns um that way it's easy for the attention to cancel out values because it's predictable if you want to cancel out the value and you know where an outlier is you can have the appropriate sign basically set the soft Max to zero for that dimension okay I think we have one more question um does the impact of 8-bit training and on activation function used for the sigmoid so um yeah these papers mostly about inference but I have results on a bit training so what I tried in particular for activations function is a soft Max in the attention layer because the attention layer is very unstable for 8-bit training and so um there if you have different activation functions they are much more stable so um if it's the same thinking like um we will replace sigmoid function with rectified linear units because they have easier gradients and um yeah they are they are if you have a sigmoid you want to saturate it your gradients get extremely large and so that can cause sensibilities and so if you replace sigmoids with rectified linear units you fix this and so I want to do the same by replacing the soft Max with the equivalent of the rectified linear unit that is basically an arc uh what you call an arc Max a soft Arc Max and so if I do that I get much stable more stable training but performance drops so that is the main problem our activation functions that we have they're critical for performance and if you change them you increase the stability that you um decrease training performance and so that is sort of a difficult trade often research often the methods that you work on they prove improve stability but they degrade performance and so um yeah that's that's sort of an active part of research figuring out how to overcome that okay and let me go on to the next uh project and that's a very recent project and that also inspired a follow-up project that that I think uh will be uh quite important and so um this project is about the question if you take um an 8-bit uh language model in 30 with 30 billion parameters and a four bit model with 16 billion parameters they have the same number of bits one has more parameters and uh lower Precision one has less parameters in higher position but overall you have the same bits and the Curious Thing about these language models is the inference latency so how long it takes to do an inference is the same and why is it the same it's because for gpus and computation is very cheap but loading memory is very expensive um particular for entrance and so model gpus can do 200 multiplications at the same time it takes to just use load one element so basically computation is 200 times cheaper than a loading memory and if we look and look at the memory that is used during inference I have your representation for gpt3 I think it is on the left you see a single Pixel that represents the size of the memory due to the inputs the input token and on the right you see the memory due to the weight Matrix and now if loading memory is much more expensive than computation then loading the weight Matrix is much much much more expensive than loading the inputs and so basically all the latency and inference um basically is due to loading the weight Matrix so if you compress a rate matrix by factor of two you get a speed of two and that's basically why the total number of bits in all weight matrices determine how fast your model is and if you have less the natural question is if you have a if you have two different models with the same number of bits which one is better higher precision and fewer parameters or more parameters in lower precision and that's what I study in this in this paper and we do a very thorough study we do 35 000 zero shot experiments on a couple of data sets we look uh between 19 million and actually 175 billion parameters we look across different models opt Bloom blooms python ux gpt2 we look um from three bit to 8-bit Precision we skip two bit because here we get random performance uh we introduce blocking already before we look at how block size improves performance and then we look at four different data type integer float the dynamic exponent I already introduced and another data type quantile quantization which is information theoretically optimal and this is our setup so we try to find out if we have the same number of bits what's the best model we can get how many bits does it have per parameter and this is basically also a question very closely if you're related like how much information um is there per parameter per bit and um This Is The Answer um so here we have a plot on the x-axis we have the total model bits uh um in in these models and the newest models opt models from 125 million to 175 billion and on the y-axis we have mean zero shot accuracy across four or five tasks that I think it was and now we have four Curves in um red orange we have our radish we have 16 bit and now you have different positions uh in 8-bit and green and and yellow four bit in the blue three bit and we can see actually Four bit is more efficient so if you have a 60 billion model and four bits it's better than a 30 billion model in 8-bit so what this plot says is basically you want if you want to have the most efficient model with the fixed size you want to use the largest model that you can fit in 4-bit and that is sort of the answer if we look at free bit what we see is performance degrades the model becomes unstable it's very Jagged and this jaggedness basically means that and instabilities were encountered similar to before and llm and H and now the performance drops to a very low level even random level for some for someone else and so um one actual question is if we fix these issues in free bit is three bit better than four bit and so we use a similar technique like an llm and eight called proxy quantization where we find the outliers and isolate them and we do them in 16-bit instead of um three bit and so here you see the curve blue and three bit and a yellow three bit plus proxy quantization so there the outliers are considered and what we see is four but it's still better so three bits plus considering outliers is not better than 4-bit 4-bit is optimal and if we and use four bit plus outliers it also has no benefit so um basically 4-bit is the best method we can have um you don't need to consider outliers and I mean what I should have said before is the inputs in this case are still 16 bit because it's a small we can have them a high position because almost the entire um inference latencies due to the model size uh to the weights the size of the weights we have the weights and four bits and the inputs in 16 bit and so yeah if we look at cable weights four but it's best um and so if we have a setup we can also study um how to improve this further so now we know four bit models are best if you have a fixed size or a fixed budget for an inference latency you should always use four bits but can we do better and so if we look at block size if we decrease the block size We have basically more quantization statistics the more quantization statistics we have the larger the footprint per parameter so for example if we have a quantization statistics with 32-bit floats and a block size of 32 then we add one bit per parameter and so here we have a block size of 64 and quantization statistics of 16 bit so we add 0.25 bits per parameter and so if we trade off these things what we see is that a block size a smaller block size of 64 is better than larger block sizes and at 64 we're almost at the optimum 32 doesn't have any benefit and so um yeah what we want to use is 4-bit small block size and this improves things further can we even go beyond that and so for that we can further look at data types what data types are good for forward quantization and so um I introduced this Dynamic exponent data type and we have a normal integer data type these don't do so well then we have two other data types of floating Point data type Focus load and then we have the information theoretically optimal data type is quantile and these two data types perform better so what do you want to use is basically four bits a small block size and either float or the information theoretically optional data time and um yeah this you get the best performance and that is what I have for this talk so I talked about 8-bit optimizers and that makes training and fine-tuning um more accessible um it's use particularly of the stable diffusion models and it's integrated in the bits and bytes Library the library now has more than twenty thousand pip installs per day so it's widely used and mostly for the Apert Optimizer but then I also presented LM and 8 and that makes it possible to have large models and use them on a single machine uh for example you can have now a single server and do opt 175 billion inference or a single um desktop computer and do llama 65 billion entrance um and then finally I showed like if we analyze sort of different cable position for inference we actually find four bit as the optimal Precision if you want to optimize your inference latency or your memory footprint always go for 4-bit that's that's the most efficient and um yeah that's that's what I have thank you so much and I'm happy to take more questions all right thank you very much Tim that was awesome and we covered a lot of ground in about 45 minutes so it's fantastic uh thank you everyone for the Applause emojis coming in um while people are digesting the end of that presentation there is okay we've got a couple questions coming in yes it's great uh maybe we'll just start from the top any intuition any intuition behind why the performance between four and three bit quantization is so jarring or rather why it degrades so quickly compared to the jump between eight to four bit yes so it's mostly so I remember we have 16-bit inputs and so um basically and the outliers and the input Matrix and the weight Matrix doesn't have big outliers but now if you multiply an outlier by weight Matrix and it also needs to have higher precision and this what we're basically seeing in these experiments if we consider the outline as plus three bit it improves performance considerably and so with that we know the main problems he outliers some inputs need higher Precision weights than others and that's why it's difficult to go through three bits we can go through three bit if we consider the outliers but it's still not optimal 4-bit gives the best results um yeah and so um certain inputs need certain positions but it seems forbid is enough if we go from 8-bit and go lower um and the performance doesn't change so if I go back to the previous work we show here with 8-bit we can replicate the full performance actually this also works up to six bit and if you then go five bit you have a little bit decoration and four bit is still pretty good and as I show here optimal if you care about model size and 3D becomes unstable um yeah yeah and I I think related to that the follow-up question is any intuition that can explain why 4-bit is the optimal that's that's like very difficult to say um this is this is very empirical but I cannot and I could sort of approve why four but it's optimal um it's it's very interesting um but um yeah it's it's it's difficult to say so for some worlds we actually see that I don't have it I think in in the slides but the free bit can sometimes be as good as four but in terms of overall efficiency so there are some exceptions but it's very regular for most models so I mean and overall we try five models and uh in total like uh basically a hundred different models with different sizes and different architectures and only in like two or three cases pre-bit is as good as forbid so um it's not super regular but I mean at this point it's sort of um very empirical all right so we've got a few more that have come in maybe we'll dive into if we want to go further into 4-bit with fine tuning methodology how can we effectively recover from quantization errors resulting from extremely low bit quantization and we've got some extra contacts there um yeah so yeah if we want to do that um so uh what I show in this work is on the quantile quantization data type this is a data type that basically looks at the empirical in this input distribution of the weight so floating integer are input independent they they just do the same for the same weight and but console quantization is actually optimized for each individual weight and that improves the performance quite a bit on average not not in this case but on average is the best data type now we can go a step further instead of empirically optimizing it for the weight we can imperatively optimize it for the input and the weight and that is actually what gptq does um I think right now it's sort of the best quantization method method um we developed something that's a little bit better than gptq and sort of follow-up work but yeah the key part is if you want to improve further you need to consider input and the weight Matrix together and for that you need to sample some inputs put it through the language model and sort of um use that information but if you have just the way the loan it's very difficult to do better than this um yeah Mr shall I maybe just read the questions or go for it okay so in the next question um is um you mentioned that perplexity is almost exactly negatively correlated with visual short learning performance you think it's possible that there are other facets of emerging llm performance not captured by zero shot performance that may differ even though two months to achieved the same per pack so as usual performance yes so and there's a little bit of a hint um in the data that I have so if you look here on the right plot it shows a smooth exponential increase and so you see even from small models that something is changing something is changing exponentially but if we look at this plot it's sudden and so um this looks very similar to for example arithmetic performance in Palm models I think in their paper they had a plot that at some points is a performance is almost zero at some point performance of mathematical arithmetic increases rapidly and this is and and then they also show how this um um this uh sort of they've shown exponential trend of some underlying dynamics that are related to arithmetic they find a very similar plot to this so um what I what I can say is basically um there are some underlying Dynamics they are difficult to find but they're exponential if you find something that looks exponential you know at some point it will grow so quickly that it looks like a phase shift and so if you find something like on the right you can say something like something on the left will happen and so um and with this um basically you can relate it in that way um so I guess to further go into details there um if you want to relate zero shock performance and perplexity you can look at exponential Trends and subset of perplexity data sets and that might indicate changes in zero shot performance okay then the next question does dynamic exponent data type refer to the floating Point data type of the indicator bit that you've introduced in the first part of the webinar uh if so is is losing one bit from already a small four bit the reason that's performing worse than others um yes I think that sets the main thing so um just to go back and maybe clarify one thing here so the main difference is that we have an exponent to the power of 10 rather than the Power of Two um or the base power is 10 instead of two so we have um sort of a wider range but yeah you're right basically um the the overall performance um is so low because you sacrifice a single bit for that's this plot here a single bit for this indicator and you only have two other bits for basically the non-signed parts and that's just not enough to do really well and so it's just not a good data type and as I said it's good for extremes if they're very large or very small values but it's not good for intermediate values and at a four bits you need to get a good error for all kind of the entire range of data thank you and we've got time from one last one there okay um there is there any reason to believe that the results for Effective quantization schemes for inference may carry over to quantization during training um yes so so um I believe that um the problem is some of these are difficult to carry over um uh sort of training is a different Beast because you need to repeatedly update your weight you need to basically de-quantize it while you're doing matrix multiplication and so um if you have sort of small block sizes they work really well but um um matrix multiplication if you have a different block so it's your animation multiplication of inner products and if the inner product if two vectors are separated by two blocks and then um their multiplication and dequantization will not result in the same output as a normal matrix multiplication and so to do the correct thing you need to dequantize the sub Matrix multiplications and matrix multiplication on the GPU that's very difficult to do efficiently and so it's very difficult to implement efficient training matrix multiplication with a certain block size on the GPU so it could carry over but it's too inefficient to implement that's similar for other data types the contract quantization type is very efficient but very difficult to implement for training so it's good for inference but for training more challenging so in general that's that's the trend um some things work but it's very difficult to implement them for training efficiently and that's the main problem there right thank you everyone for your questions today and thank you Tim for an awesome presentation and uh giving such great answers to everyone going really in depth here uh thanks for joining us today if you want to catch up on this because lots of really great information covered today replay will be up on our YouTube in about a week thank you again Tim thank you so much for joining and sharing today thank you so much all right thanks everyone have a wonderful day [Music]