Deep Dive on PyTorch Quantization - Chris Gottbrath

Key Takeaways

hi my name is Chris Scott breath I am a technical program manager at Facebook and I work with a regu Krishnamurti and the model optimization team on quantization but also with other teams on performance optimization and various different compute backends used within pi torch it's my pleasure today to be able to talk to you about what we've been working on in the area of quantization and review both the basic functionality of quantization that we've been having over the last year or two to my torch and also some of the new things that are available in preview in Pikul 4.6 the outline for this talk will be pretty simple we're going to talk first about the motivation I'll talk about a couple of different forces and factors that are driving our interest in quantization then I'll talk about the three basic workflows that that you'll probably want to use if you want to quantize new neural network model in a torch dynamic quantization static quantization and quantization alaric training and then finally I'll talk about some new features and new capabilities that we've introduced in my torch 1.6 these are graphic quantization which makes it kind of radically simpler to apply quantization to a neural network model and also the numeric suite which gives you a much deeper view into the numeric behavior of different choices in quantization so in terms of motivation the first motivation for quantization is simply that she learning fundamentally it's the machine learning is technique deep learning is a technique that is very computer intensive right like we we know the models neural network models started getting really accurate and also got very large and that trend has definitely continued maybe even accelerated in the recent years so that some of the most cutting-edge accuracy models are also extremely large if you look at bird Megatron and and other kinds of things you'll find that they they have just an enormous number of parameters they're extremely accurate but they use a significant amount of compute resources if you want to train them and if you want to run them for inference so this this is a trend that seems to be continuing and moving forward and I guess I feel say a couple words here but there's probably pretty obvious to anyone who's used let's just say a speech recognition system like Siri or or ask Google as those machines and it's those capabilities and those models have improved the user experience has improved greatly and of course about you but I feel like there's still a long way to go for those systems to be able to comprehend more and more speech more and more accents and more more different noisy environments and so there's there's a lot more that can be done but the experience is qualitatively different as you're as you achieve different Plateau is a quantitative accuracy so obviously as we're trying to provide better better services we're going to we're going to need larger and larger models and that's something we need to we same deal with we think about face book sale scale that can be a huge deal we have data centers all over the world and they are quite large because we are we're serving billions of users and if we start talking about deploying neural network models especially really big neural network models in order to provide features in those data centers the cost and resource usage you know can be can be very omits at Facebook status scale so we're already seeing data center power consumption doubling every year if we start thinking about apply so these advanced models that's going to push that trend up even even more and that's something that okay I guess it's fair to say we're highly motivated to try and avoid look isn't just what happens in the data center though we're a highly mobile in a mobile-first environment and instead of apps and of course we want to put capabilities at your fingertips but that you know that's kind of a hell you know mobile devices cell phones you know they have a lot more memory now than they did before but they're still very small and low power usage compared to data center servers so there's there's a lot of interest on our mobile side as well to try and drive efficiency try and drive model size reduction and try and make sure that we're being appropriately using things like the battery resources in mobile devices so there's a lot of interest in techniques that we can use to optimize get better performance and reduce the energy usage in so the technique I will be talking about today is quantization and with quantization the idea is we're going to use a reduced precision to store the individual values within a neural network and by doing that we can unlock model size reductions and also often much more efficient and faster kernels so this is a technique this technique that I'm gonna be talking about it has opportunities to provide really significant efficiency improvements but I don't want to like make the story seemed like quantization is the only thing to do to improve the efficiency of neural networks there's office there's there's a whole host of different techniques and at Facebook we're pushing on all of these kind of simultaneously in order to pull the hold that energy consumption curve and cost curve down for our data center and to enable more complex models to be deployed in mobile devices so the other tools that were using our optimized kernel so even within a given precision even without thinking about our quantization we can we can optimize the various new for a numerical operations that we use within within pi torch and reduce data center power usage we can also use different hardware accelerators there's this cool things like the Google TPU that are out there there's there's an advances in new generations of GPUs that are available and so we're trying to apply and also on CPU side so we're trying to apply those those new technologies into into the data center and into into what we're trying to do it here at Facebook there's also our research organization fair is is pushing the envelope on efficient network architectures so there are there are things that are available and there are big advances that have been made especially on the on the compet convolutional neural network side with mobile networks and mobile net FB net as net cetera that they simply require less computing horsepower to get deliver similar results for certain workloads so we're pushing on all these fronts at once the quantization is a really important tool in the toolkit so to provide a little bit more context on on how quantization actually works if you think about neural networks they're fundamentally doing a you know boil down in a lot of cases to a lot of matrix multiplication tensor tensor multiplication tensor addition kinds of operations so you're taking lots of floating-point you're typically floating point 32 values you're multiplying them by all those lately 32 values that you're adding the result together and accumulating about in order to create a new new floating point tensor the question or the promise of growth quantization is but what if we approximated that we got an approximate solution by by multiplying into eight value representing the weight tensor and intake value representing the the value in the activation tensor to get the resulting output tensor and and that an take value can be can be basically representative of the floating point value you could have a linear transform for example but we end up so you have a quantities in taint value and zero point and a scale factor you can you can represent floating point value this turns out to work and be actually really powerful so it allows you to to take up less memory on the device because your your tensors now are suddenly being represented with int eight values and every and every point rather than FP thirty-two values that every point so it's simply four times less memory plus some scale factors which which are usually very small compared to the size of sensors and then with modern hardware on the CPU side on the GPU side also on the DSP side all three of those Hardware families are our partners are rapidly adding new instructions and new date new capabilities and and things like that that allow us to really efficiently run compute annotate or in it in FP 16 or be float 16 so so basically on the hardware side and by looking at the way neural networks run we've discovered that we can apply these these approximations and actually still get relatively robust results now you have to be careful in doing it so there's some networks that are really forgiving and it's kind of amazing you can basically just quantize them duper tensor quantization and the end result is almost as accurate as the as the floating-point value and in those cases that's great we can apply that really quickly and easily in some networks they're slightly more demanding they're they're slightly less forgiving of the approximation and so you'll see as I go through the rest of this talk there are some techniques that you can use to achieve a pretty high accuracy even in networks that are a little bit more demanding so maintaining accuracy is can be hard can be easy but for some networks it's hard and we provide you lots of tools in pi torch in order to be able to maintain a better accuracy even in the more demanding models okay so that's kind of the high level like mathematical what are we doing when we're quantizing how do we deliver this to you in pi tour so this three three I guess principles that kind of drive we're doing this one of them is easy views so we're really trying to to make this be as turnkey as we can so we document we've clearly documented the three different workflows that you use and and in each one of them we've tried to reduce the number of steps as much as we can it's okay so there's still a number of steps but we'll talk about that later but this is key three key we're closing we'll talk about here in a few minutes and those are well documented we also recognize that we don't know everything you have to write quantization is still kind of an active area of research and high torches is loved by researchers because it gives you the ability to get getting a tinker with things in Python dynamically so we've similarly implemented quantization invite torch by giving you a lot of tools directly in accessible to you know the workflows will make it easy easy to use you don't have to dig into the details but if you want to dig into the geo set tinker and find your own we have quantizing things if you want to be apply some really fine grained control some things you totally can do that and you can you can push the envelope of quantization research using hydroids quantization and then we this is important to us so we have put I torch quantization into the core of Pi torch so the cue tensor was a fundamental thing in PI George and so its shouldn't feel bolted on or an afterthought or a third party modulus in black tights it's a core functionality within pi torch and weave for the key compute backends that we use a lot like fpga m and Q an impact the quantized kernels are just right there next to all the other kernels within within pi torch excuse me on the call okay let me talk you through a little more detail of how this is organized in its inside of Pi torch so if you're familiar with with with private orchards internals you'll be able to kind of map where you'll find things so we want everything sort of found I'm gonna focus on kind of the right-hand side of this slide here everything is founded on the notion of the quantized tensor so the q tensor in pi torch is basically just like the tensor you know and love except for that there's a scale factor and a bias sort of stored as part of the tensor and you can operate on those q tensors in the same way you would a normal tensor and just have stored in such a way that they are quantized and they can take an advantage of with quantized ops now how do you get it Q tensor well we have torched quantized per and then what the basically conversion you tell you used to create a quantized tensor from a non quantized tensor there's true variations of this which are kind of shown here on the slide there's a per 10 sir quantized or a per channel quantized let me quickly explain what the difference is so when you're creating a quantized tensor you don't need some scale factors because you're going to get a map between the antique value and the FD 32 value it's basically a multiplication by a scale factor now we have one scale factor for the whole tensor and that would be a quantized per tensor and that will give you a single scale factor for the entire tensor and all the values will be if you think of starting with a floating point value you're dividing by the scale factor and that gives you the quantized value right or we could do a scale factor per channel so that's the torque quantized per channel operation will create a tensor which is quantized in a per channel fashion so you now have instead of a single scale factor you'll have a scale factor for each one of the channels so if you have 20 channels then you've got 20 steel factories we've got 150 channels you've got a hundred 50 scale factors pretty simple okay then we have operations that can operate on cue tensors so those live in in the code base as under torch ops quantized so you'll find all that you'll find like 2d coms there the linear there and and things like that now you also have higher level operations the torch studying in module which has the sort of key modules that you use in a lot of under all networks and so you'll find that torch died in quantized and torch done in quantized dynamic contains sort of analogues of a lot of the routines that you're used to using in pi torch to create your models you'll also need some tools to convert to work through the different workflows that I'll talk about in a few minutes you'll see those are those live in PI torch in the torch quantization namespace so you'll see like a prepare and a convert and things like that those are live and torched quantization some examples of those and actually to that I'll just highlight here our torched quantization observer and torch colonization to fake quantized observer is used to choose the scale factors which we talked about before in the quantized answers based on observing the extensive like activation cancers for a television data set and tourists are quantized fake quantize is usually in the quantization we're training to simulate the effect of quantization so these are basically helper helper functions that you use at different points in the workflow so this is also it's worth noting an extensible API so we've got this sort of layout here but you can extend it and create you add your own quantized ops you can add your own your own observers you can also go in and tinker with exactly how you map based on observed characteristics of the tensors during calibration how you actually choose the the quantized values so there's different techniques numerical techniques so you might want to use we implement a few of the basic ones but you can if you can implement your own pretty easily in Python so okay now based on those building blocks let me talk you through the three different workflows and this is kind of the introduction to the main body of this talk so we're gonna we're going to show you in detailed slides how to do apply each one of these three different workflows of a dynamic quantization static post-training quantization or aesthetic quantization aware training when this in this point here I just want to highlight the pros and cons that when you use each one so dynamic quantization this was one you'll use for basically IO bound networks such as illest MS and transformers in this case what we're going to do is we're going to statically sorry ahead of time we're going to quantize the weights so that they never can store all the weight parameter tensors in you know say F P 16 orientate format so in half the space work order of a space as the the FP 32 network plan immediately gives you benefits when you're loading in those parameters and imprints time you're loading in a half or quarter as much memory memory bandwidth the activations those get quantized dynamically at runtime and so the pro of that is that you are always getting a good scale factor are chosen given the data you have it's always sort of fine-tuned for specifically what you what's what the input data set is at that given time the downside is that means you know it has to dynamically make a decision at runtime so runtime has to look at the ring usually the range of the input tensor and figure out what was scale factor and offset will work best to quantize that range of input data she's a little bit more math every time it's transforming the data from for a new q tensor on the activation side but it's kind of like always well tuned for the data so this tends to get good accuracy and it tends to work well when you're the performance is memory bound due to weights stunts post training quantization on the other hand in this case we are quantizing both the weights and the activations ahead of time we're choosing the scale factors for all of those ahead of time this tends to work really well for CNN type networks which are heavily compute bound and where you have you know kind of large input data sets that you want to give to really quickly quantize so the easy open question well how do you choose it's easy to see well it's not easy to see but you it's relatively straightforward to see how you how you choose how you quantize the weights because they're known ahead of time the activations are our novel each time you point a camera everything you've got a new set of input tensors so what we do is we do a calibration process so you'll add a new step to your your model development or deployment workflow where you calibrate the data or so you calibrate the quantization based on some some input data and that's used to determine the scale factors for all of the activation tethers this tends to work pretty well for a lot of a lot of neural networks there are a few networks in particular like mobile CNN's that tend to not give the best accuracy with static post-training quantization so what we we encourage you to do in those cases is to do this quantization aware training and this requires a little bit of fine tuning of a model so it's a little more complicated and a little more labor-intensive but it can often recover a great accuracy for these mobile style relatively fewer parameter CN n type models okay and I'll again I'll take if you didn't follow all that it's okay because when I walk through the details of each one of these in the next couple of slides or just try to give you the overview so let's talk about the first one of these three top row of our table a moment ago was dynamic quantization so this is for your sequence to sequence type models your transformer is your birds your L STM's and your multi-layer will do their email Pease multi-layer perceptrons sorry I stalled building up that name okay so here this is gonna do there's no extra calibration required for this so this essentially is really streamlined so you start to go over the code here on the right hand side you're going to start with a language model you load that model in you load in the parameters for it but this for the state did so now you have a full model in PI George I'm going to pass that into the dining the quantized dynamic helper function in the quant I want is a ssin module and that is going to and you're gonna pass in the the data type that you want to quantize to you like in this case it's a qu in Tate and now you have a quantized model and you know what you're pretty much done you can now use this quantized model and it will run fast and give you your quantized accuracy you can also save this model a good script it so you can run it in an environment for the C++ environment such as what we use in our data center for deployment so this one actually fits on one slide it's a super simple process and you basically just quantize the model and you get a quantized model again underneath the covers what's happening the areas it's gonna look at the weights because it knows them and it's going to figure out what the right scale factor is for those and it's gonna create quantize tensors for those weight tensors and then the activations when you run this for inference it's gonna look at the range of the values and those activation tensors and dynamically choose the appropriate scale factor to represent that data with the minimum loss of accuracy and it will use those it'll it'll apply those skills actors right away so it's it's determining each given time what the right scale factor is this works again well for L STM's transformers MLPs things like that tends to give you we're seeing things like two times faster compute you know one point five two you're somewhere in there and then generally four times less memory because these tend to have lots of parameters okay second workflow post-training static quantization so this is what you use for most CN NS in particular CNN's that run in the data center here I'm gonna have a multiple step process and this will come a couple of slides to describe at the high level what we're gonna do is we're going to make some minor model modifications we here we describe that it's kind of tweak the model on the left then you're going to calibrate the model based on some data and then we're going to convert the calibrated model to a quantized model and then you'll be able to run the quantized model and get the high performance what's happening underneath this covers is that this is quantizing the weights and the activations for the entire model based on the based on what it saw in the model in terms of the parameters and then what it saw for the calibration process in terms of the activations this works really well for for cnn's particular data center type cnn's like ResNet rows next etc this tends to give a faster compute and also has uses less memory both for runtime and storage the calibration calibration prior process you're basically taking basically putting around the quantized kernels like calm 2d you're putting these observers in and they look at the values that are they're coming and they look at the range they basically store the statistics and then they choose the right scale factor and then when we quantize then we'll have you'll have you'll go through a convert operation to quantize the input values and then you'll you'll just use the ante values to into through the entire network so you're not reconverting back to F - 32 between each individual layer so it's one of the more the nice benefits of this post-training static quantization now how do we actually apply this so this is now the first and this would be like a three slide sequence maybe four we're going to just show you they show you the pseudocode for for the mean operations you do when you do this so the first step is you're going to we're going to load in our model over here we're going to load in the weights and we are going to put the modeling evaluation format that turns off some of the training characteristics of the model so it'll replace your batch normalization for example with with a scale factor and so that's what the model eval does and then the next thing we're going to do is we're going to slightly tweak the model with this fuse models sorry fuse modules routine it gives us a helper function in the quantization namespace and in this case we're telling this for example that comp best or Mandrell you we see that pattern we want that to be replaced by a kind of a wrapper module which will be a con value 2d and that con will you is what we actually have a nice fast version of con relic on Braille you for quantization so we want to be able to apply that the only the other thing that you do with this stage is you make sure that you have unique names for various different layers which could be made some like minor module sorry minor bottle rewriting it's kind of kind of routine but you have to you want to do that and then the other thing you do do here is you as you may apply a you can go in here and choose where you want to convert to and from quantized forms you can actually choose to quantize only a subset of the of the whole neural network and you can do that by doing some minor tweaks to the model at this stage of the process okay once you've done the sort of minor model modifications that we talked about in the first slide the second slide we want to set our quantization configurations which is the cue config parameter you can see mentioned here there are a couple of different things that are included in this one is the apply some blanket rules to the pattern matching that happens later on to do basic identify which modules within the model you want to have quantize so you can turn on and off sections by using cue config here the second thing that you can do is you can choose like the exact form of organization that you're taking so you're doing things here like determining whether or not you're going to have a non zero bias or whether whether you have a constraint that it's in case you always want to use a bias of zero those kinds of things are part of the cue config a set of options nice thing is you don't need to kind of know the details a lot of cases so if you're on the CPU my torch on the CPU uses the generally by default use the FPGA fast math library matrix-multiply library and we have default queue configs 4fb Jim so you pull that in that will tell it a set of modules that every job has nice operations for and it will set the numeric parameters to the right thing for FP Jim there's a similar set of default parameters for the queuing impact library which we use on mobile devices so so yeah pull those in and you can you can override anything you want to but you'll at least start with reasonable defaults and then the important step here you want to prepare the model for calibration so you can see the line here that shows the cue model and then we apply the quantization dot prepare function to that and we get a new cue model which has basically observers a placed and around all of the important quantize routines to observe the the activation tensors and see what their ranges are now that we have the observers in place we want to put our Humala into the evaluation mode to make sure that that's that's good so we've done it but we want to do it again here just make sure and then we want to evaluate the model over some some sample data and this doesn't have to be your entire training dataset it can be a subset usually usually relatively small so that's enough but you want basically enough data or the statistics to be gathered reasonable statistics to be gathered for the values of activations in all the different layers so you know having done that it stores the values right into in the in the in the queue model right there in pi torch the print statement here on the right hand side shows what that looks like so this comm rail you 2d which we created in through the fuse module pass you can see that it's decorated with the min map observer and the min min Max Observer has a min value of 0 and a max value of 4.5 5 so it's that's basically recording some statistics about the values of that are in the floating point tensors that are being used for activations as an input to this combo Lu 2d and we'll use that to set the scale factor in the next step and you can create your own observers which might have different statistics if you don't want to do min max you might want to do something which is more statistically based on the on the the centroid rather than the extreme values so you might do something that that takes 80% of the maximum and allowing some things to saturate that's your choice and you can create those observers and implement them okay having prepared the model and then run that which are inserted observers and then you run that over the calibration data set now we have the statistics we need to be able to actually create the quantized model so the these command here quantization convert we pass in the the Q the Q model that has already been calibrated with about the calibration data the convert command does does the rest so it goes through their places all the the operations that have that it can take a pattern match with the appropriate quantized versions of those operations and it uses the statistics that were gathered when the observers and are recorded in the model there to select set the right scale factors you can see when we printed this out here you can see it says quantized congra 2d so it's now using the quantized version of that fused conrail you operation and you can see that there's a skill factor in there and that scale factor was informed by the statistics we gained into the previous step so it has a scale factor of 0.35 and a zero point of zero so I knew to do that because you calibrated the network and the previous step and yeah the end result of this is a is a if you the next slide here just shows in order to use it you just use it right like you've already got a quantized model you can call it right away and it will call into the quantized operations and it will do all the right thing to convert all the tensors into the right format you can also use the JIT script mechanism to to script this so you can run it before C++ imprints so you deploy it to a mobile backend for example and that's all it tastes now it was a few steps before slides worth of steps but they're pretty straightforward you start with a while and you tweak it a little bit to make it more meaningful for quantization you make sure it's in the evaluation mode you do the prepare step to put it into the to put in the observer so it can be calibrated then you calibrate it on some data and then you convert it at the end of the convert step you now have a fully quantized model which you can immediately run in quantized form or you can save it to disk all right so that let's just step back here a little bit so we've talked her way through the two of the three workflows we've talked about dynamic quantization that all fit in one slide that you use that one for LS TMS and for transformer and Burt type models the second set of slides were the post-training static quantization which is generally what you will find pretty good behavior for CN NS that are used in data centers and things like that the sometimes you will encounter a CN n that just doesn't give good accuracy when you do post training static quantization in those cases then you want the next technique that you can reach for is called quantization aware training the idea of quantization aware training is that we're going to fine tune the network in such a way that it learns how to give the correct answer when it's inputs and weights have been quantized so basically we're going to do we're going to go through of fine tuning or training several epochs where we're touching up the weight values such that when they're a leader quantized they're gonna do better job of giving the right result so basically the network learns a little bit how to work better it's been quantized I believe it also will do back propagation on choosing the scale factor so instead of just mathematically determining what the scale factor should be we can actually learn those as well I'm not a hundred percent sure on that but I think maybe we do as well in any case the result of this is that you will give a model which often has higher accuracy closer accuracy to the FP thirty-two value even though it's using quantized inputs how is it how do we do this we do this by basically rounding the values in a way that emulates the quantization so if you think about intake quantization you basically you're choosing 256 values you only have twenty four six value vocabulary for all the activation and weight values so we can emulate that way and basically have the same behavior if we continuous the news floating point but we round them such that they are rounded to two to one of 256 discrete and evenly-spaced values and then that will that will basically give the same the American characteristics as if it had been quantized but it's still floating point so you can still do back propagation in the normal way so now let's show you how you actually apply that it's actually very similar to post turning quantization except for the instead of calibrating you instead doing fine tuning so this looks should first part here should look very similar you're starting with your ResNet you're loading in the parameters then instead of turning it into evaluation though you want to leave it in training mode so model dot training that says leave yourself in training mode and then you want to use the fuse modules to again fuse ACOG Bachelor and rel you and you can see here it's putting it's creating a calm bachelor all your 2d with all three of those different layers and then you mix you know similar make sure you have uniquely named layers and things like that okay next slide here you're going to prepare your network just like we did before so again we get our model cue config for FB Jim but we now you three now I usually get default qat keep it Vic because we want to have the quantization aware training version of this and then we want to prepare I'm going to use the qat version of a prepare command which will instead of putting in observers it will put in fake qualm nodes so now we're going to be able to then subsequently train the model and have it in basically around the values that it's using so that they look like they're quantized and that's what the next line done does here we train our model using training data and the instead of training normally it's going to basically use these skate quanta nose to around the values that they don't looks like it's been quantized as you can see this is a print out of the values and you can see that this have been decorated now with with these state quantize notes they have their they're accumulating different parameters as they are doing the rounding and the rest of the workflow is pretty much the same so once you have done this once you do that so you prepare and now you train instead of calibrate once you have the trained cue model you then just do a convert and then you have a quantized model so instead of being informed by the observers you informed by the data that's gathered by the fake quanta nodes which give you the ranges and the scales and things like that and it'll create a quantized model but the training process is also changing the weights in the model so this so this cue model now not only is quantized it's also been fine-tuned so that it will run better in a quantized fashion so that's the third of the three models so let's talk about let's look at what the accuracy looks like for each one of these so let's look at three different types of models in different domains so resonators are as are frequently referenced data center convolutional neural network you a standard FP 32 accuracy for that is seventy six point one when we apply post-training quantization to that using the work flow that I should just showed here you might see an accuracy job at point two that's that's becoming so when you get for that accuracy drop you get the runtime performance the latency dropping from 214 milliseconds to 102 milliseconds and the memory use dropping by the about a factor of four modulo some scale factors so you use a quarter of the memory and you're used half the time and you lose 0.2 of your top five top one accuracy sorry top one accuracy okay so that's pretty awesome I'd say that's in many many times that'll be a win right now maybe you maybe you're the really really accuracy constraint state and you don't want to drop point to percentage in the back you see but in many case that'll be an acceptable trade-off mobile net be - now this is one where we we find that we need quantization of where training and on my next slide I'll do an ablation analysis so you can see you know how much benefit you get if you did with just post training quantization but in any case with the best technique quantization we're training applied to mobile node V - we typically see 0.4% of accuracy loss on the top one accuracy and here looking at what the performance looks like on mobile device like snapdragon 835 we see that you can run in France in 80 milliseconds that would have taken 75 milliseconds so that's almost four times faster so that's pretty awesome you get four times faster you get a quarter as much space and you only have to give up point four percentages of points of accuracy Bert this one dynamic quantization with per channel quantization actually gives you pretty much identical accuracy results so zero drop in accuracy it's 90 point tube for 90.2 after and it runs in a you know not quite twice as fast on the server with the batch size of one he runs in three 230 milliseconds instead of 581 milliseconds and again in terms of memory usage it should be a quarter of the memory usage modulo some scale factors so it's a pretty big benefits for relatively small drops in accuracy the quant highest torch occasion models are available we've gone through almost all the torture vision models we've created quantized versions of them I think you've even uploaded quantized reference weights for those and those are available right in torch vision so just look for a duck look for a quantization subdirectory of the models directory and you'll be able to see all those and use them right okay so I mentioned that we would show an ablation analysis and last slide so here's that relatively quickly so this is mobile net be to embrace that the winning technique was quantization aware of training and you dropped 2.4 percentages percentage top one accuracy but if you were to only have applied Herr Chancellor quantization you would have dropped six percent accuracy again I mentioned this before but I'll just quickly touch on it again there's two ways you can you can there's two forms that cute tensors can take the simplest form is the per tensor quantized and there we have a single scale factor and offset for the entire tensor so you have all your different channels all your different heights and with values and you basically put all of them in a bucket you look at what the right scale factor is to express all of them and then you use that scale factor you apply it to all of them that would give some good results for some cases some cases you want to use per channel quantization in that case instead of having a single scale factor you have a single scale factor per channel so you have you have 100 channels you have a hundred different scale factors and each one is tuned to the data that that we observe in that channel or that we will you quantization or training that you train and you see happening in that channel so that's the difference in per ton serum per channel you can see that for tenser is dropping 6 percent per channel is dropping a 5 percent so it's better but it's still not awesome quantization aware training where the network learns how to give you the right answer even the presence of quantization removal that V 2 seems to be necessary and it gives you a much more respectable 0.4 percent accuracy drop so you can see kind of the relative importance of each one of the different techniques okay again kind of what we've the patter we've seen is that for data center type like full-size CNN models usually post-training quantization with per channel is with or without per channel is enough to give you good accuracy with some of the reduced parameter mobile type networks where they are I guess they have fewer fewer ways for the signal to propagate through the network maybe maybe maybe those seem to be the ones where you need quantization where training that's the this is sort of pattern we're seeing I don't know that we can promise okay so with those slides I've gone through all the current capabilities in in a choice quantization you've got a lot of tools in your tool kit let me talk a little bit about some of the stuff that we're building to make things even easier so you will remember that when we went did post-training quantization it was several steps you had to prepare your model I talked about different up different things you have to like manually fuse your kagra you in order to to get good good behavior we'd like to make that go away that's a it's a routine thing that you should have to apply to every single model we should automate that so what we were doing then we were doing something called graph mode quantization so here what you do is you script your model first so you can see the first row in this instruction series to take your model run torch it script on it you get a scripted model what's happening underneath the covers when you do a scripted model is we actually construct a graph for that model execution so in this case I torch Iger is my torch is generally an eager mode framework which defines the network by running and it does not try to construct a network a graph ahead of time representing that network for scripting and running it these networks efficiently on on deployment hardware we do create a graph once we have that graph though we can use our graphing localization to modify that graph so the graph mode quantization flow starts by scripting your model creating that graph then you can use prepare underscore JIT and pass it in a model and you'll get a it'll automatically apply all the things you need to do to preparing the model you don't have to go through and manually use your modules or make sure that you have unique named modules or things like that you can just use prepare JIT and I'll take care of all the details and you can do calibration same as the normal workflow and you do convert underscore JIT and it will automatically convert it into a quantize model and it's already been JIT just hid so you can just save it you don't need to do secondly a JIT process this is work work in progress we think it's stable enough for people to use it's a prototype feature but we're still tweaking around the edges so if you'd like to use this then you you go to the pie charts download mechanism where you would normally download the stable release and instead check nightly and you'll get a nightly build the most recent nightly build and that nightly build will have this capability alternately even just go and clone the github repository for pi torch and build it yourself if you're building off of master you're gonna have this a graphical quantization available so either building yourself from github or go ahead and download the Knightley's if you want to take advantage of this and then the other new feature which is in prototype is our numeric suite so we have found in some cases when we're working with kind of tricky networks that you want to go in and maybe turn off quantization of a particular layer or make a choice between luck I use FP 16 or intake for this particular network for this part of the network for those kinds of analysis we found that it's really helpful to break down and basically look at the accuracy impact of different choices on a layer by layer level so the numeric suite are some some tooling that allows you to do that it will basically compare the intermediate outputs between a FB 32 model and basically you run the model in the numeric suite and it kind of automatically runs the model both in floating-point and in qantas quantized form and it allows you to compare the results between the two I think it actually runs it in both and it propagates forward the FP 32 values and it reak want Eliza's it passes it into the quantized so you can see on a layer by layer basis exactly how much each layer is being how much accuracy lost these potentially sort of building up in each layer of the network it's really cool technique it's available right in again a Knightley's it supports static and dynamic post-training quantization we're working on quantization we're training it's not there yet but that will come in the future right so I've covered now our two new features I've covered the basic three workflows this this legends you an idea of how much what coverage we have the different operators so I think the the key operators you need for CN NS for transformer style models for RN n type models and for multi-layer perceptron tank models are covered here but we definitely don't have every single operator and PI torch so there are some gasps do let us know if your if you have a network that falls into one of those gaps and also feel free to contribute quantized implementation so you'd like I've mentioned these two backends several times previously in the talk I'll just kind of reiterate here that we have quantized support in both FB gem for x86 CPUs and also q an impact for our arm platform so for mobile mobile PI torch is quantized using all the same workflows that we've talked about here for datacenter hydrogen right and here's my final slide thank you looks like we're about 15 minutes into the recording so took a little bit longer than I thought but I just want to reiterate that height which has been we've been working on quantization since version 1.3 we're at version 1.6 now so it's had a little bit of time to stabilize and bake I think it's very much ready for for you to use we're using a lot of close with the end Facebook we have full documentation available in in the regular places and on PI torch we also have quantized tutorials walk you through all that if it workflow as we've talked about here we also worked with the torch vision team to make quantized versions of all of those key you know frequently used towards vision models available basically that means you don't have to figure out exactly what when we said the prepare model step you to take that step you're using a rest net or a or Google net or one of those you grab those from torch torch vision and and and you can start from there you can modify it if you use a lot of a variant of one of those networks you can look at the at the torch occasion wants for reference to see what the quantization changes where we made and then I just wanna reiterate that we have these two new porn pipe features and one point six one of them is numerical suite which gives you better visibility into how how you model is responding to quantization and then the other one is the correct mode quantization which is a big step along the path of making us all very extremely turnkey for a wide range of models so thank you very much it's been good talking with you and I encourage you to try out quantization and definitely give us feedback on the forums or through other channels as you do so thank you very much