SLMs on CPU | Amazon Web Services

Key Takeaways

Hello, I'm Nolan Chen, partner solutions architect at AWS. And I'm Andrew Wal. I lead the field engineering team at RCI. Andrew, earlier you talked about how we can run SLMs on CPUs. Forgive me, but that sounds a little bit too good to be true for me. Could you talk more about that? Absolutely. And that's the natural reaction is these models require so much parallelization that the idea of running them on CPU almost does sound too good to be true. But in fact, we have run use cases that have worked really well. So let's talk about why that is. So we'll have SLMs on CPUs. And there's really three primary reasons why this has been able to work. One is that SLMs are continuously improving and what I mean by that is we are seeing drastically improved quality of models at lower parameter sizes as time goes on. One clear example is at RCAI we released a model about 6 months ago that was 70 billion parameters. Two months ago, we released a model that was 10 billion parameters that outperformed it. Wow. We're talking about a model that is seven times smaller and outperformed it in only a couple of months. And at that rate, we're able to get drastically improved performance for models at smaller and smaller sizes. Second is CPU architecture improvements. architecture improvements. There we go. Over time. And specifically what we're looking at here is the improvements to instruction sets within the CPUs that allow them to utilize deep learning frameworks. This has been able to improve serving uh frameworks such as VLLM, Llama CPP and others. And then finally, there have been improved quantization techniques. And what this has allowed us to do is run models more efficiently at smaller sizes which allows us to ultimately get them down to run effectively on CPUs. Okay. I think I heard you earlier say talk about how you guys develop models that went from 70 billion to 10 billion parameters. Is there a magical number or sweet spot where in terms of parameter counts where you can actually start running it pretty effectively on a CPU? Yeah, absolutely. And we found two primary model sizes right now that work really well. The first one is an 8B model that has been quantized to four bits as well as a 4 billion parameter model that has been quantized down to 8 bits. And this has been a really good sweet spot in the middle between what is still able to run on a CPU and still produce accurate and reliable results that are useful for a business. And can you tell us again what quantization is? Absolutely. It's the idea of being able to take models and running them at a smaller size so that they can have improved latency, but also and what's most important in this use case is a smaller memory footprint. And what's important to remember here is that when we're talking about models that are less than 10 billion parameters, we've found that quantizing down from 16 bit all the way down to 8 bit has virtually no impact in overall accuracy. And quantizing all the way down to 4bit has very minimal impact when we're talking about models of this size. Got it. So to keep some people from getting too carried away, can you remind us though what are some of the limitations of trying to do inference genai inference on CPUs? Absolutely. So when you're thinking about what impacts you're trading off when you're running on CPU versus GPUs, one is going to be how many parallel requests you're able to do. So typically when you're running SLMs on CPU, you're going to use a batch size of one. So you are very limited on the throughput that you are utilizing. Additionally, you are going to have much greater constraints on the context window for your model. So you are providing less context and smaller throughput. But if you remember you have that massive massive reduction in cost and that is really where the benefit lies. Can I ask you now just give an example of a real world use case where it might make more sense to run an SLM on a CPU. Absolutely. So think of being able to run a model in a grocery store. Let's say that a company wanted to have a model that they ran where cashiers would be able to look up inventory levels, for example, and they wanted them to be able to interact in a natural language form there. That's very simple. The information that's provided to the model is what the inventory level is for the product that they're asking about. And you're only having a few requests every once in a while. Let's say that model actually runs locally on the cache register or the computer that they're running there. And that is one that would make sense because again very limited throughput, small context size, and you're able to run it very efficiently from a cost perspective. Wow. So, who knows, maybe in the very near future, an SLM will be at every checkout the grocery store potentially. But that said, being that we're AWS, we're always trying to find more cost-effective ways for our customers to run their workloads in the cloud. What would be a way to take advantage of running SLMs on CPUs in AWS? Yeah. And one of the great things that AWS provides is instances with a really good cost to performance ratio. And specifically, I'm talking about Graviton. So, we actually did have an experiment where we ran this model here, our 8B quantized down to four bits on a Graviton 4. And we were actually able to achieve upwards of 30 tokens per second when running this model on a Graviton 4. Wow. So AWS customers today can start at least experimenting and running SLMs on Graviton. Absolutely. Awesome. Thank you, Andrew. Thanks, Nolan.