LLMs vs SLMs: A developer's guide + NVIDIA insights

Key Takeaways

Hi, it's AA here for Real Terms with AI and we're on the road today. I'm here with Jay from Nvidia. Jay, why don't you tell us a little bit about yourself? Hi there. I'm Jay from Nvidia. I'm a senior developer advocate for large language models and help developer accelerate their uh LLM application. Thank you so much for answering some questions for us today. We're going to talk about AI because it's real terms for AI. One of the things I hear a lot from our audience, from devs I talk to is my boss said I have to add AI to it. What now? I would say just focus on the uh application like what's the use case for and there are plenty of uh ways or uh techniques or toolkits available the in the LLM ecosystem. So uh for example uh is it a a industry specific application like healthcare or finance then I would want to fine-tune and large language model to understand our lingo or that specific jargon of that industry where is that application running is it u on device uh at a industrial location or is it running on robots then I think using small language models would uh benefit or yeah so it's basically depends like try to identify what's the use case and what are the uh problems that have been already solved in the ecosystem and then use that tool. So what I'm hearing you say is we need to start with the use case which is something we say all the time on this show. If I understand my use case where do I go next? Are there particular tools like it feels like there are a ton of tools these days. Correct. Are there any that you particularly like? Yes. Yes. Yeah. I would like to like encourage everyone to just look at the open source or just use open source tools and frameworks because those have been really great at coming up like some state-of-the-art demos as well as uh toolkits to build that application and few uh uh frameworks that I would like to highlight are like hugging face which basically helps helps you download uh model weights and also fine-tune model. There's also Nvidia's open source Nemo framework which helps you uh train, fine-tune and deploy models as well. Uh but yeah, I think just look at open source. I would start with hugging face if if I just want to start. We've talked a little bit about on the show and I know you know some stuff about small language models. What the heck is a small language model? Right. So traditionally large language models have a really high uh parameter uh count. Basically the bigger the model the more accurate the answer but sometimes your use case demands that the intelligence is required at the edge which can be a laptop autonomous robot or maybe an industrial machinery. So in that case it's really hard to use those large language model uh and to tackle that we have small language models which are orders of magnitude smaller than uh LLMs and we are talking in terms of like 8 to 10 billion parameters. Oh wow. Okay. It's a lot smaller. Yes, exactly. But uh they help you get the job done by uh running all the uh LLMs locally be it for image understanding or just text understanding. Okay. So, so what let me sum that up. Make sure I'm understanding. Sure. Um so a small language model is a large is a language model that is orders of magnitude smaller than the LLM we think of when we think about generative AI. And that means that we can run it on device, whether that device is a robot or potentially a phone or something like that. Um, and because we can run it on device, that means that we can have significantly improved response times because we'll have lower latency. Um, and also it means that since the model stays on the device, there's potentially advantages for keeping the data local as well. That seems like it'd be really cool for some use cases and probably complete overkill for others. Yes. Yes. uh some of the use cases for that is diagnostic devices in healthcare which basically requires uh image understanding for the scans and the GPUs is there on that system. So everything is local nothing leaves out of the system. Privacy is intact. It's real time. So helps with the use case. That's a really cool use case. Okay. So we understand why we would use one. What are why would you not use a small language model? Yeah, I would not use a small language model uh in a use case where uh the accuracy of the responses are highly required or u really important. So uh for an example, if you're solving a math problem, it requires stepby-step uh reasoning and u uh execution. So in that case, I I wouldn't think SLMs would be a cool use case or good use case. uh I think you are better off using a large language model which has higher probability of getting the answer. That makes sense. So there's going to be use cases where having the extra parameters of the large language model gives us some advantages and there's going to be use cases where the advantages of a small language model in terms of ondevice runnability uh lower latency are better and I assume you can probably fine-tune those small language models as well for specific use cases which will increase their accuracy on those use cases but not necessarily on general use cases. Got it. Cool. I understand. That's awesome. Thank you for explaining that to me. One of the things I get asked a lot is, "But what if it's non-deterministic?" And I'm like, "It's it's an LLM. It is nondeterministic. Otherwise, it'd be really boring." Yeah. Um, but that makes people nervous. And it makes people nervous for completely reasonable reasons. Like, we need we know we need guardrails around this. We've all seen AI say some really, really weird stuff. What kinds of tools are out there to help developers? Um, what kinds of guardrails, what kinds of techniques, what do you know if someone's worried about, you know, putting some guardrails around their LLM based application? So there are few tools in the ecosystem where uh you can basically add guardrailing to your application which basically uh limits your uh app or LLM to only talk about that specific topics. And there are a bunch of tools uh there's one tool from Nvidia it's called Nemo Guardrails which basically uh has pre-built tools to add those guardrails uh onto your application and there's also Google is doing great work with uh recently announced Gemma 2. Uh so earlier it was just um trying to look at text but it also does image understanding. So any image which isn't uh in the context of the use case can be flagged and maybe uh given uh appropriate responses for that application. So you're saying that there are tools that out there that exist. you gave us a couple examples. And the other thing you were hinting at there is that we can also potentially have the LLM provide guardrails on or have the model provide guardrails on the model by saying, "Hey, flag this if it doesn't fit." Yes, that's neat. That's a cool technique. I know that when we do that stuff though, there's potentially some trade-offs between speed and latency and accuracy. We're talking about this with, you know, ondevice models versus models in the cloud. Okay. How would you help someone kind of think through that? you know, accuracy, guardrails, speed, latency, all of the trade-offs you have to make to build an effective application with AI, right? Yeah, that's a tough question to answer because uh if you want to deploy in the production like there are so many parameters to fine-tune for uh but I would say just uh if you are just getting started stick with uh basic optimization uh like quantization. So there are good open source tools like WLM, Olama is also good. There's also tensor RT from Nvidia uh which helps you do that. I would say just focus on quantization. Quantization is a technique where you basically reduce uh the precision of a large language model from floating point 32bit or FP16 to down to integer 8 or integer 4 which basically reduces the memory consumption while also keeping up the accuracy that a higher uh uh pre precision helps with. Okay. Okay. You got you got a little mathy there on me. So, I'm going to repeat back what I heard. Okay. So, what what you're saying is that we can reduce the data type from a high precision float to a smaller integer and that will reduce the size or speed. Help me out here. Yeah. So, uh that will reduce the uh memory consumption of the model while maintaining the accuracy. Oh, neat. Yeah. Good trick. So what you're saying is that it depends which is one of our favorite phrases here on real terms with AI but that there's some basic optimization techniques and you gave one example of red of changing the data types that can help improve the uh performance of your application without sacrificing too much accuracy. So you work for Nvidia. Nvidia is kind of known for this stuff. Is there anything in particular that you're excited about that's either recently announced or you know is coming that you can talk about with us today? because I'm curious what's coming up, what's new. There's two things. Uh one is hardware uh the new 50 series. So basically it helps uh developers to uh fine-tune or just uh work with LLM applications locally and the second thing is the open-source uh work that Nvidia is doing in the LLM ecosystem. So there's Nemo framework for fine-tuning uh models and then there's tensor for optimizing your uh LLM applications. So those two things I'm really excited and uh really excited where developers will take this and use it in their application. The new hardware is going to make it so that people can test things and you know have a more realistic dev environment which will hopefully that should improve dev cycle time for folks because they can run it all locally right which that should mean that building AI applications gets easier and faster. Yes. Yeah. I think it empowers developers with uh the AI uh superpowers basically with your execution and also the code writing. So what about tensor RT? What are some of the things that developers you think will be empowered to do with that? The main thing Tensor RT helps with is model optimization and inference aspect. If you're writing code and if you need some AI powered coding assistance, you can basically have that running locally uh with the help of Tensor RT because it helps you optimize uh so that it can work real time on your laptop. Okay, so making sure I understood um Tensor RT is going to help folks optimize their models which should hopefully help them make smaller application specific models that can then run on device. Yes. So within just few lines of code, you can see model uh performance improve by 3 to 5x. Wow, that's some those are some stats. Neat. And that's open source. Yes. Oh, that's so cool. I'm now excited. I'm going to have to go check that out. We'll see if anyone who watches the show comes up with something cool and they can share with us. Awesome. Well, let's close this out. Uh I want to thank you for hanging out with me today, answering my questions, letting me stop you to redefine stuff when I was confused. I appreciate your patience with me. Uh, we have a tradition here. We always end with a happy prompting. So, will you give me and the audience here a happy prompting? Okay, cool. Happy prompting. [Music]