Bringing Generative AI to Life with NVIDIA Jetson

Key Takeaways

hello everyone welcome and thank you for joining us today I'm Dusty from Nvidia and I'm really excited to be here and share some amazing advances with you for bringing the latest generative AI models and llms to the edge since the introduction of Transformers and then the Amper a100 GPU in 2020 we're all aware of the hockey stick growth that's occurred with the size of the models and their intelligence which feels approaching that of a human it's been a huge leap forward in a relatively short period of time and things have really seemed to hit warp speed with the open sourcing of Cornerstone foundational models like llama and llama 2 there's a huge Community out there of researchers developers artists hobbyists all working furiously on this day and night it's incredibly exciting and the pace of innovation is hard slimpossible to keep up with there's a whole world out there to dive into and discover and not just llm but Vision language models and multimodality and zero shot Vision Transformers alone are a complete game changer for computer vision thank you to everyone out there who's contributed to this field in some way or another many have been at it for years or decades and thanks to them it feels like the future of AI just kind of poof arrived overnight well the time's now for us to see the moment and bring this out into the world and do some good naturally due to the extraordinary compute and memory requirements running these huge models on consumer grade Hardware is fraught with challenges and understandably there's been comparatively little focus on the work done in deploying llms and generative models locally outside of the cloud and even into the field and embedded devices yet in the face of it all there's a growing cohort of people doing just that shout out to our local llama and our stable diffusion at the edge Jetson makes the perfect platform for this because it that has up to 64 GB of unified memory with Jetson agx Orin and 2048 Cuda cores or 275 ter Ops of performance in a small power efficient form factor why bother though why not just do it all in the cloud well the reasons for that are the same that they've always been with Edge Computing latency bandwidth privacy security and availability one of the most impactful areas that underlies the other application shown here is human machine interaction action or the ability to converse naturally and have the robot autonomously complete tasks for you as we'll see you really need to be Geared for latency especially when real-time audio and vision are involved and of course anything that's safety critical it would also just seem good to know how to run this stuff yourself while keeping all your data local and fortunately there's a massive compute stack openly available for doing just that we've been hard at work for a while now on Jetpack 6 and it's easily the biggest upgrade that we've done to the Jetson platform software architecture which is now underpinned by an upstream Linux kernel and Os drro that you can choose and we've decoupled the version of Cuda from the underlying ALR tbsp so that you can freely install different versions of Cuda CNN and tensor rt2 we provide optimized builds and containers for a large number of machine learning Frameworks like P torch and tensor flow and now all the llm and viit libraries too there are worldclass pre-trained models available to download from too NGC and hugging face that can all be run on Jetpack with unmatched performance along with Edge devices like Jetson we're bringing more components and services from Metropolis to Jetson for video analytics with deep stream and we just released Isaac Ross 2.0 with highly optimized Vision gems including slam and zeroc copy transport between Ross nodes called nitrous for autonomous robots jetpack 6 will be out later this month and supports Jets and Orin devices and going forward should provide a much easier ability to upgrade in the future with that let's dig into the actual models that we're going to show you how to run today first up are open vocabulary Vision Transformers like clip alvit and Sam that can detect and segment practically anything that you prompt them for using natural language Then llms followed by VMS or VIs language models multimodal agents and Vector databases for giving them a long-term memory and ability to be grounded with real-time data finally streaming speech recognition and synthesis to tie it all together all of this we're going to run on board jetsen or so we've optimized several Critical vits with tensorrt to provide real-time performance on Jetson these have higher accuracy they're zero shot and our open vocabulary meaning they can be prompted with natural language expressing context and aren't limited to a pre-trained number of object classes clip is a foundational multimodal text and image embedding that allows you to easily compare the two once they're encoded and easily predict the closest matches for example you can supply an image along with a set of complex labels and it'll tell you which labels the most similar contextually without needing further training on object classes meaning it's zero shot clip's been broadly adopted as an encoder or backbone among more complex vits and vision language models and generates the embeddings for similarity search in Vector databases likewise alvit and Sam or segment anything are also used clip underneath um alv alvit is for detection whereas Sam is for segmentation and then uh over here efficient vit is an optimized uh viit backbone that can be applied to any of these and provide further acceleration again these are using uh tensor RT and it's available today in Jetpack 5 and of course will be in Jetpack 6 as well so let's dig into some of these we have a demo video here showing the capabilities of Al vit so you can see you could just like type in what you wanted to detect and it will start producing those bounding boxes should it find them previously you would have had to capture your own data set annotate it train a model like SSD mobet or uh YOLO on your uh training data set and U it would have a limited number of object classes in it well you know alvit was based on clip which just has a huge amount of images and different objects in it so you can query it for practically anything here and it's a real GameChanger not to have to train your own models for each and every last uh detection scenario that you want to do this is a really impressive technology to be able to deploy in real time on jeton getting up to 95 frames per second on HX orm so not only that but when you combine the detections from alvit with a secondary clip classifier um you can do further semantic analysis on each object so here you see within each um detection Roi it's doing further subd detections uh in Brackets means you wanted to use alvit and in parentheses that we'll see in a second you wanted to use clip so this is not dissimilar to primary secondary detection pipelines that we've done in the past but in this case it's all you know zero shot open vocabulary and much more expressive um so you can see here not only is it detecting these different objects but it can also classify them individually um so happy phase sad phase other types of things like that and you can perform some very powerful detections and subclassifications this way with just writing a simple query on your video stream um no code required even so all the code for running lvit in real time is available on GitHub here it's called the Nano project um because we've taken the original alvit models and optimized them with tensor RT which is how we took it from I think 0.9 frames per second to 95 uh and there are various different backbones that you can run different variants of alvit that have um more accuracy less accuracy higher performance lower performance and this also is a very simple P python API where you just put in your image uh put in your text prompts and it spits out the bounding boxes and their classifications for that so I highly recommend that you check out this project on GitHub if nothing else uh that you take away today because object detection is still by far the most popular computer vision algorithm run and this can completely revolutionize that so the segmentation analog of this is called Sam or segment anything model and it it works very similarly uh basically you just provide some control points uh like click on different parts of the image that you want to segment and it will automatically segment all of those different uh blobs for you no matter what they are it used to be that you would have to manually go and make a segmentation data set train the model on that and those segmentation data sets were very resource intensive um to to annotate uh but now you know you can click on practically anything and it'll segment for you and when you combine it with tracking there's another project called Tam or track anything and that can do segment anything over video for you we also have containers for this available um on Jetson that you can take and and run today so everything that I'm going to cover in today's presentation is available online what we call the Jetson AI lab which is a set of tutorials that we put together that can easily guide you through running llms vits VMS Vector databases the whole thing and all these tutorials are backed by pre-built Docker container images that we have for all these packages and you can take those containers combine them mash them together build your own custom versions and build your own applications with them and deploy them that way so it it takes a lot of the guest work and bugging from porting what normally are GitHub projects for x86 and discret gpus and porting those to arm and AR 64 to be able to patch all that and you don't have to worry about getting Transformers library to run and so and so and we have a lot of different packages here available at your disposal and some really awesome tutorials to follow and you can chat with your own llama models you can run l avva and all the vits that that we cover here and also stable diffusion too stable diffusion is really a lot of fun to to do image generation with okay so now we're going to jump into the large language models here is a chart of the llm performance that we currently have on Jetson uh this is for agx overin and it shows a bunch of different apis that we've exhaustively built tried and optimized with Cuda and it turns out the fastest ones are currently awq optimization out of the MIT uh San lab and also the M mlc TVM project out of Carnegie melon so these two are by and far the fastest llama C++ is also very popular use that too uh in the community but mlc is about twice as fast and you can see here we can run everything from lamus 7B up to the full 70b uh at interactive rates and we just showed the Llama models here but you can pick basically anything from hugging face that you want to run and run it through these and U do quantization so these are all running with 4bit quantization so they're much more memory efficient which is how we get them to fit in the memory capacity uh cuz the weight alone for 70b can be up to you know 140 GB if you're doing doing fp6 uh but if you 4 bit quanze it you're down to like 35 GB or so um so here's a video of me chatting with llama 270b hi llama how are you today I'm doing well thanks for asking how about you is there anything you'd like to chat about or ask I'm here to help with any question I actually gave you a voice with text to speech and automatic speech recognition so we are actually talking to one another wow that's amazing I'm glad to hear that we can communicate in a more natural way how's the voice synthesis and speech recognition working for you so far is there anything you'd like to chat about it's working really well you're running here on an Nvidia Jetson embedded board that's interesting in viia Jetson boards are powerful and efficient making them a great choice for EDI applications are you using any specific software or Frameworks to run the a models on the Jetson board and how are you finding the performance of the board so far that's funny that you already know about it yes um you are running on the GPU in real time that's impressive so this was using Reva for the ASR and TTS and runs in time on Jetson agx oin as we'll see in some later videos you know the smaller mod smaller models they generate text way faster um but uh it's just like a lightweight web gooey that I put together with web sockets that's Geared for low latency and it takes the mic from your web browser transmits it to your Jetson locally uh over websockets and then sends the TTS audio back to it so so obviously being at the edge most of these Jetson based embedded systems have cameras or other Vision sensors attached to them and as such everybody in the community is very interested in Vision language models like lava mini gp4 and a lot of these multimodal embeddings that are coming out and how they all work is essentially they use an embedding model like clip that will combine text and images is into one common embedding space where contextually you know concepts are very similar so if you have a picture of a dog and the word dog in this multi-dimensional embedding space those two vectors are found very similar location to each other I.E they convey the same thought or sentiment to the llm and then after that embedding is complete it in the case of lava it uses literally the same clip in viit encoder uh that that we mentioned previously there is a small projection layer that maps from the clip embedding Dimensions into the Llama embedding dimensions and then they also fine-tune the Llama model um to be able to understand uh those combined embeddings more and what we found is if you use the larger clip model that uses 336x 336 resolution instead of 224x 224 it's able to extract much smaller details and even read text out of images and there are lots of other uh image embeddings out there it's a very active area of development like image bind for example which combines way more than just images and text that can do audio um inertial measurement units Point clouds uh all types of modalities uh that the llms can be trained on and be able to interact with essentially what we're doing is enabling the llms with all the different senses so they're able ble to you know assimilate a holistic world view and a perception World model um so they can greater understand things rather than just have to do it all through text uh so you can see here the performance of the lava models is very similar to that of the base llama models is is actually the exact same model architecture it's just a a smidge slower a few te tokens per second slower um because it turns out that these image embedding are a lot of tokens like 512 tokens for a 336x 336 image embedding or I think it's 256 tokens for a 224x 224 embedding so and all of those tokens come at the beginning of the the text chat so every one of those you you know your text input is at least 512 tokens long as opposed to just uh a blank normal text chat starting at zero tokens so that's why it's just a slightly slower but otherwise it gets all the same performance so with the latest lava 1.5 model that came out that some of you might have seen it's got some really exciting new features in it including the ability to Output constrain Json format so you can essentially tell it you know detect certain objects or I want you to pick out this and that and give it to me in Json so you're able to parse it programmatically and actually do things with it and whereas the vits you needed to prompt with specific things like I wanted to detect a face or uh hands or whatnot in this example you can just tell it to detect everything and it'll spit out bounding boxes of all objects or whatever it is that you're trying to do and this will be very powerful for making closed loop visualizations that are deployed in real time embedded systems for example um smart traffic intersection crosswalk monitors or blind assistant devices or um you know B B crib monitors or anything that is an open-ended problem that you want to extract information from without having to train all your own models to do that in addition to things like autonomous navigation being able to query it on like Hey where's this path go are there obstacles in my way are those obstacles Dynamic all types of of things like that uh and this is really exciting another point to point out is this whole model lava 1.5 is a huge improvement over the previous versions of it and but it was still only trained in one day on eight uh a100 gpus so if you collect your own data set it's actually within the realm a possibility to fine-tune one of these yourselves uh for your own applications so here's a video demo of chatting with lava 1.5 on agx [Music] oin what kind of fruits do you see in the image I see a variety of fruits including oranges bananas grapes and apples how much do the peaches cost the peaches cost 2.50 how much do the oranges cost the oranges cost 1.50 what breed of dogs are in the picture there are two dogs in the picture one is a Husky and the other is a gold retriever where are they the dogs are sitting on a rocky Hillside likely in a mountainous area what does this sign say the sign says no Outlet private road what kind of trees are these the trees in the image are pine trees where's the path go the path is in the woods surrounded by trees and vegetation what kind of bird is this the bird in the image is a robin who has the ball the boy in the black and red shirt has the ball what game are they playing they are playing soccer can you describe the image the image features a young girl riding a brown horse in a Corral the girl is wearing a red shirt and is positioned on the horse's back the horse is Galloping around the Corral showcasing its agility and strength the scene captures the excitement and joy of the girl's horse riding experience here's a look at some of the example code that powers these demos and it's basically a lightweight wrapper that I put around mlc and awq since those weren't supported in uh some of the other um llm libraries out there and in in addition to all the multimodal embedding management stuff which which we'll talk about um but it's got a very simple API for text generation essentially you load the model it will quantize it for you if it's not already done you create this chat history stack uh and then you can append either text prompts or images to that it will automatically perform the embeddings for you whether uh you know depending on what data type uh those inputs are uh and then it generates a stream of output tokens so everything we do here is geared for real-time streaming so you can get uh that data presented to the user as soon as possible um and then you basically just output the the bot response to the chat history too now if let's say you want to do your own chat management dialogue you can totally do that you can just pass in strings of text uh to the model generation function what you should be aware of is that the the chat histories work best when you keep a cohesive stack going because then you don't have to go back and constantly regenerate every single token in the chat for example we know uh llama 2 models have a Max token length of 496 but if you were to generate the full 4096 length chat every time it would take really long instead you can keep that cached in What's called the KV cache and if you do that uh between requests then the whole state of that chat is kept you can run multiple chats simultaneously but it's highly recommended to keep the chat stack flowing as opposed to going back and forth and just assimilating it all from scratch every time and the reason for that is because there are two stages to llm text generation the first is the decode or what's called prefill where it takes your input context and has to essentially do a forward pass over every token in there and it's a lot faster than the generation stage but it still adds up when you're talking full 4096 tokens here um so you can see if we're running you know llama 70b on a full 4096 token length chat it'll take 40 seconds to pre-fill that whole thing that's before it even starts responding but if you only pre-fill the latest input you're looking at you know a fraction of a second that's typically like the dot dot dot that shows up in the chat or like agent is typing what it's actually doing is pre-filling your input into it before it can start generation so this is why managing the KV cache between requests is actually crucially important to keep a very consistent chat flow going going likewise here's a look at the token generation speed and how that varies with the input length that does decrease slightly as well once you get up into the higher token length so that's something to account for as well so obviously a big concern with all of this is what are the memory utilization requirements and that along with the token generation speed are what really drive everyone very heavily towards quantization methods so a lot of these llm apis that we talked about like llama C++ and others Auto gptq X Lama they have lots of different quantization methods you can go everywhere from Two Bits to 8 Bits or um most of the time below 4 bits you start to see degradation in performance but at you know uh Q4 a16 quantization I've not really seen any difference in output from that which is which is is really good because it takes llama 70b from being 130 GB of memory usage down to only 32 gigs and that is much more Deployable uh for Jetson and likewise for the smaller Jetson is too you can run llama 27b on an or 8 GB board or llama 2113b on the or X1 16 gigabyte and we we can see that here that we have a whole lineup of different and modules uh that you can deploy and each of these conveniently has um a typical model size that is well fitted to its memory capacity uh so as I mentioned the 7B models are good fit for the over Nano 8 GB the 13B models are a good fit for the ox 16 gigabyte and so on and so you can basically mix and match the level of intelligence that your application requires along with its performance uh and other swap see uh requirements like size weight power and cost of the embedded system that you're deploying and be able to pick the Jetson module that's that's appropriate for you to deploy those so a few slides ago I showed some code that was basically a low-level API for doing text generation with the llms once you start getting more complicated and adding things like ASR and TTS and retrieval augmented generation and all these plugins the pipelines get very complex and if you're only making one bespoke application you can absolutely code all of that in like one python application yourself in fact the first version of that llamas speak demo I did was just like that but eventually when you start iterating make different versions of that let's say I want to have one that's multimodal or I want to do a closed loop visual agent you're going to start having a lot of boil plate code in there um so I've written a slightly higher level API on top of that text generation uh function that you can Implement all these different plugins in it's very lightweight it's very low latency it's it's not it's meant to get out of your way and make this all easier instead of harder without sacrificing even one token per second of generation speed and with this you can very easily chain together all of these different text and image uh processing methods and use them with other uh apis as well uh so these are just two basic examples of what the pipeline definitions look like and it can be a completely directed open-ended U multi-input multi-output graph uh and get some pretty complicated setups going in there so another cool aspect that this enables is what we refer to as inline plugins or the ability for the llm to dynamically generate code for apis that you define to it for example how is it supposed to know what the time is or do internet search queries or what the weather is or perform actions like turn the robot left or right all of those core platform functions you can Define in the system prompt and explain the apis to the llm and when it needs to it'll Dynam ically call that this is good to do on top of retrieval augmented generation which we'll uh talk about in a second because it doesn't just retrieve the based on the user's previous input it can do so as it's currently generating the output and insert that into the output as it's going on that is a good benefit of you know maintaining um lower level access to apis as opposed to you know just going back to the cloud for everything cuz you need the ability to stop token generation run the plugin and then insert that into the output and then continue the the bot output in addition to doing things like uh token healing or implementing guard rails uh and guidance functions things like that all of that is very good to have very good uh granular access to the Token generation so you can stop it right when you need to and then restart it completely asynchronously um so you don't destroy all the low latency pipelining that you have this is just a basic example here of when I was toying around with this to see if it actually could generate it and this was just with the Llama 27b using its baked in code generation abilities and this is like basic stuff for it I do recommend using Json format even though it's more verbose and results in more tokens uh especially if you have functions that have multiple parameters with them because Json allows it to keep the order of the parameters straight so it doesn't confuse them if you don't really have parameters or just very simple like the example shown here you can just do like a simple python API and there are other open-source framework uh plug-in agent Frameworks that you can use like Lang chain and hugging face agents and Microsoft Jarvis that do this as well they're not quite as geared for low latency low overhead which is like why I've uh gone and done this but um at at the same time those can do U very similar things um too in fact the hugging face agents API has the llm generate python code directly which then it runs in a limited sandboxed interpreter um so it's able to like interface directly with python apis which is which is really cool and is is like not not hard to do in this case you know I prefer just to keep it Json or text and like manually parse that and call the plugins um so you know it's not having full access to to python so I've mentioned retrieval augmented generation a few times and that is big not only in Enterprise applications where you want to index uh a huge amount of documents or PDFs and be able to have the llm query against those uh because remember the context length is Max of 496 with llama 2 or there's a lot of you know rotary embedded encodings that go up to 16k or 32k or even more uh but there's always there's always a limit uh and you might have hundreds of thousands of pages of documentation that you want to index against and when we start talking about multimodal you can have huge databases of thousands or millions of images and video that you want to index against and not all of that can be included so what happens is basically you search the you take the user's input query and search your vector database for that it uses a very similar Technologies to llms and the vit encodings in fact clip embeddings are are used in a demo that I'll show you next um but it essentially uses similarity search to determine what objects in the database are closest to your query and it's it very similar concept um to the multimodal embedding spaces how that all works and there's some very fast libraries out there for this uh called feice and Rapids WRA that are able to index like billions of records and retrieve them Lightning Fast uh based on your queries and those are are very good libraries to use and I've used those on Jetson here um to do a multimodal image search um Vector data atase I basically made this demo to prove out the abilities of uh the clip Transformer encoder and just to be able to understand what I could actually query for retrieval augmented generation before integrating that into the llms so you can see here you can um not only quer with text but you can do image searches as well and it's pretty Advanced image search and it's completely real time uh uh real time refresh here uh running on jet in this is um indexed on the MS Coco data set 275k images from Ms Coco that took about I think it was like five or 6 hours to index the whole thing uh but the actual um retrieval search only takes on the order of like 10 or 20 milliseconds which means it's not going to add lag to your uh llm generation pipeline which is very important because we don't really want more than uh a couple seconds of lag between the users's query and the response especially if it's a verbal um communication ongoing so here's a chart of the um retrieval augmented generation uh search based on how many items are in your database and some of these databases can get really huge especially with uh like uh corporate documents things like that at the edge I think it'll it'll be smaller because you only have so much space available on the device but you can see here it's just on the order of of milliseconds for most people's applications and I also break out different embedding Dimensions here too so some of the higher end embeddings like image bind use uh every single image or text as a 1024 element Vector that describes it in this uh multi-dimensional embedding space clip uh uses 768 CL clip large uses 768 um so then that was the one shown in the demo here so this scales very well I think it would be pretty rare that you would get up to 10 million um entries on an embedded device like Jetson but hey if you know if you're doing lots of data aggregation have 30 HD camera streams coming on it's entirely possible and is still only like you know um on the order of a fifth of a second or less uh to do all that which is is completely reasonable all right so tying it out with Reva to you know coordinate how I actually made these demos with the Texas speech Reva is an awesome SDK that's openly available from Nvidia that incorporates state-ofthe-art audio Transformers that we've trained along with tensorrt acceleration uh and it's completely streaming it does streaming ASR and TTS you can do 18 ASR streams in real time or 57 uh TTS streams in real time on Jetson AG GX Orin and nobody nobody's really going to do that many streams at the edge um unless you have like multi microphone devices or setup something like that but what it does mean is when you're only doing one stream you're you're going to take like less than 10% of the GPU to be running all that which is great because that means that our llm token generation rate is only going to go down by less than 10% there because the llms they will consume 100% of the GPU all that you throw out with with them uh and Reva has lots of different ASR and TTS models that it chips it also has neural machine translation and I've seen some people do some really cool demos with this where you can do live translation between different languages and it turns out that uh you know a lot of the llms like llama are trained in English um there are some llms out there that are multilingual but if you're working with an llm that's trained in English but you want to be able to Converse in other languages you can use noral normal machine translation in your pipeline uh to essentially translate between ASR uh the llm and then uh the llm output back into the language that you want before it goes to the the TTS another really cool thing that Reva has is these what's called ssml expressions for TTS so you can speed up or speed down or change the pitch or ADD in like emojis or laughs or all types of cool things to make the voice sound more realistic and and overall it sounds really good for just being done on an edge device locally none of the demos that I've showed you so far uh rely on any Cloud compute or offboard compute whatsoever you can run these entirely without any internet connection once you download the containers or have your application built here is a pipeline block diagram of essentially what the interactive verbal chat management looks like and it turns out there's a lot of nuance to live chat back and forth mainly the ability to interrupt the llm while it's outputting we know these llms they like to talk and they will just keep going uh and you can instruct them to be very concise in their output but in general they they will like to to Ramble On A a little bit and it's important like in the video to be able to speak over them and have the llm either resume if it turns out you didn't want to query it or you know stop itself when you ask it another question and the best way that I found to do this is a multi-threaded asynchronous model where there's just a bunch of cues everything is going into these cues and being processed and you need the ability to interrupt and clear those cues based on things that happen so for example um the Reva ASR it outputs What's called the partial transcript as you're talking those are in the videos where you see the little bubbles come up and it's always changing because it's constantly redefining uh with beam forming what you think uh it what it thinks you said but then when you get to the end of a sentence it does What's called the final transcript and that is that final transcript is what actually gets submitted to the llm um but if the partial transcript starts rolling in and you speak more than like a handful of words it will pause the llm and if it doesn't get any more transcript in like a second or two the llm can resume speaking if it does go final the if the ASR transcript does go final then you know that previous llm response is cancelled and a new one is put in its place which is important because then you don't want it to keep spending time generating an old one um when you are already on to answering your your next question so there it turns out there is a lot of nuance in here and we Tred to do it with the least amount of heuristics as possible because that's just lends itself to Corner cases uh and in general it's really nice and fun to be able to chat back and forth with these models and I really highly encourage you to go onto the Jets and AI lab download these containers start playing around with the different models and discover their personalities and um you know go from there build your own applications and I think before long we'll see these out there on robots in Real World embedded system so yeah let's uh let's all do it together if you need any help or support at any time um I'm always available on GitHub the forums LinkedIn my email Dustin f nvidia.com and we're all out there to to help each other and keep it going um so yeah it's great to be part of this community and thanks again so much for joining us today um at this point we're going to get set up for Q&A so if you haven't already please feel free to to type in your questions if you are watching um a replay of this um yeah feel free to to ask questions uh another way on the forums or or GitHub or anywhere all right thanks