NVIDIA Dynamo Developer Office Hours 7/10/2025

Key Takeaways

Okay. Can you just want to make sure that you can hear me? Okay, great. Uh yeah, so thanks everybody for joining. Um this is our our first uh Dynamo office hours uh about LLM inference. Um really appreciate uh anybody who who who took the time to come out. Um this is kind of a new format for us. So um you know, please uh bear with us as we kind of work out some of the kinks and and make sure that that everything runs smoothly. I know we started a little late today. Um but yeah, uh basically the idea here is that you know every uh every couple weeks um we're going to get together. We're going to um answer any of you guys' questions about Dynamo. We're going to talk a little bit about what's new, what's uh you know coming down the pike in terms of uh new features for Dynamo and just anything um about LM inference and uh you know what we do here in at NVIDIA to to accelerate that in general. Um so my name is Neil. I have with me Anishh. Um I am a technical marketing engineer on the Dynamo team and Anishh is a product manager. Um and uh yeah if you guys uh you know have any questions feel free to put them in the chat or or in Discord and um we would we would love to answer them. So uh yeah I'll hand it over to Anish and and we'll we'll get started. Hey everyone. Uh thanks for tuning in. Um I figured that I actually have some questions for Neil uh that I think would be useful for the audience to hear. So I'm going to kick it off with a couple questions to start off. Um, and then from there, again, like Neil said, feel free to drop in your chats and uh we defin or def feel free to drop in your questions in the chat and we're happy to answer. Um, but you know, we've released a ton of new updates over the last month since our last office hours. The biggest one in my opinion being the the completely new UX that we're overhauling for Dynamo. Um, and Neil, I was wondering if you could talk about like the new UX V2. Um, what exactly has changed, why we've gotten that, and uh anything you want to share about the the whole new UX? Yeah, definitely appreciate that. So, um I I think uh you know, one of the things that that we're really trying to to do with uh with Dynamo is make it really easy and make it um you know, sort of uh like like a good user experience in terms of launching these really complicated inference graphs, making you know uh very sort of uh integrated graphs and make you able to do a lot of different things um on the inference side uh in in a straightforward manner. Um and so when we uh when we started Dynamo, we had this concept of um basically having this this command line interface um with with a couple of different uh actions that you could take. So you could we had um Dynamo run which was um a system for basically launching a single component. We had Dynamo serve which was launching a whole graph of components um into a single node. and then Dynamo build and Dynamo deploy which was for taking uh for packaging up a bunch of different um you know components of your inference graph and then launching them into a Kubernetes uh cluster for production inference. Now um one of the pieces of feedback that we got is that you know having having these different verbs was a little bit um it was a little bit complex for people to understand what the differences were um and which ones that they needed to use um at any given time. uh and also it made the um development process um a little bit more complicated because they had to reason about uh you know these sort of extra layers of of abstraction. Um and so what we're doing is is we're um sort of simplifying that um that process for uh deploying components into a um into an an inference graph. And um I have some slides here uh if I can figure out how to uh present them. Let me see uh share my screen. Oh, it's all good. Couple couple techn Okay. All right. In the meantime, while you're figing out screen sharing, I kind Okay, try it. Yeah. Hello. Can you hear us now? [Music] Can you see something? Hello. Hello. Oh, somebody said yes. Uh, there's Vicram says yes. I'm trying. Yeah, just uh hello. Yeah, so uh Neil was mentioning that he was going to bring up some slides. Okay, perfect. All right, we finally got the technical issues figured out. Um, so Neil was kind of going into detail about uh some of the UX changes that we made, which is the question I asked. But to take a step back, I wanted to chat a little bit more about some of the concepts that he mentioned, components of graphs. So Dynamo itself is a system of components that makes up the Dynamo runtime. And this is actually how you then serve your Geni workflows that involve these inference graphs. Uh this could be serving a model like llama or deepseek or actually a graph which could be a complex uh you know geni workflow. Um, so some of the feedback we got is customers quick enjoy the the quick ease of use to quickly spin up something uh in the Dynamo runtime using our CLI commands. Uh, but the problem that we saw was with the abstraction. It made it very hard to actually adjust the knobs and dials to adjust for example the tweak number of prefill workers or decode workers you had or number of replicas or the underlying um runtime etc etc. Uh and so what we've done is basically reformatted our two workflows directly through uh Python 3 uh serving of components or through the Kubernetes CRDs. Um and I think Neil here has some uh really nice slides to kind of showcase uh these changes and what they actually mean for our developers and customers. Neil, you want to take from here? Yeah, definitely. Uh let's just uh Zach, can we make sure that we see the slides? We're good. We're good on the slides. All right. Thank you. Okay. So, um there's there's a couple of uh you know uh yes, let me full screen it. Uh how's that? Great. Thanks. Okay. So, um one of the sort of you know more um uh you know basic um workflows that that you can deploy with uh with Dynamo is this kind of simple um uh you know pre-fill worker, decode worker um uh setup. So it's uh basically you have this this front end that kind of listens for those um you know HTTP incoming requests. You have a router that distributes um incoming requests to the actual LLM workers. And then we have this disagregated serving setup where um we're separating out the prefill workers and the decode workers. Um and so previously you know as as we had mentioned there were a couple of different ways that you could deploy this um with Dynamo. So if you wanted to use the Dynamo run command, then you would deploy each of these components individually. And then we we also had this Dynamo serve command which basically um you know allowed you to take all of these components into sort of an inference graph and then deploy it all onto a single node. Um one of the one of the issues though is that when we started moving to um multi-node setups uh the um the abstraction of Dynamo serve became a little bit more um uh a little bit more difficult to follow because Dynamoserve was only able to deploy things onto a single node. Um and uh you know we we could have gone down the road of um you know having Dynamo serve start doing multi-node uh you know deployments and orchestration of components across multiple different uh servers but we realized that you know it would be better to um rely on Kubernetes to uh you know handle those sort of multi-node multipprocess orchestration. Uh and so what we've done is we've uh we've moved to a new way of of launching components. So if you're launching them on a single node, it's going to look um like uh what we have here on the left in this in this bash script um where you're you're launching each individual component um with with a separate sort of um invocation of that actual Python script and then when you want to launch into Kubernetes um you can write this YAML file with uh you know the Dynamo graph deployment uh that that lays out what the different um you know what the different components are and some of the different arguments for those components. So this is kind of stripping back some of the um the layers of abstraction that we had uh that that were adding some of the complexity for um you know people to to understand what was going on. Um and as we sort of see how people um use these new components and deploy their new inference graphs um you know we can we can start thinking about ways to make that process a little bit easier and a little bit simpler um you know based on uh based on what everybody's doing. So um that's just a little bit about uh yeah some of the the changes that we've made to to the component deployment process. So I'm super curious. You know, a ton of developers and people have already built uh you know, inference graphs and Dynamo workflows using the previous, you know, 31 versions and everything before that since the GTC launch. Uh how do you actually go and change your existing workflows to this new paradigm? What changes you have to make? Yeah, great question. So um if you were deploying things uh with Dynamo in sort of the the the previous versions uh one of the things that you would have to do is um create these YAML files that were configurations for each individual component and then you would also create this kind of graph file this uh you know graph.py file that kind of uh link together all the different components into into an inference graph. Um and you had to manage both of these uh both of these files separately and um you know there was some uh overlap in terms of the the different configurations and the different settings that that you would be putting in each of those different uh in each of those different files. So now what we've done especially with the when you're deploying into Kubernetes is everything is consolidated into this one file. So um instead of uh having separate you know YAML files for your um your component configuration, these are now just going to be arguments that you place for the um the invocation of uh your of your component. Um and then uh for your um you know for the graph for like stitching together the different components, what you're going to do instead is just lay out um those components as resources in the um in this in this YAML file uh that you know sort of just like lists out what all the different components in your in your inference graph are. Um and so it's it's more um you know it's a little bit more verbose. Um, but it's also, I think, a little bit easier to understand what's going on and reason about what the actual um, what that actual graph process looks like. So, what if I'm not a Kubernetes developer? What if I don't want to deal with an orchestration layer like that and I'm more of a Python person myself? What can I do there? Yeah, good question. So if you if you don't want to deal with Kubernetes, right, like um we're you know, we're still going to keep uh the ability to launch each of these components separately um you know using you know just like like a Python 3 invocation command as you can as you can see in in the um in the presentation here. Um, and so if you want to, you know, deploy things on multiple nodes or deploy things on, uh, you know, like slurm or some other kind of like ECS, different kinds of orchestration systems, you you'll basically just need some way to, um, you know, to to to launch these like Python 3 commands and launch these different um, scripts on your your various different nodes. Um, and so and and and we're definitely still thinking about ways that we can, you know, make that a little bit simpler. Um, you know, right now it can be a little bit, you know, a little bit cumbersome to like SSH into every single node and then, you know, launch them, uh, launch each component individually, um, if you, you know, if you're not using Kubernetes. Um, and so, uh, that's definitely, you know, something that, you know, we're looking for feedback on. Would love to to, you know, to to hear if you guys have any ideas about, you know, ways that that can be, um, a more, uh, a more convenient process. Yeah. And I think taking one step below that uh actually uh creating the components themselves and registering them with the Dynamo runtime has become a little bit clearer too because we've gotten away from this like class-based abstraction and now allowed you to register uh very easily with the Dynamo runtime. You can now actually create components way easier and that's one of the side effects of actually doing this uh change uh back to the Python 3based way of uh you know serving these things. And once you actually create these Python 3based flows for actually deploying these components, you can actually just take those exact same commands and drop them in your YAML file, which makes it super easy to actually port your workflows over to a Kubernetes CRD style setup if that's what uh you're used to. Um, so we're really excited to to to implement this change. It'll be coming out uh with.3.2 um and the following releases. Uh, but like Neil mentioned, we want to hear feedback. So, drop it in the discord, drop it in chat, go try out some of these new uh examples that we have uh and let us know what you think about this new flow and if you have any suggestions on changes for us. Um, I guess going to that point uh what have been like some of the your favorite examples or what have you actually done with these new UX changes um that's been like an unlock since we've done this? Yeah. Yeah, great question. I mean, I think uh one of the things uh that makes it that that these new um sort of UX changes have unlocked is is making a little bit simpler to to create like a wide variety of different examples because now you only need to have that one YAML file um that you know again contains everything uh that contains your you know your orchestration that contains the uh the configuration that contains the actual like graph definition. Um, and so what we're what we're working on is, you know, really creating a lot more examples to show people how to deploy different kinds of models, you know, um, with different kinds of configurations in different kinds of environments. Um, and so um, if you keep a lookout on our um, on our GitHub, you'll start seeing some of those pop up. And I think that'll be really helpful for helping people to understand, you know, some of the different components, but also like some of the different use cases that they can um, uh, that they can use Dynamo for. Damn, that's really awesome. Thanks so much, Neil. Uh so another question I had was when I first started using Dynamo, I had to run the container build script uh in order to actually build an image from scratch from source. Uh and that took me about 30 40 minutes and sometimes even longer. Um and so we released a really exciting change to to help speed this up. Um and Neil, I was wondering if you could tell us a little bit more about it. Yes, absolutely. So um you know, one of the things that we're really excited about and actually let me uh go back and share my screen again. um is that we now have these uh we now have pre-built containers um uh available for for you to use um from NGC the NVIDIA GPU catalog which is um where we uh publish you know a lot of our containers. So um that you know sort of lengthy process of like building the container and compiling all of the libraries and sort of making sure that all of your dependencies were um you know synchronized and and all of that um that's that's going to be a little bit simpler now. uh you know you'll be able to see um uh you know these different these different runtime containers that we have. So like the VLM runtime container that has everything that you need to be able to run you know your disagregated serving setups with VLM um using Dynamo um and any other kind of uh inference um you know uh inference setup that that you would like to that you'd like to see with with VLM. And then we also have you know these Kubernetes operators and um the deployment APIs again to um make it a little bit simpler to deploy things on Kubernetes so that you don't have to you know go through this process of like building a bunch of containers locally and then deploying them to you know your own private repository, your own private container registry uh and then you know launching things from there. So again that's hopefully something that's going to be a little bit simpler um for uh you know for for you to be able to use and um make it a lot quicker to get started. uh really excited to be able to have these and we're going to continue publishing um you know more and more of these containers that have um you know right now we only have the VLM runtime. We certainly have plans to um you know deploy container or to publish containers that have different kinds of runtimes um so that you can um you know experience inference on on all these different sort of uh inference frameworks um with Dynamo. Yeah, this is specifically a game changer in my opinion because previously you were kind of forced to build the image from source, push that to a private registry and then whenever you did uh try to deploy your graph, you'd have to call from that private registry. Now in your YAML um you can literally just specify this NGC URL uh to the uh NVIDIA registry uh and then drop in your args um your entry point args for whenever you want the uh how you want the inference graph to deploy itself. Uh and then you're up and running, right? And so this is a critical component of Dynamo that we want to continue encouraging is how can we increase and make it as easy and quick as possible to deploy your Dynamo graph. Um and this going this is going to be something we're going to continue iterate on as we move forward. Um the next kind of focus for us is actually improving on our examples. So uh if you see the examples folder in our um Dynamo repository, there's a ton of different examples uh on how you can use the different runtimes but also how you can run maybe multimodal models uh or just hello world examples themselves. uh we're looking to contribute a lot more examples to show what you can do with this new UX and with the ease of use now that the inference graphs are on the NGC uh private registry or on the public registry. So um we're really excited to um release these over the next few weeks. Um and so we'll stay tuned on the GitHub, stay tuned in our Discord as we drop uh some of these new examples. Um Neil, I'm curious, I guess on that topic, what are some of the new examples that you're looking to potentially contribute back to repository? And are there any asks that you want to make to uh the open source community to see if we can get any examples in? Yeah, definitely. We we have a couple of really um you know exciting uh exciting things coming that that will help uh people that we hope will help people adopt Dynamo. So we've got some examples of how to deploy in various different cloud platforms. So like on you know Google Kubernetes and uh in Amazon's EKS um platform. Um you know we have uh various examples of um uh different kinds of models. So, um, you know, we're going to have, uh, examples of how to do like, you know, really, uh, wide deployments and like, um, for for very large models, um, and in a very performant way. And, and I know that's something that people have been asking a lot about. Um, and, uh, we're really excited to to to be able to to to show people some of the best practices for for deploying very large models in um, very large uh, in very large deployments. you know, that's that's really like one of the um the key things that we want Dynamo to to um to be good at, right? To to be able to, you know, to manage like deployments that have uh everything, you know, all over the place with a ton of different nodes and a ton of different servers and kind of uh a lot of complexity and, you know, orchestrated in in a in a in a simple way. Um we've also got um you know, some examp we've got examples for all the different kinds of um all the different kinds of frameworks. Uh so you know uh SG lang tensor TLM and VLM are the main ones that we're looking at right now. Um certainly if you guys have um thoughts about other kinds of inference frameworks that would um you know be beneficial for you. We'd love to hear about it. Um we'd love to to integrate some of those into Dynamo or if you guys are interested in contributing anything. Um you know we would love to uh we would we would absolutely love to to to accept those contributions if you guys um uh you know are are are interested in that. So, um yeah, that's that's some of the stuff that we have uh we have we have coming down in our pipeline. Yeah, on that note, we have a ton of open good first issues uh that you can go contribute to. I know Graham does a really great job of curating these for the community go tackle. So, if you're interested in learning about inference and want to get involved, this is open for anybody in the world to go contribute to and help with. Um, and we're on a mission to make uh inference as easy and accessible as possible in the most performant way. And a great first way to go do that is by contributing to Dynamo. Um, so on that note, I think we'd love to open up kind of questions and start looking into the chat. Neil, you want to start reading out some questions or see if there's anything uh that's been asked in the chat? Yeah, let me let me take a look. Um, you know, again, definitely if you guys have any questions, uh, please uh, you know, uh, feel free to uh to pop them into the chat here anywhere. Um, let's see. I see a bunch of no audios. Yeah, I think. Okay, so there's there's one question here. Can we expect KB routing support in VLM v1 as well? Um, yeah, that's a that's a great question. That's definitely that's definitely something that we're working on. You know, we want um we want KVAware routing to it's really one of our sort of flagship features for for Dynamo and we want to make sure that that works um with with all the different frameworks. Um I don't know if we have um a specific timeline on um KVA or routing for for VLM v1. I know there's some complexities and uh there's changes in the ways that like the the sort of KV events um are published to um to to our uh to our data store um which you know then get like uh that that get reasoned about in in in the router itself. Um but yes, that's definitely something that you can expect and it's it's it's something that we're we're actively working on to make sure that um yeah, that's uh that's not a blocker for people. Um you know, right now we do still we do have it available for VLMv0. So, if you haven't migrated over to VLM1 quite yet, um you can definitely still use the um the KV uh router in uh in V 0. Uh let's see if there's other questions. Okay, I am okay. Um, yeah. Okay, here's another question. Can I use Dynamo along with Olama to deploy multiple Olama instances over multiple GPUs? That that's that's another really great question. Um, so we don't have an integration with Olama um through Dynamo yet. We we we do have integrations with um with Mistrol. RS, which is sort of similar to to llama.cpp CPP um which is what Ola is is based on. Um but um there you know right now we've um we focus a little bit on these um on on VLM, SGLANG and on Tensor RTLLM as our as our inference engines. Um that's that's really good feedback though to to hear that people are interested in using um Olama with uh with Dynamo. I know this a really popular inference framework um especially for local deployments um and you know people who are who are deploying things um uh you know in like like at home or like in in you know like smaller labs and things like that and so um yeah I I think it's definitely something that we could we could take a look at um and and see if there's a there's a good way to unlock um uh to unlock that use case. Yeah, and I think there's actually an open issue to integrate Llama CPP uh if I remember correctly. Uh so if you want to contribute back, feel free to uh to do so. Yeah. Okay. Another question. How is a Kubernetes operator support with Dynamo? Uh so there's a there there's a couple of different um there's a couple of different things here. So we do have a Kubernetes operator for Dynamo. Um and uh I showed that um that that custom resource definition that CRD a little bit earlier that um had the inference graph and all the different you know arguments that um that you could put there and and and what that is basically is that's you know that's the um the custom resource that gets deployed and then the Kubernetes operate or the Dynamo operator for Kubernetes kind of um you know parses that uh that that resource definition you know understands what all the different components are and then deploys that into into Kubernetes. So um we definitely do have a Kubernetes operator for Dynamo. We we do we definitely want to make sure that um you know it's it's easy to use, simple to understand and is um robust for um you know anybody who wants to to deploy um inference on on Kubernetes. Um so yeah uh please uh definitely check it out um and and let us know if you have any feedback on you know the the experience of using um on using Dynamo with Kubernetes. Yeah and we're going to publish a roadmap for the Dynamo operator itself as we kind of mature our Kubernetes platform. So already when you install the Dynamo platform, you used to have installed the operator and the API store which is where you would you know register uh kind of all your inference graphs. Uh we we're making a lot of changes. So we actually remove the API store API server in 0.3.2 and focus on adding a lot more rich features into the Dynamo operator itself. So we're like openly trying to figure out hey how can we improve some of uh uh these features in Dynamo Operator based on customer feedback. already we're hearing uh some feedback around like scheduling um uh and some of the feature parodies with some of the other one other operators that are out uh on the market today. So we're actively discussing that and uh we'd love to share discussion with the community. So if you have any thoughts feel free to drop it in the discord uh or open up some GitHub issues. Yeah. Yeah, definitely. Um okay, so another question here. Will these changes induce any change to the way NVIDIA NIM microser is is leveraged? Okay, that's another really great question. Yeah, so for those of you who who aren't familiar, um, NVIDIA NIM is a set of microservices that we publish which are basically like package packaged up versions of um, of a single model that you can take and deploy you know in a very like simple way um, on on any kind of different platform. Um, and so we are definitely um, looking at the ways uh, to sort of deepen the integration with um, with Dynamo and and NIM. there's already um under the hood um uh you know a lot of um uh a lot of sort of like shared learnings and and shared components between Dynamo and NIM. Um and we're going to continue with with some of those integrations. So you'll what the way the way that um that'll probably manifest is that you know if you're if you're a NIM user you'll start seeing a lot of really cool new features appear for for NVIDIA NIM. Um and a lot of those will be powered by um some of the components that are coming from Dynamo. um and just you know uh make things a little bit easier for to use, make the make NIM work um a little bit uh better for you know large scale deployments and you know distributed deployments um deployments on multiple nodes uh that sort of thing. So yeah definitely definitely something that we're um yeah that that we expect you to keep an eye out. Yeah, real quick I just want to add to that. Uh I think uh one of the things that we learned about nims is they're not necessarily aware of like the large cluster or GPU environment that they may be deployed on. Uh and that's the difference with Dynamo, right? is it's actually very aware of the system that it's on uh so that it can reap the benefits of like disagregated serving or KVAware routing and so I think as we think about adding the benefits of Dynamo into NIM you'll start to see that NIM can take some of these benefits with disagregated serving and uh KV cache management KBware routing so that uh it can take advantage of the deployment target it's on and it's not just the model package into the container all right um let's see are there any motivation for building Dynamo from llama file That's I I think that's an interesting question. Um so I'm I'm not so familiar with llama file. My understanding is that it is mostly for um like a single user um deployment. So like uh like running on a local machine. Are you do you know much about llama file? Uh I don't know much about llama file other than it's kind of like similar to like the llama user base. Very useful for running uh locally. Um, I don't think we have any immediate plans to support that, but please feel free to open up a GitHub issue or ask in the Discord and we can get some other team members to share in their thoughts. Yeah. Yeah, I would I would love to continue this conversation on the Discord. Um, you know, we can we can talk a little bit more about um, you know, what what we can do to to to help with with Llama file deployments. I think it's something that we haven't thought much about yet, but um, yeah, definitely definitely a really interesting question. That's that's cool. Llama file. Yeah. Okay. Um, does Dynamo adopt a more custom fault tolerance handling for the scalable workloads or uses torch FT? Uh, so it doesn't use torch FT. Yeah, we um we we are um working on some of uh some improvements to to the fault tolerance uh capabilities of um of Dynamo. Um I actually don't have a lot of details on that. I think maybe that's something that we can um we can have as a topic for for a future uh a future office hours cuz there's a lot of really interesting work going on there. um in terms of some of the different things that we can do in terms of like checkpointing and restarting and sort of like having like a um like spare like hot spares and things like that for um some of the different inference workers and uh you know just to make things like uh like not crash as much and then be able to recover a little bit more quickly if they if you know if there are errors that that come up. Um but uh yeah maybe maybe we can um think about having that for for a future. Yeah and I think what would be really useful for the team uh working on Dynamo is like getting some feedback too because for example there's some uh developers who've told us that uh they don't necessarily care about like handling invite requests. They're fine with just like uh quitting it. There are some who have told us let's just add it into the queue um and and let that uh client request go through. So this is something that we would love to get more feedback on. So um if you specifically are interested in fault tolerance, please feel free to open up an issue. Again, uh Discord has been my mantra, GitHub issue or Discord. Uh but we'd love to get some more information and thoughts from the community um as we're kind of engineering some of the fall tolerance paradigms into Dynamo. All right. Um I'm seeing a question. There is an approximate KV router as well in the PRS. Will that also be supported for V0 only initially? Um, I don't know. I haven't uh I'm not so familiar with the with the approximate KV router. Um, I think we'll have to Yeah, maybe maybe take a look at at what the details are for for that PR unless Anish, you have any Yeah. Um, that's a it's it's a good question because yeah, I don't I don't know if it has the same um complications in terms of the the KV events that the regular KV router does. Um, yeah, it's a good question. uh we'll um we'll take a look at that and maybe respond in the uh in the Discord um with some with some follow-up. So yeah, if you if we if we can't answer your question today, we'll we'll we'll try and answer it on the Discord. So just uh some motivation for you all to to join the Discord and you know, continue these uh continue these conversations and uh yeah, we'll uh um you know, we'll we'll keep it going over there. Okay. Um I do see another question here. We can scale GPU workloads in Kubernetes after virtualization. Which codecs are reliably supported if we need alternatives? H um I'm not sure if I understand the question here. Um which codecs are reliably supported if we need alternatives? Yeah, I if you could like uh you know expand on your question. Um u uh that that would be helpful because I I don't think uh yeah I'm not I'm not sure if I if I have a good answer for that. Um yeah. Okay. Let's see. Um let me check if there's anything in the Discord. Okay. Uh looks like we don't have anything in the Discord. Let me go back. How about Oh, okay. Uh developing an AI mortgage loan application that anyone can go and provide the required information, documentation, and get almost instant pre-qualification. Um I think I I think it's an interesting idea for sure. Um there there's like a lot of complications um that go into that. Um but uh if you need that to run faster, you know, you could definitely use Dynamo. Um that's a yeah um it's a it's an interesting one. Yeah, I'd imagine there's a lot of privacy involved for that which means you want to self-host a lot of these pipelines and run inference on your own cluster. Uh which where Dynamo would come handy. As for the actual utility of the application, I don't think that's really our uh our thing to to say. Uh we should go talk to some customers. Yeah, definitely. Definitely. I think uh we're we're all very excited about you know some of the different um a lot of the different applications that uh that AI is is is unlocking and especially you know some of the really interesting stuff that we've seen around test time compute and you know being able to have like uh you know these more sort of um more sort of involved um more sort of involved inference graphs uh with you know a lot of different components. Yeah, actually on that point, uh it's really interesting to kind of think about what AI application would run best using Dynamo, right? And so when we think about this paradigm, we think about uh high ISL, low OSL uh use cases. So what this really means is like uh Dynamo is best uh performant uh when you have really high input sequence lengths and really low uh output sequence lengths that need to come out. And so for something like a mortgage utility application where you're probably uploading a ton of PDFs, you're probably uploading a ton of documents uh and maybe the answer coming out is just hey you summarize all this. This is actually the actual output. I think that's actually a perfect use case for Dynamo. Uh kind of just like thinking off the dome. So um really like diving into like how you can tune Dynamo for your use case is really important. Um but from our experience we found that Dynamo is best performant for these high ISL low OSL use cases. And um in the discord there's some resources that I actually just dropped in today on how you can kind of understand some of these key concepts and what you can tune with Dynamo. Uh there's a really great YouTube video um that a couple of NVIDIA employees have made around like the the basics of LLM inference, but also Kyle uh one of the chief architects of Dynamo has a really great reading list on some of the basics of distributed inference that tie into some of these concepts uh and how you can tune the Dynamo distributed runtime for uh your end application like this mortgage uh lending advisor. Yeah. Okay. I I I saw a question here which I think is a really important one. Um what does Dynamo team think about open source projects like LLMD? I recently saw that Dynamo supports LLMD's operations and I'm curious about their opinion on such open source contributions. Yeah, like I said, it's really important question. Um and and we're we're huge fans of LLMD. Um I I think uh one of the um one of the sort of like key motivations for us building Dynamo and and so one of the the things that we're really trying to keep in mind as we um as we expand uh the the use cases for Dynamo is making sure that it's really modular, right? Making sure that each of the components can sort of be standalone um and and can integrate with um you know with the other uh obviously the other components within the Dynamo project, but also different um uh you know different tools that are out there in the ecosystem. And so, um, you know, we've we've already contributed a lot of, um, a lot of components, uh, to LLMD and, um, you know, we're going to we're going to continue to do that, right? So, we, um, we really anticipate PE people being able to come into Dynamo and, you know, like see which pieces, um, make the most sense for for their projects and be able to integrate those into their own projects. So, yeah, we um, you know, we yeah, we're huge fans of LLMD. Um and uh that's definitely something that you know we want to keep uh um we want to keep contributing to and and you know building that sort of um that uh production inference ecosystem. Yeah. On that note uh LMD actually use a piece of Dynamo called Nixl uh which is actually how you do pointto-point transfer between the GPUs themselves. Uh and so we're really proud to have been able to help LMD come about. And again like I mentioned with the previous question, Dynamo is really good for certain use cases but not everything, right? Like Dynamo is not going to solve every single inference problem for you. And I think you're going to have to take different approaches sometimes. Like I mentioned, Dynamo is really good for high ISL, low OSL um use case. And of course, you can tune the system to match your use case. But there may be new frameworks or new approaches for different use cases. And I think uh our hope is that you know you don't have to just take the entire Dynamo system. You can take some of these modular components uh and implement it in uh your own framework or your own use case uh for that matter. And that's essentially what LMD has done with Nixl um the Nvidia uh inference transfer library. Yeah, great. Okay, let's uh maybe do one more question and then um yeah, we can uh continue the conversation on on the Discord afterwards. Um but uh I see a question. Are there any plans for integrations with the Run AI Kubernetes AI scheduler? Um yeah, another another really good question. Um I don't know that we actually have any um uh plans at the moment. Um but uh it's definitely something that we should think about. No, actually we do have as of recently. Um, but you know, scheduling is a really really uh hard problem that I think Runi has done a really good job of tackling. Uh, they've recently open sourced their uh Kubernetes AIuler, their Kaiuler. They also have a topography aware called Grove. Um, which are two really interesting piece of technology that we're really thinking about with Dynamo. Um uh one of the biggest things and pieces of feedback that we've got is we're not actually being super aware about the scheduling uh when we orchestrate these inference graphs on Dynamo. And so I think uh Kubernetes AI scheduler from Runi and some of these other uh techniques could get integrated in uh very soon and we're really excited to kind of reap the benefits of that. Um so yeah, if you have any uh thoughts on this uh you should make a Dynamo enhancer proposal or add in some uh open issues. I know internally at NVIDIA we've been discussing quite a lot and I think we're really excited to kind of share uh what our thoughts are here to get the community's feedback. All righty. Thank you guys so much. Sorry, I just have one more question that I want to just does does Dynamo support VLMs? Yeah, Dynamo does support VLMs. We have some examples in the in the Dynamo um GitHub repository um under the sort of multimodal uh label. So um yeah, Dynamo definitely supports VLMs and there's some really cool stuff that you can do with the disagregation of the sort of like vision and language parts. um when serving VLMs. So, uh yeah, huge it's it's it's something that that we're we're focusing more and more on with Dynamo. Um and uh we think Dynamo is really great for for serving BLMs. Okay. Um uh so yeah, uh I think uh that's that's all we've got for for today. Um you know, again, we've we've dropped a bunch of links in the chat uh to the Discord and to some of the um the Dynamo enhancement proposals that that we're putting out there um for, you know, the community to to be able to see and and to see and give give feedback on. So, um definitely would uh appreciate um you know, anybody who who uh would like to take a look and and and leave whatever feedback that they've got. Um you know, again, we um we're always really happy when we hear from from Dynamo users or potential Dynamo users who are, you know, curious about Dynamo and curious about inference. Um and uh yeah, would um would love to continue some of these conversations in uh in the Discord. Yeah, thanks for tuning in. We're going to try to be setting up these bi-weekly office hours, so definitely stay tuned for the next one. Uh, and we're gonna try to do an inerson one soon, so stay in touch for that. Um, but again, appreciate you guys tuning in and, uh, thank you again. All right.