Optimize Your AI Agent Workflows with NVIDIA NeMo Agent Toolkit Profiler

Key Takeaways

my name is Dro and I'm a Solutions architect Nidia today I'll be showing you how to use the Nvidia agent IQ profiler to get deeper insights into the performance and behavioral characteristics of your agent workflow for this demo we're looking at agent Morpheus an AI agent example for a analysis let's kick off a r of agent IQ before we explore some of the configurability of the agent itself the agent Morpheus system built by Nvidia can quickly triage vulnerabilities and software containers and code this is incredibly helpful as analyzing software for vulnerabilities is often a critical but timec consuming step in the development process the input to the agent is a vulnerability that needs to be triaged this also includes information about the documentation as well as where to find the code itself a large language model then creates a checklist of things it needs to investigate in order to triage the vulnerability the same checklist is then processed by another agent in parallel the agent performs retrieval augment of generation on the code in addition to the documentation of the container to understand if a particular vulnerability exists once all our checklist items are executed in parallel there are two other model steps that occur first all of the findings are summarized and then created into a Vex justification which is a justification about a vulnerability existing in a container or not that is then passed to the user this process end to end saves a lot of time with agent morphus where would usually take security analysts and operators hours if not days to do agent morphus will finish it in a minute as we'll see as we see the agent thinking and observations on the right hand side let's explore how the agent is configured in agent IQ you have functions which are the tools and agents themselves that do various things for example we have one to search a vector database another to search the internet to search the software builds and materials Etc we'll also have llms that are used in the execution for example we'll have an LM for a checklist for a vector database for the retriever execution and so on the first step to create and run a profiler on a workflow is to ensure the profiler is part of the evaluation harness that's native to agent IQ so you must add a configuration item at the bottom of the configuration yo file that tells agent IQ how to use a profiler then we'll tell agent IQ where to save the data as well as where the input data exists that's going to be used by the profiler then we'll tell agent IQ what kind of metrics we' want to look at for example if we have hyper parameters with the each of those metrics we should add them here let's try Computing estimates for workflow runtime we can also forecast how many unique tokens are created by the agent and llm in the execution of the workflow at each step we can also identify bottlenecks in workflows and we'll explore all of this as we look at the output of the profiler we see on the right that the profiler has successfully completed and now we can explore some of its output one of the primary outputs of the profiler is Trace of the execution of the agent itself we can look at all of the intermediate steps thoughts actions tool usage for the agent including where it came from what its function ancestry was how long it took and even what framework it came from for example if an agent system was implemented on Lang chain as well as llama index we'd still be able to trace all of those in a single view now this may be a little hard to read so we also standardize a data in a data frame that contains all of the same information just in a columnar format that may be a little bit easier to interpret but more importantly we also have open Telemetry Integrations that allow us to view this data in a UI such as Phoenix we can look at each of the agent tool calls and the execution look at the latency of the llms look at what went into a tool and what the outputs were we can do this for llms tools retrievers Etc all natively and transparently to the user so whether they're implementing a lang chain llama index or another framework we're able to comprehensively observe intermediate steps another metric we can look at is inference optimization which is helpful when collaborating with engineering teams to optimize workflow run times for example we could identify commonly used prompt prefixes so that we could pre-populate key value caches for NVIDIA Nim to increase throughput and reduce latency we can get a better understanding of how workflow runs or how many unique tokens there are between every agent calling an llm which may allow us to make smarter decisions about the serving of the llm M itself we can also look at workflow run times and latency confidence intervals from a developer perspective we provide additional metadata this allows us to look at understanding how workflows Run in real time we also provide a profiling report this allows developers to look at for example where the bottlenecks and their workflows are is it with the llm or is it a certain tool the last example we'll look at is how workflow runs at various concurrencies we can look at llm latencies at multiple concurrencies so we can make smarter decisions about how to deploy these workflows for example if we're seeing a lower latency at a high concurrency that's good but if we wanted to reduce our latency at a high concurrency maybe we'd make smarter deployment decisions about the cluster on which this runs but all of this is to allow developers to pinpoint where their bottlenecks are where they can optimize their workflow so that the next time it runs it will run faster