Decomposing the Agent Orchestration System: Lessons Learned

Key Takeaways

[music] >> I got one more talk before we hit lunch. I'm going to invite up from Union Mr. Neils, where you at, dude? Here we go. You put your This is yours. >> I got a I got a mic here. Okay, cool. Hey everyone. All right. >> [sighs] >> So, I'm Neils Bantilan. I'm the chief ML engineer at Union. And um I've been building and maintaining a open-source ML orchestration product called Flyte for the past 5 years. Um and through this product I've been helping uh companies like LinkedIn, Stripe, Spotify, and more recently uh Mistral help in build and deploy production-grade ML and AI systems, you know, from fraud detection to uh oh Let's see. Ah, here I am. Fraud detection to recommendation systems to uh LLM foundation model training systems. I don't do the hard work. I build the tools. The our customers and our open-source users do all the hard work to to do all that. Um but today I want to share with you some of the lessons that I and we at Union have learned uh productionizing agentic systems um with with our orchestrator. And in particular, I want to tackle the question of what is the orchestration stack for observable, debuggable, and durable agents. And to tackle this story, I want to take you through my agent adoption timeline. Uh I guess just out of curiosity, who here has installed Open Claw? I haven't, so I can't. Cool. Who is willing to install it on their company infrastructure? >> [laughter] >> Okay, no one. Cool. There there are the more secure versions that and I know I know now like Zero Claw, a few of them out there. I haven't checked them out personally, but basically in 2022, I like probably many of you made first contact with LLMs that have coherent long-range text generation, and we were all like, "Oh, wow, it's kind of working now, right?" In 2023, I went about sort of fine-tuning LLMs. That was a big story back then, open-weight models, fine-tuned it on Flyte code, see if it can generate Flyte code well or any other type of DSL or subset of a programming language. Also started looking at RAG as uh a system to overcome uh the context window problem, which I guess is still a problem, but you know, now we have 1 million to token context window models. Um oops. Then in 2024, I you know, just I'm not an early adopter in these things. I continued learning and prototyping, so LangChain had come out already, so I I tried building some LangChain apps for internal use cases. I started using Cursor, and at the time long-range or or end-to-end tasks weren't really working then, so it was mostly like tab complete live coding with like editing in the loop, right? Um also started using tools like Perplexity and Claude desktop and things like that, and that was starting to give us an inkling of of agents that can do more than just like turn-by-turn chats. Then last year, we made the critical decision to modernize Flyte, and so we set out to build Flyte 2.0. And this was our answer to some of the things that I and we as a as a company had learned building these things internally. So we internalize all the feedback we were getting from users, our own internal taste of how we like building agents. And so Flyte 2 was our move to answer the answer the question of durable, fully dynamic, crash-proof with some caveats uh infrastructure-aware uh orchestration. Not only for agents, but also good old classical ML and data science and ETL. So this is where we built Nody. This is an internal uh agent ambient agent that sort of um uh reconfig- configured uh K- Kubernetes node pool configurations for our customers. So customer on Slack would say, "Hey, I want this kind of node pool." And then Nody would go and you know, it's kind of like the Rick and Morty butter robot. All it does is change node pool configs. We built a few more of these internal tools. Started using Cursor agents and Claude code uh kind of at the tail end of that year, and so I was like pretty much agent-pilled at that point where you can mostly do end-to-end tasks, and then obviously you have to review the code and fix all the crazy bugs and things that it would introduce. Um and now it's 2026, and it arguably is the year of agents. And so what we're doing now and what I want to share with you is how we are thinking about productizing all of our learnings, and in particular, I am uh leading a team to help build agents for the thing we know and love uh the best, which is data science, machine learning, AI engineering, ML ops. Okay, so the crux of the problem here that I want to highlight today is the agent infrastructure problem. So you built an agent. Great, that's the first step. You optimized its prompts, you engineered its context, built an eval harness, and it works beautifully in production, but what we found uh in development, sorry. But what we found in production is actually often times the infrastructure gets in the way. Tools uh oops, this is not the latest version of the slide, but it's okay. I'll make do with it. So tools need secure and least privilege access to your and I'm talking not personal use on your laptop, right? This is like you're building stuff agents that have access to your organizational data assets and databases and data lakes and whatnot. Oh wow, it fixed itself. Great. Uh wow, what was that? Uh there's an agent loop in there. Uh tools and agents that require variable compute resources. So if you're building a machine learning agent, you you a single node runtime's not going to cut it. You know, the agent may run on a small CPU box, but it's going to train a model that's loading a several gigabyte data set, right? So the agent needs some way to provision its own compute like in its inner loop, right? Same thing with parallelized sub-agents and tool calls. Uh if you have a single node or some like fixed set of nodes, you're going to get resource contention, you're going to get degraded performance. Containers go boom. Anyone here who's trained a model or done anything with data has seen this type of thing. Uh the your nodes are killed by the scheduler, spot instances are preempted. Then ultimately these kinds of infra failures lead to agent memory loss or corruption, and this wipes out precious context. So agents fail at multiple layers of the orchestration stack. And the problem isn't that agents fail, it's that recovering from failure is challenging without the full context of how infra, networking, logical, semantic layers, all of these interact so that the agent can figure out how to dig itself out of the hole it's found itself in. And so what agents are saying, agents are saying telling us, "Help me help you." Agents now are very capable of helping themselves if they have the right context, even infra infra-level failures, if you give them the right context and capabilities. And so for the rest of this talk, I'll be sharing not like things that are written in stone. These are just kind of design principles we've distilled down into six of them. Uh I'd like three or four, but you know, we got got six. Um and these are more like rules of thumb, things that you should think about when you're building agents. And so I'll dig into these more, but basically number one, use plain Python, TypeScript, JavaScript, or whatever like general-purpose programming language that an LLM knows. So pick that. And those languages will have frameworks that you can, you know, quickly build agents on, but what we found is uh for us, for example, Flyte 2 works with plain Python. That's just really easy to to access all the constructs, programming constructs, for the human and the agent to use in building. Uh the second design principle is providing durability and observability hooks. Then make failures cheap so that when you do fail, that your run, your agent rollout can recover really quickly and and continue where it left off without having to recompute all the other stuff. Um provide infrastructure as context. This is a a key thing that um that we found that's quite quite important when you're building sort of variable compute uh agents that require variable compute in the inner loop. Uh providing agent self-healing utilities. Um so these are things that where the agent can fix its inner loop, which I'll show you what what I mean exactly later. Self-healing is a little buzzwordy, but it it's what basically how do you recover from failure in the inner loop? And uh uh the utility in this case would be sandboxes so that they can uh inject a little bit more creativity in how they solve the the particular problem that the user's given them. And then finally, human in the loop as the final recourse. So agents break at, as I mentioned earlier, multiple layers, right? So from infrastructure to network to logical to semantic to tool execution and context. Um and actually what we found is there there a lot of techniques and a lot has been written about how to handle network to contexts. Those are those five levels and it's it's not no by no means a solved problem, but there's I think we found that there's still a lot of pain in the infrastructure layer of this. And I'll show you an example in a little bit. And I think one of the problems is that there's a context of an evaluation gap. So your your evals typically test semantic correctness. Does it answer correctly? Is it hallucinating? Are I am I is the tool call arguments formatted correctly? That kind of stuff. And likewise, agents typically focus on recover recovering from these kinds of logical and semantic failures. But agents don't often directly handle or reason about surviving a network outage or a timeout or recovering from system level errors or recalling state across retries. So the one insight I want to give you today is that agents can actually recover reliably from all the errors in this the stack that I mentioned earlier, even infrastructure level ones, but only if you give them the right context and the right capabilities to dig themselves out of that hole. So context engineering is not just not just you know shuffling tokens and getting more data sources and and those are important activities. These are like the the key ones that you need when you're initially building your agent. But context engineering is also an infra problem. If a failure wipes the agent state, then all the hard earned earned context that you've implemented in your agent loop is is worthless. And so we think about durability as making it super easy to do the following. To immediately resume an agent rollout where it left off. To share to not re-execute shareable compute and IO bound workloads. And to save and restore uh state and memory between tasks through intermediate state persistence. So these are just basic building blocks that benefit not only agent builders, but MLEs and data scientists uh alike. And so these three elements I want to dive into before going into the design principles just because I think they're important um and they're the core building blocks of flight and any other system you may want to implement uh I think. So what is a replay log? This These aren't like official terms. This is like how what we call them um but a replay log is essentially a service that records the state of an agent and its subtasks at a super granular level at each step. And the benefit of this is you can avoid re-executing tasks, prevents memory and context loss, even recovers from crashes in the root agent process. So in this case you have an agent. Does it have a pointer? That's fine. Agent that does tool call one, subagent call one, tool call two and it executes the first two steps and fails in the third one for whatever reason, right? A replay log will keep a log of all those steps and all the intermediary outputs. So in the case of a crash, it recovers. Tool call one's done already, doesn't have to do that uh recompute it. Subagent one tool call subagent call one is done, don't need to recompute. Now then it just can pick up from tool call two. Um and so imagine this is like a thousand steps and uh I'll I'll tell you about a case study of a customer who had you know this type of very long running agent with many many steps. So this is super important because at this gives you this like fine grained micro cache per run that if there are any network or system outages that you can just take for granted that your agent is going to just going to pick up from where it left off. And this is similar to go global caching, but as opposed to this run uh run level replay log, global caching is as the name suggests is work shared across all agents and all agent runs. So in this case you see an agent where I'm calling it with the same prompt, write a report about LLMs. It does a web search that in flight you can say I want to cache that step. Uh does a DB read tool, I want to cache that step, but then I want to disable it for the composer subagent which then synthesizes the outputs of the two previous tool calls. Um and so you know maybe the the AI builder says I want to inject a little bit more creativity or entropy to that last step and so each time I call it I want it to reuse the the research kind of deterministic tool calls, but then regenerate a new report each time. And finally you have intermediate state persistence, which is the idea that you can just take for granted that once you use what I'm calling well describe later as these functional uh durability hooks, these decorators you see on these functions that you know when you call the LLM the first time, the output of that agent state is stored just to object store in S3 or something uh somewhere, right? So you just take it out for granted. You don't need to write any of your serialization deserialization code. It's just there. You get data lineage just out of the box. Same with your first tool call. Same with your second LLM call. So all of these under the hood in flight will be stored to uh uh your object store. So then those are the three building blocks that sort of unlock the implementation for these six design principles. And so I won't go through these line by line. I'll show you a little bit of code. I'm not sure if you can read it too clearly, but the first one is to use plain Python and the benefits of this I think are fairly obvious. You get loops, fan out, conditionals, try except uh async synchronous programming in Python and whatever the equivalents are in whatever language you choose. There are no DSL surprises. There's nothing new to learn, but if you do choose to use a framework, you can use it, right? It's it's pretty straightforward. And this is the point I'll I'll hammer through the rest of this is that exceptions are actually a perfect delivery mechanism for critical context about failures at all layers of this of the stack. So this you can squint and you can start to see, oh, what if I can catch an out of memory error here at the at the system actually bubbles up into my into flight, right? And leading to my next point, providing functional hooks make it really easy to add durability to your system, whatever implementation that you choose, right? So assuming that you're using functions or whatever class methods, that these hooks give you the ability to trace, checkpoint and do the persistent intermediate uh state serialization deserialization I mentioned earlier. And so in flight, the first thing you do is you define your task environment. I'm mentioning this partly to show off flight, but partly to just show you how we're thinking about this type of problem, right? So this allows you a task environment allows you to specify your container, your image, your dependencies, your resource requests. And tasks you can think of as as these containerized functions. These run on a Kubernetes pod. These run on their own container. So agents running multiple of these tasks have container isolation. You get a lot of security benefits out of this as well. You give its own AIM role. You get a lot of nice downstream benefits of this. And flight trace you can think of as a helper function so that you don't have to pay the cost of spinning up a new container. This is just a helper function that gets you all of those persistence and the crash proof guarantees uh that I mentioned earlier. And this makes failures really cheap. So if you have an agent that writes code, that runs the code, and then finalizes the answer. And so this is our first iteration of an MLE agent, you have all the tracing and caching enabled to be able to have build an ML agent MLE agent that just has all of these crash proof uh guarantees built into it. And as an added bonus, failed runs become training data or additional context from the agent to learn from. And not only is this an out of band process where you can you know query the flight database to grab all the failures and create a training data set, you can actually react to them in line in your agent loop. And so this unlocks infrastructure as context. So this is where the agent can see things like out of memory errors, uh system level ephemeral outages and retry based on those without you know totally crashing. And so here you can catch an out of memory error. You can have the agent respond to that and provision more resources based on the error in the code that it wrote. And make adjustments that are sent back from in the for loop so that the the platform can provision more resources for for example in this case a training job. >> [snorts] >> But not all failures can be done in such a structured guardrail format, right? So in this resource example, you're just outputting a dictionary of some predefined outputs and and values. What if you want the agent to orchestrate the actual tools in its toolbelt? So this is there are two types of sandboxes I'll talk about. There's a third one that we don't quite quite support yet, but code mode is something that I think Anthropic, the Anthropic team, coined. This is where you replace all the context load associated with tools and formatting tool arguments and getting the results out of that. And you have the agent, you give it a toolbox and say, "Write Python code that's sandboxed. So, you can't do IO, you can't do network calls, you can't do extra imports. You're writing pure function calls and conditionals in your code mode mode loop. Right? So, the agent writes this pipeline. The orchestrator can run this code generated by the agent securely in this limited subset of Python. Shout out to Pydantic Monty, by the way. I forgot to put in my slide, but we're using that here. Um it's a a cool project. Um check it out. And then you have this really tight error iteration loop to fix the orchestration code bugs. So, you're you're you know, you're no longer having to do an eval kind of out of band, right? You could do that, of course, definitely, but you have the agent just fix its own errors. But sometimes you don't want just orchestration, right? So, what if a user asks the agent something that for which a tool does just not just doesn't do the job? So, that's where a stateless code sandbox is super useful. And so, this is where you can introduce additional third-party libraries, you can do imports, you could do limited network IO, you could do limited read-writes on the file system. Um but so, this stateless code sandbox is not that. So, this is just a one-shot. The agent writes the code, it runs end-to-end, uh and then produces some output. So, it's it's kind of like a pure function. Um so, this is what I just said. And you can even do cool things like have it write its own unit tests, right? So, and then finally, there are a class of errors for which maybe the prompt is purely poorly formatted or the system prompt just uh doesn't work, or there's just it the agent just does not have the context it needs to solve the task. So, human-in-the-loop recourse is uh ultimate recourse to uh course correct an agent when it's kind of gone off the rails. So, in this case, when the agent has exhausted its max iter budget, you can just have a very Pythonic way to say, "Hey, I need more context." Uh and then in the UI, you can type in whatever additional context. You can upload a new file, you can give it a a zip file of a directory uh with uh more markdown files, which is apparently how we do things now. >> [laughter] >> And then you recursively call the agent. So, the flights supports recursion. Uh there's a cleaner non-recursion way of doing this, but I thought this was cool. So, you basically call the agent function itself with the additional context, right? And this is the coolest slide cuz it has Wolverine on it. What you get out of all of these building blocks is an agent with a healing factor, right? It's not perfect, it's still going to fail, it's still going to fall over, but in this case, I don't expect you to read this, but this first error was an out-of-memory error. And then the red box there, it adjusted the sandbox resources, it gave itself more memory. And this error, uh it had an import for Pandas that was not specified that it did not specify in its dependencies. Um so then, it writes more code, it updates its dependency structure, and adds Pandas. Then finally, for all those logical semantic tool execution failures, uh here's just a a simple division by zero error. So, that's the error, it just you give it the error and it fixes the the code. And so, I don't have much time left, but what does this look like in practice? We worked with a customer called Dragonfly who provides a basically a a deep research SaaS product. So, you go to them, you say, "Hey, I want a Jira-like thing, but not Jira." And then it'll do do research to figure out what stack you want to use for your company, right? So, their challenge was to build an automated solutions architect, an agent that creates a living knowledge graph of SaaS products. And so, these were 250k or so software products. Each agent call had about 200 steps. Each product a little call each product had about 100 LLM calls associated with it. Um and so, we helped them with a kind of a tiered architecture. And so, this is this is like scaling agents, right? These are not the the agents you may be used to in very tight uh chat interfaces. These are agents that just run ambiently, that just power their database, right? So, this thing will just run at some cadence and populate their database that powers their their um their SaaS product. And so, at the top you have the agent driver. So, you have four replicas. You have research coordinators that have eight replicas. You have underneath that 12 researchers. And yet underneath that are tool 12 replicas of a tool layer that each researcher has access to. And so, we helped them do cross-run caching, right? So, this is a global cache I mentioned earlier. So, they don't have to re-double pay for LLM API calls given the same research prompt. They did this really cool thing that I need to learn more about, but it's basically some semantic conversion detection. So, every once in a while, the coordinator layer would look at all the research threads they were spinning up, and based on some similarity semantic similarity, they would group and consolidate them so as to avoid duplicate work for the researchers. Checkpoint-based recovery, this is what I talked about with the durability piece. Uh they use spot instances heavily to reduce costs, and it became basically a non-issue. Spot instances would go away, they would come back a few seconds later, and you wouldn't lose any progress. And you have full auditability. Every single LLM call or every single tool call is traced. And so, the result is I actually onboarded them, so I helped them, and within an hour, they onboarded their local Babel agent prototype into production in about an hour. And this got them to 2,000 plus concurrent runs. And the onboarding was really because it was just Python. Just like, here's the task config, here's the decorators, and just go, do your thing. Reduced their recovery failure recovery time by 50%, increased their development velocity by 30%, and uh they saved 12 hours a week on infrastructure maintenance because of this. So, in conclusion, if you're an agent builder or starting to build agents, do yourself a favor and your agents a favor. Um consider that observability is necessary, but not sufficient. Durability helps root agents recover quickly. Don't aim for failure-proof, aim for cheap failures, fast recovery, and a very tight quick agent loop feedback. Think about infrastructure as context. So, when you begin, you're starting probably with a single node, but when you you start scaling out, this you're going to start feeling this pain. And secure sandboxes facilitate self-healing in that in the inner loop, the agent can fix bugs and build itself new tools as needed. Help your agents help themselves. Thank you. We got time for a question. Who's got a question around here? Anybody? Let's see, let's see. I think people are hungry. >> [laughter] >> We got one over here. Wondering if you've uh got got an opinion on what the right size and number of agent tiers are. You you showed an example of four tiers, is that typical or That was special just because they they had a pretty scale out use case, like um and so, as you saw, it was like 200 steps, 100 100 LLM calls per software product. And that's non-deterministic, too. Like, for any given prompt, high-level prompt to populate a particular SaaS product, you don't know how many research threads are going to happen, right? So, I think that to from my experience at least, that's kind of like the upper limit of scale that I've seen so far. Um generally, it's just an agent and a sub-agent. Like, I wouldn't go generally deeper than that. And even then, just one agent with a bunch of tools. Like, I feel like that's for for at least the internal use cases we've built, that works really well. Um especially if there are uh like the butter robot, they just do one the one thing, and they just do that one thing, right? But if you're doing deep research and you're you have like this coordinator layer, and and I think they arrived at that solution because they were finding a lot of duplicate research threads. And so, that, you know, they implemented that semantic conversions um algorithm. Yeah, but generally, I'd just start with one. Um and then one sub-agent uh layer of sub-agents. So, the the sub-agents can do different things, right? Yeah. Excellent. Let's give one more round of applause for Neews. Thank you. >> Thank you, dude. Yeah. Cheers, man. >> Yeah, thanks. Uh leave this here? No. All All folks, go eat. We'll be back here at around 2:00, I think. You should have the agendas. You probably know better than I do. Enjoy. Hopefully, you make some new friends. You munch your lunch with the new bunch. I don't want to see anybody eating with anybody they already know. All right? So, we'll be reconvened back here and yeah, awesome. >> [music]