How agents broke app-level infrastructure - Evan Boyle

Key Takeaways

How agents broke infrastructure. Particularly, we're going to focus on the compute layer today. My name is Evan Bole, the founder and CEO in Genisx. I was an early employee of Palumi and I've spent my entire career working on cloud and developer tools. Typo warning. In the spirit of embracing AI for this talk, all diagrams are AI generated, but 40 still has some spelling issues. So, thank you for your patience. We all know this journey. We start off as AI engineers with a simple prompt and a couple of tool calls and typically this is enough to raise our $10 million seed round. But from there, when rubber meets the road, we end up building workflows, right? We take that fully non-deterministic code and make as much of it deterministic as possible. That might mean chaining a couple of very specific tailored prompts together that we've written eval uh and more carefully controlling the context. This takes the runtime of our workflows from 30 seconds to a couple of minutes. But eventually we all end up becoming data engineers because the most challenging part of the problem is getting the right context into the prompts. And that means that we might have to crawl a user's inbox, ingest code from GitHub or many other artifacts that require lots and lots of LM processing. But our assumptions around infra have changed. In web 2.0, you would build a simple web service where you'd make an API request, maybe talk to a database to fetch some data, and return all that back in a few tens of milliseconds. In AI applications, our P1 is a couple of seconds at best, and that's only if you're using a fast model or if you hit a prompt cache. In the previous era of infrastructure, if you had a request that took a couple of seconds, your on call was getting paged. Um, and so the infrastructure that was designed for the last 10 years of the web is not really designed for the applications that we're building today. LM applications are really built on shoddy uh foundations. It's really hard to build reliable apps today. And if you're like us and you run a tight ship, you might have a Slack channel that looks like this where you subscribe to, you know, issues and outages from your dependencies. But luckily, this one only lasts, you know, an hour. By 10 p.m., it's resolved. But, oh, at 1:20 a.m. we're back at it. And 4:00 a.m. it's still going on. But surely, you know, Chat GPT and OpenAI have have better availability, right? Well, I wouldn't really trust this 99.9% uptime number. And no, it's not lots of yellow and red in this graph. But maybe we could have failed over to, you know, Gemini during this time period. Nope, we have an outage during the exact same time period. And even if you don't hit an outage, you still have rate limits to contend with. A lot of these traffic patterns are going to be extremely bursty as you batch process documents from a user or you onboard new customers. And not everybody has tier five rate limits that require spending thousands of dollars just to get to that point that you can start experimenting with high levels of traffic. So we start off nice and happy with our prototype. Maybe we built something on top of the next.js chatbot template. That's easy, right? But as our workflows start running for minutes and hours, us full stack AI engineers accidentally become data engineers. So how do people solve this today? You know there this used to be something that was rather niche longunning workflows data engineering. So there is an existing set of tools. People build these Rub Goldberg machines on top of cues like SQS. They might use batch processing data engineering tools like Airflow or durable execution engines like Temporal. But me personally, I'm a Typescript developer. I'm a full stack engineer. And frankly, I don't want to use anything that was designed by a Java engineer. No offense. I'd much rather use serverless. What about existing serverless providers? They aren't really well suited for these kinds of longrunning workflows. Uh most of them time out after 5 minutes and some of them even limit outgoing HTTP requests and for all of them they don't really have native streaming support. Streaming is something that you have to bolt on at the application layer and not something that's a native part of the infrastructure that you're building on top of. something that's really important when you're building an application that might run for multiple minutes. Users might refresh the page. So you want resumability. And so let's look at some of these common product experiences. We're going to look at both onboarding where we might ingest a bunch of data from the user with an LLM upfront and then also a product experience where we're going to run an agent that's going to take multiple minutes and we have to keep the user engaged during that time period. So during this onboarding experience, the user puts in their URL, right? Right? And we use a single LM call to grab some information from that web page and then detect all the pages that we're going to scrape. Right? And now we've kicked off that scraping job in the background and it's going to take multiple minutes uh and we're going to be making hundreds of LLM calls to extract content from those web pages and enrich them. And we want the user to be able to start using the product right away. So this ingestion can take, you know, a varying amount of time, but we want users to use the app immediately. If we wait 10 minutes, that means we're going to have increased fall-off in the funnel. So, we have to run this in the background, right? And we want to show users the status of this ingestion as they're using the product. We want them to know, hey, as time goes on, we're going to enrich more data and your experience is going to get better. Next, this is an example of building a content generation app, right? We're going to write a blog post uh about building high performance web apps with Versell and Nex.js. And this agent, we tell the user up front, sit back and relax. This is going to take a couple of minutes. And on the right, we're showing them all of the different steps, right? We're running research. We're writing an outline. And we're writing each section step by step. Since this takes multiple minutes, it's really important that if the user leaves the page or navigates away that they don't lose this context, right? And so that workflow that we built when working on a previous product took 3 to four minutes. And we really felt a lot of friction experimenting with longer workflows. Like we were deterred from making our workflows longer than five plus minutes because that meant that we would have to make deep infra changes due to limitations from our serverless provider. Eventually we did rip it out and it was a bunch of work but it took us much longer than we should have to run those experiments uh because of the amount of work that was required to migrate off of you know our easytouse serverless offering. Also, we need to stream both the final content as well as the intermediate status. And if the user refreshes the page and comes back, we want them to see that same thing, just like you get with deep research, right? And if there's an intermediate error, users are going to be way more frustrated if it takes them 5 minutes to get back to that same point uh versus if it's just, you know, a simple web page that you have to refresh or a form that you have to resubmit. So we built a lot to make this easy in our core product uh when we were working on you know agentic workflows uh and we ended up turning a lot of this into an open source library. So we built a simple component model that's infra in infrastructure aware right uh the the framework itself is very aware of the infrastructure it's running on and the infrastructure is aware of the framework and that's so that we can provide things like resumable streams for intermediate status uh and final output and we call this a bit of an anti-framework. It's unopinionated and focused on building blocks. uh it's it takes inspiration from React's component model and applies it to the back end because we know that abstraction is bad when building agents. You don't want to abstract. You want to share, you want to compose, you want to reuse code. So we have components that are reusable, item potent, independently testable steps and then workflows that are collections of components that run together. So this is a simple component that uses a wrapped version of the OpenAI SDK. It's wrapped so that we can get a bunch of other tooling for retries and tracing and things like that, but it's the same exact OpenAI surface area that you would expect. We give you full access to the model. So, it's just a prompt, a little bit of context that's inject ingested and it returns a response. So, a simple function that takes in some prompts and returns some output and it can be independently run or tested. We can also combine it together into what we call a workflow. This workflow fetches the front page of Hacker News, analyzes those posts, generates reports, writes a tweet, and then edits the report. And with this, every single step, we're going to get a retry boundary, an error boundary off each component. And we're going to get traces both at the top level of the workflow, but also all of the components that run within it. Each workflow can be turned into a set of REST APIs automatically by the framework that support synchronous invocation as well as asynchronous invocation and the relevant API so that we can retrieve the intermediate stream and the final output stream if needed. When we run this workflow, we see that we get a trace. This trace shows us all of the nested components that run inside the workflow, including all the tokens, all the details on the OpenAI call, user messages, system messages. So, it makes it super easy to debug. And we have built-in retries. Every component has a fluent API hung off of it that allows you to configure a component with things like retries and a cache. So if I have one component that's particularly problematic, I can add an exponential retry policy uh with and and configure the number of retries that run. And so this all gets deployed to a serverless platform that we built that's tailor made for longunning workflows and building agentic UIs uh for processes that run in the background. So we have an API layer and a compute layer that are completely separate. And this is really important for a lot of reasons. The API the API layer invokes the compute layer and passes it in a reddish stream ID. That's the last time that the API layer is going to have to talk to the compute layer. The rest of it is going to happen via communication with this reddest stream. Status is going to come from the reddest stream. The output is going to come from the reddest stream. And as the sandbox program executes, it's going to emit heartbeats so that the background processes within the system can monitor and make sure that a workflow runs to completion and doesn't die or doesn't crash. And if it does, we can automatically restart it or notify the user. So key architectural points of this is this separation between the API and the compute layer. It allows us to scale things independently, scale the API layer independent of the compute layer. It also allows us to plug that compute layer. So we can provide uh a compute layer by default on top of ECS, but we can allow users to bring their own compute layer as well since it's completely stateless and just talks to the API layer and the reddest streams to store output. All that API goes to the Reddus stream both for sync and async workflows which means that we get resumability, right? The API layer only reads from the reddest stream, not the compute layer directly. So that means that we can build UIs that allow users to refresh the page, navigate away, transparently handle errors, uh, and they still get the full history of status messages and all of the output. Um, something that you wouldn't get from a typical serverless like infra platform. Like we said, key point here is being able to refresh the page, resume directly from the Reddus stream, and get all of your intermediate status messages. none of the work is lost if the new user navigates away or if the web browser terminates the connection to the API server. So things and lessons to consider when you're building your own infrastructure for agentic workflows. Always start simple. You're not going to write a workflow that runs for an hour on day one. So don't build for it. But do plan for a future that's longunning. In my opinion, this is where agents are going. And I think that we can see this with all of the recent coding agents that have come around experiences like deep research. Like the future is giving your agent instructions and letting it go off and do work for you and communicate with you when you need it. And to make that happen, think about keeping your compute and API plane separate and lean on Reddit streams for resumability. Make it easy to you for users to navigate away from the page, not lose progress and handle errors transparently. All this is also to say you got to take care when deploying. If you're going to let your workflows run for 60 minutes, you have to be very very careful about how you drain workers and how you uh do kind of a blue green deployment pattern. And overall, the devil is really in the details. Easy to get started, but very hard to get this right. Now, it's really, really fun to build all of this infrastructure yourself, but if you don't want to build it yourself, I'd encourage you to check out GenSx on GitHub where we've implemented a lot of this for you. Again, my name is Evan. Thanks for having me at this talk. Uh, and happy building.