How Agents Changed Vibe Coding Forever

Key Takeaways

So our our primary objective is to save human time. You saw a lot of like interesting demo videos, but I I would say like nothing that really worked dayto-day. The more we played around with uh tool use models, the more we realized, hey, the the assumptions here have changed around what the model's capable of, you know, users of AMP, which is our our new coding agent, uh generating like 80 90 95% of of their code. we we do spend a lot of time thinking about how to nudge the user uh toward the the right way of doing things. So uh one specific example of this is >> when we talked last what right when you released a coding agent I think >> that might have been shortly before or shortly after we released Cody which was >> it was right around when Cody got released. >> So I think that was maybe like a little bit after the initial release of Chad GBT. >> Um and so that was at least like one AI era ago. That's a good way of putting it. >> And so what has changed since then? Uh like I would say like at that point the the dominant uh modality of using LMS for coding was still this kind of like copilot autocomplete mode where you just like type a couple of characters of code yourself and then you get uh a line completion. Um and at that point what we were really excited about was uh the whole like rag model. So we figured out pretty early like hey if you combine uh search and information retrieval and uh use that to fetch relevant code snippets in conjunction with a really high quality chatbased LLM. It was a really powerful way of of doing code generation and technical question answering. >> You guys were uniquely like well suited for that if I remember correctly because >> yeah so source graph of course like our our first product to market was a code search engine. >> So it's like this makes a lot of sense. >> Yeah. So like we're our bread and butter is helping developers read, understand search code. And turns out that uh that's that's very very useful still in in the AI era. Um but the big shift since then has been the uh um uh the emergence uh shall we say of uh coding agents and really it's it's the tool use and uh reasoning uh models that have uh driven that or enabled that because a lot of people tried building agents in in the era of chat LLMs and you saw a lot of like interesting demo videos but I I would say like nothing that really worked day-to-day but now we have really good agentic tool use models that are enabling a uh essentially a new application paradigm that we call agents uh that are are kind of taking the the level of automated code generation to the next level. It's it's going from you know 30 to 50% which is what we saw in the the chatbased uh LLM era to now um you know users of AMP which is our our new coding agent uh generating like 80 90 95% of of their code. Your theory is the IDE environment is already dead. >> Yeah. So I think this is like a general theme which is the application architecture that was made possible by chat LLM. So think LLMs in the era of GPD 3.5 or 4. There's sort of like chat GPT era. There was a very specific type of rag uh application architecture that was ideal for that kind of model. Um, and our our our coding assistant, uh, Cody followed that model. So did many of the other tools of of that era. Um, so the AI IDE, the VS Code fork, uh, was kind of the the pinnacle of that era, I would say. >> Um, but what has basically happened with this new generation of LLM is that they have uh unlocked this new capability uh, tool use plus reasoning. So those two together provide this kind of like agent capability and that in turn unlocks a new set of interactions at the application layer such that a lot of the UX that application builders built for the old era of chat LM is now outdated and in fact I would say in direct tension with the ideal UX of uh coding agents. >> All right, this is where it gets spicy. So then what does the new world look like? So the new world is much less manual context management and a lot less uh kind of like guey chrome and like different toggles. So I I think what you've s what we've seen with the uh the chatbased LM application architecture, the ragbot uh application architecture is just there's so many toggles now like you got to like manually specify uh through different like rules files and different different ways of like tagging in uh relevant code snippets. There's a lot of UX Chrome around managing uh what goes into the context window >> so that the LM can do a single shot with agent >> with that just to sorry to interrupt the funny thing there is >> if you know certain tricks it performs better and so there's some people that are doing it really well because they've played around with it and they've been able to tune the knobs and then there's others who are like yeah that kind of helps I guess but it also kind of messes up. >> Yeah, exactly. So, there's this kind of like strategy for getting the most out of uh like a a chatbased LLM tool. Um, and now a lot of that strategy has essentially gone out the window because uh with agents, agents have the ability to use tools themselves, fetch contexts themselves. Uh, and so it's much less the human manually managing what goes into the context window and more you describing the higher level of what you're trying to do and letting the agent go figure that out. The the analogy I like to draw is um it's roughly similar to the transition from like the Yahoo era of the internet to the Google era of the internet where like in the Yahoo era it's like what what did uh what did uh what did Yahoo look like? It was like a million hyperlinks and the the the UX was like I'm clicking through a nested series of links to find what I'm looking for. So there's a lot of pointing and clicking and kind of like manual following of links. there's a lot of like knowledge that I have built in for, you know, what pages are good. When Google came along, they essentially took that whole UI and are like, you don't need that anymore. Just type what you're looking for and we'll get you to the right thing. And I think agents, coding agents specifically, are are very similar in in uh in that regard. Whereas wherein like a lot of the strategies that people develop for getting the most out of chat LLMs, uh that's like the pointing and clicking in the Yahoo era. You no longer need to do that. Um there is a new skill set that you need to learn which is uh not as trivial as using Google. It's more like how you prompt agents to do uh their work most effectively without active human intervention. But that's sort of a a very different skill set often in direct tension with the skill set that people learned in the chat LLM era >> with the guey. >> Yes. interesting because >> what the guey leads you to do is it really leads you to kind of like micromanage the LM which was necessary in the old world but now with agents you kind of want to just give it the appropriate context the appropriate feedback loop and and let it run. Ah, yeah. And you want to let it do what it does and almost >> give it that freedom that if you're putting too many micromanagement on it. >> Yes. >> In a way, it's not able to get what it wants done because it's feeling restricted. >> Yeah. And the other the other point uh or the other thing that happens when you micromanage an agent is that you as a human also get frustrated. Uh, so like a common failure mode that we see is people who have overindexed on sort of like the the cursor way of doing things. They're like, I want to be in there and like I want to direct it at every turn because that's what that UI trains you to do. It's like uh at every turn I want to review the change before I apply it. >> Um, but with agents, what you want to do is you want to give it enough context to figure out itself what the right things to search for, what the right feedback loop is. It's similar to like >> um another analogy I like to draw is like the the previous generation it was like the AI was was like a coding uh student like you had to be there to review every single little thing they did and they're like okay you did this right and now let's go apply what you did. Now, it's gotten to the point where it's more like a a professional engineer. Um maybe like junior or mid-tier engineer. Uh maybe maybe even senior in some domains. But with with an actual professional engineer, what you don't want to do is you don't want to babysit them. You don't want to be instructing them at every turn like okay now read this file. Now go do this change. It's more like hey here's the overall context. I think you should, you know, use this command to run the tests in this case or uh use playrite to take a screenshot because you're iterating against UI code. Here's the general shape of the feedback loop that you want to construct. Now go off and figure it out yourself. >> It is very much like a declarative way of doing things. >> Yeah. Yeah. I would say it it's it's less like it's less in the weeds. It's more like let me articulate at a high level the key points and I'm going to let you figure out the low level. And how have you seen the best folks being effective in getting that context that it needs? >> I would say there's there's a wide spectrum uh of how far people have able been able to stretch the coding agent. So the the top 1% um we love those users because in some sense they're kind of like discovering the future along with us. So when we look at the AMP user base, uh a lot of the forward-looking things that we build are directly targeted at emergent behaviors that we observe in the kind of like top 1% of users. These are people whose token consumption uh is like 10x in some cases like 100x what the median uh user is. And there we we observe a couple different things. One is there's an emergent set of strategies or tips for instructing the agent to get as far as possible. This is like you know what sort of details do I do I put in the upfront prompt to enable it to construct the right feedback loops to to search for context in the right places. >> And then the other thing we we notice is is more and more parallelization. So AMP is available both as uh an extension inside VS code as well as a CLI. And so there there's a lot of people who use the editor extension for uh more complex tasks, tasks, tasks that involve, you know, more like complex chains where you want to kind of like remain in the driver's seat, so to speak. And then they'll use a CLI for parallelizing a bunch of shallower tasks. So like you'll have like a T-Mux window where they have, you know, three or four AMP CLI instances going on different, you know, shallower issues or bug uh bug fixes. That is so cool, dude. That is so wild to see. It's crazy. >> And how are you? So, I guess you're just doing product uh feedback with all of these power users. >> Uh so, we've we've hired a good number of them. I like to say like Yeah. So, um it's a great source of uh you know, forwardinking devs. Uh some of some of whom we've welcomed to the AMP core team. >> Uh and we also talk to them a lot. you know, we interact over uh social media channels, we have a discord, we hop on phone calls. Um, but it's it's great to to talk to that set of users because in some sense like we're so early right now. >> Um, like people talk about AI as if it were one monolithic block, you know, that has, you know, the wave has is but one giant wave since CHBT, but it's actually um a succession of of multiple waves, I would say. And we are so early on the agentic model era that a lot of our product development process is really kind of like partnering or sitting down with our our power users and discovering alongside of them, you know, what the possibilities are. >> Is that how AMP came to be? Because you saw that folks were clicking around too much and you realized maybe this isn't the best UI and UX that we can have. That was a big part of what motivated us to to build something from from the ground up. So, you know, we had Cody, which is uh an assistant that was really good in the uh chat LLM era. Um, but the more we played around with uh tool use models like, you know, sonnet 37 and now uh Claude 4, the more we realized, hey, the the assumptions here have changed around what the model is capable of. And if you're if you're holding it properly, you can actually get a lot more out of it than you could in in the previous era. The problem is that a lot of the old UI paradigms are kind of like actively working against you uh getting the most out of of coding agents. So this is something that we kind of realized in using it ourselves heavily and also talking with a lot of our power users. In fact, one of the one of the folks that we hired uh this guy by the name of Jeff Huntley, he was at Canva at the time. He actually wrote a blog post about how he thought most people were using AI coding tools incorrectly um because they were still kind of using it like you know a Google search++ or in a very like chatbased paradigm. >> Um and we brought him onto the team because we're like this is a guy that gets it >> different >> and that really understands hey you should be instructing these things. you should be it's almost like you're programming them through natural language uh if that makes sense. So like you're you're articulating a very set of you're articulating a set of very precise instructions uh in much the same way that you would kind of like articulate those instructions to a smart but still you know junior engineer. So like you're giving a lot of context up front and you're allowing them to get much further on their own. And how about the idea of just validating when code is working or not? >> The beauty of agents is that they have this kind of like built-in ability to construct these feedback loops. Um, and so when you're using AMP, for instance, when it's generating code for you, uh, as part of that code generation process, it will seek out an appropriate feedback loop. So if you're doing front-end code, uh it can use uh a tool like playright for instance to screenshot the front end of the application as it's working. So you say like hey go make this background red or green or blue can actually take a screenshot and verify whether uh a change it made to the code had the intended effect. Uh similarly for back-end code, it might be like a unit test suite or some other uh command line invocation that it can use to validate whether it did was uh uh whether something it did was correct. Um in much the same way that you as a human developer would would seek out these feedback loops, right? It's like readaluate, print, that sort of like core loop. >> Um that's what agents are good at figuring out. Um, some cases they need a little nudging. Um, just as you know, humans need need a little uh nudging or uh some pointers in in some cases. But by and large, if you can get the agent to uh figure out that feedback loop um with with very high confidence uh it it will iterate to something that uh is is is mostly correct. Why do you feel like coding agents had this breakout success and were uniquely positioned for such a lift with LLMs and the whole AI revolution? I think the the the answer to that question comes down to uh the im the immediately proceeding question which is uh you know how do you validate something is is correct and coding is one of those domains where you have a very strong validator in the form of a compiler or a unit test uh runner and because you have that validation point um you essentially have a way a very reliable way to generate high quality synthetic data. So, model evolution is is ultimately a data game. Uh, and there's two ways to acquire data. You can either collect it from the wild uh or you can create a synthetic uh learning environment in which you place your kind of like robot or agent in there uh and allowed to do stuff with feedback about what's good and what's bad. Sort of like reinforcement learning environment. And uh I think at this point we've exhausted the amount of of publicly available large uh corp of of of data. So like you know that that uh those sources of data are are largely played through. Um but coding is one of those domains where it's like you can create a simulation environment with unit tests and and the >> comp doing that >> uh to to to a certain extent. Um so for for certain special use cases uh we we don't do foundation model training uh as of yet but uh for for certain targeted use cases uh we we we do that sort of uh uh kind of like validation and and training. >> Yeah. I've just been hearing about how more and more people are doing simulations more. It's more common to do that just to figure out where you have strong capabilities and where you're you maybe are are failing silently sometimes even. >> Yeah. It's it's essentially what you're doing is you're designing a game that approximates the what you want in real life. So in in all the domains where AI has gotten really good, you know, think about like, you know, playing chess or playing some other form of game, it's because you have this feedback mechanism that tells you, hey, you're winning or you're losing. Uh, as long as you have that feedback mechanism, you can turn that into a reliable source of training data because essentially what you what you do is you take your model at a given snapshot uh and then you just run it. you you you say go play the game, you simulate the game and based on the moves that the model takes uh you say like okay you know plus points or or minus points and that's essentially what you're doing in these coding reinforcement learning environments when you say like oh compiler error or oh unit test failure. >> I like that way of looking at it. You you just sit around all day and you're thinking of a knowledge thinking of analogies. Huh. The the thing that I'm also wondering is it feels like we had a big jump from these rag chat bots and the way that we were co-pilot writing code. Yeah. To then all right, we're in the IDE and we are doing this almost like click ops type of stuff and very micromanaging. Now you're saying we've got a whole new era that's being born with AMP and how you're giving it this context, as much context as possible, and then letting it do its thing. >> Is that the last era or do you feel like there's another one that you want to get to? It's just not yet possible or it's in the works. >> I don't think we're in the the final era. I think things will continue to uh evolve. So you know one of the things that we're doing now is we're thinking about how to combine multiple models effectively in this new agentic paradigm. So uh in the old world uh the the name of the game was was simple rag. So like every uh AI coding assistant had like a model selector where you just like you know whatever model you want to use you can use that and then we'll just fetch the relevant snippets put in the context window and generate the response. Uh I think in the agentic area you have to be much more thoughtful about the models that you use. So you know we use one model for the core kind of like tool use and agentic driving of AMP. Uh and we just shipped uh a feature that allows you to use uh another model 03 actually for for in-depth reasoning because turns out there's certain types of like nuanced uh problems that you might want to tackle that where where these like uh reasoningheavy models can do a lot better than the the models that were uh kind of like trained primarily for agentic tool use. So that that's one way in which uh the the paradigm continues to evolve. It's like now moving beyond just simple agents to maybe like reasoning uh agents or agents that can uh use different types of models to to do more things in depth. >> And you want to abstract that away from the user or you want to have it that every time I go and I give a task to an agent I can say here's your three or four models you can choose from you figure it out. Uh I would say if you uh the we want to enable kind of like a a spectrum of use. So for the the first time user you know you don't have to know that we have this uh so the tool is called Oracle because uh 03 is is such a powerful reasoning model. It's it's like talking to an oracle of sorts. Uh and uh you know we we don't want that to be a prerequisite to being able to use AMP. So if you don't know about what tools AMP has access to, you don't need to. Uh it will just select what it's what it thinks is the best tool for the job. Um but at the same time, uh instructing coding agents, uh in our view is a pretty high se ceiling skill set. So uh you can get good at coding agents in the same way that you get good at your editor of choice or you get good at your programming language of choice. And for our power users, we do see uh uh prompting or or query patterns where they're saying like, "Okay, I want I want you you to use the Oracle in this case because this is a little bit of a hairy problem. It's it's more nuanced. I want some more in-depth thinking." Um so there is some exposure, but it's not at the point where it's like, "Okay, decide what LM you want to use for this case." That is now an implementation detail. I think it's it's almost, you know, it was a best practice to expose that to the the end user in the chat LLM era, but now I think it's it's a more needed. >> Yeah. It's funny. Are there any other anti-atterns that you're starting to see and that maybe surprised you? >> The number one antiattern is people trying to use coding agents in just the same way that they use the the chatbased uh coding assistants. Um, and I would say those antiatterns largely fall under this umbrella of in the chatbased world, you wanted like the human had to be in the inner loop of like back and forth between you and the model out of necessity, right? Because uh each, you know, model invocation was like a roll of the dice. Uh, and in the chatbased world, you know, the the probability of it just working was probably lower than 50%. like more likely than not it would make some subtle bug and it had no way of correcting itself because it couldn't iterate against feedback. It couldn't use tools and so it couldn't fix its own mistakes. Um and so as a consequence you wanted to be in the loop uh so to speak as a human to constantly course correct it. Um with agents it now has the ability to gather that feedback on its own. And so uh if you if you instruct it properly in many cases the fidelity you get from a single model invocation uh like a single file edit or a single uh you know bash command that it runs is closer to like 90 95 99%. So like you can actually like you can get get out of the way much more uh if you use it properly and and it can do more for you. uh but that almost requires uh almost like an active rejection uh of a lot of the best practices that people learned in in the chatbased LM era. So it's almost ironic like some of the people who are struggling the most to use coding agents effectively were the ones who early adopted uh chatbased uh coding tools. this little microcosm of the macrocosm or that it's a >> Yeah. Just let go. >> Yeah. >> Trust the process, man. It's going to work out. >> Yeah. Oh, that's hilarious. So, uh I had I had a question about >> that. I can't remember now. It's not coming to me because I was thinking about trusting the process and not the Hold on. This is the power of post-production. We can trust the process. >> Yeah. Trust the let me trust the process of my question asking capabilities. So, you mentioned the different power users and how they're paralyzing different things, and that seems to be one very advanced way of doing it. I wonder if you've found nice tricks other than that that maybe aren't the power user tricks, but it's just in the way that you're prompting or you're asking the agent to do things. Like my mind instantly goes back to the early days of chat GBT and we started asking it like think step by step on this and everybody was like whoa it's so much better when you do that and have you found any of those almost like prompt tricks or maybe there's other tricks that aren't even in the prompt or in the way that you're asking. >> Yeah. So um by and large and and the best way to to discover these things is really through experience. So, I will do my best to like tell them to you in the moment, but it's it's no substitute for actually like using it and kind of like building the intuition. Um, but in my experience, there's kind of like three uh buckets of of like prompting tricks. Um, there is what I would call like uh context hints. Uh, number two would be feedback loops and number three is uh kind of like structured uh approaches uh slashplanning. So the the the first bucket is really about helping the agent uh figure out what tools uh it needs to invoke or or essentially where to look for the relevant context. So especially in in large code bases often times it can be a little bit tricky even as a human to find the exact spot that's relevant to a particular task. And so agents um agents agentic LMS uh the context windows these days are much larger than they used to be. Uh you know I think you know sonnet uh 4 has uh 200,000 uh tokens total. Uh Gemini now has a million. Um and so the context widows are larger but they're still finite. And what that means is um the the the more information you give to the model about where to look, the fewer tokens it has to expend uh finding uh uh the sort of like general vicinity of of uh what's going to be relevant. Um and so the the more hints you can say like oh like look in this part of the the codebase or I think it's under these directories. uh or maybe like use use this tool to fetch the context. That can help a lot. The second thing is feedback loops. So, you know, I was telling you before about feedback loops and how they're critically important. Um these are essentially like, hey, you know, uh use this tool or use this command to uh validate your approach. Um often times it will will infer the appropriate tool to you use on its own. But in cases where it's it's not trivial to figure that out, again you can save on on the main context window um by by uh kind of like nudging it in that direction. Uh and then the the third approach is just uh adding structure to the overall approach it uses to uh to solve the problem. So, in the simplest form, uh this is just like, hey, before you go do this large and complex task I'm about to give you, first write out a plan of steps, uh and and maybe even let me as a human review that set of steps so I can ensure that they're correct. Um and then and then more and more uh we've we've sort of like built additional features into the product where knowledge of those features can help. So one of the things that we've shipped uh uh well I guess not so recently now it was like a month and a half ago so it's like you know eons ago in in uh uh AI land um we ship sub agents so sub agents are essentially uh I mean as the name suggests they're agents within an agent so like the the main agent can invoke a sub agent to go do uh a subtask like you know searching the codebase or uh going and implementing a feature in one part of the uh the codebase. And so having knowledge of what sub agents are good at uh and kind of like nudging the main agent to use them where uh appropriate um can help you conserve the context window because the beauty of sub agents is uh once they complete their subtask uh they don't uh essentially like the the tokens they use the context window gets garbage collected. They don't use up the context window of the main agent. So that allows you to get uh further in complex tasks because you're you're essentially chaining together these sub agent calls that don't eat into your overall token budget in the main agent. That makes sense. >> Yeah. >> Yeah. I've heard that described as agents as tools. A lot of people are that's like the hot buzzword these days. It's like oh agents as tools. It's coming. And >> yeah, everything is a tool and tools are just function calls uh at at the end of the day. Yeah, I was laughing with a friend cuz I was saying, you know, even humans are tools at the end of the day when you're asking for the human to give you the feedback. It's like invoke the human tool. >> Yes. Yes. It's like and is is the agent the tool or am I the tool for the agent to get his job done? Sometimes the line blurs a little bit. >> Yeah. Yeah. the part that you just broke down really well on how to conserve that context window in a way >> so that you're not using it all. A because it's finite and maybe you don't have the ability to throw everything at it, but B it's really good for the cost and keeping the cost lower. I can imagine when you're looking at agents and folks that are using agents, I think there's probably two lenses that you look from. You look from like, okay, the consumer is trying to keep their costs low, but they're interfacing with source graph in a way and AMP and so yeah, >> you also have to be weary of costs and passing on the right costs to the users and the pricing and all of that fun stuff. >> Yeah. How are you looking at all of these different costs and how do you feel like we've all heard this idea that oh well >> LM calls are just basically going to zero >> right and so it's just getting cheaper and cheaper but now if you're talking about agents using sub agents that are doing super complex tasks >> it still could be like 50 or 50 cents for a task to get done and >> or maybe even five bucks who knows So our our primary objective is to save human time because that is still the most valuable resource by a huge margin. Um and so one of the the core principles we we've adopted is essentially to to not to not uh to not worry too much about uh keeping the cost of the agent super super cheap. So, you know, agentic coding tools, they look expensive relative to chatbased coding tools. Um, like your your average token spend is is growing from on average $10 to $20 per user per month to uh you know in the hundreds in some case in some cases in the thousands of dollars uh per user per month. like when when we talk about the top 1% of users who are really like redlinining uh the what the model can do and what what the coding agent can do um often times they're they're pushing into thousands of dollars uh per month territory. And so that looks expensive uh to people, but if you look at the amount of human labor that's being saved uh given how productive people are with these tools, uh it's like a no-brainer uh trade-off. And um I I think a lot of other tools I I think they narrowly focused on they they overindex on like hey how can we keep the cost low as compared to the chatbased uh LLM era. And I think that's that's a that's a very poor trade-off to make because um it's kind of like saying like hey I have this magic tool that can save you the human hours of hours per day. Your time is super valuable. like human developer time is is still by far and away like one of the most precious resources uh within your engineering or uh and even you know if you're spending like $1,000 uh per month that comes out to what like you know 30 $40 $50 uh per day like how how how much how many how many minutes of human developer time does that translate uh to to saving and so I think what you're going to see is uh more and more people start to have this realization over time that oh it's I shouldn't be thinking about the the baseline cost of this because really the upside is far greater the amount of additional productivity and the the amount of additional feature velocity that I can unlock in my engineering team with coding agents is far greater than uh the cost that I will be uh paying for them. So like right now you you you still see you know CFOs and people in accounting and the finance department being like you know oh it's it's difficult to forecast but I think it's just going to play out over time where the companies that trust the opinions of their of their developers and encourage people to uh make the most out of coding agents. They're just going to move much quicker and over time the market will reward the people that uh prioritize uh developer developer productivity essentially. >> Yeah. Do you feel like we are not going to be >> doing much of the >> or actually so I've played around I've played around with a lot of the different tools and I feel like there is something that's happening right now where I no longer want to click around and get things done. I also don't even want to like code to get things done. I just want to say it and then it goes and does it for me. >> Yeah, >> that is very easy to do with almost coding tool type things. Like if I say, "Hey, connect my website to this database." Great. That should be possible. >> Yeah. I don't know if there is a possibility for the world to look like this in the future, but I would love it if I could do that with any application. >> I no longer want to have to write out >> the or click around to get things done. And >> I almost feel like we are getting spoiled in a way. >> Yeah. with the software being able to do it when you're coding or when you're creating your your application with whatever. >> But like are we going to be able to do that in Jira and Confluence at some point? Are we going to be able to do that with just like >> voice next? Right? Like I it you got to think it's going there. So, I saw a tweet the other day. Uh, it was something to the effect of the guey, the graphical user interface was a blip in between uh command lines and uh agents. >> Oh, interesting. >> Um, which I think you know rang very true in in the sense that um I I I think it's exactly what you said. I I think the what we think of as a software application today is is going to look much different in uh a few short years from now. So it's not that like I don't think like visual interfaces are going to go away entirely. Uh I would I would say like precisely what I think is going to happen is that the primary input modality to uh computers or to to software applications is going to shift from graphical modes of input like pointing and clicking to more textual forms of input like typing or speaking. Now the output what you get back from the computer might still be visual. It's like you know I type in what I want and then show me the results. Maybe like uh you know >> Airbnb listings with photos. >> Yeah, exactly. Certain things that chat cannot describe. >> Yeah, exactly. It it like I don't want to read all that text. Just show me like a picture uh a picture is worth a thousand words. But in terms of like how like articulating what I want out of the application, I I I just think of it in terms of like bit rate, right? Like what what what's the bit rate of these input modalities? Pointing and clicking is a very low bit rate. It's like couple bits per second at most because you have to like you know takes time for you to drag your mouse cursor to the right button and then you got to it's like it's a very primitive form of of communication. It's like, you know, monkeys and apes do that, right? Humans, we have the the we have this innovation called language. And language is beautiful because it's it's a relatively speaking a high bit rate form of of communicating our intent. And now computers can actually understand language and also translate that language to a series of actions that actually perform what you want it to do. And so I to your point I think a lot of the application input uh experience is going to shift toward textual or or voiced driven uh forms of articulating what you as a user want. >> So what are some gnarly things that you encountered while building out AMP? maybe on the infrastructure side, maybe not necessarily the the users and the evals and all of that stuff, but stuff that you as you're building it out, you're like, "Oh man." Like, "I didn't think it was going to be this hard." Yeah. There there's a lot of like trickiness and and nuance to designing uh the the user experience. Um and I I think it really requires thinking from first principles what you're after. So like we've given a lot of thought to how to uh bake in the appropriate feedback loops, help it get to the right feedback loops, help it to uh get out of like common failure modes where it kind of like loops and tries the same thing over and over again. That's a that's a common issue with with a lot of agents. Um how to conserve the context window. We we've given a lot of thought to that. Uh where to use the appropriate model. So like what models are best for for which task. Um, but these are very different questions than the questions that we've traditionally asked around UX design because traditional UX design is very visual. It's like, you know, how do I lay out the set of how do I lay out the button panel, so to speak, right? >> What color should the button be? >> Yeah, exactly. >> Where's people I remember reading a blog post on how people look at web pages and there's like the Fshape of the where their eyes go and the attention goes. >> Yep. And so, you know, like AMP doesn't have uh any of that really like the the input interface is very simple. It's just a text box and like you write write what you want or or some cases in some cases people like to do the voice mode. So, they'll use like the uh like Mac OS like voice input to to to speak to the the agent. Um and and that's the the primary way of of getting what they want. Um but it's a very different set of questions. So you kind of you can't rely on the rules of thumb that uh people developed in in the kind of like point-and-click uh guey world. You really have to think uh from the ground up like okay I as a user what do I actually want? How do we want to guide the user to this behavior that's kind of like new in this this new kind of like paradigm that unlocks a capability but still feels familiar enough. Um, fortunately developers are are accustomed to using command- driven interfaces a bit more than the average computer user. Um, uh, so so that helps. Uh, but we we do spend a lot of time thinking about how to nudge the user uh toward toward um the the right way of doing things. So, uh, one specific example of this is, um, AMP is maybe one of the only I I think we're one of the first, uh, agents where, um, typing enter in the input box does not submit the query. So, typing enter just introduces a new line. Uh, you have to hit command enter to actually submit uh, your request to the agent. And the reason we did that was it was a subtle nudge to encourage users to create longer prompts because the more information you give the agent, the more reliable it becomes, the more it can do for you. Um, and so that was like a subtle nudge to users to say like, look, >> don't stop here. >> Don't stop here. Uh, this is not Google search. Don't type, you know, four four keywords and then expect it to read your mind. Yeah. >> Um, in in many cases, uh, getting to what you want, especially if what you're trying to do is is more sort of like out of or unique, right? You actually have to give it the information because it's not a mind reader. Like the the information has to come from somewhere. Either it's baked into the prior learn during training or it's it's it's going to be embedded in the the words uh and tokens that you give it. right now. >> Did you think about adding certain shortcuts or hotkeys? And I've seen this done where you have the prompt box, but you also have little boxes underneath that you can click on where it's like, here's some common workflows or some common questions, that type of thing. >> So, I think this is one of the the the things that we did very differently. It was sort of like a contrarian take. Um, we we actively didn't want to add additional toggles or modalities at the bottom. Uh, because number one, that's like mental overhead and it it makes it so you have to point and click again, which is I I think, you know, we now live >> trying to get away from >> Yeah, we're trying to get away from that. Like, it's the age of agents. You should just describe what you want and and be able to get what you're looking for. Um the second thing is that with a lot of other applications that do this, the the more um toggles and switches you add, they essentially um exponentially uh complexify the the interface surface area of your application to the point where like if you if you introduce like a single binary toggle that you know like other other um coding tools for for instance they have like an ask mode or an agent mode or like a planning mode. Um, like if you introduce a toggle that has like three different modes, now you essentially have three different like mini product experiences that are all very different from one another. If you introduce another toggle again with three modes, now it's 3x3 because you have this like cominatorial like, okay, what if I choose option one from the first toggle, option three from the the second. Now there's like nine possibilities for what the product experience looks like. And it's very hard to maintain a high quality product experience if all your users are using essentially different products, right? Different product experiences because they all have some like different uh configuration that that they're using. >> And how about the times where you want to zoom in on specific things? Because I know like if you're if you mess around with lovable, they have the >> the bullseye and you can click on that and it's this new mode that all right, it's got the specific parts of the web page that I can click into and then prompt it so that it changes that. >> Yeah, >> maybe you thought, well, we don't need this. We're just going to have the user put that into the chatbot. the the way we like to do it. So if there's some like capability or some uh behavior that is more um specific to a particular use case, uh the way we like to do it is put that into a form of a tool that the agent can use. uh and and then enable the the agent to invoke that tool in the the right situations or or enable the the user to nudge the agent to use that particular tool in the right situation like hey you know go use playright uh for iterating on this UX and the reason why we think that is better than modalities is that um tools the complexity of tools uh it's it's more like linear growth in complexity rather than cominatorial growth and complexity. So like each additional tool it's not like uh it's not like modes where it's like bless you >> each each additional tool uh that gets added it's it's just another tool that that can be used. It's like you know O of N in terms of complexity whereas if you added a new modality now it's like you know M by N by you know K. It's you get this kind of like exponential blow up in in terms of the service area. You have tools that are standard off the shelf for the users to use and there's a description and the documentation about it. >> Yep. >> But then users can also bring their own tools I imagine. >> Yes. So we have uh three kinds of tools. There's the built-in tools uh that are just you know as the name suggests built in uh you know basic things like reading and writing files and executing bash commands as well as more advanced built-in tools like like the Oracle that use different models uh for for advanced reasoning and and other use cases. Uh there are what we call connections uh which are tools that call out to third party APIs. So think about like you know bringing in your issue tracker or bringing in your observability tool uh pulling in additional context from those sources. Uh and then the third type of tool is MCP server. So MCP is, you know, by now like everywhere, right? And so the the beauty of that is that you have all these different other tool builders out there that have built MCP uh servers that front their application or their service uh that can then be pulled in uh to to to AMP. >> How are you thinking about the tools that you support and are natively giving to users versus the other two? Yeah. So, I would say all the the tools that are necessary for the day-to-day uh like core interloop of software development where it's like you know you're in the code, you're reading and understanding the code, you're writing code. Um that's our bread and butter. So, we like to bake those in as as really good first class tooling experiences. Um, and then there's also like a second wave of thirdparty tools that are just like so common that we also want to make sure that uh those integrate well and that's why we have these like connections uh these kind of like firstparty connections to third party services. So like bringing in uh things like linear or github or sentry um the the way we bring these tools in the the description of the tool and the set of arguments and and how to invoke them um it's very difficult to abstract fully. It's not like you can have the same set of tools that work really well in a coding agent as works well in I don't know like a generic uh you know enterprise knowledge retrieval agent or or whatnot. Uh and so we we try to refine the the tool definitions for those. Um but then we also recognize that there's going to be a long tale of things that people want to integrate especially because you know we we serve a lot of the the Fortune 500 and and there's uh uh you know each each Fortune 500 uh codebase is it's like its own special uh environment with all its like uh different internal tools and uh unique combinations of external tools. Uh and so we we also want to enable our users and customers to build uh tool providers, MCP servers uh that connect out to the unique tooling environment within within their own uh company. We bring that into AMP as well. [Music] Heat. Heat. [Applause] [Music] Heat. Heat. Heat. [Music]