Coding with AI // Chip Huyen

Key Takeaways

Okay. So today it's an Asia tech event, right? So I think that one of the biggest use case for AI agent is like coding agents because I'm extremely excited about I feel like I cannot really I know I I feel like I'm one of those people where serious engineers hate when I say that oh I cannot really code without it anymore and I want to talk about it. So uh so this is a bit tricky because I usually try to get my talk in front of the audience because one of the fun part about AI coding agent is that we still learning about it and we still want a lot of interactions like discussions like what is going on. So I like to get feedback from the audience and since I cannot see you um I don't know maybe like you have any feedback or thoughts about the process feel free to send me uh an email like on my website on LinkedIn or Twitter. Okay. Okay. So today we're going to talk about coding with AI. Uh and um and usually when I ask people who has been using it for coding at a tech conference, it's like almost almost everyone is using it. Um and from the hiring manager perspective, um someone recently told me that if he's interviewing a candidate for software engineering role and that person has not experimented with AI coding before, then he would consider that a red flag. Uh and the reason is that we feel like okay it show that um that persons has not is not eager it's not willing to adopt new technology um so when I ask this question online some people say that not everyone has opportunity to try new tools um and that's totally reasonable but regardless of what uh of of what's the view about is uh about AI for coding is I would say that a program foring is everywhere and if you have not experimented look at that it might actually hurt you more than help so for AI coding. Uh there are very many different types of toolings and interfaces for AI coding tool and the most popular one or the earliest one is probably the IDE base when you have a um when when when you have like a putting like editors like VS code uh cursors when you can do like autocomp completions and stuff and then the next generations of a coding tools that I really love is CLI based like terminal. We open the terminal and type in a CL code or like codeex and it can help you do stuff. And another interface is very interesting that I was so really excited about but I have not seen other use is like GitHub based. So the idea is that like let's say you have the GitHub repose or maybe GitLab say have a repository online and maybe somebody post a an issue. So you might invite a coding agent in and say, "Hey, given this codebase and this issue, maybe you want to try to like submit a PR to like address the issue or submit a um a PR, you might invite an agent in to review it." And of course, like the last one is like web interface. Uh maybe so it's especially common on a web applications. So let's say you have a mockup of the interface you want. So you can put it inside the uh you can put it inside the uh you can you can you can post the image in the this tool and then I can have you generate a an application with the interface that you want. So when I talk to companies um I I usually ask people hey which interface that you prefer. So this is one of the the companies there's quite a lot of engineers uh I would be sorry like hundreds like thousand of engineers on the call and they they they did this uh survey when I was asked and the vast majority of people like say oh it's ID and then at the same time I also want to like ask my friends so I actually put all my friends on a zoom call and I made them answer the same questions and I was like wait a second so the the preference like can vary widely from like people to people and it's actually there are several reasons uh for it. So first for the companies they told me that oh we have only adopted IDE based toolings. So that's why like people have not had experience trying out other tools. So yes so they prefer the IDE and when I talked to my friends and they told me that oh it's because my my preference actually evol over time. So um when people work right like I I do there's a book I actually really like uh the title is the principle of least effort. The idea is that as humans when we face with a challenge uh we we try to progress from a solution that requires a less effort and when it doesn't work we try out with something with more effort like that how a lot of people do but some people just slice the suffer and jump straight to the most effort and that's totally fair. Yeah. So so so so the least effort when like let's say you have a coding challenge right so the least effort solution is that you might start with something that extremely that that is extremely automated right like say hey here's a spec build something do it like so you might want to give to like uh um clock code or or codeex or like gemini cli and it work and then it if it doesn't work then you might want to actually then you might actually have to jump into the codebase and look at it and that's why you need to open the the editor and actually look at the code and that's when you use something like cursor or like other stuff. So um people told me that like the more they use AI actually the less time they spend on actually looking at the code and the and so that's where why their preference for ID coding tool actually reduced um so uh so I think it's like when when people talk about AI coding tool one question that's always arise is like how do we measure productivity because like with coding before we have uh traditional metrics like engineuring time spent like I'm not sure about you but like when when we used to make like design documentations or like API doc like okay how much time did this take like this t will take maybe two weeks of engineering time three weeks of engineering times um and maybe some people use the thing of like lines of code to measure like the codebased complexity however this metric don't quite hold up in the area of AI first engineering time is not the same as meal energy and um I want to explain this more because it's a little bit uh more abstract so so for Let me write like let's say a task take uh a day um if I have to spend all my energy or if I have to spend if I pay attention to it for the whole day then I cannot do anything else but let's say the task I take a day but all I have to do is that like I enter the task I give the task to the agent and then I go and do something else and I check back like a day later and see the result. So, so like even though it takes a day, the mental so my extra mental energy I spend on it maybe just like maybe half an hour or like an hour and I'm totally fine with it. So I realize now with like with AI coding agents I'm okay with things taking more time if it require me it require fewer mental energy for me because if I don't have to babysit it I can actually spend time doing other stuff and and I can actually spin up like instead of just spin up uh do like one task at a time I can do like 20 30 or like as however much money I want to spend on it. So actually allow me to do things a lot faster. I can like start a lot of task go to bed wake up and get a result. And another thing to lie of code. Um so AI is really really good at jaring new code. Like whether the code is good or not is is another question. But like AI is extremely good at jaring new code. And before when we had to write code by hand, we we feel very protective of it. Like we said, okay, if we need to fix an error, we we want to reuse as much of the code we had written as possible. But now a lot of people just like try to get rid of it. Like sometimes with exiting codebase it just become too hard to try to fix it. So we just try to like generate like just get rid of the code base and like build a new code base instead of fixing exiting one. Um so okay so I cannot see anyone so I'm not sure how we are doing but I'm going to continue. If I need feedback feel free to reach out on LinkedIn or email or Twitter or whatever. So so another things that um there another kind metrics I look into is when we talk about coding is automations because coding AI toolings is to help us automate our work and I do want to understand like how autonomous AI coding agents can be. So I I looking into this framework. So this is borrowing. So I borrowed this framework from Graham Mubic who is a hands creator and a sim uh professor and he in turn borrows this framework from self-driving car. So we said we got to track the progress of self-driving cars. We have different level automation to see like where the industry is at and how far away from like the the holy grail of like of automations. So number one uh is autoco compilations is is when like AI only suggest like show suggestions for code like auto compilations and and you can accept or not accept it. So it's is we have been doing that for a while. I think it's the first tools I use was like back in like 2017 2018. So so it's not new. It's been around for a while and it's still very strong with like cursor and stuff. auto res day four is that you still have to write code by hand but for very very specific task >> I just want to say that and I'll pop off by >> oh that is interesting I would love to see like it's just a zoom hole I would I'm curious to see the audience here like what what preference do you have okay so so we're talking about level two like partial auto automation so like you can very specific task like writing documentations for certain functions uh then then you can use um then AI can do that for you but it's very limited and the next level is more like conditional automations when yes uh you can do full automations for a wide array of tasks but like still pretty specific first of all like building a new features uh or like a new app from scratch and as the next level is like very high automations so the day four would be AI can do it it's a for very very specific type of task that AI fail like let's say that you have like a legacy software like a lot of Oracle or like I know software and maybe maybe I cannot do it for you or you want to do very low level like cuda like kuda optimizations there's somebody been trying and it's they realize that there not a lot of code like good um cuda code to train the model on so this task is still pretty hard and of course level five is a full automation to do like everything um and I would say that like nowadays most are like between level one and three um and I would be very excited to see like when we reach level four have not seen level four Um so when you talk automations um I was like okay so so what is like higher domations right like is there a way like we can like a threshold or you reach that threshold you can say it's higher donations so what I I also borrow this concept from self-driving car which is like interruption rate um so let's say you have like a Tesla and you turn on phone uh phone self driving mode um and let's say that you turn it on for like a week and you realize it's like okay every 10 minutes I have to like take control of the car because it's doing something dumb. Then it would al constantly be in this like paying attention mode. It has to understand okay how is this going? How is this um you cannot relax like you have to like constantly pay attentions. But let's say you turn on self-driving mode and you can go on it for a month without any you have to jump in to take control of the wheel. Then maybe you can say okay now I have a lot more trust in this car and I can just like let it go. So we can have a very similar concept with like AI coding agents. So let's say you assign a task to the AI uh to write to fix to fix an issue and you say okay it's doing just fine you jump in I can just like completely uh check out the result when it's done then then you have high confidence of it. Um so when I when I was um I tweeted about this and actually use this tool to like measure my own interruption rate with AI coding agent. Um and I I tweeted about it and um actually got quite a lot of support uh like um not not industrywide but like quite a few people messaged me that they are you adopting the um this uh this metrics for example chat is a CEO the CEO of like consu.dev DEP which is like a coding um AI coding tool and he was talking about like intervention rate is as a new build times. So in the early day of software delivery you want to measure like how fast how long it takes you to like build and deploy the software because the shorter the build time like the faster you can iterate and it takes the intervention rates are like very similar like AI automations for coding. So there's several reason why it's important to reduce inversion rate like first uh if you don't have to interrupt AI, if you don't have to um spend mental energy, then you can do more things. Uh I'm not sure about you, but I realized that I'm really really bad at like parallel processing. Like when I was babysitting uh quitting agents, I cannot like keep track of like more than like three at a time. And I have a friend who can do it like five at a time. But like I I haven't seen anyone who can keep track of so many so much contacts like across many uh many task um efficiently. So so I think the highest interruption rate uh is actually limit my ability to like do a lot of things at the same time. Also it's like uh if you reduce rate you can like increase the potential for sub agent. Um so um when I ask people as they familiar with sub agents a lot of people will say yes uh like the the first like code has a concept of task it's like the good agent can spin up a task independently and only returns to use a result when it's done um and I was looking at as a system instructions for like sub agent and task and something that stood out to me is that user cannot interrupt a task in or a sub agent in progress. So like because the main agent spin up the task uh and then it collects the result, users only see the result when it's done. So that means that like if the sub agent is doing something dumb, it can waste a lot of tokens uh and a lot of money without giving any good feedback. So like the the main agent only spin up a task or sub agent if has high confidence that they can complete a task. So like if we have more tasks that agents can just like do like if you have high confidence like that more task that agent have high confidence of like doing without requiring human interactions then it can spin up more task in parallel and of course the last thing is like reduce context loads because every single time you interrupt an agent you get more context to it right maybe the agent has something in mind once you execute and I interrupt it and like okay wait I need to reevaluate it didn't work and blah blah so so so yeah so um so reduce context would allow allows the agent to use memory more efficiently. Um, so I can't ask people here but when I ask uh the the when I talk talk in person, I usually ask people of like what's the what everyone's current interruption rate is and it varies like very very widely from people to people. So I actually built this tool so I can like look at people AI coding um lock and like I see how often they interrupt the thing and so these are just from people who contributed uh to to like dep I I do not I do not know them and I can see that like the task go from anywhere between like I know 0% to like almost like 30%. Uh or like 50%. Um and something I I started asking people and I realized like there are certain areas like impact someone interruption rate. So so the first ones is just user background right like um let's say you have like engineer users and people who come from a nontechnical background. So it's obviously like if people come if someone comes from a nontechnical background they are less likely to interrupt the air coding agent because like let's say they they give agent uh a task and the agent start writing code and they could be like and because they don't know how to write code it could be hard for them to like know whether it's like doing the right thing or not. So they are less likely to like interrupt um the the agent. Um another thing is so engineers are like way way more likely. But then the next part is about senior versus junior engineers. And we usually ask people like who do they think could like interrupt who who do they think could be more inter more likely to interrupt coding agents and a lot of things are like oh senior engineers are more likely but in my very limited data which mean totally biased. Um I find it's like senior engineers actually less likely to interrupt and the reason is that like senior engineers a lot of people who work in software for a long time and are used to writing design docs are used to communicating technical requirements to like other stakeholders. They actually much better at writing prompt uh writing specs for agent to do it. So then they have a pretty good picture in mind of like what they want the AI coding agent to do and because they give very good instructions AI can do the task better and they're less likely to like have to interrupt. Whereas people who are like more juniors uh they usually like they don't know like what kind of risk uh what what kind of like pitfalls what kind of errors is might commit. They might not have a good understanding of like the stack or like the ST requirement. So, so they might just like do it as they go, right? They started putting in some short prompts uh and then things don't work out, they interrupt it and they change it and and stuff. So, they less likely to interrupt. Um and I like another uh behavior that emerged that I felt uh that I find very interesting is that um let's say that you you start with a with an idea of an application you want to build. So, you write out the specs for it. So you give AI to do it and because it's like you're still building and thinking about it, you don't have a very good picture of what it should be. So So you just like figure it out as you go. So like you AI started like building this uh maybe like suggest like an interface like maybe that's not the interface I want. I want this to have like separate pages. This have to be like admins. They have to be like stuff or like you uses okay uh maybe this they are it overeng engineering it. So I want to specify the scale. So the the people like I personally I learn as I go like I I I have a dry run just like put things out there and learn from like s as agent building it and then after that I have a pretty good pictures of the specs I want. So now I have a very fully fleshed and very detailed specs and requirements. Then I would like just spread it of the codebase and give the the new complete spec to coding agents. And this time the agent can just like build from scratch like very very nicely without me having to interrupt or interrupt it a lot few pure. So yeah from the first run when it's still exploring the second run know better what you want to do the interruption rate like I reduce significantly. Uh and of course there is another thing is about like task type. Um so depends the complexity of the task or or like the text stack that AI might do well or not do well. So I think people have found out that I I think it's not a secret anymore um is that um AI usually is much better at like working on new features on a on a new code base like writing new code from scratch instead of trying to work with existing codebase and um and the aer of that is that now there are a lot of people are discussing how to make existing codebase more friendly to AI coding agent. So for example, if you have a code base that extremely poorly structured with a lot of intertwining components and it's not modular at all, you might need to rewrite restructure or refactor it to make it modular to make it a to make it easier for AI to work on a certain small part of the code base. So yeah, there's a whole school of thought and discussions on like how to make existing large code base AI agent friendly and the text. Um so there are different tools. Um so obviously AI will be better at like using popular tools where there a lot of tutorials online and like less good at like using um less good at using um new tools when not documentation but only there are languages. Um I could say this like for example um I usually like the example of um of like three different languages. So let's say JavaScript, Python and Rust. And I ask people of like which language do you think AI would be the best at? Um like the answer always surprising. Um I I think it's like okay I'm going to leave this as an exercise for for video here but I'm curious what what people think. I can say about JavaScript and Python. So AI oh de I saw you jump on screen. Um do I still have time? We need to like run >> that and it we need like 10 15 seconds before people in the chat get the feed and then they write it in. So I'll come back momentarily to give you the results. >> Yeah, no worries. So, so yeah. So, so JavaScript and Python is interesting because personally I I I tested it out and I found out that like JavaScript like I was pretty bad at JavaScript and I talked to a bunch of friends um who also noticed the same thing. So I actually asked other friends who are actually model developers and I asked them like what is going on? Why is why are AI models so bad so much worse at JavaScript than at Python? And they told me that oh it's because there's a lot of bad JavaScript code on the internet. So because like AI models I mean we train a lot of like code on the internet right like if people like the average JavaScript code on the internet is just like way worse than the average Python code on the internet. Uh and the question of like whether it's better Rust than Python script I would I would like leave it to your imagination. Uh but I do have some interesting data point on that as well. >> Uh so so yeah. Huh. >> So >> a lot of people >> a lot of people in the chat are saying Python. They think Python's going to be the best and that's hilarious on JavaScript. All right. I'm out of here. The interruption rate from me is going down. Sorry. [laughter] I'll be out of here. >> Yeah. Next. Can we make this more interactive? I would really love to like interact with the audience. Yes. Um so yeah that's the beauty of like events right okay so another metrics that I also track when I work with a coding agent is how long does it take for a coding agent to complete a task I give it so so when I work with a coding agent it's usually like for for a lot of task I for a lot of things I wanted to do it's not simple anymore it's like require a lot of back and forth for a small one of the project I use like it require like 5,000 back and forth like it's like it's a shittita sorry it's a It's a lot of comment >> and every time I give it I give instructions I have to wait for it to like return the results right so when you give me instruction it start thinking okay how do I complete it so I start coming up with different steps and like the more steps it take to complete a task the longer I have to wait so I started counting like what is the average number of steps the agent does in the background given each comment and usually the the the the graph look at this you can see so it grows over time and the reason is simple is because the codebase gets more complex over time and a lot of task uh a lot of uh a lot of steps it does is just search so let's say as is like hey go fix this chart somewhere it would first like okay where in the codebase is the chart written so it start looking into different fries and then from the fries it's like okay maybe the the function name is this is start searching for it so the more complex the codebase the more search steps have to do And I get annoyed because I'm very impatient and I don't want it to do things. So I don't like this chart of like growing upward and upward. So what I try to do is to make it very flat like I want to do my codebase in a way it's like it does not increase the complexity for each instruction and agent can do more faster. Uh so so here's an example like when I did a task uh an app and is the AI is the first somehow first um intuition like first reaction is to build it in the back end JavaScript and it was just like why would you do this in JavaScript it's like horrible so I asked you to change to Python you can see that the complexity immediately dropped I think it's great because now it can complete things a lot faster um as I try to re refactor the long code files uh because I use the longer the files the harder for AI to parse and understanding. So I make the code very modular. So extremely technical about the the structure of my code base so that AI knows where to find things and where to put things like do not do like duplicate code. Um so yes so so I I hope I have convinced you that interruption rate is like um this something like important. Um so it's important like not just to help us be more productive but also it can be a mirror like for us to understand how how how good we are using AI. So sometime when I feel like I have to you interrupt AI a lot it's not because AI is not good but because I'm not able to give it like clear instruction enough like a lot of time I realize the issue is me because I just fail at specifying what I want and of course AI misunderstand it. So okay so so so yeah so reducation rate is like of uh one one level is from model developers like you can build model with more reasoning codability more thinking logical planning then it can help with a lot more task um another layer is from like tool developers uh so like given a base model they can build better agents for example like have better tool design they can give like the right set of tool maybe grab instead of bash or like some some other like modular tools the tool design is a very challenging Uh so because you want it you want to give agents a tune um a ton set that is large enough that they can do a lot of task but not too large the agent will get confused. Uh so when was talking to like developers um usually they avoid like at this point they avoid giving agent like more than 20 tools because it's very hard for agent to learn to use that many tools and that actually a challenge with like things like MCP because when you on board an MCP server like that server can bring out a lot of tools. It's just like add you just keep it make it so so it's so easy to just keep adding MCB servers and then you get a lot and a lot of like tools that agent has no idea how to use and it can make agents perform worse rather than better. Um and of course there's the last step is like users right like we need to learn how to work agents we need to uh we need to understand like what mistakes it makes uh so we can like structure our workflow and code base in a way that make it not to commit the mistakes anymore. Um so yeah so I think it's like when I look at it um look at the users human AI workflow um I think of it as like three steps the first step is like plant uh so so you give the agent the spec and then you can generate like step-by-step plan and then once you execute it's like you write code for it and then after they have to verify whether the the executions the code generated satisfy the original goal and plan um so yeah users you do need to write spec instructions I don't think is going to go away. Uh so so I I remember someone just talk the set from like just now in demos imagine from like the panel the thing you learn is that like pro engineering is going away and for me is a very strange concept because saying that pro engineering is going away is like saying that communication is going away right like AI is not the only intelligence entity out there in the world like we have like 8 billions people like walking human intelligent out there in the world do you think we stop communicating with each other. I don't think so. I think a lot of the challenge is like being able to communicate to AI what we want to do. And for me that is problem engineering. I'm not sure. Um so so so yeah. So for me I don't think it's going to go away. Um for executions uh I do think it's like people have a lot of people have reached the point when they don't write code and like they let AI do it and they only interrupt when the automation fails. So they actually a lot was just the time they spend on like an ID is very very limited. So they spend mostly on like terminal or like some kind spec uh provide like markdown file reading spec and then they go to GitHub to review the code and they don't write code anymore. Um and for the verification process um it's really depends on like the task. Sometime it's very easy to verify like if you write a simple like if you if you write a simple app you can just look at the app okay it's working just fine. uh but sometimes it's a lot harder. So, so it really depends um on on the kind of task. But I do think that we are reaching the point when um a lot of us are actually reviewing code more than writing code. And where um so so um a company that I worked with recently and they told me that they have been like structuring their uh their team so that like they let like juniors people and AI agent write a lot of PRs and senior people spend a lot of time like writing down architectural design specs and then reviewing code and got upset uh when we had the discussions maybe because we're like wait so what am I am I just the person reviewing Because I'm not sure about you but I have never met a single engineer who enjoys reviewing code. Everyone considers a chore like people like building. No one like reviewing. But uh but if you think about it, if you think of from like a career progressions, uh we do move more toward reviewing as we uh as as as we get um older uh like let's say like when we beginning right we are more junior we do a lot more hands-on execution stuff but if you become a manager then a lot of time you don't actually do things yourself anymore like a lot of times of task is like okay how do I assign tasks to my team and then how do I evaluate the performance of my team and how do you guide them towards doing the right thing so so I do think with AI Asian we we are moving like another layer of attractions like we do more we we let we do more less hands-on stuff and more uh reviewing and like guiding um and verifying so so yeah so I think that's what wait if I delete this one but I just repeated it um oh no I I it's not that I just wrong um thing um okay so so I do think this one is like I do think I'm I'm very bullish on my spec driven development it. So I think the idea is that you just give it very clear instructions having a very good understanding of what we want to build why you want to build it. Um and also like site rules for example like in my rules file I usually get things like what uh what to expect that I want to use for like if I eat something I don't want it to use it. uh I have to make it like forces to like do a lot of like understand documentations like don't just make up like API calls uh and and stuff like that like I make it very clear to scale because if you don't tell it to scale like hey this is just a test project I'm like I'm the early users it might go and go crazy with like package design and like because like handling like thousands of like requests which is unnecessary um I do think it's very important like to do a analysis like looking at the errors that AI agents make and try to reduce these errors. Uh so for example, here's what I um kind of chart I make with the tool. When I look at like the ways um you like look at when you interrupt or look at the arrow messages from the coding agents and it tries to break down like what kind of errors out there that it make and then you try and just like fix like try to reduce it. Um so so yeah um I do think this like with all of that like spec driven development and analysis I do think it's like for problem solving. Um system thinking I'm very bullish on system thinking uh that I do think is one of the most important skills that we need nowaday um um yeah that's pretty much my talk for today. Um thank you so much. Thank you so much everyone. Uh if you have any question feel free to reach out. Yeah, there are a a lot of questions coming through here in the chat and this is how we're going to do it because I promised some of your book to be going to the attendees. We already gave away some AirPods. We're now going to be giving away some straight knowledge. Chip, that was awesome. First of all, I got to say that before we go any further, that was brilliant. you articulated things that I've been feeling and seeing in such a way that it's like ah yes, especially with the like time doesn't equal mental effort piece. It's so well put and I'm probably going to steal that and re try and recquote you later on because it it's really well done. >> Here's what we're going to do. You want >> you want to play along with me? will go and everyone that is now in the chat, there's a Q&A section and you can vote thumbs up for the questions that you like the most. either write your question in right now or go through and look at all the questions because there's a whole ton of them and upvote your favorite one and I'm going to ask five questions of Chip and those five people that I ask your question. Reach out to me so I can send you one of Chip's books. All right, let's do it. I'm going to give them what should we give them? Like 10 seconds, 15 seconds to upvote, read through all the questions. >> You're the expert. [laughter] They're going to speedread all of these questions and then they're going to be able to upvote whichever ones they like. I'm going to start with my favorite first uh because I'm making up the rules. That's what we can do. [laughter] And the next one I'm going to ask is going to be the one with the most up votes. All right. So, and then we'll go sequentially down the row with the most amount of up votes. All right. So, it was this one. Let me find it again. D. How do you prevent over automation from creating brittle ML systems? >> How do I prevent over automation to create brittle? Um so I think what is over automation means right it's like when you're trying to rely on AI to do the tasks that is not able to do So I guess over automation right? So, so I think it does require understanding like what AI can and cannot do and and this is tricky because I first it require you to actually spend time with the AI and then try it out and see like over time it's like okay what is good and what is good but also AI is evolving so fast that like you actually do that constantly right because like maybe I chart um because um before I think one thing that I try to measure is like how how how much how complex how much complexity AI can know with planning. So it looks like okay maybe like in early 2024 I found out that for a lot of cast a cannot reliably sold things that require more than like five or 10 steps right >> but then like just5 a year later I can see they can reliably perform things like I don't know um 14 so I feel like it's growing so fast so AI is impinging very fast uh so so yeah I don't think there's a good way of doing that but like just like with a lot of like learning. Uh I don't think it's a process you can just solve instantly. >> Yeah, I do like that. It goes back to that slide that you shared which showed how complex you can let the agent go and create how big of a system or how many steps and as time went on kind of has gone up and up. It's it seems like it was a little logarithmic like >> it's not going straight up now but who knows maybe that >> you know the new model I can see. >> Yeah, I can see that newer models just like can perform better, right? Like older models like drop at like three or four and then newer models can go up to like what 78 here. So, it's quite interesting. >> Yeah, that that is fascinating. All right, next one we've got this is awesome that somebody asked this. They want to know Chip's opinion. Is MCP overhyped? Um, do you want me to make enemies? I cannot how could I possibly say anything negative by anything publicly? Um, >> it's a lose-lose situation. >> Yeah. Um, I do think that standardization is good. Like anything that make it easier for people to collaborate and reuse things is good. But of course um I think like standard I think like we will never have a standardization that can meet every edge cases. >> So we need to understand the limitations and we have seen this like MCP right users as well. So I actually work with companies they have very very good instruction internal guideline on what type MCP server to adapt right for like they they would prefer MCP created by the tune developer like let's say like they first for example let's say have the MCP for Google calendar from Google versus MCP Google calendar from someone else they could probably prefer the one from from like from the developer because like because the tool like Google calendar would change over time ap you need to trust that the MCP server would would get up to date. You you should trust that the MCP server is not going to try to steal information and do something crazy or something like that. Um yeah. So, so I think like it's really up to the users or like we had an example in the talk about like people just keep on adding MCB server without looking at a set of tools and like the AI agent suddenly have a session maybe 100 different tools with very similar sounding names and has no idea what to do. So you have to be very mindful mindful of like the tools you're giving the agent. So yeah. So, so I think it's like I like standardization, but standardization can never be perfect and also really depends on the way people use it. >> Well, that was a very diplomatic answer. I like it and I'm going to go with it. Well [laughter] said. All right, we've got two more really top like they've been upvoted the most. I'm going to go with Abel's. When using AI to generate production code, what are the most overlooked failure points in data flow design or system architecture that engineers should proactively guard against? >> Oo. Um so, so I think there uh it's like different threat uh there are two different threats here. Uh because like AI could um sharing code for you and also like building the infrastructure. Um so I think it's like a lot of I don't I don't think AI is a point is any can automating building infrastructure for you like it is not going to help you um um because because for a lot of company you don't build infrastructure from scratch you usually have okay I resign the contract with like Google GCB so so now we have to stuck with big queries now we have to stuck with that I'm not it's a bad thing people love bigquery I think a lot of people um go okay I'm not saying that like it's infrastructure not about what is the optimal tool but it's it's usually like a legacy system that we have to work with. Uh so so I don't think AI can automate that. Uh so so yeah so so I think you just need to be able to specify the system you have so you can think of as a constraint for the application. So let's go into the specs like okay here is what we want to do here's what we have and here's what we cannot change >> like help us come up with a plan to do it. Yeah. >> No AI migrations yet. Uh yeah, it's weird thing because when we move with our customers on the first thing we ask them like hey what company is a very big contract with >> because like they don't like I don't know I don't know who they don't change that very often. I think like some company I think uh this one company just like they change providers uh every time they change CTO and the engineers absolutely hate it. They change like back and forth like GCP to AWS and then they change back to like I don't know did I break and was like what is going on? Um yeah. >> Oh that's change so easily. >> So painful. Yeah. Okay last one for you because we're starting to run over time and I'm going to serenate everyone good night as our closing act. Rashab is asking, "Given that senior employees are more experienced prompters, do you see the early career job market decline further in comparison to the rest of the market?" And there's a little bit of an assumption. >> Sorry, I didn't mean to cut you off. There's an assumption there that uh it's declining because I don't I don't actually see it declining. But let's take Rashad's question because I got a lot of hope, folks. as is given the senior employees you remember it all right go for it >> it's actually really really hard problem and people like debating it a lot uh so so I think some companies told me that right like they think it's like okay like if AI can just automate a lot of junior work uh worker uh junior job so we just hire senior engineers and like let them make reviews the code like build design infrastructure for for for AI agent to do and the question is like but if we don't have junior engineers then how Can we have senior engineer down the line? Uh right like if nobody like you come like yeah like in in 20 years how do we have senior engineers if no one is entering the market and and it's a hard questions um and I I don't I don't know the answer for it. Uh but I do have uh I want to bring out this model like internships. So in in college um I had a lot of friends who like got very fancy internships at like fancy companies and yes I also got a fancy internship with like fancy companies and we got paid a lot. When we look at the paycheck, it's like this. No freaking way. We're doing that much stupid. Okay. [laughter] If my own employers like please do not take it like seriously. I appreciate it. But I was saying it's like it's very hard to justify crazy paycheck for people coming in for like a few weeks and like have no context, right? But companies do it because they think of it as a nurturing talent. So it's like okay now they get this internship interns in the pipeline. They train them. They understand the workflow and they have a chance to evaluate how good they are. And then maybe after they graduate when they become a much better engineers they can join this company. So so I do think they would need a pipeline for nurturing junior talents even though if we don't necessarily need them right now for a lot of task. >> Yeah. And I would say I'll go out on a limb and say that a lot of the newer devs or junior engineers are learning how to use these coding tools in different ways and they're coming up learning it and being able to lean on it much more than you would necessarily think. So it is a it's a fascinating one. You always got to learn the fundamentals and the principles and you got to put in your time and get burnt and have those experiences to learn what not to do. But it's cool to see too that there's now new opportunities and new doors opening for the folks who are junior that can use this right out the gates. >> Oh, I love that idea actually. So what you mentioned about like maybe like junior people can actually use they actually level up quickly. So I do think it's like AI makes this very easy to build thing to end up. You can just launch things yourself, test out an app, send it to friends and try to grow it yourself and you can learn a lot about the process and become much more experienced. >> Exactly. Chip, this has been great. Folks who I asked your question to, please hit me up in the uh messages. I'll send you one of Chip's books. Thank you so much for doing this. There was no better way that we could have ended this [music] conference than with your keynote, Chip. I learned a ton as I always do and I really appreciate you coming on. >> Thank you so much for having me. Uh I think this I really love the conference. So yeah, thank you so much for having me again and people if you need [music] any question feel free to reach out. Have a good day.