TNS Agents Livestream: Keith Ballinger, Google Cloud

Key Takeaways

All right, we're back with the New Stack agents, our show about all things AI. And as so often, with me today is Alex Williams, co-founder of the New Stack, and Keith Ballinger, VP and GM for Google Developer Experiences. That's a mouthful. >> And thank you both for being here. >> Yeah, it's my pleasure. >> Good to see you both. >> We get started. Google developer experiences. That seems like there's a lot in that. Can you talk for a moment before we started what it all includes? >> Yeah, the G actually stands for GCP, but um funny enough, it does include a lot of nonGCP pieces, so I should probably rename it. Um but what the GDE or does is uh a bunch of developer tools and services. So there are um you know classic tools and services that you use in GCP like cloud build or cloud deploy. Uh but we also do a bunch of uh you know AI agentic coding um tools as well. uh for instance, Gemini CLI which is you know not a GCP um only product essentially and then I you know like the Go team uh works for me uh the people who run that project uh in the open source world flutter dart um the Google developer program um that everyone should sign up for and um so it's it's you know a lot of Google's you know primary stuff when it comes to to development and of course our big um push these days and a lot of our focus is on you know uh agentic coding you know um how to how to make developers you know have a lot of fun um using AI. >> How do you keep all of that straight and how do you balance this current push for AI with other projects like Go or Flutter or Dart you know programming languages and and other kind of primitives in the programming world. >> I mean it's it's a good question. I think one of the one thing to think about is in the end AI isn't this incredibly separate thing. It it helps with like all parts of building and operating uh code. And so in many ways even when you're investing in something like the go team you're actually helping um agentic coding as well. So for instance um you know the Go team has uh you know a lot of really great static analysis tools you know an LSP there's an MCP built on top of that to make it even easier to do Go coding when you're using um uh Gemini CLI for instance and what's really nice is you know those tools whether they've been built now or in the or have been built you know in the past it turns out that they're just still useful for AI and so you don't have to always think about things as I'm going invest in something that's purely about AI coding or not. It's almost every investment uh can can help both in a lot of ways. >> Now before we move on, I was just thinking about your history too because you were at Zamarind, which some people probably don't remember anymore at this point. It's uh about 10 years ago or so. It was a big open source project. Zamarind, I assume you went over to Microsoft when Zamarind was acquired. >> Yeah. uh you went to GitHub, now at Google. Talk to me a little bit about your career path, how you ended up with this. >> Of course. Um uh you know, my first dev job was um at Intel and I was there for a couple years um in the late 90s and then in 99 I joined Microsoft actually that time and I spent uh eight years at Microsoft and about half that time I spent uh working on net and C#. So, Andrew Hosberg, the um the creator of C#, you know, he was in the office next to mine. Um and so I' I've always been in dev tools ever since then. And when I left Microsoft, I just started doing a lot of startups. You know, I really had the startup bug. Those startups ranged wildly. Um you know, because I wanted to get lots of experience doing lots of things, but the ones that were focused on um coding and developers were the ones that I enjoyed the most. So for instance, I was the first engineer, first employee of a startup called Standard Treasury, which was building APIs for banks. um you know targeting developers uh that was bought by Silicon Valley Bank and you know I did Zamron where it's really funny actually uh Nat Freiedman who's the CEO of Zam was the CEO of Zamarind you know uh he asked me to kind of help him you know he hired me as a consultant one day to help him uh find a VP of product and you know I kept trying you know grabbing candidates and bringing them to him and he kept saying no and I was like on the side starting my next startup and then I was unable to do that startup because my co-founder got sick and I was complaining to Nad about it and he's like why don't you just be my VP of product and that was a little surprising to me because I've you know I've been an IC product manager at times and I've been mostly an IC engineer um with a little bit of engineering management and so I said sure why not you know and then I read a lot of books about how to be an executive and a manager um yeah and then you know Zamron was bought by Microsoft and at Microsoft you know I ended up um you know being the general manager manager of uh developer services which was things like Azure DevOps um and the internal um one engineering system they called it and then um you know Nat and I got the the bug ear to try to get GitHub bought by Microsoft and we wrote like a one pager that was mostly bullet points for Satcha and um it it went incredibly smoothly I have to say um it was it was amazing how all the pieces lined up so easily and you know then then we moved over there and became the CEO and I became the SVP of engineering. >> I I >> it's a good fit because I'm actually I was at the GitHub Universe conference. That's why I'm here in San Francisco in a hotel room as I so often am for these recordings. And it's actually it was funny. I was walking through the kind of the startup hall there and there was a lot of startups and companies in the ecosystem at GitHub and there was Google jewels as well which you know is you know not quite part of your orc I think but >> no jewels is a it's a lab project so Google labs um tries to do cutting edge you know services and apps um you know and uh you know being part of research and whatnot they kind of get to do cutting you know like I said cutting edge stuff uh riskier ideas and then traditionally when labs comes up with something that's really good, um the tech or the you know the thing itself or whatnot will transition into a product team uh at some point in time and so Jules was like an example of hey what if we built something that was like tied to Google uh sorry to GitHub especially GitHub issues and could do those you know semi-autonomously and it's it you know it's been very useful to learn how to apply that kind of methodology and that that tech to you know our other products um that you know we you know that we have free versions of but also that we sell to people including enterprises. >> Yeah. Yeah. And Jules now becoming with a couple of other projects like OpenAI's codeex and anthropic code other ones you know. >> Yeah. I think a little bit Jules differently than that. I think you know codeex has a CLI and a web a web version of the CLI and cloud code has the same thing. Jules, I think, is like taking a different approach where they're much more um you know, they have a much more specific workflow um around how you work with um GitHub and GitHub issues and whatnot. Uh you know, the Gemini CLI is is decoupled from Jules. It's not the there are different projects, but you know, the Gemini CLI will obviously also have like a a web interface to it like codeex web and whatnot. Those are more general, right? They just kind of do anything. >> I have a quick question. Alex, jump in there. >> Yeah, I just have a quick I >> I'm curious on, you know, with all your background as a developer and in engineering, >> how did you come across AI and how did you approach it? Because you were saying earlier, you know, >> Keith, >> sorry, >> it's Alex muted. >> Am I muted? >> I can hear him. >> Oh, I cannot hear you, Alex. Very weird. Okay, great. It's a live stream. Everything works perfectly. I'll just wait for I'll just wait for your answer, Alex. >> Sure. I I was just asking about uh you know how Keith you really started to you know unfold AI so to speak. So how you started to really use it. Um because you were saying before before our recording like a lot of people just don't know how to use it and they're using it kind of in a very singleshot kind of way. >> Yeah. >> What's what have you learned and like you know about that know and tell us about those early days when you were first getting started. >> Yeah. Um, and it's interesting, you know, I while I've always been involved in like developer tools and and technologies, you know, as a teenager, I actually would um I was like an email pin pal of Marvin Mvin Minsky. Um, and so I, you know, I I had the AI bug very early in my life and you know, like I put together a hack team once um like 2012 or something for Baze Hack, which did like, you know, machine learning for, um, things uh, for, you know, for nonprofits. So I've always been really interested in that world and then at GitHub we started building Cop-ilot and the idea there was you know OpenAI had come along and and finally had a pretty good model um that funny enough they called Codeex at the time and it could it could do coding tasks um pretty well if if you managed it carefully and so the very first thing we did actually was purely code completion you know there was um there was a chat soon uh and interestingly I think the code completion um starting with that starting with something very interactive was very useful in understanding how people would use AI and kind of getting them comfortable a little bit with it and then you know of course then people started uh using the chat and you know sometime some people go like a chat GPT or a Gemini even today to do AI coding and then of course like agents came along right where all of a sudden the model was doing things on your behalf with tool usage and that was like a really interesting you know um growth in our industry And uh you know I kind of followed along on it and I remember like maybe mid last year um all of a sudden tool using models started really taking off and so you know you things you know people like cursor all of a sudden were able to make a dent because they weren't just code completion and they weren't just chat all of a sudden they were like I said like doing these kind of semi-autonomous tasks for you with tools and it was funny because up until then for like a year prior to that I would just use chat. So I would go to Gemini, the you know what we called Bard at the time and I would tell it what I wanted to do and then it would come back and like give me code and I would take that code and I would paste it you know into my um into my project but it would also tell me to do other things like you know um run a test and give it the the output of the test and I was basically acting as the tool for Gemini for the model itself and then once that was automated uh all of a sudden you could do things so much more quickly and it really expanded on what you could do. Now, it's interesting, one of the things you talk about is um how people use it, right? And there I think one of the things I'm seeing in the industry right now is there's a big variance in people understanding what the best practices are and even maybe what the best practices, you know, should be. And and that's something I think that we have to kind of do more education to help people understand like how to use AI for assistance, you know, for for coding because if you just single shot stuff, you're not going to really be successful. You have to treat the AI, you know, kind of like a co-orker. Um, and you if you follow best practices from development in general, things like writing a you like a a user guide or a spec, things like writing a technical design, creating a project plan, just those three things. If you do that when you interact with AI, which you should do anyways on anything complex, you actually have really great results with AI. And I can uh I can show you an example of that actually. I have a little open source project that does that if you want to see. Go for it. >> Okay, cool. I will share my screen. Um, let's see here. So, I um I just want to pull up the right thing here. So, this is my this is my desktop. Tell me if you see something called line counter there. >> Yes. >> Cool. And you can see right now I just have one file called conductor MD. And if you go to my GitHub, you can find a repository called conductor. This is a file that I use to set up a project. And the the way that works is I'll just cap conductor so you can kind of see what's in there. Um it specifies um to the AI how to set up the project. There's like a plan file, style guide for pros, style guide for code, a workflow, um a status file, the prompt that you can use to launch the AI at different times, user guide, architecture, and then with with more details. So what I can Sorry, go ahead. >> So just to go in there, I saw a similar I mean this is your project, which is kind of interesting that you're coding yourself obviously, but >> yeah, and code is really just markdown files for what it's worth, but yes. >> Yeah. That's but that's what I was thinking about is that it's still all very verbose and we see similar projects right where we agents.mmd files >> do you think that is that kind of a just a step right now in a different direction where we don't have to do that anymore or >> kind of the future of that >> I hope not I don't want to I think I like creating you know I like being a craftsman um and creating things and I want to be involved with that I don't necessarily necessarily want to write every line of code um and debug every, you know, test failure or whatnot, but I I like the act of creation. And for me, the act of creation is, you know, um thinking of a problem I want to solve, you know, and then thinking about how I want to solve it. And I think those things, you know, AI could probably do that as well, but you still have the problem of how do I as the AI um or how do I as a human give you enough of like what I want without the AI, you know, you have to do that somehow. It can't read your mind, at least not yet. >> And if it doesn't read your mind, you know, then you can't get that. And I I want to be involved in that part. Um so, let me show you. I'm going to go back to that um uh my terminal here. I'll just show you how this one works because it's it's it's manual to your point. Um there are MCPs that do this kind of thing. I like doing it manually because I find it to be more enjoyable. Plus, I can really control exactly how things happen. So, I can um and I had this prompt ready to go. You know, I'm like a tool smart line counter counts lines per language prints out a spark line or histogram. And then I just say use the process outlined in conductor MD to set everything up. So, it's going to read that file and uh as you'll see in a second, it's going to set up a folder called doconductor and it's going to put into that doc conductor folder um you know say yes always uh it's going to put like all these documents that really will help me and the um the model work really well together like that architecture MD and the style guides and whatnot. And this means I can be involved in the parts of creation that I'm really interested in um without necessarily, you know, having to do everything. And so, you know, we can see it's um now it's creating these. And one of the interesting things about this project and again it's just a just a project. I think the the idea is what matters is um you know how you want to control the agentic experience. Okay. So now it it needs to write the user guide. I could just write a user guide if I wanted to. Um, but I actually have it ask me questions and you know I could I won't go through all of this right now, but after I answer all those questions, it then has a very specific um task plan and architecture doc and like user guide. And then I can iteratively I can come back anytime I want. I can do a session. I can do a couple tasks with it where all I really need to say is what's the next task and it'll continue on with that. But these are very detailed which means hey we can um we can achieve a lot of success and you know I have um using this methodology whether it's with conductor or just like similar tools I've been able to build I think very interesting and complex things um so for instance I if you don't mind I'll show you um one of these and I I got really interested in um languages I've always mentioned in languages but I had the idea and I so I just wanted to programming languages. >> Yeah, programming sorry, programming languages. So I I got really the idea of like could we build a programming language that um was for LLMs and one of the problems LMS have is programming languages aren't you know they're ambiguous at times and things like that. That's where a lot of the code generation tasks fall apart. So I I created what is basically a really ugly uh language to humans. Uh it's a lisp. um but it's like incredibly um specific and doesn't have much uh ambiguity at all for for the LLM. And I built this using that methodology I I was talking about. And it's like a Rust project, right? So here you can see this is all the Rust code. It's using Rust and the LVM to compile essentially a systems language um that is very, you know, it's very fast and it contains a lot of capabilities. And I did all of this just with that methodology, creating an architecture, you know, creating a plan and going through it. Um, anyway, someone wants to check out this, I call it aether. It's on, uh, GitHub under the Google Cloud um, uh, organization. Um, and I have no idea if it actually makes LMS better. Um, you I haven't done that part, but what's interesting is this is a a complete language and and it works pretty well. All of those examples work. All of the tests pass. There's thousands of tests. Um, and you know, I built it probably in a month. Um, just some nights and weekends. And that's the interesting thing is if you spend a little bit of time up front, you can just really, um, you can, you know, slow down to speed up as they say. You can just get so much done. So how do you then make sure that it's working correctly and that you know often when you put in a you put in a prompt and then the the output that you get from the prompt and then you apply it maybe to your dev environment it doesn't really work very well. >> So how are you verifying everything? >> The way I do that is I use the AI to help me with that. So I'm not I don't look at every line of code. Um, and with something like aether, I don't really know Rust all that well. So, isn't like I would have, you know, done a good job of that anyway. I know LLVM, okayish, but I created, you know, that architecture doc, I did collaborate along a lot with it. And so, and then the the task plan was very minute. And after each task, I I basically had it do like a test-driven um it was it was like TDD. Um, it would write tests and then it would build implement the feature until the test passed. And I did spend time looking at that in terms of uh Oh, we lost Frederick. Um, but I'll keep going. >> Yeah. So, you know, like I would look a lot at the tests, but I also spent probably most of my time looking at the examples. I had to create a bunch of examples covering all kinds of scenarios. file IO, HTTP, you like a blog, you know, um, you know, simple, you know, memory management things and those examples I I made sure always ran. So after any task that was done, I would make sure like the examples ran, all the tests passed, and I would make sure the AI was doing those things, too. And so in some sense, it's a black box to me. All I really care about is in the end, did it work? Uh, but at times I'd go casually inspect things as well. As you can see, it's, you know, perfectly fine rest code. >> So, it's interesting because you've discovered some new experiences through all this, haven't you? Um, >> and this came but, you know, a lot of development comes from, you know, problems people have. You know, I'm doing a little research on on Jupiter projects on Jupiter right now, Jupyter notebooks. And you know, just like a lot of things, it was like it came from a need, right? You know, the files were everywhere. They couldn't keep track of things. They couldn't keep them updated. You know, I'm curious about the experiences that have been troublesome for you in traditional development and now how you're seeing them transform your own thinking about software development overall. >> Yeah, it's a it's a good question. I haven't thought about that actually. Um, so it's it's a novel question to me. I do agree with you though in general. A lot of the best software that's been created in the world came from some engineers personal pain points, right? Um, you know, Linu Trovalds created Git because he needed better source control for Linux. Uh, I think it's a classic example of that and it just had so many uses outside of outside of Linux, right? And it changed how people in our industry worked in a pretty fundamental way, especially with like once you combine it with the GitHubs pull requests. And I you know I think about like the kinds of problems I have are you know I generally speaking I don't code a lot to solve problems I code a lot to have fun I guess is comes down to that you know if you think about business issues >> um because of like my position leading all these teams I do want to use tools a lot to understand them and make sure the quality is high. So part of this is just dog fooding. >> I like to code a lot to make sure we're building high quality products. Um, and then when I come across something like a best practice, then you know I'll hand it to the team. Um, but at times I have ideas that I think could be applicable. Um, for instance, uh, I wrote a I'm still writing an MCP server. I call it DevSense. I should probably call it something else that essentially like looks at Git history. um and then does um the same kind of analysis that the book uh your code as a crime scene does where you know you can do analysis of like the tip of your source code just like what it looks like now but the much more interesting analysis oftentimes is how the code has changed over time and so that was something I thought was very interesting and so I you know like I said I started building an MCP for it but yeah a lot of a lot of stuff I do is just for fun like I built a I built a comic book um gener generator using nano banana nano banana um because like I was like oh this seems fun. So like you know it lets you create reference images and then you give it a comic book script and it goes and creates the panels and and all that stuff. And so that was like you know that's the kind of thing um and I was inspired by that by my fiance who like made me a comic book uh for my birthday. That was pretty cool. Um and it it just got me thinking like oh you know how can we make this even better? um agentically. >> So inspiration comes from everywhere. >> As you're doing that, you know, you clearly, you know, you're really thinking about, you know, not just oneshotting and other best practices, but where have you struggled so far with using these agents? What what hasn't worked for you? >> I mean, I think it goes back to actually the CLI. um before the CLI existed and or other tools like it, you had like I was saying like I the human had to be the tool. I would be told what to do and then I like I would read the file and I would paste it into the chat or whatever. But when we decided to make the CLI all of a sudden we had um we solved that problem and we made it like it wasn't as tedious or laborious anymore. Um, and that was like I think one of the drivers for Gemini CLI or other tools like that. >> Mhm. Why do you think this CLI has become so popular again lately? You know, we've we've got pretty much all of these agents has some kind of CLI version at this point. Why? Why is that? >> Yeah. Um, I think honestly we never abandoned this the terminal. I think it's part of it. We didn't think about the terminal as a primary interface. More and more I think people uh you know you would go to the terminal to run things like npm um or just other commands like ls or whatever >> but we didn't think about it as like our primary interface more and more you know there there used to be like a rich ecosystem of like TUIs like terminal user interfaces like midnight commander right um and those are >> old enough to remember. Yeah. >> Yeah. Right. And it's interesting like those are we we just like kind of stopped thinking about those and just had you know command line script but we always used the terminal. Um and especially in modern development you know espec you know where there's so much OSS like you know people use the terminal forget and so it was it was really natural I think to use to the terminal as the way to do these things. It gives you a lot of control. it focuses on the job to be done versus lots of UI, you know, um, and so it kind of, you know, focuses you in on what you need to do. And it, you know, it just kind of follows along with what you're already doing or what you have been doing. >> The CLI is an interesting topic now in in, you know, in the world of AI, as Frederick is noting. I'm curious about how you see the terminal itself evolving. I mean, you know, I mean, the Gemini CLI brings AI directly into it, doesn't it? >> Yeah. Yeah. I think the it's a really interesting question. Um, will the terminal evolve uh to be more AI um capable itself? Uh andor will there just be more and more AI capable or influenced um command, you know, tools within it? Uh like a year ago, I I built a prototype of um KubeCuddle. Um, I built an extension for it called coupubecuddle AI >> and that was that was actually a they team rewrote it. My code was awful, but it it now it's not part of cube cuddle core and you can give like a natural language query and then cube cuddle can do things with it. Um, like how what nodes are failing right now or whatever and so that's one place we could see things happening is more and more tools do that. The other thing you could see we could see is terminals becoming more AI and agentic themselves. U warp is an example of that. And then also, you know, the the most common shells today, you know, absent the terminal um are bash and zsh. And one of the things I've been thinking a lot about and I I did a prototype a while back and my team's done another prototype which is even better, which is like could we create like an AIA shell and you know each of these are ways of interacting, right? Just like you could have a a desktop app that you know understands things and you interact with or a web page, right? Like there's likely all of these things will happen is my guess. Um because they all have, you know, interesting use cases. >> How do you think about that kind of handoff between the CLI and the the IDE and maybe some other surfaces where I want to use these? Yeah, >> the last time we talked we talked um about the integration between Gemini CLI and Zet if I remember. >> Yeah. Yeah, exactly. >> Yeah. >> Uh and then let me actually you know what let me share my screen again and I'll show you this is like a thing uh I'm seeing this more and more across all the tools. Um but it's really interesting to see. So let's see here. Um if I'm in VS Code like I said works this works with Zed too. I can just start the CLI and in the terminal that exists here in um uh in VS code, right? And of course this, you know, is working with more and more IDEs all the time. And then I can say things like um what what does this file do? Like it knows what files I have open. >> And so I have mod.rs open. It knows that, right? It's going to read that and then send it. and you know tell me what it is. And so you don't have to like think of these as I'm you know even flipping from one thing to another. You can actually use these deeply in concert with each other. And I think that's like we'll probably see that being the workflow a lot of developers prefer. >> Makes sense. I'm I'm curious again about programming languages and are are you seeing you know you know if we're talking about the evolution of the of the terminal right there's the evolution of lots of other things that are associated with the terminal >> um and you know with generative AI you can like you you said earlier you don't really need to know Rust very well right you can you can you can use the machine to help you put in you know to to generate the right code. But is is there a shift in like how we're thinking about code itself and you know and and then with that you know how do you see the evolution of new programming languages? I this is a fascinating question because I I think there's a wide variety of opinions on this and no one knows for sure and I I I have generally I think a very optimistic opinion that's probably in the minority. Uh but Acer was of course an example of that. I think we should be building programming languages um that are designed for LLMs where we never look at them um so to speak. Like I think we can get to a place where as a developer I'm no longer dealing with lines of code. I'm dealing with the system design, the architecture, the you know the user interfaces, all those kinds of things. I'm crafting it together. Um but the actual language underneath is just a black box to me. And this to me feels like a natural progression of abstractions. You know, we originally started with punch cards, right? You know, essentially machine code. Um and then we had you know languages like C which were you know thinly um thin thinly on top of things like assembler and then you know we went to uh more and more abstract languages like Java right or C which you know continued up the abstraction level and I think the next abstraction level is essentially my intent what I want um and how that you know composes together um to make to make an experience to make an application and in those cases I don't think you know um you'll necessarily need to have the code. Now I always think there'll be people whose craft is the code who love writing code you know line by line and honestly they'll probably have a a competitive advantage for certain avenues certain areas of code writing you know for a long time and you know and that's if that's what they enjoy you know I want them to enjoy it but I think we're going to get to a place where we don't see that you know as much um and again controversial opinion I'm sure but you know there's like there was a paper I read not too long ago about Someone put two agents in communication with each other and over time they started like sending gibberish to each other because they negotiated like a a highly compact language that they invented on the fly to talk and you know that kind of thing is is fascinating to me. I'd love to see where that goes. Is that abstraction layer by default natural language though or is it kind of a meta programming language because you want some degree of precision, right? Which natural language isn't, you know, at least English isn't always good at. >> Yeah. I mean, it's a this is a question I ultimately don't know the answer to, but >> that's why I asked it. >> Yeah. I used to be >> uh I used to be kind of anti-natural language. I felt like there were going to be other metaphors for interacting with AI that was more precise and than natural language. But, you know, as I as I've worked in this field longer and longer, I kind of feel like, you know, when you look at a a textbook um or any like a non-fiction novel, we don't worry about the precision of what's been written. You know, no one ever says, "Well, you know, that um you know, that that textbook on calculus was imprecise." It was we feel like it was pretty precise. Um, and of course there's bad writing that's vague, but it kind of I think there's just so many examples of people in natural language writing precision um, uh, and doc um, language and I think that that because you have to kind of meet people where they are, right? And people could learn some other syntax, right? um you know something that is formal that allows them to talk to an AI or something that's visual that allows them to communicate with an AI. >> But then we're asking people to do learn something new and do something new. And I think we just have more and more evidence that just have really good writing skills and have really good um have have the ability to decompose problems well and understand designs, right? Like design patterns have existed for a long time now and they've always turned out to be incredibly useful and not just in coding, right? Like architects use design patterns. I think that's kind of where we want to go is let's just help people be better and better um writers, right? And better and better problem solvers, which again it's like about decomposition of the problem. >> Yeah. A different kind of skill though from sometimes the skill that programmers have today. I would argue that's not true. No, I don't think that's true. I think that >> a a great programmer is a great writer. You know, they they write down what they're going to do. They document the the problem, the solution, options for the solutions, you know, the design, all of that stuff. I think great programmers do that and then uh you know, and they certainly understand they can they're able to go from business problem to sol technical solution really well. you know, understanding what requirements are, understanding what's possible, decomposing it into small pieces so that they can kind of iterate and pivot around as they need to. I think those are the exact skills needed in this agentic coding world. Um, and maybe or maybe not, the ability to write lines of code is the one that is is not as um needed in the future. So I just would like to you know just ask a question about what are you seeing as the tradeoffs for using like for instance systems that depend on inference. You know, because when you're depending on a system that uses inference, it there's some limitations and you know, you're basically relying on machines to form conclusions. How how do you you know, how do you address that? >> Um, I guess could you could you elaborate a little bit more? Um, I think I'm not quite getting you. Well, I mean, you know, AI can essentially be a, you know, black box, right? You know, we can't trace how an AI, you know, reached its, you know, its its imprance. >> And so, you know, when you're thinking about that and kind of in your work, how are you approaching that, you know, really that, you know, that known inherent weakness that comes with using AI tools because they're following patterns. you know, they're not doing reasoning. >> Um, you're trying now agents are trying to do some reasoning in itself, but now you've got to set up a whole system to make sure the reasoning is accurate. >> Gotcha. Yeah. Um, I think there's a couple thoughts I have about this. One is that in terms of the blackbox piece, right, we don't know what's going on inside of them. There there is in fact a lot of research going on that's essentially kind of like an MRI on an LLM to like show you the the circuits so to speak of what's being activated during token generation. So we are gaining more and more insight into kind of like how they work if if that makes sense. But the other thing is like if I'm working with another programmer I don't really care how their brain works right like I don't need to understand the you know the neurochemistry of their brain. um what I need to do is be able to communicate with them effectively and clearly and for them to be able to do what they're doing. And if I kind of humanize AI, well, hey, I just need to know I can communicate effectively and clearly with them. They can do what they're doing. Um now, there is of course, you know, you're kind of hitting on it. We need ways to kind of evaluate and uh understand how these systems are doing so that as uh they're in production and doing things like are they doing the right thing or not? Are they going off the rails? And you know there's a bunch of techniques that we're creating in the world. I think we have a long ways to go here because it's far beyond of course just like you know HTTP errors. It's like you know hallucinations and whatnot. I will say that I feel like a lot of these are happening less and less. I think the the the in teams working on inference the teams working on the models have done a pretty good job at creating more reliability and but you know we do need to create new techniques you know both like during let's call it CI/CD or AB testing rollouts but also just generally in production >> for how we like you know sample um the work and make sure that it's just you said not going off the rails. Let's bring it down maybe back to the the CLI itself and I just want to talk a little bit more about Gemini CLI too and kind of the state of things there like what what's new there and to some degree like does that feel like a solved problem right now like Gemini CLI you've got the CLI talks to Gemini is that kind of where we are now and that's the end game. >> Yeah, it does not feel like the end game. I think we're in I think we're in the first inning or maybe we're just warming up and the game hasn't started yet. Okay. >> Um there is certainly the basic thing that we need works. I can talk to Gemini and it will respond to me in a conversation using tools and it's good enough about reasoning around what tools it needs and so on and so forth. But there's so much more we need to do around that. You know, models are going to get smarter and smarter. I think some people believe that the models would get so smart that we can always have like a thin wrapper um you know the agent code so to speak around it. Uh I think actually we'll always have more and more interesting things to put there in the wrapper and some things will graduate into the model and I think they come in a you know a few different categories. One category is just UX, right? Um whether you're in a CLI or some other agentic um app, right, like inside of an IDE, how you do it, right, is uh is interesting. And like how you tell it things, how you approve tools or how you spawn a sub agent, you know, those are all like are interesting problems that we can iterate on and make better. Do we even need chat always? Is chat the right way to do these things? I think we kind of fell into chat and it works okay, but are there other UX paradigms that we could be looking at? Um, and then I think there's just like the the next piece is how to tackle bigger problems. And there it's like there's sub agents, right? There's things like conductor, how we decompose problems. Um, I think there's a lot of work we can do there. And then there's things like um monitoring my runtime system and gathering that data back to me, right? which is very much like a data processing issue. Uh you know I I love to use Gemini CLI to write my code um check it in run CI/CD and then deploy it right to cloud run or something and then have it across that look at the data that's coming back to tell me things. So for instance, I will tell my I'll tell Gemini CLI use GH the the GitHub command line tool to monitor the the checks you know the actions runs on some pull request that I worked on with it until it passes and when it fails automatically fix it you know and ask me if it needs help until the the pull request fit um works and then the next step after that is okay let's say it's in cloud run download um you know using the G-Cloud tool uh download the the the logs, right? And even send up requests and see if the right things are happening and if not, uh, make those changes. And I think all of this is like workflow issues that we can still invest a lot in. And certainly with Gemini CLI, we're those are areas that we're doing a lot with, right? We're doing a lot with like, you know, um, UX, we're doing a lot with workflow, we're doing a lot with optimizations, make it faster, you know, all of those kinds of things. >> Yeah. How do you think about the relationship between Gemini CLI and kind of the background agents we're seeing a lot of at this point? You know, those two working together because to some degree in the CLI I'm still in that mode of watching the >> Gemini do its work. um you know and we'll release soon um and we have I think in a branch the you know sub agents right where you kick those off but already it's very easy to use the CLI as um a background agent and you know you can I I often will have the CLI spawn another instance of the CLI and then monitor it as it as it does something and this is like I think a common pattern and of course we're making that better from a user interface perspective and I think that will be more and more how it works and I think that's a place where we can probably leave it to the GM CLI to figure out how much does it need to do with these things right >> so so how are you monitoring then the sub agent then >> um the every CLI tool out there is is um building this kind of infrastructure where they can you know kind of have some command that gives you then um lets you look at what's going on like the chat transcript and all those kinds of things the systems I've built before those existed and I kind of still use because I like them is I give a prompt to the other instance um well in my prompt to the other instance when I describe what I want to do. I also tell it to update a file with its status and then the one that spawned it I have given it a prompt that basically says look at this status file every now and then or you know I have like a custom slash command that says look at this you know status and whatnot and a lot of these things like I said are there are product features that are you know coming and already exists a lot of the time for these you know this is actually what's really interesting is Gemini CLI is an open source project right and you know not all of our competitor CLI are. But I think it's been really nice because then we get ideas like this. I've seen quite a few really cool, you know, sub agent um, you know, PRs and discussions come along for people that, you know, again, you could, you know, run it in a branch, which is great if you're, you know, in an oss project or that becomes that feeds into what we, you know, think is the the final kind of official version. Um, and there's so many different ways to do this. So that's why you know kind of a superpower being open source because you get to iterate on all these ideas >> and a large part of Gemini CLI is probably now written by Gemini CLI to some degree. >> Sure. You know it was it was pretty early on in its development where we were able to bootstrap and and start doing that. Um certainly I you know I was um I was coding on it uh pretty early on and most of the code I wrote for it was using it. Um you know there's parts that you know we didn't um either because you know it wasn't as capable yet or you know we just really had like fine grain details that we need to to figure out. Um and there's some developer preference but as you can imagine essentially the whole team is using the CLI to build the CLI on a very regular basis. So does the CLI then better understand the code context too? >> Yeah. Um the CLI I mean you can generally you know people manually give the CLI code context that it cares about right you can just at some file like uh if I'm working on something I might say look you know look at this file and then I use the at symbol to kind of um intellense to the exact file name and you know summarize it for me and then write a unit test and at that point once it reads that file it's in the context and Gemini has this huge context window so it's kind of always there um but also So you can just tell it to do something and it will determine it'll like it'll look at readmes or overview files. It'll do ls you know on different directories and it'll decide which files to read and uh and then build up its own context. So what's interesting is as an industry we need to do a lot more around context management. Uh I don't think models are incredibly good at their own context management and I think that's something that for quite a while we'll need to like continue to iterate on tools because the bigger your context gets the kind of the the more things can go off the rails and so we can give the user you know developers who are using Gemini CLI the ability to craft the context exactly what's in there for any particular task and that's where sub agents are actually super useful is you can like give them exactly what they need for that very specific task. Um, but I think that's like a a set of innovations that we're going to see a lot of to, you know, more user control over exactly what's in the context. >> Now, you know, does this bring up issues around state and state management and how you think about state and state management and then how you know you then why you might need better context. So you know I mean because this can get into some pretty challenging technical situations for instance with you know you know thinking about the overall infrastructure and how it's managing state across you know a distributed environment. Yeah, that's a great question because when you're building a project, right, you have the whole SDLC and you have to manage like you said the state of that pipeline, right? all before production and that's where I think you know I like to for instance have a status markdown file and I think you know there are MCPs that you know store it in different ways but this again is another place where you know I might have a set of repos a large team of developers all using AI and we need a way to coordinate them all together so that those agents know what they need to know and maybe there's even master agents who kind of understand everything and I think we're kind of early in the industry um figuring out what that should look like. But we are seeing rapid progress. I I think you know my team's done a lot of good work uh around for instance um AI generated but also AI consumed documentation which really helps a lot but to your point it's static here's how things look right now there's the need to uh you know there's a need to like understand what your deployment topology is right there's a need to understand you know what the load is on your on your service and so all of those things I believe we need to continue to iterate on going back again to why Gemini CLI is in the early days along with all of our competitors. This is a a place we really have to iterate on. So it's a it's a really good insight. >> Thank you. Um, yeah, it it's it's curious to me because then that gets into all kinds of questions about, you know, the processing that you do and and, you know, of all those agents that are, you know, might each have their own state and, you know, and then that gets into the questions about, you know, what kind of infrastructure do you really need to do this? And it seems that's where like a infrastructure provider like you know Google um you know is really almost you know required for this kind of thing because you who who else is going to have the processing power to be able to manage agents in such a form and fashion. I >> I think you're right. I mean obviously I'm at Google so you know maybe I'm biased but it's one of the reasons I came to Google was because I thought Google had everything from the TPUs on up to to you know do an amazing job here and really >> but it's also the cloud service providers really. Yeah, other others for sure. But you are right like we have in in GCP we have something called um cloud assist and there's an investigations feature where you can literally go to that and you know from say the you know when you're looking at logs and say what's this error or hey my users are reporting like a bunch of 500 errors or something and it will use AI to uh gather a bunch of context about the runtime environment to help you diagnose the issue and it'll look at both like you know uh environmental things that are just with GCP in general but a lot about your environment where you have code running and this is exactly what you're talking about like you need a cloud provider to pull off that kind of state and uh but when you have it the agents are so effective right I it's it's amazing to me how you know every time I want to troubleshoot an issue in at runtime uh just how easy it is um when you have you know using Gemini and yeah using investigations and the way it gathers the state and then it just It always knows what the problem was. It's actually it might be like >> the you know I think we're sleeping on it. It might be the case the use case that works better than anything else right now. >> Wow. >> Interesting. Well, let me um switch gears a little bit as we're coming up on time here. I know you've been really interesting. He talked about it as we were chatting before the show here about using these models for science and math and kind of those kinds of problems as well where some people would say maybe they're not even that good at it them yet but you know getting better obviously but where does that interest come from for you and kind of >> uh I mean I'm a >> yeah I'm a big math and science nerd and I'm always looking at um new ways to do things. So, one of the things I've been spending a lot of time on in the last couple months is using Gemini to um do like math and science like paper writing and research and formal verifications and things. And I'm by no means an expert, but you know, there's lots of scientists here who can um can help me. Uh in fact, I can show you an example. Um I'll share my screen. Let's see here if I have it open. Um, oh yeah. So, here's an example of a paper that uh I wrote with Gemini and uh I use something called co-author which I'll talk about in a second. But it basically you know it it does a lot of latte management and it does like code to do modeling um and to make figures and things like that. But then you know you >> tech maybe for those who who are not in scientific publishing. >> Exactly. Yeah. Exactly. But then in the end I have a paper here, right? Like that's >> crazy error correction effects with black holes and you know it's uh this was vi not vibe coded but this was iteratively built with um with Gemini and uh with experiments and like I said formal verification and uh the thing I the thing I built to do that uh I'll share that with you is called co-author and let me see if I can find it real fast. Um, here we go. So, co-author. Um, and I'll close this. It's basically a set of prompts here and, uh, playbooks for how to do things along with handbooks on like how to use latte and how to use lean. Um, along with a user guide and, you know, code style guides very similar to conductor. And then I actually have a script here. um you know to install it and then once you install co-author there's a you call co-author and it goes into it calls the Gemini CLI to actually um control it. So instead of me just like being in Gemini CLI and trying to do things, I I run the co-author command and it then figures out a bunch of context that it feeds into Gemini CLI when it starts it and then I can resume things and all that kind of stuff. Um and I think that's like a really interesting way to go in general. I think with science and math there's been a lot of work on how can we be autonomous? How can we create like a autonomous scientist? uh which is I think interesting but you know in in the coding world we didn't start there and there's a lot of similarities between science math and coding in the coding world we started with like I said code completion then chat and then chat with tools you know and then semi-autonomous and and you know starting to see more autonomous agents science and math needs kind of go through the same thing where we incrementally um build up on it but it's very clear um both from the work I've done but others that Gemini CLI is a very useful tool with this kind of thing because it's interactive and you need that interactivity um before you move to the completely autonomous you know oneshot idea. >> Have you seen a paper yet that was accepted after peer review based on co-author? Have you submitted a paper yet for >> No, I I'm not I I I uh I don't feel like I'm a scientist uh you know to do this kind of thing. So what I'm interested in is the process, right? not so much like I'm not going to invent new science or math. >> There's gonna there's going to be an agent symposium someday where all where all the attendees are agents and all the presenters are agents and they're representing their their papers that they've generated. I mean >> I would not be surprised. I would not be surprised at all. there is like um I do know that various journals have varying levels now of AI input into the creation of the paper and generally you have to like in the acknowledgements um note how you used AI to build it and you know I think there's some journals want you know very little like they're they're explicit that hey AI if you used AI to like you know touch up your writing and whatnot it's fine other ones are more broad and I think we'll see more and more broad ones I think, you know, relatively like that that paper I showed you, you know, it was it was very much like the AI did everything. I I directed it along the way just like I did with with Ether, the programming language, but um you know, it was very interactive, you know, at the same time. So I assume that physicists and mathematicians who I I said I'm working with them now and they're um you know starting to use these tools they'll be able to accelerate their work versus just hand it off because I really do think that maybe someday they'll be independent autonomous agents doing science but why not make scientists and you know researchers and mathematicians more effective now right like that would that would be great >> waiting for the moment where we're standing around at an AI poster sessions with our, you know, beers and the robots will explain their papers to us. >> I can't wait. I think I I can't wait. I think, you know, there's just so many cool things that can go on with >> and then they'll drink their motor oil. >> Well, it's what the the robot folks always tell you is the robots keep breaking down and that's why they can't make progress as fast as they want to. >> Absolutely. you know, it'll bother us once entertainment is heavily AI dependent and then it's breaking because we really like our entertainment, right? >> Yeah. >> You know, you're gonna see more and more of that. You like like the comic book creator, right? Like, you know, it was like kind of hard for my fiance to to put that together. And uh then now it's a lot easier. And um but you know, we'll get dependent on these things, not just for comic books, but for you know, video content, right? And you know, audio content and all kinds of things. And the more independent we get on them, the more reliable the systems better be. >> Absolutely. Now, we've got um we're pretty much up on time here. We've got one question in the audience. If you have time for it, we'll just take it. >> I got time. >> You see it down there in the out of the podcast later, but you know, the impossible computing that you've talked about at some point in the in the past. Um >> that's a good question. it is and it's from Joshua so maybe you can address that you know >> overall how much of an impact do you think full vibe coding will have over the next year um it you know when I say impossible coding by the way I mean it kind of aspirationally um what I'm you know I like to say that I like to use the term because I want people to to not think anything's impossible right that we can do anything with with code essentially now the question is how much of an impact do I think full vibe coding will have over the next year I think really the question is what's meant by full vibe coding But I do think there's a world uh where knowledge workers and consumers vibe code epheral apps, you know, um a party website for a single birthday with, you know, people rvping or whatever and they're just a pheromal and they go away. Um you know, and but on the knowledge worker side especially, I could see people vibe coding, say, you know, um business apps for themselves, right? like using Gemini Enterprise, a marketer creates um an app to help them do copywriting. Uh and I think that, you know, we'll see a lot of those. You know, Mackenzie said there's something like a 100x software demand for supply. So there's that's a lot more programmers or a lot more efficiency we need. So it' be great to have people those knowledge workers for those marketers be able to build those things. And then we need to make sure that you know there's a handoff, right? If uh the marketer builds some cool app, you know, at some point it and the internal engineering team will need to take it over and can we use AI to make that better from step one, you know, like capture the intent that went into what was built. um all those kinds of things because we saw this right with like Excel spreadsheets in Access databases right like in the 90s you know people without much technical skills or none would build all kinds of sophisticated tools and then those tools would become some line of business app but oftentimes they just had to start over and maybe we're in a world where we don't need to start over every time >> or maybe we're in a world where you want to start over every time because you just create that app without >> bothering you know to even save it and just build it every time. >> That's true. That's true. Like, you know, your desktop OS in the future might just dynamically create the user interface for everything you're doing on the fly all the time. You know, actually, who knows, right? That could easily be how it goes. >> But I do think we're entering a really weird world that's exciting on this kind of thing. >> For sure. Um, one thing I was thinking just as a final question, I was looking at your old blog from 10 years ago and you, you know, you talk about impossible computing there, but you always had reading recommendations back then. So, I was wondering, >> what are you reading right now? Is there something in science, math that you're reading? Any recommendations for >> Let me uh I use Kindle. I'm going to look at the look at, you know, because there's a few things and so I just, you know, I never remember the A Shards of Earth by Adrienne Takovski is a novel I'm reading right now. I'm about halfway through. It's It's been pretty good. I enjoyed it quite a bit. Um >> fan. >> Yeah. You know, I haven't read a lot. >> I never got into his stuff. >> I haven't read a lot. >> Never clicked for me. So, >> I think this is the first novel of his I ran I read. Um I I recently read Seven and a Half Lessons about the brain. Um that was that was really good. Uh and let's see, what else have I read recently? Oh, um you know, Scott Meyer's work is always really good. Um, I think a lot of people liked The Fourth Age. Um, I think that was really great. Um, I read or reread um, last year John Rolls is a Theory of Justice. That was really interesting. Um, you know, I think, you know, I I have a pretty broad reading. I would say that the most amazing book I read in the last two years is um, Tomorrow and Tomorrow and Tomorrow. >> Oh, that's great. Yeah, I love that. I got it as a gift in fact for my staff I think last Christmas. Um it is I I think if you're a creator, you know, if you're someone who creates things >> Yeah. >> Uh you know, for your job or whatever and you work with people to do it, it is the most beautiful novel I've ever read that captures that the spirit of it, you know, >> and and it's just a you know, you laugh, you cry. It's a just a such a good story. >> Yeah. Yeah. That one I like I I recommend that to everyone right now. I think it's the best book I've read probably in 10 years at least. >> I would say so myself. Yeah. >> All right. I've got something to do on my 10-hour flight to Berlin for next week. So, I'll just take that book with me. >> Yeah. You'll love it. You'll love it, Frederick. It just does it captures the spirit of being on a team and creating so well, you know, >> and relationships, you know, and the between people and, you know, and what they mean and how they transform and >> just so cool. >> Yeah, it's a stunning book. >> On that note, I think we're out of time for today. We're well over our time, I think, but it's a really good conversation. >> Yeah. Thanks, Keith. Keith, >> of course. I' I've thoroughly enjoyed it. Um yeah, thanks for having me on and um for everyone listening in or watch this later, um you know, always feel free to to ping me if you have thoughts, feedback, ideas. Um Keith Ballinger.com. Um I'd love to hear from everyone. >> Awesome. Thank you, Keith. Alex, thank you as always. >> Thank you. >> We'll be back next Will we be back next week? I'm not No, I'll be traveling next week. We'll have a rerun or something pre-recorded next week, I think. So, >> thank you guys. >> Thank you.