Aishwarya Shankar

Key Takeaways

Our next speaker is Awaria Sankar who's the founder and CEO of Intelligence AI. And let me just say um it's really funny. First of all, this young lady is local and I've been here 50 years. Monte Vista High School, which is one of the most competitive high schools in the state. In fact, competitive doesn't really do it justice, okay? And I won't go any further than that, but just know it's kind of brutal. Um she then went on to do a dual degree in electrical engineering computer science at Berkeley. Right. Get that right. Yeah. But more importantly, she's not competitive at all. Top 100 tennis player in the in the world. Right. >> In the US. >> In the US. Sorry. Okay. So, not quite the world. And uh felled by injury, she turned to a less competitive space, technology. All right. So, I'm gonna let you start and talk about intelligence a little bit rather than me doing the intro since we're running late. >> Sounds good. >> Okay. >> Hey everyone. Um, thanks for the intro. I'm Awaria, the founder of intelligence.ai where we're taking on the grand challenge of actually building artificial engineering intelligence for engineering teams. So, obviously you heard the previous presenter. A lot of these AI coding tools work really well for individual engineers trying to build projects. Um, and they also work well when you're working in a team. But how can you actually streamline the entire engineering org to ship faster? So in our point of view, we work all the way from the low level of trying to build background agents that can do specific tasks for your engineering org such as improve your observability instrumentation and so forth. And then the next step of actually reviewing your code and then for engineering management being able to actually tell you what's been shipping um and give you kind of detailed engineering performance reports. So that's what we believe engineering intelligence is and how we can actually get large engineering orgs to move the needle faster. Cool. So yeah, as you all know, AI coding is here to stay, but that doesn't mean we've ironed out a lot of the gaps and issues, right? So how to actually deploy code safely and securely, mitigate security vulnerabilities when they are actually created um is kind of still an ongoing challenge for a lot of these organizations, right? And then how to ensure that individual engineers are actually still collaborating together or catching issues and not just five coding things directly into production code bases. So yeah, so these are some of the critical gaps that we uh focus on is performance uh visibility. Writing massive amounts of code no longer is actually an indication that you're a great engineer. Um making sure that you can kind of address security vulnerabilities. review PRs as fast as you write them. And yeah, just kind of address engineering across the entire stack from when you're writing it, making sure it's safe and secure, when you're reviewing it, and then ensuring that your entire team is collaborating, working well together. Um, yeah. So, I wanted to first talk about a bit of the challenges of actually building background agents. So how can you give and give an LLM a prompt and say hey I want you to build this feature and actually get it to execute this well right so as an engineer probably one of the most exciting things that I've been able to do is us and our team is actually figure out how to reverse engineer what engineers do to build in the first place and I think one of the biggest challenges is obviously context right rag misses a lot of details cloud code kind of famously is one of the first ones to dispose of rag entirely Um, and I think one of the main unlocks there is code is a very structured um, language, right? So you know exactly what each file, each function is referencing. Um, so integrating that allows agents to search based off of function definitions, classes, etc. And then now that you have all of this context, so maybe sometimes it's up to 100 files that are relevant to make a change. The next thing that we found really useful was actually being able to rerank these efficiently. Because if you're passing in a 100 files, it actually makes coding agents way worse at being able to make specific changes. And then the last one is actually condensing all of this based on. So we identify like which functions are important, which functions we only need the like function headers and the return types. Um and all of that intelligence actually yeah kind of makes those incremental improvements on these agents. So here's a little bit of a live demo of how it works. Um yeah, some of the use cases we focus a lot on are actually going in and improving for example instrumentation or improving alerts um etc in a large codebase. So here what it's doing is it's going ahead and searching through our codebase. Um coming up with a plan. Um it initializes the entire thing in a sandbox, identifies which files and which things to start looking for. And it will go through and like sequentially make these searches based off of all of the file context that it was able to pull. And then from there it will go ahead and come up with a detailed plan. And we make this interactive so that the engineer can then go in and add additional steps. Right. Um, >> so yeah, I'll pause it here. So, as you can see, what it's done is we've kind of looked through the several hundred files that it needs to do and it identified in each place where to make the appropriate changes, right? And gives you this entire plan step by step and you can go ahead and add things. So, this way engineers kind of collaborate with it um while still being able to make sure that they don't have to go in and review every line, right? And then from there um it then transfers over to the programming agent which will then draft a PR that you can look at and review in the M as you go. >> Yeah. So as you can see here it is actually going in and doing the replace um edit tools to go ahead and make each of these changes one by one. Cool. And then the second part of this is now that you have all these PRs, obviously making PRs production ready is one of the other big challenges, right? Is a lot of people, it's great that you can now ship out like a 100,000 line code file changes, but actually being able to catch these issues um and deploy it safely is I think one of the next big bottlenecks in making like an entire engineering team more efficient. So again, this also comes with its own set of challenges, right? I think the first thing again integrating code-based intelligence makes reviewing code way better. Um a lot of the initial code reviewers just looked at in like that files context but we again enable agents to search based on headers definitions for every language has its own parser. So it's able to look at this cross repo um and so forth. And as you can see it allows you to do this to generate diagrams. Um and then again we've done a lot of analysis. I think the biggest few of the biggest challenges is again being able to find and this is even harder because it's like a few line changes the the bugs can be very minute and it needs to figure out like which other files are going to cause these issues. Um, so yeah, so I the things that we found really helpful is actually being able to search context based off of file function headers, definitions, code context, and identifying which of those um files work best. And then the last one is making the user experience as seamless as possible. Right? So this is within an IDE. Um, no longer have to wait until you push it all the way to GitHub to get the context. um it'll take the changes, tell you which files you've edited, and give you the feedback in real time. And the same thing with security. A lot of large orgs only think about security as kind of an afterthought. But what we've done is we will kind of create PR fixes in real time to address security vulnerabilities and gaps. Cool. And I think this is one of the ones that I think is also most exciting is obviously AI is making engineers 10 times faster, but if you're engineering kind of management and leadership isn't getting faster or more efficient, the entire team is still going to be slow, right? Um so we've been thinking about this in a few different ways. One is can we actually consolidate all the work that engineers are doing and put it together in like a real-time feedback loop so that engineers know the impact that they're creating um their team leads know it without spending like hours every week reporting and saying hey I worked on this I worked on this this is what this person did and a lot of the information gets lost in that translation in meetings so what we do is we will actually go through all of an engineer's work across PRs tickets etc and automatically put together the synthesis and give engineers and everyone on the team feedback. So it cuts out a lot of those extraneous meetings um and kind of just loss of information in the process. And the second one is as mentioned earlier right more lines of code doesn't mean more impact anymore. You can very easily get an AI system to write 100,000 lines of code and it can all be poorly written code or it can cause more issues. It can just be architected poorly. So we've built a separate model that can actually go in and evaluate code for code quality, code security um and give you those insights on a per PR level. And the same thing for um for sprints, right? Is like every week it'll give your entire team insights into what's working and what's not working. So the whole goal with this is to make engineering teams themselves be able to automatically reflect and improve their process so that it's not just individual engineers shipping faster. the entire team can kind of autoimprove and fix itself over time. Um, cool. And then the last thing is being able to see how AI is actually improving or helping your engineering team, right? So we kind of attribute which engineers are using the most AI code, how that relates to kind of shipping metrics and code quality. Uh, cool. Yeah. So that's it. And again, our main goal is to improve the entire engineering life cycle to ensure that larger orgs can actually get the full benefit and value of AI engineering. Thank you so much.