Understand the AI Engineer's Tech Stack

Key Takeaways

Hello everyone and welcome to today's session. My name is Ree and I'll be your moderator for today. We're about to get started. Uh we are a little bit late starting today. However, we've got a couple of minutes before we do actually get going. So, if you haven't done so already, make sure that you register for today's session. You can do so by scanning the QR code that's on screen. that'll take you to all of the future webinars and the today's webinar uh that we've got scheduled for now and the future as well. So yeah, make sure to scan that QR code. You can also head over to datacamp.com/webinars where you'll find this session as well as all of our future sessions as well. I'll put a link to that in the chat now. But yeah, make sure you register for today's session. That means that uh you can get the recording and resources sent to you when we're done just in case you need to pop off for the live session today. Uh with that being said, uh we are also going to be uh sharing some resources that will be uh useful for what we're going to be talking about today. So uh please do keep your eyes peeled in the chat for those. They'll be posted very shortly and they will also be in the video description on YouTube as well. I'll notify everyone in the chat as well. If you have any questions or comments at any point throughout the session today, let us know in the chat. We're going to be running through your questions for the last 10 minutes of the session. So make sure you stick around for those as well. However, if you want to leave a comment and then see if we answer it afterwards and you need to jump off, leave the comment and catch up with the recording. So, yeah, make sure that you register for today's session so you don't miss a thing. Brilliant. I'll run through these messages one more time for everyone else that's just joined and then we'll get going. So, hello. If you've just joined, my name is Ree and I'll be your moderator for today's session. We are just about to get started uh for today. Um, if you haven't done so already, make sure that you sign up and register for this session. The link to do so, uh, is in the comments. You can head over to datacamp.com/webinars. That's where you'll see all of our future sessions, uh, aside from the upcoming campaigns and virtual conferences we've got coming up. However, we will mention those at uh, a later point during today's session. So, yeah, make sure you registered for today's session. If you have any questions or comments, let us know in the chat. We're going to be running through your questions for the last 10 minutes of the session. Uh, and I think that's pretty much everything from me. Please do check out the resources as well. They'll be in the YouTube video description as well as the comments very shortly. Also, uh, the slides uh we are going to be able to uh potentially share with those with you at after the end of the session. So, the only way to get them is to sign up for the recording. Make sure you sign up uh for today's session either by going to datacump.com/webinars or scanning the QR code that's on screen. Brilliant. And with that, I will hand you over to Richie for uh today's session. So, yeah, Richie, please take it away. >> Hi there, data scamps and data champs. Welcome to the webinar. We'll get started in just a few moments time. In the meantime, please do let us know where you're joining from. Uh great to see a few of you saying hi in the chat there. So, uh we've got Ove saying happy to be here. We've got uh Runa Runa is that uh saying hi everyone. Dehane saying hi. Uh great to have you all here. All right, so uh today uh let's kick things off. Um in case you haven't noticed, there are a lot of AI tools. It's crazy. So for any given problem, there are at least half a dozen pieces of software to help you solve it. And that's brilliant, but it's also confusing. So today, we're going to explore the AI engineers tech stack. That means we're going to figure out which combination of tools you need in order to build AI products effectively. Our guest is Alex Kerr. He's an applied AI engineer at Base 10. Welcome, Alex. Great to have you here. >> Hi, that invite. >> Wonderful. So, Alex is an AI engineer. He's an entrepreneur. He's also an investor. Uh, so at the AI inference platform base 10, he helps customers build AI solutions. He's also a venture partner at TA researched capital where he invests in early stage companies and he previously founded the AI project incubator uh P AI where he continues to serve as chairman and pre previously Alex was a decision scientist at launch darkly. He's also been an editor at Stanford High and a machine learning scientist at Neurable. So uh uh quite a career history. Uh I'm going to let you take it away. Tell us all about the uh the AI tech stack. Awesome. Okay. Um, hi everyone. My name is Alex. Uh, currently I'm an engineer at Base 10 and I'm very excited today to talk to you about um how open- source models are secretly powering um a lot of AI coding at scale. And I'll be sharing uh my personal favorite models uh for coding that are nonobvious but highly valuable if you've only considered uh closed source options like anthropic and open AAI and then share some workflows that I use day-to-day um as a developer and and with some remarks on how um I should uh we should sort of adapt as developers um as the ground beneath our sh uh feet are shifting very quickly with um all all the new updates to to the stack. So, um, so here's a little bit about me. Um, I joined B 10 earlier this year at the intersection of go to market and engineering. I split my time between contributing to open source projects and creating content to help developers build better AI applications in production. Um, and uh, Richie went over some of my previous experiences, so I'll skip that. And uh, fun fact is in my free time, I like uh, being obsessed over calisthenic skills and reading as many uh, books uh, in fiction on non-fiction to get closer to my good reads goal. Um and I want to start off to say that um what um we work with a variety of customers today at base 10 and a large number of the most popular AI applications today that you might have used uh is on this list and because of this we have a great front row view of how Agentic systems are built across different domains such as healthcare coding and um sales tools And um today in pace of AI which is coding um it has proliferated over the past uh few years because it has well- definfined examples uh across the web and uh patterns and um ways you can check the correctness of code that is being generated. Um so yeah let's dive in um to go over the agenda briefly. Uh first we'll go over some of the limitations and walls uh builders are facing today with closed source options like GPT5 and Sonnet um 4.6 and why we still continue to use closed source models. Uh I still love jamming with claude on some tasks, but I tell you why it might not be the best option in production. Then we'll contrast this with the rise of uh open source models. I'll touch on three of my favorite state-of-the-art models uh today. Then we'll move into how to take advantage of these open source models uh in your current dev workflow uh from something that is the least handholding and opinionated to some of the more opinionated options and uh more custom options. And lastly, we'll go over a brief case study of how at base 10 at the infra layer in inference, we increase the performance of these coding models to bring users to light. U it's sort of a under the hood sneak peek of the optimization we make uh to making these models run fast. Um that I think you will find very interesting. And lastly, I'll conclude with some takeaways. uh as uh how to think about the world as a uh like AI native builders. So the starting with the problems of open source models, I usually say these are great for prototyping your applications, but the issues really start to emerge at scale. Uh the first is rate limits. So when you're not paying a foundation lab uh millions of dollars, they'll limit the amount of tokens you can push through per minute or per second. Um the second is the costs. Uh there are huge costs that are multiples more than open source somewhere on the order of 5 to 10x when you compare um something like a cloud opus 4.6 versus uh Kimmy or GLM, which I'll dive into later. The third is reliability. Um reliability is concerned as your mercy at these uh closed source labs uh service level agreements. Um if they have outages then you you sort of are also out um if you're building an app that customers are actively using. And lastly, for uh speed and control, um often these um closed source endpoints are throughput optimized, which means they're trying to serve uh millions of developers. So your individual uh token throughput could become an issue if there are particularly uh noisy neighbors around you. And then for control, if you want to build a truly differentiated AI product in this market and want to use techniques like RL or SFT, um closed source have limited offerings and you often um cannot fine-tune slash do not have ownership of these models at the end of the day. So why do we use closed source models to start with? It's because um they're really smart and with high quality.