Finetuning: 500m AI agents in production with 2 engineers — Mustafa Ali & Kyle Corbitt

Method Financial has run AI financial agents in production for over 18 months, handling hundreds of thousands of interactions daily. In this talk we'll review some of the "battle scars" we gained while deploying and scaling this system. We'll discuss specific techniques we've found critical to keeping this system operational and effective including fine tuning, realtime guardrails and "sanity checks", observability platforms, and intelligent caching. Recorded live at the Agent Engineering Session Day from the AI Engineer Summit 2025 in New York. Learn more at https://ai.engineer and purchase tickets to our next event, the AI Engineer World's Fair, in SF June 3 - 5 here: https://ti.to/software-3/ai-engineer-worlds-fair-2025 About Mustafa Mustafa is a founding & senior engineer at Method Financial, a financial platform that makes it easy for fintechs, banks, and lenders to interface with consumer debt. At Method, Mustafa leads their efforts around building robust and scalable real-time data integration systems with top financial institutions. Prior to Method, Mustafa worked on product-facing and on-chain applications for the Topl protocol. About Kyle Kyle Corbitt is the co-founder and CEO of OpenPipe, the easiest way to train fine-tuned models and deploy them to production. OpenPipe has fine-tuned thousands of customer models, and serves tens of millions of inference requests every day. Before founding OpenPipe, Kyle led the Startup School team at Y Combinator, which was responsible for the product and content that YC produces for early-stage companies. Prior to that he worked in engineering at Google and studied ML at school.