Patronus AI with Anand Kannappan - Weaviate Podcast #122!

AI agents are getting more complex and harder to debug. How do you know what's happening when your agent makes 20+ function calls? What if you have a Multi-Agent System orchestrating several Agents? Anand Kannappan, co-founder of Patronus AI, reveals how their groundbreaking tool Percival transforms agent debugging and evaluation. Percival can instantly analyze complex agent traces, it pinpoints failures across 60 different modes, and it automatically suggests prompt fixes to improve performance. Anand unpacks several of these common failure modes. This includes the critical challenges of "context explosion" where agents process millions of tokens. He also explains domain adaptation for specific use cases, and the complex challenge of multi-agent orchestration. The paradigm of AI Evals is shifting from static evaluation to dynamic oversight! Also learn how Percival's memory architecture leverages both episodic and semantic knowledge with Weaviate! This conversation explores powerful concepts like process vs. outcome rewards and LLM-as-judge approaches. Anand shares his vision for "agentic supervision" where equally capable AI systems provide oversight for complex agent workflows. Whether you're building AI agents, evaluating LLM systems, or interested in how debugging autonomous systems will evolve, this episode delivers concrete techniques. You'll gain philosophical insights on evaluation and a roadmap for how evaluation must transform to keep pace with increasingly autonomous AI systems. Links: Percival Launch: https://www.patronus.ai/percival Docs: https://docs.patronus.ai/docs/percival/ Paper: https://arxiv.org/abs/2505.08638 Chapters 0:00 Welcome Anand! 1:15 Percival! 17:20 Online and Offline Agent Tracing 20:40 Complex Agent Traces 23:05 Quick Insights and Deep Research 24:47 Automated Agent Tuning 31:19 LLM-as-Judge and Scalable Oversight 42:24 Agent Inbox for Evals 45:49 Causal Inference and AI 51:24 Percival and Weaviate 56:04 Exciting Directions for AI