AI Agent Debugging: Four Lessons from Shipping Alyx to Production

Building AI systems that actually work in production is harder than it sounds. We built Alyx, Arize's agent for AX, and it broke in ways we didn't expect. This post is about what broke, what surprised us, and the patterns that actually worked -- with enough implementation detail that you can reuse them. Alyx is an AI assistant that helps people use Arize AX, a platform for observing and evaluating AI systems. Users ask questions in natural language: "What's the bottleneck in this trace?" "Which experiment had better accuracy?" "Why did this eval score drop?" Alyx is an LLM-powered agent wi…