Voice Agent Use Cases

This episode is brought to you by the @mlflowossteam. Check out more information at hyperbolic.ai and MLflow.org. What does it actually take to build voice AI at a billion-interaction scale? This episode features an ex-Amazon voice AI engineer who built customer support systems handling 2 billion+ interactions — now working on next-gen voice agent platforms. Anurag digs deep into the real engineering tradeoffs, design patterns, and use cases that separate production-grade voice agents from demos. Voice Agent Use Cases // MLOps Podcast #374 with Anurag Beniwal, Member of the Technical Staff at ElevenLabs 🎙️ Topics covered: 🔹 Cascaded vs. speech-to-speech — Why cascaded systems still win in production, and how to make them feel natural without sacrificing control 🔹 Latency masking — Foreground/background model architecture and how to buy yourself time while deep retrieval runs 🔹 Constellation of models — Using Haiku for tool calling, fine-tuned smaller models for response generation, and why "one model for everything" breaks at scale 🔹 Turn-taking & ASR challenges — Why voice is harder than chat: accents, noise, silence detection, and domain-specific fine-tuning 🔹 Level 1 vs Level 2 customer support — Why today's agents max out at Level 1 and what it takes to capture Level 2 expert judgment 🔹 Inbound vs. outbound sales agents — Where voice agents are already winning, and why inbound lead qualification beats cold outbound 🔹 Booking, reservations & concierge — The clearest near-term wins for voice agents across hospitality, home services, and SMBs 🔹 Continual learning from natural language feedback — How to build agents that improve from real operator feedback without ML expertise 🔹 Conversational TTS — Why passing full conversation history to your TTS model changes everything for tone consistency 🔹 User tiers for voice platforms — Non-technical business owners vs. developers vs. enterprise: why one interface doesn't fit all If you're building production