When Agents Learn to Feel: Multi-Modal Affective Computing in Production // Chenyu Zhang

March 3rd, Computer History Museum CODING AGENTS CONFERENCE, come join us while there are still tickets left. https://luma.com/codingagents Thanks to @ProsusGroup for collaborating on the Agents in Production Virtual Conference 2025. Abstract // The next generation of AI agents won’t just respond to what we say—they will sense how we feel. As large language model–powered agents move from research prototypes into production, a critical frontier is the integration of multi-modal affective computing: combining voice, text, facial expressions, and interaction patterns to detect the learner’s or user’s emotional state in real time. This talk explores the challenges and opportunities of deploying emotion-aware AI tutors in production environments. Drawing from ongoing research at MIT Media Lab and Harvard, and from startup experience building GlowingStar, I will share how multi-modal signals—speech tone, facial micro-expressions, response latency, and even silence—can be fused into affective state estimates that meaningfully improve user experience. We will unpack the technical lessons learned from moving affective sensing beyond the lab: designing architectures that combine ensemble LLMs with sensor inputs, diagnosing when modalities conflict or sabotage each other, and establishing guardrails for privacy and consent in sensitive domains like education. In parallel, I will highlight multi-agent orchestration patterns—including critic–rewriter loops and role-based ensembles—that make it possible to personalize instruction, generate equitable feedback, and sustain engagement across diverse learners. By the end of this session, attendees will have a clear picture of what it takes to move multi-modal, affect-sensing agents from demos to durable production systems: the architectures, the pitfalls, and the metrics that matter. More importantly, we will consider how these lessons extend beyond education to any industry where AI agents must not only think, but also feel with