Cracking Causal Inference: Why It's Harder Than Regular Machine Learning!

Machine Learning, AI Agents, and Autonomy // MLOps Podcast #282 with Egor Kraev, Head of AI at Wise Plc. Egor elucidates the challenges inherent in estimating causal impacts, which differentiates from regular predictive modeling. Unlike traditional models where outcomes can be directly observed and validated, causal inference requires understanding the effect of specific actions (like sending different types of marketing emails) without the ability to observe all potential outcomes simultaneously. Egor discusses the inadequacy of simple A/B testing in such scenarios and mentions the advanced causal inference models developed by entities like Microsoft, highlighting their diversity and the complexity involved in choosing and tuning the right model for specific applications. The conversation reveals the depth and intricacies of applying ML and AI to real-world problems where causality must be inferred indirectly. // Abstract Demetrios chats with Egor Kraev, principal AI scientist at Wise, about integrating LLMs to enhance ML pipelines and humanize data interactions. Egor discusses his open-source MotleyCrew framework, career journey, and insights into AI's role in fintech, highlighting its potential to streamline operations and transform organizations. // Bio Egor first learned mathematics in the Russian tradition, then continued his studies at ETH Zurich and the University of Maryland. Egor has been doing data science since last century, including economic and human development data analysis for nonprofits in the US, the UK, and Ghana, and 10 years as a quant, solutions architect, and occasional trader at UBS then Deutsche Bank. Following last decade's explosion in AI techniques, Egor became Head of AI at Mosaic Smart Data Ltd, and for the last four years is bringing the power of AI to bear at Wise, in a variety of domains, from fraud detection to trading algorithms and causal inference for A/B testing and marketing. Egor has multiple side projects such as RL for