Designing Machine Learning Systems | Chapter 9: Continual Learning & Test in Production

About this lesson

Read the detailed version on: https://onepagecode.substack.com/ Use this url: https://onepagecode.substack.com/p/large-language-models-architectures In Chapter 9 of "Designing Machine Learning Systems" by Chip Huyen, we explore how to keep ML models accurate and reliable after deployment through continual learning and safe testing in production. Get the book on amazon here: https://amzn.to/3T5SBTV This chapter starts by addressing one of the most common questions in the industry: “How often should I update my models?” We discuss the value of data freshness, how to quantify the performance gains from retraining more frequently, and the trade-offs between model iteration (changing architecture or features) and data iteration (refreshing the same model with new data). We then introduce the concept of “Test in Production,” explaining why offline evaluation alone is not enough and how to safely evaluate models in live environments. The chapter covers multiple production testing techniques: • Shadow deployment – running the new model in parallel without serving its predictions • A/B testing – randomized controlled experiments to compare model variants • Canary release – gradually rolling out a new model to a small subset of users • Interleaving experiments – exposing users to recommendations from multiple models at once • Bandits (including contextual bandits) – a more data-efficient alternative to A/B testing that balances exploration and exploitation We also discuss the importance of building automated, standardized evaluation pipelines so that model quality can be consistently assessed rather than relying on ad-hoc testing by individual data scientists. What you’ll learn in this chapter: • How to measure the value of data freshness for your models • Model iteration vs data iteration trade-offs • Shadow deployment, A/B testing, and canary releases • Interleaving experiments for recommender systems • Multi-armed bandits and contextual bandits for efficient model

Original Description