Designing Machine Learning Systems | Chapter 7: Model Deployment & Prediction Service
About this lesson
Read the detailed version on: https://onepagecode.substack.com/ Use this url: https://onepagecode.substack.com/p/large-language-models-architectures In Chapter 7 of "Designing Machine Learning Systems" by Chip Huyen, we explore one of the most critical and often underestimated parts of the ML lifecycle: model deployment and building prediction services. Get the book on amazon here: https://amzn.to/3T5SBTV This chapter starts by debunking common myths about ML deployment — such as “you only deploy one or two models,” “models don’t degrade over time,” and “most teams don’t need to worry about scale.” We then compare the two primary ways models serve predictions in production: batch prediction (asynchronous) and online prediction (synchronous), including the emerging pattern of streaming prediction that combines batch features with real-time streaming features. We discuss the challenges of maintaining separate batch and streaming pipelines and the industry trend toward unifying them. The chapter also covers model compression techniques — including low-rank factorization, knowledge distillation, pruning, and quantization — which are essential for reducing model size and inference latency, especially when deploying to edge devices. Finally, we explore the trade-offs between running ML on the cloud versus on the edge, how compilers and intermediate representations (IRs) help optimize models for different hardware backends, and the growing potential of running ML models directly in browsers using WebAssembly. What you’ll learn in this chapter: • Common myths and realities of deploying ML models in production • Batch prediction vs online prediction vs streaming prediction • Unifying batch and streaming pipelines • Model compression techniques (quantization, pruning, distillation) • Cloud vs edge deployment trade-offs • How compilers optimize ML models for hardware • Running ML models in browsers with WebAssembly This chapter is essential for anyone moving models fr
DeepCamp AI