Designing Machine Learning Systems by Chip Huyen | Chapter 3: Data Engineering Fundamentals
About this lesson
Read the detailed version on: https://onepagecode.substack.com/ Use this url: https://onepagecode.substack.com/p/large-language-models-architectures In Chapter 3 of "Designing Machine Learning Systems" by Chip Huyen, we dive into the foundational concepts of data engineering that every machine learning engineer and data scientist working in production must understand. This chapter covers the full spectrum of data engineering fundamentals required to build reliable ML systems. We start by exploring different sources of data in production environments — from user input and system-generated logs to third-party data. We then examine key data formats (JSON, CSV, Parquet) and compare row-major versus column-major storage, along with text versus binary formats and their impact on performance and storage costs. The chapter also explains different data models including relational, document, and graph models, and when to use each one. We break down transactional versus analytical processing (OLTP vs OLAP), the ETL and ELT paradigms, and the growing trend of decoupling storage from compute. Finally, we explore modern modes of dataflow — passing data through databases, services, and real-time transports like Apache Kafka — and compare batch processing with stream processing for building dynamic ML features. What you’ll learn in this chapter: • Common data sources in production ML systems • Row-major (CSV) vs column-major (Parquet) formats and when to use each • Text vs binary data formats and storage efficiency • Relational, document, and graph data models • OLTP vs OLAP databases and modern lakehouse architectures • ETL vs ELT processes • Passing data between services using databases, APIs, and real-time transports • Batch processing vs stream processing for ML features • Static features vs dynamic (streaming) features This chapter provides the essential data engineering knowledge needed before moving into sampling, labeling, and feature engineering in later chapters.
DeepCamp AI