Manage Schema Evolution in Real‑Time Data

External: Coursera Courses ↗ · Coursera

Open Course on External: Coursera

Free to audit · Opens on External: Coursera

Manage Schema Evolution in Real‑Time Data

Coursera · Intermediate ·🔄 Data Engineering ·3mo ago

Key Takeaways

Teaches the management of schema evolution in real-time data using compatibility policies and automation

Original Description

Ship data and schema changes without outages. This hands-on course teaches you how to treat schemas as contracts, evolve them safely, and keep producers, consumers, and warehouses green end-to-end. You’ll design compatibility policies in a Schema Registry (backward/forward/full, transitive), automate checks in CI, and practice expand → adapt → contract rollouts. In streaming labs, you’ll capture OLTP changes with Debezium, deliver Avro-encoded events to Kafka, and route malformed records to a DLQ with actionable alerts. On the analytics side, you’ll evolve BigQuery/Iceberg schemas additively (NULLABLE/defaulted columns), shield downstream users with views/contracts, and validate correctness with queries and time travel. Realistic scenarios walk you through enum expansions, type widening, null/tombstone semantics, and subject naming rules. This course is for data engineers, backend engineers, and analytics engineers who work with real-time or streaming data systems and need to evolve schemas without downtime. It’s also useful for platform engineers and architects responsible for data contracts, CDC pipelines, or Kafka-based platforms. Learners should have basic SQL knowledge and a general understanding of streaming systems such as Kafka, along with familiarity with Git and the command line. Experience with schemas, CDC, Docker, or cloud data warehouses is helpful but not required. By the end, you’ll have runnable templates, governance checklists, and a portfolio-ready project that proves you can design zero-downtime change—confidently and repeatably. For more information, check out the document.
Watch on External: Coursera ↗ (saves to browser)
Sign in to unlock AI tutor explanation · ⚡30

Related Reads

📰
I Built My Second ETL Pipeline. This Time, I Started Thinking Like a Data Engineer
Learn how to build a production-ready ETL pipeline with Python, Docker, PostgreSQL, and Kestra by thinking like a data engineer
Towards Data Science
📰
JuiceFS Sync for PB-Scale Data Transfers: Resumable Sync, Encryption, and Bandwidth Control
Learn how to efficiently transfer large volumes of data using JuiceFS Sync, which offers resumable sync, encryption, and bandwidth control, ideal for PB-scale data transfers.
Dev.to AI
📰
How Airflow is using AI to make data engineering more resilient, not more complex
Airflow uses AI to make data engineering more resilient by detecting data drift, resuming failed pipelines, and fixing issues automatically, reducing complexity and improving reliability.
Medium · AI
📰
What Can We Do When Memory Becomes the New Bottleneck in Data Engineering?
Learn how to overcome memory bottlenecks in data engineering using Pandas chunking, Dask, and Polars, and why it matters for processing large datasets
Towards Data Science
Up next
A Moment Frozen in Time | Arnav Iyengar | TEDxJenks Youth
TEDx Talks
Watch →