How to deal with late data | Event vs Processing Time | Stream Processing

Code with Irtiza · Beginner ·🔄 Data Engineering ·3y ago

Skills: Streaming Data53%

About this lesson

What is Event Time? What is Processing Time? How can you process “late data”? 🥹 If you found this helpful, follow me online here: ✍️ Blog https://irtizahafiz.medium.com 👨‍💻 Website https://irtizahafiz.com 📲 Instagram https://www.instagram.com/irtiza.hafiz/ 00:00 Event Time vs Processing Time 03:55 Which one to use for computation? 05:30 How to deal with late data? 07:40 Watermarks #streamProcessing #flink #beam #kafka

Original Description

What is Event Time? What is Processing Time? How can you process “late data”? 🥹 If you found this helpful, follow me online here: ✍️ Blog https://irtizahafiz.medium.com 👨‍💻 Website https://irtizahafiz.com 📲 Instagram https://www.instagram.com/irtiza.hafiz/ 00:00 Event Time vs Processing Time 03:55 Which one to use for computation? 05:30 How to deal with late data? 07:40 Watermarks #streamProcessing #flink #beam #kafka

Watch on YouTube ↗ (saves to browser)

Sign in to unlock AI tutor explanation · ⚡30

More on: Streaming Data

View skill →

Stream Real-Time Geo

Stream Real-Time Geo

Build Streaming Data Pipelines on Google Cloud

Build Streaming Data Pipelines on Google Cloud

Creating a Streaming Data Pipeline With Apache Kafka

Streaming Big Data with Spark Streaming, Scala, and Spark 3!

Streaming Big Data with Spark Streaming, Scala, and Spark 3!

Handle Late or Duplicated Data and Archive Events for On-Demand Replay | 5/5

Handle Late or Duplicated Data and Archive Events for On-Demand Replay | 5/5

Building Resilient Streaming Analytics Systems on GCP IT

Related Reads

What Can We Do When Memory Becomes the New Bottleneck in Data Engineering?

Learn how to overcome memory bottlenecks in data engineering using Pandas chunking, Dask, and Polars, and why it matters for processing large datasets

Towards Data Science

Migrate from Ponder to Envio HyperIndex

Learn to migrate your indexer from Ponder to Envio HyperIndex to scale your data management

Dev.to · Envio

Data Backfilling with Apache Airflow: Architectures and Implementations for Historical Data Processing

Learn how to implement data backfilling with Apache Airflow for historical data processing and improve your data pipeline's accuracy and reliability

Dev.to · Wangila russell

Building a Production-Style Weather Analytics Pipeline from Scratch: ETL, ELT, Star Schema, and…

Learn to build a production-ready weather analytics pipeline from scratch using Python, DuckDB, and Apache tools, and understand the importance of ETL, ELT, and Star Schema in data engineering

Medium · Python

Chapters (4)

Event Time vs Processing Time

3:55 Which one to use for computation?

5:30 How to deal with late data?

7:40 Watermarks

A Moment Frozen in Time | Arnav Iyengar | TEDxJenks Youth