Python Tutorial : Components of a data platform

Want to learn more? Take the full course at https://learn.datacamp.com/courses/building-data-engineering-pipelines-in-python at your own pace. More than a video, you'll learn hands-on coding & quickly apply skills to your daily work. --- Hi! I’m Oliver Willekens, a data engineer and instructor in this field at Data Minded. In companies today, people are trying to extract value from the tons of data they’re gathering. They’re doing this in an environment called “the data platform”, which is the start of our journey to create robust data pipelines. While working through this course, you will learn * how to ingest data into the data platform using the very modular Singer specification, * the common data cleaning operations, * simple transformations using PySpark, * how and why to test your code, * and how to get your Spark code automatically deployed on a cluster. These are the skills you will be able to apply in a wide variety of situations. And because of that, it’s important that you standardize the approach. You’ll see how we do this. Note that there is a lot to be said about each of these topics, too much to fit into one DataCamp course. This course is only an introduction to data engineering pipelines. Many modern organizations are becoming aware of just how valuable the data that they collected is. Internally, the data is becoming more and more “democratized”: It is being made accessible to almost anyone within the company, so that new insights can be generated. Also on the public-facing side, companies are making more and more data available to people, in the form of e.g. public APIs. The genesis of the data is with the operational systems, such as streaming data collected from various Internet of Things devices or websession data from Google Analytics or some sales platform. This data has to be stored somewhere, so that it can be processed at later times. Nowadays, the scale of the data and velocity at which it flows has lead to the rise of what we call