Python Tutorial : Running an ingestion pipeline with Singer

Want to learn more? Take the full course at https://learn.datacamp.com/courses/building-data-engineering-pipelines-in-python at your own pace. More than a video, you'll learn hands-on coding & quickly apply skills to your daily work. --- Cool! Now that you know how to create schema messages with Singer, let us continue with the record and state messages, so that we can create a full ingestion pipeline with the Singer library. Here are the user records we saw earlier. To convert one such user into a Singer RECORD message, we’d call the “write_record” function, like this: The “stream_name” would need to match the stream you specified earlier in a schema message. Otherwise, these records are ignored. This would be almost equivalent to nesting the actual record dictionary in another dictionary that has two more keys, being the “type” and the “stream”. That can be done elegantly with the unpacking operator, which are these 2 asterisks here preceding the “fixed_dict”. That unpacks a dictionary in another one, and can be used in function calls as well. Really useful. Now, I did say “almost equivalent”. The truth is that Singer does a few more transformations to make better JSON, but the details are not that important. Simply use the functionality that is offered to you to benefit most. When you would combine the “write_schema” and “write_record” functions, you would have a Python module that prints JSON objects to stdout. If you also have a Singer target that can parse these messages, then you have a full ingestion pipeline. In this example, we used “write_records” instead of “write_record”. It can simply deal with many records compared to the single one of “write_record”. We’re introducing the “target-csv” module, which is available on the Python Package Index. Its goal is to create CSV files from the JSON lines. The CSV file will be made in the same directory where you run this command, unless you configure it otherwise by providing a configuration file. By th