What is Apache Airflow?

Key Takeaways

My goal for this video is to explain you Apache Airflow in a most simple and intuitive language. Apache Airflow is a widely used tool in the world of data engineering. And if you go to any job portal, you will find tons of job. On their website, they say that it's a platform that will allow you to programmatically author, schedule, and monitor workflows. But, what does that really mean? Let's go through a simple example. Let's assume you're working in a finance company as a data engineer, and you are given this task to extract the stocks information using some API or let's say you're doing web scraping and extracting information such as ticker, price, book value, etc. Then, you are doing transformation, where you are calculating a new column price to book, which is price to book ratio. You are removing invalid records, right? For PB ratio, this doesn't have book value, so you probably want to remove it. You're doing some currency conversion. Reliance is a Indian stock, rest of it dollar, so you're doing rupee to dollar conversion and so on. And eventually, you want to load that data in Snowflake. Snowflake contains huge volume of data. So, daily you are loading these prices, and you are doing this computation nightly. Okay, your manager said that every day at night, we need to do this computation. So, what you can do is you can write simple Python code, okay? Extract, transform, and load functions, and then you can call them in sequence. It's a simple file called etl.py, which will perform these three tasks in a sequence. And then, using Unix cron job, you can set up a nightly schedule. You can say, "Okay, daily at 2:00 a.m. in night, please run this file." Now, this approach is straightforward, it works good, but there are a couple of issues. First, let's say there is an exception in extract, okay? Let's say the website or the API from where you are extracting the stocks data is down for a few minutes, only for a few minutes it is down. And when you're running this at 2:00 a.m., this will fail. And let's say this is throwing exception. So, then remaining things will also fail, right? So, there is no inbuilt retry logic or failure handling. Now, if you're a smart programmer, you'll argue that, "So what? I can code up my retry logic. I can have a for loop, I can have three retries, and if there is an exception, I will wait for 10 seconds and I will retry again." And it will work. Let's say if the temporarily this API or let's say website is down, then after waiting for 10 seconds, it will uh come up. This works fine, but this approach is very manual, repetitive. And you must add this to every function where there is a chance of failure. So, it's not very scalable. Second problem is there is lack of visibility and monitoring. What if this script fails? In that case, you have to set up all the monitoring and logging on your own using some external tools. You have to go to the Unix machine where this is running, and you have to, you know, examine all these logs. I used to do this a lot at Bloomberg. Uh so, this is not a great approach in terms of visibility and monitoring. So, at Bloomberg, we used to have all these external monitoring tools uh to capture the failures and success, etc. Another thing you want out of this is what if you can represent this uh particular data pipeline in form of a DAG? So, DAG means directed acyclic graph. Directed means the data is going into one direction, right? So, there is a direction to it. Acyclic means there is no cycle. It's not like, "Okay, you are doing transform, load, and again you're going back to transform." There's no cycle to it. And graph means, if you know computer science fundamentals, this uh structure is called graph, okay? So, what if you can represent this workflow, this pipeline as a directed acyclic graph with retry and all other features, okay? This thing you can't get in your plain Python script. So, there is no dependency management. See, here I'm saying that transform depends on extract. And by the way, this is a very simple graph. You can actually have a complex graph. You can have even more complex graph than this, okay? You have multiple steps, some branching logic, some things you are processing in parallel. Now, if you do this in Python, of course you can do this in Python, but it is not very intuitive, it is complex, and you can't get this kind of visualization as well. So, Airflow tries to solve all these problems. In Airflow, you will import DAG. So, you'll say, "From Airflow import DAG." And then, with DAG, you will give a name to your pipeline. So, this is called a DAG ID. Then, you will have start time, okay? Start date actually. From this date, this pipeline will run every day at 2:00 a.m., right? So, this is a cron syntax. So, schedule is set. And then, you will specify your three tasks, okay? So, first task is extract function. So, extract function, you will put it in this Python operator. You call it task T1. Then, transform function. You call it task T2. Then, load function. You call it task T3. And you simply run this pipeline. You will say T1, T2, T3. It will execute these three things in a sequence. Now, the beauty of this kind of programming is it's very declarative, right? Here, you can say, "Okay, retries is three." And it will automatically do retry. Tomorrow, you want to change retries to five, let's say. And it will do that. You want to specify retry delay? You can specify it. It's like you are having some kind of configuration file here, and you are just changing bunch of parameters. So, this way the possibility of introducing bugs will reduce. And also, the code is readable, very much extensible. Once this code is written, you can visualize this DAG in Airflow UI, okay? So, this is the Airflow user interface. And if you go to this DAGs, you will be able to see your pipeline. My pipeline ID is etl_pipeline. So, see here I am able to see it. etl_pipeline is the DAG ID. And you can see the schedule here, next run, latest run. See, you all See all this information here. And then, uh you can also trigger it manually. By the way, when you go to this, see you will be able to see all the runs, right? Now, you can go back, and you can also trigger it manually. See, if you want to trigger Usually in production, you don't trigger it manually, but just in case I'm just showing you. Uh it will be usually run on that nightly schedule, okay? Now, let's say at night 2:00 a.m. is running. See, this is running now. So, when you click on it, you will be able to see the status of your process in a nice UI. See, now it is saying success. See, all these three steps, you are able to see how much time it took, like 8 seconds here. You're able to see what are the sub steps, uh how much time each of the sub steps took, okay? If there were any failures, if there were any retries, you'll be able to see all the information. You can also see the logs. See, you're you are seeing logs for extract, you can see the logs for transform, load. It makes debugging so much easier, it makes tracking of your pipeline runs so much easy. You can also click on this particular icon and visualize your DAG. So, just to summarize the differences between your plain Python approach versus Airflow. See, Python script with cron job is very easy to set up. Airflow will require some setup of Docker and all that. But, once that is set up, you reap a lot of benefits. First one is you get nice web UI for your monitoring and observability. You also get uh retries and alerts built in in a declarative way. Dependencies are visualized through DAG graph. Then, it is scalable. Apache Airflow is scalable because it uses distributed computing, and then there are distributed workers, okay? Whereas here, if you want to do distributed computing, you have to do multi-threading and all of that on your own. Then, there is extensibility because Apache Airflow has many integrations. See, we saw that Python operator, right? Similarly, let's say you want to do something with AWS or ADLS, uh it has a lot of integration in terms of those operators. And it is best suited for production-grade multi-step pipelines, okay? Uh and in the industry, usually your pipelines will be complex, uh and for that, Apache Airflow is more suitable. That's it, folks. I hope you have some understanding of Airflow now. [Music] Okay.