Kafka Simply Explained

Key Takeaways

In this lecture, we will understand what is Kafka in a super intuitive language. We all use Uber Eats, Zomemetto, etc. for food delivery. Have you ever wondered how things work behind the scene? Let's say you are a back-end engineer working at Zomemetto. Now, somebody is placing an order of paneeta masala through their app. The order will go to the back end. Let's call it order processing service. It will take a payment and it will insert a record in a database for your order. But does it stop there? No. The system is much more complex than what you think. It will uh send an order to a given restaurant. Let's say you order through this gulati spice market. It needs to send the order to that restaurant. And let's assume that restaurant has some kind of API or HTTP call. So it will make a call. It will also uh call delivery service because it needs to schedule the driver etc. so that they can deliver the food once it is prepared at the restaurant. It will also make another call to analytic service, another call to notification service so that you can get an SMS on your phone regarding your order. All of these are microservices. Okay. So assume that these are microservices and this is the back end for your app which is calling all these microservices in a code which may look something like this. So this is a Python code and it is making bunch of HTTP calls to all these services. Now watch this. What it is doing is it is making a call to restaurant service and it is waiting for the response code. Then it's making a call to a delivery service and waiting for a response. So this is a blocking call. It is a synchronous processing. Okay. So this is called tightly coupled synchronous architecture. Now let's say Zomemetto is in its early days. The programmers created this kind of architecture. It is working fine. But all of a sudden now Zomemetto has become more popular. There are more and more users using their service. And there is a world cup final match and everybody every single user of Zomemetto is now ordering food through their app and their load has increased. Now they are getting millions of request from those users. Okay. And this architecture which was tightly coupled synchronous architecture was not designed by keeping in mind this kind of a huge workload. Okay. So what happens is basically since this is a blocking call remember that R2 has to wait for R1. What will happen is similar to a situation at a traffic junction. You might have seen this kind of traffic situation where there is a huge traffic coming in and let's say there is outgoing traffic to these three different roads and there is some kind of congestion. Let's say there is some kind of road work going on and there is a congestion here. So due to this congestion the traffic will get blocked here and these people will also get impacted. Okay. So this is number one issue in a tightly coupled synchronous architecture. If something goes wrong with one service, right? One call is slow or let's say one service is down everybody else will be impacted. The second issue with tightly coupled synchronous architecture is that when you want to add a new service, let's say you want to add a new service for loyalty point service. If you're using Zomemet or Uber Eats, you will get this loyalty points, right? Let's say they decided to add that feature later on. Now you have to add one more call. Okay? So it requires a code change into your main code base and also this R5 has to wait for all these services. All right. So there are these two key issues with tightly coupled synchronous architecture. Number one, one slow service can bring down the entire system. Number two, there is little flexibility because you are making a core change every time you are adding new service whether it's loyalty service, notification service and so on. So how about we design an alternative architecture where this part remains same. Okay, a customer places an order, order will go to database, you take a payment and that's it. But instead of calling all those four dependent services, you write an event to a message buffer. Okay, it's like a message cue where you will write an event saying that order is placed, customer ID is this, order amount is this. And then your restaurant service will simply monitor this. It's like your postman delivers a mail to your post box, right? And you will get it whenever you are free. So your postman is not blocked right. They are not waiting. They will put the mail in your post box and then you will get it whenever you are free. Same thing happens in email. Somebody sends you an email, it is in your inbox and you consume it whenever you are free. So here this one is a producer. Okay. So this is producing the events and this one is a consumer. It is consuming those events. And once this event is consumed right see orders are coming continuously second order will come third order they're coming continuously they will pile up and this one once it is consumed they will have some kind of offset right so right now the current marker is here once it is consumed they will move the marker to here okay and same thing can be done by other services they will have their own different markers okay so let's say this delivery service when restaurant service is at this delivery service will be at this and once that is consumed they will shift this pointer here and so on. Okay, tomorrow if loyalty service needs to be added, it's easy. Nothing changes here, right? This order processing back end is putting a message in this buffer and you can have any number of consumer consuming the messages from this buffer. It is like I want to play a volleyball. I love playing volleyball by the way and I have my friends who come to play volleyball with me. Now one option I have is let's say tomorrow weather is nice at 4:00 I want to play volleyball I can call all of these friends individually but that's not what we do nowadays what we do instead yes you guessed it right you create a volleyball group you post a message and all these people will consume the message whenever they are free so this message buffer that we referred to initially is nothing but Kafka now you'll think Okay, so basically it's a message Q. Well, not quite right. We will look into the difference between message Q and Kafka later on. But Kafka is actually an open-source distributed event streaming platform. Okay, you have to remember this. It is a distributed event streaming platform. Apache Kafka, it's an open-source platform. So in terms of code, you will have the producer. So our producer is that order service. Correct? So after the order is saved into the database, it will simply send an event. Okay, it will send an event. This is an event and this is the topic. We will see what exactly is the topic. But topic is an entity related to your business scenario. Here is an order, right? There can be another topic called payments. So you are pushing an event in uh Kafka using this producer. Okay. And once you do this, see this is asynchronous. You are not waiting for people to consume. You push it and then you are done. So this way this code is pretty safe. It is not going to be impacted by what what is going on in consumer side. On the consumer side, see this is a restaurant service. You will create Kafka consumer and this is Python code by the way. You will create this Kafka consumer and you will pull the messages. So it's like an event loop, right? For messaging consumer infinite loop. So you're constantly checking. It's like I have a post box. I'm constantly checking every five minute. Okay. Do I have a postal mail? Do I have a postal mail? I'm checking every half an hour, let's say. So this is that for loop. I'm checking. Okay. Do we have a message? Do we have a message? If you have a message, you will get this message here. And same thing for other consumers, right? This is delivery service. Similar code. So there is producer, there is consumer. Now just to highlight the differences uh between both the approaches without Kafka when you're doing tightly coupled synchronous architecture sync HTTP calls your order service calls four services directly. If one is down the whole flow will break. Adding a new consumer means changing the order service code. Okay. and latency increases with each downstream call because you're waiting right, R2, R3, you're waiting. So the R5 will have to wait until all the four calls are done. Whereas if you're using Kafka, order service will just write to one topic. It's like you have a message buffer, you are putting your event and that's it. And then any number of services can consume without order service knowing. Order service doesn't need to know who is consuming my messages. You're just pushing the messages to Kafka. And if analytics is down, others still work. Analytics can catch up later and you can replay the event. See, this is very important. In Kafka, you can replay the old events. So events are persisted to disk and you can configure when you are configuring your Kafka cluster. You can set up a retention policy. You can say last 5 days, last 1 year, last 2 years. I want to maintain all the messages. Okay? And you will be able to replay those messages. When I was at Bloomberg, we used to have uh this kind of event streaming platform and let's say if some uh something fails in production, what we used to do is we will go back and we will replay all those events. So let's say there is a production bug due to which all the processing in last half an hour failed but we still have those events in Kafka. So you deploy new software and then whatever events occurred in last half an hour you can replay them because events are persisted to disk and you can use it for reprocessing backfilling and so on. Okay. So Kafka is a distributed event streaming platform. If somebody asks you this question in an interview, this should be your answer. It's a distributed event streaming platform where applications write events and other applications read them with events stored durable and in order. Okay, in case if you're wondering what exactly is event streaming, well, it's a practice of capturing data in real time from sources like databases, sensors. Okay, this is used a lot in IoT devices by the way, mobile devices, click stream, software applications in the form of streams of events. So here are the real world use cases. Number one is Netflix. So I love K drama on Netflix and this is the series I recently watched Born Appetite Your Majesty. So when I click on it or when I say thumbs up etc. It will push a message on Kafka and there will be a recommendation engine. There will be a IML algorithm which will listen to that event stream and it will get updated. There is a billing system which will track the usage of this. There is analytics services you know which will track okay how many people are watching Kdrama, how many people have given thumbs up to born appetite etc and so on. Second use case is autonomous driving by Tesla. So Tesla cars can drive on their own and they have lot of sensors installed. So every moment when they're driving these sensors are generating the IoT data. It's a stream of IoT data which can be GPS speed battery all kind of metrics on on this and that gets pushed to Kafka and then there can be a real-time monitoring dashboard which will consume these events. predictive maintenance alerts, long-term analytic storage. Okay. The third use case is credit card transaction. See most of the credit card transaction companies have this Kafka setup where when you make a transaction the message will will be pushed to Kafka and there will be fraud detection model which will run and if the transaction is fraud within few seconds or few milliseconds you will get a message that oh did you do this transaction there can be user notification service this is a separate service okay micros service and it will send you a SMS saying that you have made this transaction in this amount at this place and there is a third service which is analytic service which can be capturing all these transactions and the this service will be used by some kind of analytics dashboard to view overall analytics on credit card usage. If you talk about industry adoption, Kafka was invented at LinkedIn. Okay, LinkedIn is a company that invented it and they process 7 trillion messages every day. Uber processes 1 trillion messages every day. You see trillion, do you even realize trillion is a huge number folks? Trillion means 1,000 billion. 1 billion means 1,000 million. 1 million means 10 lakh. Okay? So it's humongous volume of data that all these companies are processing and 80% of Fortune 100 companies use Kafka today. If you have worked in software engineering, you will have a natural question. What is the difference between Kafka and traditional message cues like Rabbit MQ? See in the architecture that we saw before let's say here if we are using rabbit Q what will happen is when restaurant service reads this particular message this message will be deleted other consumers won't be able to see this message. Okay. And even restaurant service let's say after some time if it wants to replay it it won't be able to replay it. Okay. It's like you're going to a restaurant and the chef has served you samosa, you eat samosa, samosa is gone. It's like message is consumed once it's gone. There are ways by the way in Rabbit MQ where you can enable multiple uh consumers. But the fundamental design is this. It's oneonone message is consumed. It's deleted. You can't replay it. There can be five different consumers consuming it. Okay. So message Q is more like I want to send a letter to my brother. I will send it via postal mail. Okay. So there is a letter. I send it via postal mail. My brother will read it and then he will throw this in a garbage bin. Okay. So let's say he throws this in a garbage bin. So this later is gone. So that is Rabbit MQ. Kafka on the other hand which is a distributed event streaming platform is like me uploading a video on YouTube. Now if my brother wants to learn data bras he can watch the video he can replay it when he watches the video doesn't get deleted right video just stays there for a long time and if he wants to replay it rewind it he can do all of that not just my brother but millions of other people on internet can see that video and they can replay and rewind. Okay, so here are the key differences between message Q and event streaming platform. Message retention. In message Q, messages are deleted once consumed. Q will act as a buffer whereas in event stream processing, messages are retained for a configurable period regardless of consumption. Okay, so this acts as a durable log. Okay, some people call Kafka a durable log as well. The consumption model in message Q is pushbased. Okay. So broker will push the message to consumer. We'll look into this broker concept later on. And messages are consumed once. There is a little star because there is a way in Rabbit MQ to kind of tweak this behavior. But this is a fundamental behavior. And then in Kafka it is pull based. Consumers pull messages. multiple consumers can independently read the same message by maintaining their own offset. So every consumer will maintain the offset. Okay. So I have read let's say first three messages so my pointer will be on a third message. Somebody else has read only first message so their pointer will be on a second because first one is read so it will be on a second. Somebody else has read 10 messages then their own pointer will be at 11th position. So there is a specific offset. We will go over uh this concept in detail as well later on. Now typical use cases for Rabbit MQ is background jobs, email sending queue, task distribution. Whereas Kafka you use it for realtime data injection, IoT sensor, click stream and in all these cases the volume of events is humongous. It's like millions and trillions of events. Whereas if you have like thousands of events, you better use rabbit MQ. Okay. So when to use Kafka? Well, realtime data pipelines, event-driven architecture, log aggregation, stream processing. See, you will understand some of these concepts as we do some hands-on practice. Okay. So if you're not getting all these jargon for now, don't worry. When not to use Kafka. Well, when you need complex SQL queries, don't use Kafka as your database. Okay? If you have a database thousands and thousands of records, you want to perform complex SQL queries, don't use Kafka there. If you have small data volumes, less than thousand messages a day, you're better off using Rabbit MQ and your request response patterns, right? So there are cases where having tightly coupled architecture actually makes sense. Okay, so direct calling REST API can be better. If you're a software engineer, you will understand my point. In your coding, right, you would have done this so many times where you are calling a rest API. You wait till you get the response and then you do further processing. It is perfectly fine. You should not throw that away because now you know this new concept behind Kafka. Okay? Use it only when it makes sense.