Monitor Your Python Applications with Prometheus & Grafana

Key Takeaways

What is going on guys? Welcome back. In this video today, we're going to learn how to professionally monitor our applications and also visualize what we're monitoring using two tools called Prometheus and Graphfana. We're going to learn how to use them in Python. So, let us get right into it. [Music] All right. So, we're going to learn how to professionally monitor our applications using two tools called Prometheus and Graphfana in this video today. Now, both tools are open source and free to use. This is not sponsored. This is uh commonly used in production. So, this is uh these are two tools that are commonly used in production. And this video today is about getting you started with this setup. So, it's not about giving you an in-depth tutorial. We're not going to do a crash course here. I just want to show you that these two tools exist and how you can set them up, how you can get started with them. And if you want to go into more details, you can do that either on your own or if you are interested in learning from me specifically, let me know in the comment section down below that you want to see a crash course on Prometheus, Graphana or both. And if there is enough demand, I can make a crash course on these. But this video today is just for getting you started with these technologies. Now in a nutshell, Prometheus is a tool for monitoring and for getting metrics out of applications and Graphfana is the visualization of these metrics. So the idea is we have an application, this application uh send certain metrics to Prometheus and then Graphfana takes them from Prometheus and visualizes them. That's the basic idea. And again, what I want to do in this video is give you a basic example. We're going to build a Flask application and we're going to attach Prometheus and Graphfana to them or to it. And we're going to do all of this with Docker and Docker Compose. So, we're not going to install this locally on our system here. We're going to do everything in containers. And to get started, I want to go into my working directory, which is my tutorial directory here. And we're going to start with a very basic setup. We're going to start by creating a requirements .txt file. Now, usually I would recommend using UV at this point, but since I don't want to confuse people that don't know about UV, we're just going to use a basic pip setup. So, create a requirements .txt file. And we're only going to need two packages today. One is going to be Flask, and the other one is going to be Prometheus_client. So, this is what we're going to use in Python to communicate with Prometheus, which is the monitoring uh tool, so to say. And then we're going to say app. py. This is going to be our flask application. And in here now, we're going to have some very basic endpoints. Something like hello world, something for working with numbers. So we can do something with the input here in terms of monitoring. And also an endpoint that is going to deliberately crash the application. So we can also handle exceptions and monitor exceptions. Uh so we're going to use a couple of examples here. We're going to take a look at a at a couple of examples here. So let's get started by saying from flask import flask itself. We're going to also import request and we're going to import response. And then from Prometheus uh_client we're going to import counter. Now there are different metrics that you can export. So a counter would be something where you just export a raw number. So how many times was an endpoint called? how many exceptions did we raise? Whereas for example, summary would be more for aggregation. So I could have numbers that I process and maybe I'm interested in averages or something like that. And this is what I would use something like summary for. So we're going to take a look at these two. Then we're also going to say generate latest to get the metrics endpoints. So we need an endpoint for the metrics for Prometheus. And then finally also we're going to import this content type latest. All right. And that's it for the imports. Then we're going to just say app is equal to flask and then underscore_ame very simple. And now we can do a bunch of things. So one thing that I would like to do here and I'm going to just implement everything right in the beginning. So we can use it uh in Prometheus and graphana. Let's say I want to have some exception counters. So I want to see what kind of exceptions are unhandled in my application. How often are these occurring and what are the different types of exceptions and where do they happen? So what I could do here is I could say app.error handler. This is now specific to flask. If you use some other framework, you of course have to adjust this to the respective framework. But we can say here error handler for any sort of exception is going to be a function called catch all. It's going to take e as a parameter here. And then we're going to say actually before we do that we need to create uh the counter. So the idea is we're going to have a counter that is going to keep track of the exceptions. And this counter is going to be incremented every time an exception happens. So this here is an error handler. It's going to call this function here when an unhandled exception occurs. This can be something like internal server error. This can be something like not found. This can be something like we are raising an exception. Whatever that exception is. And this is going to be handled by this function here. And in this function we want to increment a counter. But this counter needs to first be created. So we're going to say exceptions is equal to counter. We're going to give it an identifier. So, app uh exceptions total for example. And then we're going to give it a description total number of unhandled exceptions like this. And then we need to pass two things here in square brackets. First of all, the end point. So, where did it happen? And then the exception type which is what kind of exception actually was raised and was unhandled. So that is going to be kept track of. And here we're going to say now exceptions dot labels. And here now we're going to pass the information. We're going to say endpoint where does this occur? We're going to take that from the request. So it's going to be endpoint is equal to request.path. This is the endpoint like /hello for example. And then the exception type exception type is going to be equal to the exception that we get here. So this catch all parameter E. We're going to get the type of that. So type of E. And we're going to take the name of that class. So it's going to be a class like key error for example. And we want to have the actual string name of that class which is key error. So not just the class itself but the name of the class a string. And then we're going to take that and increment this. So we're going to say inc plus one essentially and this is going to group it by exception type and by endpoint. But of course then in graphana we can also have a query that sums this up. So at the end of the day we get the total total if we want. So we have here the app exceptions total. But we can also get the total of that by taking the sum of that. We're going to take a look at this later on. But that would be one classic example. So I would have the number of unhandled exceptions group by exception type and endpoint. Now let's do something else. Let's say we also want to have a request counter in general. So I could do here app.befor request and then I could say uh every time before a request actually I don't need parenthesis here. I'm going to do the count requests function here. And what we're going to do is we're just going to count the number of requests. So I can copy that here. I can say request uh count. It's going to be a counter. Let's call it app requests total. And then we're going to say total number of requests like this. And here we're going to say method and endpoint. So we also want to know if it's get or post. So uh one endpoint can have multiple HTTP methods. We could have a get uh and a post call to the same endpoint. So it makes sense to distinguish that. So here we have request count and then what we want to do here is basically the exact same thing as here just that this time we don't pass the exception type we pass uh the endpoint and the method. So here I'm going to say method is equal to request do method endpoint is equal to request.path uh we also don't need the name and then we just increment this. That is how we count requests. This would now be for every single type of requests. Now we can also uh do that for only a specific type of request. So we can say here if request do method is get then we only want to count this otherwise uh and of course here I need to change by the way request count uh otherwise we don't want to count that for example so whatever you want to do here but in our case we're just going to count all the requests and group them again and then what we want to do is we want to have some basic endpoint so let's go and define app.rout route hello. This is just going to return a text. So we have something we can call without any other effects. So I can say here return hello world like this. Uh and then maybe what we could do is we could say app.oute. And we want to have one of these endpoints where we can pass numbers. So let's call this something like print number. Very simple. Nothing too complicated here. And this endpoint should accept arguments. So I want to have here print number. We're not going to add a parameter. We're going to get it from the request argument. So from the question mark in the URL. So from the URL parameters. Um and we're going to just keep track of this number. We're going to of course print it or return it. But then we also want to get the average for example. So this is what we would use summary for. Uh and I'm just showing you here some examples. You can of course do whatever you want with that. There's also additional stuff you can do like additional plugins and you can also get CPU and RAM information. uh but you need to use an additional tool for this. I'm not going to get into this here in this video, but you can customize this however you want. I'm just going to give you a basic example. Then you can go deeper into that and customize it to your specific use case. But now what we're going to do here is we're going to say number string is going to be equal to request.arguments.get want to get the number argument. Otherwise, we default to none. And then we're just going to without uh I mean we're going to check if that's none. So we're going to say if number string is none we're going to return none or let's say number needed whatever status code 400 and what we're also going to do here now is we're just going to try to convert this without checking. So I'm going to just say float of number string which of course is going to result in an exception if this is hello world for example. So if this is a string that doesn't contain just a number, it's going to of course uh raise an exception, which is what I'm trying to do here because I also want to trigger this exception counter here. But what I also want to do is I want to keep track of the numbers that are passed to this function. So what I want to do is I want to define up here a print number for example, which is a summary object. And this summary object we're going to call it as an identifier here. We're going to call it app_print underscore summary underscore uh sorry not summary print number summary. This is going to be just summarized information about numbers past whatever. Let's just use that as a description here. And here actually I don't really need uh any any square brackets. So I don't need to have any grouping here. I can just say give me the summary and I'm going to get a bunch of things in return. And what I want to do here now is I want to say print number dot observe. So I just observe that number and it's going to be uh added to this to this object. You could say to this summary object. So it's going to be observed and it's going to allow us to take things like the average from these observations or of these observations. And then I'm going to just return here number whatever the number was if it didn't crash up until this point. Uh and finally let's also create an endpoint or not finally but uh before we create the final metrics endpoint let's also create an endpoint called crash which is just going to crash the application on purpose. So this is going to be a crash function. All it's going to do is it's going to raise a key error. Let's say you can make this whatever you want. And finally then we're going to have the endpoint for Prometheus. So app route /metrics is what we're going to call this. And this metrics endpoint here will return a response object. And this response object will get the content of generate latest. And we're going to say mime type here is the one that we imported. This content type latest. And the rest is just if_ame underscore is equal to underscore main then we're going to run the application app run. The host is going to be to make this compatible with all environments 0000 and the port is going to be 5,000. That is our basic flask application. If I didn't make any mistakes, this should work now. And we need to combine this now in a docker compose setup with Prometheus and Graphfana. So what I'm going to do now is I'm going to create a Docker file for this uh backend. So for this Flask application, I'm going to use Python 3.12 as a base image here. So I'm going to say from Python uh and then colon 3.12- Slim. So a more minimalistic version of that. I'm going to say the working directory is going to be slash app. I'm going to copy then the requirements requirements .txt file to requirements .txt. Then I'm going to run the command pip install d-n no cache directory-r requirements txt just so we have the dependencies installed. Then I'm going to copy the app.py py itself and the command for running this or actually maybe for documentation purposes let's also expose 5000 here it doesn't really have an effect but it's uh yeah documents that port 5000 is used here and the command is going to be python app py and of course in production you don't want to do it like this um by the way let me use proper quotation marks here Not sure if that's necessary, but I usually in Docker always use these. Um, but yeah, of course, you would use something like Gunnyorn or something else when you're serving this in production. I'm just going to keep it simple here now with the ordinary app.run. But that is our Docker file. And now we also need a Prometheus config. And the Prometheus config is not going to be very uh sophisticated or comprehensive. It's just going to uh mention the interval. It's going to set the interval. Then we're going to also point it to the flask application. And it's important that we always use host names of docker or that are specified in docker compose when we're working with docker compose. So we are not going to pass local host. We're going to pass the names of the individual applications. So what I'm going to do here is I'm going to say prometheus.yaml. And here I'm going to say now global and I'm going to set the scraping interval. So, how often do I want to update the monitoring or do I want to scrape for the new values? I'm going to say scrape interval, let's say 5 seconds. You can adjust this depending on how important the information is to you or how up to date it has to be, how granular and frequent it has to be. Uh, and then we're going to say scrape configs. Uh, job name. I'm going to call this here now flask application or flask app. And then we're also going to say static configs and we're going to configure that this has to listen to the following targets or the following target which is and here now we're going to use a host name that we're going to define in docker compose. So I'm going to say flask instead of localhost and port 5000. So this is going to target our application. And now the final thing that we need to do is we need to combine all of this into a docker compose file. So I'm going to say docker compose yaml. We're going to create this file. Now I'm going to say services and we're going to have the three services. We're going to have flask the back end. We're going to have Prometheus and we're going to have graphana. So I'm going to say flask. In order to build flask what we need to do is just build the current directory. Uh we're going to have the container name. Let's call it flask app like this. By the way, I probably think I need to prune my container. So let me briefly do that. docker ps or docker container uh prune to get rid of all the containers that I'm no longer using or actually maybe I did that already so I don't need to do that. Um but you specify a container name that you want to use. Then very important we need to expose the ports here. Um 5,000 needs to be mapped to 5,000. This will expose the application. So this is actually going to map your local 5000 port to the Docker network 5000 port which is the default bridge network that is going to be created or used. So keep in mind if you do that with something like a database or something where sensitive information uh exists this is going to expose to be exposed to the public. So be careful with that setting. Um and then we're going to say Prometheus. And here now we're just going to say the image that we want to use and we're going to use the official prom / Prometheus latest. Probably in production you want to fix a version. That's like sort of best practice to do. Uh then container name is going to be just Prometheus. Then we also want to take our config and put it into the container. So we want to say here volumes and we're going to map our uh local dot / Prometheus YAML. We want to get this onto Etsy or etc whatever you want to or however you want to pronounce this Prometheus. And then here Prometheus.yaml we want to make this read only. So colon R O and then here also we expose again be careful with that 9090 like this to 1990 in the container. And then finally we have also graphana which is going to also be based on the image called graphana slash graphana latest then container name no surprise is going to be graphana actually I don't think we need to use these quotation marks so let me get rid of them briefly um and then ports Here what we do is we expose the ports 3000 3000. So the port 3000 is mapped to 3000. And then important this depends on Prometheus. So we're going to specify that Prometheus is what we need first to be able to use Graphana because Grafana is going to visualize what Prometheus gives us. So what Prometheus scrapes so to say. Now, if I didn't make any mistakes here, I should be able to now say, and again, I'm not going to go into the Docker installation. I assume you have Docker on your system. You know how to use Docker or at least, yeah, you have it on your system and you can just type the commands that I typed here. If you want to learn about Docker, I do have a crash course on this channel. You can watch it for free. Just go to Neural 9, type Docker and you're going to find the crash course. Uh, what we're going to do now is we're going to say Docker Compose built. This is going to build everything according to the Docker file. So our flask backend is built. And then we're going to say docker compose up. This is going to produce a problem because we don't have Okay, so I had to turn off the already existing containers on my system. You will not have this problem unless you've already done this before. Uh and now that they're turned off, I'm going to say again docker container prune just to get rid of them. Delete all of them. Uh and hopefully now I can say docker compos built again. then docker compose up. And now you can see everything is started, everything is running. You see a bunch of logger messages here. Uh and what we want to do then once this is done is we want to go to localhost uh not 3,000 but 5,000. This is going to be an internal server error. Why is that? Okay, of course because we don't have uh a default route. So we have to go to localhost/hello. That is our flask application. Now how can we monitor this system? First of all, we can of course go to Prometheus itself or we can also just go to the metrics endpoint. I can go here to metrics. You can see here this is the Prometheus output. Uh this is what we can do. But we can also just go to localhost and then port uh 1990 which is Prometheus. Here I can use Prometheus directly with queries and all that. But we're not going to get into that because we're going to use directly graphana to get the information and to visualize it. So actually we're going to go to port 3000 which is where graphana is uh running. And here we can log in with a default admin admin. So username admin password admin. We can skip changing the password. And now here I'm in graphana right now. So here I can set up everything. The first thing that you want to do here is you want to add a data source. So you want to go to connections. You want to go to data sources and you want to create a new data source which is going to be Prometheus. So you can see there's also other stuff that you can use, but we're going to go with Prometheus here. And we need to connect to it. So we need to provide the URL. Since we're using Docker Compose, this URL is going to be http/ Prometheus port 19990. So not local host, but Prometheus 9090. Save and test. You can see it worked. So now I can go to dashboards and I can create a new dashboard and add a visualization. So I can say for example take Prometheus and then down here I can choose a metric. I can say select metric and you can see I have a bunch of things that I can choose from. I have some default stuff here and I have my app uh metrics. So I can see here app exceptions uh created total app print number summary count created sum. So I can use sum and count to compute an average. I have here app requests total. So let's go with that. Let's click on this. Let's click on run queries. And what you see here then is I have a graph. And you can see that this graph here is split also uh by different endpoints. So depending on which endpoint I used, I see a different uh number here. I can also aggregate them by just going to code here. So instead of using builder, I can go to code and I can just use sum. So I can say sum app requests total to get the overall um metric. So here I would now just get the number of requests total without any grouping. So I can just uh take this and I can say save dashboard uh maybe let's call this my dashboard whatever save this and now if I click on dashboard my dashboard you can see I have this panel here showing the sum app requests total and I can click here on refresh every 5 seconds then I can go here I can make some requests click on this a little bit do a couple of hello requests here And hopefully here if this refreshes we're going to see or actually did it already refresh? Maybe that was it already. There you go. 41. So this is basically increasing over time. There you go. 71. I think that was the increase. So of course I can also say give me here the time frame the last 15 minutes and I'm going to get a more zoomedin view of that. So I can see here the number of requests over time. You can see this is where I clicked a bunch of times to hello. Um, of course, metrics is also being constantly pinged by Prometheus. So maybe you want to exclude that because that's going to happen all the time. You're always going to get calls to metrics uh which are going to increase the number here in the graph. Uh but let's try the other ones. Let's say here add visualization and let's try to add a visualization for the exceptions. So what I can do here is I can go again metric and I can say exceptions total. I can say run queries. In this case here, you're again group by exception type. Now, if you don't see this by default, you might need to cause an exception because if you don't cause an exception at all, it's not going to maybe realize this metric is a thing. So, it's not going to show it here in graphana. So, cause an exception once to be able to see this here. Then you can choose that or you can go back into the code. You can say sum and uh you can say again run queries. Everything now aggregated for exceptions in total. Save dashboard. Go back here. Go to dashboard. We can also rearrange this. So I can do something like this here. Uh so now we can see here exceptions total. Let's cause some exception. Let's go here to slash uh crash which is going to lead to an internal server error. But actually for the wrong reason, I think because crash did not return a valid response. Did we actually raise a key error or was this just was this another exception that I caused? Now, not sure, but we got an exception. Okay, so that's good enough. That's the only thing that crash uh should do. Actually, I don't want to refresh here. I want to refresh here. And you can see now from four, we went to six exceptions. So, we got two exceptions in a row. Um yeah, this is something you can monitor here. Now, let's go and do something more uh I wouldn't say complicated, but which requires an extra step. Let's compute the average of the summary data. So let's go here and we have here the metric print number summary count and print number summary sum. So what we can do here is we can show those individually. So we can show the sum which is going to be zero because we haven't done anything yet. So let's save that and let's also go and create another one here for the and I'm not going to do any naming here. I'm just going to use new panel all the time. I don't want to waste the time here with properly designing this. Uh but we can go here to what did I use count or created? Uh count or sum? Not sure. Let's go with sum. Let's uh run the query. Let's save the dashboard. Maybe now I have two times the same. Save dashboard. Oh, actually maybe let me just see here. Discard. I used the sum already. Okay, so I have the sum twice. Let's go to edit and let's turn one of these to count. Wrong queries. Save dashboard. Save. Go to dashboards. Uh why does it always want me to to do something here? Okay. So, we have the count, we have the sum, and what we can also do now is we can calculate the average by dividing. So, we can say sum over count. Now, it's not going to work if the count is zero. For obvious reasons, we can't divide by zero. But I can go here to print number and I can say question mark number is equal to 50 for example. Okay. So there seems to be a problem. The view function for print number did not return a valid response. Let me just briefly fix that. App py. What's the problem here? We have return number. Not sure if that's the problem, but maybe we need to add a response type here or a a status code. I'm not sure if that is the issue. I mean, this should raise. Not sure if that is what messes up the rest of the program. Let's give it a try. Let's try to build and up again. Okay, now it says n is not defined. Okay. Uh I think Okay, got it. The problem is of course we are not using n here but numbum. So there was an exception. We just didn't see it because the catch all didn't rease. So that is something we need to fix. I'm not sure what the problem with crash is. Uh or actually maybe there wasn't a problem with crash. Maybe it just didn't raise it. So first of all let's build. Then let's up. And let's try again. Now, now everything is going to be reset since we restarted the application. But now here you can see number 50. So what I should be able to see here after a while hopefully is that the sum is 50 and that the count is one. So we should be able to once this was properly recognized here. There you go. We have 50 here. We have one here. Now we can do some division. So now I can go and say visualization Prometheus uh and now I want to go here to code and if I don't know what the names are anymore I can go here to app print or to the metrics and see app print number summary count. So here I can say now app or actually I also see it here app print number summary sum divided by app print number summary count that is the average. So I can say run queries and I'm going to get here the average which is 50. I can save that. There you go. Go to dashboards. And now I have this. It's here. So we have 50 here as the baseline or as the as the current uh status. And now if I do something like 100 here what should happen is it should go to 75. So the average is 75. The sum should now be 150. So this year should go up to 75. This here should go to two. And this here should go to uh 150. There you go. So we can see here 75 150 and two. And you can play around with that. There's many things you can do. Again, I don't want to make this a crash course on Graphana or Prometheus. This is just showing you how um this can be done professionally, how you can monitor systems professionally. There's so much more you can do here in terms of visualization and different plot types uh and interpolation all sorts of stuff here that you can do. I don't want to get into that. Uh you can choose a bunch of different plot types here. But that is a basic setup. And if you guys want to know more about this, you can let me know in the comment section down below. But this is a super easy to setup stack where you have a Python application that can be whatever you want. This can be Django, fast API, Flask, and then you have Prometheus, which is monitoring everything in Graphana for professional visualization. So that's it for today's video. I hope you enjoyed it and hope you learned something. If so, let me know by hitting a like button and leaving a comment in the comment section down below. And of course, don't forget to subscribe to this channel and hit the notification bell to not miss a single future video for free. Other than that, thank you much for watching. See you in the next video and bye.