Cutting Costs, Not Performance: Optimizing Databricks at Scale

Key Takeaways

Hey everyone, thank you all for coming today especially after that gamechanging keynote that we just have and we are everyone is just trying to assimilate everything that we have just heard and thank you for coming here hearing a little bit about cutting costs not performance. So let me start by asking you a question. How many of you have ever been asked where are my data pipelines costing so much money? Or how much money is a specific how much money is a specific data data process costing? Or how much money are we investing on this specific business unit to give them their data? And it's difficult to answer these questions, right? My hope today my my goal today is to help you a little bit on this. So let me start by introducing myself. My name is Peter Fer. I'm project manager in entity data Portugal. I'm actually the practice leads in datab bricks and uh during my the last few years I had a chance to follow up really closely most of our data bricks projects not only in Portugal but also some of them in other geographies. Okay. And based on that experience, I built this presentation. In addition, um, last year we had a a project, a really interesting one that the goal was exactly this one, cutting costs. And based on that project and with a close relationship with data bricks, I also was invited for the champion program. And a few months ago I was I had the honor to be recognized as um solutions architect champion. So what are we talking about today? So first of all we will cover some of the most common challenges that we face on almost every um organizations. Also, we will uh uh I will share with you some of the best practices that you can take in consideration while while facing these issues. And at last, I will also try to share with you some real world use cases that we found throughout our projects to show you that uh and to show you the real impact that these things may have. Okay? But my main goal is not just to share this with you. My main goal is to plant a seed in you and for you to go back to your organizations and challenge your teams, challenge your data architects, challenge everyone to cut the costs. So let me start with one sentence. Meaning meaningful cost reduction begins when your platform aligns precisely with business reality. And to be honest, when I was preparing for this presentation, I shared this with with a colleague of mine and he told me, Peter, that's trivial. Everybody knows that. But can you really say that all your organizations are and your all your data platforms are 100% aligned with the business reality business requirements that that they have and I think this is the main question that we we have to answer. So let's now start to dive a little bit technical and let's all start talking a little bit about how data bricks costs work right so data bricks it's pretty much the cost is they are pretty much divided into two in two areas DBUS that is actually the cost of you using data bricks right and the cost related to infrastructure That's actually the cost that you will pay for renting all the infrastructure to your cloud provider. And there are some other uh concepts that I want to also to to introduce to ensure that we will all be on the same page throughout the rest of the presentation. The first of one data bricks is pay as you go. And this one is a concept that I want you all to keep in your mind because I will bring you bring it back a few times during the presentation. The other one is that you can actually prep purchase some DBU. So you can prep purchase the datab bricks units that you have and by committing to that to to that usage you end up saving some money on those on those DBUs. Well, if you can save on on prepurchasing DBUS, you can also do it by infrastructure. And that's actually the reserve instances that you can do with your cloud provider. And in addition, there are there is also spot instances. Spot instances are are just machines, virtual machines. They're out that are out there. They're not being used. and the cloud providers enables you to to use them for a fraction of the price. However, there is a catch. They may need it and the process may fail. But what I challenge you to think on these situations is I mean could you uh just what would be the impact if the process fail? Aren't there some process that you can just put it on a on a machine it runs for example every minute or it has a really nice retry policies that ensures that it won't affect your your pipelines in a lot of them you can so let's start to addressing one of the main issues uh when we see uh the cost on a on a platform and the the the main issue is for what we see out There are a lot of data engineers that they don't know how to choose the right compute. I mean most attention engineers they focus on the pipelines but for what we see out there most of them they end up choosing the same compute that other processes are already using on the data platform or just overload overloading everything to the same cluster that there is and they don't really wait for a second and try to think about which one could be the best. So this list is pretty much the one that we that we use to to to try to decide the pro the most proper one. So let's start start talking about job clusters and allpurpose clusters. And in fact just to to to introduce you a little bit about the difference uh job clusters are ephemeral meaning that they are attached to um to a job and when the job ends the cluster also stops. You are not paying for any more than that. Actually uh data bricks will advise you to and always advise you to use these ones in production because they are you you you have the insurance that the clusters uh are not idle. They are not just there waiting for something to to be run and they are also isolated meaning that it won't they won't be impacted for some other process that may have a peak on the data that it's arriving and may have impact on a critical process that you may have running at the same time on the other end all purpose clusters they were designed to be interactive I mean these are the clusters that probably most of your teams are using on the notebooks to do their their developments or the on their day-to-day. And my my message in here and it's pretty much if you can automatize it, move it to job clusters. There is one thing that it's also good to take in consideration on this. Actually, the cost on DBUS on job clusters is smaller than on allpurpose clusters. The other thing that we also take in consideration is the uses of spot instances and reserve instances. I mean I just introduced the terms but like I said in the beginning most teams use the always the same clusters right and if you are using the same clusters meaning I mean that's actually you are using the same virtual machines. So why don't you go back to your cloud provider and com and commit with a with a number of of machines that you will use for a year. You don't have to be you don't have to be aggressive. You don't have to to commit to use another percent of your of your the machines that you used last year. You can you can have a different approach and start by 20%. But I'm sure that if you were able to save at least 5% on this or only 5% on this on your on the cost of your data platform that would be huge. The other one that we we that we also take in consideration is the is fotton the usage of fotton and I know that fton by itself it's more expensive the dbus are the price of the dbus are higher but remember when I said before for you to keep in mind pay as you go in fact for and if you check datab bricks documentation and some studies that is with fotton your process will most of your process will be faster. So my message in here is try it. I mean if there is a small chance that your process will run faster and you'll land up saving money with fotton why not worst case scenario it will be like one day or two days that the cost will be a bit higher but you can you can measure exactly how much uh higher it can be and wor and best case scenario you'll end up save a lot of money. In fact, this was what we did in one of in one of our projects that just by applying these these strategies, we were able to reduce up to 50% on the DBU count and DBU costs on a on a data platform. This was basically this organization had the a different orchestration tool. They were not using data bricks jobs and they had a huge cluster to compute everything every single day. The thing is they needed a large cluster for the data ingestion most of the data of the the initial data pipelines. But for the rest of the day, a simpler cluster would be enough. And just by moving this to jobs in data bricks, the other the other orchestration tool was still being used to trigger this to trigger these jobs. But just by moving everything to data bricks jobs instead of having that huge cluster that I was telling you about having smaller clusters to do most of the pipeline and only have this big cluster up and running for like a portion a small portion of the time. This was gamechanging for for for these guys and we did the we didn't did just this. I mean we also had to like I just was saying we had to to pick the right instance types. So in data bricks you have different types of virtual machines that you can opt right you have memor memory optimize compute optimize storage optimize you even have GPU clusters that you can or GPU machines that you can opt mainly for um your AI workloads and just the fact that you sometimes you can check what you are actually doing on your on your processes and wonder which can be the best cluster for that process. It's huge. I mean the impact it will be sometimes these machines are they are a bit a bit more expensive than than than the ones that are not optimized. But again, if you can run the same thing much faster, you you land save money. Also, it's important to take in consideration the cluster sizing, meaning the the number of nodes, the number the the size of the machines, the the the amount of memory that that machine will have. And in fact my my best s suggestion in here is start start with a a small cluster and monitor everything check if you if you need to to increase I mean you know your processes you know the process the clusters that you are using if you don't know if if you can if you if you you'll be able to to reduce to a smaller one check the metrics that you have you have uh spark UI and now the metrics in data bricks the UI is completely different from I think a few months ago and it's much easier to to understand and to monitor all these things in the cluster. So I challenge you to to check it out and if if needed or if you feel that it would be it would be appropriate just get a cluster to a smaller one and you I mean there's also autoscaling and auto termination and autoscaling sometimes when I when I uh I'm checking um one of our clients uh data platform I see that sometimes they have the the number of workers ers and then uh fixed and why are they doing that? I mean most times you can have you can start with one or two workers and then let it autoscale to the number that you you think that it may be required throughout the the your data pipeline mainly when we are talking about development uh environments. I mean sometimes when I when I uh when I check the development environment of a client they have a huge cluster in there. I mean that's not actually being used most of the time. And my other suggestion is to be aggressive on auto termination. I mean what's the worst case scenario that can happen most like I said uh before if it's in production my advice is to go with job clusters if you are using allpurpose clusters if you set an auto an aggressive auto termination meaning in development if no one is using the cluster the cluster read will die if your data team is working if they if they continue work, the cluster will be there up and running all the all the time. But if for some reason they stop working for 30 minutes, 40 minutes, 15 minutes, they don't run anything on the cluster, it will die and it will take five to six minutes to to spin it again. And let me give you an example on this. Imagine that you have two hours on auto termination and it's a number that I I've seen quite often. Imagine that your data team is working at 6:00 p.m. everyone lives. There is one guy, one of your data engineers that he had um a query that stuck in his mind. When he arrives home 7:30 more or less, he opens data bricks, he tries to run the query, it worked, close the computer and see continues the next day. If everyone stopped working at 6:00 p.m., the cluster if it has an auto termination time of two hours, it will stay up until 8:00 p.m. just idle. And if someone did something at 7:30, this window will grow until 9:30. Meaning that just on this day, you end up paying three and a half hours more for nothing. The other thing that um that we also suggest to take in consideration is the use of SQL warehouses. I mean you saw in the keynote the the performance on these ones it's huge. I know that they are a bit more expensive but they are they were designed to be used for BI and analytics. They have forton and predictive IO enabled in most of them. And I mean it's if you are if you have something connected with a PowerBI or if you have some of these or PowerBI or any other uh BI tool or if you have some of these analytics and BI workloads that it's pretty much SQL that the everyone is doing there. Why don't you try to check the SQL warehouses and at last the serverless computes and the serverless I mean for most of the sessions that I I seen throughout this event and some of the conversations that I had I feel that we will we'll have to be quite um we'll hear the name of the word serverless quite often. Because I mean you you saw on the keynote if you want to just to to call an API. I mean call it with a serverless. You don't have to to spin up a cluster you that will take five to six minutes call an API and just a few uh just for a thing that may last just a few seconds. Another example on this and it was actually when we end up saving up to 30% on the on the costs actually on the cost of SQL warehouses was by enabling serverless on the communications with PowerBI because on this case uh we end up having to to have all our clusters up and running at least eight hours a day. If a user wake up in the middle of the night, couldn't sleep, you really wanted to to check the their PowerBI, it would take six to five minutes to start. We would receive an incident uh that that day. So, everyone would arrive in the morning, there is an incident, no one happened, no not there is nothing there. It was just the machine that took a while to to to spin up. And just by moving to serverless, we could ensure that we were not receiving these tickets anymore. We could we ensure that everyone could use the their dashboards anytime they wanted and we ens we ensure that we were no um no overprovision overprovisioning anymore. In the end, we we had the business unit happy. We had everything running faster than than before and everything actually cheaper. So, win-win-win. There is also something that um I mean I was talking about compute and we were always probably most of us were on the keynote and we were hearing about delta life tables as well or lakeflow declarative pipelines if I'm not wrong and on those my main uh suggestion for you is try to understand exactly what they are What are the requirements that you you have when you are spinning up them? Because at at least at this moment you have three types of of data life tables. The cores, the advanced and the and the pro. And in some of them it's basically CDC then data quality. And if you are not using the full capacity of those DTS and sometimes it's quite difficult. I've seen some clients losing a lot of money because they were not cautious when they spin this uh DTS or any serverless pipeline because it starts autoscaling and sometimes it starts to to control it. If you if you know exactly what you need, just provide it th those needs to to to your data platform. Don't try to to think what if I if I need it in the future. If you need it in the future, you just change the the kind of the the machines that you have. It's as easy of that as that. Actually, like I said before, uh in the beginning, one of the biggest challenge that we have is to be able to answer those questions that we we all had of why is this costing so much money? How much money are we investing on on a specific data pipeline? How much money are we wasting on or or is this costing? And this leads me to one of the main things that I suggest you all to do that's governs all the cost of your data platform. And the main thing the the first one the first suggestion that I have there it's actually just actually something that you can do to ensure that your your data teams mainly the the most junior uh data engineers they don't just choose a random cluster and they they use it every every time. It's enforce cluster policies. I mean if if if your data team is not does not need a huge cluster don't let them create one. If they want to create one they can talk with you or they can talk with someone in charge and they can create another cluster policy for that. The other one and this one I mean I could have created a slide just for this one is tags because if you want to monitor you can only monitor based on tags. You can only filter based on tags, right? You can see everything together, but if you don't tag, you won't be able to to divide those dashboards and to check and to to split everything and to identify exactly how much money the data process that is trying to identify new leads is costing me. So, tag everything. tags. You don't pay for tags. What What's a worst case scenario? Creating so much tags that you cannot you don't even know what's there, right? And the next one is actually related with Unity catalog that I mean we just saw that 97% of the companies use it. So um and is to use the system tables because the system tables they have the billing the the billing tables that allows you to check exactly how much the views you are using. But it's not just the the the billing tables that I'm advising you to use. It's actually also the audit ones and the usage ones. So you can also check if your tables are being used or not. And I mean we see quite often and I think in all organizations this happens then a business unit asks the technical team I need this table I need this process I need something the engineering team creates it it's there and sometimes they are not they use it for one or two month months and then they stop using it and they don't say anything to anyone it's just there we are wasting a lot of money to have that pro those processes running and that the that data updated but no one using that and with system tables you can actually check on the check if the table is being used or not if it's not go back to the business unit and ask them can I delete this or not the other one is dashboards and alerts so build the dashboards around this actually if you if you are using unity catalog I mean you can create a um a dashboard related with your costs in just a few clicks. And the dashboard is quite good. It's quite complete. And I mean it's just as easy as that. We all know that data bricks does not work just by itself. It needs some other things. So it needs the the rest of the infrastructure. So it needs storage. And in here I mean after all the sessions that we have heard delta lake should also be there um it's delta lake and delta format or iceberg right we all heard about it same optimizations and my suggestion in here is if you can use manage tables I know that some organizations for numerous reasons you cannot move to managed tables you are you have to use external small ones, but even on those ensure that you have your regular optimizes and vacuums set to ensure that you don't have um data that is not used anymore stored on your storage account and if you use manage tables I mean use predictive optimization it will optimize everything for you. The other thing that I want you to take in consideration and this one is not only inside data bricks. It's actually related with the also the cloud provider that you may have is the usage of data life cycle policies because as you know the usually the actually the the cloud providers they have different kinds of or different tiers. So there is um a most expensive one uh a most expensive storage but cheaper right reads and writes and there are some other tiers that's ex exactly the opposite. So you almost you'd pay almost nothing for the storage but you pay a lot for reading and writing for that that year. And there was um a project that we we had that we were able to identify that the vacuums were happening only after the data was move was moved to uh a cool a cooler tier. Meaning that um we were moving the data from a place that it's actually cheaper to read and write to a place that the source is cheaper. But only a few days later we would go there and delete that data. Actually on this case uh when we moved to um to the second tier of data we had to rent and commit for at least 30 days of the data. If we remove the data prior to the to that period we would have to pay the entire time the entire period as well. So on this case, just by ensuring that everything was aligned, we were able to save all that money. But this one that's right here, that's a reduction of 75% of the of the costs was just by applying regular vacuums. I mean, we entered in a in a project that there wasn't they were not vacuuming their tables. meaning that all the chances that they have done throughout the entire life of that data platform all the park files were still there they didn't add deleting vector so I mean park is immutable so all the the the the files were there in addition databicks also needs network and in here I know that this slide. It has a lot of information, but my main my main idea in here and my biggest goal is to to ensure that you that you understand the different connections that may be happening in here. And also to to share with you which of these connections they have the the biggest amount of data going through because between users and application and data bricks account there is there is most organizations wants to to ensure that it's contained and only people with in the right networks can access it. And actually there is not a huge throughput in there. There there is low data that's going through that connection. But imagine that you have a private link or private endpoint between the connection uh on the connection for between your clusters and your storage. All the time that you will try to access or to run a query or to access your your tables, you'll have to pay for that connection. And for some organization that is a requirement but if on your organization that's not a requirement and you are actually doing that because sometimes on on on the processes that I'm talking I'm using here as an example we saw this this connection was built in a way that we were paying for for using this network and just by changing the configuration we we were able to save 20% not only on networking but on the entire data platform costs. Imagine if you could save 20% of your data platform. I think no one would say no, right? With Unity catalog and the way that Unity catalog manage the connections with uh the access connectors, this uh this architecture is a bit easier to to to manage. Before we had a we had some new some other challenges but now it's a bit easier. So what am I what do I want you to keep in mind and like I said go back go back to to your organizations and challenge your teams. So reassess workloads and justify costs. I mean regularly check it check it again. If you see that some some pipeline is not being used, if some pipeline the cost of that specific pipeline is increasing a lot, ask ask your teams why. If I was a when I was a data engineer, I hated these questions because I mean all the process can be improved and all the process can be optimized and people often they don't like for when you check their pro when you ask and question their the process that are running. Um but it's a work that has to be done right and people don't say that I was the one to advise you to to to go there. Um so choose also the right size and optimize your compute. It will be critical. It's one of the biggest source of costs on your data platform. Also tag everything and track everything. And just because it's not on on on DBUS, don't overlook storage and networking. So that was a lot. If you have any questions, feel free. Don't forget to to to complete the survey. Um, and thank you. Thank you all for coming.