How FedEx Achieved Self-Serve Analytics and Data Democratization on Databricks

Key Takeaways

All right, good afternoon. I'm Pat Brown here talking about the FedEx journey to self-service analytics and data democratization today. So, we'll spend the first few slides showing you what the company vision is here, the overview of the platform itself, and we'll dive deeper into the actual data sharing, governance, and analytics we built for our different analytics users across the enterprise here. So the vision within FedEx is to build the smartest and most intelligent supply chain for everybody here. So what does that mean? We basically have physical assets. That's our planes, trains, automobiles and digital assets. That's our billions of scans come through every day. It's our marketing data. It's our sales data. That's our finance data. Basically everything we have that actually powers that network is being run through the centralized data platform within FedEx now. And so recently we made a decision to centralize all those assets into one data platform itself. So the solution we've done here is basically where we've got thousands of analytics users across our enterprise who have to do anything from a one-off SQL query to find manufacturers shipped last week to a production dashboard AI model whatever it is to empower the business to run every day there effectively and so what we delivered here is called the valuedriven analytics platform it's very much meant to standardize our technology ecosystem and data assets beyond thatism as well so on the top here we have Atlas that's our enterprise data platform up there that's hosted on Azure effectively basically what happens is we have data coming into Azure every day effectively every minute in some cases and that is basically powering VDAP in the bottom there effectively and so VDAP is mainly put on data bricks today and as you move from left to right in the bottom there we have workspaces that are very much meant for what's calling on demand problem solving and then as we move to the right there it's more about solution development there effectively. And so if you're an analytics team member FedEx today, you basically have the ability to do a one-off diagnostic analysis. Let's say that's looking at, you know, how many shipments were delivered last week, for example, or did we miss our sales k last week from what was that that case there? Right? The next one is more for machine learning. And so if you now get asked, all right, we know we missed our KP last week or we know there was issue last week, we'll step again in six weeks, for example. So more of that predictive analytics capability there as well. These two are very much meant for mention is called question day if you will. You're not going to go in first two work is build up what's called an always on solution. Very much meant for doing kind of that that point in time question got answer for the leader there effectively. The lesson over here more advanced. These have basically the ability to do more production workloads effectively. So anything from experimental dashboard AI model automated report whatever it is effectively can be quickly spun up there for our leadership team given to them iterate on there effectively and then once they're ready to graduate to the production environment that's more of where we can do with the always on reporting AI work um stuff that is used to give to business leaders technology partners anyone across the enterprise area right so basically the goal here in the bottom is how much do the obviously the technology complexity and governance rules from the users who don't care about that stuff, right? What they have to do is answer a question for the director of sales or VP of logistics today and not care much about how we spin up compute or manage data access or manage anything that is underlying that is not core actually solving that mission for analytics teams there right this here is the enterprise data platform so basically we have on the lefth hand side is different source systems across the enterprise that are either bashing data into the system or streaming it effectively and so with that means is we have data coming into our platform every five minutes every five days doesn't really matter the platform itself it says give me all data you've got here effectively right so it's all about getting data in one spot and then from there we've got the medallion is designed here obviously so we've got basically the ability to go from bronze silver to gold as an analytics user what you really care about is gold there right so you don't want to really spend time analyzing the unstructured data there you know it's called a JSON fire XML format there for example it's all about very tabular very structured data in gold there. Once in that format, it's basically used downstream the analyst platform itself. And so we have platforms today that are being used to power labor insights, financial insights, operational insights and that's basically the the main workflow of getting data from a source system into our data platform and downstream the analy platform as well. I will say we've also got data bricks plugged into this what's called inestion workflow as well here. And so as data is pulled in, database is used basically to process data and store it as well as well. But that's a quick plug for daticks there. All right. So this is a more technical view of actual analytics setup here effectively. And so again lefth hand side you get basically how it's called the the basic technology platforms, right? So very simple SQL Python analytics there to find out what happened last week. a little bit more advanced ML workflows as well to kind of build up that model, answer that kind of onetime question for leadership there. As you move to the right there, obviously you've got more of the advanced stuff. So everything from Azure, Kubernetes, Flank, PowerBI, ADF, these different Azure tools are very much centralized to our workflows today. On the bottom, we've got the data panel itself. And so you can see basically we've got data coming in from customer service data. We've got equipment data, revenue data there effectively. Basically we've got a ton of data that is coming in every day effectively and it's all about how we make that data easily exposed to our analytics platforms upstream. And obviously on top you've got the user itself. And so depending on what they're doing, they might just be doing ad hoc work dayto day there, right? But they may also get asked, all right, this is a great ad hoc report that I got last week. I need you to automate that for me effectively. And so graduate from the first workspace to the last one there on the end there. So it's all about giving them the the tools again to see all the business needs as as there right. All right. So this is a few examples of how they're using it today effectively. And so first example here is sales leader support here. So let's say we have a a segment that's underperforming in our sales organization and then we find out why that happens. So basically get access to the first two workspaces, run some queries, find out what happened there effectively and then go back to their leadership with this is what happened and that is how we want to basically shift our sales KPIs based those results there effectively. And so it's kind of that mix of diagnostic and also predictive analytics workloads here effectively in the lefth hand side right hand side is more production workloads here. And so let's say we have a developer working with customer service agent. And now basically what happens is customer service is inundated with different calls IM's emails day-to-day there effectively and they basically need a a production system to collect all the data that is used for customer service issue resolution there effectively and answer that kind of in a self-service manner there. And so basically what happened is we have a developer analyst who would collect the data for the customer service agent there. They would build up a dashboard for example to give visibility into everything that's relevant there. So what is basically the the package status there? Is it late is on time? What is what is going on there right and obviously we can monitor SLAs's over a period of time. So the main thing here is how do we give the tools to our downstream customer service agents to do their job more efficiently. We don't want them having to message our team kind of every day there effectively. It's very tough because there's so many of them, so few of us obviously. So how do we give them the tools and technology and data to act kind of on their own there, right? All right. So that's what's called the overview of the company, the data platform and the analyst platforms itself. Now we'll talk about the actual data sharing between systems as well. And so the problem statement here is how do we collect data and make it easily available for the downstream analytics platforms, right? And so we have some issues here as well because obviously it's a big company at FedEx. We need to be able to give our engineers who are ingesting data a very seamless way to get that data into the platform itself. And so we don't want to block their development work there. We do have to enable them to basically go quickly there effectively. And so basically have a few solutions here. The first one is we use catalog basically organize our data. We also implemented a common data pipeline and configuration based approach to expose data as well. And so there's no manual work done in this at all. It's all automated in the back end there effectively. And so as data is pulled in from actual metadata and data standpoint, it's automatically exposed to our data platforms downstream. And the last thing here is DT. And so we basically use DT in the enterprise data platform to store and expose data with the with the ability to analyze downstream there. Right? So how it looks here again. So you just saw the the first on top part there about the enterprise data platform on the bottom we've got the actual workflows that actually allows us to expose the data and metadata as well. So what's happening on the first section here is the actual data ingestion itself. And so let's say this is um delivery data from the west coast that came in last week for example. So the engineer basically create a YAML file that's used to spin up injection infrastructure itself. And so everything from containers to VMs to security components is all put in that ammo file itself then run through a data pipeline to actually ingest data. And so as system record exposes data to Atlas enterprise data platform it's basically automatically collected via that pipeline itself. Second set is the metadata exposure here. And so this we have some partnership here between our enterprise injection engineers and data governance analysts. And so basically what happens is they collect the metadata from the source system itself. They look at it make sure it's accurate from a description data type. Uh it makes sense obviously that's put into a very similar EML file and fed their data pipeline and exposed in the analyst platform itself. And so we basically have what's called the role level data being exposed from the platform on top and then also the metadata being exposed from the analyst platform on the bottom there as well. And again it's all automated. So as they finish that work for the configuration there, it's deployed and then it's running there on automated schedule there. So as you do add a new data or new metadata, it's automated pick up there via the uh pipelines there. All right, this is an example of how it looks in the metadata file itself. And so really what we're doing here is collecting let's call it the core information. And so that could be US liver packages. We've got source systems up there. We've got a table summary as well. We also got two very important parts here to use. So there is a data classification sensitive nonsensitive there's more as well that I just didn't put here but that's a very important part here as far as controlling data access and then also data domain as well and so like I showed you before we have shipment data we have finance data we have logistics data so these domains basically used to organize data in UC itself and so as we do pull data in that it's not in some big massive blob of whatever data right it's it's very well organized by basically data domain there on the bottom you see the the different columns as well and so we basically have let's see uh tracking ID for example so just tells you what that data is the data type which is good for you know running queries on that data itself we also classify data at the actual column level itself and so what that allows us to do at uni is basically to control access the column level as well if we want to as well so we have it basically at the table level if that makes sense also the column level if we got to as well on left right hand side you can basically see the workflow as well. So what happens is we basically collect that metadata. We configure it within that metadata file. It's basically reviewed and pushed to production by the engineers themselves there. And so those are the let's call it the eight most important uh attributes there. But there's obviously a lot more that we put in this as we go. All right. Now data mockation. And so once you have data into the environment you have to have the governance behind it effectively. And so the main idea is how do we allow our users to basically discover their data without jeopardizing the access controls in the back end there. And so we basically have the ability to show them what data is out there, find it for their own use cases there and then request additional access beyond that too. And so the way we basically did this is by organized data via in catalog within data domains there effectively. And so there is a data domain for marketing data, data domain for shipment data. there is a metadata exposure by basically collecting all of that metadata so it's visible and then exposing that to users themselves and the last part here is data access controls and so the ability basically to see data and actually query data is managed separately and so basically that means is if I'm an analyst today and I think I need need marking data but I'm not sure which table I need for example I can see metadata in the catalog today but I cannot query that there without having addition access approved by our different governance uh team there effectively. And so that's just how it's set up to be sure that we do get them the ability to see data but not to expose all the enterprisewide data across the enterprise here. All right. So this is just a logical view on the lefth hand side of how it's set up in uni catalog. So we have the catalog enterprise data platform itself. We have schemas. Let's say that's pricing, shipment and finance data. and then tables that are delivery and other types of data there as well. And so basically what you can do is you can browse all levels of that data the metadata itself with one image role that's our access control within FedEx and then there are controls based on the table data domain and sensitivity as well and so basically there are different access roles if their data is sensitive and shipment data for example versus if a table is not sensitive and also shipment data same thing for finance and so on and so forth there. So that basically allows them to scan through the entire catalog as it exists today. But if they need access to finance data, there is an additional access control for that different data itself. We also have views here too, but I I put that here, but it's it's basically the same thing where we basically have views of the data itself. And so let's say we have sensitive data in that table itself. Because we've categorized the actual column level itself, we can actually redact that sensitive data so that someone can see the sensitive table itself elements in that table. So that basically lets them say I can query this data just with sensitive data but if I need the additional image role for data access there's a that's also available for them as well. Alrighty. So enabling self-service analytics. And so we have a few things going on here. The first thing is we've got to be able to support on demand problem solving also different solutions itself. And so that's basically a very simple what's called IT setup and a very complex IT setup there as well. We also have people who are very technical and less technical. So we have people who can do very complex SQL and Python and Spark scripting. People who just know how to do let's call it analytics without the coding itself. And so we've got to be support those two different kinds of personas there. The last thing here is rapid deployments. And so we don't want teams waiting on us to get access to their data or technology itself. And so we basically have to find a standardized way to spin up these different environments so that they don't have to wait for us effectively. So what it looks like from a solution standpoint is we have different analytics products that can support on demand and let's call it reusable solution development here. Effectively we have native data bricks tools that is to use for technical non-technical uses there. effectively is basically the the SQL and notebooks type setup for uh technical folks and then Genie for non-technical folks. And the last thing here is we have what we call usage driven templates that are basically used to spin up these environments as well. And so as someone comes in and says I need access to do an ML experiment for example that's basically populated within a few minutes user can basically get that access to data and technology pretty quickly there as well in the back end. So very similar to the AML file set up for the metadata itself left hand side you basically got the file used to spin up these environments to and so this is very much template where you see the bolded text is what we basically populate you know periodically here and so basically we name the workspace we assign access to it based on the group itself and we do a bunch of other stuff as far as setting up compute setting up unit catalog setting up schemas can be used for data storage as well this again goes automated pipeline it's automatically provision the back end there and spin up for the user in a few minutes. Then the bottom there we've got basically the the workflow of data ingestion let's call it and exposure to the analyst environment itself. And so again data is coming in from finance source systems, shipment data, uh pricing data for example, that's all production data flowing to the environment. And so that's a big thing for us as well was we don't want latency between our different data ingestions and the analyst environments itself. And so the main idea here is that as data is exposed in the platform itself, it's automatically synced to work as it stands today. So this all about basically giving our analytics users the ability quickly find their data and analyze it in real time there right near real time obviously it's uh still uh some workflows that are complex from an IT standpoint so what happens is we have data coming in from the production systems unit catalog is basically used to do two things there so it's basically used to expose that production data and also expose metadata as well there and so as your new today you basically see the production data role level there and the sample data view There I'll see the actual table descriptions, column summaries, etc., etc. And you see so you can basically find that data and analyze in a pretty quick and efficient manner there. All right. So kind of combined diagnostic and predictive analytics here into one one little box here. But conceptually they have the ability to use SQL environments uh AI genie notebooks and obviously the out of the box ML tools autoML and feature store as well to to build models. This is all about doing let's call it the analytics setup for the actual user itself. Beyond that we also have data viz options here too. And so if they want to build a quick dashboard to look at their model results or look at their trend results there for a period of time. PowerBI is an option there. There's some listing there that can be prohibitive obviously. So we do have the ability to build the PowerBI dashboards in addition to the data bricks dashboards as well. So whatever works them but data bricks dash afterwards are more more than good for a lot of what's called the ad hoc work they got to do their day-to-day. All right so beyond that we've got some in cataloges in the bottom here. So if they basically want to create storage accounts and store data in un catalog they basically have the ability to create table results to store what's called feature training data or SQL results whatever is basically used just to go back and look at those results in the future if they want to. So it's all about giving them the ability basically to do that quick analysis save it for you know another day there effectively and then that's just there for their their analysis through there. Last thing here is serverless compute options. And so we spun up different compute based on the environment itself. But conceptually they have the ability to activate a medium a larger 2XL cluster here effectively. And so what that what that's important is because one from a cost standpoint we don't want teams trying to figure out like all right should I use larger 2XL for this? Medium is good for most things. Basically we tell them is if they are in medium getting bad performance it's much much easier to do that query in larger 2XL there. Right? So we don't want them waiting an hour to get that query result. It's all about giving them the ability to run most of their stuff in medium. If they want to do larger 2xL, it's also available for them as well. So good for cost there. And performance has been pretty good for the the past uh past year or so here. So this is a few insights we've learned from the different setup here effectively. And so the first one here is service compute. And so this is all about giving our analytics users the ability to quickly spin up resources without thinking about it much, right? So when they get access to the environment, it's already let's call it bolted on there, right? So they don't have to really think about like how does it autoscale, how does it spin up, they basically press one button and get access to the compute they need to run that query. So that could be analyzing let's call it a thousand rows on a table itself or 100 billion rows, right? We have data coming in every day, every second effectively from scan data, from finance data, from marketing data effectively. And this is all basically giving the ability to quickly analyze that data and find out the best way to solve the problem there. Right? Next we spin up here is self-service monitoring cost for costing users here effectively. And so we don't want our teams in the dark about what they're spending the environment. Obviously when you're running a query it's costing FedEx money, right? So we don't want to give them let's call it the the blackbox approach here where they don't know what they're spending. And so we basically spun up customized dashboards to show them spending the user level and also compute users as well. And so this basically good for two things. The first thing is if they do want to be sure they're not overspending on their budget, whatever it is, they basically have the ability to find out if someone is doing that in their workspace. So I'm a manager, I want to be sure no one's overspending in my workspace, right? Second part here is compute level usage. And so we don't want them again thinking that medium is is always the best for think for most workloads. If they are running workloads in larger 2XL, that's another warning sign for the manager to say maybe you should downsize from those jobs. Be sure you're not uh overallocated on that job itself. The last thing here is minimizing resource usage and overallocation here. And so we basically set up predefined rules in the workspace itself. And so as they do spin up cluster, it will all terminate after a few minutes. So we're not let's call it running that cluster for a period of time. We also have the ability to set data retention as well. And so if they do store data in the environment, it'll be there for about a quarter or so. And it'll actually auto automatically purge that data as well. And so it's all about let's call it making sure they're not overallocating the environment. And we do have the ability basically to downsize things if you have to there. All right. So as far as results go, so we've onboarded about 1,400 enterprise users over the past eight months here. We've got over a thousand tables catalog in the environment. This is growing every day effectively. Got over 18,000 analyses executed here. And our daily viewers is only growing here. So the main thing here is we want to give teams the ability as they migrate from their legacy workloads, legacy tools across the enterprise into data bricks, be sure they've got the need to do their different workloads there. Key learnings is mainly just how we install UC across our workspaces. And so we get so much value just using UC for everything that is let's call analytics related and so we don't want them have to think about like if they wrote a new AIP at Genie tool today for example can I access that answer is they can because we've automatically enabled everyone sees union catalog here. First one is just obviously metadata. So we have good metadata practices. Before we started this, we found some gaps in that process as well. And so investing time and resources into actually collecting metadata, making sure we have a good data access process as well is very important for us too. So basically want to give the ability to think about what data is out there and then we can basically manage the back end from them from an asset standpoint. All right. So make sure you submit a survey here. Um obviously data bricks is want to tracking all these different results here. So appreciate you take time to submit a survey here for sure legal responses just this is all forwardlooking statements obviously all veryformational hope for you guys and uh that's uh that's the show for today thank you for your