Hema scales time to market developing a data mesh on AWS (Technical) - Cloud Adventures

Key Takeaways

Hello everyone. Thank you for joining this technical episode of cloud adventure series. My name is Am Singh. I'm working as a senior solutions architect at AWS. Today with me I have Tomaso from HA and we are going to talk about HA's data intelligence platform journey. But before we start Tomaso, can you introduce yourself? Hi Amit, thank you so much for having me today. My name is Tomaso Parachani and I'm the head of data and cloud platforms at HMA. Thank you Tomaso. So can you walk us through Ha's journey from on premises to cloud and specifically how you have built the data intelligence platform? Yeah, we wanted to take advantage of all the perks that come with running a cloud-based data platform and as part of that initiative, we decided to decommission and deprecate the existing onremise data platform. That also gave us like a unique occasion to sort of like rethink a little bit like the way that we carry out our data operations within HAP. uh and uh we knew that we wanted to start treating data as a product within the business and really start building what we internally refer to as a data marketplace where we have producers and consumers and data transactions happening. Uh this this vision really fitted like you know the construct of data mesh uh which was additionally like incentivated by the existing uh architecture enterprise architecture which is based on microservices which fits really well the data mesh construct. Nice. Can you tell us the core components of data mesh? Sure. The central component of the data mesh is uh the team that I'm lucky to to manage at HA which we refer to as the core data team. The core data team has two main responsibilities. The first one we refer to as the platform enablement and governance responsibility. As part of that we do build the platform, we do build the infrastructure that we then offer to the different domain teams to carry out their own uh data operations. We do this to allow consistency, optimization and alignment in the way that we do carry out data operations. Then after that we sit together with the teams and we enable them in in achieving their success uh by helping them out with any sort of like you know technical solution that they need assistance with. Last but not least, the team is also responsible for the governance framework and as part of that we strive to ensure the availability, the security and the quality of the data that is being produced in the mesh. Next to that, the core data team also is a producer within the data mesh and we do produce what we internally refer to as the core data uh products which are all those data assets that are central to the functioning and the operation of the business. Then rotating around us as the core data team we have all the different domain units team which also carry out their own data operations running their own data pipelines based on the platform that we have offered them. So this results in a situation where we have like many data producers and many data consumers which are looking to meet each other and start sharing the data and using each other's data. So that really like fitted our motivation to build what we internally refer to as the data marketplace where data producers and data consumers could meet and put in place those transactions. And after we initially launched the platform, we realized that we were coming short of a solution that would seamlessly allow uh this to happen. Uh and that's actually when I started engaging with AWS and I got preview access to Amazon data zone which ended up being our solution. Could you explain the architecture of data marketplace solution particularly how Amazon data zone has addressed the data catalog challenge? Absolutely. So when we look at the data marketplace um we're looking to have in place and have an answer to four core pillars of data management. The first one being data catalog. So the ability to build an inventory of the data that has been built and published across the mesh. This enables the second core pillar which is the data discovery which is the ability to discover and to understand which data is being produced across the different domains of the of the of the data mesh which then leads to the third point which data sharing which is the necessity and the ability to start consuming data that is produced by somebody else within the mesh. All of this though needs to be wrapped by a strong governance framework which is the fourth and core pillar which is the ability to have like policies in place that can manage the logic of these transactions that happen within within the data marketplace. uh so with that we actually started looking at our options and that's where uh you know like Amazon data zone stood out as a great product as it would have like an answer for each of these four pillars of data management in a single solution which you know for us was quite unprecedented. Um so we decided to invest into building uh the the Amazon data zone, implement it, launch it to production and make it also the central business data catalog. And this was in conclusion an very important item for us as it also came as the central piece of a pretty complex ecosystem where in data mesh we have different business units that also have the freedom to work with different technologies, different tools and a different stack. For example, in our ecosystem, we have business units working with data bricks to run our ETL uh workflows and other teams working with native AWS services. So, we were really looking to have a central uh piece that would allow this two separate worlds to communicate and Amazon data zone offered that possibility. So, what specific benefit Amazon data zone brought to HA's data management practice? Great question. Uh so when we look at the tangible benefits that data zone brought especially like on the business side first of all we talked about like the enablement of the data discovery which is quite priceless but I want to put emphasis on the data sharing practices because this is really an area where we have like a quantifiable uh you know like uh you know progress that we made using the Amazon data zone. So we estimated that in the past using a lake formation pipeline that the core data team built to share data across different environments the data sharing turnaround time would look anything between four and five business days. You had to create a ticket, you had to refine it, you had to wait for its turn, you make it happen. With Amazon data zone, we fulfill one of the main criterias of data mesh, which is self-service. And this allows domain teams to talk to each other, use the catalog and with a click of a button get access to the data that they need. So metadata is a key feature of Amazon data zone. So could you tell us how this feature has helped with the data catalog challenge? Absolutely and you're right, it is a very key component and key feature that comes with Amazon data zone and it helped us massively both on the technical end as well as on the business end. Starting from the technical end, the main advantage is the save in time invested by human to write descriptions about like a certain data assets. And you know, it sounds a simplistic answer, but we're talking about hours of work that are actually saved uh by the AI made generated metadata. Uh but even more importantly on the business end the the metadata really gives the data consumers uh the opportunity to have nuance information about like the preposition and the scope of a certain data asset uh with descriptions that uh also cover all the different possible use cases as well as deep dives into the schema definitions. So what impact this platform had on the data producer and the data consumer? It's a great follow-up from our previous question. Uh, and to answer this question, I will actually um use as reference a couple of different features that I think really show uh the benefits both on the consumer data consumer side of thing as well as on the data producer side of thing. Let's start from the data lineage. uh from a data producer side of things, data lineage allows us to have uh a a picture of the who is subscribed to the data and what the status of their subscription is. And on the uh data consumer side of things, the data lineage feature even more importantly allows them to understand the journey uh of a certain data asset from its source through the different layers of transformation all the way to the point it is published into the catalog. Second, we have the data quality scores. With data quality scores, the data producers is on top of the quality of the data asset that has been built. Uh giving them the possibility to set their own data quality rules. While on the data consumer side of things, uh the data quality core feature uh allows the uh the consumer to have like a clear picture on the reliability and the quality of a certain asset that might then be used within their own operations. Last but not least, we have like fine grain access control uh which it's it's a feature that again uh comes in and on on both sides but mostly benefits like the data producers and this allows them to really share only what is needed and what is strictly required to be consumed by the consuming team as well as the consuming team to keep you know the the the within their data inventory just the minimal viable product in terms of the data that they need to consume. So looking ahead, what is the plan? So first of all, as we have been talking about data catalog, you know, we're fascinated by the announcement of the SageMaker Unifi studio, which offers also a new version of the SageMaker catalog. As part of that, we're going to be looking into the possibility to migrate from Amazon Data Zone catalog to SageMaker catalog. But from a broader perspective, now that we really nailed and invested time in building solid data foundation and a solid and healthy data organization, we have been already starting to work on some new exciting projects uh to deliver some important new data products uh such as AI solutions and then data science solution that will continue to power our operational excellence. Excellent. and we will continue this journey together and thank you for explaining us HA's data intelligence platform journey. Thank you everyone for joining this episode. If you want to learn more about HEMA's data intelligence platform, please do watch the business video. Thank you.