Apache Iceberg with Unity Catalog at HelloFresh

Key Takeaways

Hello and welcome everyone. Uh it's pleasure to be here and uh we will be talking today about Apache iceberg with Unity catalog and how we are leveraging that at HelloFresh. Um before we dive into that just just very show of hands, how many of you are actually using HelloFresh? Okay, cool. That's that's nice like glad to hear that. Uh I hope it's going to be more in the future of course once you learn what of an amazing architecture we have here up and running. Um but let me quickly show you what um we have actually prepared for you here for today. So origin at the beginning um we will be talking a little bit about data engineering at HelloFresh in general how our architecture looks like what was our starting point um on the journey towards iceberg and unity. we will dive a bit into the decision of like why we started using iceberg specifically at HelloFresh. Um and then I will be handing it over to my colleague Adam who will be talking about Unity catalog for iceberg in specifically. Uh so like really diving into the technical details of that um and also of the catalog federation aspects that we have with that. Before we do that quick word about myself I'm Max. I'm the guy in yellow. Um I'm the director of data engineering for the data platform of HelloFresh. Um so I'm leading all the areas related to the underlying data infrastructure as well as the productivity layers that we have on top of that. Um and um I've been very actively working on productionizing large scale data both infrastructure and data management platforms. Um over the past couple years I've got the chance to speak here at the summit already a couple times in the past. They're mostly focused on aspects around data mesh. Um, and that's also where I'm still like very actively involved with. Adam, do you want to couple say a couple words about yourself? Hello everyone. Uh, my name is Adam. I am senior staff data engineer at HelloFresh. Joined linked to HelloFresh for about four years now. Um, I built and help our internal teams to expand data platform offerings for internal users, customers and I am quite passionate of making the complicated um, challenges easy for the end users. and I started my career as a backend engineers, front end engineers and I decided to spice up the challenges and I moved into the data world couple years ago. Thanks. Just a couple words about HelloFresh. Um, we are not just offering HelloFresh as like a core offering. There's a bunch of other brands that we have and like probably very few people here also know that HelloFresh is actually a German company. Um even though the US is our biggest market and so we are very active here and as you can see like there's a bunch of other brands that we also have um that we are operating here um including factor for instance which is pretty big here as well for ready to eat um but also some other offerings like uh pet food um if that's something that you might be interested in. Uh just so you get an idea about the the the numbers um and the scale that we are operating in. Um last year we shipped almost a billion meals worldwide. Um and like if you combine that with like the total number of like 20,000 employees that we have worldwide. Um that gives you like a rough idea in terms of the the scale that we are operating in. Um we have like a pabase a pabyte large data lake. Um we have like various different tooling um that we are using on top of that. Um but yeah, let's dive right into that aspect. Um so data engineering at HelloFresh, data engineering usually breaks down still like into two large categories in that sense. There's the the batch processing on one side. There's the the stream processing or more like event driven real time um side of things and that is um still how it breaks down for us as of today. When we start on the batch side, um there's a bunch of different sources of where we are getting our data from. We have a companywide Kafka based event bus. Uh we get a bunch of data from backend databases for instance of the different services that we are operating. We still run a lot of processes on Google Sheets. So that's a lot of data that we're also getting from there. Um and of course there's also a couple manual sources um that people are directly uploading um sometimes even directly interacting with some of our partners. So first thing is you understand the sources and then you start processing them right like you start setting up the infrastructure um you start building your data pipelines you start setting up your orchestration um the core setup that we have is on the one hand side we are almost exclusively AWS-based uh so most of our data sits in S3 when it comes to the the the analytical plane um and we use a bunch of AWS native tooling as well when it comes to that processing specifically spark on EMR uh is something that we are very active ly using for data pipelining. We even wrote an in-house framework that allows people to simplify how they are writing data pipelines and that automates a lot of the infrastructure creation for them. Um so using that using airflow largely for orchestration writing the data back into S3. Um and then really getting into the part of then using that data to actually create value. Um where we have like a little bit of a split approach. Um we do have Snowflake on the one hand side as a data warehouse. um you can move the data into that. Um we also apply data vault modeling in there that allows you to to give like a bit more structure specifically applied to like the let's say top 100 metrics that are really being used for decision- making companywide. Uh so that is very actively practiced on that side. Uh but when it comes to data activation that that's also where we really leverage data bricks as well. um we use serverless SQL as like the main distributed SQL engine with like about 1,200 users uh on on a monthly basis and um but we also have like various other offerings from data bricks that we are leveraging there specifically when it comes to the the machine learning space um where we have like a bunch of data science teams that are also looking into that and leveraging those features and then we use Tableau on top of that when it comes to the visualization um so that you get a bit of an understanding how the stack looks like end to end. When it comes to the streaming side of things, um we use Flink um when it as like a stream processor for the one aspect of course directly interacting also with the data that we have in Kafka. Uh so like really reading data from there doing some modification on the streaming data itself writing it back to Kafka um so that other services can also interact with that. Uh but we are also leveraging Flink when it comes to operational monitoring. Uh so some information that is then directly going into some some Postgress databases that are then being leveraged for for visualization specifically monitoring and alerting um on the operational side where we are largely leveraging graphana um to for that entire setup but of course also that is used then for further analytics um so like we are entering exactly the same space that we just previously showed on on the slide um to cover that aspect. What are some of the challenges that we currently have with that? Well, on the one hand side, if you look at the big picture, we have now four different engines that are like interacting with our data on a regular basis. And permission management across these different engines is quite complicated, right? At the same time, we also have a lot of different touch points with the data on S3 itself. And data management on S3 prefix level is an absolute pain. And that goes one level further as well. When you look at the data integrity of the data that you have on S3, it's an absolute nightmare, right? Like trying to operate on these things on file level, you engage all these challenges like corrupted files, unintended overrides, like something like a small file problem. That's something that you that you're always encountering on that side. And that's specifically what drove us into the direction of looking into table formats, right? First things first, table formats in general, there's like a bunch of advantages. Of course, when you look at the different things, Delta Lake, iceberg, hoodie, um, as a transactionality was super important for us. Uh, but also getting some of the features like time travel or or the data versioning in that sense to get rid of these data integrity nightmares that we had um, up until now. um schema evolution but also schema enforcement in that sense is something that was very important for us and generally having that level of metadata management um so that we could process the data in like much more integrations that went into that um well on the one hand side we were really looking forward for this engine interoperability like of the of the data engineering landscape that we just showed before right um but we were also really looking towards having a cross engine rest catalog um that would be supported by all the engines so that we have like a single point of entry where we are actually maintaining our metadata so that it can be leveraged across the board but then also looking at some of the industry dynamics right like we already had some tables in delta format as well which was usually data that was natively created on the data bricks side um but with the acquisition of tabular last year there was also a big push on our side where we saw okay data bricks is doubling down on iceberg and a bunch of the other vendors that we've been working with have already been very actively working with iceberg before. Um so it became like a huge push into into that direction for us and on the one hand side that also re like resulted in the situation that we saw that also for the data warehousing aspect working with exter external tables and iceberg format we saw that there were minimal performance penalties like was was really almost as good as working with like internal tables in in that respective setup. Um and then also seeing that the Unity catalog integration with Uniform was pushed a lot more forward. Um that really brought us to the point that we said like okay maybe we don't even have to have like this this clear separation between Delta and Iceberg. Um because they are getting closer and closer together. If you've seen the the iceberg 3.0 announcement that just like launched one or two weeks ago um it's really showcasing that there's more and more moving into that direction, right? And that in combination with like the wide community support that we have on that um really like pushed us um towards the decision um to go uh to go all in uh towards iceberg. There are still some format specific features like specifically hidden partitioning like after the 3.0 release. I think like now that's probably the last big thing that is really still pointing on that side. Um but that those are the the last couple things um why we really wanted to move into that direction. And with that um let's take a deep dive into the actual setup of Unity catalog for Apache iceberg. And for that I'm handing it over to Adam. Thank you Max. So before we see how easy it is to connect iceberg unity with our various tools, let's start with couple of opening notes. Uh so first we heavily relied on the private preview feature. Um this was enabled for us at the beginning of the year. We received couple of updates. Uh it's still in private preview. uh hopefully it will be released quite uh quite soon so everyone can um get those features enabled for them. We are also deployed in AWS. Uh we are using single region for our data processing. So this might also matter if you have different cloud provider and different setups. And now we also need to uh create special permissions on the datab bricks uh level. First is external data access. This needs to be enabled on the whole meta store. Without this is not going to work. And also each Unity catalog which requires to be accessible by the external tools needs to have external use schema enabled. And it's quite important to note because um initially like the troubleshooting was not that easy like we have got all the same setup for the tools why one catalog works and another doesn't work. So those settings are quite important to get in place. And also all the examples I'm going to show I added them to my private uh GitHub repository uh accommunity catalog examples. So feel free to copy use them however you want. Take inspirations uh also uh let's cover how we sort of govern access to our tables and how we create even maybe the the tables. Um so although we are AWS um house we use Microsoft enter ID uh previously known as Azure active directory for our user identity management and the group creation. Uh so to simplify this for the users we created like custom modules uh Terraform Python which allows you to easily create a new groups in an id. Uh however the usertog groupoup management worries often uh sometimes we do manual addition to the groups. Uh in some of the repositories we are using uh terraform. Uh and of course after that we use uh skin functionality to sync all of our users and groups uh to the datab bricks. So it allows us to attach permissions to the groups for appropriate assets. So of course we don't want to rely on um the access being granted directly to the users or something like that. Uh so for um ease of management that we are using uh the group level permissions and what's important is also uh something we spend quite a bit of time uh troubleshooting is that whenever you create the groups manually for example for testing purposes those groups they need to be like account level uh in data bricks and not the workspace level otherwise the unity catalog and uh that's not going to work appropriately. Uh we also hit some problems with terraform provider as this is private preview feature. Uh there is no support for iceberg as of now. Um I'm also not sure if the terraform is the best tool to create a tables. Uh but because there was not clear and immediate solution for us uh we rely for now on like ado table creation. So the users most likely are creating a tables just inside uh the processing scripts. So with that out of the window uh let's start with the beginning of the journey Flink and our streaming uh data. So Flink has got the idea of the catalog and you can create a new catalog uh with couple of configuration parameters. So we need to configure that our rest catalog uh is of type iceberg. They also required you to provide catalog type um in our instance rest. Um and we need to specify the URI which is our unity uh URL. Most likely it will be always the same. The only difference is like you need to provide your workspace and after that it's the common URL for everyone and you need to authenticate appropriately to the catalog. So here we are using on behalf of token for the service principle. Um when we were initially designing this in working on this uh this was the only supported way of authenticating. I think also now OAF 2 is supported. Uh but for now we are still using the the token authentication. Uh so essentially this service principle has needs to have access to our catalog which is configured as a warehouse uh parameter. So with that in place uh we can then have this super simple Flink application uh which is querying source data. So for example it can be our Kafka topic. Uh we provide some transformation on our uh on our data coming from Kafka. In this instance we use tumble window. We group by five minutes. Uh we are interested in number of items maybe uh some value of our uh order metrics. And with this processed we can now invoke insert into statement rest catalog which is our local catalog name which we have just uh created and then schema name and table name which are now unity catalog resources. And this with this simple setup we have got now continuously inserted values into our unity table every five minutes. But now let's assume uh we want to build on top of this batch processing. So we are interested in aggregating every day uh overnight. So uh we are using spark spark for it. From spark perspective the configuration is slightly different. You essentially need to uh configure specific spark settings and the important setting here is the spark SQL catalog and after the dot we also have got the local catalog name. So how we are going to refer to that catalog in our spark application. So in this instance we have got unity uh and the first parameter is essentially java implementing the iceberg catalog which is what comes from the appropriate dependencies. So we have got orc apache iceberg spark and spark catalog. Uh again type is rest. Of course iceberg supports different versions of catalog but we are interested in rest here. uh and the same parameters as with fling URI warehouse and token. Uh nothing different here. Uh two points to note. Um I also configured extensions. Extensions essentially allow you to enhance your Spark uh SQL with additional like iceberg functions. So for example for uh metadata management and things like that. Um and also what I'm going to showcase uh later we have got the catalog.spark Spark catalog which is like default catalog for spark still configured for delta and this allows us to essentially use two catalog at the same time and ensure that our older jobs they still work as expected and they can still query uh all the data as expected uh without any interruptions. So with this configured we can now invoke our spark SQL statement insert into appropriate uh unity table and again unity is our configured local spark table spark catalog um and then uh order metrics and order statistics daily are essentially the resources which we created in the unity catalog um and it's just a simple example but we are also actually selecting from another table which was prepared by flink Uh so of course based on the appropriate permissions uh depending on how you configure your users um that user which is invoking this the spark uh process might need to have the right to appropriate table and read to another table and with this in place we can as you can imagine query this quite easily on the datab brick site without any extra settings. So of course assuming that I as a user who was invoking this query can access my or can access order statistics daily. I have got read permissions. Um I can see the results of my transformations. Um and I also I clearly see in the datab bricks unity catalog UI that this table is managed and the most important thing it's of type uh iceberg. So Max mentioned that we had got a lot of issue with the file permission. So why this works and I haven't mentioned anything about giving access to file. I mentioned only giving access to the table. Um and it works thanks for this idea called credential vending. So this is probably where is the biggest difference between various cloud providers. Uh as we are AWS house I'm going to explain this on AWS uh example. So in order to start we need to cover two datab bricks resources um and one uh AWS. So from the data bricks perspective we have got something which is called external location and external location um you can attach this to like meta store or maybe catalog or maybe schema and it's essentially location where the iceberg tables will be stored in S3 under some kind of prefix and it also has got information what credential will be used uh for this particular external location and the credential is like a key it's information what role data bricks will need to use and the role is AWS uh resource is some kind of special identity which doesn't have a password and it's used because it can be like impersonated uh by either real users or by machines and that role will have access to appropriate uh data location. So essentially to summarize the role is like two thing it is information who can assume that role who can impersonate that role assume is term AWS uses and what resources that role can read. So with this in mind let's see how our external tools can use this to get access to data without actually having explicit access to data physical data granted. Uh so there is a special um endpoint called temporary table credentials and that endpoint uh will return you uh appropriate uh credentials for that table. So how it works is that the datab bricks obviously uh knows what is the table location and what are credentials for that table and it will check if you have got appropriate permissions as a user uh to that table and it will invoke assume ro on the special AWS service called AWS security token service and what's important here this assume ro operation uh uses the idea of session policy and the session policy is essentially mechanism to lower down the permissions you can do and um you can imagine this is required because our like role has got permissions to read and write uh data maybe it has got permissions to access other tables. Uh so during this assume role with recession policy process data bricks requests new credentials but only for example for read access and this then those credentials are returned to our spark application and you can safely access data. you won't be able to access anything else. You will only be able to access files for this uh particular table. And this is great. This is something you can also test manually. Uh this is just rest a point which you can invoke from the command line. Uh it's it's quite great for for testing and ensuring everything works as expected. So um that covers our LE journey. Um let's see also how we federate our catalog. Uh so this is important for us from two angles. First is going to be snowflake and another is going to be our legacy fi our legacy catalog which we use AWS glue. So let's start with the snowflake. Uh snowflake is quite similar to what we had in flink. Uh it uses the same idea. There is special resource called catalog integration and we need to specify catalog source being iceberg rest. We can we need to specify table format being iceberg a bit weird specifying the same thing twice but it needs to be known. Catalog URI is something we already know. Um and the new parameter sort of is catalog name space which is our schema name. So snowflake sort of like links your catalog integration directly with the schema. Um and then to continue our configuration uh we need to specify rest config which is essentially warehouse our catalog name and this access delegation mode is something which I have just explained. Uh vended credential will allow snowflake to access physically the data without going uh through the loops of configuring access from the snow snowflake itself. Um and then we need to finish with the rest authentication. Uh again we set this up using on behalf of token. Uh I'm pretty sure it supports OP2 uh at at this moment as well. And just need to flick flick the flag. You cannot see this I think at the bottom but there's like enabled true uh as well. And that's it. With this in place you can query the data almost. You can query the data. There's this extra step needed on the snowflake side where you need to create a table uh create iceberg table but of course it doesn't copy any data. just some kind of metadata information for for Snowflake that the table you will create on the iceberg sorry on the snowflake site is linked from the catalog you configured and the table name in that catalog is for example in this instance order statistics um and snowflake sort of like refreshes metadata periodically uh so it doesn't actually go and and query the iceberg table completely so this auto refresh is also required otherwise you might need to specify some kind of manual refresh which is not uh ideal. And with this in place now we can finally query our order statistics table and it also give us uh appropriate uh results. Again everything without touching the the physical file authentication but only one thing which you had to configured is this on behalf of token for snowflake having appropriate permission to the schema and to the table. Okay. So now let's address our big elephant in the room uh our legacy hive and the options we tried to federate that this catalog. Um so our main legacy catalog is AWS Glue data catalog and we have got over 46,000 tables there at the moment of speaking. Um we we tried AWS glue for catalog integration uh with the unity catalog it works but only for the readonly access tables like you cannot write when you use the foreign catalog integration. Uh so it's not ideal for all the use cases. Um so we still sort of need to rely uh on this that you can configure the tools to support multiple cataloges and for example still rely on this hive unless you can really go u out of it. Um and for us really problematic are quite old jobs which are writing directly to glue. And what I mean here it's uh some jobs can even directly invoke glue API uh to create a tables um or even they would dump rope paret files or even JSON or CSV files and people would have like crawlers to to scan the physical files to just create tables on on top of them. So for us it's quite a big journey to get rid all of that all of the use cases and help our users to build this better with the unity and more modern table formats. So let's summarize this on the picture. Uh so we have got our uh external tools flink and spark and they would be interested in both a iceberg catalog and AWS glue uh data catalog. Um so adding a couple of like sort of intection points uh writing read to iceberg no problems. Um we still need to rely on writing parket JSON assets. Ideally we would like to avoid right but probably for some time it will be inevitable. Um so on the external tools it's actually not that difficult. Um but when we start looking into data bricks it comes up a bit more complicated. Uh so we need to add those two ideas into the picture. We need to add our federated catalog and we need to add our legacy hive meta story integration. So they work slightly. There is a subtle difference. Federated catalog sort of like copies the catalog and and crawl uh our uh data catalog on the AWS side. However, the legacy hive meta store is just like sort of permanent link. uh but the permissions are completely different like for federated catalog you can already get some benefits of the unity uh which you don't have when you use legacy hive metas store integration option um and when we start adding the our compute important point to notice u that we are predominantly using on the data side warehouse as the read workloads for our analytics engineers and we have got a bunch of compute clusters uh where data engineering teams would create or update uh new new assets. So adding to picture the the inter interactions from the SQL warehouse, it's actually quite easy. Uh we don't have many issues with that. You can read iceberg uh directly. You can read legacy tables assuming there are all the permissions are in place that works as expected. However, uh the right workloads are a bit more problematic. So first at the bottom I added even like red um arrow going to this legacy hive meta store. Uh with the unity catalog enabled workspace we have got some permission issues. Uh there are existing workarounds but it all it means that the migration path it's not that easy for us. The yellow line from to right um iceberg tables was yellow couple weeks ago but because it didn't work as expected but now in the latest updates it actually works. uh which I'm going to cover right now in the limitations and configuration section. So a bit of that will be a bit of updated. This is actually super quickly moving space and we were working on the slides and and preparing that uh things didn't work which they changed in the meantime. Uh so let's uh take a look. So from the snowflake perspective um we would like to close this loop and also enable writing into our iceberg tables. Um they didn't support that initially. they have now in like private preview which I managed to test like two weeks ago or a week ago even uh it works but it doesn't support this credential vending idea so it's not that simple as we would like yet uh so we are keeping an eye on this as well and and also some important features are only available in preview from the snowflake perspective as well so you cannot rely on properly partitioned iceberg tables um initially they were using like their own algorithm which made iceberg files not properly readable uh by the other tools which was not ideal from the datab bricks perspective writing to iceberg table was not possible either via compute or secl warehouses this has changed at the beginning of May and it's now possible to write to iceberg tables um from compute but not from SQL warehouse at least as I tested this two weeks ago um this like reading iceberg tables also was converting underlying tables to delta. Maybe it doesn't matter for the end users but when we were like implementing that uh it mattered uh because that conversion required modify permissions for the user calling this operation. So it didn't work for us because our read workloads actually had to have modify permissions on the table. But again this is actually now fixed uh from the beginning of May. So we were uh so happy to to see it. Um then sort of the last couple of of things uh troubleshooting of all of this was sometimes quite difficult as I mentioned like missing some of the uh setting ended up with like S3 related errors. Um I think some of them were also improved in the last weeks or months. Uh but maybe there's like odd uh not properly handed scenario there and there. Um and as you could expect because it's like quite a new way and approach of doing things um the information on the internet is quite scarce and AI is not doing great to help you and suggest proper solutions. It's hallucinating quite a lot and uh from the asset creation perspective this is something we try again to find how to work best with that. Um data brick supports terafform base on fly leaf although still not for iceberg unless maybe uh it changed a week or two weeks ago. Um we are not using liquid base or fly leaf. So maybe we will need to end up writing some of our custom scripts. Uh but this is something for us to also consider going forward. Um and for our users uh like this whole migration uh it's not migration only to unity but also iceberg as I mentioned we have got a lot of quite legacy use cases and trying to get everyone on boarded will be quite a big journey for us. So with that I would like to hand over to Max for the wrap-up section. Yes exactly thanks a lot Adam for like walking us through all those technical details. Um so let's just like summarize what we were talking taking taking a look at um throughout the session of today. Um so first things first, iceberg with Unity works quite well for us because it actually integrates quite well with the entire tech landscape that we have at HelloFresh, right? Um and specifically also having the Unity catalog integration with Uniform um thus also supporting iceberg and Delta as as Adam just mentioned behind the scenes um is something that that works quite well for us um because of some of the some of the legacy constraints that we still have on that. Um just recapping like the this that there's almost no performance penalty also for the data warehousing use cases that we have. Um so that was of course like a pretty important point to us. Um but one aspect that I really have to highlight here once more is that like there's wide community support um that the topics have right now and that there's really very active development from from all the vendors that are like involved um into the the pictures that we showed earlier today um is really allowing us to make like huge progress in this space, right? Like a lot of these cases that Adam even mentioned that that didn't work for us originally and even like within a couple months have already been addressed and fixed. um that only showcases like how much of an importance um that is to to to all the companies that are involved in that space and that gives us also a lot more confidence that this is the right choice and we are moving into the right direction here. um some of the learnings um integration is very much possible across the entire ecosystem but it's still quite techy let's say right so there's still like quite some limitations that we need to go through there's like a bunch of things that you need to figure out sometimes with hacky workarounds um sometimes also by by by pushing the vendors a bit more um to to address some of those shortcomings which again they are very happily um able to oblige um but generally Generally speaking, it's a very fastm moving space, right? Like so some of the core concepts um that were still true six months ago have been like completely addressed uh already like throughout the last couple months. Uh so it's great to see that there's like huge amount of innovation that is continuously happening and it's like spreading more and more also across the industry. Yeah, as I already mentioned, some of the issues from three months ago have already been resolved by now. Uh but there's still a lot of moving pieces to really fully productionize this at scale, right? And when I'm looking at the at the next steps here, um also for us, this is only the beginning, right? We have end to- end production setups now that that are like up and running. Um we have everything um throughout the entire tooling landscape um that we have here. Uh but we still need to scale this across the entire organization, right? So there's still like a lot of things where we now need to migrate away from some of that legacy so that we can fully be committed to the the end to end setup um that we have now showcased uh to you folks today. We are very closely working with the Unity catalog team um as well as with the iceberg teams um to drive this forward um and this also really helps us to engage in the follow-up conversations and that's also something that I can very much encourage you folks if you have the needs if there's something that you want to double down on if there's something where you just want to discuss like the potential options reach out to to the community become part of the community reach out to us uh we're very happy also to to uh have some follow-up conversations on that. And with that, there's only one thing to do, and that is to say thank you. [Applause]