Get Data Into Databricks - Data Collaboration for Ad Tech

Key Takeaways

If you're an adtech, this is probably a familiar scene. Your company collaborates with a ton of people. There are ton of ad hoc asks, and you rely on your partners to provide the best-in-class insights to your customers, all under tight timelines. On top of managing all of this, there are a lot of friction points that can arise, especially around data collaboration. Hello, my name is Amelia and I'm a solution architect here at Data Brooks. If you clicked into this video, chances are you personally experienced this the intersection between data collaboration and sadness. I mean, so you may have been asked to send or receive flat files with traditional methods such as SFTP or cloud storage. There may have been duplicative reprocessing such as when a data provider restates a portion of the data and the recipient also needs to reingest and reprocess the exact same thing. There could be inflexible and bespoke file naming conventions, meaning there may be some very specific rules for every partner required so that their bespoke systems um know how to process new files or send data. And finally, there could be opaque data retention policies. So, when data providers and recipients are kind of unclear about um what each other's costsaving retention policies that their companies have or how it's enforced, it can end up feeling like files are being deleted or removed without warning. And sure, you can accommodate this. It's not that it hasn't been done before. As a matter of fact, you might have already built a bespoke system um or an analyst team may have painstakingly compiled all the rules for the different parties in a document. And you know what? It works. But you know, can it be better? After all, you don't want your data engineering team to be focused on, you know, the mundane every day. Um, but rather you want them to be focused on solving your toughest problems. You probably want to reduce the time to turn on new pipelines and limit the time something is an edge case. You want it to be easier probably to reach more folks in a more controlled way. And overall, you probably want to minimize a copy of data to lower the total cost of ownership. You probably want to make crosscloud collaboration turnkey or have more fine grain access when provisioning. As a matter of fact, I'm sure you're on this journey already. And that's probably why you're here. Um, and there are definitely options in the space that are trying to address these issues. So, I'm going to show you what better looks like on data bricks here. We want to provide a more efficient way of collaborating with existing partners and an easier way to get from 0 to 60 with a network of new partners and customers who are on data bricks or not. So today I'm going to go through a quick demo of what it looks like to set up a data share and a clean room when you have a more nuanced use case. I'm also going to introduce this term called lakehouse federation which is a mechanism within data bricks that in simplifies data collaboration with data sets outside data bricks. So let's hop to it. So here we are in the data bricks platform and I've hopped on to this catalog section. So the first thing we're going to do is going we're going to try to make a data share. I've selected this um synthetic crosswalk that I've gone ahead and created. And so we're going to pretend that we are Meggaore and we've commissioned someone to create a crosswalk for us or an identity bridge between Meggaore and partner one. In this table we have four columns and the data looks kind of like this. So for each individual identifier that Meggaore has um we have a synthetic identifier for their partners. So they can communicate with each other using this data set. All right. So let's jump in. To start sharing, all you need to do is click on the three dots here and then click on this button. You can choose to add something to an existing share or create a new share. So, today we're going to create a new share. We're going to call this um the crosswalk and segments shared to partner one. Um, and we're also going to pretend that partner one has um given us their sharing identifier ahead of time. And so we're just going to paste that in. Oops. All right. And voila, we have a share. Um, so now we're going to go over to the partner one side. We can see here that partner one is using an Azure instance of data bricks. So, we're going to go over to their catalog explorer. You can see here completely different accounts. This one's Ager, that one is AWS. Um, and we're going to go over to the Delta sharing um page on partner one side or the Azure side. We're going to look for shares that have been shared with me. um and we're going to look for the provider. So here we will take um the provider's sharing identifier. We're going to pretend that um they messaged it over to um partner one. And so partner one can now search um for the share provider um with using the sharing identifier. You could see here um many people on my team have experimented with it. But the share that we have created is this crosswalk and segment share. So we're going to go ahead and create a catalog and we're going to use my name demo partner one. All right. Great. Right. Cool. Well, now we could see from here the entire crosswalk schema. Takes a second to load. So now we could see that the entire table is available in the Azure or partner ones workspace. All right. Wasn't that quick? Um so a few things that we I want to point out before um we move on. So let's go back really quickly to um the data provider side or mega core. We're going to go back to that share that we created. Um so here we have a standard table that we're sharing. But what if you wanted something say more granular or you had a view that you wanted to share? Um perhaps with partner one there are certain tables that you don't want to provision entirely. Um but you also don't want to make many copies of tables for each partner. Right? So, I've created this view here um that's based off of another table, a larger table called mega core campaigns. Um and so mega core campaigns contain anonymize ID list um for all of the campaigns that Meggaore runs. Um, but here you can see that I've used a separate allow list table and selected specifically on partner one to surface the mega core campaigns that are specific to partner one. And this is only one example that you can use. There's plenty more in our documentation um of how you can leverage um the the uh secure views that we have here um and share it out to your partners to limit the amount of data copying that you have to do and to be able to share things in a dynamic way. All right, so we're going to go ahead and also add this table to the share. So we know that our share is existing. What did I call this share again? Right. All right. And then one final thing that I want to point out is that the data doesn't necessarily have to live within data bricks and be delta format to leverage delta sharing. So, my friend Megan has prepared a table leveraging um lakehouse Federation, the term that I mentioned earlier. See here. Yep. And she's got a couple ones here. So, we're going to use um the MySQL uh database. So this is a um catalog that is leveraging lecast federation and is a direct connection to a myql database. So there's you know no cop no copy of the entire table um but still you're able to access um information in here. So here we're just going to take the generic ticket features table. All right. So we're going to go ahead and enable that with delta sharing. All right. You can see that we've added all these things to share. And this is actually using a MySQL data source. And there are plenty of others that you can look up on our documentation online with Lakehouse Federation. All right. So, let's verify that over at Partner One's workspace. Yep. So, we could see that we're in partner one's workspace as it's in Microsoft Azure. And we're going to hop on over um to this Delta shared uh catalog. So, we're going to refresh it a bit. Oh, actually, we didn't even need to. I didn't even see that. You could see here the view is also available um all within partner 1's workspace. And that's it for this section. Um so, you might be thinking this is great, right? like um you can limit what people's what your partners can see using leveraging views. Um if you have a data set that you want to share alongside other tables within um Unity catalog um you can be it if it's in my SQL or another cloud um data warehouse. So this is great, but perhaps you have some data sets with uh sensitive data or PII or perhaps you only want to provision access to tables under specific circumstances. So perhaps you have specific use cases that you would be eager to share with your partners that um a table on, but then you don't want them utilizing the table for other um any other use case. Um so actually the flow that we went through and making these tables available in Unity catalog puts you in the perfect um position to actually leverage our cleaner feature as well. So I want to show you what that looks like. We're going to jump back to Megaore or the AWS instance of data bricks here and we're going to go to the catalog explorer is our friend today. Click into cleaners and then click create cleaning. All right. So, we're going to go ahead and give this clean room a name. And then we're also going to pretend that partner one has um messaged us their clean room sharing identifier and keep the default security settings. Hit create. And this is going to set up our clean room. All right. So, our clean room is ready um for us to add some assets to. So, I'm going to walk you through the marketplace um clean room starter. Um and so I've already imported those assets here. So, I'm now I need to add the assets into the data clean room. Um let's see. I think I put it under my name. Yep. And it's right here. H2 demos megaore. And so we're going to go into this audience overlap and we're going to select the publisher identity graph. So actually let's take a second to backtrack on what we're trying to do here. So we're pretending here that we are Meggaore and Meggaore is a publisher. They perhaps have a um census level identity graph. um that they want to um share out with their partners in a very extremely managed way. Um so for this in this particular use case they are willing to share this identity graph for the purposes of doing an audience overlap exercise. So on the partner side they're going to be the ones that are experts at analyzing what um the overlap is. So they're going to go and provide um some data assets and a notebook to run the analysis on. So now we're going to jump into partner one or the Azure workspace. Um and we're conveniently already here in the clean rooms tab within the catalog. And because we created this clean room so recently, um the clean room is right up on top here. All right. So, we're going to now add partner ones um data assets and notebooks. Right. So, we're going to open that up. Okay. And you can see here we've created an audience overlap specifically for Megaore and Partner One. Add that in here. All right. And then within this notebook, we I know um that there is something that partner one needs to provide. So let's see here. I'm going to look up schema. All right. And we're going to add a part one here. Um and you might have seen that catalog level provisioning and um schema level provisioning is not supported and that's because we really want to make sure that you're only sharing the data sets that you intend to um and it's a mechanism to limit mistakes. All right. So yep. So here we have a data set of meggaore customers that actually made a purchase. And so we're going to go and add in this data set. All right. So then now we have both these data assets in the clean room. Um, and we're going to let Mega Core know and have them run the notebook that partner one provided. So, let's jump back to the AWS or Mega Core workspace. Um, we're going to go and refresh the page and we're going to see that all the notebooks and data assets that um, our partner provided. All right. And so we're interested in running this audience overlap exercise. And you can see here we can see any like plain text code um for transparency. And then we can also hit run. So we're going to go ahead here and click confirm that the notebook and any private libraries used correctly reflects the workload that you're expecting to run. Um, so kind of a little bit of a consent and the job has successfully started. So, we're going to give that a few minutes. Um, but as you can see, um, in the few minutes that we've spent together, we were able to go um, from essentially 0 to 60. And as a reminder again um uh today we walk through the point andclick or gooey version of clean rooms and data sharing but there are APIs and SDKs um that I've linked below that can help you do this at scale when you're ready. By the way, if we want to here, we can also monitor the run as it goes. And this run um is viewable on both sides. So you can see here we're back in partner one or the Azure workspace and we can click into the run. You could see that was started um by Mega Core or datab bricks field engineer. All right. And after a few minutes um we've jumped back in here and as you can see there's a status check mark. So we can see that this job has been completed. So for this particular case um within this notebook both Megaore and partner one has agreed that it's okay to share with each other um a list of house anonymized household identifiers. So you can see down here a excerpt of the resulting overlap analysis um and also a place a command that creates a table within um each data bricks workspace um that contains the overlapping households. To access that data, you just go back to our clean room. Create a location for the output information. Jump to that catalog. All right. So, we're going to click into um the schema here. All right. Um All right. So, we can see here that we have access to the exact same data. And we can even copy this name, take it into our SQL editor, and run any further analysis that we need. All right, so this kind of takes us to the end of the demo. Um, if you're interested, let's get you started. So for if you're looking to deep dive and develop right away for prospective customers, we have something called data bricks express to help you on board on the platform very quickly. Um for existing customers, um be sure to reach out to your account team to learn the latest and greatest about this stuff. There's constantly new partners and data connections, etc. that we're making available. Um, so please reach out and let us know so we can put you with um the people that you want to collaborate with, put you in touch with the people that you want to collaborate with. You can also ask your if you're an existing customer, please make sure that to reach out to your account team to learn the latest and greatest. There's we always have new partners um coming on to data bricks that make things a lot more easier. um and also new data connections using u lakehouse federation. So if there's a particular connection that you would like or a particular partner that you're really interested in leveraging these tools with um please um let them know. Um you can also just you know go ahead try it out yourself. We have docs. Um make sure that if you aren't a meta store admin yourself um that you flag down one to help you with some of this stuff. And if you just are interested in reading more, um, I have some links for that as well down there. Um, and finally, if you want to just have a very low barrier of entry, try try out. Um, there is an interactive demo of the clean room. No account required. Um, you just click on the link and it's a reprise app that will allow you to go through and experience the clean room flow yourself. Um or if you do have an account um or you've created an instance of data bricks express you can use the starter asset in marketplace which is the um demo that I ran you through today. All right. Well, thank you again for your time and hope to see you again soon.