Unity Catalog: Setting up Catalogs and Workspace Binding

Key Takeaways

Hey everyone, my name is Shre and I am a scale solutions engineer here at data bicks. Today we are going to talk about unity catalog and setting up cataloges with workspace binding. So with unity catalog we have a three-level name space. At the top at the data bricks account level will live the unity catalog metas store which is a cloud regional concept. So you can have one metas store per cloud region. Underneath the metas store you will have your cataloges. Cataloges are a logical separation of data. Underneath the catalog you will have our schemas or databases. And then within the schemas you will have your actual data assets. These could be tables, views or any other assets that could go underneath the schema level. And in order to actually query your assets, you will have to use the three-le name space which will be the catalog. Schema dot the data asset. With Unity catalog, you are also able to restrict data by workspace. So if you have multiple workspaces that are underneath the same Unity catalog meta store, you can bind your cataloges so that they're only tied to specific workspaces. This is a common practice that we see amongst many customers in which they have multiple workspaces broken down by software development life cycle. So for instance, if you have a dev workspace and a production workspace, it's common practice to tie your dev catalog to your dev workspace and your prod catalog to your prod workspace. But there might be cases where you want to make your prod data available as read only in your dev environment in which case you can achieve this using workspace catalog binding. And we will see this in the demo as well. Now if we jump over into our data bricks workspaces, I am going to showcase how we can set up our cataloges with workspace catalog binding. So here I am in my dev environment. Within my dev workspace, I have set up a dev catalog. And this dev catalog is tied to an external location which I have also set up which in turn is tied to an S3 bucket. So within my dev catalog, I have two schemas, the default and information schema. If I navigate over to the details tab, we can take a look at the storage location for this dev catalog. And if you take a look here, the storage catalog location is my S3 bucket, which I have set up already. And in order to set up the external location and storage credentials, you can follow our public documentation to see how you can set those up. If I navigate over to the permissions tab, this is where I can delegate permissions at the catalog level. It's important to remember that Unity catalog permissions follow a tree hierarchy. So if you give broad range of permissions at the most top level, which is the catalog, all those permissions will inherit down into the schema and data asset level as well. Now if I take a look at my workspaces tab over here, you will see that the default all workspaces have access has been turned off. Instead, I have bound this catalog to this dev uh workspace. So if you take a look here, my dev catalog is only read and write access level to my actual dev workspace. Now I will navigate over to my production workspace and we will create a prod catalog. So here I am in my prod workspace and if I navigate over to the external data tab at the top we see that I have created a external location which is referencing my prod S3 bucket and that is known as the prod catalog external location right here. Now if I go back to catalog I will create my production catalog. So first I will click the plus sign click on create a catalog and I will give my catalog a name. So I will call this prod catalog. For type I will choose standard um and I'll uncheck use default storage and for external location I will choose my prod catalog e. I will click create and what it will do is create my prod catalog. Now, now a catalog has been created. I will click on view catalog and we can see that our prod catalog has been created. If I navigate over to details, we will see that the storage location for this catalog is my S3 bucket which is tied to my S3 prod catalog bucket. If I go over to permissions, here is where I can delegate permissions to my users and groups. And now, if I navigate over to workspaces, we'll see that the default behavior is that all workspaces underneath this Unity catalog meta store will have access. I don't want this behavior. So I will uncheck this and I will specifically grant workspaces access. I will click on assign to workspaces and these are the two workspaces that I have underneath my UC meta store. I will choose both of them but I am going to update the permissions for my dev workspace. I'll click on assign. Cool. So now in order to actually manage the access level for my dev workspace, I'm going to click on the demo environment dev workspace. Click on the manage access level and I'm going to change my access to readon so that this catalog will only be read only in my dev workspace. If I click apply those permissions will take place. Great. So now if I navigate back over to my dev workspace and I hit refresh we should see that the prod catalog shows up in my dev workspace which it does but this will be only read read only. So if I go to the workspaces tab we'll see that this catalog is read only at this workspace level. And if I try to create any tables underneath any of the schemas, it won't let me because this is readon. Awesome. So you can see that the catalog has been designated as read only in this workspace. So I won't be able to actually write anything to this catalog from this workspace. So with that, we have successfully set up our Unity cataloges in our both of our dev and our production workspace and also bound them to the respective workspaces. So this is how you can achieve this using AWS data bricks and unity catalog. Thanks for watching.