Fine-Grained Access Control for Unstructured Data With Volume Permissions

Key Takeaways

My name is Adrian. I'm a software engineer at Datab Bricks. Um, and today I was supposed to give this talk together with my colleague Leanne and uh talk about um fine grain access control for unstructured data. Unfortunately, Leanne is not feeling well, so she's not going to be able to make it and I have not prepared for presenting the slides. We were supposed to do a part with with slides and then a demo part. I was mainly here for the demo, but I'm going to have to improvise a bit. Um, so let me just get straight into it. Um, UC volumes are the data brick solution for managing and governing unstructured data. And uh the main challenge we have uh today is that um the when it comes to governance the permissions that you can define on volumes are relatively coarse grain. You can just decide who gets access to to the volume in its entirety and uh that's works very well for the typical cases. But in more advanced use cases, um people run into uh into challenges with regards to their data setup and u the lack of flexibility that the current model offers. So right now we want to announce an upcoming uh preview for a feature called fine grain access controls uh where we're going to try to make the situation better. So um hopefully my demo is going to be self-explanatory and uh yeah happy to take questions at the end. Now um for this demo let us um pretend for a minute that Lean and I are um not datab bricks employees but rather um datab bricks users as part of a large pharma company that's uh trying to develop and uh put out uh new treatment for breast cancer. uh say we're running there a so-called drug development program or DDP for short and as part of that there's a bunch of u work streams going on or so-called studies which may be of different types preclinical clinical uh translational etc. Now let's assume Leanne is the data administrator and she has organized things such that we have one UC volume defined for each of these studies um and with a purpose of managing there um all of the unstructured data related to this study all in one place. Um and in in this example of ours uh this uh unstructured data would be typically clinical trial data for breast cancer patients. Uh and more specifically it'll be um breast uh breast MRI scans um stored in this specialized file format called DICOM uh which is the industry standard for managing u uh medical images. Now for ease of use on data bricks we also um extract out um just regular PNG files uh out of these uh diccom files and we store them uh in the same volume alongside the diccoms. Now one more note I should make uh before moving on is that um clinical study typically progresses through a sequence of four phases and that's why you see this uh this structure here with a with a phase one two three under study X that is because our study has reached um phase three which happens to be the most involved and costly u phase where human large scale human trials are run and extensive analysis are performed in order to demonstrate the drug's clinical benefit. This is the last step before pursuing um regulatory approval from bodies like the FDA or EMA. Now finally this is where I come in as one of the pharmacologists or researchers in the group who is assigned to work on one of these phase three trials namely this one here uh called labeled trial 3B and uh my job concretely is to process these diccom files and uh extract those uh regular images out of them. Now starting out I actually don't even have access to this volume. So if I try to list it like this I'm going to get this permission denied error. What can I do? I'm going to ask Lean to grant me access to it. So for that she would be typing this uh following grant statement. grant read volume on volume study x to adrian databicks.com. Um so in case you've ever used datab bricks before this is probably familiar to you that's because it's just the typical way of granting privileges to UC entities. However, uh as is um Lean is not going to run it because this would open up the entire volume for me and that may go against regulatory uh requirements and in general it will violate the principle of list privilege. Instead, what Lean wants to do is um to restrict my access to this particular trial directory and maybe also the the trials under phase two because I may need to pick at those in order to do my work properly. So fortunately for that all she needs to do now with this upcoming feature is to add this with paths clause with paths and then list here the exact paths that she wants to grant me access to. So phase two and phase three trial 3B. Um so there we go. Uh easy as that. no more need for um workarounds such as copying this data out to separate volumes in order to manage permissions there. That would have been the the old way of of doing it. So now let me try listing again. And indeed this time I uh I'm getting access to this uh to these directories that I need to be working on. And for example, if I go into phase three, I'm going to see my uh trial 3B here. However, notice how um other directories in the volume that I did not get access to are not showing up in my listing. So, in particular, this trial 3A here is not showing up. And even if I try to list into it explicitly like so um that's still going to fail with a permission denied error. And similarly if I try to access the raw data in there and so on it's all going to fail and that just confirms that the access granted to me was a fine grain was granular. Um okay so now let me get back to the work I need to do which is to process those uh diccom files in my trial. uh for that I can uh just simply invoke this um library function that is available because I've set things up beforehand and uh all I need to do is to specify a trial directory um and I'm just going to copy paste this one and then output directory which is going to be uh exactly this one that was uh suggested which is a subdirectory inside the tra same trial Now if I try to run that um it is going to fail and that's expected because indeed I am lacking right volume permission. Uh all that lean granted me was read as you can see here. So now again I'm going to ask her to to help me out and what she can do is to run a similar grant statement but then update the read here to write and then u yeah she can specify the path that I should get access to and here she can be very specific she can u grant me permission to only this images directory and this way she'll have some peace of mind that in case I make a mistake uh I will not be able to for example delete those dycom files. Okay, so let's say she runs that and grants me access. And now if I try to run this again, hopefully it's going to work. And yeah, it looks like it will. Uh it just may take a moment. So in the meantime, let me prepare um showing you how one one of these images looks like. Um okay, there we go. So the processing is complete. the files were generated. If I refresh here in the UI, I should see them there as well over here. [Music] And now if I u am to copy this, copy paste this first one and display it, you're going to see how a typical breast MRI looks like. Um, all right. I'm almost done, but just one more thing to show before I wrap up. Uh, which is um how Lean would go about revoking my uh right access as soon as I'm done with the processing. Uh, because it's important to ensure the principle of list privilege as I say. Um so for that she can simply type a revoke statement like this revoke paths and then here she needs to specify the same path as before uh for write volume on volume stud x from adrian databicks.com. All right. Uh so now she has revoked she she revoked right access to me and to confirm that um first I can try displaying this image again and it is going to work. But then if I try to for example delete it like this with FSRM that is now going to fail with a permission denied error. So that concludes my demo. That's that's all I have. Um thank you so much. I'm uh happy to stick around and answer any questions that you might have for us. Uh maybe I should say a few words though from the from the presentation uh what's there at the end. Um I'm just skipping through the slides now and this is where the demo would have been and now after the demo. Yeah. Uh so as I said in the beginning this is this feature is um approaching private preview. We're now um collecting uh like subscriptions for this private preview. If you're interested, please uh scan this QR code um and fill in the details there. I will uh keep it uh on the screen for a moment. And um yeah, there's a bunch of exciting stuff that would uh come afterwards, which is to uh allow for more flexible um grand clauses uh with maybe globe paths and and maybe tags support and uh also we're considering adding new privileges like browse only uh capability. But uh yeah, please uh engage with us. Let us know what what you're interested in and we're happy to collect your feedback. Thanks again.