Automating Container Patching with Copacetic and GitHub Actions

Key Takeaways

So Copathetic is a tool that is able to do uh real-time patching. So without rebuilding your container image, it will be able to patch it. Hello everyone. I'm Joeru cloud advocates team at Microsoft. We are back here on the open at Microsoft show with Josh. Hello Josh. Hello. It's nice to have you here. you uh we work on the same team at Microsoft and we are going to talk about um automating uh container patching with Copathetic and GitHub actions. Mhm. Uh I think we're bringing back Copaetic to the show. Can you tell us what's Copacetic and how that can help someone on the cloud matrix space? Absolutely. Absolutely. So my favorite first question is a kind of a preface to this though which is so you hear a copacetic and then you hear that it patches container images uh and then some people need a little bit more context to well why would I patch a container image you know why don't I just update the base image and then have to not have to worry about it and the reason you would do that is to reduce CVEes um and that was kind of my dream you know maybe 10 years ago when containers came on the scene where hey there's this immutable thing I don't have to patch it anymore But now we have these long r long running workloads and we need to patch them. There's containers out there running for for months years and they are they're vulnerable and so that's where copacetic came along. So copacetic is a tool that is able to do uh real-time patching. So without rebuilding your container image, it will be able to patch it. And so what I mean specifically by patch is right now it only handles OS level vulnerabilities. Let's say you've got some kind of package that your application's depending on in your container. Copacetic will be able to patch that for you by using trivia results. So trivia would be a scanner that say, "Hey, your container image has these four CVES or 46 CVES, whatever the case may be, and they need to be remediated." So instead of having to go through the whole pipeline of rebuilding the image, Copaetic can be run on the container image and patch this just those specific vulnerabilities or kind of all the packages on there if you want to. We could talk about that more later. Um, but that's that's the gist of what Copathetic is able to do. It's able to take a an existing container image and apply fixed or non or targeted or non-targeted patches to the image so that way your container images aren't vulnerable or aren't as vulnerable. So that's kind of the the gist of what Copacetic does. Yeah. And I I love that thing that we don't have to have the whole application source code to do that. We can do that complete separate process without having access to what was done before was deployed there because we're just patching the dependencies the operating system not the uh the application itself. That's correct. Absolutely. Yeah. I mean that's kind of the two big scenarios that come up in my mind are well what if you don't have the Docker file anymore or you can't rebuild the container image and then the other one would be um what if the container image is out out of your control? What if you're taking a dependency on another base image and it's up to a third party, someone outside your company to update that image? You're you're kind of dead in the water. And so there is I I do agree with chain guard like you should have really So they had like a little advertisement for the last conference I think that said uh you know like patch patch the current or patch the present with a cross through it and then it said uh um build the future. And I don't disagree. I do think you should have really strong low CVE or no CVE images. However, there's just certain scenarios where that's not possible and so patching is still a very valid use case uh for a lot of workloads. But yeah, that's the general gist of Copathetics to patch your images without having to rebuild them. Yep. So, let's share your screen. Yep. And then we we can go straight to showing I think you have a GitHub hippo with example how to do that with GitHub actions that nice. Yep. Yeah. So here um here on the screen this this is a this is kind of a little solution I've been working on. There are some other projects that are are working on building something similar, but I wanted to take the existing copacetic action and then use uh GitHub workflows to create a continuous patching workflow. So the world that I want you to envision is okay, I have all these containers on my registry. I'm on a platform team and I'm responsible for keeping them up to date. There's a lot of CVEs in these images and it's really hard to wrangle all the developers to get to update their images with inside their pipelines. So what's my other option? So this is one other option. So this continuous patching workflow um we'll kind of just walk through we'll walk through the workflow so you get an idea of it and if it helps I can switch to a matrix or a flowchart. Um but it all starts with this the general dream and vision that I want to set for you is we have images on a registry and we want to patch them continuously on some kind of schedule and cron is arguably the best way to do that. I have that commented out here for the sake of demo. I'm going to use a a dispatch where I'm just going to manually trigger it. Um, but we're going to have a schedule. So, every day at 4:45 a.m., we're going to go and patch all of our containers on a registry. So, that's what this workflow does here. So, it starts with this cron trigger um when it's not commented out, and it'll go through and trigger that. So, then there's two steps in this workflow that I want you to put in your mind first before we look at the code so you don't get lost. The first one's going to be the setup, which is okay, I have a registry. How do I know what tags of a container image I need to patch? And we'll talk about an incremental tagging strategy here in a second. But the first step is just get me the images from the registry that I need to patch and give them to me in a matrix uh a JSON matrix so that way I can ingest it in. Um so that's what this first step does here. So, if you look at this, um, it's going to check out the GitHub repository and then it's going to log into Docker. And I'm using this because this little CLI tool that I'm about to talk about uses the credentials from Docker to authenticate to GitHub or GHCR, the container registry for for my GitHub user. And then I I created um I just can't help myself these days. I just I'm constantly creating CLI tools. And uh so contiguous because it's going to get container tags. So, I thought I don't know. I thought that was clever. You can correct me if I'm wrong. Um, it's a little CLI that logs into your registry and then pulls down all the images and then it has a little bit of logic and massaging in there to determine what versions of the image or tags in that image should be returned back. And then it essentially just gives you a list and we'll look at this in the output in a second. Um, but it'll echo out the images. And so what it'll do is it'll say, "Hey, this container image is viable to be patched." And then it generates what that next patch tag is. And that you'll see that in a second where that's the incremental patching strategy where it's still semantically versioned or semantically correct as far as versioning is concerned. It just adds a hyphen n like hyphen 1 hyphen 2 at the end of it. Um, and so that's the setup like get all my images that I need to patch and get them into a JSON file. And that's important for the next step. So the next step is the patch or job in in GitHub workflow uh nomenclature syntax. And patch does a number of things. The first thing that it does is it generates a matrix. It generates a matrix from those images. And just let me give you a quick snapshot of what that looks like. So if we just go to a previous run, it's just way easier to see this visually. So I had nine jobs that run. So what a matrix does if you're not familiar in in GitHub workflows is it'll basically populate the data that you need to run for all the jobs. It creates a loop for you. So what I'm doing is I'm saying patch is now going to loop through for every image that I found in my registry that was returned from the contiguous um setup task prior. And so if we expand this we'll see we'll see all the images that I have in my GHCR. So, I've got some Alpine, the contiguous action itself is in here, uh, because it's under my GitHub repo right now. And then I've got Rust and some other ones that are relevant to the demo app that I'm going to show you. I, uh, I actually stole, uh, Paul you, so he's a co-orker also with us, George. Um, and he he rewrote the Azure voting app in Rust. And so, I just I stole that code and I made it used it for my demo app, but it relies on Debian and Rust container images. So, that's why they're there. But that's the matrix. So let's go back now and look at what happens in the patch step and this is these are important steps here. So the very first step inside this is it's going to generate a trivia report. So I mentioned at the beginning of the call t um copaceta can either patch all the packages on your container to latest or it can do targeted patching and it does that by taking trivia results the scan results that have vulnerable CVEes and then using that as a way to say okay well that package has an update available I'm going to go and get the next version. Um so we need a trivial report to be able to do that targeted patching and that's what this does. So, it's going to format it uh as a JSON and it's going to um output it as report.json. So, this is going to for every image in my registry, it's going to run trivy generate me a report. Uh and again, I'm filtering by only OS level vulnerabilities because copacetic currently can only patch OS level vulnerabilities. It's not going to do anything like say I'm writing a Go application. It's not going to go in my go mod file and do anything about those different um vulnerabilities that might exist there and the dependencies of the code itself. I believe that's on the road map u but you can check out the copacetic project in the road map to be sure of the open issues. Um, I know it's been discussed, but anyway. So, it's going to scan and then we're going to do uh a vulnerability count. And this is important just so we're not rerunning things in our workflow unnecessarily because right now in our workflow, if we don't have any vulnerabilities, there's no reason to attempt to patch because there won't be anything. So, we check the vulnerability count. So, I have a little uh bash magic there. bashfu there that will set a variable of vone count to the github um outputs and then I'm going to run copacetic but only so I have this condition here only if the vulnerability count is not zero so if we have some vulnerabilities on our report we'll continue with the workflow otherwise we'll just stop there's no reason to continue and waste compute resources if there are no vulnerabilities so this is the copacetic action Um, this is a a GitHub action for the Copathetic CLI. So, Copacetic, the CLI you can run just in your terminal. You can download it. I believe there's even a brew install for it. Um, but we want to automate this process. I don't want to have to download a download a JSON file and then manually go through and update these. I want this to be fully automated. So, that's why I use the GitHub action. And this will go through and patch the container image. Sorry about that window keep popping up. I'll stop clicking on that. Um, it'll go ahead and patch the image. So, it takes tripport, patches it, and here's the important piece. Copacetic allows you to name what the new container image will be. Again, it didn't rebuild the whole image. It's just pass patching the existing one, but it does generate a new image for you that you need to uh tag and then push. So, that contiguous action is going to determine what the new version is. So, let's take a look at that real quick. So, let's just head over to my GHCR packages and we'll take a look at the the net monitor because it's been patched a few times. So, if we look at this um don't mind the duplicate patch or tags here, but anyway, I started with zero uh one zero and 011. And then you'll notice that there's this hyphen one here. So, this is the first instance of that version being patched. when it gets patched again, it'll bump to two and three and so forth. But this is still semantically correct. And that's very important for later on in the demo where I'll show how to use dependabot to submit PRs into the repos to update the Docker files with patch tags. So still semantically correct, but that's how it works. And that contiguous is generating that for you. It's going to say, "Oh, well here's your base tag 010. Here's what your next patch is going to be." So if my base tag was dash one, next take will be two. It's it's sounds super simple but in the coding logic it was actually a little difficult uh because all the different varants of tagging uh tagging standards that we have out there in the world. U but that's what that looks like and that is a value being passed in here and then that container image is going to be local to the GitHub runner. So we need to push that up to our registry. So we log in to GHCR again because they're two separate uh jobs that can't share that login and then we push it. And so like that's pretty much it from the continuous patching perspective. Now we have this job running and it'll pull all the images and rotate through. Um the only thing that you might need to do differently if you're not using GHCR is swap out this contiguous for anything. You could write a shell script, you could write a PowerShell script. Um, you can write your own CLI. I am working on support for ACR and other registries, but right now it's just a GHDR because it's a more more of a demo or a prototype so to speak. So that that's the continuous patching workflow. So that gets us to the point where we now have a registry and we know that every image on that registry has been patched within the last 24 hours, which is super cool. But that doesn't fully solve our problem. The remainder of the problem is the Docker images and the manifests that exist in our code repos that reference the unpatched tag still exist and they need to be updated and so that's where dependabot comes in just before we go there Josh. Yeah. Yeah. Let me do like a you know recap here means that getting your container history I'm looking all the image that I have there all the image with like any CVS and then I'm going to patch all those image and then push back to my history and now I have a new tag for those image that change and I want you to complete now and show the last step that you going to talk about that's how you going to get those new image And now patch the running applications to make sure that they are the latest uh container image version. How is there? Let's share your screen again and show that for sure. Yeah, let's take a look. So uh the easy way is to to abuse dependabot again. So if we look at um let's just look at this demo repo. I forked it from dependabot and it's really really simple. So whatever repository that the Docker file or Kubernetes manifest that has the outdated image that needs a patched version, we can use Dependabot's version checker to be able to automate that and do PRs into the repos that are using unpatched images. So if we go into GitHub and Dependabot, it's really this 16 lines of code. We're going to set up uh version updates for Dependabot. In this case, it's looking for a Docker file. You have to change it for Kubernetes manifest. It's my current directory. That's where my Docker file is at the root. Uh, and then this is just extra stuff here because I'm I'm logging into GHCR. Even though these are all public images, I still need to treat it as a private repository for dependabot. So, this does require a token. But if you add this dependabot and you give it a token in your GitHub settings, now you're golden. And if you look up here, I've got two PRs now from my previous run of continuous patching from yesterday that I need to bump up the versions that this particular image is using. And so that's how you go full cycle. You set up a project that goes and patches all the container images in a registry and then you can use dependabot with your projects to be able to notify your development teams when there's a new patch version available. That's great. then all the process will be getting updated using that dependent boat and that's the goal. Yeah. Let's go do a call for actions uh for anyone that want to be a contributor like yourself for the project. We have here project copacetic copacet on github and so look for the issues normally you know a first issue that's very good for you to help the project and yeah search for for something that be first shoe um a good first shoe and and try to help the project maybe documentation is something for you and watch the previous video here on the show we're going to leave the the video description on the uh more details on the video description that you can you can follow and follow the channel. Subscribe to the to the the Microsoft developer channel and follow the opening at Microsoft show. Thanks Josh again for for coming and showing that demo for us and see you all next time. Thank Josh.