Lightning Talk: Beyond Zero: Eliminating Vulnerabili... Patrick Smyth, Dan Fernandez & Srishti Hegde

Key Takeaways

good afternoon everyone uh thank you for joining us we're going to be talking about Beyond zero uh eliminating vulnerabilities in the pie torch container images this is a an effort that concluded a couple months ago uh that focuses on minimizing uh the bber first uh my name is Dan Fernandez I'm a product manager at a company called chainu guard today I'm joined by Patrick Smith who's a staff uh developer relations engineer at chenard and Tristy he who's uh deliver engineer uh we wanted to start by going over a little bit about why containers are also ideal for AI applications just like they are for other applications and it has to do with a few of their properties such as their portability so you have a consistent development and production environments which makes it easier when you make the transition uh it also offers efficiency because it allows you to scale with the growing demands and lastly it offers uh some isolation not uh full isolation uh and this encapsulation of apps in general uh that is consistent across environments also allows for uh the possibility of Hy workloads which a lot of large organizations are starting to uh or continuing to transition into we also wanted to share some metrics around the overall adoption of containers for AI applications and start with as of the end of 2023 is the latest data on it there was a 58% increase on GPU instance hours usage this was by a report uh from data dog uh and this has to do with as you may imagine the increase in the need for both uh training and inference workloads associated with Gen applications there's also an interesting metric here and this is more of a forecast but while AI components have not made it to every uh Enterprise application even though it it sure feels like it it is estimated by 2025 90% of most Enterprise applications uh will have some AI component within them and lastly the hosting or the spend for cloud resources associated with uh Cloud applications is estimated to be uh $200 billion doll by the end of 2030 and this was by a uh Cloud Revenue estimate that was offered by Goldman Sachs so this kind of gives you an idea of like why is it that we uh decided to focus on this but obviously we're uh uh pytorch conference and we wanted to highlight that really the the important part here is that pytorch has become uh and it has a key role in the AI supply chain and is because it has widespread use for both deploying and developing models the flexibility the strength of the community and the ease of use uh has made it one of the most popular container images across the board and that means that it has now become the foundation to a lot of libraries and projects um and due to the far-reaching scope and use cases for the the this specific technology it now also means that the attack surface for AI applications via pytorch has also increased significantly over time um so any organization any uh Enterprise that is deploying an AI application that is concerned with data privacy uh also now has to be concerned with maintaining all the components in this case container images uh making sure that they're up to date and that's what we're going to focus on the rest of this presentation where Patrick is going to walk us through some of the metrics around vulnerabilities associated with the pytorch image all right thanks Dan yeah so pytorch has been downloaded over 10 million times in the last year um and so you know this is an application that really matters in terms of you know if you secure pytorch the piech container image you're securing literally millions of deployments across the planet so let's dig a little bit into to pytorch in terms of security so um the last build of the pytorch container image had a critical uh five high 40 medium and over 50 low uh cves um and you know you might be like hey that sounds like a lot that's actually not that far off industry standard which is you know maybe even slightly unfortunate um unfortunately cves do really matter um what are cves they are common vulnerabilities and Expos exposures they are known vulnerabilities in software that actually affect the security posture of that software um and cves are you know they can be looked up in a database so these are vulnerabilities that uh can should and in some cases must in the case of for example fed ramp compliance be remediated um so if you're doing Fred ramp you need to fix them within a month and that's that's what you need to do so uh unfortunately there's an upstream problem so if you're um someone who wants to run a model in friend wants to develop an application uh then probably about two maybe 2% of the cves that are in that application that code um that production deployment or might be introd introduced by your team the rest come from Upstream whether it be language run times whether they be uh OS uh and you know as as the the person at the end of that you're responsible for it but how do you fix all of that it becomes very difficult you have to employ cve remediation teams and so on so at chain guard we uh create low to know frequently zero cve images and how do we do that we do a couple of different things um we build fresh uh we patch when needed uh we issue advisories and we really strive for M the images to be as minimal as possible and when you're aiming for zero every uh removed package really matters um because every package is a potential source of CVS that could pop up literally any day um and so in terms of zero well is zero just a marketing thing I mean maybe some of you have used CV scanners maybe you've played around with this um if you're responsible for this you probably you may never have actually seen a scan comeb back zero CVS but but and when I joined chanard I was like is this just marketing but actually you know we work really hard at this um we do it every day and we actually do get to zero cves uh which is kind of remarkable uh and I'm still surprised by it but it's possible um so let's get a little into the nitty-gritty so um just to talk about minimal for ex for a minute um so this is a comparison of the uh back surface of the current P torch image versus the recently built chain guard uh or chain guard P torch image and I'm speaking specifically speaking about the runtime PCH P torch image here so if you look here we have about 75 in the uh chain packages in the chain guard pytorch image versus about 200 in the uh current pytorch image and uh we have about 400 executables versus 1400 in the current pytorch image so I'm going to hand things over to shishy she can go go into a little more detail thank you so uh talking about the minimal set of packages what it really means is a reduction in the image size uh and case in point uh our chain guard spy torch image is actually uh nearly half the size of the Upstream image as you see there uh so talking about how we arrived there uh there are a couple of things so we also ship a prod and a Dev variant of the image and within our prod variant we've rried a lot of things uh that that's uh your shell and development utilities diagnostic tools Network libraries we've also tried to like greatly reduce the complexities introduced by package managers by stripping them down to the bare minimum again but uh not all use cases are proud use cases there are cases where you need to have access to development tools so you can always use the dev variant of our image which has access to a larger uh set of Dev utilities I think one of the biggest challenges we had in arriving at this place is um the very complex uh version Matrix of all the components that together constitute torch uh and as many of you might know most of them are quite tightly coupled and it's down to the tool chain that's actually used to uh build torch uh there's a lot of you know back and forth involved I think uh as of now the Upstream P torch releases uh Cuts in release every 5 weeks or so uh every time a new release um is uh is let out chain guard Builds an image for this particular variant uh in all versions of python that's supported and other libraries uh these images are constantly scanned and patched whenever there's a CV that that shows up and our images are built nightly so really they're fresh as they come uh in addition to zero CV and minimal images we also build uh fips compliant images so a number of you might have a requirement for these images as well uh albe it a very minimal set of packages that we are shipping with our image it's always good to know what's actually running in your system the code that's running in your system and to that end the py toch image that we ship out comes with an s bomb that tells you just that uh the py toch image just like all the other chain guard images is compliant with the salsa standards which means that you get a verifiable history of the build giving you information about how the image was built the dependencies that that went into it the source code and the build system itself so we wanted to provide you with a couple of starting places um so if you're interested in taking a a test drive for the recently built pytorch image the Zer CVA pytorch image um you can take a look at this QR code on the left that's our py image there in our image catalog um another great place to start is the recently released securing the AIML supply chain course which has seven modules they cover all sorts of different things from the comp from the compliance ecosystem tooling um scanning and also some training and inference so using container images it's a great place to start uh similarly uh we do a monthly learning lab uh focused on different secure application Frameworks and run and language runtimes I just did one on our AI images including pytorch and we have if you want to take out the next one coming up next month uh you can scan this QR code so we've discussed a few ways that this is Wolfie or the tiniest octopus in the world you know we go for minimal here so he's going to his hole but um well before we get to questions I'll just say the techniques we've discussed here uh you know from building fresh not going from five weeks to every night uh and including s bombs going minimal these are all things that could be applied to the uh current pytorch image to make it more secure and that could really affect security uh in production environments around the world so thank you all right so I'm told there is a microphone right here in this aisle um and you know you may think that everyone's going to hear you it's actually a pretty big room so you might want to use the microphone um so my question is how do you determine um you know what packages and executables um are not needed in the uh in the image without knowing you know what applications are going to be built on top of that a very short answer is testing but that also sounds like a shy question so we try to uh get a little bit more insight about how exactly the customer is using the application but uh for most part we've come to realize that the strip down version of torch suffices most of the customer needs yeah yeah another way of saying that we change things if customers complain anyone else I think we have time for maybe one more is that okay thank you very much