Private Security Patching Is A Nightmare In Open Source

Key Takeaways

in a completely public project one where everything is open source how do you handle private development as in not blasting out all of your security vulnerabilities out to the public before a patch is available and shipping to the distros this is not something that I or many of you likely spend really that much thinking about but it is incredibly important especially for maintainers like Daniel Stenberg the maintainer of a little-known project called curl yes that curl one relatively common method to handle this is develop the patches in private do not push them out to the public repo at least for a period of time and then discuss with the affected parties on a private mailing list one of the most popular mailing lists to do this this is something called operating system distribution Security contact list and a lot of the major distros are on this list you've got your FreeBSD your netbsd Oracle Solaris along with your Linux distros like Arch Linux Chrome OS Debian Gentoo Microsoft Linux systems group Oracle Red Hat slackware Susa Ubuntu Wind River basically every Upstream that matters and a lot of projects that do their own thing as well this is a very important mailing list if you are on this mailing list and you're not going to be on it unless you maintain a distro and that distro has a reason to be on that list you are expected to be keeping an eye on it to you know keep track of what's actually going on but more importantly when a report is sent into this mailing list it may be requested to be under embargo for up to 14 days preferably shorter in the range of seven or so days however not all vulnerabilities are going to be private you know when it's something big like a heart bleeder Specter and meltdown all of these really big name vulnerabilities that are blasted out to the public sometimes there's going to be a blog post that breaks a story sometimes it's already been patched publicly in those cases obviously they can't go under embargo like you can't just embargo the entire world and there's no point trying to put the distros under embargo when it's already a public story for those vulnerabilities they shouldn't be sent to this mailing list instead they belong in another mailing list they're still very important but that mailing list is the OSS Security mailing list Charter now for the past 12 or so years Daniel has been working with this mailing list sending up vulnerabilities whenever something comes up that is important with curl so far he sent up around about 130 vulnerabilities most of them were low or medium but there are going to be there's occasional high-level vulnerabilities that need to be dealt with basically as quickly as possible however recently he hasn't exactly been happy with how the mailing list operates leading him to write this blog post pre-notification dilemmas so until very recently they had a very different policy about how these vulnerabilities should be handled we recently updated our security processes in the curl project we have noticed that we have previously several times landed fixes to security problems that were defective and in some cases did not even fix the reported problem correctly I believe one reason for this is that we had a policy this is an eternal policy in the Cal project to make the fix into a public pull request no earlier than 48 hours before the pending release 48 hours is enough to make all the tests and see I verified the fix but it is a very short time window for the community to react and to be able to test and find any problems with the fixes before the release goes out so he would go to the mailing list get an embargo it'd be like you know 10 days or something and then two days before everything goes public then the patch would need to be done but a patch that doesn't fix the problem or introduces a whole different problem isn't exactly a great patch and you pretty much want to avoid that whenever you can so to combat this issue curl introduced a completely separate policy for how this should be handled this is documented over on their GitHub in a pull request I'll allow low plus medium issues to be managed through plain PR's update the bug bounty to reflect current reality as an attempt to do better we have tweaked our policy if a reported security problem is deemed to be a severity low or severity medium we will instead allow and rather push for turning the fix into a public pull request much earlier we will however not mention the security aspect of the fix in the Public Communication about the pull request but only talk about the bug fix aspect so don't make it really obvious this is fixing some security problem just say hey we dealt with a bug this will allow us to merge fixes earlier in the release cycle to give the bug fixes more time to mature and ripe in the repository before the pending release it should also increase the chances that we can do follow-up fixes and truly make it a good correction by the time we do the next release hopefully it leads to better releases with further regressions but this approach isn't without its risks of course the risk with this is that a malicious user somewhere finds out about a vulnerability this way earlier than 48 hours before a release and therefore gets an extended time window to perform nefarious actions that is why we also limit this method to severity low and medium issues as the ones rated more serious are deemed too dangerous to risk and that makes complete sense to me if you have a known high or extreme severity issue that you know could take over every server in the world you don't exactly want to make that public before everybody that needs to patch it has the ability ability to patch it so with those things where it's like okay maybe this causes like a overflow in this case but it doesn't really cause anything dangerous okay maybe we should probably fix that it doesn't really matter if someone finds out but anything more than that maybe it's a big deal but those smaller issues where maybe it's exploitable in this really weird circumstance but not really doable in the real world it doesn't really matter too much if someone happens to know about that exploit yeah maybe a bit of damage is going to be done but it probably makes way more sense that any patch that's going to address that works and the problem just goes away and with the release of curl 8.0.0 he implemented this new method but he still felt like it was a good idea to make sure the mailing list was well aware of these vulnerabilities to make sure they can go and Patch them as the patch is coming out I emailed the destroys mailing list again like I've done so many times before and told them about the upcoming six vulnerabilities we're about to reveal 2D world this time turned out to be different because of our updated policy where the fixes were already committed in a public git repository they destroys mailing this policy says that if there is a public commit they consider the issue to be public and thus they refuse to accept any embargo which at least at the face of it makes perfect sense you can't embargo something that has already been made public and anyone who seriously malicious will probably be checking out the commit diffs and will notice that something is out of the ordinary and over in their policy it specifically States this right here please note that in case a fix for an issue is already in publicly accessible source code repository we generally consider the issue public and thus you should post the OSS Security right away not report the issue to Linux distro's this mailing list as we'd merely redirect you to OSS Security anyway and insist that you make the required posting ASAP there can be occasional exceptions to this this is what he's trying to do such as if the publicly accessible fix doesn't look like it's for a security issue and not revealing this publicly right away is somehow deemed desirable in particular we Grant such exceptions to Linux kernel issues concurrently or very recently handled by the Linux Kernel Security team in all other cases you'd have to have very sound reasoning to claim an exception like this and be prepared to lose your argument and if so to post to OSS Security ASAP anyway so he decided to shoot his shot anyway and see what happens what they call embargo I of course call heads up time I argue that while the fixes are public the actual vulnerabilities and the security issues those fixes Rectify are not as in nobody's written a blog post about them nobody's talking about them they are still only contained within this repo there's no public CV or anything like that the only way you would know about the security issue is if you go and read the repo it takes a serious effort and pretty good insights to just detect that one or more of the commits for the pending release are done because of a security problem and then even more so if you want to convert that suspicion into an actual attack Vector now considering this blog post exists I probably don't need to tell you um how it actually went down so basically they were like yeah uh we don't care they maintain that while they could make an exception for me slash us this time this is an exception and their policy says this is not acceptable for embargoes if we make commits public before telling distros we may not ask for an embargo and he doesn't mention this in the blog post but they probably told him to just go and post on OSS Security instead he has a uh a different idea he still feels like he's in the right and they should place this under embargo obviously until someone else goes and discovers it so instead of doing the open source security mailing list he says I thought we were doing this for their benefit I was under the impression that we actually helped Distributors of open source operating systems by telling them ahead of time what was going to ship very soon that they might want to get a head start on so their users stay protected I've been told in very clear terms that they do not want to be notified about vulnerabilities ahead of time if the commits are public I have informed them that I will not tell them anymore until they change their minds because I think our opt-in security process can make our releases better and I think improving curl and making better releases is more important than telling distros ahead of time I cannot understand how this stubbornness makes anything better for them for me it takes away some amount of work so I will manage just fine for Kell users in the wild this will probably mean that they will get security patch kill releases from the distros a little slower in the future we rarely see Carol vulnerabilities rated High higher than medium so this means we'll effectively stop emailing distros about pending flaws we are still allowed to tell them about critically scored vulnerabilities but I must confess I feel less inclined to do that than I used to I think this is a really interesting case because the mailing list is pretty much in the right here and from their perspective it makes perfect sense why they wouldn't want to place this under embargo but I can also understand the dev's perspective where this patch while it is available in a repo it's been named in a way that unless you know it was for the security vulnerability that you don't even know about in the first place you probably wouldn't know that it has to do anything with security anyway and that's ignoring the fact that if someone does Discover it there's no reason to keep the Embargo around anyway so the Embargo can then be lifted do you feel like this is one developer stepping out of line trying to do something that doesn't make any sense in this system or do you think a bureaucratic system is holding back something that otherwise would be incredibly important I would love to know your thoughts so if you like this video go and like the video and if you really like the video and you want to become a one over these amazing people over here check out the patreon scrub solid Bearer pay Link in the description down below that's gonna be it for me and wget is better [Music] [Applause] foreign [Music]