Fedora Linux Finally Kills Off Delta RPM

Key Takeaways

as I look more and more into Fedora I discover a never-ending stream of really neat additions added to the distro over the years one of those additions is being considered for removal before I'd ever really heard of it that being Delta RPMs this was proposed by the Fedora project leader Matthew Miller proposal drop Delta RPMs for real this time the for real this time is very important this is not the first time this has been discussed all the way back in 2017 this issue was made all the drpms Delta RPMs not synced as noted in this post it seems we are only syncing out the drpms made in that specific compose which keep a ideally configurable amount of older ones around I think for Rawhide we were keeping two weeks we could start with that as it is now on less someone updates on the day an update is pushed they won't get the advantage of a drpm basically as the new drpms were coming out they were ditching the old ones so you couldn't actually use them and someone mentioned hey uh I half wonder if we should just stop generating drpms entirely and then following this in 2021 this was made Delta RPM usefulness Heil I think Delta RPM is not really useful anymore and goes over all the reasons why they simply don't really matter this discussion didn't really go anywhere and sort of just hit a dead end and just ended that's not really that uncommon with a lot of projects but now it's being considered again but hold up a second what is a deltra RPM why would anybody care about them and why did they exist in the first place okay hopefully we all know what an RPM is in the context of Fedora RPM is the packaging format used by the DNS package manager and by extension used by yum this is the packaging format used on Fedora Linux and various other distros out there but Fedora Linux is the only thing we care about today now a Delta RPM sometimes just called a drpm basically builds off of that concept with one really major difference so when you update a regular RPM like updating any regular package on pretty much any other distro whether that's Ubuntu Arch Linux void Linux anything else out there the old package and all of its content are going to be deleted and replaced with the new package and all of its contents this is the easiest way to handle an update no fancy computation just get rid of the old and bring in the new this is not the way a Delta RPM Works instead what it does is downloads the difference between the current version and the new version and then you take the current RPM and the update difference and bring them together reconstructing the full updated RPM locally on your system now as for the name Delta the reason for this is different is represented in mathematics as the Greek symbol DOTA this does have some drawbacks but it also has some pretty major benefits let's say we have an application that is 200 Megs in size and has an update now with a regular RPM you can't do anything besides downloading the entire up updated application downloading the entire 200 Megs but with a Delta RPM you only need to download what is changed this might be like three or four megabytes this can be incredibly useful for those that have a slower internet connection however there is a reason why you don't see every District out there doing this nothing in the world is free so by having to reconstruct that package locally on your system this has a much greater cost on your CPU and your i o in many cases it might not actually work out to be quicker now even though we're talking about Fedora Delta RPMs didn't actually start there they actually started over on the Susa side in 2005 and then made their way over to Fedora in 2009 this was a time especially the 2005 date when even in really developed countries like the US Germany UK Australia especially Australia it wasn't uncommon to have absolutely awful Network speeds in 2005 I think I had a speed of maybe one down possibly maybe a little bit less at that time you could legitimately save yourself a lot of time sacrificing a little bit of CPU time to not have to download a couple hundred megabytes on a one down connection but over the years procedures started to slip the issue pointed out in 2017 is they weren't actually keeping the Delta RPMs around long enough so people could actually download them now the difference between having a full RPM and a Delta RPM is with a full RPM you can wait six months skip 20 updates and update no issue whatsoever with a Delta RPM because you're applying the update and trying to reconstruct the package you need to download each of those individual updates and if those individual updates are no longer in the repo you can't build the fully updated RPM and then in the 2021 post another issue was pointed out there are very few drpm files in the repository see for example now at this point all of this has already been deleted but the problem is still happening today this is every single drpm are currently available it looks like a lot of packages but this isn't everything this is not everything that should have a drpm if drpm is covering the entire distro so this leads to a bit of a problem this leads to um now the problem getting worse this is an example from the 2021 post this person installed four packages upgraded 161 and removed three the total download size was 751 there were three drpms out of all of these 3D RPMs the Delta RPMs reduced the download size from 750.9 megabytes to 747.5 0.4 saved and this seems to be the case for most of the people somewhere between 0 and 2 ish percent sometimes I have seen up to about five percent but that's pretty much as far as it goes and this is a more recent example the last update just now on three different machines gave me this is a couple of days ago Delta RPMs reduce 284.9 megabytes of updates to 281 1.4 saved 14.3 megabytes of updates to 3.3 76.9 saved that does sound like a lot but keep in mind downloading an extra 10 megabytes would maybe take I don't know a couple of seconds more and then the third had no Delta RPMs whatsoever this and this are very typical results now here's the fun thing about missing Delta RPM updates so if things are being deleted before you can download the entire chain you can't use the Delta RPMs so you're just going to download the regular RPM anyway so even if there was a Delta RPM for one of the things you're going to update it's very likely that you weren't gonna use it anyway this also happens if the Delta RPM fails constructing the full RPM it will just go and download the regular RPM and be good to go at question then becomes what do you do do you go all back in on Delta RPMs and make them legitimately useful or do you ditch them completely and then focus on other things instead well Matthew didn't even consider the first thing so second option it is then I was asked to weigh in on this right here as a priority last we talked about this we didn't really get anywhere that was this one here and that ticket hasn't moved because fixing it isn't trivial what we're doing now as has been the case for several years already noted in the previous discussion has very little end user value providing a Delta RPMs that stick around for a short amount of time for a tiny amount of applications that basically don't do anything besides slowing down your system also as noted in that thread as in the ticket that's unfortunate because it did bring some real benefits and could possibly do even more but I think it's time to move on we have os3 and various container Delta approaches we should swive on those and give Delta RPMs a sad fond farewell but even though Matthew is the project leader he realizes that getting rid of them at this like very second might be a bit of a problem so my normal response would be well I missed the Fedora 38 deadline by a wide margin so Fedora 39 plus uh Fedora 38 I believe is supposed to come out in April the beta should be in a couple of days correct me if I'm wrong but I think we could stop producing Delta RPMs for current releases Without Really affecting those releases there would just not be more created which as previously explored would not really have a strong effect so I wouldn't leave the other options out of the question so hey infrastructure people what do you think how easy is it to shut this off if shut off can it be turned back on again in case of regrets uh once shutoff is decommissioning infrastructure around a project or just more shutting it off what risks are there and how much would be saved in compute resources storage delays ongoing maintenance work and other and Kevin fenzie of the Fedora infrastructure team chimes in to answer these concerns and uh it turns out they'd already half decommissioned it anyway just FYI we do not produce drpms for Rawhide Branch at all since 2017-ish so um that's a big part of the reason why it was really useless so shutting it off is just one line in a config and then turning it back on is just turning false in the true as far as I can think nothing else needs to be done they will just disappear what risks are there none uh what would this save some but not a lot as we only guilt against the previous update composers and thus don't do too many storage hundreds of megabytes lots of storage uh anything else nothing comes to mind so it seems like getting rid of this is a pretty seamless thing maybe it would take like an hour possibly and then turn it back on possibly about the same but even though as it stands it's not really that useful not everyone is in favor of the decommissioning this person said dislike minus one because everyone else was saying like plus one which is not really a coherent argument but hey don't throw it in there there is a much better argument from Neil gomba saying our tooling has been broken for a long time and contributions to that tooling is just not going to happen since nobody can run this stuff outside of Fedora infrastructure it's a sad State of Affairs indeed please don't try to equate those things to Delta RPMs unless you're trying to equate their General uselessness OS tree and container Delta stuff is not generally useful or applicable for Fedora users and they won't be for a very very long time and they might never be useful because os3's approach requires us to use OS tree remotes which we're killing for oci remotes and there's no standard for container Delta since the Baseline oci format isn't amenable to Delta fetching but what do you think do you even use Delta RPM do you even know if you have them enabled in your dnf config it's very possible you don't it's very possible you've had them existing all this time and never even remotely touched them but even if they are enabled it's possible you've never seen one so let me know down below and if you don't happen to use Fedora um let me know if Delta packages on another distro would be cool I would love to know so if you like this video I'm gonna go and like the video and if you really like the video and you want to become one of these amazing people over here go check out my patreon scrub Stanley therapy linked down below that's going gonna be it for me and I'm out [Applause] [Music]