ByteDance makes Linux kernel reboots faster

Key Takeaways

the company behind tech talk bindance proposed few fixes to the linux kernel to improve the speed on which the linux kernel actually reboots i find this fascinating well while this is not really relevant for you on me and uh you know normal engineers it is quite significant to large companies such as ticktock google scale or you know amazon scale you know and so shaving few milli hundred milliseconds can make big big performance to their slas so we've seen this with google like few months back where they improved their linux kernel to reboot faster when you have large amount of ssds because you see the the previous api was synchronous that means you call in an ssd reboot and then you once that reboot go to the next one and reboot and i believe if i'm not mistaken each ssd takes like five seconds to shut down so that really adds up so they moved it to a asynchronous api which obviously significantly improved the reboot process but now we're seeing the same thing with tick tock slightly different but i don't i wanted to discuss this how about jump into it so this comes from phoronx how about we read a blurb and then discuss bindance working to make it faster que exec booting the linux kernel binance as a chinese company behind tick tock has been working on a number of linux kernel optimization a few years and their recent work is for faster k execute rebooting the kernel so if you you might say it's like what is that and that is exactly what the question i have like what is k execute because you see i don't work with the linux kernel and i don't develop that kind of linux kernel but here here is one problem you know when it comes to kernel development and kernel updating like if you want to if you have a linux operating system and you want to update its kernel because you want to stay up to date right with this enhancement new kernel features security so it needs to stay up to date and as a result you need after you apply the new kernel you need to restart it same thing with kernel development if your kernel like you you compile your kernel and it crashed because you had a bug what do you do the kernel is down you have to restart your whole machine and restarting a machine takes time like minutes right especially if you have hardware you have to do this post thing like power on self-test you have to initialize hardware so it takes time so imagine you are a kernel developer and you have to do the restart every single time you crash or every time you have to restart so people invented this thing that's called k execute the kernel execute which is a very clever way you know what it does is basically you take the the new kernel image that you want to load and then you give it the old one and just execute that and when you execute that the operating system will load the new kernel image and then discard the old one without going through the boot sequence processes and doing the hardware initialization you might argue that oh your hardware can put in a weird state but if you can't live with it and you know the consequences that's a huge save time so today right without any of the optimization this takes around 500 milliseconds so that was like a summarization of whatever is being said here right apparently that's not enough for tick-tock it's too slow you might wonder what tech talk is doing i don't know are you updating the kernel every every day that seems like a little bit excessive if you ask me unless they're doing something else with the kernel right or maybe they're using the kernel reboot to reboot their servers and instead of actually applying a new kernel images i don't know so they're planning to optimize this to down to 15 and that is the part that interests me right as an engineer i like to see that part like what do you what did you guys do to optimize that you see when i when i explained the kernel image right the kernel execute i said it takes an image that image by default was designed to be compressed here's one optimization because it's compressed first of all you have to decompress it and in order to decompress it you have to copy it so there is a copying involved and there is decompressing involved and those two things significantly slow down and that contribute to the 500 milliseconds apparently right and if you guys are interested this is their poll request or whatever it's called the commit the proposal so this has come coming from one of by dance developers hanji albert in many time sensitive scenarios we need a shorter time to restart the kernel however in the current k execute fast restore code or that's what it's called today there are many places in the memory copy operation verification of verification so there's a verification as well that i didn't talk about right so apparently we're verifying if the images is that kernel image is actually correct or not so that takes time wouldn't you still need to verify the image regardless even if it's uncompressed i don't know and decompressing the decompression and permission which takes more than 500 milliseconds through the following patch series machine kxs start kernel only takes 50 milliseconds so so here's some more of the patches so four patches in total pretty cool thing right and yeah like a bunch of stuff and guys yeah and guys i absolutely love to cover these things because i learned so much you know i before i read this one news this morning i didn't know even what kernel execute is you know but just reading that it's like wow there is a thing that was designed to solve a problem and that thing is now being optimized to make it even better it just fascinates me i don't know maybe maybe this is just me you know the ground of seems a scheme of things you know it's like shaving half a second of kernel boot you know it's not really matter for me and you you know if i'm gonna upgrade my linux setup i don't care if i can wait an extra second or half a second who cares right but think of scales right tick tock have is being is as popular as ever you know so they have tons of servers so if the a new kernel security patch has come in and they have let's say uh to be to be conservative 100 000 servers 200 000 servers you know definitely more than that you know if they want to apply you know the the security patch to all 200 000 servers let's do the math here so i just did a quick math guys if tick tock assume that they have 200 000 servers and just to be you know conservative they definitely have more than that you know and uh there is a new kernel that they need to update you know there's a new linux kernel security patch origin they have to do it or there's an optimization in the new kernel and they want to load the new kernel using today's method which takes around 500 milliseconds half a second to load a new kernel they are they need 27 hours 28 hours to be specific so yeah 28 hours to update all 200 000 servers and you might say they do it in parallel it's not really serially right it's not you're just not waiting for one server after another to reboot it depends on the script and how they wrote it but if you do it in parallel yeah it's going to be slightly less than that obviously but this really matters for them shaving that from 500 milliseconds down to 15 brings this down to what it's nuts this brings down this to less than an hour so yeah if they implemented this and it's approached by linus travolt then they can bring this reboot sequence to 15 milliseconds they can bring down the 28 hours down to less than an hour and obviously with in parallel it just it's gonna be a few minutes probably to do the entire thing so yeah it really adds up and i just i just love enjoy reading this article and go uh check out the for onyx really i love their content this this is an amazing website that just you know focuses on details you know they try to stay stay objective about the thing itself and discussing this stuff so guys as a summary i think this is i i absolutely love this thing because it it changes the way i think about anything i think i talked about this in another video like the other day right i talked about this video where there's a linux function that have been sitting there for ages you know and someone decided to take a look at the code and improve it yeah just by looking at what is it doing and they took advantage of new cpu architecture and just they improved it by what why three times they made it really fast you know it's just i absolutely love this thing it just changes the way i think when i look at michael and says why am i doing it this way it just opens up you know different avenues for you to think and especially as a back-end engineers where we have we we deal with operating systems right we deal with networking we deal with databases we deal with proxying so any latency really adds up and if you think holistically about this stuff it just changes the way i think and i'm i'm learning ever new things every day yeah it's just fascinating to me this is indeed c back in engineering is an unending beautiful sea of things to learn see you in the next one you guys stay awesome