High Fidelity Neural Audio Compression | Paper & Code Explained

Key Takeaways

what's cracking guys Alexa here in this video I'll be covering a new paper called in codec High Fidelity neural audio compression from The Meta team and basically I'm gonna do a Code walkthrough as well as as the paper walkthrough and hopefully that combination is going to be useful for you guys so so basically um what they've done is they've shown how they can preserve the quality uh after compressing and decompressing the audio file even though they are using a neural networks and they can do so uh and and basically save up a lot of space so like the the bandwidth of of signals encoded using this neural method uh are are much lower compared to say MP3 format Etc so it's very cool results we're gonna see uh the actual numbers a bit later but uh for now go ahead and basically clone this repo and you you should probably uh take a look at the installation for development section in the readme if you want to follow along and uh do a stepping through the the code base okay so there's a couple of uh samples they've shown here I'm gonna play a couple of them so they have two models basically one is the 48 kilohertz stereophonic model that's uh non-streamable we'll see what that means in in a moment and then they also have the 24 kilohertz monophonic so monophonic just means a single Channel stereophonic means in this particular case two channels but it can be even more channels and basically this one is streamable which means you can use it to real time uh encode uh send the data as as it's so as the raw samples are arriving you can encode it real time uh and and basically that's what streamable refers to in this context so here is uh the grand truth signal uh let me played would say I am you namaste in the Galaxy Globes connect like a piece of art it's the same in your brain so now if I play the encode X so so that's like uh I don't know what's the original size of the audio per second but it's it might it must be much higher than than the numbers you can see here so here we have like six kilobits per second and that's the encoding so that's this method let's hear it okay and finally here's the MP3 and the funniest thing is we'll see this in the table a bit later is there's this particular method with only six kilobits per second achieves the same subjective uh score as the MP3 so let's hear the 64 kilobits per second my state in the Galaxy Globes can act like a piece of art it's the same in your brain cool guys so let's uh dig into the paper and let's see what the magic is about so again as I said High Fidelity neural audio compression by The Meta team uh and there's a couple of papers that uh that are well that this paper is building off of so one of those is the Sound Stream and then to end the neural audio codec and the second one is audio gen so that's that's kind of a paper I covered a couple of videos ago and that could be a very useful prerequisite because I've covered some things like what are male spectrograms uh etc etc so maybe do check out that one uh in case you you struggle understanding what's happening in this video okay having said that let's let's dig into the paper so first of all why why is audio compression important well this sentence kind of clarifies it so recent studies suggest that streaming audio and video have accounted for the majority of the internet traffic in 2021 82 according to Cisco report okay that's kind of huge number and basically if we can manage to better compress our data then it's a no-brainer that we can save a lot like the bandwidth of the internet that's the energy money Etc et cetera so like it's a huge hugely impactful uh thing to be working on okay so then uh because we we mentioned we have a neural audio compression so what's the alternative well the alternative are these signal processing hand engineered uh methods and we'll see some of the baselines a bit later but basically as you can see here the audio codecs typically employ a carefully engineered Pipeline and like I've kind of highlighted the signal processing part because they are not learned they are not neural instead they are handcrafted okay so uh what's the problem what's up potential Pitfall of these neural models so here you can see that the loss in neural compression has a couple of problems so the first one is the model has to represent a wide range of signals such as not to overfit the training set or produce artifact Laden audio outside its comfort zone so basically outside of its training data set and this is something we are well familiar with and the way they cope with this problem is that they have a diverse set of data sets and diverse domains such as general speech and noisy speech and music etc etc okay and then the other problem is that uh instead of compression compressing efficiently both in compute time and in size so for the compute time they mentioned here so for the former we'll limit ourselves to models that run in real time on a single CPU cord so I think they've tested all of these models on a MacBook 2019 or something and that's kind of cool and then for the latter we use residual Vector quantization of the neural encoder floating floating point out okay so um some of these ideas um are probably familiar to you in case you watched some of my videos like vqvae or vqan or or any of the derivatives of those models but I'm gonna still cover and explain what those means uh mean in a second okay so let me quickly walk you through the the the higher level diagram here so let's see how this thing is constructed so on the left side of the diagram here we have a raw audio signal and we also have the Mel spectrogram representation so that's basically on one axis here we have frequency on the other axis We have basically um uh time axes and then uh the the the the color of the dot on this diagram tells you the amplitude of that particular sinusoid of that particular frequency at that point of time if that makes sense so again do watch the audio gen in case you're confused but this that's basically uh equivalent representation to the raw audio so this is like the time representation this is the frequency space representation of the audio signal okay and then there's a couple of things they do so first they they kind of pass the the signal through the encoder which is built up as you can see here out of these com 1D layers and they additionally have the lstm a two layer lstm sub module here which helps to integrate the temporal information because otherwise uh we would be basically so so like given that the particular architecture they are using they basically encode 320 samples from the raw audio into a single Vector here that's 128. Dimensions large okay and they call they refer to this thing here as the um as the frame so this thing is referred to as frame but I also refer to some of these things here as frames that's kind of inconvenient okay so once you have the the this representation at the output of the encoder what you do is you quantize it so that means each of these Vector quantizations is going to have Associated code book table so basically that means they're going to have something like this so like they have a table uh and the dimensions of this table are going to be 1024. so they have 1024 vectors and each of these vectors are so the dimension here is 128. the same as the output of the encoder okay and each of these so 128 and each of these Vector quantization modules is going to have a particular independent Code table like this one and so what these do then is you you take this particular Vector at the output of the encoder and you find the closest Vector in this Code table and you then just grab its index okay so when I say close I think they're using the L2 Norm so basically just find the closest Vector here and then let's say let's say for example this Vector here is the one that corresponds which is the closest one to this one then you just grab its index here so that might be like I don't know like 238 and now you use 238 as a representation of this whole vector and that's how you kind of compress down the the output representation here and uh then additionally they have as you can see they have multiple of these and that's because they do something called residual quantization so they first snap so snap they find basically they they find the closest vector and then they take this vector and they they subtract it from the original vector and then the residue is encoded by the by the by the next uh Vector quantization module and then rinse and repeat and that's how they basically encode this thing so you end up in this particular drawing here you would end up instead of 128 Dimensions you would end up with four dimensions because you have four numbers being allocated to that particular vector and we have again in the background we have four tables okay and then finally so I'm gonna ignore the the Transformer module here for a second but like finally you can basically decode those signals because you have the numbers and then you can just grab the vector from the associate tables you can sum them up and if you formed your representation which you then pass through the decoder okay and decoder is just basically you can see symmetric to what we had in the encoder so I'm going to skip explaining it and finally we end up with the reconstructed audio signal here okay so how the thing is trained is there are multiple losses and they also introduced this thing called a balancer which uses the gradient information to balance and scale up all of the low losses as needed so um first loss here is the LT which stands for time domain loss they basically do L1 between the raw representations of the reconstructed audio here and the input audio uh here and then they have the LS which is the uh basically a spectral domain L1 basically Norm between the the reconstructed Mouse spec program and the input Mouse spec program so that's what you do here and then finally they have the LD and LG which are the generator and discriminator losses and you can see here there there are like a couple of identical discriminators and so what you do here is you just convert the audio signal into the real and imaginary stfd which are just again the spectral representations and these are strictly more informative so let me just change the color here so these are strictly more informative than the metal spectrogram down here because given this information you can easily uh get the the Mel Spectrum information because you just have to find the amplitude of these imaginary complex numbers and that's what you what gives you the amplitude of the Mouse bike program anyways um you kind of pass that through this encoder through the discriminator and then you train uh using the adversarial loss as usual so you just basically contrast you give it the real signal you give it give it the generated signal and it teaches the discriminator to discriminate between those two and you teach the generator which is in this particular case this whole pipeline here how to generate uh signals that up like look as real ones finally a couple more losses here so we covered these ones and the the final two ones are this one so this is the commitment loss and again if you watched any of the vqba evq gun papers uh you're familiar with this one and finally there is the the the loss that corresponds to this Transformer here so what the Transformer does is it helps us to uh basically compress this uh latent representation to even lower dimensionality at the cost of of having to spend more more compute and and uh and basically increase the latency of your system but uh I'm gonna I kind of go through probably if you have enough time I'm gonna go through and show you how the language model part fits into this pipeline once we start walking through the code okay guys um let me quickly show you this this residual Vector quantization logic uh there is from this from this uh before I mentioned this Soundstream paper and they have a very neat um pseudocode um for the for this residual Vector quantization so let's kind of run through it so you can see here Y is denoted as the output of the encoder we have multiple of these quantizers and there is NQ of them so in this particular drawing NQ is four because we have one two three four of these components here okay so let's go back here um and so here is how it works so initially we initialize the residual variable with Y which is the output from the encoder and then you can see here we quantize it and then and and basically then we add that to the reconstructed signal then we subtract that quantize those quantized vectors from the initial residual vectors and then we use those residual ones and we feed them into the next quantizer and that's basically how it looks like on a high level I also want to uh walk you through the the rationale why are we doing this so I think a couple of these paragraphs might help elucidate the the idea behind it okay so the vector quantizer VQ proposed in these papers here in the context of vqvaes meets this requirement this Vector quantizer learns a code book of n vectors so n was 1024 in our particular case to encode each D dimensional frame and D was 128. of the output of the encoder here denoted as n of X the encoded audio as you can see the dimensionality being s times D so if if we have a one second uh audio signal s will be 75 because we have the Hop being 320 and I'm just using particular numbers because it's easier to kind of create a mental picture but basically imagine you have a 24 kilohertz signal and then one second means you'll have 24 000 samples if we divide that by 320 we end up with 75 so that's that means that for a signal of uh audio uh for for one second of the audio and the input will end up with 75 frames at the output and then these just the the as I said 128. okay so that's their mapped to a sequence of one hot vectors of shape s n so n is again 1024 which can be represented using these this many bits okay so because this is uh 1024 uh this is going to be uh 10. so once you do the log you'll you'll end up with uh 10 bits here as that's how much you need to index into those code book tables okay so let's see a concrete example here so let us consider a codec uh targeting a bit rate of 6K so kilo bits per second when using a striding factor of 320 uh each second of audio at sampling rate 24 kilohertz is represented by 75 frames at the output of the encoder this corresponds to 80 bits allocated to each frame right so if we have if we have the target bandwidth of 6000 and we have 75 frames that means we can allocate up to 80 bits and we will still not uh basically uh go over the target bandwidth so using a plain Vector quantizer this requires storing a code book with 2 raised to the power of 80 vectors which is obviously unfeasible okay and then they say to address this issue we adapt our residual Vector quantizer AKA multi-stage Vector quantizer which Cascades and Q layers so we have four in the drawing below of VQ as follows the unquantized input Vector is passed through a first V Cube and quantization residuals are computed the residuals are then iteratively quantized by a sequence of additional NQ minus one vector quantizers as described in algorithm 1 which was the one we saw here uh the dollar rate budget is uniformly allocated to each V Cube I.E we have r divided by NQ so R was 80 bits and then we divide that by the number of quantizers and that's how many bits each of the quantizers have to has to to to to to to basically allocate for example when using nq8 each quantizer uses a codebook of size at 1024 and that's the number we'll be using in this paper as well okay so hopefully that makes sense so that's why that's why uh this approach was basically uh developed um it makes it easy to get to the Target bandwidth without having to have huge ginormous uh uh basically uh code Books Okay so let's now uh quickly go through some of the details uh basically uh as I said they have the non-streamable and the streamable architectures uh the streamable ones so so they say here thanks to this spelling scheme uh which is basically left betting the causal padding the model can output 320 samples which corresponds to 13 milliseconds given at 20 kilohertz 24 kilohertz sampling rate as soon as the first 320 samples are received so that means as soon as we have 320 samples on the input you can already encode it into a frame and then you can quantize that frame and you can send that down the line and then you can just keep on doing that as the signal keeps on coming as you're receiving your your input signal you can keep on encoding it and sending it so that means that's why it's streamable okay and then they mentioned by selecting a variable number of residual steps at train time a single model can be used to support multiple bandwidth targets that's something we saw up there uh with this particular example so basically uh depending on how many of these um uh quantizer models you add you can you can achieve a particular bandwidth so here we picked eight because we want to have eight 80 bits per frame and thus we want to have six kilobits per second okay okay so I did mention the language model uh that's being used here so implemented as a Transformer to further uh reduce the bandwidth so let's kind of read this paragraph and then later we we probably will have enough time to just go through that and see it in code so for a Time step T the discrete representation obtained at the time T minus 1 is transformed into a continuous representation using learned embedding tables one for each codebook and which are then summed so we'll see that in the code a bit later but basically so the script representation are these numbers I mentioned so for example if this is uh step T minus one you can see that the discrete representation is basically these four numbers here and when I say four numbers even though it's it's kind of uh float initially which means you have like 64 bits we know that it doesn't have to be larger than 10 bits because it just needs to index into this table that has 1024 uh vectors and that's why these numbers are actually just 10 bit numbers right okay so let's go back here um and then for T equals zero a special token is used instead okay the output of the Transformer is fed into NQ linear layers with as many output channels as the cardinality of each code book which is 1024 in our examples giving us the logids of the estimated distribution over each code book for time t okay so basically instead of just finding the closest Vector uh in the in the codebook table and snapping our output Vector to that particular code Vector instead of that we basically do we form uh like a probability distribution across the whole codebook so this is the codebook table and we form a probability distribution across its indices and then combine with arithmetic uh coding they show that you can further compress the your your latent representation using this approach okay let's continue here um so here are the formulas for the losses I'm not gonna spend too much time here but you can see here L1 Norm between the time domain signals and then you can see here a combination of L1 and L2 Norms between the male spectrograms across various scales and then there is the the discriminative loss basically you can see here uh the generator loss and you can also find the uh discriminator loss somewhere in here so here it is but I'm not going to spend any more time on this because I've covered this uh also I think the audio gen or VQ gun paper uh basically have pretty much same losses as this paper so I'm gonna kind of skip all of those details here is also the commitment loss uh fairly familiar to those of you who who watch those videos and who who know about those papers and finally the final loss is just that we did some of all of these various losses and in order to avoid having to basically do uh like a search and find the right coefficients they make it adaptive and so they basically calculate the gradients of these losses with respect to the input and then they use the norm of the gradient to automatically rescale uh the the importance of each of these losses so you can see here if a gradient of a particular loss was huge so we we have this this is basically a norm and exponentially um and they've done that exponentially moving average on the norm of the gradient and we can see if that was huge historically then that means this is going to be down weighted so this the gradients from that particular loss are going to be down weighted because this is a big number uh in the um denominator and because of that that loss is going to be uh down weighted so that's the kind of the intuition behind it okay so as I said um they use diverse uh domains uh uh to to train these models that's how they ensure that they don't overfit your particular domain and uh um and and yeah so so they they mentioned your speech noise speech music and general audio and here are the results so you can see here um how they compare against some of the baselines such as Opus and EVS which are basically signal processing baselines and then we have Lyra from Google which is a neural codec as well and you can see it basically uh outperforms all of those models for all of the target uh bandwidths additionally this this entropy coded one which which uses the Transformer to further compress the representation uh basically outperforms for a lower Target um bandwidths outperforms the the the normal one the normal encoderk uh without the Transformer okay guys um let's see what else is interesting here uh yeah this result here is kind of uh cool so I think I mentioned this in the beginning of the video basically you can see here that MP3 uh with the target bandwidth of 64 kilobits per second has the same this mushroom score which is just basically subjective score app and has the same score as this this particular uh encoder with six kilobits per second so you can see these are fairly uh on pair and you can see that already with 12 kilobits per second it's like heavily outperforms the uh MP3 format and also if you include the entropy coding then you can even further reduce with 4.2 kilobits per second you'll perform 64 kilobits per second MP3 and you have comparable uh basically subjective um metrics that's very cool okay having seen all of this haven't seen the paper let's now switch our Focus to the uh actual code and uh hopefully that's gonna help us understand this much better so I just created a launch Json and I basically passed as you can see here the arguments I just have this test 24K uh I'm gonna pronounce this as wave signal I'm not sure how this format is pronounced but wave sounds easy enough basically that's 24K is already a part of the code base it's checked in so you just have to clone the repo and you will have it out of the box so that's it uh that's everything we have to set and later I'm just going to add this LM flag to see how uh how how the language model fits into this whole picture and there there is also the HQ which is the the 48 kilohertz non-streamable model but I'm not sure I'll have enough time I don't want to go over one hour in this video so let's let's basically start I'm going to enable all of the breakpoints here and uh let's run this this thing okay again I'm going to focus only on the on the crucial parts so um you just make sure that these audio signal exists and then here you can see that uh because we have a wave signal and this is basically ecdc which is the compressed format suffix because of that we are going to go into the compression Branch so but by understanding this compressed function this decompress function is going to be fairly trivial to understand so I'm going to focus on this particular branch of the code okay so we form the output as just we just changed the suffix basically it's also going to be as you can see here test 24K that oops that's 24K ecdc that's the that's the output and then we check that the output exists and because it already exists I'll just have to first um basically delete that signal I'm going to delete this um output compressed file and now it's going to work so it's going to happen to you as well so I'm not gonna edit this out I'm just gonna restart the program again and this time it will not fail so let's skip to this check output and this time this will not fail so everything's fine there cool so because HQ is set to false we're going to deal with 24 kilohertz signal let's enter this let's enter the uh uh the encodec model 24 kilohertz function uh and basically as you can see here we just set some variables and finally we we need to first uh basically get this model so the model consists as you can see here from the uh encoder and decoder so I'm gonna enter into the encoder just to briefly show you how everything looks like so we are in the encoder here and you can see uh basically so just some variables a location there and then we we have this ratios list which basically tells us how we are reducing the the temporal uh dimensionality as we are progressing through the as we're propagating through the encoder and if you sum all of these up you'll get the number that's probably familiar at this point and that's 320. and that's why we have three we need 320 uh raw samples to get a single output frame that's the the particular reason why this is happening because of the architecture of the encoder okay so let's continue so you can see here we just have these com 1D layers with this s prefix we just uh suggest that they have they kind of modified it a bit I'm going to show you how it looks like in a second uh but uh after that they just keep on adding um like these uh resonant blocks and bunch of which consists basically out of a bunch of these com 1D layers and some uh skip connections so basically nothing nothing fundamental there it's just your encoder architecture and that's it I'm going to quickly just show you how this thing looks like so they have this as prefix as I mentioned and so the thing here is they'll have this um depending whether we are dealing with a causal model or or non-costal model they'll have a bit different padding here and I think I think that's pretty much it they also have some details like uh how they wrap up this Norm layer with the com layer so you can see here just some idiosyncrasies but nothing nothing crucial really to understand how this is working so let's treat it as a black box and we have our encoder a decoder is fairly similar so I'm going to skip it completely and finally we um let me zoom in a bit in case you you can see a bit better now okay hopefully this this works let me let me know whether uh the the visibility is now better than than before I Zoomed In by the way I'm a bit sick so that's why my throat might sound a bit different uh but hopefully the video is going to be high quality nonetheless um so let's see what's happening here um you can see here that the uh this enqueue is going to be the max number of those quantizers we had in that in that bottleneck of the encoder decoder uh that this model can can can can can have in case we have the largest Target bandwidth and you can see here Target bandwidths go from 1.5 kilohertz all the way to 24 kilohertz and when we indexed by -1 we fetch the largest bandwidth which is the 24 kilohertz okay and because it's 24 we multiply with thousands such that we end up with Hertz so this is now the number of Hertz for the Target bandwidth and then we we divide that number with this thing and so what's this thing so so basically we we have here um the number of frames you can see here sample rate when you divide that by The Hop length you end up with the output number of frames it's going to be like 75 for our particular case here and then we multiply that with 10 because we they kind of hard-coded that because they assume that the code book is going to be 1000 24 slots and they they basically does assume that we need 10 bits to index into that code book so again that means that a single quantizer will have 75 times 10 bits so that means uh 750 bits for a second of of the video and so when we divide these two we get the number of quantizers we need for the Target bandwidth of 24 kilohertz as you can see here it's going to be 32 so that means we need 32 of these quantizers uh to be able to uh achieve this particular Target bandwidth of 24 kilohertz so again just to clarify as this might be a bit confusing imagine we have a signal as a second of the audio uh raw signal that's going to be 24 000 samples right and then we encode that and we end up with 75 frames and because for each of the frames we allocate 10 bits that means we have 75 times 10 bits right and then if we want to figure out how many of these do we need to get to 24K we just divide 24K by that number of bits and that's precisely what you've seen here okay um second uh thing I want to show you is this residual Vector quantizer although all of the fun is going to happen in the forecast not not here but let's see how it's constructed again just a bunch of uh variables here and then there is this uh residual Vector quantization let's enter that thing and you can see here we have 32 of these Vector quantizations uh basically wrapped into a list and let me see whether I have a breakpoint here so let's enter in into one of these Vector quantization uh modules and let's see what's going on there so uh here is the codebook dimensionality it's going to be 128 so that's the number we've been using as a running example throughout the video and then because we don't require any projection because the dimensionality of the output of the encoder and the code book are the same because of that we don't need any projection and that's why we're going to end up with identities here okay and finally we have this euclidean code book and the whole font is going to be in the forward pass I'm going to skip the actual initialization but that's that's how that's the implementation of the code book I've been mentioning uh this whole time and it's called euclidean because once we do the um once we try to figure the closest Vector in that Echo table we use the L2 Norm to to figure that out hence equity and okay I'm going to hit F5 here and let's exit this and now I'm gonna disable all of the breakpoints and let's exit this one and we are back here I'm gonna re-enable all of the breakpoints and let's continue here okay so we have the encoder we have the decoder we have the quantizer and finally we just have this encodec model which is just basically gonna wrap up all of those variables nothing fancy I can I can I can move this breakpoint I can remove it and let's continue here so let's just get to here and that's it so we are out of here and we have our model finally uh ready so now we just fetch the the the actual uh State dictionary so the pre-trained weights we initialize the model we set it into the eval mode and we continue here okay guys so now we set the target bandwidth okay so the the target bandwidth by default is going to be six kilobits per second and what it does is basically just sets this bandwidth internal variable of the encoded model 226 and that's it so nothing nothing fancy happening there okay so now let's just load the the audio file it's gonna be a 24 kilohertz 20 second audio file um so if you take a look at the root of this Repository uh we can find that signal here let me see where we can hear it [Music] when the second audio file that's it okay so let's continue here let me just show you the the shape of that wave file so sheep is 48 uh no 480 000 and that's because if you just think about it if you multiply 20 which is the number of seconds of the audio file times the 24 000 samples per second we end up with 400 uh 480 000 uh samples my brain is glitching man okay so let's continue uh we now do the conversion but because um it's gonna be a no up basically I'm going to enter here just to show you but basically because the target channel is set to one we do a mean across the zeroth dimension but because we only have a single channel it's going to be a no op if we had a stereo sound then we'd have two channels and we'd just do a mean across the two channels to get the final output Channel because the target number of channels is set to one here right but in our case it's just no Ops we can ignore it and then we do resampling but again because both our signal and the target uh signal sampling rates are the same 24K uh we do nothing so we just basically uh it's a no-op okay so here is the fun part we do the compression of the uh signal of the loaded signal and we pass it through our model let's see how that's going to work okay so let's kind of skip over all of these details uh what we do here is we are basically going to load uh the whole the whole signal um so this whole Loop is going to have just a single iteration and um I think worth mentioning is even though they support the streamable setting they don't have it implemented here neither do they have implemented the training code so what you can find here is the inference you can also find some details about the balancer and about the multi-scale sdft discriminators okay so basically you can see here they also use the terminology frame for the input raw audio signal and not only for the output of the encoder so just keep that in mind and not don't get confused by the terminology here so frame is again is going to be simply the same as our input signal so because we fetched every single sample it's just going to be 480 000 samples okay and now the fun happens in the code frame function so let's see how that's going to work so we don't do it in normalization so we can skip that and now we pass we pass the X so the input signal through the encoder so let's see how that's going to work so again this is just going to be a series of these com 1D layers and some of them are going to have a stride of Two And so our our input signal is slowly going to be reduced across the temporal Dimension and the number of channels is going to be increasing so what I'm going to do here is I'm going to enter the column 1D layer here and I'm going to put the break point with a particular condition and this is why I love Visual Studio code is you can basically added the breakpoint and you can set a particular condition when you want to hit this one and I want to hit it when stride is bigger than one right and so because I want to see how the the number of of uh the temporal axis is being reduced okay so now let's hit F5 and here we are so currently if I just print the Sheep the shape is as you can see here we already have 30 32 channels because we had some previous componential layers which didn't have described by the but they still had uh increased the number of channels but once I hit so now I'm gonna put the regular breakpoint and hit F5 so once we are here now if I do x dot shape we have as you can see here 2x a smaller dimensionality across the temporal axis and we have 2X more channels and that's our regular pattern we've been seeing all around cnns do that all the time as soon as you reduce the spatial dimensionality by two you also increase the number of channels by two that's a common pattern you'll see all around okay guys so now I'm gonna remove basically all of the breakpoints and I'm gonna exit from this function I'm going to exit here I'm going to exit here I'm going to exit through all of these and we are done okay so this is the encoder I'm gonna hit F10 and here is the final Dimension now it is going to be 1500 128 and that's the output of our encoder right and it's thousand five hundred frames because again we have 20 seconds and each second is 75 frames and that's why we end up with thousand five hundred hopefully that makes sense okay so now what we do is we have to quantize uh those those vectors and let's see how that's going to work so here we are so we first have to to to figure out for the desired Target bandwidth how many of the quantizers do we have to actually use to to get that Target bandwidth and so that's what this function does so let's enter into the function and let's see how it works so okay so they say get bandwidth per quantizer so does that make sense let's think about it so we have 75 here so that means we have 75 frames it's going to be 10 because uh business is going to be 1024 so this is 10 times the number of frames so this is basically the number of bits for a single Quant uh quantizer module and then we divide by a thousand because we want to have kilobits right and so you can see here bandwidth per quantizer and that's going to be 0.75 kilobits per quantizer okay and now what we do is we take the the target bandwidth which is six kilo uh kilo bits per second and we divide it by 0.75 and we end up that we need eight of these quantizers to achieve that particular Target bandwidth okay and that's it so now we actually do the encoding with these eight um with these eight um quantizers let's see how that's going to work so you can see here we just take the first eight quantizers instead of the full list which is like uh 32 so remember we have actually we have 32 of these and we can only deploy those when we have Target bandwidth of 24 kilohertz okay so let's enter here let's see how the encode works so it's fairly simple so first of all uh this pre-process is just gonna do some reshaping nothing fancy there so we end up with X being of shape 1500 128 and then this is where the fun happens in the quantize function so in the quantize function let me enter the quantized function you can see this is the code booked uh the codebook table I was mentioning the whole time so let's see the dimensionality of the codebook table it's going to be 1024 because we have 1024 vectors and each of those is 128 uh dimensionality right and so what we do here is we basically do uh we compute this distance Matrix which tells us for each of our output vectors how close are each of these code book vectors from that particular vector and because of that we're going to end up let me just show you this let me show you the dimensionality of the distance matrix it's going to be 1500 024 because as I said you have for each of the rows you have the columns telling you how far away is that particular codebook Vector from that particular output encode encoder Vector right and and because of this minus sign if we do a Max we actually find the closest vectors and that's it so we end up with we basically end up with as you can see here 1500 indices simply print the actual numbers so so those are the indices of the closest vectors in those code tables hopefully that makes sense and again post process just does some some reshaping and we are done guys that's it so now we do the decoding that's the second interesting part so let's enter it into the decoding part so let's see what happens there so um decoding is gonna call this the quantize method and the quantize is basically just going to fetch the indices and you can see here we just grab the vectors from that table uh that correspond to those indices and we end up with we end up with the quantized vectors whoops let me just show you the dimensionality here we have 1500 128 so that's when we snap our output vectors onto the closest vectors in the Code table that's what happened there okay and then projection is identity so we can ignore this and that's pretty much it and now that we have the quantized vectors we do the residual minus quantized and then we we save the indices from the first quantizer and we repeat the process right we now take the second code book and we we take the residuals and we just encode them and rinse and repeat so I'm going to now disable all of the breakpoints and I'm gonna just put the breakpoint here hit F5 I'm gonna re-enable the breakpoints and that's it so the out indices now are 1508 so that means for each of the frames we now have eight numbers uh that's how we encoded our output of the encoder okay let's continue here and that's pretty much it uh we can now keep on exiting here and we have everything we need we have the data uh the the compressed version of our audio file we now write some metadata into the header of this ecdc file I'm going to ignore that completely not that important important for us and then because we're not using LM we are gonna uh construct this bitpacker and so why the bitpacker so again the problem is uh these numbers are going to be uh either ins or floats let me just see for a second let me let me tell you the exact thing here so frames again frame shape is okay frames is a list let me just see okay it's a list and then I think I just have to do something like this yeah okay so here are the frames and and if we do the D type let's see the data type so it's going to be in 64. so that means each of these eight numbers currently take 64 bits even though we only need 10 bits and because of that we only because we own out of those 64 bits only lower 10 bits have the valuable information that's where the speed Packer comes into play to handle the low level details I did debug it and understand how it works but it's going to be um I think redundant if I do that in this video because you can you can do it in at your own pace but nothing fundamental there other than low level details of how do you extract the 10 bits that are actually valuable and pack them into the output bit by stream okay and that's it so here what we do we just keep on pushing those values and values are coming from the frames and so basically um that's it we're just taking those numbers from this structure here and we're just basically encoding them into this this byte stream uh and so let me hit F5 here let me remove this thing and we are done we are done we can exit here we can exit here we just write the bytes and that's it guys so that's it so let me let me just find the file so the file is here as you can see here test 24K ecdc is again created and that contains uh our uh compressed audio file you can see that we started from 937 kilobytes and we ended up with 14.7 kilobytes does that make sense let's see so we have 14.7 uh times I guess eight and this is how many kilobits we have and that's roughly uh well that's roughly 20 times 6 right which is 120 because we have six kilobits per second target bandwidth um and we have 20 seconds so that's roughly it so everything basically uh fits okay so now I'm gonna go to the launch file I'm gonna quickly uh add the LM and I'm I'm kind of afraid that I will not be able to explain this uh to give it justice but I'm gonna try nonetheless um because I think it would require quite a lot of time to actually uh explain all of the details behind how and why it works okay so now I'm gonna um start this again and let's just see the diff between the last one so I'm not gonna focus on on on the the things that that are the same obviously I just want to focus on the differences so um the difference is here are gonna be only as you can see here we pass the LM argument to the compressed um function so I'm going to disable the breakpoints I'm going to enable this one I'm going to hit F5 whoops again I forgot to delete the file God damn it so I'm gonna delete this one and I'm gonna restart so let's hit uh you can add the uh Force argument to avoid having these errors but I was lazy so I I kind of skipped it so I'm gonna enable the breakpoints let's now enter this let's enter here and let's see so use LM is where the differences are so there's a couple of places where there are the differences so I'm gonna um get here first so let's see what the get LM model does so basically we instantiate a language model here so let's see how that thing is implemented uh I'm gonna hit F5 here here we are we have something called streaming Transformer encoder and we have as you can see here 32 of these embedding tables and we also have 32 of these linear layers that project from the 200 which is the dimensionality of the of the Transformer 2024 because that's the uh cardinality that's what card stands for the cardinality or the size of the code book table that's why we have 1024 there right so I'm gonna okay I have a breakpoint here already uh let me see whether there is something interesting to mention here let's quickly walk through the the Transformer here um okay uh basically there are these uh streaming Transformer encoder layer um modules and uh let me just see whether those are interesting so those are basically just as you can see here just stock Pi torch Transformer encoder layer with some differences like um yeah this might be too complex to explain in this shorter video but in any case I'm gonna skip this I'm gonna skip the instantiation so we just have a Transformer we create those embedding tables we create the linear layers and we continue here so that's the LM okay so now what happens is we'll load the actual um pre-trained weights we put it into the eval mode and we return the language model okay so now we do the encoding the same way as before so nothing changes here we still end up with 1500 times 8 numbers so let me show you that so we have oops again I have to first do the zero zero indexing and then we end up with thousand uh 508 so that's the same shape as before and so now uh let's see where the difference is so the first difference is here we form this thing called arithmetic coder and it's fairly uh complex to to explain as you can see just by the description of how this thing works and some of the low level details uh here of bit shifting etc etc but basically in a nutshell what it does uh is uh using the the well I I'm gonna get there in a second so it's gonna be wasteful for me to try and explain it like this so in any case we have this arithmet decoder we're gonna see how it works a bit later okay so you can see here we first initialize the input as all zeros and K is equal to eight which which is the number of our quantizers so this is gonna before we get the actual eight numbers uh that are Inc that are the encoded version of the uh first output Vector of the encoder uh before that we just assume we have all zeros right so again let me just show you what I mean uh let me find the um diagram here so here we are so again before we have the actual encoding for this first Vector so we will have like four numbers here or eight numbers in our particular case we just assume we have all zeros we have all zeros and that's what what the program assumes here okay that's that's what this thing is okay so let's now see how this is gonna work so we pass the input the states and the offset into the the language model and then we just basically do a forward pass through the model to get the output probabilities okay let's enter here just to let me just show you one thing um whoops let me just say one thing and that's the largest part so after we just do a forward pass through the Transformer which I'm gonna skip so I'm gonna skip this part so I'm gonna put breakpoint here so I'm going to disable everything enable this one we're going to do a forward pass through the Transformer and we end up with the output okay so output is going to be of the sheep let's see so 200 because that's the inherent that's the the uh inner dimensionality of the Transformer so we we passed in all zeros and we basically how we how we um convert those all zeros into a continuous representation is on this line here as you can see here we just do eight times we grab the the number and we basically grab the corresponding embedding table and then that's how we get the representation and we then we just sum them up and I think that was also explained in the paper here so let me just show you uh where that thing is explained um I think somewhere here so okay okay for a Time step T the discrete representation obtained a Time T minus one is transformed into continuous representation using learned embedding tables one for each codebook and which are summed so that's basically this line this this line describes this piece of code here so again let's go back to the diagram it's going to be easier so we have these um let's say four numbers here and each of these four numbers is going to index into a particular uh table so we'll have four tables associated with this Transformer and we get their corresponding vectors then we sum them up and then we pass when then we do forward pass through the Transformer and we end up with a 200 dimensional Vector that's what we do there okay that's the idea and now we just map those using those linear layers I mentioned before that are going to have the output dimensionality being equal to the cardinality of the code book which is 1024. see if I do just a forward step over this we're going to end up with logits here are the shape the shape is 1024 8 and that makes sense because for each of the eight numbers that we are supposed to output for this particular time step we have thousand we have a distribution which is 1024 uh bins uh across our codebook table uh corresponding code book table I know this might be confusing so you'll have to go through at uh through the code code base at your own piece but hopefully this helps you some somewhat okay so probabilities are there so what we do next is this so we have this um we grab the next eight numbers and we add one and one is added because we have because the zeros as you see here have a special connotation they're just like initial values which cannot actually be we can't get zero as the output and that's why there is this plus one here anyways a small detail there um so here is the complex part we we built this uh quantized uh CDF which is the cumulative distribution function and basically how it works is that the higher the probability at a parti