OpenAI Whisper: Robust Speech Recognition via Large-Scale Weak Supervision | Paper and Code

Key Takeaways

open AI just open sourced whisper which is an automatic speech recognition system that's approaching the human level Baseline on both the accuracy as well as on on the robustness wise for the English transcription and that's super awesome so in this video I'm gonna cover the paper as well as I'm gonna go through the code behind the paper which they luckily open source so the inference code as well as the model checkpoints have all been open source and that's kind of cool so yeah without further Ado let's dig into the video so here is the tweet that uh kind of announced the uh release of whisper they say we've trained a neural net called whisper that approaches human level robustness and accuracy on English speech recognition it performs well even on diverse accents and Technical language whisper is open source for all to use you can see that whisper can do many tasks uh and it's so it's a multilingual multitask system so here's a couple of examples so English transcription so given an audio in English asks not what your country can do for uh blah blah blah you ask and then you see the transcription here ask now what your country can do for you so that's kind of trivial and then we have uh any two English speech translation where we have Maybe audio file in let's say Spanish and then you have the translation the quick prawn fox jumps over and then we have non-english transcription where the audio file is again in some foreign language that's not English and the the transcription is also in that same language so I'm not going to try and pronounce Chinese my Mandarin it's not at its peak uh so there is the last test here is the no speech one we are given a background music playing or or like whatever it's important that we don't have human voices the model outputs a special token uh signaling that there is well no no human speech going on uh other than these tasks the model can also do language detection which means it can given the audio detect which language does that um particular audio file belong to okay let's uh jump into the paper let's see um the abstract and then uh I'm gonna first give you some high level overview and then uh we can dig a bit deeper so the name of the paper is robust speech recognition by a large scale weak supervision we're gonna see that this is one of the main components uh of the of the paper so just like the data set itself and let's kind of skim through the abstract first so we study the capabilities of speech processing systems trained simply to predict large amounts of transcripts of audio on the internet when scaled to 680 000 hours of multilingual and multitask supervision the resulting models generalize well to standard benchmarks and are often competitive with prior fully supervised results but in a zero shot transfer setting without the need for any fine tuning when compared to humans the models approach their accuracy and robustness okay so let's first start by just showing you the the high level diagram how the architecture looks like and these special tokens uh that you use to prompt the model to do specific tasks so the architecture itself is literally uh like off the shelf um Transformer encoder decoder Transformer module from 2017 attention is all you need paper so they wanted to kind of remove that confounding factor of just tweaking the architecture for this particular task of speech recognition and instead they just took the off-the-shelf model and they've been playing with the data like data is the the main the main part here as well as these special tokens we'll we'll see those in a second so again how the system works on the high level is you grab an audio file and then you convert it from its raw form into something called log mail spectrograms so spectrograms are these representations where here on the x-axis you have the time information so the the time steps and then on the y-axis you have the actual frequency and then the color basically tells you uh what's the amplitude of that particular frequency at a particular point of time so like this point here like this would be the the particular time and then you can see the frequency here and the frequency is usually in hertz but in the case of male spec programs it's in Mel's and Mel is just like a logarithmic scale because that's how we perceive pitch and not like linearly like like hurt scale so that's uh tldr but I'll I'll link some resources uh for that explain very nicely how Mel spec programs work because that's not the topic of this video okay so we once we have that representation of the audio then we literally do a feed forward pass through the encoder stack so you can see here just blocks of simple Transformer layers that consist of the self-attention and MLP multi-layer perceptron and out comes the final representation here so once you have the final audio representation you use that to condition the decoder the causal decoder model here via cross-attention mechanism and then the whole Magic is in these special tokens so you can see here we have this startup transcription token we have then the English token and then we have the transcribe token and then the model generates like this is first the the the timestamp and then out comes the actual uh transcription of the audio file you can see that the here is the target so it's a supervised data set we do have we do have the the the label for the audio obviously and then we just do we shift by one and do the classic language modeling objective here there are some details that we can kind of dig into a bit later like decoder uses learn positionally codings here we have sinusoidal ones but as I said architecture is not the main contribution of this paper it's the data set and it's this uh multi multitask setup with the tokens so ignoring this part for the time being on the left let's focus on on the right part of the pipeline so we start with the start of transcript token and then the model will Auto regressively decode either this no speech token which basically means that we are not detecting as I said previously I think there is no speech going on the audio the model is only detecting Maybe music or something without human vocals or the model could generate language stack and these are by the way the only things that the model can generate because they they have various suppress like masking mechanisms to make sure this is the case okay so let's say the language stack was was decoded in that case uh we basically then have two options either transcribe or translate we've seen the differences so main differences uh in the translate option whatever the The Source language of the audio file is we'll end up with the English translation okay and then we can do the the node time stamps route or we can do the one with the timestamps where you additionally predict uh timestamp information and then you do some some transcription inside of that segment and then I rinse and repeat and finally you have the end of transcription token okay so all of these uh give rise to multiple tasks and uh we'll see how those play out a bit later in the code as well okay so having explained all of this let's uh dig uh deeper into the paper let's start in the beginning here so there are roughly two research directions that people have been pursuing before a whisper was published so one big uh research Direction was uh the unsupervised Direction so let me read this one for you so these pre-trained audio encoders learn high quality representations of speech but because they are purely unsupervised they lack an equivalently performant decoder mapping those representations to usable outputs necessitating a fine tuning stage in order to actually perform a task such as speech recognition okay so that means on one part of the spectrum we have models that are trained on a huge amount of data so we have let me denote the data set like this so we have huge amount of data I think they mentioned here like you can see here scaled up to million hours of training data so that's like and that's the worst visualization of data ever um let me delete that for a second and draw it properly so as I said we have a data set of 1 million hours of audio here this is a bit better and that's on one part of the spectrum so let me now show you the other part of the spectrum here um and again what I've said here is that they train these uh in a non-supervised fashion which means you go from from um from audio to audio representation I guess from male spectrogram to male spectrogram something like that and then the problem is how do you decode those representations uh to actual human language that's the problem with these models that's why you have to do the fine tuning part okay and then the second direction is this one so speech recognition systems that are pre-trained in a supervised fashion across many datasets domains exhibit higher robustness and generalize much more effectively to held out data sets than model strain on a single Source okay so that's just stating that we want to have a multilingual multi-task setup which we kind of know already especially if you if you have bigger models however there is still only a moderate amount of this data easily available these authors mix together seven pre-existing data sets totaling only 5 000 hours of supervision okay so that's the other part of the spectrum we have tiny tiny tiny models here I mean tiny data sets so this one has like 5K hours compare that to one million hours there okay and then they say this so whoops uh by relaxing the requirements of gold standard human validated transcripts uh these authors make use of sophisticated automated pipelines to scale weekly supervised speech recognition to 10 000 and 30 000 hours of moisture training data this trade-off between quality and quantity is often the right call okay so that means those authors um scaled up this by using the the weak supervision but they only went to maybe a bit like a couple of times bigger data sets so now we have like something like 10 to 30k and finally let's get to this paper so what these authors have done is whoops what's going on here uh yet these new data sets are only a few times larger than the sum of existing high quality data sets and still much smaller than prior unsupervised work in this work we closed that Gap scaling weekly supervised speech recognition the next order of magnitude to 680 000 hours of labeled audio data which means they literally close this Gap they have almost as much data as the unsupervised approach obviously there are the trade-offs of the quality and the quantity and so here are some of the heuristics that I've used to make sure that the quality is as high as possible considering that they are using these automate automated Pipelines okay so initial inspection showed a large amount of sub-pair transcripts In The Raw data set to address this we developed several automated filtering methods to improve transcript quality so let's see a couple of them in order to avoid learning transcriptase we developed many heuristics to detect and remove machine generated transcripts from the training data set and one of those is if you detect all uppercase or all lowercase then it's a very unlikely that that's a human generated transcript and then you're just going to remove that pair from your data set a second heuristic they've used is ensure that the spoken language matches the language of this transcript According to some pre-trained cld2 model which does the language detection basically and if the two do not match we don't include the audio transcript pair as a speech recognition training example in the data set etc etc and then finally once they had this big data set they trained the initial versions of whisper and then they used predictions of whisper to figure out what's wrong with the data so it's kind of very iterative process so they say that here for an additional filtering pass after training an initial model we aggregated information about its error rate on training data sources and performed manual inspection okay so I can assume that that was a fairly laborious process and that's probably one of the reasons that they did not uh release the actual data set that was used to train whisper here are some details around Mel spec program I'm going to skip that maybe we can get back to that later I don't want to dig into too many details that are not that vital here here's some details worth mentioning though during early development and evaluation we observed that whisper model set a tendency to transcribe plausible but almost always incorrect guesses for the names of the speakers and the reason being is usually in those transcripts you have something like um something like this predictive so open bracket and then a name of a person and then close bracket and then the model learned to because of the way how models learn statistical learning to just predict these even though you cannot infer what's the name of the person from a 30 second audio clips that they are using in most cases unless the name was actually referenced so they say here to avoid this we fine-tuned whisper models briefly on the subset of transcripts that do not include speaker annotations which removes this Behavior okay so uh one more thing and then we're gonna get to the robustness part uh they have problems with this wer metric so that's the word error rate metric and it's a similar problem as what to what people are encountered with when they deal with machine translation for example with the blue metric Etc so let's let me read this part so a wer which is based on string edit distance penalizes all differences between the model's output and the reference transcript including innocuous differences in transcript style which kind of sucks we want to focus on the semantics and understand whether whether the well in the case of audio transcription every single word matters that's true but the actual maybe punctuations or or style that's superficial and not changing anything you kind of want to ignore that right so they say here while this pose is a problem for all transcribers it is particularly acute for zero shot models like whisper which do not observe any examples of specific data set transcript formats so that means those other baselines because they're actually trained on that particular data set and then basically they do the the the the evaluate on the validation portion but the distribution is very similar they kind of can learn and overfit the particular style and thus they they um have an advantage compared to zero shot uh whisper model so because of that they developed various laborious uh normalizers so they say here our text normalizer was developed through iterative manual inspection to identify common patterns where naive wer penalized whisper models for an innocuous difference because I will not cover this in the code bit later so I'm going to just quickly show you how those look like let me just try and find those normalizers so here they are and if I open up English normalizers you can see how laborious this must have been like there is just a lot of rules that they had to do to to make sure yeah you can see it's kind of messy so like let me go back to the paper and let's continue here uh so uh there's they introduce this concept of effective robustness and what it means is it measures the difference in expected performance between a reference data set which is usually in distribution and one or more out of distribution data sets so obviously you want to have this difference to be as small as possible which means that you are generalizing to out of distribution data sets which is a desirable Behavior right you want to we want to have General models so a model with high effective robustness does better than expected on out of distribution data sets as a function of its performance on the reference data set and approaches the ideal of equal performance on all data set on all data sets so here is the actual diagram so here we can see um in an Ideal World we have this line here uh basically the the the the metric on the Libre speech uh Dev clean should be uh whatever the performance of your model there is you want to have the same um error rate on on the other data set so here is the average wer on these other data sets so you can see here uh some of these models that have much better performance compared to whisper uh on the Libre speech so you can see this point here for example whoops this point here I'm terrible at drawing today um so it has much lower error rate here compared to let's say this point but then you can see how how poor the performance actually is the error rate is huge for these data points of these uh Libre speech models so you can see all in all in all when you when you fit the fit like a line uh across these data points um basically you see that that it's much more robust whisper is much more robust compared to to the other supervised Liberty speech models even though it's not actually soda on on the Libre speech as you can see here but that's I guess less important we want to create robust General models okay so um they mentioned that even the smaller zero shot whisper model which only has 39 million parameters and a 6.7 wer on Libre speech test clean is roughly competitive with the best supervised Liberty speech model when invalid on other data sets so I guess that's uh like like literally this model here probably is competitive as you can see here a performance wise the Y Point here so the average performance on on the other data sets is close close to these supervised Liberty speech models they have much better performance on the actual Libre speech Dev clean test did a sad story okay so here are a couple more interesting observations um probably not that surprising at this point of time and that's that with more data you can see on the x-axis hours of transcribed audio you can see that these languages that have more data available in the data set have much lower error rate and you can see it's super predictive like the the the the uh the correlation coefficient here is 0.84 and um they mentioned somewhere here a couple of these outliers you can see here um Z8 which I think stands for Mandarin Chinese or something Korean are kind of outliers here given that even though they have this this much data the error rate is somewhat higher and they uh kind of uh answer that hypothesize why that might be the case here so checking the regression coefficient for a linear fit to these log log values results in an estimate that the the error rate halves every uh for every 16x increase in training data so that's the the general Trend we've seen and then they say that the many of the largest outliers in terms of worse than expected performance according to this trend are languages that have unique scripts and are more distantly related to the Indo-European languages making up the majority of the training data set such as Hebrew Telugu Chinese and Korean so I did point out to these two languages here so they speculate that these differences could be due I mean it's not the speculation it's a valid hypothesis I guess um could be due to a lack of transfer due to linguist distance or simply the problem with tokenizer being a poor match for these languages or variations in data quality so all of those factors could be confounding okay so they have additional interesting diagram here showing that the translation performance is not as as predictive given the hours of a translated audio it's not that easy to infer what will be the blue metric and then the the hypothesis here somewhere below here um so they say here so we suspect this is partly caused by the noisier training data due to errors in audio language identification so as an example Welsh Cy is an outlier with much worse than expected performance at only nine blue despite supposedly having 9 000 hours of translation data but as it turns out and let me just show you on the diagram here where the c y lies so you can see here it literally has 9000 but it has around nine uh blue which is like super low compared to what this this this trend here uh would suggest but then they say here inspection shows the majority of supposedly Welsh translation data is actually English audio with English captions where the English audio was misclassified as Welsh by the language identification system resulting in it being included as translation training data rather than transcription data according to our data set creation rules so that's a problem because that means you were using the Welsh token but actually you had English audio and English transcription which messes up the results so yeah I guess that's that's uh suggestive of not uh taking this this this plot here too seriously and uh just like I guess the general point of the paper is given enough data that's high quality we can achieve human performance on speech recognition pretty much um there is some saturation phenomena but I'm going to show you that in a couple of minutes okay so let's continue here let's see a couple more things here um things worth mentioning is this one so they do the following thing they add some noise on top of the input the audio signals such as for example on the left diagram we see white noise and then on the right the the pop noise so that's kind of background noise from a restaurant or above or something and you can see that the the the whisper which is this red star has um you can see it definitely does not have the lowest error rate on the Libre speech test clean but then as the noise levels keep increasing it all of a sudden becomes one of the best models and especially here on Pub noise um you can see that the model outperforms all of the other ones so it has lower error rate so that means it's more robust to noise and especially the natural noise compared to this synthetic white noise okay let's continue here um they they also compare uh whisper to other bass lines in this long form transcription where because the model can only see 30 second audio inputs uh they have to do uh sliding window approach and they they show here that they are their own pair and even better compared to many of these even Commercial Services you can see the blue the blue candle bars are for most of these uh long form uh transcription data sets lower than than the baselines given there are some exceptions obviously like here you can see that this company B is outperforming them but yeah that's kind of impressive um this is also cool they show how how they perform how they compare to to human transcription so you can see how whisper is very close to other even even human transcription services so so here they they took actual uh well uh human humans to to transcribe some some uh I think like they had 25 recordings of text or something and then they compared them and you can see it's on pair pretty much but then there's this computer assisted uh human transcription which is the best out of all of these approaches so that again means that this combination of humans and machines is still the best way to to to to go about many many problems let's see a couple of more results here um obviously model parameter this is again not that surprising although it's very noisy let me show you uh what's going on here so we have English speech recognition on this leftmost plot you can see that the model parameters as it's growing we can see that the error rate is obviously falling down but there is some saturation going on and there is also a lot of variance if you take a look at these um lighter lines uh the the dark blue one is just the average of those so that kind of means the performance varies a lot across across these 12 data sets and we can also see saturation I guess that's that's the main um I guess that's the main uh takeaway here we we see situation on English in particular and then on multilingual speech recognition we can also see like a trend downward Trend General down downward Trend and again a lot of variance if you take a look at some of these alight blue curves considering that the models are approaching that we are doing a supervised learning approach here and that the models are so where was the diagram here so that the models are on pair with this human um with human services with human transcription uh that might explain it and the final results I want to show you are these ones here um basically you can see two lines here two fits and one is the English only um training data set and the second one is multilingual and multitask and if you're in this regime where the flops are lower that means you're training smaller models and um you can see that with smaller capacity you you're suffering uh the so you want to have this is the error rates you want to be lower so you can see the English only version is better but the multilingual multitask models scale better so that means when you go into the bigger model cap like bigger scales you can see that the error rate is actually lower for the multilingual multitask which is um uh not that surprising I guess and um and encouraging given that we can just scale it even more and have higher quality data and we can achieve even better results uh these are some candle bars about the talk the the actual normalizer I'm going to skip that and finally it's important to realize that there are still a lot of heuristics that they had to use to make this uh decoding uh reliable so we have developed a set of heuristics that help avoid failure cases of long-form transcription in particular so they say here first we use beam search with file beams using the log probabilities as the score function to reduce repetition looping which happens more frequently in Greedy decoding and that's a common problem you have with greedy decoding in general like you have the repetition problem and then they say we start with the temperature zero I always selecting the tokens with the highest probability which is basically greedy and then increase the temperature by zero two up to zero 1.0 when either the average log probability over the generated tokens is lower than -1 or the generated text has a gzip compression rate higher than 2.4 which would probably mean that if it's higher than that that means the text is too random and too hard to compress which means it's probably gibberish I guess that's the the rough uh reasoning behind the heuristic and there is a lot more details here I'm just going to skip that for now um and you can see here like in the tabular form the same thing adding beam search adding temperature fall back and in previous text conditioning and adding this voice activity detection initial timestamp constraints all of those help improve the average performance across uh different transcription long-form transcription data sets here so I guess the main takeaway from all of this is if you take a look at the diagrams here this plot is kind of suggestive that given a higher quality uh and abundance of of audio files of audio data for for Lang for a particular language of interest uh and even off-the-shelf model you can achieve remarkable performance and we also want to do the multilingual multitask setup okay guys having seen the the main ideas from the paper let's now switch to the code and let me walk you through and and try and show you the correspondence between the code and and and the diagram you see here I'm gonna see a bit in a bit more detail how this uh special token uh pipeline looks like so um I already went ahead and cloned the repo created a condo environments so all of that should be fairly trivial everything is written down in the readme quite nicely so yeah um let me just show you the the types of models they have so they support a couple of models you can see here tiny base small medium and large you can also see the dot and versions which are English only and these are ones without the suffix are basically multilingual multitask um models okay so you can specify multiple audio files here we're using small model by default default task is transcribed you can also choose translation language is set to none which means we'll infer automatically what the language is so that's going to be interesting and then a bunch of parameters around decoding and these thresholds that help robustify the decoding pipeline now they just pop some arguments I'm going to skip all of these parts that are not relevant and crucial to the understanding of the model so this is like temperature goes from zero to one with increments of 0.2 so that's something they they said in the model they use to robustify the decoding if that's even a word and uh load model so let's let's kind of go here uh and let me jump all the way to to here so I'm gonna disable all breakpoints put a breakpoint here and let's just go to whisper okay so um basically in the background we we download the model uh actually I already downloaded beforehand so it just kind of loads the the weight so here is how the model looks like uh let me maybe now enable the breakpoints and enter here so we have the audio encoder and we have the text decoder so those are the two stacks we saw previously in the diagram so let me now um enter the uh let's enter the uh encoder and let's see how it looks like so we can see here we have two com 1D layers uh we have the positional embeddings which are not clearance so they're sinusoids and let me just put the code and the diagram side by side it's going to be easier okay so here we are you can see the the diagrams uh on the on the side here so again we have the the two com cone 1D layer so that's this part here the stem of the encoder and then we have the non-learnable uh positional embeddings you can see those here and finally we have a bunch of blocks which are just residual attention blocks so that's nothing nothing new there and finally we end up with the layer norm and later we'll see in the actual forward pass how we passed the the Mel uh the male spectrogram through the com1d and then we have the W blah blah blah the stem and then the the Twitter part so everything is fairly trivial here okay so let's exit the construction of the um uh audio encoder let me just um make sure that I have everything enabled let's hit F5 and enter the decoder and now let's just go back to this side by side view so here is the text decoder you can see a couple of things uh of course embedding table here and because we're dealing with obviously a decoder model that has a vocabulary we have positional embeddings which are as you can see here these ones are learned so it says here learn positional embeddings and that's why we have parameter and we just initialize as you can see here so 448 is the context length that they are using and the state is around yeah 768. okay and then we have a bunch of blocks uh these ones are our cross attention true so that means they're additionally going to have this third component in the cell phone in the in the Transformer layer the so-called cross attention block but yeah that's it fairly fairly uh similar to to what we saw in the encoder part okay and uh we have a mask which is going to be basically a causal mask this is going to be a causal decoder so if I uh if I were to print this you can see minus infinite Infinities on the upper triangle which means we're going to mask out the future tokens and this particular token can only attend itself and the previous token and the previous token Etc okay nothing hopefully nothing new there just like standard attention stuff uh Transformer stuff sorry and now let's go back so let me now kind of uh expand this again so we load the dictionaries uh we just push the model to the uh device uh let's exit here and they have a small like uh well it's not a bug but like they do the two device two times which is kind of unnecessary but yeah okay so here we are we popped the audio path so I'm gonna use the the one that they provided in the uh code base so maybe it would be useful for me to show you um hot that sounds so here it is can do for you ask what you can't do for your country okay so that's the other file we we are using and let's now transcribe it so again let me make sure we are enabling the breakpoints let me hit F5 and here we are so um let's see what's interesting going to happen here so the first interesting part is the log mail spectrogram so let's see how we convert an audio file which is just a path into a mail spectrogram so let's enter here so the first part is loading audio audio file so we just use ffmpeg for that and then you can see blah blah blah from buffer because these are bytes integer 16 and that's we divide by 32 000 because precisely because we're using int 16 so that's like I guess 2 raised to the power of 15 is is exactly that number and that's why we are basically putting the amplitude in the minus one one range just normalization detail okay so once we have so we are in the log mail spectrogram so once we load the data which is just raw data at the moment so that's going to be just a sequence of numbers 176 000 um floats now we form the hand window just some signal processing stuff then we apply the the short um uh basically a Fourier transform it stands for what's the what does it stand for a short time yeah short time Fourier transform okay so we applied the short uh time for a transform which is going to basically convert this audio file into a spectrogram which is your regular spectrogram so if I were to print this particular um variable here the spectrogram the shape is going to be as you can see here 201 and 1101. so 201 is the frequency Dimension this one is the time Dimension the reason being they have some uh they're doing sliding window approaches so that's why we don't have as as many uh time stamps as as in the original raw raw audio okay so then then we do the magnitude because we want to have the magnitude component of the of the Fourier and then we form the these filters as you can see here we have we just what they do is they they use this uh librosa to generate these male filters and then they saved the mail filters into this particular file so you can see here this assets Mel filters mpz so if you go to the assets directory here you can see that the amp is out so they're just gonna save them there and then they don't they can avoid having the dependency anyways it's already too many details let's get out of here uh filters and then we uh do the matrix multiplication again that's a detail where by when once you multiply using this triangular filters once you multiply that with the actual magnitude Fourier spectrogram then you end up with a male spectrogram and now we're gonna see how this is going to have a 80 bands so if I were to print the shape of this thing it's going to have oops let's just grab this variable it's going to have as you can see here 80 bands and we have 1100 um again uh time Dimension and if you go to the paper you'll see all of these details there so again let me just quickly jump to that part where they mention it somewhere here so all audio is resampled to 16 000 Hertz and an 80 channel log magnitude metal spectrogram representation is computed on 25 millisecond windows with a stride of 10 milliseconds and because we're using these windows of 25 milliseconds that's why we end up with a much shorter representation in the time Dimension compared to having the 176 000 or something okay let's go back to the uh code some normalization I'm not going to focus too much on this and finally because the language is none we are now going to predict the language let's see how that's going to work so detecting language using the first 30 seconds blah blah blah we first do the padding uh which is just going to pad this to a 30 second uh audio file so that means it's going to have uh three thousand uh three thousand um basically dimensions in this in this time along the time axis and now let's basically do the detect language part so we pass the segment which is just a padded male spectrogram which means basically we paired it with all zeros uh to the right and let's now do the detection so here is how it's going to look like uh we get the tokenizer first it might be interesting to just briefly uh jump into the tokenizer and let me show you how it looks like so we specify that it's a multilingual tokenizer to ask is transcription we have English and now we do the build part okay so here is how building the token editor look looks like we are basically taking the pre-trained gpt2 tokenizer fast from the hogging face library and we just pre-load it and here are the special tokens so here are the tokens we saw before in the in the paper so we have the start of the transcript token sot we have as you can see here I think this language's Keys is like 99 keys in total or something let me just double check that so you can see here at 99 Keys precisely so that means We'll add for each of those languages We'll add this special token for example this and uh like the the a smaller sign and then bar and then n like en and then the rest and for each of the languages a similar thing so you can see here how the keys look like here for the languages and then we have translate transcribe start of LM start of prev no captions and no timestamps so each of these have some some meaning we're gonna see maybe some of them a subset a bit later so we end up adding those to the pre-trained hugging face tokenizer and we return back here okay so now uh let's just see what's interesting none of this is going to be interesting for the time being they just create a sot sequence which is going to have as you can see here they start with the sot and they added the en token and they and added the transcribe token and so that's basically the sequence we saw before here so let me just go back to the paper so that's your classic sequence here so the start of transcript language stack and transcribe let's go back to the code but we are not going to use that for the language detection that we are currently doing so that's why I said it's not that important at the moment so we just do unsqueeze on the mail uh nothing fancy and then we pass the mail through the encoder stack so let's now get the features so you can see here the the everything is the same as what the diagram says uh we do a forward pass through I guess 12 blocks or something so we can just skip that let me just see how many blocks this one has I guess it's like 12 or something 12 yeah um let's just pass through all of that and we ultimately end up with the following shape so you can see here um 1500 so because we have one of these has stride of two stride is set to two and that's why we went from three thousand two thousand five hundred and the dimensionality is 768 so that's the model dimension of the of the encoder okay let's exit here number of odd is going to be one because we only have a one audio file and here you see so here we just set the sot sequence sot token and uh we create a tensor that has sot and now we're gonna pass the Mel so the encoder features um from the output of the encoder and we pass the S of T and let's see what's the model is gonna generate so let's see which language it gives the highest probability to and that's how we pick the language so let me uh just go into the code instead of explaining it like this so here we are we are in the text decoder forward pass I'm going to ignore the cache part those are just implementation details of how they optimize this uh execution of the the forward pass of the decoder and let's see what's going on so they do the token embedding so again remember this is just just the sot token as you can see here and then they just add a positional embedding so all of that is kind of standard stuff and then you can see here uh they're doing the cross attention so XA XA is our um the output features of the encoder as you can see here and so that's that's what's going on here so because of that I'm just going to skip all of that and let's get here now the output features are multiplied by the we do matrix multiplication with the embedding table and that's gonna project those um from 768 Dimensions into the vocab space which is like let's see what's the dimensionality so logits shape is going to be 51 080 865. okay so those are the logits and now comes the interesting part so we create a mask that initially just has um basically once for every single single token in the vocab okay so that's that's the mask initially and then for the language tokens let's set that mask to false which means we do not want to mask the language token so the en the d e all of the 99 I think this should be 99 let me just make sure that's the case so let's see what's the length of this list 99 precisely 99 and so that's how we modified the mask and now we put the logits of every single token except for the language tokens to minus infinity which after we apply the softmax is going to be is going to set them to zero so that means they'll have zero probability and only the language tokens will have non-zero probability okay so here we do ARG Max to find the the highest probability language token we end up with I think English or something so this is I think English and uh we also returned the probabilities uh for for for the other language uh tokens we just shipped that and we return back those results so we returned the actual language token and we return the probabilities um but uh I think yeah this one is not even used so we just grab the the this is kind of wasteful they could have just passed it directly here but anyways um so they end up with the language uh we just grab the key so you can see the probabilities are language token and the probability you can see that en English has 0.95 uh roughly and that's why we're gonna pick uh precisely English as the language here and we output detected language is English language okay that's the first part so we now have the English predicted now let's continue uh doing the actual um transcription I'm going to skip over all of these details um let me just um disable the breakpoints let's get to the interesting part these are just some details where so the Precision is as you can see here 20 milliseconds that's why this is 0.02 those are I guess not that vital okay we just Pat again the the Mel so that we end up with 3 000 here so let's just check the shape of this segment so you can see it's 83 000 because we have 80 male bands and we have three thousand because we have a 30 second padded uh audio clip okay we calculate the duration it's going to be 30 seconds and then this is where the magic starts happening the code will fall back let's see how that's going to work okay here we are temperature um again we have the the list of various temperatures we grab the first one uh which is going to be the zero temperature which means we are basically uh greedy decoding and um well actually because we'll be actually using the beam so let's just continue here and see what's going on um decoding task so this is how they they kind of uh Design This they form a decoding task and then they they call run on that task and they pass the Mel spectrogram so let's see what's going on I'm gonna skip some of these details because there's too many stuff going on just want to focus on on the stuff that's be that's going to be useful um for whisper understanding of course so here this one is um implements the this this penalty with the length so when you're doing beam decoding you want to take into account how probable that beam is but you also want to penalize if it's too long so that's the this is what this this maximum likelihood Rancor uh does here as you can see here so penalty and we are appending the log prop divided by the penalty I'll skip this but in case you want to learn more and I think this this repo is amazing when it comes to understanding how beam decoding works if you wanna uh I might create a separate video where I just go in a lot of depth and explain how this how this how this works but here I'm gonna skip those details so we form a beam decoder here and then this is the interesting part so here are some of the heuristics that they are using they are suppressing certain tokens during the decoding process so one of those is going to be suppress blank which means we're going to suppress the blank token as the first token after we start the transcription so um we'll get to that a bit later then they have a specific list of of suppressed of tokens that they want to suppress so because we have let me enter this one so let me enter this one so suppressed tokens are set to -1 because that's our input argument uh to the program and now let's see how um that's going to be mapped into the actual tokens so here you can see we extend by these non-speech tokens and here they are so just a list of tokens that cannot appear in the in the spoken language and again I'm going to skip because there is a lot of details but that's that's the rough point we don't we want to help the decoder by just kind of mapping but by masking out those particular tokens okay and they add some more we don't want to predict the start of transcription because we already know we are transcribing so so that's kind of a no-brainer but yeah um and finally the no captions which is the token that tells you that we don't have any human voice we just have like music or something going on okay that's a second heuristic they have and then they also have um these applied timestamp rules which do what okay I have a break point here we'll see what it does once we hit the apply so let's just continue let's continue here let's exit here and we are now in the run method so all of that was construction of this decoding task now let's see how the actual um run looks like so we first fetch the audio features and this one is gonna again just pass the the Mel through the encoder and return back the audio feature so I'm gonna skip uh all of that so I'm just going to disable that let's fetch the audio features by passing the mail through the encoder stack uh and by the way I'm running this on a CPU because my current environment is was not properly set up that's why that's why um it's a bit slower but it's still working fairly fast which is kind of cool um okay so we set the initial tokens to D3 so that's the sot that's the English and that's the transcribe that's our initial sequence and then we um this is going to be a no-op uh basically we just passed the language we already detected language we know it's English so there's going to be a no-op um now we do repeat uh by five because we are doing B again I'm not going to focus on the details of beam but let's just see the shape before and after so the shape now is 1500 because that's the time and then this is the actual uh embedding dimension so let's see what's going on after this call we have five so we have five because we are doing we'll have five hypotheses that we are keeping track of during the beam decoding and we do the same for tokens and then now here is the main Loop so here is where the magic happens okay let's start here so we Now sample for 224 samples and we first passed the tokens as well as the audio features to grab the logits so there's going to be a simple pass through the decoder again um some hook stuff that's again optimization detail here is the decoder call so what we do is again we pass all of the we embed our three tokens so here let's see what's the shape now as you can see here five three seven hundred sixty eight because we have three five because we have beam and this is the dimension of the model now we pass all of those and we also have the again we are cross attending to the output uh features of the audio encoder okay that's the idea and finally we end up with the logic so we can skip all of this this is just like regular Transformer stuff whoops and let's exit here let's exit and we have our logits now this is an interesting point let's see what happens here because we are in the iteration zero and because no captions is set to to to yeah because no captions token exists we're going to enter this branch and what we do is we grab the logits for the sot index so that means what's the what's the once we we condition the model on the sot token what's the distribution across the next tokens after the sot and that's in case that that there is a high probability for the no caption probability then that means we don't have any audio uh any human vocals in the audio clip and so again we index here we grab the sot logits and that's going to be let's see the shape so here is the shape so you can see here uh because it's beam we have five but we just have the logits above the sot token and then additionally we grab precisely this particular we are interested in the probability of this particular token and that's going to be the no caption probability and in our particular case it's very low but if it was high enough then we would basically ignore the decoder output and just say that okay there is no voice happening in this audio clip and let me remind you here that's this part so we either have the language act or the no speech stack and that's the voice activity detection so that's that's how that part that task breaks so now you see it in code okay so having collected that let's now grab the because remember we actually have the sot English and uh uh transcribe tokens and we're gonna do the decoding even if this was higher probability we're still gonna do the decoding here and assume as if as if we want to transcribe the the audio but in case this was high enough then we would just ditch and ignore all of that um output which is probably wasteful uh maybe some just a break statement here or something will be uh more optimal but but yeah that's jus