Automatic Speech Recognition (ASR) with Facebook AI's wav2vec 2.0 model using Huggingface

Key Takeaways

hey everyone hello and welcome to yet another video in today's video we will take a look at of model that has been making the news in the nlp world uh wave to week 2.0 this is a framework for self-supervised learning of speech representations this was released earl towards the end of last year by the facebook ai team and what's really interesting about this is that this paper claims to require way lesser training data than that is usually required for us to be able to train automatic speech recognition models so just quickly go through the abstract before we go on to the code implementation uh which i have done using hugging face so we'll first take a look at the abstract so the abstract is we show for the first time that learning powerful representations from speech audio alone followed by fine tuning on transcribed speech can outperform the best semi-supervised methods while being conceptually simpler and if you see here i think somewhere they mentioned while lowering the amount of label data to one r wave to vect 2.0 outperforms the previous state of art on the 100r subset basically a 100x improvement right in terms of the amount of label data that is required using just 10 minutes of label data and pre-training on 53 000 hours of unlabeled data we still achieve four point eight slash eight point two w er we are as word error rate this demonstrates the feasibility of speech recognition with limited amounts of labor data and this is a very important problem important uh what do you say this is a very important advancement because a lot of languages around the world don't have the kind of label data that you need to do automatic space recognition because what you need is basically speech samples and a transcribed text of what the speech is about right so that is not readily available for many languages in the world and building that dataset takes time takes a lot of effort lots of lots of money uh and this this paper says that we can reduce that by a lot of margin and and that is why it's really exciting uh now let me quickly move on to the hugging face wave to wake base 960 h model this is the model that has been trained by i don't remember the name and that is not mentioned here but it has been trained by somebody and upload it pre-trained rather pre-trained by somebody and upload it to hugging faces models repository and we'll be using this for our demo today so what i'm going to demo today is basically i've already recorded a hello world basically kind of uh sent in just one sentence of our 34 seconds i'll first play that out for you and then we'll see how that gets transcribed using this wave to wick base 960 h model okay let me quickly see if i can play that out for you and yeah hello world dot wave hello world this is a test i'm just going to play that out once again hello world this is a test okay so let's move on to the code where i had written this down and i mean i worked on this on a demo that somebody else had built but i've tried to uh write down comments and put it in a more readable format so let me just run these installations basically here checking for the python version nvidia sma and then i'm installing a few libraries that i will be using in order to get this demo going okay i i really like this new arrow that collab is showing these days okay so we've already installed the requirements now let me just import the libraries i'm going to import nltk which is a natural language processing library librosa is a audio processing library arc bars again libros uh you don't need multiple times other parts also i don't need right now because arc path is mostly for taking in command line arguments import torch this is by doors uh and this is a very popular machine learning rather deep learning framework import sound file as i said this is another library you can use to manipulate sound sound files from transformers import wave to vect to for ctc and wave to vector tokenizer and these are the two uh functions we are going to use from the like hugging phase library transformers lltk.download.pu and kt you need to download this in order to not run into any issues later this is a tokenizer from ltk so let's just import all of these okay so we have run the cell now i have a few functions here i'll just run this functions first and run the code first and then we'll go through each of that so this now starts downloading uh the model onto the machine okay error opening hello underscore world.wev that's possibly probably because it doesn't exist here anymore so what i could do is go to downloads say hello underscore world or wav and put it here yeah so you need to upload your sample file and run this again okay let me rename this to hello underscore okay we have a transcribed version over here it says hello world this says a test and what i had said was hello well this is a test and this was what we were able to transcribe now quickly let me take you through the functions so this is where we are giving file as input so hello underscore world.wav and this is what i had uploaded then i'm calling this load underscore wave to vect underscore 960 h model function uh and the this function returns a tokenizer in the model let me just quickly take you to that function this is the function so the iterations tokenizer in the model from free trained tokenizers and models what it does it goes and uh fetches so hugging phase maintains a repository of different models from bird to roberta and all these kinds of nlp models right so every new model that comes out hugging face almost has it in the vehicle source it's really fast that way so we have a pre-trained wave to back base 960 h model which if you see here it in more detail the base model pre-trained and fine-tuned on 960 hours of delivery speech on 16 kilohertz sample speech audio when using the model make sure that your speech input is also sampled at 16 kilohertz so this is the model we are using and uh yeah this has been pre-trained and fine-tuned on 960 hours of labor speech on 16 kilohertz okay so this is the model that we are fetching here and this is the tokenizer that we are using okay so after i call this i call this function asr underscore transcript automatic speech recognition just go transcript and we here give tokenizer model and the wave input file as parameters and if i show you that function here okay so returns the transcript of the input audio recording your output is the transcribed text your input is your tokenizer model and wav file so you're reading the file and that is the wp file you're getting speech and sample rate making it one dimensional because your input is supposed to be one dimensional array and then you're resampling it to 16 kilohertz and like we saw over here when you're using the model make sure that your speech input is also sampled at 16 kilohertz so we are resampling it to 16 kilohertz tokenizing it this is uh pt pi dots uh getting the input values from uh basically tokenizing and getting the tokenized values taking logits over it and after that taken argh max which is basically finding the most probable word id out of all the predicted ids and then we take this protected ids and decode using the tokenizer to get the transcription okay and this is how we are able to get the transcription but the transcription once you uh take the arg max and then you decode using tokenizer is all caps what you saw as the output was not all caps it was hello world this is a test so what we've done here is that we've tried to correct that using correct upper case sentence function so we first change the transcription to lower and wherever we need to uppercase we uppercase and this is the function that does that for us okay we so tokenize the sentence and then for the first word we capitalize and return that over here this is the transcription you see and at the end of this function you just return this transcription after all of this processing has happened and then yeah basically store that as text over here and then we just print this text and this text gives you hello world this is a test now this is not very accurate we could perhaps train it on more data or uh or actually we don't really use a language model here right so most of these problems are solved by using the language model at the end so perhaps trying that out along with this could be a useful thing to do but uh that's it for now i mean we've seen the journey from uh giving a wave input file as a wave file as input to seeing a transcribed text output using facebook's wave 2 vector model that has been pre-trained on and uploaded on hung face so this is what we've seen today and i hope this video was useful for you to take a look at how you you could also train your automatic speech recognition system using this model i'll be sharing the links of all of these things that we've seen today in the description box below thank you so much for watching please do subscribe to the channel and keep supporting like you've always done thank you