Step 1: Data Prep for Speech Recognition using Zipformer (Icefall)

Hello everyone. Uh now I'll show you what I'm going to teach in this video. As you can see on the screen what whatever I'm saying is getting transcribed. Uh this is because of a model which is running in the background locally on the system and uh as you can see this is real time transcription. Uh in other words we can say it is a speech recognition system. And uh I'll show you how to train the model from scratch from data preparation uh till data processing how do we do augmentations and uh uh what are the models what kind of models in in current industry we use and uh um basically how you can deploy that also but I don't think uh all of that will get um covered up in this video only one video so I'll start from data processing and then we'll slowly go into the training. Let's get started. Uh I have created a project here. Now I need to download the the repository icefall and uh it has multiple models. Uh in current scenario we are going to use a model called uh jeep former uh that's based on transformer architecture and uh let me download it first. I'll I'll show you how to download and everything. So, let's use git uh I mean the project directory. I'll clone the repo here. Download is completed. If you open the folder here, you will get a folder called ESS. In ESS, you'll find many different repos. So these are uh basically data sets uh which are getting used to train the models. They have done multiple uh experiments. Uh usually we use lib speech that's a very large data set. So in ASR if you go into the ASR folder here you will find multiple folders. Uh these are different kind of models. uh ignore all of those just go into the zip former folder and uh in zip former you you need to open the script train.py. So this is the script which uh we are going to use to train the model. This has like all the settings which you need. These are like default settings. You don't need to change anything for now. If you want to deep dive into the model struct structure and uh depending on a data set also you can change uh different parameters but for now you can keep everything uh as default. So here if you see it is loading the training data and uh these are the training cuts these are getting loaded here in these functions and uh there are different functions to load different kind of data sets. This function is for uh loading different three different data sets. All are in one function. Um 100, 360 and 500. These are the data duration uh in hours. If you open this, if you go into this function, it has one cut uh JIP uh file. It is in the manifest directory. I'll go into this and uh you need to change all of this later on. This is in ASR data module. I'll open this as well. Okay, let's download the data set and start from there. This is the OpenSLR website. Here you'll find the different data sets. You need to go to the resources folder. Here you search for English and uh if you go down English speech recognition corpus there are multiple speech recognition corpus. These this is the thousand liy speech which directory we are in right now. This has 100 hours of data set uh in English. If you click on this, you will find different sets. These are uh this is the validation set. And uh this is the 100 hour uh data set, clean data set. You can you can download this 100 hours data set. If you want all of the data, then uh you can download 500, 360, 100, all of this. Uh let me let me grab the 100 hour data set from here. Uh yeah so I have downloaded the data set here. Uh this is the zip file that I have downloaded and I have extracted out uh the data set as well. Uh I'll show you this is the folder and uh this is the folder structure. It has uh different uh sub directories and inside all the directories there are audio files. uh these audio files along with uh the transcription and I'll I'll open it in VS code. Yeah. So this this transcript file contains the name uh audio name and uh their corresponding text and uh these are space separated. So you need to process uh this transcript files and then uh format it to a manifest. Uh let me go into the into this folder first. Uh before that uh I need to copy the path and then let's go into the folder. Let's prepare a script like uh prepare manifest py. Um so here uh in this script we we need to open this folder. So let's say root directory. We'll take the root directory and we need to import OS also. Then u we need to iterate over the directories inside this folder. Uh let's use a for loop root directory and files in OS. So this work function will go into all the subfolders inside the directory and uh let's print print the root here. If you want to see what is happening here, uh let's do ls here. And uh this is the folder. Okay. So you need to give the path of the folder. Here the raw folder is there. Okay. If you save this and uh if you run this sorry Python 3 prepare manifest. So as you can see it has uh printed multiple folders uh and all are like subfolders of this raw directory and uh so these folders contains uh different files. Okay. So if you want to see the files and then be inside the root directory then you need to uh use one more for loop. Uh let's let's say for uh directories in os dot list directory and uh you use the root folder here and then pin print this directories and if you run if you run it again uh uh I'm sorry this would not be directories I think these will be files let's say file Okay. And if you run this again then you will get all the files inside it. All the audio files as well as the text file. But uh here we need to ignore the audio files. We will just use the text files. And if you in this in this directory if you see this is the root directory and inside that the files are there. So we can use this root uh to concatenate with the transcript. So here we need to use this transcript file. So let's say if u txt is in the file in file then we we know that that's a transcript file. Uh so here we need to open the script transcript uh for line in open. Let's open it. Uh but to open it we need to join it uh join the path root and file os.path.join join and here uh we need to join root and uh file. So this will open the the transcript file. You need to read the lines. Uh this read lines function will read it. And here uh you can print the line here. Uh okay, let's save it and run it once. Let's see. So here if you see it has printed multiple lines these are the lines transcript lines as well as their corresponding audio names. Okay. So let's see here we what we need to do is we need to take the line and each line has one ID right so we need to uh split it first. Let's split it uh using the space delimiter and uh this will split the line. Okay. Okay. First we need to strip strip it because it has the slice at the end. So that's why we need to strip it so that can remove that slice and and uh it will strip uh it into the words word sequence. So we know the first word would be the ID of the audio. uh let's say audio id equal line line and uh zero index would be the audio id and uh line uh line would be we can we can say that we need to join it space join this function will join all the words we don't we need to join all the words again because uh we want it as a single line uh so but we don't want the audio id so that's why we need to go from the index and then till the end. So this will join all the words u all only the transcripts not the uh audio id this audio id will have the id that means this name only but this is not a full name of the audio where you'll see these are the audio files it has flsc at the end this is the format of the audio uh so you need to append that uh with this with this name this since this is a string you can append it using plus and then you need to append a root as well. So you can use root plus plus but here you need a slash. So that's why I have given two classes. So this is the audio ID. You can you can use OS function. Uh let's say OS dot path you need to check that if this audio exist or not. If it doesn't exist then you don't need to worry about that. You can ignore that audio. Uh so this is just for uh uh exceptions always.path.exist. So this function will check if that path uh is really exist or not. Let's say print uh exist. Sorry. And then okay that's fine. So we'll save it once and then run it. So if you see it is printing exist exist every time. So that means all the audios are uh uh are there. So you need to do everything inside this loop inside this if function only. So if it exists then we'll process it or process audio otherwise we will not process it or we will not include it in the uh manifest. So here you got the line here you got the audio ID. Now we need to uh create the manifest the output file. Okay. One is recording file other one is supervision. Recording file will have the information of the audio and the supervision file will have the information of the transcript. Let's open these files first. Let's open it. Open. And then first you need to provide the language. What language you are going to use and then recordings. This recording should be there. record uh recordings and then data set type the format would be JSON L. So uh we need to open it in right and as uh C and uh then we need to do this for the supervision as well. uh let's say sup and uh this this name would be supervisions supervisions and then the rest would be same and uh everything should be inside this width and here we have opened two files um in this directory wherever this script is there and after that uh we need to calculate the duration of the audio But before that uh we need to open open the audio first and for that uh we can use uh torch torch audio. Uh for that we need to import torch audio torch audio and uh uh we know this is the audio path. So we can use uh let's say it it will return audio information. So let's name it audio info. and then torch audio dot info and uh then we'll get the audio path. So what this will return this will have uh some information of the audio. We can print it here audio audio info and then let's exit it here first only one iteration will be there. Um okay so here if you see it has printed this uh information metadata it has the sampling rate number of frames number of channel bits per sample and encoding uh uh let's say audio duration this would be uh audio info dot uh number of number of frames divided by audio info sampling rate sampling rate. So if you print if you print the audio duration this will uh this will give us the audio duration in um seconds. So if you see the audio is 13 second long and uh now we can define we can define the structure uh this is the structure of the manifest files. Let's let's comment out those print functions. Okay. So here uh we source type would be file and channel would be only one. So it has a list of zero. And then u the source would be the audio uh audio id sorry uh audio ID is the path you need to give here. And then here uh you need to give uh unique ID. Audio path and audio path would be this this one and audio ID would be the index of zero. And this audio ID uh we will use that this audio path would go here here. And this audio uh id we will use in this ID. Okay, we got it here. And uh sources would be this type data. And the sampling rate would be audio info. Number of samples would be the this number of samples. Then audio duration. Then audio duration is this one. We can give it here. And then this is for the supervision data. This is this is for recording data. And here it here the supervision comes here. We can use the same ID. Uh this audio ID and we can paste it here. Recording ID is the same one. So we can paste it here as well. Then the audio duration comes this one. We can use it here. So to dump this data set we we need uh JSON. We need to import JSON and uh then we need to do JSON dot dump. The data would be the recording data. We need to give the recording data and this will go into the uh recording file. Ensure there is a flag ensur you can make it false. So this data set will be one single uh single line and then you need to put one slash and after that because uh otherwise all the all the recordings will be in single line and you will that will not be human readable properly. So you need to write slash slash n at the end at the end of each JSON. So this same thing you need to do for supervision as well. Uh here I have copied and pasted here. This is the supervision data. You just use that and uh this is the output file. You use it here and this is here. We need to give the audio path instead of the audio ID. Uh I changed it earlier but I forgot to change it here. Now I have changed it. And then uh uh we can use DQDM as well uh to see the process what kind of whatever data has been processed yet. Let's use that from TQDM. import TQDM and then we can put it uh here dqdm and then uh that's it. Okay, let's run it once. Let's see uh let's see the output. Uh here as you can see the two files got generated. Uh let's open the recording file. Uh if you see it has the JSON in single single lines. If you analyze one line it has the id individual ids for uh each audio. Uh sources would be whatever we have given. This has the path and then uh after that it has the sampling rate, number of samples, audio duration and channel ids. These are the these are the things you need to train the model. And uh similarly here if you see the ids uh here and then recording ids would be the same and then starting and ending these are the duration starting and ending. If you want uh if you want to start from a different point rather than zero then you can use it you can change it here but uh for now since we are going to use the whole audio from the initial initial uh point here we have given zero and then the duration is there uh channel is zero and then the text these are the supervisions uh uh and this data set is mostly clean so we don't need to clean much um in later part we will see the data But in this video I just wanted to cover only the data preparation part. Yeah. So this is this is it. We have prepared the manifest and uh now what we need to do is uh we need to create a folder in this data folder. Let's use mkdir and data the folder name would be manifest manifests and then u you need to move these two files. Let's use MV and EN uh EN uh recordings and uh EN supervisions uh to manifest. So we have moved it to manifest and then if you open this folder sorry what happened it's not refreshed. Uh if you open the manifest folder here you'll find these two files. uh we can what we can do we can separate out some data from this training for validation set uh let's go into the manifest and then uh just give me a moment it is not visible for you I think um yeah I'm in manifest now uh if you do ll you will find these two f two files so let's take the tail of each uh each file and uh number would be Let's say how many 200 and uh en recordings and we will put it in a n file recording and then we'll change this train uh to test. Okay. And uh if you do l again you can see it has this test file and then you can do the same for uh the supervisions. EN supervisions and then EN supervisions and then we'll change this train to test. Okay. Now if you see that there are two four uh files and if you count these lines uh if you do wc minus l and then star you can see these two test files uh has 200 lines and then others has 28,000 and then similarly in the supervision also 28,000 files are there. So these are the audio files that we are going to use in the training process. Um so this is it in this video and in the next video we will uh start the feature extraction. Uh we'll go into the feature extraction. I'll explain what is feature extraction and uh what kind of features we are going to use in this training process. Uh after that we'll dive into the training of the model. Uh stay tuned.