Audio Data Processing in Python

hey YouTube my name is Rob and I a data scientist I make videos about machine learning and coding in Python in today's video we're going to be learning about working with audio data in Python there's so many cool things you can do with audio and python but it can be overwhelming at first because you can't necessarily visualize the data in the same way you can tabular or image data by the end of this video you should be able to load in an audio file and explore some of the features of it and prepare it for a machine learning algorithm but before we get too far if you enjoy these videos please consider liking and subscribing to my channel I'd really appreciate it now everything I'll be doing today I'll do in a kaggle notebook I'll link that in the description below so you can click that link and explore the code yourself all right with that let's get started so here we are in the working with audio and python notebook that I've created before I get started in this though I want to show you the audio data set that we will'll be working with and I'll show you that here I've imported it here on the ad data side of the right side of the notebook the data set we'll be working with is called the ravness emotional speech audio data set and it contains a bunch of different voice actors saying the same phrases in different emotional tones so let's just go here and show you an example of one dogs are sitting by the door okay and a different version dogs are sitting by the door interesting so this is the audio data set we'll be working with so before we get too far we're going to have to do some imports and we're going to do the standard import so import pandas as PD import numpy that lets us use work with Vector arrays we're going to import some [Music] some packages for plotting like matap plot lib and we're going to import caborn as SNS we're also going to use a package called glob and glob allows us to uh list out all the files in a directory which will be helpful when we want to read in a bunch of the wave files from this data set um and then the main package that we're going to import for working with audio data is called Li Rosa so we're going to import Li Rosa and we're going to also import libr Rosa display I want to be able to play some of these audio audio files in the notebook so for that I'm going to import uh I Python's display module so import IPython display as IP PD and then I'm going to actually paste in a few more Imports and these are mainly just for colors and making my plots look a little bit nicer so let's go ahead and paste those in here run that cell all right so we got all the Imports now we're going to just briefly talk about some terminology you'll need to understand in order to work with audio files in Python so so not to get in too much detail but we're going to need to learn about frequency of an audio file so the frequency just think about it as what describes the differences in wavelengths in the file so every audio has some sort of frequencies or multiple frequencies within it and this image sort of shows you the difference between a low frequency audio and high frequency or short wavelength now the second thing we have to keep in mind is the intensity of the audio file itself so that's not only the wav length which is the frequency but how high do each of these waves get for each frequency itself it's change in Pitch is is uh frequency here on the left but the change intensity is the loudness and the way we measure that is in power or decb and then the last thing we need to understand is when the computer deals with audio it actually has discrete observations of the audio file and it's not continuous like true sound that's coming out from um something that you might hear and the way that we measure how frequent those samples are taken of the audio file is by using some something called sample rate so think about the sample rate as the quality or the detail um about the audio file that we're taking in or the resolution of the audio and this image I think is great because it sort of shows how a high sample rate kind of gets more details into each wave of the audio file and more than the low sample rate and there are certain sample rates that are commonly used because at those sample rates our ears can hear most of the sound um at or above that sample rate so this data set consists of a bunch of folders that each have wave files inside of them and we can use the package glob to find list out all those files so I'm just going to go to this directory and you can see that there are folder for each actor and and inside those there's actually a file for each recording we have of that actor and I am just going to make a list of all the wave files in this entire data set by putting stars in here where I want glob to replace this with any files that exist um so if I run this I'll have a list here of all the different files in their file location so I'm going to call this audio files and let's go ahead and try to listen to one of these audio files so if you remember we imported IPython display module and I can do ioon display. audio to one of these audio files let's do the first one in the list and it's kind of nice here in the notebook let's call this play audio file it's nice cuz in the notebook we actually get a player down here where we can listen to it kids are talking by the door okay so we can hear it fine but we haven't actually read in the file and and don't have the data to manipulate when we do this this is nice cuz we can listen to it but we want to actually deal with the raw data and the way we load that in is with li Rosa so lib rosa. read allows us to read in a file and we're going to take this same audio file pass it into the read function and it'll output what we'll save as Y and Sr for sample rate now Y is going to be the raw data of the audio file and Sr is going to be an integer value of the sample rate that we were talking about before so let's go ahead and run this and this is load not read and now we could see we have a numpy array like this that's all the audio data really is it's a long numpy array and we also have the sample rate here so I'll print this out and it's 2205 why don't we go ahead and print this so let's print y looks like this let's just print the first 10 values and print the shape of Y is that and print Sr is Sr so we see here the first few values of this array are just zeros we have the shape of the audio file we brought in and we also have the sample rate very nice all right let's continue on from here and actually plot this data to get an idea of what it looks like so the numpy array um is a little bit easier to plot if we just make it into a panda Series so let's take pandas and wrap this array as a series now we have a panda series of the audio data and then we can just plot use the plot method off of the panda array so let's go ahead and give it a fig size something that'll make it nice and wide oh there we go we have our audio file and a few other things so just because we want to make the line width smaller so we can see the audio better let's do lwal 1 that's the line width and let's give it a title which is raw audio example go ahead and plot that we do plot that show so it actually um displays without giving this this result here and looks nice well I guess we could do color is color pal zero which won't change anything but we'll be using this again later we notice in this audio file that there's a lot of Silence be at the beginning and the end of the audio file and liosa provides some nice effects that we can apply to it that will remove those blank spaces so so let's do librosa effects. trim and this will allow us to trim this audio we'll just feed it Y which is the array that we had and what it outputs is the trimmed value and we'll just not use the other output the index of the values that it returns we don't need those so we'll trim it and then we'll go ahead and run this exact same code as before and call it trimmed example let's also change the color of this and we're going to run this on the trim data there we go oh it actually looks pretty similar and that's because this trim effect has uh that we can give it and that that's the top deciel that it decides to trim off at so we actually want to lift up that threshold or actually lower that threshold and we can do that by changing top DB to 20 instead of the default 60 and now look we could see that it has trimmed it might have trimmed a little too much but it looks like we've cleaned up the audio file and only have the area where with the the sound that we're interested in so with this plot we're only looking at the raw data that's only uh not too helpful to us because of there's so much detail in here we're not actually being able to visualize but just as an example we can try to zoom in manually on one of these areas so why don't we take this plot from before for y and we'll make this called zoomed in and we'll take this and actually look at let's look at from 3,000 here or 30,000 let's just slice this at 30,000 and go to 31,000 and now we can actually Zoom having zoomed in see some of the Audio Waves that we were talking about before so you you can see there are multiple different frequencies at different times overlapping a little bit and this gives us a little bit more um of an idea of what's going on let's zoom in even more yeah so this gives us a real good idea zoomed way in of what the audio data looks in looks like so that's nice but we want to take it another level and actually look at the different frequencies by how powerful they are are and the way that we can do that is by applying a 4-year transform to the audio data basically what that allows us to do is extract out which frequencies are sounding at different parts in the audio file the way we do that is we run librosa sft and that's shorttime forear transform and we're going to run that on on our Y data and save it as D for our transformed then we're going to take that output and we're going to apply another transformation to it and that's the amplitude to decibel transformation that'll take this these values from an amplitude value and convert it into deciel which is commonly used transform for audio data so we're going to take the absolute value of this D apply this transform and then we just do this as a reference for what the max volume volume would be um and let's go ahead and save this as our sound in decibel form now we want to visualize this data and we can first just check and see what shape it's in uh so you can see that the 4year transform has converted it now we have a numpy array of shape 1025 and 1 153 and let's go ahead and plot this and this is the sort of data that we could actually feed into a machine learning model so we're going to with map plot lib just make a place where we'll throw our plot let's make the Fig size something reasonable and we can use libr roa's display package to make a spectrogram image of this data so we're going to pass in sdb that we've created before and the x axis is going to be time and the Y AIS will be uh the log or the decibel and we'll feed in that axis that we've created up here in the subplots this is just our way of visualizing it wow so interesting we can see the different frequencies here on the left side and over time how they change in intensity we probably want to add a title to this so let's set the title and we should also add a color bar so we can see what values relate to what colors here in this so let's do color bar on our image accesses that and do a format of uh 0.2 F there we go and let's go ahead and do plot that show great so now we have a spectrogram example of this audio data and this is something we could use on a machine learning model so next we're going to create what's called a Mel spectrogram what's a Mel spectrogram you might ask well same thing as a spectrogram we did in this previous example but the Mel s actually stands for melodic because we're going to use this transform to express the frequencies that we can here in audio usually so we're going to go ahead and do a very similar thing to what we did before but instead of using the sof forear transform we are going to apply a straight Mel Sprint spectrogram onto this so librosa feature. Mel spectrogram we're going to feed it in our audio file and we're going to give it the sample rate which we pulled in from the audio file when we we read it and provide it a number of mels so uh 128 sounds fine so we have our Mel spectrogram data set here and we can see that it has 128 by 153 the 128 is the number of mels that we asked it to provide and then we're also going to take this and apply the same um transform amplitude to decb like this and we'll call this sdb Mel and then we want to plot and we can use the same code from above let's just pull this in call this Mel spectogram and we're also going to have to provide it the subply there we go let's up this let's make this twice as many Ms so we can see here that the area of the audio that we can hear is a little more accentuated and um now we have a Mel spectrogram now this data here that we have this sdb ml we could run on the entire data set and create a new a bunch of features to feed into a machine learning model and train thanks for taking the time to watch this quick tutorial on how to work with audio data in Python I hope you learned something new and if you enjoy the video please give it a like And subscribe see you all next time