Neural Network Voices

Key Takeaways

five days left to enroll in the decentralized applications course link in the description hello world it's Suraj or as my computer generated version would say a little world it's Suraj I don't know how well you could hear that but that was the generated version of my voice he says a lot of things what is this madness this is crazy I like to travel he says a lot but basically he is a deep learning generated version of my voice the demo for today is gonna be us generating our own voices what I just played was this web app called Lyrebird behind the hood they're using what's called a an encoder decoder architecture to generate voices after having trained on your voices so definitely check that out it's called Lyrebird AI but we're going to talk about the technology that it uses and it uses something that is very similar to what Baidu released recently Baidu really recently released a paper called neural voice cloning and it's really rare for any of these big Chinese companies like 10 cent or Alibaba or Baidu to release something open-source so I'm very grateful we should all be very grateful and we should find ways to incentivize these companies to share more of their findings because China is really really really going in on AI and we want to see more open source work from that region of the world so what we're gonna do is we're gonna talk about deep voice 1 be poised to be points 3 and then the latest version of deep voice which is Baidu's text-to-speech system called neural voice cloning basically you can say something and then translate it into somebody else's voice like Kate Winslet for example that's why I have the picture of Kate Winslet one of the authors of this this github library that we're gonna look at later he translated his voice into Kate Winslet's voice oh you could say something enter it into the system and then Kate Winslet will say the exact same thing that you did right so we're gonna look at the code as well it's tensorflow code we're gonna figure out how this architecture works how its evolved over the past two years that Baidu has been working on this and hopefully you'll get an idea of how you yourself can do this as well the link to the github repo is gonna be in the video description ok so let's let's go into this so deep voice 1 so in February of last year Baidu released deep voice Baidu by the way is like - version of Google it's like the biggest Chinese company is focusing on deep learning right now but they released a system called deep voice and what it was was a production grade system that means that means that it was a system that was made for use in production environments for people to actually use that applied deep learning to text to speech synthesis that means you type a sentence and a computer then reads it reads that sentence and it says what that sentence is right so the traditional TTS text-to-speech pipeline consists of hand engineering all these features of like what a voice sounds like what a single word sounds like etc but deep learning does away with feature engineering as you probably know if not it does away with it and said it Lauren's all the features that it needs and so the system could be trained and retrained in just a few hours and here's that here's a really interesting bit Baidu claimed a 400 X speed-up over wavenet so wavenet was deep minds really you know kick ass paper of the Year last year on TTS systems and that's a 400 X speed up on the world's leading AI institution which is really impressive if it's true it's very impressive so yeah China we got it we got a we got to get in on China see we see what's going down in China you know I'm saying so uh-huh so let's talk about this architecture so what they used was an encoder/decoder architecture it was inspired by famous paper called sequence the sequence learning that came out of Google but the model looks like this here's what the pipeline looks like so you say something right you say something it could be like hello world it's Raj and then it it has a text version of that right so there's a text version and what it does is the model will first it'll take that text write all those words and it'll convert those words into what are called phonemes so phonemes are the way of describing what a word sounds like so here's here's what I mean by that so think about the words though and then rough they both end in this Oh ugh right but they sound different so how do we represent that for a computer to read right so the way we represent that that is by using what are called phonemes and there are dictionaries of phonemes online so a a so check this out the phrase white room and the phoneme version of that is w a y1 so a y1 means a certain way of saying H I so you see this sound here H I when we say it in the context of white it makes a certain sound like I write white but if we say H I in another context like hi it's different it's high instead of I rides there's a H in front of it so we could differentiate that using phonemes so one would be like a a y1 and then another would be a Y - right so it's different than just saying like here's what those letters are that consists of the word here's what it sounds like so the first part of the model would convert the text into the phonemes right once it had the phonemes it would use another model to convert those phonemes into a prediction of what the duration was of those phonemes and then the frequency of those phonemes as well so how long do we say those phonemes in the context of of a sentence right how long does that phoneme last right it could be small it could be long and there's you know it could it could be a millisecond difference but there's still a duration for how long it is frequency means intonation like how high how low think about languages like Mandarin right intonation matters like you know there's like four tones in Mandarin like mMmmm you know like that so tone matters duration matters and then lastly once it had all three of those things right it had the phoneme count it had the duration it had the frequency it combined all of those and fed it to a decoder and what the decoder would do would be it would reconstruct the original audio but in the context of the original voice right so you have some text it predicts all these three features and then it's going to construct the voice so what would that the spoken version of the text be and that's what it did it wasn't converting anything into a different voice it was just reading the text but this was a really cool model and this the second step like I said was predicted durations and frequencies and by the way the way they did this the way they predicted the durations and frequencies frequencies by is by using that model in the middle that we talked about up here but that model in the middle is called F segmentation model because it's segmented the audio clips of each phoneme right so you've got the audio of the text you got the phoneme that we're feeding into the segmentation model and it will map one to the other once we've got those will combine the phonemes and durations in the frequencies to output a sound wave that represents the text so they they'd modeled this after deepmind's wavenet so they were inspired by deep minds wavenet architecture which looks like this so the deep mind wavenet architecture we can make an entire video in fact I have one on this it's called generating music using tensor flow so check out that one so wave Mnet's architecture is really interesting because it's able to say how am I supposed to pronounce this in the context of what I'm about to say so it uses what's called an attention mechanism to look at the input data and say okay so let's say we're trying to have a system that generates a speech of hello world and Suraj it's going to look at the word Suraj in the future to see how it should pronounce hello world it's because you know how phrases like they depend on like that there's there's all kind of flow like an intonation flow when it comes to phrases and saying something is always dependent on what what else you're saying if that makes sense so in a sentence you're saying a couple words and the intonation of later words depends on the intonation of previous words it's kind of complicated but it's basically looking at the future to try to predict the past and that's the attention mechanism in this case but basically it was an encoder decoder architecture that's it it had two different models and then in a few months later they released what's called deep voice to basically it was a same model but they just scaled it up meaning they they trained it on even more speakers so they were say scaled it up from 20 hours of speech and a single voice to hundreds of hours of speech with hundreds of voices and each voice corresponds to a single vector so about 50 numbers which summarize how to generate sounds that imitate the target speaker right so if we have some target speaker like me we train this model on my voice it's going to generate a single vector that means a matrix with 50 different numbers in there that's it and that represents my voice and from that the model can generate my voice like what we just did which is really incredible so deep voice 3 was a couple months later so nearing like closer to closer to now which they said let let's do away with that model let's create a new model and it trained an order of a magnitude faster and it allowed them to scale to over 800 hours of training data even more voices 2400 voices so it looked like this so it was still a sequence to sequence model right they had an encoder and had a decoder but the difference was that it used convolutional blocks so recall that convolutional networks are used for image processing however if you look in papers recently convolutional blocks are being used more and more in sequence learning in general right in recurrent networks in encoder decoder architectures because they done so well are CNN for example was used by Facebook research recently to map someone's body in a video and then turn it into a 3d model in real time so convolutional networks are being used more and more in sequence learning like sequences in that case would be frames of a video and what they used was a low dimensional speaker embedding so if they represented every single speaker with the different embedding and once it once they had that embedding they could retrain a model into somebody else's voice which is kind of what led up to neural boys cloning which is their most recent paper which came out just a few weeks ago so neural voice cloning is really impressive it's really impressive because it was an example of few shot generative modeling of speech that means that using only a few samples of my voice like a few sentences it will be able to then generate speech in the style of me so it's like style transfer right taking van Gogh's paintings and applying it to novel paintings but doing it for speech and not only doing that but doing it with just a few examples which is incredible it's incredible right so let's check this out so so what they did was they said let's segment this problem into two different categories there's speech speaker adaption and then there's speaker encoding so speaker adaption is saying let's say I want to say something in the style of Kate Winslet speaker adaption is the problem of saying I'm gonna say something to this model it's gonna listen and then it's going to output something in the style of Kate Winslet but in to do that it's got to have an embedding a representation of vector essentially a vector of what I sound like remember that matrix with 50 numbers in it that represents Raj that is an example of speaker adaption how do we train a model to adapt to not only my voice but to kate winslet sports and then there's a problem of speaker encoding how do we encode my voice how do we represent my voice into that matrix so segmenting the problem into those two different problems helped them build this model so in the case of speaker adoption what that what it meant for them was fine tuning a multi speaker generative model with the few cloning examples and they trained the whole thing using back propagation so it was using back propagation for both the encoder and a decoder and so what they found was when they mapped out all those speaker embeddings that they learned from all the different speakers that they trained on they found that there were clusters around speakers who had the simple who had similar regional dialects there were also clusters around speakers who had similar genders like males and females or clustered together so males from a specific region all clustered together in terms of where they were in this latent space right this embedding space the learned space which is really cool if you think about it there's a lot we can learn from that in terms of how language works and and how different people relate to each other in terms of their language and where they're from so there's that and it was called neural voice synthesis so I thought ok cool let's see let's see what we can do here obviously they haven't released their code but they did release the paper I took a look at the paper it's really the paper check it out links are gonna be in the description but Lyrebird was a way for us to you know really do this with a web app there's also this really great repository that I found it's a it's it's only four months old so it's relatively new and their architecture is very similar to neural voice cloning which is why I picked it for this video but basically they had two networks right just like the neural voice cloning like an encoder decoder type architecture so the first network is an encoder and the second one is a decoder we can also think of it as the first network being a classifier and the second network being a synthesizer or a generator let's just look at the code here so I've downloaded the code and we're gonna look at it right now so under models up pie we can see some of the code so let's let's make this bigger and see what see what's going on here okay so they've got two networks here they've got network one and network two and they're trained sequentially so we were first gonna train Network one and then we're gonna train network two in this code so network one is gonna act as a classifier what they did was they trained it on what's called a timet data set these are six hundred and thirty speakers with the label being the phonemes for each speaker right so if a speaker says you know hello world its Raj the label for that input data is going to be H e1 l2 right the phonemes for the for that for those words and when it comes to input data with labels it's a form of supervised learning so what the first network did was it essentially it was just learning the mapping between the input data and the phonemes right so if you give it some novel speaker like it's whoo-hoo it's never seen before it'll be able to predict what the phonemes are over time so remember neural networks are just a series of matrix operations right and these are just numbers and operations that are being applied to and using some optimization scheme like back propagation or whatever genetic algorithms it will slowly update the weight values over time so that whenever we have some input data it's gonna hit the right numbers in those matrices so then output the the most likely prediction right over time it's gonna be trained by minimizing the error like how far off it is from the actual label so when you give it some novel speaker it's never seen before it's able to predict what the phonemes would be just because there's a likelihood that it's going to be that based on what it's seen before so if we look at this further through the first Network it first loads up that vocabulary right the phoneme - ah DX - phoneme idx - phoneme and basically that those are the labels and the phonemes like the words and the phonemes for all those speakers and then it does some it does a form of data pre-processing but using a very small neural network just to make sure that the data is normalized so they call that a pre net then once it's once it's got that normalized data it feeds it to the network so they call it here in this code see bhg but if we look in the modules see bhg it's actually just a convolutional network right so it's just a it's just layers of convolutions over and over and over again convolution one normalized convolution to normalize convolution three normalized and then if we go back to the models we'll see that okay it's it's it's got the input data it's got the labels and it has the it has the output right so what is a predicted label and then it lastly it does a final linear projection by using a fully connected layer this is usually this is like a very common thing and neural networks to have your last layer of your network actually just be fully connected or a dense layer in the case of tensorflow a softmax are then output a class probability and then the predictions are going to be formatted using two and thirty two and then we return all of those right so that's the first step to train this first network on the input data and the phonemes and once we've got that fully trained network then we can train the next network by the way the loss function they're using for that first network is cross-entropy loss which means we're just going to do them essentially the mean squared error that's just a different way of saying mean squared error taking the mean of the of the actual the prediction minus the actual label squaring it and then returning that for the second network it's gonna say or we've got the first network trained now it generates some new some new audio right so how is it going to do this so when it comes to the new audio that what they used was they used some speaker data set so for Kate Winslet they used two hours of audio books read by Kate Winslet okay so so what I mean is they said let's first input what Kate Winslet says that waveform into the first model right so it's a fully trained model they fed the what Kate Winslet said not what the timid dataset speaker said what Kate Winslet's said to that fully trained model now what's it gonna output after having been trained it's going to output the phonemes exactly yes once it's got the phonemes for what Kate Winslet would say then it's gonna feed those phonemes to this second network right which is the speech synthesiser network again it's going to do some data processing using this pre network again then it's gonna feed it to the same kind of convolutional network right it's a similar convolutional network but it's in the reverse order right so remember neural networks are just a series of matrix operations right you have some input data you do some matrix multiplication you do an activation function and then you repeat and everything's really a glorified version of that this is a matrix multiplication over and over and over and over again and when it comes to deep learning research deep learning research is just playing around with what types of operations were using what sequence were doing those operations in and in the end some of the best models are just researchers playing around with what should come first or should we tried this multiplication here or should we divide this and then use the mean there's this great xkcd where the guys just like how'd your deep learning model work so well and the other guys just like I don't know I just jumbled up all this linear algebra together and it made its great output that's basically the appointing research in a nutshell when it comes to the second Network it was able to generate audio from the phonemes so the way it did that was it used a different type of loss function and the different types of the different type of loss function was called a reconstruction loss so we could see it right here so it used a reconstruction loss it had the phonemes and using those phonemes it applied the series of matrix operations in this convolutional Network and at first it knew nothing right so it's just here's the output a jumble of numbers and so that jumble of numbers they compared it to the original what Kate Winslet's said right what did it what did Kate Winslet originally say that's the actual output so then it computed a reconstruction laws which is how well could it reconstruct what Kate Winslet said from the phonemes that were generated from the first model you see what I'm saying it fed what Kate Winslet said to the first model output phonemes fed the phonemes to the second model output what the reconstructed version of what Kate Winslet said which is actually just a bunch of numbers at first it compared that output to what Kate Winslet's voids originally was that's the Reconstruction era the loss between those and it used that difference and it minimized that difference over time using back propagation okay so over time then the second model was able to better reconstruct the Kate Winslet's voice over time now that's just Kate Winslet if we kept feeding at different speakers it would learn multiple speakers right so because we construct all of those different speakers over time but they just fed at Kate Winslet so eventually right after training both models in that way so the first was a classifier trained using cross-entropy laws the second was a generator and the loss was instead a reconstruction loss the first was trained on a million speakers the second was trade on just Kate Winslet so then if I say something it will go through the first create a phoneme list that PETA to the second and then it will read it will reconstruct it as Kate Winslet after training so that's that's how they did that if they didn't using tensorflow the model works the guy even has a demo of it as well work with ends working with units on Soundcloud he called it voice but the culture the revolution thus far had exhausted the Janka but to culture the revolution was fired and exhausted beyond er its Cynthia so yeah you can check out the demo there as well but yeah pretty cool stuff all the links that of but the stuff I've talked about are gonna be in the video description and I hope you found this video useful