This AI Makes "Audio Deepfakes"!

Key Takeaways

your fellow scholars this is two minute papers with this man's name that is impossible to pronounce my name is dr. Khurana if I hear and indeed it seems that pronouncing my name requires some advanced technology so what was this I promise to tell you in a moment but to understand what happened here first let's have a look at this deep fake technique we showcased a few videos ago as you see we are at a point where our mouth head and eye movements are also realistically translated to a chosen target subject and perhaps the most remarkable part of this work was that we don't even need a video of this target person just one photograph however these deep fake techniques mainly help us in transferring video content so what about voice synthesis is it also as advanced as this technique we are looking at well let's have a look at an example and you can decide for yourself this is a recent work that goes by the name techo Tron 2 and it performs aei based voice cloning all this technique requires is a 5 second sound sample of us and is able to synthesize new sentences in our voice as if we utter these words ourselves let's listen to a couple examples the Norseman considered the rainbow is a bridge over which the gods passed from Earth to their home in the sky take a look at these pages 4 Crooked Creek Drive there are several listings for gas station here's the forecast for the next four days Wow these are truly incredible the timbre of the voice is very similar and it is able to synthesize sounds and consonants that have to be inferred because they were not heard in the original voice sample and now let's jump to the next level and use a new technique that takes a sound sample and animates the video footage as if the target subject said it themselves this technique is called neural voice puppetry and even though the voices here are synthesized by this previous taiko Tron 2 method that you heard a moment ago we shouldn't judge this technique by its audio quality but how well the video follows these given sounds let's go the president of the United States is the head of state and head of government of the United States indirectly elected to a four-year term by the people through the Electoral College the office holder leads the executive branch of the federal government and is the commander-in-chief of the United States Armed Forces there are currently 4 living former presidents if you decide to stay until the end of this video there will be another fun video sample waiting for you there now note that this is not the first technique to achieve results like this so I can't wait to look under the hood and see what's new here after processing the incoming audio the gestures are applied to an intermediate 3d model which is specific to each person since each speaker has their own way of expressing themselves you can see this intermediate 3d model here but we are not done yet we feed it through a neural renderer and what this does is apply this motion to the particular face model shown in the video you can imagine the intermediate 3d model as a crude mask that models the gesture as well but does not look like the face of any one where the neural render adapts the mask to our target subject this includes adapting it to the current resolution lighting face position and more all of which is specific to what is seen in the video what is even cooler is that this new rendering part runs in real-time so what do we get from all this well one superior quality but at the same time it also generalizes to multiple targets have a look here you know I think we're in a moment of history we're probably the most important thing we need to do is to bring the country together and one of the skills that I bring to bear and the list of great news is not over yet you can try it yourself the link is available in the video description make sure to leave a comment with your results to sum up by combining multiple existing techniques it is important that everyone knows about the fact that we can both perform joint video and audio synthesis for a target subject this episode has been supported by weights and biases here they show you how to use their tool to perform phase swapping and improve your model that performs it also weights and biases provides tools to track your experiments in your deep learning projects their system is designed to save you a ton of time and money and it is actively used in projects at prestigious labs such as open AI Toyota research github and more and the best part is that if you are an academic or have an open-source project you can use their tools for free it really is as good as it gets make sure to visit them through w and be calm slash papers or just click the link in the video description and you can get a free demo today our thanks to weights and biases for their long-standing support and for helping us make better videos for you thanks for watching and for your generous support and I will see you next time