You Only Need Audio To Deepfake Now! Might look slightly cursed tho [PCAVS]

Key Takeaways

deep fake has come a pretty long way it started from stitching people's faces onto another person's face then came along the process of animating a still face with just a video and now you can just simply use audio to create basic lip sync i have a pretty horrific video about the lip sync ai wave 2 lip but i thank you not to check it out because it's so cringe to make the above evolution chart easier to understand the bakamitai meme stands around in the middle it was hilarious in its own way because of how badly the faces were animated but what if it's decently animated i'll probably ruin the fun but i'll show you later in the video so in this episode of beating a dead meme let me introduce you to pc avs short for post controllable audio visual system it's like a first order motion model and wave to lib had a baby not only it can lip sync any videos with just audio listen i will be more ready than i was in 2012 because i will have done my job it can also copy the head movements from another video too the flexibility of this ai is just incredible and this is because this ai model is built upon three main parts first is the input identity which is the part that focuses on generating and manipulating the face you choose as an input this part controls how the face looks like and how consistent it will be when animated and moved around to put it in perspective fom has a worse identity generation consistency when the face moves too much this is what caused the face to deform in the meme while pc avs generates new inconsistent facial details with limited information from the input but there will be a consistency problem on regenerating the faces even though it's a pretty good solution the second part is basically the information about input posts you want to use as a reference whether the face looks up or down this part will make sure to transfer the head movements aka the pose onto the input identity the last part uses the audio spectrogram and synchronizes it with the visual features which are the lips combining all three of these parts a much more consistent and flexible face animation is then born with respect to the three parts three files are then required to use as an input an audio a target face and a pose you can use it in the intended way like on their official demo or you could turn something into a speaking image by making the target face and the pose both the same thing or just slightly create some movements to make the speech look a bit more natural this offers a lot of possibilities and functionalities unlike wave to lip which can only make the lips movements in sync with the audio or unlike fom which can only move around the facial features that may make it look slightly awkward the only functional wise downside is that currently you cannot just use a video without an audio to generate a talking face like how fom can just animate other faces with its driving video since the audio is the basis of this ai it means that the lips movements are based on the audio spectrograms so you need a clear speaking audio to make the lips move as intended this means i will need to manually sing bakamitai myself if i want to reproduce the memo this is some really fun stuff even though the output quality is pretty low unfortunately we won't be able to increase it for now as it depends on the identity generation part so basically the resolution is stuck here because of the architecture of the ai model if you want to try it yourself i'll link my tutorial down in the description it's a pretty messy setup as the original codes were all meant for the linux system but if you are patient enough you can give a try or you can just work with today's sponsor 27 stars to set up and run these ais for you or for your business 27 stars is a linden based development company that creates custom tailored web and mobile applications for individuals or businesses of all sizes they are really experienced nice and friendly people to work with and they are also providing an exclusive 10 discount for all of you guys if you choose to work with them all you have to do is to include my name in the initial email to receive the discount and by working with them you are also indirectly supporting me too which allows me to dedicate more of my time to work on these fun videos thank you so much for watching also a big shout out to md and many other patreons that support my work through patreon you can share your generated results over on my discord channel or if you have any questions feel free to put it there follow my twitter if you haven't and i'll see you in the next one