Stable diffusion Animation tutorial with AUTOMATIC AUDIO SYNC. Make your own AI music video!

Key Takeaways

Hello friends. Today we're going to talk about how I made this AI music video which is uh automatically audio synced. So I thought we were going to have a look at that and I'm going to show you how I made it. Uh let's skip to skip here so you can actually see the the audio syncing [music] happening here is that every time the drum hits the AI zooms into the animation. So I'm going to show you how that uh how that works. So, first of all, what I did is I created a a start. We need a starting frame for the animation. And I made this by setting a prompt, this robot Android. And I found one that I liked. I picked this one. And I saved all these settings. And I put them into uh the forum, which is a stable diffusion collab that works great for AI animation. And I saved that prompt. And I I put it down here, animation prompts. We'll look at that later. So, first of all, we got to decide what part of the music is going to be audio synced. And I choose the drums because that's the most efficient way most of the time. If you have a song, you can fine-tune it in Premiere or Adobe Audition or whatever you're using. There are also AI tools for this. I have used this site in the past. And uh what you can do is you just um enter your file here and it will give you an input. So if you listen to this, this is the original song. And if you skip up to where the beat starts, this is the song without the drums. Now they actually separated the drums. So if you listen to this, you can hear. [music] So this is what we want. You can have 10 minutes free trial here. So just get these drums. And when you have that file, you're going to input that into this audio to key frame string generator. So I've prepared the file, put it in here, it loads the file and it generates key frames or actually the volume of what's going on. And if you change it, we are going to use disco. But if you change it to CSV just to see what's happening, you see that for every frame, this is one frame, two, three, four, five, etc. For every frame there's a value. Right now the value is one. This is the default setting is 1 + x which which is the volume that is reading to the power of four. So when the volume strikes you should get 1 something. If we scroll down we can see here. Yeah we're starting to see some values here. It's 1.47 for example. And the reason that it's one is that in the forum there are different choices when it comes to animation. You can work with angle zoom translations uh which is um moving the camera, left and and up. I've set this to 2D, but um for 3D you can move it in in um all three dimensions. If you're using a value like the zoom that I've been using, which is moving into the image, it's a multiplier. So one is nothing. So 1.1 would mean that it's moving in and 0.99 would mean that it's moving out. So what we have as the default value here is actually working pretty good for the zoom. If you would want to change for example uh the translation which is not a multiplier but works from from zero and up or minus you would need to change this one to well nothing. you would have the value x depending how you know strong you want it you can make it to the power of two or three or four or multiply it by you know value. So x will always be your volume and then you can just decide how much more you want of it. But we want one uh plus x by the power of I think I set two for that one. Uh but the default is four. Anyway, so what happens here then is at every frame you've set the value of one which is coming from here and then X which is zero because there is no volume in the first frames. So you have no zoom, no zoom, no zoom, no zoom. Um, and that's, you know, up to like frame 300ish or something. And then you start zooming in. And then you have the angle parameter here, which is basically a 2D rotation. So at frame zero, which is the start of the animation, we have an angle rotation of one, which is a positive. So it's going to move to the right if I'm not mistaken. And then at frame 50, it's going to go to minus one, which is a rotation to the left. And then at frame 100, it's going to go back. You see a pattern here. It goes 1 - one one. And then it ups to four. So that's a faster rotation. Then it goes slower. And then it reverses reverses again faster, slower, reverses, reverses, angles faster. So you can play around with with these values between different frames and and whatever. Uh it's a lot of fun, but it also takes a lot of time to change these values as the they're manual. All the angle values I set manually and the zoom value is based from this um audio to key frame string anyway. So let's go back. We need the disco value here. That's what you're going to use for the forum. And then it's very important that you change the frame rate to what you're going to be using. So you have to consider what you will be using in the final product. So if you set like 24 here and you end up using 25 or 30 in your video, this is not going to sync and you render the full animation, you know, completely useless. And due to the render time of these animations, you you're wasting a lot of time. So think about what frame rate you you want to use and um stick with it. And seeing as let's talk about this mathematical function. Seeing as this is a power of so you would think that a power of four for example is bigger than a power of two but is not because this x value is actually 0 something. So let's say here you have a volume strike. Let's talk about the x. So the x could be zero. It also be like 04 0.1. So here's a big volume change. Here's a small volume change. And then you have the one which is our multiplier we have here. So you have one plus the say you have a big volume change of 04 and the power of two. So that would mean let's bring up a calculator 4 * 04 1. So let's do 1 + 0.4 4 4 and again 0.4 to actually get a smaller value here. So it would be 1 0 0 2 56. So because this is smaller than a one the power of you know can I reverse go smaller. If you're a math wizard please explain better in the comments. I'm not it. I tried to the best of my ability. If you if you're using the default to the power of smaller makes bigger and vice versa because it's a 0 something value. So this just copy all of that. Go back into your forum. Put that into the zoom. Now let's talk a little bit more about the settings here from from the start. Uh you need to set the animation mode to 2D which was set to nom at first up the max frames depending on how long the uh your song is. So you can just talk about how many frames per second you want um your video to be. So you just you know multiply the frames by your um by the seconds of your song and you have the value for your frames. Uh the strength schedule is how much each image changes. So we're going to have a sampling step of 50. If you have a sampling step 50, you would take that times 75. That would mean each subsequent frame would have uh this minus this. So 12.5 samples per frame depending on the value here. Each um new frame will get more or less samples. Play around with it to see how that works for you. You don't need to change a lot of here. This is a new setting for the forum version before. It's basically blends frame into the next frame. So if you want to save a rendering time, you can set this to two or three. That would mean like if you set this to two, it will render frame one. Then it would render frame three and it will blend frame two between them. And then we will move forward, render frame five, render frame seven, render frame nine, etc. And it will blend the frames in between. So it's a timesaver. It can make uh your animation smoother, can also make it messier. So play with that. Um for this animation, I haven't changed any of these values. They're default. This, let's talk about this. When you're starting your animation, this doesn't matter at all. But if you quit it, and it might, you know, if you're doing a big scene, might take hours and you uh need to go back, you can resume where you started. Look at your images, your files, take the time string, put it in here, check that box, and it's going to resume from wherever you were. Here's where your prompt's going to be. Here are the prompts for the still images, but we are not going to be using that at all. We're going to be using animation prompts. And these are based by frame. So here at the start of the animation, this is our frame that we had from this image. So if you start having a starting frame, that's what you should do. You should have the same settings and then match that up in the forum. And then if you want changes in your animation, make new prompts dependent on where you want them. So we're starting here at frame 200. We're getting robot birds flying up towards the sky. at frame 400. Clouds and planets in space like a nebula. You see how that works? You know, when you want to change, put a frame number, put a new prompt in there, make sure you have a comment. Pretty self-explanatory. Uh again here, make sure you get the same settings in if you're having the starting um frame. I ran this at 6 by 320 and then I upscale it, I think, four times. uh batch name if you want to be saved into a specific folder. So, I'm going to run this and um show you how it works. It's going to take some time to start all of this. And I if I haven't mentioned that already, I'm going to put all the links down in the description, but I strongly recommend the forum if you want to use AI animations. And it has a super active uh community and a discord. you can ask questions and u there's some great help there. Now, we didn't talk about setting this up, and most of the time, uh, collabs are easy to run, but there's one thing. You need to download the model. You get that from hugging face. I'm going to put a link in the description. And then you need to put it in, uh, your drive AI models. There you have it. And that's what's uh, referenced here, model checkpoint. And as you can see here, the rendering has started. Rendering animation frame zero of 5,500. And this is our starting frame. You remember it from the first. And as you can see, the image is starting to rotate. And that's our angle value that we set cuz the zoom isn't happening till frame 300ish. So, it's very cool that you can see this live and uh the files are saved in your Google Drive. And uh there's actually an option here to create the video uh from your frames. So you could just easily uncheck that, set your frames and when you have uh rendered all the images, just press run here. What I did, I took all the frames and ran them in Premiere to have more choices when um post-processing the video. So yeah, this is how I made the AI music video for Grammatics, the ghosts of Pyon, which were automatically audio synced. Hoping it's going to help you make something cool. If you like this content, please press the like button and subscribe and that will help me a lot so I can create more of these explainer videos. So yeah, good luck, have fun, make some cool animations. And uh if you make any animations based on my tips here, please, you know, give me a link in the comments so I can look them. Would be super cool. Bye-bye.