Easy Text-to-Video in Python | Python Tutorial with Damo-vilab Model

Key Takeaways

hi everyone this is Smitha from assembly Ai and in this video we're going to be looking at how we can convert text to video in just a few lines of code in Python so let's get started for this tutorial we're going to be making use of the demo B lab model which is created by model scope and this model essentially converts text to video using a diffusion model I'm going to explain how this model works but before that let's actually jump into Google collab and download this model to start running it once you've opened a file in Google collab this is where we'll be writing all our code it's going to be very simple the first thing we want to do is make sure that we are using GPU resources because we need that to run this model so the first thing you want to do is click on runtime and click on change runtime type and just ensure that you have selected GPU and click save once we have done that we can start writing the code to actually download this model and set it up the first thing we're going to be doing is installing three different python libraries that we need the first thing is diffusers Transformers and accelerate so that's what we're going to be doing so pip install diffusers Transformers and once you have written this command let's run this to download all of these models next we want to write some import statements to import different libraries that we need so the first thing is torch and then we want to create a pipeline of how this will all play out so create pipe equals to Fusion pipeline Dot from free trained and we can actually go back here and copy the name of this entire model and we're going to paste it right here so in this code what we're doing is we're importing torch we're importing diffusers refuses as a library which helps to do diffusion and in the context of generative models such as this what diffusion actually helps us with is generating videos from noise so from gaussian noise we are trying to generate some sort of video or output and it works alongside the model as well in order to do this and and we're going to create a pipeline of all of this along with the pre-trained demo vlab model which we have installed and once we've done all of that we can go and hit run if you are downloading this for the first time it's going to take a little bit of time and while you wait for that to download let's actually go back to the model page and let's take a deep dive into exactly how this model works so first off the demo vlab text to diffusion model operates in three separate stages first off we have a text feature extraction model which means that the model takes in the text input that we give it and then extracts its core meaning to understand exactly what we're looking for in the video next is the text feature to video latent space diffusion model and this phase actually Bridges the gap between the text information that we have and also the videos so it actually is creating an abstract definition of what a potential video output can be so in this stage the video has not been created but it's just trying to get an idea of what type of video we can be creating in the final step we have the video latent space to so what this means is that the abstract representation which was created in that second stage actually turns into a real video and this video generation starts from your gaussian noise and essentially that refers to something like white noise if and this final stage actually takes this gaussian noise and turns it into a final video now let's hop back into Google collab and actually give it the prompt to create a video so we're gonna do prompt equals to Spider-Man is surfing so a video of Spider-Man surfing and we have to set some parameters such as the video frames number of inference steps to 25 if you increase the number of inference steps that will increase the compute time as well foreign so in the very first line we give it the prompt and next we're creating something called video frames which is a pipeline which includes our prompt and also redefine the number of inference steps that we want which we set it to 25 we also Define the video pad so where exactly are we planning to export it to and then we we create the video name which which ensures that this video will be created in a folder called TMP which exists right here and we're also printing out the video name so we know what what the name of the video will be the output video and finally what we're doing is we're actually emptying the cache of the torch cudas because we obviously don't want it to be blocked up in case we're running multiple prompts once you've done that let's hit run once it has generated this video you get an output of the name so we can go into folders click on TMP and you should see it right here and you can actually download this into your local machine and this is the video in my local machine and this video only plays with the VLC player so make sure that you have that as well and there we have this video it's a very short video as you can see and it's not super high quality but this is a great starting point for generative Ai and being able to actually create videos directly from text and this is an awesome starting point so feel free to check this out hope you found this tutorial helpful let us know what you guys thought about this in the comment section below and like And subscribe for more amazing AI content