Stable Video Diffusion - First Impressions!!!

Key Takeaways

everybody has been talking about SVD I was surprised why would people talk about SVD is there a new innovation in recommended systems because SVD the one I know is singular value decomposition it's a very popular technique in recommender system but that's not what we going to talk in this video this SVD that everybody is talking about is stable video diffusion stable video diffusion why is it stable video diffusion maybe they don't want a lawsuit from stable diffusion original research team so anyways the point is stability AI created a new model that is called stable video diffusion where it can take an image and it can turn that image into a video it's quite impressive I stability AI team claims that it is better than or it is on par with the close Source models that they have tested like something like let's say Runway ml or P models in this video we're going to break down how the model is and we're going to break down what does it take and also we're going to see finally some demos that they have built using stable video diffusion let's get get started with the video the first thing is what is the what is a product this is the announcement that came couple of days back uh stability AI the company that develops a lot of models that we have used has created a new model the open open they call it the first open uh not necessarily the first open this is stability ai's first open generative AI video model and you can see that you can upload an image and it does a pretty good job in animating the image like you get the image as a video so the end result is 2 to 5 Seconds um you get it um maximum 2 to 5 Seconds of the video and the way you can get this is you can get this at different frame rates so because you can get this at different frame rates you can do a lot of different things with this so stable video is released in two in the form of two image to video models capable of generating 14 SEC 14 frames and 25 frames at customizable frame rate between uh 30 and 3 and 30 FPS so it can generate at 3 FPS and it can generate at 30 FPS once again if you're working in the animation industry if you're working in the movie industry or if you are familiar with videos you know that having uh the ability to get it at different frame rates can help you do a lot of things like for example you can do certain slowmo thing you can do certain fast things uh it's it's quite good uh when you get it at different frame rates and it says that it processes at 2 minutes or less but if you go to their demo you can see that on A1 100s it takes 60 seconds I'm not sure if you can fit this model on Google Golab I've not tried it yet but um this is this is where it is so from a single image it can generate like a 4C video at uh in this this particular demo is at least 25 frames at 66 FPS so for every second it is six frames and totally it generates 25 frames and this they' have shared a research paper I don't want to start cribbing at the start of the video but I cannot stop from cribbing because stable video license is what the license on which this is released so the license is called stable video license it's not an existing license it is a new license that they've created now what is the stable video diffusion license what is this license if you see this license you can go read the license but the main point is this license is under a non-commercial Community license that means unlike the stability ai's previous model of stable diffusion Excel SD XL where you can use that model in production to build any product that you want this particular model you cannot use it to build a production application or commercial application to be be specific if you're going to make money with this you should not do it you cannot use it for any commercial purpose other than research purpose which I kind of understand stability this is what they're calling as research preview they don't want to make everybody use this model and build on top of it which I can understand at this point I'm definitely looking forward to to see stability AI making a clear statement about what is it going to be in the future that is one thing but if you read this the license reflects stability dual commitments to making its research widely available while working to ensure that the AA models are used to benefit Humanity I never I don't know maybe this is part of their mission or Vision but I never knew that stability AI is working to make AI models to benefit Humanity I don't know why every company on the planet has to work to benefit Humanity I mean is this is this some um Silicon Valley uh story I don't know um but anyways the point here is that um you cannot use it to build commercial model so if you have been dreaming at the start of the video to make commercial model out of it you cannot do it and I'm definitely looking forward to hear from stability AI or Imad to see if this is not going to be what the case in the future because if that is what they're going to do then you know it's it's a a totally different comp company from what I dreamed of but at least now it is available for research non-commercial purposes you can get the model code and weights directly so the model weights are available on hugging faces model Hub you can just directly go here and you can see the model weights like the safe tensors are available for you to use it you've got two different um files one you have got the normal SVD file and then the second one is you have got the image decoder and you can go here and then see the specifics about what you can do with this and uh you can see some of the user valuation that they've done so they have uh taken the two models one is the SVD stable video diffusion 14 frames model and then the second one is a 25 frames model the 25 frames model scores much higher than the existing Runway and peaps Runway is kind of like the the boss of video creation at this point AI video creation so you can see the stability videos SVD is 25 it sounds to weird to say SVD because uh in my mind SVD is always is the the one that I said like singular value decomposition for recommended system so anyways SVD U for SVD is um better than Runway and P labs and the 14 frames is almost on par with uh Runway and P Labs um so that is where it is like uh the user preference stability has some good user preference one of the things that I honestly wanted to point out from the paper is uh while all these things are good is um if you see the paper there's one very interesting specific detail that you can see that this model while this being a pure text to image generation model I think this model has capability to do much more than just being a normal text to image generation model in fact you can see that it does uh something called multi view synthesis so this is I think um they've done fine tuning but you can see multi view synthesis you can give an object and then you can have a view of that object of different directions I remember way back when we had um stable diffusion um like I think before even stability AI I remember covering a video about a Lura that was doing like this um all the 360° capture of an image that you generate from stable diffusion I think this has been a dream for a lot of people because that gives an ability for you to enter into the 3D World and this model has the capability of doing it which they also acknowledge and then mention it in the abstract of the paper so you can go here and then see uh what all things that they can do so finally we demonstrate that our model provides a strong multi view 3D prior and can serve as a Bas to find you a multi- view diffusion that jointly generates multi view of objects in a feedforward fashion uh outperforming image based Methods at a fraction of their compute budget so so the point here is that even though this is like a pure play text to video Generation image to video generation model at this point this can actually become more than that having multi view different camera angles because this can probably give you like lower options to create different camera angles and it also can give you an ability to create like a 3D world in the future the second thing that I wanted to point out here in this is um the way they did image um frame interpolation so if you see frame interpolation uh so that is one thing that you would see lot of text to image Generation image to video generation model suffering so typically most image to video generation model works like this you give an image and based on that it will generate like 25 30 images and uh it's going to stitch together so you have a 30 FPS uh 1 second video this is typically what most of this video generation models do most of the models that are not Runway that are not peers suffer from this image frame interpolation problem because one frame and another frame are not very coherent so when you see it you would easily say that you know something bad is going on so they have taken a nice approach so to obtain smooth videos at higher frame rates we fine tune our higher resolution text to video model into a frame interpolation model so they have a frame interpolation model which they fine tuned from the higher resolution text to video model and um we follow a particular approach they are mentioning Blackman's approach and concatenate the left and the right frames to the input of the unit unit is one of the neural network part that is available masking the model learns to predict three frames within the two within the two conditioning frames effectively increasing the frame rate by four so this increases a frame rate that ultimately helps you get much smoother video this is something that I found pretty interesting apart from the multi view generation you can go to the end of the paper and then see a lot of different details about what they have done and how they have done and uh you know uh different sample images you can see like for example in this case you can see this is the input image like the conditioning image based on the that you can see how different images are generated which is finally going to be stitched together and create the video I came across couple of demos which are like mind-blowingly impressive so this is one of the videos somebody created with stable diffusion video SVD so this entire thing is stable diffusion video it is so surprising to see how far we have come in terms of image to video generation and um um generally AI video generation you can see somebody has tried to create like a Steven sped ber Style movie with this and uh you can also see the frame interpolation sometimes you can see still there are problems you would see when uh these frames as being stitched together but I would say like this is being this is the first version of this model I I think like this is a pretty brilliant model at work it's a shame that this model is not available for commercial use but I understand that I don't think they they want their competitors to copy this but um but yeah I think it's a fine balance in terms of Open Source research and Commercial commercializing your products this is a pretty good um um video the second one I saw which I link it in the YouTube description you can see here is that you can see the jelly movement and uh this is like straight out of stable video diffusion um I think the only thing that has been done is like the FFM part but I think everything else is like directly out of it without much of an enhancement and Improvement I I feel like this is super amazing this is super amazing I've been always fascinated about what Runway ml has been doing with respect to the images like whether it is a panning whether it is a zooming whether it is a different scene but I think stable video diffusion will take this AI video generation or AI cinematic video generation into a totally different world and different experience I'm definitely looking forward to see how this is going to go and before I close the videos this is a real picture so typically AI video Generation image uh Solutions do well when you have stable diffusion or any AI image so you have an AI image it does a pretty good job when you upload an image of a real human being not every time it does a good job because one um it has to understand that IM humans it has to create recreate this humans in the diffusion latent space that's how it can do it but I found pretty fascinating that this image which um I uploaded sha and Sam Alman and when you see this image being a video you can see this uh this is almost like a live photo of iPhone or a Google pixel phone so when you take a photo sometimes they take extra two three frames to give you the live video action a live action this is not a live for video it does a pretty good job of taking the real human beings real human beings taking these living human beings and making them into a video I think Sam is not doing pretty much but uh you can see the smile the movement and Satya going in the front and coming back and I think it's quite amazing and Brilliant and uh let me know in the comment section what do you feel about this SVD that every body has been talking about see you in another video Happy prompting