Building a Video Understanding Engine (Moondream Tutorial)

Key Takeaways

this video we're going to learn how to use moondream to understand images and also JF so as a start we're going to build a Google collab notebook and we're going to make the Google collab notebook explain what this image means and also explain what this animation means at the end of this video you will be able to run this code yourself on free Google collab and understand any image and any GIF animation and if you want as a take home assignment you can also extend this Google collab to understand videos this is definitely the start of running a very competitive Smalls size model on a really good hardware and make it understand everything that is multimodel to start with let's first install the required libraries in this case you need to install Transformers Tim and inops inops is just like a helper Library Tim is for image processing and also Transformers is the library which is from hugging face which we are going to use to download the model and do everything else the next thing are in in fact like the first thing that you should do is if you are not going to use the Google collab that I'm going to give you in the YouTube description you have to go to the runtime and then click runtime and select change runtime and make sure you have got GPU selected this code will 100% work even if you do not have a GPU but having that free GPU will accelerate your inference speed once that is done and after you have successfully installed the required libraries the next thing is you need need to get any image in this case I'm getting this particular image it has got three apples so I'm using WG as a Linux or shell command to get this image and then save it as image. JPEG after I save this image now we have the image that is required for us to use and now we have to download the llm or the AI model in this case so from Transformers Import Auto model for coal LM and auto tokenizer auto model for coal will let us download the model and auto tokenizer will let us download the tokenizer these two files are important for us to use the AI and do inference in this case inference means give an image and get the caption or description of the image back next you from pill import image pill is the Python 3 version of pillow for image processing and import torch which is importing P torch because we are going to use a certain helper functions from touch next specify the model in this case we're going to use the model byat Moon dream 2 which is a really good model and a very small size model so you can use it for a lot of application we are going to use the revision from the third March sorry 5ifth March and then you're going to check whether you have got Cuda which is the GPU support or not if you have GPU support then you're going to assign Cuda value to the device if you do not have GPU support you're going to assign CPU that's why I said this code will work even if you do not have GPU but having that free GPU on Google collab means you can have accelerated inference then load the model then load the tokenizer when you load the model you need to enable trustcore remote uncore is equal to true because this is not part of the core Transformers and also you can use the revision just to get the right repo or right revision and you have to send the model to whatever device that you have got this is essential if you have got gpus support but even if you do not have GPU support it will send it back to CPU which is your current memory then get the tokenizer so at this point youve got the model and the tokenizer ready you don't have to give any tokens you don't have to add any credentials authentication nothing now we have got the model downloaded we have got the image downloaded now let's do some magic first we're going to open the image as a pillow object so P object so this is image. open and this is the current path where you have got the image so if you go here and then you will see you will actually see the image. jpeg file so you're going to refer to the same one but if you have gotten a different path you can give that particular path what whatever path that you have got path of the input image that is what you have to give the next thing is you're going to encode the image and then send it to the device in which you're going to run if you have GPU GPU if you have CPU CPU and that is all defined by the device object here so you don't have to make any change now you're going to generate an answer so this is not just an image captioning model this is an lmm large multimodel model so that means it's not just you have to ask it to give caption you can ask it to do anything but here I'm just simply saying describe this image so you have got the encoded image then you have got the question that you want to want the llm or this image to the AI to answer then you have got the tokenize you're going to send everything get the answer back and move the answer back to CPU and then print it once you print it you can see that three apples are arranged side by side in this image with a single leaf attached to each the background is white and then it gives you all the information so you can see the leaf is there background is white and it does everything that you want to know from this image which is kind of cool and it doesn't take a lot of time to run in fact I'm going to run it in real time so that you can see how much time it takes it loads the image in 2 seconds imagine just in 2 seconds it gives you the image and you can play with this um like the way for example instead of saying describe this image you can say write a joke about this image so it's up to you and your creativity how you want to push this because like I said this is not a capturing model alone this is an llm and then it can do a lot of other things so you basically ask anything that you want and get the response back once you do that now we're going to extend this code instead of a single image we're going to extend this code to a gif GIF in this case is a bunch of image frames attached together so we're going to to use a gif and then do it and like I said at the start of the video if you want a take home assignment I would strongly encourage you to build the same code for videos so how are we going to approach JF so the way we going to approach JF so this this is already loaded this is just a Helper because Google collab was throwing some error now you have to download the GIF this is the GIF that you're going to download and I'm downloading the GIF from tiner.com and this is a p Day gif even though the model doesn't recognize entirely P day but I wanted to try this image and then see what it does so download the GIF and then put it inside your local folder I don't know how do you call it gif or GIF GIF whatever you call it that is what this is once you do that we're going to load the JF so the path of the GIF is here we're going to open the GIF then what we are going to do is we're going to count the GIF frames and once we count the GIF frames we're going to take every fifth frame so take all the GIF frames so GIF is a frame the frames of images as you can see here the image is there multiple images are there that's how you get the animation so the JF is a bunch of frames of images so we're going to count all the images inside this GIF and then we going to take every fifth frame and it is up to you you can take 10th 11th 12th I just like I'm going ahead with an arbitrary number five and then we going to take that and then convert it as a frame and then do exactly what we did before and get the answer and upend the answer every time you have a new frame and at the end of the day you're going to print all the answers and you're going to have the answer for this particular GIF this is the output that I got a piece of pizza is depicted in this image with text written on it what that's the first thing then you can see the pizza appears to be animated then the background is blurred the depiction of Pizza in the is presented in this image with text written on the bottom left and you can see that it would start adding more details about it you can see various symbols and lines and um like whatever that information that you have got so this way what you can do is if you have got a video you can send the video to it get the response back like this and then in fact you can use an llm like as a to-do what you can do is you can do ask an llm to summarize this so you can use that also with this so you get the response back now you have got 1 2 3 4 5 six six responses which probably means it has got like 30 frames of approximately and then you ask an llm to summarize so you get final one description of the video which could be critical what I'm going to do is I'm going to take live one new uh image and then I'm going to try it so I'm going to take this image let's say this says happy P day I'm going to copy the image address come back here and then I'm going to paste that image I'm going to keep the output file name same see the output file name is going to same tener doj I've created the collab in such a way that it's easier for you to run so that if you have a different file for format you add the path here but uh if you have the same one it should work fine and instead of 10th Frame I'm going to do uh instead of fifth frame I'm going to take every 10th Frame and then I've sent this one here and let's wait I hope it has more than 20 frames at least otherwise we'll not get more than two responses or more than one response okay we have got three respon four responses okay a table is depicted in this image with a glass ball containing a brush on the pie some text is also present and it says brush is visible on the right side a glass object is present now I want to change the prompt a little bit and then say give me the description of the image and also read the text on the image that's it let's see what it does so I'm going to send the same image but with this new prompt I want to see if it actually improves in the response that it gives let's quickly see what it does and whatever you're seeing is a real time inference speed it took 9 seconds to process this gif so everything from reading the Jeff and uh sending it to llm getting the response back I don't think it actually gives me what the text is maybe I have to change the prompt but you get the response back in such a short time and this could be extremely helpful imagine you have got birds and um youve got like a cage and you want to count the number of birds you can use this I think people have like a lot of hobbies like collecting bees they can use it there are lot of other places where you can use it because we actually counted the apples um if you see here so you can use it in production line you can use it in a lot of different places I'm yet to try this on a Raspberry Pi but I'm quite excited to try different use cases with moon dream model so definitely check out moondream here star moon dream if you like it but if you just want to straight away use it this Google collab notebook is made for you I'll link it in the YouTube description enjoy coding happy programming