Image Annotation with LLava & Ollama

Key Takeaways

okay in the last video I talked about some of the updates around llama and one of the things I talked about was that you can build a lot of little apps that actually do things on your local computer and so one of the ones I mentioned was about screenshots so people asked about that and in this video I'm going to go through show you how to do it I'm going to talk about the basics of what's going on it's very simple and at the end I'll talk about a different version that I'm using personally which is a more advanced version on this okay so the whole idea of this is that basically we've got a folder which we do screenshots for right most of us on our computers have a folder where we save automatically save screenshots and over time that folder gets very big or at least if you're me that gets very big and often I want a screenshot that I did a while back at a certain point in time and stuff like that but the only way to go through that is to actually go through and look at each of the screenshots and while that works one of the things you can do I'm going to show you here is that you can get one of the Vision language models to automatically annotate or write captions about the images which will make it easier for you to find the image later on so in this video I'm just going to show you a basic version of how to create the annotations using olama using the lava 1.6 model and then perhaps in another video I'll show you how to sort of integrate this with a rag system so that you can actually just use uh you know Q&A to basically query your screenshots and and get the the answers back that way all right so let's jump in and actually look at sort of a diagram of how this all works it's actually very simple I really we've got a folder where we've got our screenshots so the first code we're going to basically have is just something that goes to the folder and gets a list of the files out and then basically sorts those so that we've got them in some kind of order to basically use the next thing up is like the main logic of this and this is really just going to basically load up the file in my case it's loading up pngs you could set it to load up different things but in my case because it was pngs I I found that ama didn't seem to process the pgs it was more used to jpegs but what you can do is just convert the file to bytes and then it can handle bytes quite easily in here so basically I load up the file I convert it to bytes I then send it to the lava 1.6 model now here there's a few different models that you can use you can use the basic 7 billion parameter model and you'll get okay results out of that you can go up to the 13 billion parameter model which is the one I'm going to show you here and then you can also go up to the 34 billion parameter model so in my experimenting the thing that I found is that the 7 billion parameter model will often miss things that are really obvious and it can do a really good job on some images and not you know as great on other images the 13 billion paramet model is definitely better at sort of having a bit more understanding over the image and stuff like that and if you've looking for something specific in an image you could actually put that in your prompt that goes along with this and probably get decent results out of that obviously the 34 billion parameter model is going to be the best one uh for this but for a lot of people either you're not going to be able to run that or it's just going to be insanely slow for me on this machine that I'm using I've got uh 32 gig of RAM I can run that so it's not you know you don't need like a super computer to do this but it's definitely slower than the than the 13 billion parameter model that I'm going to be using here now when you send the image there you want to send it along with a prompt and you really want to customize the prompt for your particular use case so if you're just trying to index stuff maybe you have something about describing stuff I'll show you when I go through the code of the prompt that I'm using in here we then bring the results back from the lava one 1.6 model and then I basically just add that to a data frame so one of the things that I did early on was I basically check if there is a CSV file in the folder and if there is I just load it up and then we basically check to see okay has this file already been processed if not we'll process it if this file has been processed we just leave it and go to the next uh image in there but once we get the results back from lava 1.6 we basically put that into the data frame and then finally we just save the data frame out to a CSV file so I'm just showing you a really simple sort of Standalone version of this you could obviously save this to a database you could save this to a vector store which is one of the things I'll talk about at the end you've got a whole wide variety of things that you could do in here and then you end up finally with your CSV file which you could load into Excel or Google Sheets or use it wherever you want to use this kind of thing all right let's jump into the code and have a look at how this actually works okay so I'll go through the code it's pretty simple in here we've got our Imports up first so I'm just bringing in llama and then I'm going to bring in generate which we're going to use for actually generating the return where they going to use glob to basically get a list of the files I'm going to use pandas to make the data frame I'm going to use pill to bring in an image and then I'm going to convert it to bytes so that just shows you what I've got on here so first up we basically try to load a CSV file with the file name that we've got here so I'm calling it image descriptions. csb and if that exists we'll just convert that to a data frame so that we can then add anything that's new and then that will be saved at the end back to this if it doesn't exist we're just going to basically make a new pandas data frame we're going to give it two columns one is going to be the image file one is going to be description in here so when we run that now now we've got our data frame all right so we need to get a list of the files from where we're going to get them so basically here we're just doing a glob of the folder path in this case I'm going for uh PNG files just cuz I know that folder has nothing but PNG files in it but if you were using jpegs or something like that you could change this or you could use it star. star to get any and then put in a check to make sure that it's an image that kind of thing all right so we run that we've then basically got the list out I'm just going to sort the list in this case I'm just going to print out just some sort of debugging we're going to print out the first three images it gets and we're going to print out the head of the data frame if we've got one if we don't have one obviously we'll just see an empty data frame there right so now we come down to the main part of this so what we're going to do is we're going to have a loop where we're going to basically just go through each of the image files in this case I'm just taking the first five we're going to basically check if this image file is in the data frame and if it's in the data frame we're just going to skip it if it's not in the data frame we're going to process the image so this is the main function here the processing the images what this does we pass in the path name to the file and we're just going to print this out to the console so we can see it obviously you could turn these print statements off really easily and then we're going to basically load up that image and convert it to a bytes format so that we can just pass that in we're then going to basically pass that into generate you'll see this full response string I'm going to just set that up there but we're going to pass into generate so we're telling it which model we want now in this case I'm using the lava 13 billion 1.6 uh model but in my case I've got all three of the different sizes in here and if I was just running it as say a Cron job or or some sort of job that got run in the middle of the night or something I could just go for the really big model and I wouldn't worry too much about it taking that long and it doesn't take huge amount of time anyway it's more if your uh system has got enough RAM to actually run it or not and then I've got the prompt so the prompt that I'm passing in here is describe this image and make sure to include anything notable about it and then in Brackets I've got include text in the image so the idea here is that if it sees some text then that's probably going to be one of the most important things now the challenge is going to be that it often won't get that text especially with the smaller models so this is something that you want to be aware of but you can totally play Not only can you but you should totally play with the prompt here for your particular use case so sometimes if I've got it sort of looking for something in particular perhaps you've got it where you're trying to get it to do a not safe for work check or something like that play around with the prompt for those kind of things and be aware that the smaller models are just not going to be very good for certain things all right so that's my prompt I then pass in the image here and this is just taking in a list I'm just passing in the image bytes that we got in there and I'm going to stream the response out here and that is just so that I can see the text coming through and basically print out each response as we go through this and then basically just adding each of those streaming texts into the full response and then finally at the end of this function I'm basically just adding the path name or the image file name and the full response that we got back to the data frame so it's just adding it to a new row in there and then finally at the end of it we're just saving this out back to a CSV file with the same name as what we started with so if we've updated the CSV file we'll just be updating that file if there was no CSV file to start out with we're basically creating a CSV file here all right so let let's come in here I'm going to run it and let's see the sort of time frame that it actually takes to do this so you can see see that basically there was no CSV file in this case and we can see that at the start it is processing this image so it's actually loading up the lava model and then sure enough now we can see that okay it's generating text out here now this is what I mean by the smaller models will not always get the text right so that particular first image there is for Gemini right it's a screenshot of something about Gemini advanced in there and you can see that it's actually got genini so they won't always get perfect OCR or something like that obviously the bigger models will do better at some of this stuff you can get some things that will get quite nice results out here if we look at this one here about the Google logo it's done a quite nice job at sort of being able to interpret that so this is a stylized version of a Google logo it features an abstract Dragon Light creature with Chinese elements so the here has done a really nice job of being able to capture what's in there and actually on this one probably a lot of OCR systems wouldn't get the Google Google in there very well now looking at the results out we can see that okay it's basically done five different images here and and it's had no CSV file to load it's done the five images because I had that set to five there if I come out now we just saved that after changing it now to run through all the images now can actually see that okay already the CSV file has these images done in here so it doesn't need to do those images again and you'll notice that this time it was actually uh quicker to actually get the model because the model was already there and stuff as well in this case so you can run into some issues where if you're trying to load the model twice and you've got it half loaded or something like that in any of those cases you can just quit out of Al llama and come back in and it should work fine or you can actually go through and kill all the oama processes manually but it will often be in the middle of something and then restart a new process so you can see for a bunch of these it really has got the whole sort of IDE IDE the number of images in here of this sort of design and it really does understand that this is like a cad design going through and working that out which will make it quite easy for us to find this either just doing a keyword search in there or to actually use some kind of rag in there as well and you can see at this point it's generating pretty quickly like I've mentioned before I'm not using a super fast Mac here I'm just using a Mac Mini and it's going through and and being able to generate these images out just to show you some of these images one of them was a Walmart receipt that I got from online and I think it's done that this image appears to be a receipt from Walmart a large retail store the receipt list several items okay so that one has done a pretty good job what about the flying cat one this image shows an orange tabby cat in mid jump the front pores are extended out so you can see that it is actually getting some of these quite nicely and then like I mentioned it doesn't need to get them all perfectly for you to be able to then put this into some kind of search or some kind of rag for this how could you extend this and make an advanced version of this so one of the things that I've done to make it Advanced versions of this is where you get also things like the file modification or creation date and store that in the list and then when you put those into rag you can then basically use those as metadata to actually do searches and say want to search for this image it was from December 23 and it just be able to sort of hone down so especially if you've got a lot of images that are going to be very similar from time to time adding in anything that can get uh metadata in there as well can be really good so things like getting the modification date getting the owner or the username of the person who saved it those kind of things can be useful if it's not just for yourself as well but anyway this project gives you a simple example of how you could do this and hopefully you can see that this can be really useful okay so in one of the next videos I think I'll look at how to add in a custom rag with a fully open-source local model so that we can try that as well in here as always if you've got any questions or comments please put them in the comments below if you found the video useful please click and subscribe I'm going to be doing a bunch more things like this I'm currently working on a number of things to try and show people how to do the function calling with some of the open models and and looking at the different results that you get from different open models for doing things like that all right I'll see you in the next video bye for now