Say Goodbye to Manual Entry: Automate Image Data Extraction with Python & Gemini 1.5 | Invoice

hello everyone how's it going welcome back to my YouTube channel and in today's video I'm going to show you how I buil this exciting invoice extractor application powered by Gemini 1.5 a multimodal large language model capable to process text audio and images on this application I just need to fit an image to the model in this case it will be invoices but it could be any kind of images and then you can start asking questions to the model it could go from paring all the information from the invoice or asking specific questions such as the date of the invoice or the amount of the invoice let me quickly show you how this application works in the sidebar you can upload the invoice let's pick this one that I already saved here and here we can see the invoice information and now let's ask to the model to par all the information from the invoice and let's see the results let's click on the botton send and now we'll be using Gemini 1.5 to identify all these fields that we can see here and provide an accurate response as you can see the model was really fast and we can see a couple of information here we can see the date that it was the 14th of 2023 here we can see that the date is correct the invoice number also looks correct what about the total amount of the invoice there was 702 all this information we can see it here and let's say that you don't want all this information you just want to know how much was the insurance which was Zero let's ask this question now how much was covered by the insurance and now let's ask this specific question let's click on the button send and here we can see that demount was Zero which is correct so you can see that this application is working as expected now let's go to python so I can show you how I build this application and we can build it together here in Python I already created a virtual environment which have all the python libraries that I'm going to be utilizing for this project I also have an environment file which has the credentials the Google API key that we need if we want to use gini 1.5 I have a video that I already created in which I show how to create the API key I'm going to be leaving that video in the description below so you can follow it up this file is just an example but you just need to paste your credentials in the EMB file here now let's proceed to create the application so let's create a python package where we will have our app.py so let's do that now and now let's proceed to import Streamlight since this is the main library that we use to deploy our application so let's do import Streamlight as St and then let's define a function to create our application as a first step let's do SD set page config and here we will Define this information that we have at the top which is invoice extractor and then this struet so let's do that now invoice then let's specify the title the page which is invos extractor with these two emojis here is two let's copy the same ones so we can have the same exact application and then now let's create this filling box in which the user can send questions to the model so let's call it user question equal to SD text input and here let's put input prompt specify as a key input now let's proceed with the sidebar which is this information that we have here let's add as a title of the sidebar invoice image here we can do sd. sidebar. Title Here we can specify invoice image and then let's specify this information of the sidebar that we have here so let's do choose an image for this let's do uploaded file st. file uploader let's specify choose an image and now let's specify the format type so the user knows in which file formats file can be uploaded and now let's do if the file uploaded is not known let's proceed to open this image and this image I want to render it in the main area of the page so let's do sd. image and here we'll do image let's also specify a caption of loaded image and also the column width here you can see all this information here you see that the image is rendered in the main area of the page and we also have this caption so this is what we just created now let's specify a prompt for our large language model let's create another file so we don't have it here and this file looks a bit more organized and it's easier to read so let's create this prompt let's call it def load prompt in which we specify the large language model that it's a financial advisor with an expertise in understanding invoices also let specify to the promt if you don't know the answer please refrain from speculating okay so let's load this prompt in our app so we can provide instructions to the large language model to load prompt and here let's import The Prompt that we just created and now we can proceed to create a new function that will generate the responses of the large language model based on the prompt the user question and the image that the user uploads so let's go back to the utils.py and here let's generate this function let's call it generate response llm llm and we will need to import a couple of libraries in order to use this function so let's import them now let's do import OS Also let's import google. generative AI as geni and Fromm import load M so we will load all the credentials now let's load Google API key and we also need to configure our credentials in J so let's do gen. configure and here we'll need to specify the API key now let's proceed to create model so let's do model jni do generative model here let's specify the model that we will be using now which is Gemini 1.5 and now let's create the response so for this let's use model. generate content and here we need to provide the prompt the user question and the image so let's do input question prompt and image we will need to specify these parameters when we want to use this function and now let's return the response response. text okay let's go back to the app and let's specify the button and here we have it as send so in this button we'll specify that if the user clicks on this button it will start generating a response so let's do it specify if S button which is in the main area of the page well here it says submit but let's call it send as we have in the page and then let's specify a spinner which will say start processing so this message will appear while the model is processing all the information and then let's do response equal to get what basically the function that we created earlier which is generate response llm and here we will need to specify these three arguments so we have the input question which is the user question basically the image that the user uploads which we called image and then the prompt that we loaded earlier and now let's also specify a subhe header that will say response let me show you the sub header here this is the sub header that we have and then let's write the response that the model generates and now let's see how the application looks like so let's do if name equal main so Main and then let's dra stream light run and then appui let's see if this works here we can see that the application was built and if we click on it uh it says that no mod name SRC let me remove SRC from here and then let's refresh the page let's load it again and see if everything works as expected this part let's change it also because we had it earlier in the sidebar so let's go back to the code so it should be here St file up loader we will need to do sidebar also by just making this small change let's refresh the page and we can see that it should be here nice let's upload an image another invoice let's go with a second one and let's ask a different question let's ask the invoice date here we can check the invoice date that it was in 2015 let's see if it's producing the response accurately so as you can see it's working as expected which is amazing let's ask a different question such as the invoice number which we have here 20158 let's see this one now let's click on send and as you can see the information is been processed really really fast let's go with with a different file here this one is in Spanish Let's see if the model is also able to generate an accurate response in a different language let's check the What's the total amount of the invoice let's see if it's able to reply 199 let's see wow this is amazing this is working perfect Qui L let's also ask let's well let's go back to the second well let's go back to the first image let's ask the information of the vendor par the name of the vendor let's see if it's able to do that it should be local store here W it's it's amazing it's given the information of the name of the vendor and also the address which is amazing aming as you could see this invoice extractor application is impressive you're able to send images to the large language model and then this one is responded accurately it's processing images and text at the same time and that's it for today if you have any kind of questions please leave them in the comment section below also I'm going to keep uploading videos on a weekly basis so don't forget to turn on the notification Bell subscribe to the channel because it helps me to create more content and hit the like button below see you next time