Invoices & Financial Docs Parser | AI-Powered Python App with Google LLM Gemini 2.0 Pro

Key Takeaways

hello everyone and welcome to this new video last week Google made Gemini 2.0 available for everyone this is considered their Flex chip model in this video tutorial I will use it for document parsing and test its capability with invoice parsing invoices in Arabic French and English when passing an invoice we will ensure structured output that respects predefined format in order to do that we will use pent library and Google Gemini response gon mind type if this sounds a little bit confusing to you or you don't have much coding knowledge that won't be an issue because I have created a collab notebook easy to follow with grade you application where you can upload your invoice and get in no time your output as gson format and also if you want to modify the code use another model or change the parameters that won't be an issue because the code as I said it's very easy to manipulate and to modify so without further Ado let's jump to see how it works and then let's see the code together hello again so now let's see the Google collab notebook first we start by setting up the environment and we install the necessary libraries we don't need to install Google gimini because it's already installed all you need is to install the gradio we are going to use it later for the application then we'll start by importing geni and for the user data it's a good practice to save in the collab Secrets your secret Keys like the gro API key the Gemini API key and the Hing pH token and whenever you are going to use one of them you can just activate it from here so as you can see here the API is Gina API key and the client is G client API key here I stated three Gemini models the Gemini 2.0 flash Gemini 2.0 flashlight and the Gemini 2.0 pro we are going to start with the 2.0 Flash and then we are going to change to another model so the gimini models are able to process images and videos which can used by the base 64 strings and using the files API after uploading the file you can include the file u in the call directly the python API includes an upload and delete method when you are going to do the client file upload we can just pass the link for the file so first we can upload a file with the config for example display name here I uploaded my resume and one of the things that we can do is to count the tokens for example this is my resume here this is my resume the first thing we can start doing is to calculate with client models. count tokens the number of tokens in this resume which is 1858 tokens that will help us to like uh have an idea about the cost if you are dealing with many documents and also the chunking and the text splitting if you are doing a rag or other things so next we are going to use the structured output with Gemini 2.0 and pantic pantic is a python Library we are going to import base model and field it will help us to create the schema like the keys that we need to see them and we need the model to generate a value for them at the structured output format so here we Define for example the class topic contains name and the class person with the first name last name email work experience and education as you can see for work experience and education it's a list of topic it means the element of the list are topics that contains the name of the topic like description is the name of the topic here you can see the description mention the work experience if not provided please return an empty list and for the education it's the same thing so let's start with a prompt simple prompt we are not going to upload a document or any other thing it's just prompt or plain text here ahed Tunisian AI engineer working on a tutorial for invoice ping using Google gini models and now let's generate a response using the person model here there is the client models generate content the model ID the content is the prompt itself and the config is set to application gson we need the output to follow a gson format and the response schema is the person which we defined here so let's see the output it's already shown but okay so the first name is it knows that this is the first name it's so very impressive it knows that this is the first name this is a last name the email it's not existing and the work experience the only information here is AI engineer so the work experience is AI engineer and for the education is an empty list because we stated here if not provided please return an empty list here we can like create class for that person and uh we can just call it whenever we need it let's move to the next part which is extract structure data from PDF using gimini 2.0 so right now we saw how to extract from a prompt or let's say a plain text now we are going to use a document for this we are going to create this function extract structure data we give it the file path and the base model we need uh the file is the client file upload as you can see here we have the file path with a display name so the prompt is basic prompt extract the structured data from the following PDF file and the response is client. models. generate content we send the model the content which is this time not only the prompt but also the file with the response which is a gson following the model response schema that we are going to take from the input and we return the response. part okay so let's see for an invoice we need basic information like invoice number date items and total gross worth okay each item will be represented by a description quantity and gross worth for this I'm going to use this let me show it here this invoice it's from Orange uh written in French also some Arabic and uh I think it would be a good invoice for testing okay so let's run this okay so we print the type result it's class main. invoice and we also print extracted invoice result. invoice number so here we have the invoice number let's put it on the right to verify okay so the invoice number we can see it uh here seems to be right and also the date which is 1 12 2019 uh 1 12 2019 with a total gross worth of 36.4 and it's right as you can see there is no English in this invoice and the model so far precise and accurate here we can see there is a little bit ambiguity for the model because it represented the total orex value as an item and the tax value which is the a percentage of the DVA DV DVR which is the tax value as a second item which is not accurate at all so to overcome this I will create more precise fields in the identic class and also with a good description for the model to understand better what type of data should be extracted for each of the fields we mentioned for that let's Push It Forward we will update the class invoice with the right Fields upgrade the system prompt and also create grade your application at the end also we will change the model to Gemini 2.0 pro which is their best model so far you can use it with 1,000 as far as I remember 1,500 request per day and it's free to use for anyone so to get the API key I forget to show you that you need to go to uh just look for Google API key and then go to Google AI Studio API key and you can by clicking here get API key create API key and you create your own API key and you're free to go also for the data sets I'm using and the invoice there is a very impressive uh open-source website that contains a lot of images for generally for image recognition and the uh computer vision in general which is roof flow you can see here for example let's see invoice and you will see all sort of data sets so yeah that's what I wanted to mention going back here so right now we changed the model as I said to Gemini 2.0 pro and and now let's create a more specific pantic base model so I called it invoice 2 it will contain the invoice number the business information the seller business details and the customer information the client details the items list of itemized good or Services the tax the total and the payment method stated in the invoice as usual for the items it's not just an item we have a list of item which is defined here each item should have a description quantity unit price and subtotal and for each value we add a description so the model knows exactly what value should be written in front of that key also for the business information and the customer information we have more than one field to fill containing the name address phone and email for both client and seller okay so now let's try passing the same invoice that we passed before which is this one okay so uh let's run the output here we have the okay as you can see here there is the extracted invoice that's the invoice number the with total gross worth which is the total it's right again and a payment method there is no payment method included here so the payment method is none and for this it didn't print anything think for item in items because the items list is empty so far it's more accepted answer than adding the total of tax and the T as an items okay so now let's try with other type of invoices and let's see the Model Behavior we modify this function just to return the gson by uh changing the response. text as a return instead of uh response uh par okay so right now let's jump directly to the grade application and let's see it in a better way first let's import the gradio and the gson and we just copied all the code we have and we addit the gradio interface here honestly 80% of it is generated by DPS and I do the same for stream late applications the documentation is there it's very easy to follow but I found it easier for lazy people like me to generate it directly with dsck or CH gbt so let's run this it will give you the application interface here but uh I want to use it via this link okay let's this link you can share it with anyone it will as they mentioned here the link will expire in 72 hours you can share it with anyone for example if you have a project or an idea to show to your friend or to your colleague you can just share the link with them and they will have it on their site okay so first let's start by uploading the invoice let's start with an easy invoice let's say structured invoice and let's see the result of the model process file okay we have the invoice number the business information it's Kylie Lee illustrations the address the info there is no phone number the email is right the customer information it's right and for the items we can see that there is two items the nature information sign illustration and the map and brochure design map and brochure design so we have the nature thing here for the second it's the map and brochure thing 3 hours or quantity 18 and the sub tootal is right let's try another invoice that represents more the real life invoice not very structured with handwritten stuff on it and let's see how the model will behave okay we have this invoice here and let's process it as you can see it's it represents the real life invo it's not very structured and and also the image it's not very uh clear okay so first let's start with the invoice number let's check it it's 186 okay it's here we can see it here 186 maybe the video it's not maybe you not see it okay this is better I think even for me when reading directly from this image it's not very clear to me okay so the business information the name is dulco it's here I can read it in Arabic it's right but it's just behind the uploaded file okay the address the address is the right one we have the email also the customer information it's all extracted we have the items the description of them that's right the total amount is 8458 which is barely visible here but it's correct correct and the payment method we have the iban and the bank name let's check that we have the iban here and the bank name which is a bank in Dubai okay so that's very impressive to be honest and that will conclude our video for today you can find this notebook on my GitHub repository I created YouTube Project repository where I will upload all the notebooks related to my YouTube projects you will find the link in the description below also you can follow me on the hugging face repository I am uploading open source models maybe it's interesting for you and also open source data sets and if you have any other questions or you want to contact me or collaborate you can leave a comment in the description or feel free to contact me on my LinkedIn you can find also the link below in the description so that's all for today's video I hope you enjoyed it and maybe you can try it on another type of documents if you have any idea you can leave a comment for me and I hope to see you soon in the next video bye-bye