Automatic OCR Receipt & Invoice Parsing in Python

Key Takeaways

what is going on guys welcome back in today's video we're going to learn how to automatically parse receipts using ocr in python for example to track your finances based on the receipts you get from different stores to build an automated billing system this is what we're going to do today we're going to learn how to use ocr which is computer vision a computer vision technique to parse the bills to parse the receipts that you get from different stores so let's get right into it [Music] all right so for this video today i have prepared two images of receipts that i found online you're going to need at least one image that you want to process that you want to parse and the images i chose as i said i found them online are from costco and from walmart and you can see that those are pretty basic generic images photographs taken with a simple phone or something they're not professional scans and you don't need professional scans you can use images like this one to parse the information using ocr ocr stands for optical character recognition we're going to use an external api we're going to send to it images like this one or images like this one and we're going to get the information that is contained in one such receipt so we're going to get the actual store we're going to get the address we're going to get the individual items the price of the individual items the total the subtotal the tax everything that's listed here will be delivered to us in a structured way as a dictionary as a json response so we're not just going to get the text here the raw text and then we have to filter the individual um the individual pieces of information here we're going to get a dictionary that contains stuff like the items as a list and then we get the amount and we get the description of the item and so on um so this is actually quite impressive what we're going to see in today's video and by the way this works also with non-american non-english um receipts as well now i don't know um if it works with something that has a completely different font so if you have something like arabic language or chinese or something i'm not sure if this is going to work but at least with the english european german spanish whatever letters it works so um i tested this myself with austrian bills with austrian receipts in the local source that we have here and this worked perfectly fine as well um so what we're going to do first here is we're going to open up a command line we're going to install the request module this is the only external module that we're going to need for this because we're going to send a request to the api and the api is going to do all the processing for us so we're going to say import requests and import um json json is part of the core python stack so maybe let's put it above requests here uh and then we're going to say now that we want to get the url the url is going to be https colon slash slash ocr dot s price i hope this is how it's pronounced this is the api that we're going to to use today um it is not a sponsored video so this is not an ad for the api but this is the api that i use personally when i have to automate uh certain things so oftentimes when i uh get receipts from certain stores i want to track my finances so i just scan them in with my phone i just take a picture of them send them to the api get the information put it into a google doc sheet and then i do all the calculations that i need to do so this is quite uh quite handy api slash v1 slash receipt so this is the end point that we're going to use here and then we can say the image that we want to send is going to be the receipt 1.png and the response is going to be request dot post so we're going to send a post request to the url and the data is going to be equal to so data is a keyword parameter here and the data is going to be equal to api underscore key and the api key if you have one you can you can pass an api key so you probably have to create an account you can have a free plan maybe i don't know or a paid plan but you can also just test it you have a limited amount of um responses or requests that you can send a day and then you essentially get the responses and uh you can work with them but you cannot do that an unlimited amount of time from the same machine right so you have to to wait uh maybe another day or something so with a test key you can do it uh just like that if you have to do it on a large scale with multiple requests a day with like 20 requests or something or 200 requests or something you might want to create an account there um but you can do that with a test key here as well and then we're going to say recognizer is going to be auto so it's going to be chosen automatically here and then the ref underscore no is ocr underscore python underscore one two three and then after that we're gonna say that the files are going to be equal to uh okay now the formatting here is not very good let's add this to a new line files are gonna be equal to file the file is open opening the image in reading bytes mode so that's the payload here that we sent to the url to this api and as a response we're going to get data we're going to get the data that we're actually interested in um and before we now go ahead and send requests all the time i'm going to save the requests into a pickle file so into a serialized file i'm going to save for this import pickle um actually do we have to do that with pickle i can actually i think i can also do this with jason it doesn't matter i'm going to do it with pickle here but i think it's even easier with jason so you can do it with jason if you're a more professional coder than me um what we're going to do now is we're going to get the response text and we're going to dump it into a file so we're going to say actually let's do it with jason i think it should work with jason as well uh so res dot text is what we get here as a response this is going to have all the information and we want to turn this now into adjacent objects so json loads we're going to load the response text and actually we're going to say now this is now the json file so what we want to do is want to say json dump s so dump into a string the json object and we want to dump it into the file um or actually do we have to provide a file stream json dumps uh [Music] no this actually let me just see what this produces or actually let's dump it into a file so let's just say dump like this let's say with open response 1 dot json in writing mode sf json dump and i think this is the way it works right let's see if that works so we have this response json it actually worked there you go we have this json file here we're going to take a look at this in a second but before we do that let's do the same thing for the receipt 2 dot jpeg so that we have both of them to examine response to json let's run this they go response to json so oh okay daily quota exceeded i think this won't work um but this one worked because i tried i played around with the code here a couple of times before making this video so exactly for the second request this was not enough because the daily quota succeeded maybe i'm gonna cut this video then in the end i'm gonna add um maybe i can use my phone hotspot to change my ip or something uh but let's work with the first json file here first so let's delete all this here because that's just for downloading let's go ahead and say json.load and we're going to load with open response 1.json in reading mode s f we want to say json load f and we want to save the response into data so data equals json load f and then we can print data there you go so this is the response to our first request so to this image here from costco um and you can see we have a bunch of information here that we can also extract um from we can also extract from this json file here so let's go ahead and just say data.keys to see what we have here what we can actually look at and you can see now we have the ocr type the request id the ref number and so on and so forth the interesting thing for us is now the receipts themselves so here we have the collection of receipts so let's go ahead and say data receipts like this now if i run this you can see we have a list of dictionaries in this case we will only have one dictionary because we only uploaded one receipt image but you can see we have here a list so we can actually go ahead and not just get the receipts we can get the first receipt in this case by specifying index 0. so now we only have a dictionary and here you can see already we have key value pairs representing the information that is part of the receipt so you can see that we have this costco receipt and the one thing that it doesn't recognize is that it's costco so it says that the merchant name is wholesale which is this string here and then thornton and then 629 so this is what it sees on the receipt um but then you can also see the address of the store and all that so we can actually go ahead and say dot keys here as well to see what we actually have here we have the merchant name the merchant address the phone number the website the tax registration number and so on and then we have also some stuff like uh the currency that was used the total and this is the most interesting one the items this is a collection of the items that were purchased so what we can do here now and this is probably where it gets interesting for automating your billing system for automating your finance tracking is you can iterate over the individual items so we can do something like items equals data and then receipts zero to get the first receipt that we have here the only one that we have here and then we want to get the items this is a collection now i can print the items list here and you can see we have again a list of dictionaries one dictionary is one item so here we have multiple items multiple dictionaries so let's start with a simple message saying your purchase at and then let's make this an string let's say this is the data receipts zero and um we wanna get actually the merchant underscore name in this case it's not gonna be costco unfortunately because it didn't parse the logo correctly but we can start with that and then we can save four item in items um we can do we can say print and then first of all the item description this is going to be the name of the item let's put this into curly brackets so the item description is the item name essentially and then we're going to add a dash and the price so we're going to say us dollars so a dollar sign and then item and then amount or actually i think we should be even able to do this with currency we can see if that works in a second um but that alone should already list the purchases so you can see your purchase at wholesale uh thornton or thornton how is it pronounced i don't know then we have the individual items and the prices and let's now see if we can do this with the currency as well so we're gonna say uh this should actually be the receipt um no actually data receipts zero and then currency let's see if this gets us the dollar yeah okay it says usd basically so we can add a space here and instead of the dollar symbol we have usd which is fine we know that is us dollars so we have that and then we can also at the end of this say okay now print maybe a dash 30 times so that we have a separator and then we can say okay the subtotal of your purchase is and then we can do something like um or not something like it's actually the uh data again receipts maybe we should we should create an object that is receipt zero maybe we should say receipt instead of data receive zero all the time uh however we can say subtotal then we're going to add to that the tax so instead of subtotal tax then we're gonna add another separator and then we're going to say the final sum so the total is going to be whatever the total is so this is all information that was extracted by the api you can see now we have this list here and we have the subtotal and we have the tax and of course we should probably um include this here so the u.s dollar there you go there you go there you go let's run this again us dollar us dollar us dollar so this works and of course this is now just printing the information because i want to show you how to work with ocr in the api here but combined with a lot of other videos that i have on this channel i think i have a video on working with exo files on working with pandas i have a whole data science series you can think about a thousand ways to automate this into something right so in this case i showed you how to get the information how to get the currency how to get the total amount how to get the individual items iterate over them get the description get the currency get the amount and all that um and even more but what you do with that is then you can do a thousand things with that right you can take all this put it into a csv file put it into a pans data frame apply some aggregation function get some statistics you can take this and put it into a database you can take this put it into an exo file format it a certain way whatever you want to do with that information that's up to you maybe i can do a video if you guys are interested on how to build a complete um automated billing or automated receipt tracking system and how to categorize the individual purchases we can do that on this channel if you guys want to but essentially this is how you get the actual information this is how you get the actual data automatically from your images off your receipts and then what you do with that is uh up to you you can do a thousand different things so that's it for today's video i hope you enjoyed it and hope you learned something if so let me know by hitting the like button and leaving a comment in the comment section down below and of course don't forget to subscribe to this channel and hit the notification bell to not miss a single future video for free other than that thank you much for watching see you next video and bye [Music] you