Process Documents Faster Using AWS Machine Learning - AWS Online Tech Talks

Key Takeaways

hi everyone and thank you for joining us we're going to talk uh today we're going to talk about how you can use aws machine learning services to process your documents faster i am samir karnik i'm the product manager on amazon comprehend which is our fully managed natural language processing service and i'm joined by jillian armstrong solutions engineer with liberty mutual and dave mcfadden solutions engineer with liberty mutual so i'm going to start off with a high level introduction of a few aws services and a couple that we'll focus on in this webinar amazon comprehend and text and then jillian and dave are going to show you how they are using these services for their specific use case at liberty mutual at aws our mission is to put machine learning in the hands of every developer so to that end we have the broadest and most complete set of machine learning capabilities so at the lowest end of the stack here we have ml frameworks like tensorflow pytorch and mxnet and infrastructure like inferential gpus deep learning amis and containers one level above that we have amazon sage maker which abstracts away a little bit and allows you to label data pick your right model algorithms train your models as well as tune parameters and a lot more today we'll focus on the highest level of the stack here which includes quite a few services for example you can use recognition for vision-based machine learning you can use lex for chat bots and kendra for enterprise search the two services we'll dive into today are also at this level which is amazon comprehend amazon comprehend abstracts of a few decisions that you can focus on your use case for example you don't have to worry about uh which model architecture should i use or how should i tune the parameters or am i using the right instance types am i getting the best latency or throughput that i need we take care of all of that under the hood so that you can focus on your application and how to solve your use case similarly text track does a lot more than ocr it gives you key value pairs it gives you handwriting recognition table extractions and more and today we look at how you can use these services to process your documents intelligently so let's talk about idp we've spoken to a lot of customers and and we hear that billions of documents are created and these documents can be physical in nature right like these could be handwritten forms that are scanned or they could be uh electronic right these are forms that are filled out on the computer online by somebody but they're in various formats they're in various forms and the information from these documents needs to be extracted into some structured form so that it can be used effectively in downstream applications today organizations have trained experts to do exactly this right bring that information out from letters or forms and put it into a structured form or there is some level of automation a lot of this automation is rules-based because it's easiest to get off the ground or it's template-based so that works well for standard documents but if that document changes even a little bit that automation again requires some maintenance or an update to work seamlessly so for example the kind of documents that most people are working with are like the example you see on the right here structured documents we've seen here an employment application and a w-2 and organizations need to get information out of these forms like the first name middle name last name the phone number or the address and a lot of this information is presented in the form of key value pairs sometimes it's in tables there's some logical grouping for information that's within this within these tables and that relationship is also important in understanding what information is presented there in other examples we've seen very dense or semi-structured documents like the ones you see on the right here so in this case there is a natural language here so right you need natural language based awareness of the information presented here rejections don't tend to work well here because information can show up in various different ways and one regex might not capture all of it here people are using nlp based models they're using pre-trained entity extraction to get information like people dates organizations quantities some pii as well or our customers have also asked us for uh custom entity models right that you can use to extract information that's unique to your organization or unique to your domain and you have example documents you have annotated data you could you just want to do a one-click train and use a custom model to extract this information as well as you want to classify these documents to understand what type of documents you're dealing with right so both these use cases that we just saw can be solved using amazon text track as well as amazon comprehend these two services are on the top layer of our stack right the extract can get information uh can it can extract text and and data from virtually any type of document that it can be scanned it can be in not perfect form and it's able to recognize this information and extract it amazon comprehended i said earlier is fully managed it provides nlp models in the form of simple apis so you don't have to be a machine learning expert any developer who knows how to use an api can now incorporate machine learning into their application and then we're also going to talk about amazon augmented ai which is our human in the loop service so in those use cases where you need a hundred percent perfect results the the low confidence output or the low confidence results from machine learning models can be sent to a human for validation or verification or just a quick correction as needed so let's jump into jillian's and dave's use case and and see how they're using these services at liberty mutual all right jillian up over to you thanks samir so i'm jillian and dave will be speaking to you in a few minutes and we're solutions engineers at liberty it in belfast northern ireland and our company has a sole focus in delivering world-class software and solutions for our parent company liberty mutual which is a large insurance company doing business a lot around the globe and you may think that most business in 2021 would be done via the web but in lots of businesses and definitely in insurance a huge amount of transactions are still performed via paper or at least the digital equivalent of it pdfs text documents spreadsheets images scans and in fact we receive thousands of documents and emails from our customers every single day and that's true across our entire value chain right from quoting to claims management and that means that millions of hours are spent by humans working for us manually inputting that data into our business systems before it can be used and also because of the high effort a lot of data is never fully collected or connected to related data in the organization [Music] and this meant that if we could automate the data entry we could massively improve efficiencies allowing our employees to focus on more complex tasks and also ingest and connect more information than would have been previously possible now the first step in that is just pure automation no ai needed at this point and the diagrams we'll walk through today have been simplified down to some core components and are just one part of a larger platform but they do demonstrate some of the key considerations and there are two parts to the automation here firstly automated data ingestion we want to ensure that any data be that emails documents other is automatically rooted into the system as it arrives this saves someone having to manually monitor mailboxes or file systems and also removes the time spent manually saving off files which takes longer than you think secondly automated data extraction we want to be able to pull as much information from the incoming data as possible and get it into business applications where someone can actually use it and you'll notice event bridge here and that's because we made this a fully event driven system incoming data is an event we can pick up and act on and when we've processed that data we can also emit events that allow multiple systems to go and retrieve and use that process data in whatever way they want whether that's saving it sending it for review or using in other downstream processes now with automated data ingestion we just bring in the data for example a pdf file and save it off in the automated data extraction phase we want to do more than just save off the raw data we want to tap into the valuable information that is within that data replicating what a human would do when they look at and manually information manually enter information about that email form picture or whatever has been sent to us and the simplest data to work with in automation is where we have machine readable text word files excel many pdfs and one of the most common things in this category are forms so let's walk through some examples of how we would think about extracting information from a form that we had received because we receive a lot of forms now you'll see an example form on the left and the extracted text on the right and as long as we have some knowledge of the structure we can just use this information directly for instance here we know that we have a label and then on the next line the value that goes with that label so the pretty simple pattern that we could use to parse the text and would mean that we could take that text block and transform it into a tabular format that could be put into a database and you'll see if we expand out that step function we had on the earlier diagram that we just need a few steps some pre-processing to determine what type of file it is some extraction logic to pull out the text block and parse it and then any post-processing that we want for that particular type of data and as we've mentioned before the ultimate purpose of taking that raw data and moving into structured information we can save in our databases is so that we can replicate what our employees were doing when they were typing it in getting it into our business applications where we can use it in our insurance flows giving you a quote or processing your claim which all shines great except real insurance forms don't look like our nice initial example instead we see that the layouts can be much more complicated they're rarely straightforward lines with a label and a value instead we see that labels are above or even below the value they refer to multiple pieces of information are on the same line and we see a variety of formats within the cm form and although we could read the text from them parsing it becomes much much more complicated and maintaining a set of rules that can transform any textbook into structured information becomes untenable and this is where we realized we needed more sophisticated ways of handling our documents we needed to use some ai in our automation so that we could expand out to even more complex forms [Music] and this is where amazon techstract comes into the picture since we were using that step function to orchestrate our extraction it was very easy for us to add in multiple methods for extracting and mapping the data and then use that pre-processing logic to direct forms to the best extraction choice so we can go ahead and choose between basic rule-based options our own internal ml models or any combination of services both internal and external and techstract is one of those services that we added to our platform to help accelerate our automated extraction needs as samir is called out textract offers a number of different features and although it does do straight ocr of the text the big win for us was the text tracked form service because this can extract key value pairs from a form rather than just the text so for something like this slightly more complicated example we see that taxtrack can identify each of the labels and its corresponding value regardless of the layout and that makes mapping those values out into our business systems so much easier and allows us a much higher confidence in those extractions as we aren't having to maintain complex rules about how to map any text blob into structured data we just need to know the form labels that we're interested in and text track gives us this even when dealing with very complex forms and of course there is another really important reason that we need more options for extraction than relying on machine readable text a lot of our forms are not machine readable many are scans or images plots can be low quality more than you think have handwriting and initially we directed these through other means or we just sent them straight for human processing but then at the end of 2020 techstrike added in handwriting recognition and because our system is flexible we were now able to start directing handwritten forms through our text checked extraction option and being really impressed with the results we haven't had to do any filtering based on form quality we find that service can handle all sorts of issues like the form being slanted or faded or crumpled now it's not perfect we still need to have a human review on those more challenging forms but it does make it quicker and easier for those doing the input to have that initial extraction done and if we think about how we as humans deal with unclear information for instance terrible handwriting we rely on context and also being able to cross-reference other information and a generic ocr service doesn't have that context or cross-reference generally built in so it may initially seem like something that just isn't going to be part of an extraction like this but actually most of our data doesn't come in in isolation we don't need to only rely on the ocr of a single form files come to us already connected with other data that also arrives at the same time [Music] if we look again at this diagram we can see just that almost all of the forms we receive come in as email attachments that means we have additional information that we can leverage to improve and increase the information that we can extract and dave is going to tell you more about that thanks julian and as jillian said the email attachments are not going to be the only place we get information an email might contain multiple attachments but also as information in the email body that we will want some of these emails are pretty straightforward where it's a single email with no replies however we've got more complicated examples where the original request could include a trail of emails between the liberty employee and the customer for example when additional information has been requested however for the purpose of this demonstration we're focusing on the single email the importance of the email content varies the one thing we know with emails is that the text is completely unstructured as humans reading this we can clearly see the attachments the subject line and the email body it's important that we add automated extraction for emails as well as for the forms so we added a parallel flow that runs simultaneously allowing us to extract forms and email data at the same time there are a number of entities available in the email that we need to discover before we can successfully process the request one of these being the email context liberty like most companies has customer mail boxes and these mailboxes can be used by customers for any request whether it be for a claim feedback or general query so identifying the contacts to successfully extract the content is very important the email could also contain additional information some structured forms will be pre-labeled will have pre-labeled fields one of those fields might be addressed there could still be additional fields not in that structured form and the customer might still want to share these so it's important that we can successfully extract these from the email body and as humans we're not perfect and mistakes will definitely happen this example shows how the form and the email may also have conflicting information in this case the claim number differs if a human was processing this form they might need to do manual searches against business systems to validate which claim number is correct doing this systematically is no different we will want to extract the information and use a series of rules lookups and cleansing to accurately predict the values we chose amazon comprehend as one of the methods that could help us to understand the unstructured text from an email out of the box comprehend provides the ability to identify entities such as people places dates and locations as you can see by running the example email body against comprehend a number of these entities are identified take quantity for example if we had performed classification on the email and had already identified this to be in the context of a claim we could add additional business rules and successfully extract the quantity value from the email with very little effort as we showed in our architecture diagram earlier we have multiple extraction methods running in parallel by having these running in parallel we're helping to increase the overall accuracy and if we look at the values specifically for the three date entities we know from having contacts that therefore claimed it repaired it and addressed changed it it would be difficult to write business rules to identify what each of these dates meant so to help increase the accuracy of the claim date extraction we built a custom model using comprehend to do this we labeled a number of emails to create a new entity type called claim date this essentially involved highlighting that entity a lot on production data amazon has a minimum requirement of 200 annotations per entity and this is the minimum so be annotating more it's likely you'll get a more accurate result once these were successfully labeled we could create our model using custom comprehend there are a number of ways this model can be used it can be done by creating a real-time endpoint which should give you the results in a matter of seconds however this incurs a cost of roughly 40 dollars per day alternatively the model could be invoked asynchronously which takes longer but the user only gets charged for what they use this can be done as part of a step function for example allowing the user to pull the job and check the result until they get a response and this was the exact solution we picked for our specific use case but this since our system is flexible we could have multiple models deployed using both techniques so now looking at our diagram again you see we have three methods of extraction we're using these to retrieve information from the email by using custom rules comprehend ner and comprehend custom using each of these will give us insights into each individual piece of data the last step in our process is to consolidate all the insights we're extracting and create a combined packet that can be rooted to business systems or sent a human to help them make an informed decision using a combination of business rules and machine learning we can successfully create a combined packet of information that came from multiple data sources by layering the business rules on top of the extracted data we can get higher accuracy along with the business rules we can also cross-reference the extracted data points from our data sources to ensure that they match for example if we have extracted claim date from the email body on the form and the values match this helps to increase the overall accuracy of the field but with a similar approach if they don't match we might need to refer this request for human intervention the main aim of this platform is to bring a number of unstructured data sources and create a structured data source that we can use for analysis data insights and eventually have zero touch processing for a request and looking at where we are now we have a complete serverless platform that is extensible and flexible orchestrated with aws services and using aws machine learning services to help with the extraction of a number of key elements by implementing this extraction process it's helping us gain up to 40 efficiency and avoiding the slow data input that jillian referred to earlier we are also continually adding additional use cases to the platform from across the company each providing unique challenges and opportunities to help us further enhance our platform and finally looking to the future the platform is continually advancing so we're using these learnings to evolve and improve the extraction methods therefore improving the overall system but we're also looking to expand to different data types and some ambitious use cases that's a high-level overview overview of how we're using some of the amazon ml services to enhance a complex intake process and decrease the number of manual hours spent in putting data i'm going to pass it back to samir uh thank you jillian and and dave that was that was really informative for those of you who want to do something similar at your organization there's a few ways to get started here and how we can help you get there on the top left here if you're uh interested in running a poc we have a ready to use a quick solution called document understanding solution the link is provided here if you need help if you need to you know you need help building a model building out a reference architecture whatever it might be we also have the aws professional services team that can help we have a team of solutions architects these are experts in aws services they can help you brainstorm understand your use case and also get you familiar with the various aws services there's a link there to contact them and then of course you can always lean on your sales and account teams to help you build an intelligent document processing workflow with aws i hope this was useful for all of you we had a lot of fun talking about it and best of luck in your intelligent document processing adventures thank you