Reimagine document processing and understanding with generative AI

Key Takeaways

[Music] my name is ruse bpur I am one of our technical solution engineering leads uh here at gcp uh hope you enjoy the session today I'm joined today with uh three amazing distinguished speakers uh who are going to talk about the different aspects of this uh solution uh Derek who is on our vertex product team he'll share some of the details about work that we're doing both at the agent level and you heard in the opening keynote some of the work that we're doing uh on that front but also at the platform level to unlock some Enterprise applications Greg group CTO of sap Ariba will speak about the processing aspects of the solution and how to ingest large scale data and not only feed it into the front middle and back office applications but also feed the information back to sap so the existing workloads that you already have for example with sap can light up with some generative AI capabilities and Tom will uh close the session with some amazing examples of Frontline workers who could literally use the same zebra devices that we're all very familiar with in various Industries to light up these kind of use cases now why do we care about this solution let's talk about the problem statements first and I think they're very familiar for everybody in this room which is there's a lot of data and a lot of data everywhere uh but that data is not necessarily organized the number of data models that we talk about the number of databases that we have these are all different ways that we cannot access all the data that we want to leverage that is already digitized but that's not the end of it there's a lot of data that's actually stored but not accessible and we're all very familiar with the sum of all the Excel sheets and the number of emails that we send back and forth with attachments and links there's a lot of data that's still sitting in paper cabinets right that's all data that's not being used today and if you're involved in the sum of you new cases that we want to leverage to improve our operations we're not accessing those the way we should in a meaningful way what does that mean what that means is today you have workers who are trying to do their job for example in a retail setting where they just want to help a customer out who's standing in an aisle and they just want to have a simple question of what's the difference between two products I don't know it's sitting in a piece of paper either in a PDF file online or I just don't even have access to it as an employee that leads to inefficiencies in the conversations secondary if you are in the back office and you have access to those documents you're spending hours searching for those a perfect example of those kinds of use cases is sop Access employee training there's a lot of work on the regulation side of the fence and if any of you have worked with Contracting there's a lot of contracts a lot of ndas a lot of compliance related conversations whether it's on the accounting side whether it's on regulatory aspects and then there's some things like simple summarization of the vast amount of information that's out there and just tell me the basics of it so I can move forward so it all comes down to productivity and efficiency improvements in our businesses now this solution ties very well with our mission statement as a company if you think about our mission to organize the world's information and make it universally not only accessible but useful this solution touches all of those we have information that's unstructured and structured we have processing needs for the sum of all those documents to make it accessible and more importantly we can now leverage generative AI capabilities to make them useful now one thing to talk about here is trust is at the core of all of this and I think you you've seen over and over again in the past few days we continue to talk about trust and Enterprise trust this is all your data it's very important and as you will see the guests who are going to speak about this the importance of making sure that that data remains your data and in the confines of your own instance and this data does not flow back in to train or to improve the quality of any of the models that are supporting these kinds of solutions now from an architecture perspective there's multiple aspects to the solution there's the processing side which is how do we take the sum of all the different file types that we have or data for that matter maybe you have data sitting in big query alloy b or a variety of other databases maybe you have documents that are PDFs maybe there's Outlook files that have screen captures of a simple table whether you have Excel sheets EDI files name the sum of files that could be out there how do you bring all of those in process them through a variety of parsers that we're putting together including document AI how do we store those in a seamless manner manner which brings in some of the stuff we announced earlier with big query for example and its integration with vertex AI so you can actually do the embeddings directly in bigquery and have that data Loosely available followed by the capabilities on the search aspects and at the very last mile of it our vertex AI platform and the model Garden that today supports 100 plus different models so you can choose which of the models best suits your needs whether it's Gemini I pro nano or any of the third party applications like Claude llama 2 or any of those that best fits your need now with that said some of that data flows back in to other applications like I mentioned earlier with sap once you parse and understand that data you can actually flow it back so those same Integrations that you have today and the same applications you're using can suddenly leverage the capabilities here and become more meaningful to your business with that said I'm going to start with Derek he's going to speak about some of the product announcements and show a demo and then we'll follow with Greg and Tom thank you thanks ruse Bay as he mentioned my name is Derk Ean I'm a product manager in vertex AI excited to talk about some of the products that we offer powering the use cases that ruse Bay just in introduced okay so our mission uh on my team and within vertex part of it is to help developers build state-of-the-art document processing and understanding Solutions the way that we accomplish this is through a portfolio of products right across the spectrum that rusbe introduced on one side of the spectrum is document processing that's all about getting uh structured data from a document to drive automation so instead of manually reviewing an application maybe you could get the eight data points that are important and structure it in your database to automate some of that decision making on the other side of the spectrum is understanding so you have documents and you want to have a conversation and get answers to natural language questions based on the information in those documents across that Spectrum we offer options to the developers first to the agent Builder a manage service that abstracts tracks some of the complexities of generative AI away and you simply are calling apis and getting results back on the bottom of the stack are more uh platform components for model builders that want to interact directly with the Gemini apis and have full control over customization Which models their settings prompt engineering and more so I'll dive into the stack starting with document AI our manage service for document processing and with this product you can digitize classify and and extract document data you can use our OCR product uh to digitize you can also use a new layout processor to get the structure of a document back and embed it uh in a rag architecture which I'll talk a little bit about shortly you can classify document types which determine Downstream processing steps So based on the type of document what are the next steps I should do with it uh and you can also extract the structured data from a document form parer allows you to extract key value Pairs and then the custom classifier extracts entities that you specify and that you're interested in and let's double click into the custom extractor this is uh released and generally available ready for your production use cases it's really easy to get started with you just post a document along with the entities that you're interested in to an API and you get responses back uh you can see uh it's a little bit small but this is a sample invoice the generative model returned all of these uh entity these back without any training but there was one uh data point that was missed the ergonomic keyboard quantity in the middle of that line item with the blue square around it if you squint and I went ahead and annotated that and said hey model you need a little hint there's the 12 that you missed and then if I do that five or 10 times uh and I have an integrated data set now I can go back in to document AI with a simple click of a button or an API call you can fine-tune the foundation model so you then fine turn the foundation model with your integrated data set that you have exclusive access to within your project so you're customizing the results and improving uh the predictions for your documents and NY types we have multiple customers in production with this product and their overwhelming feedback is what used to take me days now takes hours to meet my quality goals all right the next use case I want to talk about is layout processing so we live in a world where there's more and more documents and they're more and more complex uh Financial reports research papers uh technical manuals they have diagrams graphs charts some of it is hard for us to understand it's also hard for uh large models to understand that's why we're introducing the document AI layout proc processor this layout processor will decipher the structure of a document return headings subheadings paragraphs text your charts your diagrams and it'll recommend how you chunk that document if you're going to embed it in a rag architecture for a search or a generative use case inside your applications uh my colleague guanga she led a uh session session 164 on Tuesday that demos this extensively if you want to check it out check out the recording all right I have shown you some of the managed services that we offer within vertex for documents for agent builders that are easy to use uh and you're just interacting with apis or simple interfaces I want to go down the stack talk about model builder capabilities and this is uh enabling you to work directly with Gemini apis uh right now the 1.0 Pro Vision and 1.5 Pro uh with documents as an input so that you can ask questions compare the document uh extract structured data and you can do that primarily through apis but we also offer an interface with the vertex AI Studio which I will test my luck and try to demo live here all right okay perfect so I'm in the console let's see if I can make this a little bigger for us okay so I'm in the Google Cloud console and in within the vertex I Studio I come to the multimodal section there's a bunch of document examples so we can see uh for entity extraction there a couple of document examples you can classify documents have a conversation with documents summarize parse tables translate and compare documents let's click into a couple of these examples we were just showing these examples but there's a myriad of use cases you can handle within this console so the Q&A the the ones that I just mentioned directly so let's look at the first extraction example so the first extraction example we're looking at a pay slip uh and it's fake data but you can see kind of in the bottom uh left portion of this document there uh is a $2 an hour rate there's 80 hours worked and it's $1,600 within this paycheck there's year-to-date uh data on the bottom and other structured data now if you open up the entity extraction uh option within the vertex I studio it'll pull up this example prompt so this example prompt and uh this document that we just looked at that is saying hey model help me out uh pull this structured data back and return it in a Json format and what you'll see at the bottom here is uh I've already run that prompt and you can see the structured data that's been returned right so if you like what you see you can simply click the get code at the top right of the prior slide and it's going to give you code you can see well you can't see it but um you can't see it but in the sorry in the in the middle of the in the middle of the code here you're essentially seeing the application PDF you're seeing the prompt and you're seeing the location of the GCS bucket where you're calling that document so you can rinse and repeat that however you'd like in your application uh and you can customize these prompts however you want all right let's move to a separate example we went back and chose one of the chat with documents example it's this fairly complex letter from the government saying I need to uh submit a certificate by a very specific date or my license will be uh revoked and since I don't like reading those sorts of documents let's ask Gemini to help me understand when I need to take action on this document and explain my logic so you can see well I guess it's a little bit small but what's Happening Here is within that document it was sent on um June 19th and 60 days from uh receiving that document you uh the receipt of the certificate is due so Gemini is able to recognize that do some math and say hey you need to submit this by August 18th right you can do this to compare complex tax documents you can use it to compare purchase orders across vendors you can do that uh on a myriad of different document types and of course you can get code similar to the pattern that we just saw uh use that in your application across a myriad of different languages okay so that is the demo that I was hoping to show you um and now I'm going to turn it over to my colleague uh over at sap Greg who's going to show you what they're doing with document [Applause] processing thank you Derek and thank you Derek and rusby for having me uh good good morning my name is Greg Greg telch I'm at sap um Enterprise application company you might know and I'm in uh the CTO of sap arba that's a procurement chapter of sap so we're doing the procure to pay solution supplier management um and also invoice management um first of all I'd like to uh tell you a little story um about my journey with Gen AI um last October we had our spend live conference um of sap Ariba and um I held a um AI Workshop a brainstorming workshop with our customers and one of the customers uh she she told uh she had the idea explained Greg uh when you negotiate a contract with your supplier and and uh you all in business um you're negotiating a a great contract with 100 pages of documents 100 page document with all the Clauses and this is a great time when you sign a document and from that you're actually driving down the value slope because it's hard to actually get then all the deliveries and that quality and um in in in this service so um we had actually then this picture generated um during the workshop that is showing us the this value slope and um so and uh when you go down the value slope you're losing the money so what you saw today here from Google and we are working with Google um on document processing and understanding is actually tapping into that and going up the value slope uh again and understanding the documents much better um and um getting the benefits so um the the example I have today for you is processing supplier invoices so this is quite easy right the supplier uh uh delivers the goods sends the invoice and they want to get the cash and the money so our task in arba is okay we want to get that supplier invoice into our operational system Mees that uh to the purchase order that we issued with the price and with the uh quantities delivered and then we we cash it out okay so what is happening today uh we get the the invoice and we saw invoice examples beforehand we get the invoice by email in a PDF printed uh let uh letter in Germany maybe even with a effect um and and then we need to get get it into the system and this is typically done in shared service centers maybe you know this from from your companies um and then really typed in into the system of course we also have the sap uh business network uh where we make that uh electronically but here in this example we we consider the documents um and this is um yeah effort for for our end users and customers and we want to automate the process of course much much more so we want to introduce now with the docu with Google um also document processing that we can uh OCR and pass our invoices um with with Gen AI uh then get to the supplier invoice and turn the user interaction then more into an uh approval workflow a Rie and fraud detection uh mechanism so that we move the needle uh move the user interaction then uh actually to the to the right side of the house and this is also how we uh uh together with Google think about flowing that document processing and AI is also flowing back into the Erp um or into the um SAP systems finally this could look like that this is a screenshot that we have um in inhouse you see a document on the right hand side the uh the invoice that was captured um and the the invoice document um in our operational rebba system is then completely filled out and uh is then a matter of reviewing and and and checking it instead of typing that in so thanks thanks a lot I wanted to introduce you to this document processing at sap rea thanks Greg and uh thanks to Google for having us I'm Tom ban with zebra Technologies and if you're not uh familiar with zebra we're really delivering solutions to the front line of work empowering those Frontline workers to have better experiences and drive more productivities in the jobs they're trying to get done so we do that through asset visibility think uh RFID reading barcode locationing technology uh through connecting the Frontline workers so android-based mobile computers which is another partner ship we have underneath the alphabet umbrella Beyond gcp and giving those workers the tools they need the connectivity they need in order to get their job done think about workers in a retail environment or uh clinicians or nurses in a healthcare environment that are doing meds Administration ensuring that the right medication is getting to the right person at the right time and the right way all of our technology from an asset visibility connected Frontline worker perspective enables that to happen and our most recent Investments to enable that at the intelligent Edge is in a category we call intelligent automation so we've acquired a few companies in the Machine Vision space and the autonomous mobile robot space uh to enable that as we're seeing more and more automation make its way in a collaborative fashion with workers into the front line so what I wanted to do to compliment the presenters that came before me is really talk about why are we in just the perfect uh coming together of a number of different dynamics that are enabling really powerful use cases and I think um Bill Bill Gates may have said it best when he said there's only two times that he's seen technology that has uh caused him to really think that we're at a fundamental shift the first one was in the 80s when he saw the graphical user interface and the second one was in Q4 of 2022 when he saw the first versions of open Ai and chat GPT so we are truly living in a once in a 50-year uh inflection point it's sometimes easy to miss that when you're so close to it we're we're living it every day but it's really worthwhile to take step back and think about that for a moment think about all the opportunity think about the Frameworks that sap and Google are building that we can all stand upon as Foundation to deliver really compelling Solutions and uh change a game from a revenue perspective from a from an overall opportunity development perspective for each of our organizations and teams so why the perfect time well uh this is from an IDC study an industry analyst in the second half of last year and I'm going to focus on a retail vertical but you could extend this if you're if you're playing other verticals like manufacturing warehousing Healthcare government just think about the analogy of of what I'm describing and I and I bet you'll you'll see the same kinds of things in your domains as well so where are organizations prioritizing from a digital transformation perspective first time I've ever seen this I track this every year uh this particular study the word experience pops up there in the top three twice improving customer experiences so if you think of a retailer uh it's the customers that are interfacing with them in a digital way in e-commerce or also in the physical store but also improving the employee experience and I'm seeing this more and more from customers that are saying my customers if I'm a retailer aren't going to be happy unless my employees have the tools they need to engage those customers in the right way so delivering really compelling experiences and this is a shift because in retail and in many verticals you know what did the convers what was the conversation about mostly in previous years right if you look at 22 and 23 what this survey um highlighted was operational efficiency and customer satisfaction right that's kind of a rung down from experience right it's like well how do I get more productivity how do I drive operational efficiency and I want to use technology to do that I'm going to prioritize that and oh yeah I have to have really you know happy and satisfied customers that's gotten elevated now to experience and why is the word experience so appropriate given everything else we just heard well what does AI enable if you think about AI for Frontline workers what does it enable it reduces cognitive load it allows me to have like an always on companion that I can interface with it allows for better decision- making whether it's just in time decision- making or in increasing the speed and the consistency at which I can make a a decision we've got devices in the hands of every single postal worker in the country uh the United States Postal Service there's 300,000 of those devices in the hands of those individuals do you think the United States Postal Service is thinking about how do they get consistency right um if you got 300 ,000 people in your operations or you're a large tier one retailer you have 150,000 or two or 300,000 employees how do you make sure at any given moment of time each one of them has access to the right information the best knowledge so the least experienced worker and this is the cool proposition can the least experienced worker enjoy the best experience job satisfaction and perform as well as nearly the most experienced worker as they're getting their job done and that's what this once in a five day five decade technology allows us to do so increase that speed of decision- making and then reduce the time to competency a faster way of saying in a world where labor attrition is higher than ever turnover for Frontline workers is massive how do I ramp up the onboarding time um or or speed up the onboarding uh to reduce the time it takes to get to a competent Frontline experience so imagine if we had an always on Enterprise Centric domain specific trained customized in the moment AI enable assistant for every one of those Frontline workers and as we started to think about use cases and working with the Google team we realized we were seeing the same things right operating procedures and policies are really complex it's my first week on the job or maybe it's my fifth week um I'm not going to know that as well and I don't want to sit static behind a desktop in a learning management system um necessarily I want to be out in the wild getting my job done and learning as I go um or product knowledge um the expectation from customers now is as high as it's ever been people are doing we're all doing it right a tremendous amount of research before we buy a product and walk into a store so if you're not speaking with someone that's at least as well equipped about that product and the knowledge about that product as you are from doing a few hours of online research it's a it's just a bad experience and so more and more we're seeing the need to be able to get this kind of capability um in the hands of Frontline workers and we kind of zered on these four areas you'll see some overlap overlap with what risb presented earlier so proactive manager employee assistance so leveraging standard operating procedures so I'm not thumbing through an sop document but I'm able to just get the steps I need right in the moment as I'm getting that particular workflow done um an everyday companion so I mentioned consistency before how can we leverage best practices from Top performing locations I've got a thousand stores as a retailer let's say um there's going to be a top cortile and there's going to be a bottom cortile what is the top cortile doing in terms of behaviors engagement knowledge that the bottom cortile isn't and how do I drive that consistency across all thousand stores scheduling so this is really interesting scheduling um you think about the Employee Self-Service experience I'm a shift-based worker how do I interact with what my schedule is how do I get more flexibility in a world where um providing those kinds of capabilities leads to less attrition because you have a more satisfied employee and then finally recommendations this is really about product knowledge um client telling cross selling upselling if I'm in aisle from a from a retail perspective working with a customer so um I thought what we do is walk through an example and um just kind of show what what what we're talking about so um if I could just go back one we just just go back one slide just have me set this up for a moment if the if the team count in the back so what we did is we worked with um a European retailer uh to ingest about 150 is there any way we could just back up one slide yep no I got to do it over here no okay uh okay so we worked with a European retailer to in just about 150 standard operating procedure documents and we trained using the Google Gemini document processing that rosbe talk talked about and the AI llm services weet trained a model uh to be able to operate in those documents and then be able to provide uh answers to questions that might be uh asked so uh let's see is there why I can go back think I see a previous button over here there we go okay so with that is the setup U you can do things like this so you you you've got your handheld device those are deployed across all the employees and you might have someone coming to the store for a return I might be a new employee so what are the steps to process a customer return and what we'll do is we'll come back with like a relatively short answer as you can see here but then you can click on the longer answer let's say if you have less experience and it'll take you through step by step and you might think well returns should be pretty straightforward well is it an online purchase was it a refund U did it get delivered online and now somebody's returning it into the store and you can also explore sources so that answer that it just provided can be linked back to the actual sop document that generated from which the answer was actually generated so this is the SOP document that that answer came from so we have that attribution between the answer to the prompt and the document that created the information that delivered the answer as a way of eliminating hallucinations in these kind of really Mission critical Enterprise use cases so another example what's the uh you know what what if the customer doesn't have a receipt so it's understanding the context of the previous question and now saying well if the customer doesn't have a receipt here's a basic answer here's a longer answer which is you can give them store credit if you read the answer there but make sure you inform them that there might be a uh restocking fee as an example or that there's certain categories let's say certain seasonality categories that that um return may not apply to switching over like another type of not customer engagement but a work a workflow within the store is how do I update the onhand quantities uh for an item so how do I update the inventory uh for a given item and you'll see in this answer there's a lot of specificity to this particular retailer right so it talks about their application open the count menu tap on the U receipt open the camera and the available quantity tap finish so think about training now training is actually happening in the moment as you're getting the workflow and the job done and then you can go back to the SOP which I was really excited to see in this particular Retailer's sop we anonymized them we call them Mambo Supermarket um but I think if you if they scroll down a little bit further there uh you'll actually see a um a reference to how do you handle inventory transactions specifically with a zebra device so they they've actually built our device capabilities um and and the way they're interacting as you see they save the permanent inventory on the zebra as they call it which is referring to our devices that are being used in the workflow um switch gears to an Employee Self-Service example so uh integrating to uh the workforce management system we've synchronized that workforce management database with the memory of the llm so now I can say things like what's my schedule for next week and it understands the context of that and actually knows well next week is um going to be the week of April 15th so it brings up the schedule for next week I can click on a given day uh it'll show what my work time is for that day and you'll see the dates with blue circles are the dates I'm working the dates without blue circles are dates that I'm free so now what if I want to swap a shift um I've got a personal commitment next week and I need to swap with someone um who can I swap a shift with next Thursday as an example so it's going to go look at that entire database uh using the document processing RS Bay described and it's going to say hey you can swap shifts with Michael Davis or oliv Olivia white here's two examples and when I click on Michael Davis you can see that on Thursday the 18th of next week he is actually free there's no blue circle um and similarly with Olivia white she's working the weekend but she's off on Thursday so those would be the two best candidates to swap a shift with and now I'm not going into a portal and trying to like kind of figure all this out navigating you know a traditional UI um I'm embedding the Gen process right into my workflow right so um in doing that it's actually really just changing the experience streamlining the workflow um and and making it much more compelling for the worker and a much better experience for the customer in the store so we were really excited to announce last week uh Wednesday of last week we announced that uh we are partnering with Google U along the lines of everything that Derek and risb presented earlier but also with the Android team within alphabet uh on our devices and then with Qualcomm such that we can run these models across a whole flexible set of deployment uh modalities so uh we have customers that are looking at how do I get as much of this model to always be connected to the cloud to synchronize right um but how much can I get to run down on the mobile Edge uh or the far Edge as certain uh retailers are calling it also be able to run some of it locally and hybridize between that mobile and local and think about the Model Management scenarios across those two hybrid uh between the cloud and what's local or on Prem or far Edge or run completely on cloud and it's going to depend on the use cases it's going to depend on the type of data that's being access accessed how static uh the train information is in terms of what you're going to be engaging with in terms of figuring out what that deployment option is but there's no doubt that we're going to be doing that and I don't know of any other of the other hyperscalers that are thinking through the engagement with companies like ourselves to consider these flexible deployment options to enable the kinds of use cases we're talking about here in the most economic way possible and of course all of that will be underpinned by the vertex AI uh Google suite and framework that Derek uh outlined in terms of the document ingestion the processing the managed services that I described the training deployment all of that will always be connected back to the cloud in order to enable these flexible deployment options so we're really excited about it we're at the cusp of an amazing wave hopefully you're feeling the excitement of you know we're we're uh here in a time that we we haven't experienced this much technology shift in 50 years and uh stay tuned for a lot more from the combination of alphabet zebra and Qualcomm over the coming weeks and months thank [Music] you