Elevate your document management with Azure AI Document Intelligence

Key Takeaways

[Music] [Applause] [Music] he [Music] he [Music] n [Music] he [Music] n [Music] hello my friends and welcome to another fabulous episode of the AI show live live holiday edition all my kids are out of school so I have to like drive them anywhere so I didn't have to wake up early but I wanted to because we're AI all the time it's pretty amazing uh this is the last show of the year uh we will be taking a two week break for the holidays because I mean I don't know I mean the AI show next week is going to be on if for those who celebrate Christmas Day the 25th and then the one after that I think is January 1st so everyone on the Western Hemisphere will still be sleeping from whatever festivities they had on the n on the new [Music] year so we thought we'd take a Hiatus but I'll tell you about the show after that the first show of the year we are going to up our game and we're going to have we're going to have more guests more Fab ous stuff more amazing things um so let's talk about what we're doing uh today let me Sher you my [Music] screen uh where's everybody coming from if you could maybe share share share share share share share uh what where everybody's from just put in the chat I'll just the side of the eye side of the eye side of the eye where everyone's coming from whilst I write down what we're going to be talking about uh today number one AI doc intelligence AI document intelligence so I bought a vocal processor a couple years ago and I I use it sometimes that was one of those times AI document intelligent with my friend [Music] BMA and he is uh he's here so uh he's going to you know help us out if you have questions we we recorded this I think the other day um so for those watching so what we do um if if you're new to the show uh at Microsoft we put out videos to help people understand what it is that we're doing with AI oh I just dropped my mouse uh and so we used to just do it live and cut it in front of you but some some people were like can I record it beforehand and then show up and then we play it and answer questions that's what we do sometimes though with some guests um they're just like we're doing it live we're doing it live um sometimes you do that so the next episode in two weeks is live live like uh and sometimes we do a lot like with people like Sarah she just shows up and we do a thing other times we uh we spend extra time in make it pristine uh and so that's number one number two uh for the second half of the hour um if you watched microsof IGN knite I did a demo with um prompt flow uh that had visual stuff in it now that gbt gbt 4 Turbo with vision has been released I need to move it over to proper bits so I think I was using like really really old like really really old Dev bits so now I'm going to use really really really newer Dev bits uh so I'm going to do a visual prom flow visual prom flow to newer [Music] bits uh so um that's what I'm doing for the second half of the show so that folks can stay then is all live you will see me do all this stuff live to the point where You' be like does this guy even know how to code no I don't that was my vocal processor I wanted to throw it in there for you uh all right so let's take a look at where everybody's coming from I knew I wanted to make sure folks could tell us where they're coming from okay here we go here we go here we go here we go Louisiana USA welcome welcome my friend gosh I'm old I need to move this up uh col Colin with a one and a five smart ESP vial oh wow that is I used to live in Spain so that is a political statement that I'll let you research all right uh so um welcome Carl juel espa I used to live in Spain it's a delightful delightful country and the food is delicious Pakistan welcome my friend uh Toronto Ontario Canada we have San Diego in the house the land of my birth californ welcome welcome uh North Carolina uh Iana artina uh welcome uh we also have Vancouver British Colombia uh from Houston barana thank you hopefully I said your name right well welcome uh Carl uh how how do PE how people build want to build easy effective rag data IND oh that's a great question uh maybe we'll talk let's talk about that uh let's save that ask again after the first the first H the first half of the show uh so that we can get to that we have Mexico uh Poland welcome welcome uh India uh Peter from the Netherlands what a we we have a wide array and swath of wonderful people here uh Malaysia uh Melbourne uh lovely Place uh Montreal Montreal Georgia uh Singapore welcome everybody it's so I'm so glad to have you especially during the holidays you there's a lot of ways you can choose to spend your time and spending with us as we're just so thankful for that so let me move my screen and let's bring up our guest BMA you ready thumbs up what's up handsome how you doing hello nice to meet everybody nice to see all these people joining also I didn't get the memo about the voice changer I would have brought mine do you have a vocal processor too I no oh man we could have like had like a vocal processor off and uh then really lost a lot of people watching so how you doing my friend you g celebrate the holiday holidays in any fun way yeah I'm gonna be staying with my family actually in San Diego so I know I saw somebody in the chat from San Diego so shout out San Diego land of my birth are you from San Diego too I wish I was but no I'm actually from Columbus Ohio so a lot a lot worse weather oh man I remember I had to do a thing in Ohio once and I landed middle of January stop by because we don't have we don't have these over here in the west you know there's waffle have you ever you ever been to Waffle House of course of course of course but on the west coast there like there's no Waffle House West of Texas I know it's insane but anyways I every time I go out west or sorry out east I have to stop at a Waffle House I get out I don't know if it was Columbus I don't know where I I was in Ohio and I got out and it was like minus like 20 and I was like what is this craziness sounds about right yeah and so then that's why again San Diego feels a lot better all right B you ready to talk about AI doc intelligence we recorded a thing but you ready to play it you okay to stay after to answer questions of course and I'd just like to point out that I did we the same outfit in the recording because I was trying to keep the illusion but you ruined it but that's fine that's fine no I I I try to do that but I forget and so I'm like oh I gotta wear the same shirt uh but everybody know like this show is is fun because uh we play the thing but then people answer questions everyone knows like the whole show is live but there's like a little section but sometimes it isn't you could have also said that you love the shirt and you have like seven copies of it that is true I I did watch it in between wearing I will seven copies we're computer people okay forgive us all right so let's play this uh take a look at this AI doc intelligence you're not going on miss this episode of the AI show where we talk all about elevating your document management with Azure AI document intelligence with my friend BMA make sure you tune [Music] in hello and welcome to this episode of the AI show we're talking all about elevating your document management with as your AI document intelligence with my friend BMA how you doing my friend I'm doing great how are you great so tell us who you are and what you do yes I am a product manager on the a AI document intelligence team and I'm super excited to show you what we've brought to our new public preview all right so Azure AI document intelligence Sounds new to me it's not a product I've heard about can you fill me in yes you are exactly correct um so the first thing I'd actually like to discuss is the rename um we are now the product formerly known as form recognizer we are we now go by Azure AI document intelligence and we didn't just change the name because it looks better on paper we made this change because we believe that the new name more accurately represents what our product does now and what we plan to do in the future um we don't just deal with forms we deal with all types of styles of documents both structured and unstructured and our goal is to empower our customers to be able to do more with their documents and enable powerful document understanding use cases and with our most recent public preview we have taken another big step in continuing to reach this goal um so you will notice that in this new studio um this change has been already reflected and that's both in the naming in the studio and also in our URL which is now documen intelligence. a.az your.com Studio lovely so if you're ready let's just hop in and look at what's new in this new preview let's do it so now I now that I know and this is cool now I know that this is it used to be called form recognizer but now it's Azure AI document intelligence the document intelligence Studio has a new URI and everything so what are some of the new things is we've done a lot of form recognizer but I bet there's also new stuff in here as well there's a lot of new stuff so let's start with layout actually awesome so we have made many changes toay to the layout model um to make working with llms easier for our customers so layout now supports Office document types such as PowerPoint and and uh Word documents as well as different document types you already supported including digital and scan PDFs um and all types of image files so you now have more options than ever when uploading documents and can parse all this data with a single API call that's cool uh the Azure AI document intelligence LE model offers a comprehensive Solution by providing Advanced content extraction and document structure analysis capabilities with this model you can easily extract paragraphs tables titles section headings selection marks font style key value pairs math formulas QR codes barcodes reading order so many things is quite the list yeah quite the list indeed and all of this in 309 printed languages and 12 handwritten languages so if we just run this here hold on hold on you said that so you said that like you could basically extract anything any language because you said 309 language you said that so fast 309 languages 309 printed languages and 12 handwritten languages and we're constantly adding to this list that is cool all right so what are we looking at so what you see here is some new analyze options for layout and what you'll notice here is that markdown is a new output format style so markdown is extremely popular with web applications and llm tooling and so now if I select this and I run the analysis we will see that the output is now in markdown which is extremely easy to to integrate in other applications um that use l M technology yeah and this is really cool I mean not to interrupt you here but this is really cool because I'm able to do this as an intermediate step I'll put mark down and put this put it directly into like an llm call for example which is really cool exactly and this helps to enable really powerful scenarios with semantic chunking or like rag scenarios and in fact I I want to jump to a little application that we've made to actually show some of the power behind what this can enable so what you'll see here is a little application that we spun up very quickly but basically behind the scenes we're using um layout from Azure AI document intelligence and we've used that to create semantic chunks that make it easier for the model to understand what it's looking at so now the model has better insights into table structure it understands sections better as you can see this is a press release for for the fiscal year 24 um it understands a lot of of context a lot better than if we were not running this um with layout behind so now because of this it it enables me to ask more specific questions for example let's ask this model can you summarize what constant currency is so as this is executing is this literally calling azure AI document intelligence pulling in the page is it's breaking it up too and then it's putting that in is that what it's doing or maybe explain what it's doing that way I can get a sense yeah so if you look at the at the structure of what's been pulled out from this document for example you'll see that this it's able to recognize the different paragraphs the different section headings break them into different semantic chunks and label them so what this does when this data is given to a large language model able to use this semantic chunking in order to gain better um understanding of exactly what is in the document and this helps because it also does things like um maintains reading order um so all of these extra benefits allow the the model to have a better understanding of exactly what it is looking at and therefore give better results and that's what we're seeing in this application here that you're showing where you're typing in questions and it's referring to the document exactly and because it has such a deeper understanding of what is going on in the document is able to provide much better answers this amazing and and and and these questions like I said it just it's it's just ging this is this is cool stuff can you do another question I have is can you do a little bit more because there's other llm technologies that people have been using can this integrate with that or is it do you have to just is this application available or how can I make something like this I am so glad that you mentioned that so if you are familiar with um a lot of llm Technology you've probably heard of Lang chain oh yes um Lang chain is one of the most popular llm orchestration tools um used to create llm applications and we are soon going to have native support for Azure document intelligence as a document loader um so one of the really cool things I wanted to show off was all of this will be available soon I promise you um in the next week or so um but you you'll be able to quickly with a few lines of code go in set up the Lang chain and then you will be able to load and semantically split different chunks and ask questions um using rag pattern and we're really excited to see um how our customers are able to integrate Lang chain and Azure open AI in order to do this process that's cool can can we scroll up a little bit because I thought I saw Azure AI search as well is that true yes so Azure AI search is part of the packages that are included in this demo but there are many different ways to do this this is just one way to quickly get off the ground um with using um Lang chain and Azure open AI to create these these powerful workflows and this is cool because um I I don't know if you're building llm stuff it's usually hard to like figure out how to do the chunking window because it's like do I take the first end characters or how do I overlap but what a meure AI document search is doing is it's semantically chunking the data into for example reasonable paragraphs which I think is really cool am I am I getting this right so what we're doing is we're providing a very good basis to be used in semantic chunking algorithms so the output of of uh layout can be fed into these semantic chunking algorithms to provide very easy um basically good data to be used to create strong chunks we all know in AI if you put garbage in you're going to get garbage out so basically what layout does is it makes sure that we provide high quality data to these semantic chunking algorithms so that when you are using them you're getting good quality data out when you're implementing these rag patterns cool so that's the layout stuff and we're super excited to see how people Implement that into their workflows what else you got my friend one of my favorite things one of the features I'm really excited about I really want to show off is query fields and let's jump into the studio so I can show you how that works okay so we're going to look at the invoice model here one of the biggest complaints that we've gotten about our pre-built models is that sometimes they are not sufficient enough for some customer use cases now don't get me wrong the pre models are amazing they require no training they require no labeling and customers able to use them out of the box but sometimes we have customers with specific use cases that need a little bit more than we offer it so let's take this UK invoice for example if I'm to run this we're going to see all the fields that our customers are used to seeing in the invoice prebuilt model we have things like customer address customer address recipient customer ID all that good stuff but let's say for example in this specific use case we have a customer that needs more so they need the campaign number we have they need the sort code and let's say the account number these fields are not fields that we extract out of the box because most of our customers don't need this information but in the case that they do we now have an answer for them and that's query Fields ah I see so generally the model is very good at a preset like General stuff for invoices but there may be specific things for invoices for your organization for example that you want to extract and this is what this does exactly and we're never going to be able to cover all of those use cases out of the box but this creates a very easy way for customers to add this to their solution so you'll notice this pretty button right here called query field so if I select this basically what we're going to do here is we are going to use camel case style naming to name the new fields that we want to extract so for example we said campaign was one that we wanted to extract so I'm going to select that and then also sort code so again just using very simple camel case style naming and then let's just add one last one let's say account number cool so now when I I just have to click that acknowledge that I am using query Fields save that and now when I run this analysis again what we should see is Boom campaign is extracted we have sort code and we have a account number and all of this is beautifully integrated directly into the model to common name so customers can now use these field in their in their solution I mean that's that's cool are you I mean there there's got to be bound to be stuff that you want to pull out that you can totally just put that in there is that something you can do programmatically as well you have to show us but I bet is that something you can do like in the post request as well exactly that is exactly the case all right those query Fields were awesome because it enables uh folks to actually pull stuff exactly the stuff out that they want uh what else you got to show us my friend yeah so let's now jump into the custom classification model custom classification models basically allow customers to to run a group of documents through a through a through a model and be able to instantly tell which documents are in there based off of classification that they themselves have made oh that's cool because then you can chain the these things to pull out the right forms now uh right forms and then do other things like for example if one form has a custom field then you would want to be able to know that's the right form then you could do the other form and then the other question I had about this because I I heard about this before does it can you is if there's a multi-page document that has multiple different kinds of forms in there does that work as well yes so you are able to classify whatever it is that you want to do so let's say that we have a form that has three different types of forms in it forms a b and c but they're all one Collective if you wanted to split these uh to split them into their respective Parts you could do this as well you just have to train the classification model to represent document a b and c and then it would work just like that all right can you show us how to do that yeah so the custom classification model has actually been expanded to different languages and one of these is French so I've set up a demo here just to show how this could work I've actually preet this up just for the sake of time but it's extremely easy all you have to do is upload a folder which has two different subfolders um one holding um document type A in this case which is Medical Healthcare type documents and then folder B which has professional service type documents and then the service will actually split these for you automatically and use the folders as labels which is why we have this nice stratification without having to do this uh um manually that's cool yeah um so once you're done with this you're going to hit train and again I've done this just for the sake of time because it does take a little bit of time and what this is going to do is it's just going to generate labels and everything so that it understands with the document structure and once you have a trained model you can go ahead and test it so that's what we're going to do now we're going to go to test and I'm going to browse for a file and let's choose this sample file and run the analysis what we should see here is the model being able to tell which bucket this belongs to and we do so we have this being recognized as a professional service document and that's super cool because now you're able to have any number of different kinds of documents you're able to have document intelligence tell you what kind it is maybe even break apart a multi-page document and then you can process those things with custom fields for each kind of document so you really got like a really powerful end to-end solution okay I know I said that this was last thing but we actually have I have one more thing that I want to go over a couple of small updates that we've made to the service if that's okay awesome so the first thing that I want to point out is that with the read API um we've actually added support for um handwritten extraction for the languages of Russian Arabic and Thai um we're really excited about this we know we have customers all over the world and so these were some really highly um requested languages and we're and we're excited to announce that those are now available for you to try in the studio today um we also have new pre-built models so us tax documents is an area that we've been investing a lot of resources into and we're happy to announce that we now have available the US tax 1099 forms in all of these variations as well again these Empower very powerful scenarios these 1099 forms are things that people use for taxes every year so we're really excited to see how our customers can use this um to make their document intelligence um processing even easier yeah and just in time for tax season too exactly cool that was not a mistake um along with this uh in the invoice model actually we have added support for kvk number which is a critical field in the Dutch local um we've also added support for bpay information which is critical in the Australian local and with these changes we now more fully support um these locals and we hope that this will allow customers in those locals to have more robust Solutions without having to rely on things like custom models that's cool and then the last update that we have here is in health insurance cards we now have support for Medicare and Medicaid numbers built into the health insurance card uh model and again we just are very excited to see how this empowers customers to have more robust solutions that allow them to do more with their document intelligence needs yeah and this is cool because it could even be a picture so you could build applications where like let's just say you're a medical place and it's just like here just pulled your card under our camera and then Boop uh Azure AI document intelligence can make that sing uh if I if I if I get right is am I getting this right you are getting this right um and that is one of my favorite things about this service is how robust it is um I even remember when I first joined the team I drew out an invoice with my hand on plain white paper and it was able to extract all the information correctly um it's amazing how far this Technology's come that is amazing so this has been awesome where can people go go to find out more yes so the first place to go would be Azure document in uh the document the Azure document intelligence studio um here you can find resources um that will help you get up and started very quickly and allow you to test out all of our services right here in the studio um another good place to go would be our documentation um we keep all the information up to date with what's coming what's new and there'd be another great place for you to check out what we have coming through the pipeline well this has been amazing my friend thank you so much for spending some time with us thank you so much for having me it's been a boss and also thank you so much for watching we learning all about elevating your document management with Azure AI document intelligence here on the ai ai show thank you so much for watching and hopefully we'll see you next time take [Music] care how about them apples I I forgot to I did everything right except turn my volume up my friend how about them apples that was fun right yeah that was awesome and if I may I just like to address something in the chat um Denise SC said no dark mode I like to say I always use dark mode okay I had to turn it off for the demo I always use dark mode just had that out there they tell and so there is dark mode dark mode or as we say dark mode that's how we say it in our heads uh there is dark mode he just we they tell us to use light mode when we're presenting uh and I break that rule all the time okay let's go to some questions there was a ton of question you okay take some questions my friend yeah let's do it all right so the first question is hold on there was more there was uh okay are there any limitations on the document size this is from barish yeah so right now the limitation is a th000 Pages um per document um if you have more than a thousand page document first of all I'm I'm sorry but right now it is a thousand if you have more than a thousand you can split it into two calls but now the limit is a th000 for a single call oh my goodness if you have more than a thousand Pages bemma [Music] says dude you got some cool lights going on back there my friend I I just did realize that that was not planned actually no no no no it was totally planned we want the sparkly to be to be there all right so here's another one uh from mitsuru uh does it support sentence reconstruction when a paragraph is broken by a page break for example oh so right now I do not believe cross page sentences are supported um we have had requests for this so is something that we are looking into um but I would say what I normally say is just try it out and and see but I do not believe it is supported right now but I do know it's a request we've gotten and that's cool because as a PM uh we it's sometimes a thankless job the way we know something is right is people will say to us and that kid you not they'll they'll say but can it also do blah and that me in my in my PM brain I'm like okay we got the right thing and now they're nitpicky if we get to nitpicky stages then we've got the right thing so I feel like this is the right thing and it's been the right thing for a long time there's a ton of people using this stuff right A lot of people and people using it in ways I never would have expected them to use it that has to be the coolest part when you see a customer use your product in a way you didn't think it could even be used so it's always great to see those use and I tell people and no offense I tell people like the best AI is the one that does the most boring stuff like imagine you are the one that has to like because we all go to the doctor right and every every six months we got to fill out the same form and I'm just like can you just keep the form I filled out it's the same one where you do you have to check all the boxes you know is document intelligence not extremely to you this is my life's passion no it's actually it's the most useful AI ever because like look reading stuff off of paper is so ubiquitous across everything we do that it's the most exciting and also the most mundane but most useful but I'm not gonna say anything all right here's another one is there a maximum recommended document size for this kind of use so it's still that a thousand Pages um if you do have a larger document it's going to take a longer period of time so if you can break it into smaller chunks and if sort of latency is something that you're worried about then I'd say you probably definitely want to go to a smaller size but a thousand is the limit yeah okay uh so uh but is there a recommended size for optimal uh like like like obviously these models are awesome but is there is there like a sweet spot for the size of the model I would say in my own personal testing I'm sure there's a there's a there is an answer um uh but I would say my own personal testing once you go over about 20ish Pages you start to see a little bit of latency so I'd say keeping it under 20 you should see the the short amount of latency that they're expecting I love it so here's another question because you talked about Lang chain but uh here's a question from Cuda how about semantic kernel support uh it's all the same thing right you could use semantic kernel Lang chain uh what say you so right now the integration that we have is directly with Lang chain um more to come hopefully um fingers crossed um but this is the the direct integration that we have now um but keep your eyes peeled for more Integrations in the future I love it but I mean if you want to use semantic kernel in general you could totally use semantic kernel and write code and do whatever you want so exactly because in the end it's just an API call right yeah uh and so yeah here's one question that we always get asked on the AI show uh and BMA this is an important one is there a risk that the AI software will grow and become [Music] self-aware no well in my case if my AI software became sentient and started processing all my documents I find that pretty cool so over here I think we're good now they start get their own bodies concerned but hey documents for me but then they'll be processing our documents exactly so it's okay I'm perfectly fine that pointed at the at the mail and all the junk mail that we get is can you process these documents for me and it's like yes has like a shredder in its mouth or something I don't this this got weird maybe we should move on here's another question uh do you know when AI will be integrated in Visual Studio that's a good question it already is if you look at uh co-pilot I think GitHub co-pilot is integrated in Visual Studio already right yeah yeah so it should be there um but for more information on that you're going to have to ask the visual studio the visual studio team um yeah let's to keep that under loocking key here's my good friend Lynn she is one of our fantastic researchers here on uh at not the AI show but just the product group is uh ingesting document uh for index creation is it a one onetime thing or does it also create a persistent index that's a good question so anything that we process we do not hold on to um so you're going to have to hold any information that we process and provide to you on your side this is more of like a regulatory thing um we don't want to hold any personal data so we just just give you the data and then you you do all the magic with it so if there's an index that you want to keep um you're going to have to store that and preferably you know some Azure blob storage yeah something like that or something or Azure AI search there you goh so so this is just a it's a transient thing just to get the because I the thing about this that we are learning B and you you know this too is that building robust AI applications that use llms in particular with documents that have been chunked requires a a pretty sophisticated pipeline of you managing your data so that the llm can do so that the search can work and the llm can do the thing and it's just uh what say you I agree plus one it's like yes yes all right uh here's another here's another uh he already answered this question will Tech Will you provide a semantic same semantic chunking as stuff to come he's saying he's saying watch the space uh here's another one uh is this Incorporated with Microsoft Dynamics 365 are planning to incorporate to fill out Dynamics 365 Fields oh yeah so we work very closely with Dynamics 365 and they normally I want to say take one to two months to come to parody with what we have after we GA a service W um so expect them to be implementing similar capabilities um in one to two months after we GA so this is a preview um but once we hit GA expect them to follow uh soon after here's an interesting here's an interesting uh uh experiment a thought experiment this appears basically a switch from a mysqldb setup to an online on thefly fix for cloud computer it's basically like like your papers have doc have information in them so why not use the information in a structured way right exactly yeah um I think this that's sort of the beauty of having these things available on the cloud um it's very convenient Carl was like the Disco lights are the best he turned turn him off though he's trying to be profession I didn't I didn't want to distract from the document intelligence this is not the kind of show we run here we're not professionals here here's another one I noticed that the paragraph class has regions that could Define itself across different pages so I thought maybe okay that's the the gentleman that did the the the try it out uh because the chunking size is like a more of like an art than a science as far as I know um here's another one um this is an interesting question uh maybe you how to show us code or anything but can you gu us how we can easily add a custom skill set search as specific date finder on page counter or so it turns out and I think I know the answer to this one when it comes to Azure AI search you can build specific skills that take information and one of those skills can be AI document intelligence is that is that is that right BMA that is my understanding um so I'm I'm a little bit confused as to what exactly the question is here yeah I what I I am too but I wanted to go higher level and be like you can use AI document intelligence in a uh search pipeline using a skill and that's something that you can do in fact the one of the first demos I ever did and this is uh true story with uh it used to be called cognitive search is I used something formally called form recognizer as a skill to ingest like a ton of PDFs and it basically reconstructed our entire order system that I fat finger deleted that's the story I did it's really funny it was like it was like there used to be there was something called the ignite tour and I went out and I was like I I deleted my database oh no what will we do well we have all the PDFs and it basically with all the orders it reconstructed the entire database wow which is which if you think about it you totally could do with AI document intelligence right yeah okay uh and someone is saying I didn't know until now that I must get a voice processor indeed yeah me either you would think that my co-host would let me know that I'm letting you know ly because I wanted the reaction to be authentic the echo is my favorite yeah there's a bunch of them like I can I can have like a a robot [Music] voice that's cool yeah that's cool I'd use it sometimes that was out of context that my usage there but that's okay uh so B anything any last things you want to add about AI document intelligence before we we uh let you go and I move on to the next thing um just we're really excited for everybody to try it out um I saw a comment about the Lang chain sort of um demo that there will be a blog post out later this week about that so have all the instructions for you to try that out but yeah just thank you guys for joining I'm really excited for you guys to try this out please let us know everything uh you want to add to the service we're constantly improving it so correct all right my friend we'll see you later I put down here the links if you want to go to AI document intelligence just go to aka.ms AI doc intelligence and then if you want to look at some of the documentation you go to AI doc intelligence docs uh to that uh for all the goodness there and hopefully B will'll have you on um next time when we have more good stuff to show hey and I'll have my voice changer ready nice night but leave the lights on okay all right buddy we'll see you soon all right my friends that was bemma uh delightful human being all right so uh the the next thing I promised to do was uh this goodness here uh number two no I didn't no hand mind's out of the potty I was not going to do number two I was going to do number two here visual promp flow with the newer bits so let's uh open this up um let me oh good I'm glad I CU sometimes I open the project and it's like wrong thing okay all right let me make sure my environment variable is not uh open so let's put this here so just a reminder uh I made a promp flow a long time ago a long time ago a prom flow that takes an image oh what's new here oh reload reload uh a prom flow that takes an image and I broke it uh because uh it's so let's uh spend some time for those that aren't familiar with prom flow prom flow is a lowlevel declarative orchestrator for creating llm inspired apps inspired or using okay uh and the way it works is there is this thing called a flow. dag and this flow. dag basically tells it it's a declarative way of of saying this is all the stuff you do we recognize that this is a pain in the tokus to edit and so there is a nice little visual editor that tells you what to do and the thing that happened with me is that this particular node here uh uh broke on me because I needed to get a new thing so let me let me show you so effectively what this does is this um this takes an image uh and looks like the image is here data data um iock 14094 small this image and what it does is it basically uh looks at the picture I say I want all all these things and then the prom flow decides what to show me so let me run this here in uh local mode here uh so the first thing it does is it goes to the image and it processes the image and decides is this image even an image like if it's not big enough don't even look at it right uh and if it if it uh if it is big enough it processes the image if it isn't then it goes and it says oh I it's not an image so let's do this I do a prompt and run it through GPT 35 turbo you are a Koso Outdoors product assistant who helps users formulate great search queries based upon their questions blah blah blah uh and then it formats stuff in the output use these categories for do not if you don't have any suggestions please return the following uh so I'm like trying I'm like literally coding with English by the way by the way I can I pontificate here for a second one of the reasons why I like promp flow and all the other orchestrators are great like Lang chain samand kernel no beef but one of the reasons I like prompt flow is because it puts the prompt at the center of all the work there's no way to go get around writing your own prompt there's no way sorry people are pinging me it's like it's like the Monday when Microsoft goes to sleep but there's still work to be done and so people ping us so notice um notice that uh this is the actual asset that I think is the most important the actual prompt uh let me make this a little bigger so you can see it uh and this is the actual prompt so this is what the prompt looks like when it does not um get an image but when it does get an image then I use this prompt here and this prompt is you are the Koso Outdoors reviewer who looks for oh let me uh how do we word wrap here view word wrap word crap you are the Koso outdo who looks at all possible products in an submitted image these products should be listed in a way they can blah blah blah blah blah do not include things blah blah blah an example might be like this oh this might be this might be in the wrong spot this should be here oops no this is not this is okay I might I might have to play around but again this is the prompt but the difference between a multimodal prompt and a regular prompt is that in the multimodal prompt we have this image uh this should be changed to customer customer customer image okay so what I've done is I've hoisted the customer image into the system message we got to get some colorizers for this uh okay cool so uh you can see this is the customer image and then it goes through there uh so looks like in this case the image was real so the image went through here once the image goes through here then we do the generate embeddings and the generate embeddings for each of the for each of the return things what it does is it creates a search embedding my by the way the reason why prom flow is awesome too is because you can actually see the output so here's the output of the um of the thing and this is the picture if you remember uh let's split right split spit right let's go to the flow. dag here so you can remember that the output of this thing is this so a Yellow dome camping tent a foldable portable folding and collapsible camping table so these are the things that it found in the picture and this is why I like prom flow it's really nice for debugging okay so that's what it found then when we generate the embeddings for each of those three things we actually create a embedding and so you saw the code for the generate embedding uh notice I'm taking a open Ai and the reason why I'm I'm using this code is because I wanted to do an embedding in one shot what does that mean I wanted to pass a list of things so that all the embeddings happen at once and then you can see I do an embedding and then I do a zip into a dictionary okay uh okay okay okay okay okay okay cool uh oh I did not know you could click on that that's handy but anyways here's the here's the output and the output as you would expect were here are the sets of embeddings so for the Yellow dome camping tent I hopefully uh input output let's do that so for the Yellow dome camping tent you'll see that this is the embedding for the P foldable porting chair uh folding chair you'll see this is the embedding and then for the collapsible camping table this is the embedding so noce at uh the the imager the thing with the multimodal gbt 4 Turbo with vision it extracts adjacent object then I parse and then push into a search index uh well I am I do the embeddings I I do I vectorize it and then I do the embeddings okay so let's uh close this and let's we'll look at the retriev products the retriev products hey I can click this thing now the retriev products takes those uh items and it does a search for each one one of them and then it removes the duplicates and that's it and so now when you go to the the dag for the retrieve products you're going to see the output is here's the request a Yellow dome camping tent a portable com and here are the products uh pretty cool right uh so now you can see like as you're building these things the reason why I love prom flow between you me and the wall is because it lets me see exactly what is going on in the flow of things so you're able to actually trace the flow of the data from the top all the way to the bottom uh so that's what this is okay cool so now that we've retrieved the products we're going to do a customer prompt and if you look at this the customer prompt uh looks like this uh so let's view here and then let's do a word wrap this is the new llm task that goes in5 35 turbo and then it also looks at look like if it us an image then it use an image otherwise don't um do you see and so this is using Ginger 2 for those that are wondering and Ginger 2 let me open a a browser a browser ging Ginger [Music] 2 is effectively just a prompt uh a a a template engine for python see you can do all of this stuff Ginger is a modern design hold let's let's sell it a little bit here let's s sell it a little bit Ginger is a modern and designer friendly templating language for python modeled after d Jango templates it is fast widely used and secure with the optional sandbox template execution environment get yours today that was it was that was the thing uh so everything that works in ginger will work um for this template thing because it's basically just a ginger template cool beans cool beans uh so in this case notice that there's a lot of if else's if else's if it's used an image um otherwise uh it this is just this is the actual thing right as an AI agent for the Koso outdoor retail and the cool thing is you can look at this uh you can look at this thing um and see what output it had which is really cool uh there you go and then finally uh this is the llm response in the llm response there's just one final um one final thing where I have the promp text and then the user question and then I for item roll whatever okay and so now the output is this thing this is the output is this the output view word wrap oh here it is awesome I found some great products that match your requests for the Yellow dome camping tent I suggest the trail Master see that is cool to pair with your tent I suggest the Trek Master camping chair and last but not least and a collapsible base camp full in table so you can enjoy meals together don't forget creating an account allows you to track your orders save your payment and shipping info and even provide access to it's a no-brainer for sure but here's the other thing that I did not show you and this is what makes prom flow absolutely delightful notice that I have a customer ID and this customer ID thing if it's zero then it just does nothing and so but if it isn't zero it gets a customer number it gets a customer uh and this is the context and it gets active user so for example the input is a customer and uh this is a customer fish so this gets swatted swatted in but here's the cool thing let me change the actual um customer to customer number let's say four and let me run it again hopefully it works if it doesn't everyone cross your fingers and toes here we go it's happening look it's happening it's happening there we go okay so this whole other thing is exactly the same like for example even if I go to open a new tab you can see oh wow I found more stuff green and gray what did I change something up here what did I change green and gray it found the wrong things oh let me look at the output again waterproof yellow rain jacket how did it find that wait a minute I'm G to run this again let me let me make sure data small custom ready number four I need to do some prompt engineering because I have not tested that side of the thing yet okay so let us draw this run one more time and let's hopefully see if it finds the right things so let's go to this node right here because this is the one that's the one that takes the longest it takes 10 seconds every time uh by the way keep your questions coming I forgot to look [Music] questions here we go let's see what the output is of this thing open a new tab wow it found a lot of extra stuff black hiking backpack where did it find all this stuff I don't like that um I don't like that let's change the let's change it as we can uh use the following uh for your assessment cool please return only the return only the most prominent only most prominent in the image they're not prominent include at most how many yeah there you go we're letting the AI tell us what to do telling the AI what the AI will tell us what to do okay so now let's run it and let's see if this makes it a little bit better by the way this is what prompt engineering is uh because I've never tried I've actually never tried this side of the prom that pulls a customer in and so something might be happening all right so let's open portable gr gas Gill did I do some um maybe there's something wrong maybe it needs to be like this image maybe that broke it but that seems weird to me that it would do that let's do it again run in standard mode okay here we go uh output of the llm here oh man it is almost time for the walk-off music so let's put this [Music] on okay let's take a look at the output now here to see uh yeah looks like it needed that maybe that there's a parer in the back for that um I'll have to ask about that because that feels like a bug by the way uh let me go back I don't think I need to change this let me just do this because I think this is an important instruction you know so let's do this and let's run it one more time by the way isn't it cool that like we found an error in a prompt we found the return thing because usually like I said uh when you're running these things with um orchestrators that are higher level it's hard to know what's going on because they do the prompt for you um but this doesn't you have to you have to do it yourself so let's open the yeah okay cool now now that we've added the customer information though notice that the customer lookup has Sarah Lee in there right and the last couple of things that she's purchased sleeping bag in a tent okay so now those things the retrieve products you can see brings all the stuff and now when it goes into the llm let's see what the output is take a look at that pretty cool you are a platinum member of Koso that's cool so let's open this in a new tab here so you can see um view uh and let's uh uh word WRA this is cool right hi Sarah cuz remember before it was like suggesting that they that uh they should join but now it's like oh Sarah after looking at things I recommend the following products that will definitely make it this is I mean um it's really cool um so notice that this is now a multimodal llm and uh uh next time we're going to be doing a little bit more with this in the AI Studio but I wanted to get this up and rolling uh also uh I I need to get the end card uh here for the uh triangles download uh the next AI show uh is not until uh let me look at the calendar here uh it's not until um the first week of January let me calendar that look my calendar that look at my calendar let me uh let me let me close this here um and my calendar says my calendar says that the next AI show is going to be on January 8th sneak peek here sneak peek join us Monday January 8th for our live stream 8:30 a.m. Pacific intro to type chat with the man himself Andrew halsberg delightful person delightful person uh so we'll be talking to Anders halsberg and then we'll do multimodal chat with prom flow and gbt 4 Turbo with vision inside of azure I studio uh the stuff I showed you today but we're going to move it up to the Azure I studio so hopefully it works so that is the next AI show uh same bat time go to aka.ms show live to get to the next one we we'll uh we'll put that one up soon but as always is a privilege and a pleasure to be with every one of you uh um next time on the AI show make sure you tune in Andrew is going to come uh and hopefully you have a restful and wonderful holiday thank you so much for watching and hopefully we'll see you next time on this the AI show live take care [Music]