Running AI Application in Minutes w/ AI Templates: Gabriela de Queiroz, Pamela Fox, Harald Kirschner

Key Takeaways

[Music] thank you so much for coming to the workshop my name is Gabriela dearo and I'm director of AI at Microsoft I have Pamela here I'm Pamela and I'm a python Cloud advocate so well done on those you who said python um but I Al I worked in JavaScript for then for quite a long time and I generally like lots of languages hi I'm Harold I'm a PM on vs code and GitHub co-pilot chat so awesome um so today we are going to be talking or showing you how to run a AI application in minutes so we are going to have a lot of like Hands-On so be ready to do like some coding not coding but like going through some coding uh using different tools uh GitHub code spaces Azure and and other tools that we are going to be talking about but now just to give a a overview of like the agenda I'm going to be talking about Microsoft for startups a little bit some of the Partnerships some of the pain points and then we'll go through the AI templates and handson uh so Microsoft has a program for startups so if you have an idea if you have a startup uh you can apply to this program and what I always tell people is you don't have to have a startup per se but if you have an idea that's enough to apply for this program and you get a lot of benefits and benefits that can be um I'll just skip it can be like credits so you get up to $150,000 in Azure credits you also have third party benefits like a lot of like different tools that you can use and then of course GitHub Microsoft 365 LinkedIn preman and more uh you can use all the different models from open AI but also like Lama uh models from kohary mistra and so on and the the piece that I like the most is about the sessions that you can get oneon-one sessions with people like me or Pamela uh that uh we volunteer our time to share our knowledge with Founders uh we can talk about maybe like I don't know you are hiring and then I'm an expert in hiring so you come and talk to me and I say hey these are some of the best practice for you when you are building your team or you can go to technical sessions and ask more like technical pieces um as well and inside this platform we have like several things other than the benefits that I mentioned and the guidance that I just mentioned is what we call build with AI and inside we have some AI templates that the idea is like you you we can help you accelerate um the the AI application piece with some kind of like skeleton in a way um so so you have something up and running in like few minutes um so again you get Cloud credits you have access to Dev tools you have the AI templates you have the one one one1 guidance um and no matter where you are in your journey if you have an idea if you are already building or if we're scaling this programs for you um you have access to All The Cutting Edge AI tools so you can innovate and streamline your AI development and on top of like the founders of this program that we have you also have like programs uh that it kind of like it's like the next step like let's say you are now scaling growing and then you use all the credits what is next there is a next like you know we try to guide you through the whole process so there is something called the Pegasus program um where we help you to coell go to market and so on and then there are some like strategic VC partners and like accelerators that we partner with so we have partnership with why commun Ator Neo The Alchemist uh Etc um pain points for startups there are a bunch of them uh one of them is like you don't have time you cannot wait to go to market you have to go like as fast as you can you have a lot of like resource constraints we have some issues with scalability you have you don't have the support and guidance and that's where we are trying to uh help you with so now we are going to go to the fun Parts it's like the AI template so that's where Pamela is going to show you all the amazing things that you can do with all the different tools all right so our goal today is potentially having you deploy maybe even three different templates okay um H so we have three different ones all like just um you know show in the in the brow which ones we're going to be deploying right so we have starting we're going to start simple with this chat application here just to make sure everything's up and working and then we've got two different rag applications one of them is a rag on a postgress database like rag on a postgress table that does a SQL filter building and then we have rag on a unstructured document so here I've got a rag on my personal blog or like a rag on you know internal company documents whatever it is that you're going to whatever kind of documents you're going to rag on so those are the three templates we're going to be looking at today and we have it all set up so that you should be able to deploy those templates without spending any of your own money and doing it all through our credits which is yay all right so um the first thing you need to do is get this URL so everybody open this URL on your computer so it's aka.ms e-workshop it should open up a a Word document in the browser that looks like the screenshot you see here so you can either type in the URL or scan that QR code and get that open on your machine so let's make sure everyone's got it open welcome welcome so go ahead once you've got your computer ready put this uh put this URL in your browser Harold maybe you can just memorize it and then help anyone who doesn't have it yeah ae- Workshop uh okay so then let me go to that actual doc here so the first thing you need is a GitHub account does anybody here not have a GitHub account okay so everyone here has a GitHub account great if you don't have a GitHub account you can sign up for one for free right now and um and that should be fine um the next next thing you need is an aure pass so this is something that we've got for this workshop for this conference and this is going to let you deploy stuff on aure without spending any of your own money so we got a passes for 50 bucks and they're valid for 7 days so if you do want to keep hacking after the workshop you can keep using your pass and uh after 7 days it'll disappear just like Cinderella and the Pumpkin uh so in order to get that as your pass you do need to have some sort of Microsoft account so you can use your like uh you could use a personal Microsoft account if you have one uh so if you're if you like how do you tell which one you're logged into right now I guess if you just go to outlook. office.com maybe you know what Microsoft account you're currently logged into um and then you can see some people in the last Workshop were like logged into their kids Minecraft account so just uh just you you need a Microsoft account and you might want to double check to see which one you're currently signed into if you are signed into a Microsoft account if you don't have a Microsoft account no big deal you can make one on the spot I made one this morning so uh if you do need to make one you can just make up a new outlook address and set it up that way um so you can also make it as part of this progress so we're going to go to this a checkin URL and that's linked from this doc here so if you don't have this doc if you just came in we can help you get this doc open so we can get this URL and uh we're going to spend 10 minutes making sure we get through this step since it can be a little a little tricky so when you go to this checkin URL right we put this in the browser it loads this is what you're going to see and it says I can either create a GitHub account or log in with GitHub so I'm going to log in with GitHub because I already have a GitHub account and I'm logged into this browser with it already so I'm just going to click on that and so what that's going to do is create a pass for my GitHub account and so we get a pass so each of us will get a different code based off our GitHub account so this is my you know basically my Azure pass promo code so I can copy that and then there's this button here that says get on board with Azure this is the next step is to click this and then we get this screen which says okay this is you can start and when I click this here it says what my currently logged in account is so this is where you should check to make sure you're happy with what account you're logged in with and you don't want to switch um I don't recommend using a corporate account if you do have a corporate account like don't just don't use it it's going to be problematic for various reasons because corporate accounts may have restrictions that won't let you deploy things so we do recommend using some sort of personal account or making up a new account so that's why you see I'm using my Gmail instead of my Microsoft uh so I'll confirm my account and then I can enter the promo code and that was from this screen so I still have this screen open so I just go there I paste it in and then we go s uh 6 x y y k I think it's case insensitive submit and then it's going to actually fail for me because I've already set this up on on this thing here um and this if you see this it's because you've already actually gone through this stage uh so for you it should work the first time and then uh it'll create the Azure account for you and if it works then what we can do is go to portal. azure.com so portal. azure.com and we'll see what it how it loads in does a bunch of redirects and then we can click on subscriptions and what we should see is there should be at least one subscription that says as your pass Dash sponsorship so that's our key that we have done this correctly and as long as we use this subscription when we're doing our deploys we will not get charged any money well Microsoft will but you won't that's the important part okay so we're going to spend 10 minutes to make sure that we can get everyone through this this stage so that we're all on the same page going forward so if you already got it that's awesome you can um you know like look at Harold's like uh Facebook profile or something so once you have that set up the next step is the proxy um so I'll just show that uh so you can start playing with that uh so here's it's the next Link in here so the reason we have a proxy is because normally when you're using as your opening eye you actually have to fill out a form and say how you're going to use as your open Ai and then somebody says oh okay yeah that's a good use of open AI because Microsoft doesn't want able to use AI willy-nilly so we you know check to make sure that something adheres to our responsible AI principles uh we don't have enough time for you to go through that process while we're in a workshop so we've set up a an Azure opena proxy that you can use during the workshop with the repos and we have special instructions for how you can use this proxy with the repos since you can't use the actual as your open AI uh so this you can follow the link from the doc and log in with your GitHub account out uh I'll log out so I can show that log in with GitHub okay it says I'm logged in and then we have an API key and a proxy endpoint and that's all we need to be able to uh to use an as your open AI instance now normally I don't like to use keys and I tell everybody to avoid them but uh in this situation we are going to be using keys and uh yeah and these keys will expire at a certain point so we don't have to worry about them being exposed uh typically with keys we'd have to protect them very fiercely so that nobody was using them so you can go ahead and log into this and see your registration details and then you can even play around with the playground this is really similar to the aure open AI playground or the open.com playground if any of you played around with this uh you can see here you can play with the system message that's how you like say like oh you're an AI assistant that constantly makes pirate jokes y uh and then we update the system message oh private I wonder what it'll do there we go and then um let's see oh enter my API key okay so we need to enter the key I actually never used this before uh so we're going to enter the key not save it select a model okay so we select a model over here uh so we've got 35 turbo I didn't know we had four too you set up four as well cool we can use four four is better all right and four is slower but better uh okay and then uh please uh tell audience about open AI okay all right and you can see different parameters that we send and these are all getting sent to the open AI SDK so we say the model right here we've set up two models gbd3 5 turbo gbd4 those are often the ones you're picking between with open AI although now you've got GB 40 that's a good choice if you're doing something with vision something multimodal I wouldn't use it otherwise just based off of some experience we've had with it um but is a great one gb40 is good for vision uh so here you can see you with the combination of the system message and the user message so this is what we call user message this is what we call system message those combined together we get back a response like this where it describes open AI with lots of RS and mes and stuff uh we can you know change different parameters here like how many tokens it should send back the temperature is roughly the creativity uh top p is also roughly about creativity and there's some more advanced stuff there and you can see how many tokens you used on the way out and how many tokens you got on the response so you can play around with this playground to uh you know to try stuff out and make sure that uh that you're able to to use the key so this is just linked off of um off of this Workshop right so if you go to the workshop proxy you log in you'll get your key and your endpoint point you can go to that playground and you can play around with the playground to check that that's working but we just want to make sure everybody now has an Azure pass and is logged in to the proxy so that you have a key and an endpoint so we'll just check to see if anyone had any issues of that this step is hopefully okay all right so here's the like these are if you're looking for the models this is generally the the page to check um so you know gd4 gbd4 and going down those are the gb4 models gb5 you're saying there's a gbd3 5 that supports Vision no no oh four tuber with vision this one yeah so we were using that one but it's a lot slower yeah so that's why we I've started using 40 this one this is the G oh oh okay all right yeah so you just want to compare those so we'll just be using the basic GPD 35 and GPD just GPD 35 today actually and then also the embedding models okay so is everybody set up with the proxy okay all right so now we're going to actually get something working so we have this repo here so you can follow the link from the doc and it has readms for the three different projects that we can deploy and these readmes are specific to using them with the aure openai proxy uh so normally you can just use the the readmes that are on the repos itself but because we are using this AZ your open a proxy we do have to use a slightly different setup so we've made readme specific uh for this for this Workshop uh so we can start off on this uh open AI chat app Quick Start and make sure that that's all working so the first step is to open in GitHub code spaces so you can do that by clicking this button here have any of you used codes spaces before okay a couple people so code spaces will open a VSS code in your browser with a developer environment for that repo so you can actually use code spaces on any GitHub repo so you go to go any giab repo you click on code and you can make a code space for it so it's a way that you can start hacking on any repo uh very quickly so you can open this button here to open in code spaces and and I'll just go ahead and make a new one and I'll say create codespace so this is going to take a few minutes to load because what it's doing is that it's creating the environment for this repository it's opening VSS code in the browser and it's also just setting up vs code so if you actually have like if you use VSS code locally and you've got like extensions that you use locally it's it's actually potentially syncing those extensions and uh enabling them them here I should probably just not do that cuz then it would load faster for me um but yeah you can see in the bottom here as it's setting up and we'll just wait for it so this is you know the slowest part of using Code spaces is just the loading you a tip if you want faster code spaces there's pre-builts available as well yeah and I do have them on the third repo but I think I don't have it on this one so I I should have remembered to do prebuilds for all the repos right and the slowest part is probably installing all the dependencies in the build it's basically it's doing all the things you would do when you install it locally just automated and with a progress bar and at some point it will just light up yeah let's see what the you can even watch can we watch the logs for this one building codes Bas code there we go so if you like this sort of thing like if you like watching Docker containers build because that's what it's actually doing everything's a Docker container so you can actually watch it as it um builds everything here and now it's downloading all the requirements so these are all the python requirements so all the examples that we're going through today have a python back end and then some sort of JavaScript front end uh this one has what we call like a vanilla JavaScript front end as in I just wrote some JavaScript in a script tag H but then the other ones are much fancier so they've got a full typescript and a build system and react components uh using the Microsoft fluent UI uh you know web framework so you can kind of see the range of front ends there okay so you can see it's you know it's still going through the process but at least now uh we can see the file explorer has loaded so we can uh explore the files here and uh and I'll show I'll go ahead and show the the code if you're interested in the code uh it is in the source folder uh we are using a court application and I think nobody has heard of qu but uh has anyone here heard of flask or used flask great so quart is just the async version of flask so it's literally built on top of flask and one day it might be brought back into flask and it just you just take your flask code and you put ayns in it and then you've got you've got court uh that's really uh how it goes so uh if you haven't done async before in Python async is a way that uh if you use async with your functions they become co-routines and then they can be paused and waited on and it's important to use async when we're building applications with AI because we have these really long blocking calls to an AI API rate so we make a call to an llm and we send off our request and these llms they can take like two seconds 5 Seconds 10 seconds right depending on what we're doing and while that's happening we ideally want to to be able to handle other user requests coming in uh so that's why we use async framework so if we use an async framework then while we're making IO calls we can handle other user requests that are coming in so all of the ones that we see today have an async backend either court or fast API anyone heard of fast API it's very very popular these days yeah so fast API is the one most people know of as the acing framework um so I you know I I like both of them fairly equally uh so I I use a mix of both um but I just want to make sure people know about the value of async Frameworks okay so that's all in the coure app folder uh if you want to look at the code there so it is now finished okay anybody else get their code space loaded get a couple okay great finish configuring so I can I in the terminal yeah we are going to be using the terminal and if for some reason your terminal like goes away sometimes this happens in the codes space just click that plus right here sometimes my terminal kind of blinks out so I just click the plus and that'll give me a new terminal right Boop new terminal okay so here we are in the terminal U but actually the first thing we're going to do is that there's aemv do sample we're going to make a EMV file based off of that so I'm going to make a new file and I can do that using this little new file button up here so I'll just click that say new file and I'll type EMV uh you could also like copy and paste um and then I'm just going to paste the EMV in there you could even rename em. sample to. EMV I think that's another way um and then we need to fill in these values to match the values of the proxy so we'll go to the proxy and let's see where's my proxy open here so here's my proxy so I'm going to go ahead and fill in this one that's the end point so the end point should start with HTTP and end with slv1 and look like that in the middle uh so that's the end point that's where we'll be sending our open a requests then we need the key so we'll copy that and it'll look like that or slightly different for you and then the deployment is going to be the name of the deployment is gbd 35 turbo uh and that's also the name of the model in this casee so if you has anybody used open.com a few people okay so on open.com you just pick what model you're going to use and that's all you need with as your openi you have to make deployments based off of the model so you actually have a bunch of deployments and you could actually have multiple deployments of a gbd 35 turbo model that have different names so when you're working with aure open AI you have to know the deployment name not just the model name so that's one of the complexities of of using Azure open AI but it does give you more flexibility cuz you can can say oh this deployment is going to have 20 tokens per minute and this one's going to have 30 tokens per minute right and then you can like say which of your colleagues can use what like if they're all trying to like use up your deployment or whatever uh so it's more flexibility but you do have to specify it okay so now myv is set up so this is just so that I can run a a local server and I'm putting local server in quotes because I'm going to run a local server inside GitHub code spaces so it's actually running a local server not on my actual machine but inside the GitHub codes spaces development environment uh so to do that I grab I'll grab the command here that's going to run the court app and just give it and when with code spaces you do have to allow so you'll see this little thing that pops up so if you ever want to copy paste you have to allow for the terminal uh and then I have okay then I paste it and then you can see that it says it's running on this URL now you can't just paste this URL in the browser I'll show what will happen so if I paste in the browser I'm going to get an error because this is not running on my local machine this is running inside GitHub codes spaces so you have two ways to get to it one way is that if you just click on it uh like uh option click at least on my Mac so I Mouse over it'll tell me what to do mouse over option click so code spaces will actually detect that you're clicking on a local URL and it'll turn it into a codespace port URL and it's say this funky URL up here improved disco for me um and uh and that's actually you know like local for that GitHub machine and uh that's one way of doing it another way that you might like more is you go to your ports Tab and you're going to find it listed here and uh we'll see the you know the forwarded address and we can click on that or we can even click the GL Globe icon and we get to the same URL so there's many ways you can get to this locally running URL uh and uh and get to the special code space URL it and you can even change your Port visibility if you want to like share it with a colleague if you just or in a class you can change it to public and then you could actually ping this URL to someone else now this is not a deployed URL like you're not going to use this for like you know your deployed dril but it's fun it's good for development so now I've got this running locally and now we can type stuff be like what's the weather in San Francisco see if it's going to lie uh can see oh good that was a good answer I think this has been trained it refuse to answer it's always good when it refuses to answer something it shouldn't know um so we could go ahead and like you know I could change this now and change the system message and let's see where's our system message in in here right so right now my assistant message is just you are a helpful assistant like you are a assistant that cannot resist a good pasta joke stra pasta jokes I don't know I love llms okay so what's the weather today is it going to make a pasta joke [Laughter] [Music] all right it looks like might getting quite softy today don't forget your umbrella you might end up feeling like a so noodle so good uh so uh so that works but so here we go so now this is running locally um and so this is a good one like when we're developing we can just test things test things locally here the next thing we're going to do once we're happy with it we're like this is the best app it makes pasta jokes we're going to deploy it so then we move on to the deployment instructions so the first step of that is ASD off login so this is going to log in to our aure account that we made earlier so I'll do ASD off login and uh this is going to give us a device code that we're going to paste into this oaf uh browser flow so let me go and open it up maybe over here I think that's my Azure account that I'm using for this and then I go and I take this and I paste it in and I'm going to pick my account I'm going to use this one continue and uh okay and then we're logged in okay all right so that was the device code flow so you just want to make sure that you log into the account that you got with the pass right whatever account you use for the past that's what you want to log into the next step is to create or Gabriel should I pause like should we get through the local step first or should we keep going with ACD deploy yeah we can pause and see if everyone's got the local local one running actually that I think that might be good to do okay so let's just pause and see if there's any questions with getting the local one ready so yes someone asked like can we just run this locally you can totally run these locally as well we like to use get Cub code spaces in workshops because that reduces the number of potential developer environment issues if you want to run it locally uh you can either run it you know just with a python virtual environment and you just have to install all the requirements uh or you can run it with VSS code using the dev containers extension and that will do the dock Riz environment for you if you want kind of the benefit of the dock rise environment without um you know being in the browser and having to pay potentially pay for codes spaces so we should know also that GitHub codes spaces you have a limit of some number of hours a month either 60 or 120 it's 60 okay I must have paid more so uh so it's 60 um so you're not going to go over that today but eventually you could go over that if you use codes spaces a lot so if you're local right I think I have mine open locally as well and I'm just yeah locally I'm just using a a python virtual M so you're also welcome to try these things out locally if you like local environments uh just you know be a good person and make a python virtual M to manage your python dependencies right um yeah okay so I saw a lot of local things so I think we can move on to the addd um so yeah I did the login so you saw me do you saw me do the login here right and that's using the device code flow uh so you should see something like this happen from inside code spaces and next step is to make a new a environment so a is this tool we're using for deployment uh so we make a new environment name you can just call it like chat app whatever you want to call it and then what that does is it actually makes this do aure folder and it makes this chat app folder inside and that's where it's going to store all of our deployment environment variables so we need to set all configure anything we want to customize about our deployment we're going to configure that now and it's going to get update it's going to update this file here uh so the next thing we're going to do is set all these ASD environment variables so the ASD environment variables are different from the ones we just saw in the m the m is just for the local server ASD environment variables are for deployment uh sometimes we use the same but a lot of times we want our local environment to be slightly different from our deployed environment uh so we have two different ways of setting those variables all right so I first set these commands so this is just going to tell it to not create an Azure open AI because we're using the proxy and then we're going to set the name of the deployment to gbd 35 turbo then we need to set the key so I'm going to paste this and then I'm going to delete delete [Music] delete gosh that's what happens when you have Wi-Fi issues actually is you see it with the typing then I got to find my key again uh there we go so that sets the key and then I'm going to set the point I'm going to delete how we're going to do this here we go and get the end point all right so now I've set all these things now if I've done it correctly if I look at my DOT as your folder for that environment I I created I should see a EMV that looks like this so this is a EMV that's inside the dot as your folder so this is what is going to be used for the deployment and it's going to tell it you know this is how it's going to set up the AZ your open AI connection okay and now I'm just going to do I'm just going to type type thank you okay all right and then I'm going to do azd up here we go so what ASD up is doing is that it's actually deciding it's doing several stages okay so I I have to select an as your subscription in this case I only have one subscription so you just press enter uh if you had two subscriptions you would want to pick the sponsorship one uh then I select an Azure location to use typically you just choose one that's close to you so Central us is pretty good uh now what ACD is doing the first step is that it's actually packaging up the code that it's going to deploy later uh in this case you're deploying to as your container apps so it's packaging up a Docker container file so it's actually literally building a Docker container right now so if you do like working with Docker as your container apps is a great fit and a lot of people like Docker so we deploy a lot of stuff there but we also are going to be using Azure app service for one of the later templates uh so we've got lots of ways to deploy on Azure so you can see it building up that Docker the step after this is where it's actually going to create a your resources so it's going to create the container apps can create a container registry create a container apps environment and create a log analytics workspace so these are all the components of a containerized app on Azure and uh you know it's multiple components and we have to stitch them together the way we stitch them together is using infrastructure as code uh Has anyone used terraform here before okay so we have our own version of terraform it's called bicep and it is a infrastructures code which means we're declaring what resources we want to make right so we say oh we want to make log analytics we want to make container apps we want to make you know the actual container apps image and then we're going to sign some roles right so all of that is declared in this bicep file so that way you have repeatable repeatable processes for provisioning and this is really helpful when you're making complex applications on Azure because you might have like 10 different things you're using right uh you might have a postgress and a key Vault and redis cache and uh log analytics and app service and you want them all to tie together so you can declare what that you know what that infrastructure looks like and then uh and then put that in a bicep file and then deploy it you can also use terraform so if you really into terraform and very comfortable with it you could totally use terraform tier as well I don't know terraform I haven't used it personally so all of my examples do use bicep but if you want to send a PR with her form I'll I'll review it and just stamp it cuz I don't know how to reason about it so what you can see here is that it is actually creating uh the resources right now so you can watch it here you can also watch it in the portal it's not really super exciting to watch so this is the point where I usually fold my laundry um uh but because it can take some amount of time H or you can even get an error oh I I used okay so I already made one in central us for the earlier demo so I should have picked a different region so for this as your pass there is a constraint of one container app per region which is why we said in the readme that you should pick a region that you haven't picked before and I didn't pay attention to I read me uh so that one won't deploy so uh what I can do is I'm just going to make a I'll just make a new environment I'll just uh I'll just copy everything over um now what you shouldn't run into this because this would be your first uh your first environment right uh so chat app 2 and I'll just copy and paste we'll change chat to and then West us seems like a good region okay and then I'll asdm select chat app to there we go and then ACD up yeah okay so then it'll up do the up again but I have one of these already already deployed so I'll just open up the deployed one so you can can see deployed deployed is going to look pretty darn similar to what it looks like locally where's the one deployed okay so this one's deployed it looks pretty much the same as what it looks like running locally right the difference is that the URL is a container apps URL and you'll see this URL displayed in the terminal once it finishes successfully deploying you'll see this uh displayed let me see if I have that in my history anywhere from earlier today uh no you never know okay so let's see how it's going here yeah I lovingly handcrafted them yeah so um I yeah I write the bicep files some of them all the ones in core are actually from a shared repo that we just copy and paste from we're trying to move towards something called a AVM as your verified modules which are bicop files that are maintained and have security best practice in them so we'll we'll gradually be moving over but basically with bicep files like you can use ones from a central registry you can use ones from your own private registry if you're doing a lot of them uh or you can just use you know ones inside the folder um so there's a lot of techniques you can use depending on how much bicep you're using okay so now it's starting over and deploying again all right so let's walk around and or any questions on what I showed here all right so I saw saw a lot of AG deployments are going I saw some issues with like naming which I run into all the time as your has very obscure naming rules the safest thing is to do short names with no symbols in them and nothing fancy uh if you do have a naming rule you can just always do asdm new and make a new environment and you know and start over uh and that should be okay uh but generally the issues you run into with deployment are usually related to naming region constraints account constraints and that's probably yeah the ones you might run into all right so um we giv me a 45 minutes left I'm going to show you the other ones and uh and these are ones that you can uh that you can also start trying to deploy now and following very similar readmes right so the first one um actually the the these two are both about rag so first I'll talk briefly about rag right so let me first motivate it right so uh let's see like uh tell me what Pamela Fox uh likes to code on I don't know let's try this I'm trying to get it to lie um this is the pasta one okay so this one clearly lied spaghetti Fon great all right so then but then if I go to um this one right here tell me what Pamela Fox likes to code on this one will hopefully be more accurate at least have less pasta jokes here um and this is so basically what we're trying to show is that if we just ask an llm to answer a question it is it's very possible that it's just going to make something up um if that one it seems like there oh good I mean in this case it says it doesn't know what I like to code in I think I should have said like code in um you know like here like what python Frameworks does pamelo use let's try this one um so you know if it doesn't know the answer it'll say in this case yeah in this case it does know the answer because this is actually using the rag technique in order to answer questions based off a knowledge Source right um so those are our last two samples are about rag uh so the general approach of rag is that we get a user question we use that user question to search some sort of database or search engine we get back matching search results for that user question and then we send those uh to the large language model and say Here's the user question here are the sources now answer the question according to the sources and so now we can make customize applications that can actually synthesize and answer questions for any domain so we've got two rag samples here so one of them is rag on postgress so this is for the use case if you've got an existing database and you want to be able to ask questions about that database and have the llm answer accurately based on that so for the example uh you know database that I'm using I have product right so these this is a chat on products so uh you know our table is storing all the products for this website so I can say okay what is the best shoe for hiking so then it's going to go and search the database rows and get back matching rows and then come back and say okay this blah blah blah blah blah blah blah blah and it's going to includes citations so one of the key points of rag is to have citations so that users can verify where the information come from and see that it's actually legit information and we can also look at the uh the process for this rag flow here when we look on the the thought process here and as as we actually you see is that this rag flow is a multi-step process so the first process is actually what we call like the query rewriting phrase or the query clean phrase so that's where we take the user's question and we ask the llm like hey here's a user question turn this into a good search query because a lot a user question may not be that well formulated right like uh please tell me about the best shoes for hiking now okay so you know there like a user query and uh you know that's probably not the optimal search query for uh for a search so if we look now at the thought process we can see that the llm actually turned that whole long thing into best shoes for hiking so that's a better query uh so that's our query rewrite phase so that's an llm call then we get back the resulting rows from the database and then this is our call to the model that says hey you need to answer questions according to the sources here's how you should site your sources here's the user question and then here is all the sources so this is basically rag and then you know we're able to use it with different sorts of uh data data sources so that's rag on postgress so you can get that set up following really similar steps to the to the other one and you can even run that one locally first as well just on a local postgress database uh so here you can run the app locally uh this one is a little more fancy because you've got a react front in there then you can deploy to azer you're going to set similar VAR Ables and uh run it up so if you're interested in that you can start uh going through those steps and then you can customize it the other kind of rag that we have is rag on documents so if you're trying to ask questions about unstructured documents like you've got a bunch of PDFs or word docs Excel files uh anything like that you can actually put those into a search index and then search that so the example we have for that is rag with a your AI search and uh it's a really really full featured sample we've had it for the last like more than a year now and we've had thousands of developers deploy with it and put it into production and so it's been used for a ton of use cases and it's got a lot of features uh Speech voice Vision user access control lots of lots of cool things in it uh so let me show that was the one I was actually showing earlier with my blog right so here's you know I made a version of it that's just based off my blog posts and uh you know it can site my blog post I've also got this one here which is for an internal company handbook which is a very popular way of using it as well and so you can see for each of them we can you know click on the citations and uh and yeah so now this is a bit more complicated because here we have a multi-page document so we've got a 31 page PDF we can't just send an entire 31 page PDF to the llm because for a lot of our llms it's going to go beyond the context window right a lot of our l m have a context window limit so typically that's around 8K 8,000 tokens uh it can go up to 32k even 120k we're seeing um but typically they do have some sort of context window and even if they don't have some sort of context window llms can get lost if you give them too information too much information there's a research paper called Lost in the Middle where they did a study to see if they throw too much information at an llm like at what point it stops paying attention so we generally want to send the llm the most relevant chunks so what we do is that we first have this data ingestion phase that will take a PDF or whatever document takes a document it extracts all the text from it and we do that with Azure document intelligence which is very good at extracting text from all sorts of documents so we extract the text from it we Chunk Up the text into like good siiz chunks usually around 500 tokens each then we store each of those chunks in the search index along with their embeddings and that's what we actually search on and send and then we send right so if we look at the search results here we can actually see that the search results are just chunks from the PDF where we say here's the chunk here's the embedding this is the page it came from and this is the file it came from and we just send back those chunks uh so this is the most complicated of our architectures because we do have to have that data ingestion phase and that means we have to have a you know a script or a process that does that ingestion stage and you know here we can do it locally or or in the [Music] cloud so those are the two rag samples so we have um you know we have another 40 minutes so and we have like a good ratio here of of helpers to y'all so if either of those sound compelling to you like sound like a use case that you're interested in then uh you can try to deploy them now and see and see how they work um so once again you just go to the app templates Workshop shop repo and you can either pick rag on postgress or rag with AI search and then start going through through the steps to try it out uh these will take longer to deploy so it's good to start the deploy now um because they take they've got a lot more infrastructure to set up and then for the AI search it's got to do the whole ingestion step and that ingestion step takes a certain amount of time as well so yeah any questions before Su are you using any libraries for the chunking and all that stuff yeah that's a great question are we using libraries so when this sample was first created it was like last April it was before there was like really good established libraries we kind of used link chain but not heavily uh so all of ours is it's actually custom coded um now if you're going to use a library the big thing I would make sure you're doing is um using a token based chunker a lot of the Splitters out there are doing character-based splitting which is probably fine if you're doing English only documents but we do have lots of international customers and as soon as you start doing non-english documents then you really want to do stuff based off of tokens and not characters because imagine you take like a Chinese document and you you'd say like oh my chunks are a thousand characters long like that's a lot of tokens you you can like go over the context window really fast so we have token based chunking that we've implanted here uh there are there is token based chunking available in Lang chain so if you're going to use Lang chain um the thing to do is is find my colleague's blog post where he talked about it okay yeah working with cjk especially if you're doing anything non-english um he basically analyzed all the Splitters from linkchain um to figure out which of them properly worked with token based splitting and with cjk uh languages in particular um so we've implemented this oursel he actually to my my manager Anthony he he worked on it um but uh Lan chain and llama index both do a lot of this stuff uh they just you know they they take care behind the scenes so what you need is this you need the splitting and you can get that from basically from link chain because llama index uses link chain so I would just say use use l chain probably with this one so you can specify the chunk size and then uh then you just have to vectorize so that that's easy you just use the open a SDK and we do the batch embeddings with that um so that we can do a bunch at a time and then you just store it in AI search so the hard part is really the uh extracting the text so there we either use as your document intelligence in the cloud uh or we do have some local parsers too if somebody doesn't want to use document intelligence we use like Pi PDF uh we use our own csb parser because that's straightforward uh HTML for my blog I just use beautiful soup which is the python package that does html parsing right because I thought I could do a better job at it so this one I just used beautiful soup to extract the text so uh so and that's so you can do that as well and we've got beautiful soup in there so yeah there is actually a surprising amount of things that we've written ourselves for the AI search repo um if we were going to do it today we'd probably use the Lang chain splitter at least yeah good question sry long answer other questions ABM I set andice something else is it toile differences so generally with bicep what it does is that it tries to figure out and bip is really compiled down to arm and arm is just Json so what what you're actually doing is called an arm-based deployment so with arm-based deployments what they try to do is figure out what does your resource currently look like what are you saying you want it to look like and what changes does it need to make happen um so yeah we're like we'll probably switch over to AVM in a lot of our samples and we're probably just going to make sure like it we're trying to make it not have a change but if you want it to change then that's that's fine so you should totally be able to switch between AVM not AVM um as as you decide as you see fit um and the important thing is just it'll figure out the difference and just make sure you are on board with any changes that come up there is like so if you're doing um there's this a deployment command that does what if and that tells you like actually tells you uh what resources will change I want to figure out how we can do that with ACD I think ACD maybe has a dry run command so that might be what we try when we consider switching to AVM because we want to switch to AVM uh so that we don't have to maintain our own modules but we just want to make sure that we aware of any configuration changes that could happen uh there just different things yeah addd is a command line tool that um you know does the does the arm-based deployment and also does code deployment code upload right so I have this a your. here I didn't show this so a your. says this is the code that you're going to deploy to this host so addd does multiple things it does um provisioning which is basically doing an arm-based deployment which is equivalent to if you're doing [Music] a AZ deployment if if you know the Azure CI it's this a deployment command um so it's doing that and then it's also doing packaging and code deployment so um if you ever done like I don't know if you've ever done like web app up that's where you deploy code up to app service ASD will also do that for us so ASD is trying to do the whole workflow of you need to provision your resources and you need to deploy your code and we're trying to make this Central way of doing it across all of our offerings because right now with Azure if you know Azure but we've got like a billion different ways of doing things across all the different things and AD is trying to make a more common wave doing it so if you look at my um my GitHub repo I'm kind of a huge fan girl so you can see all of these repos are all ACD ified almost all uh that's what this ACD column is because for me it's the best way to deploy because it's repeatable right um so if you are looking for examples that I have quite a few here um but uh yeah so we're you know we should be able to do it on different host container apps func