LlamaIndex Webinar: RAG Beyond Basic Chatbots

Key Takeaways

hey everyone uh Jerry here from llama index and excited to feature a few really awesome hackthon projects that we uh that were winners of our last llama index hackathon um they R the theme of the hackathon was basically anything but trap bots so how can you use rag to build advanc applications that are beyond the simple Q&A apps that you typically see during hack Fon projects and so we're excited to feature uh four projects um so U planning um home AI uh nothing I think that's how I pronounce it um and then counselor co-pilot and so um we'll present these projects not in the order that I just mentioned but you know they'll be in the YouTube timeline and then uh excited to share these projects with you answer any questions that you have and they also feature really cool use cases of both llama index the open source framework as well as some of our recently released features like Lama pars um so without further Ado I'm going to pass it to uh Sophie and the rest of the Adu Team all right uh hello everyone we're Adu team I'm Sophie uh I'm actually full-time chip designer uh but that's okay and then that's my teammate Justin Justin is software engineer uh and then we have another person here that's Henry Henry uh used to also be software engineer uh and then we have a first person although I don't see her that's regene regene is still PhD student uh but focusing on gener AI uh so that's four of us uh and then uh should we just do a quick demo Justin and Henry uh so that's our project ad planner so our product is essentially you enter address and then you can zoom to address and then you get the satellite image of this uh LW and then you go to this image and you start um analyzing the city building code for this law for you uh so uh how do you actually build the uh the adus uh in the backyard so this is a this was uh live uh analyzing the uh C building code and then as you guys can see there are a minimum side of the real whatever like there's some uh rules you have to follow U and then after that uh this whole thing is analyzed uh oops uh jerck do you mind just like go back a little bit Yeah of that oh yeah so after this whole thing is analyzed uh the city building code will uh be fitting to like for example GPT and then you also output like a mask to mask on top of your property uh the mask that's red that means that it's unbuildable and then I we ask also calculate like a region that's actually buildable in the green box and then we will render the buildable flow plan on the top of the buildable Lots uh as there you can see and then by clicking on that a uh on that uh Flor plan of the Adu and then user will be able to direct it to the uh Adu vendors uh to actually initiate um the the building process yeah maybe oh just to take a quick step back um what was the motivation for this project uh because you know I know you're trying to narrate over like a 40 second video but we love to hear like the high level context of what uh motivated to your team uh Henry can you give us a you know motivation so um this was sort of to um allow the homeowners to put adus in their backyard in California you can do that now in other states as well so was just kind of to streamline the process um rather than having a vendor call a person call um all sorts of different vendors and it just kind of makes it easier for uh a person to build in their backyard and Adu kind of to solve the whole somewhat somewhat of the uh limitations of um homeless um you know lack of housing in California and others other areas of the country what was the stack that your team was using uh we were using a reactjs front end and then we had a flask back end and within the flask backend we were using llama index for our uh PDF parsing and to build a like an index over the uh building codes that that we were bringing in for that particular location right and Sophie I know you had a few other things um that you were going to say oh there are a few other things oh you were just I thought you were gonna uh continue because [Music] I um is there anything else that you and your team want to share [Music] um do you guys have anything to share um yeah uh i' know to share that for the uh image analysis uh we uh we utilize the GT 4way for the uh for the core uh Pro property poll and some some key key Parts um recognition but I I saw that someone in linking have some concern that since um LM could hallucinate uh so uh they have concern that whether we should uh you uze uh generative AI to some real um scenario and uh so in our case uh I I like to share that even if we adopt jpt 4way for the um for the property recognition but for the analysis for the accurate analysis we still you uh use computer vision model to uh make the uh accurate uh prediction so as you can see for the for the mask uh it's masked very accurate that's not only based on the U generative AI uh we uh so we we utilize generative AI for recognize uh where is the what are the core Key Parts in in in R and then we uh connected this uh data with the uh Outsider uh computer vision model so that it can mask the the exact um area of that and then based on the uh combined with the data we we we got from the Adu policy based on your address we make a good combination and Analysis and then uh calculate the uh calculate the potential uh available area that you can build ATU in your backyard and was that last part was that just uh herotic was is that an algorithm or is that also using a model oh what's algor sorry what's the question oh just that last piece for you're trying to calculate where to place the the unit um is that like an algorithm or are you also using a language model uh no I think we ccate by by ourself we uh we write that not yeah we write a model by ourself makes sense what's the what was over um could you talk through the stack about like how you use GPT 4V because I think it's very interesting um you put in an image feed it in through GPT 4V uh and then you translate that you said you you feed the input into a computer version model that does some sort of like you know segmentation or or detection of these like regions where you can't build right um and and so could you talk a little bit more about what the like how you feed in the input like into that model so the output of gbt 4B is a bunch of text so how do you translate that into something into into these um red boxes right here uh yeah um so F first I try to use uh GPT four-way because what we want is a a structured output like we we we want to uh get the uh uh coordinating coordinating data of the of the um of this this part uh sometimes uh it's surprisingly sometimes jt4 we can give me some uh some some uh ma mathematical data but I try that it's not very accurate so uh I CH that to only give me a analysis of for the for the recognition and then you uh you you find you find the Keyport like Po like uh these are the this are the main parts from the from the layout because also we found that property pool Drive R are the three um major parts uh we like to take into consideration in the ad building so then we uh we just use GPT 4way to find that and then uh some computer vision they are trained by a lot of U uh data to for as an example for po uh the computer vision if you you tell them find me uh what is the uh what is the coordinating AIS for for the poll that computer vision can based on their um uh training data to give you uh to recognize it and output a structure uh output and then we utilize that uh uh to connect with a master uh compter uh model uh compter Vision model that can mask that area got it from the structured output and then connect a image uh model to give it a mask and show it on the layout great thanks for sharing the information um and I think yeah I think that that rounds out the time pretty nicely and so with the um we could probably transition to the next project so thanks for presenting the Adu team and congrats on um a winner of the haon thank you very much thank you thank you great so I think the next one is uh counselor or co-pilot um are you guys ready sure thing um I can share my screen yeah that sounds great let me stop sharing all right can you guys see our deck yeah this looks great awesome um so uh yeah Ria do you w to take the lead here sure thanks Sarah um hi everyone uh my name is Ria um I'm from the counselor co-pilot team um you have all of our names down there Amanda deija me Sharon theun and Zara I'm excited to talk to you about our product today um so the motivation for this product um was our first St firsthand experience as counselors uh with the with the Trevor Project um so for those of you who are unfamiliar with the Trevor Project the Trevor Project is a nonprofit based in the United States that aims to um end suicide amongst lgbtq youth um we serve youth via text uh message and calls um and as a counselor myself I faced a number of issues uh when being a Trevor counselor mostly around the time spent um kind of searching for resources Googling for things online digging through documents in Salesforce filling out Pace forms all while also trying to manage multiple conversations with contacts at once so um our our goal was this with this project was to empower counselors like myself um to Aid youth in crisis um and kind of be able to focus on conversations rather than have to do all of the administrative tasks around um just starting a conversation um wrapping it up and then creating the documentation afterwards um so our solution has a couple of different parts to it um we are able to uh search for documents in Salesforce and kind of synthesize those uh for context before conversations we also draft responses based on Trevor Project guidelines um also if the conversation seems to Veer the direction of needing uh location specific resources we're able to search for those and email patients without any um kind of prompting from the counselor uh we also are able to fill out Salesforce forms after the conversation and also during the conversation um and more excited to talk into or speak uh to exactly how we developed this as well um but the goal of this was again to make sure that counselors can focus on conversations and on people rather than on paperwork um just for context this is what a general Trevor case looks like so as you can see there are quite a lot of fields that need to be filled um and then these are all transformed into a PDF that then needs to be parsed um if there are previous conversations with the same contact that we want to make sure we have the contacts of um and then next slide please um and then these are some examples of the Trevor resources this is just a very small um subsection of all the Trevor resources usually as a counselor I have multiple screens a bunch of tabs open trying to navigate between um different resources to make sure I'm providing the best care um possible um so we're hoping that with Trevor co-pilot this becomes a lot easier for not just Trevor counselors but any crisis counselors who have to talk to people but also take action at the same time awesome with that we'll pass it at the run to talk a little bit about how we built this all right awesome thanks Ria so as Ria mentioned you know the goal here is uh to sort of automate a lot of these laborious tasks and uh when we when we were designing this you know the key goal was you know the end user in mind both the uh counselor and the contact of the patient uh so the idea was that we wanted to make the process seamless so we didn't want the counselor to sort of you know do prompt Engineering in real time we wanted everything to flow based on the context history based on you know where the conversation is so that's that's what this architecture uh sort of you know U how it was created so out here I'm going to start at the bottom left corner so what we do is you know we take into account patient or contact history uh this may include their uh location information this may include their previous treatments and things like that and we also take into account the chat history right where is the conversation what's going on what was discussed above and all of this is fed into a uh llama react agent and the idea is that we want this react agent to sort of you know in real time take decisions uh and then you know provide the right relevant tool to the counselor right now what are these tools and what do they look like so there were six key tools that we had created so the first is information tools so these are you know more around providing information uh right to the counselor one is PDF parsing and summary this is based on the Llama index bars U we we totally loved using it and what we did was uh we basically extracted information from llama index bars use chat GPT to summarize it and like re mentioned you know this was the initial forms uh that had patient or contact information and as the chat started this tool basically served up this information so then the uh then the counselor had uh some context on the patient second was you know rag this was around all these documents uh that uh you know we used uh Lama index Rag and sort of you know built these reply suggestions using this second was action tools uh these uh these are tools that take actions for example when search for example local search for therapists then email compos and send this is a python mailer uh to send emails out and then finally there are two additional tools one was escalation response so for example based on chat history if if the react agent thinks that the that the context is severe that the that the case is severe it might it might suggest that you know it needs to be escalated and finally the form aill which we use at the you know which is is triggered at the end which actually summarizes the whole information so based on all of these tools and the uh the uh the history and the uh patient context it actually picks which tool to use and then it suggests that tool in the co-pilot suggestions and U I'll I'll pass on uh next to uh Thea to talk about the vector database yeah in this haon we have a couple of other sponsors uh who offered us the credits to use their services and we used uh data stack ASB for the vector database and Bento ML and coming to Bento ml Bento ml is a offers open source models uh which we can deploy in the bentor cloud and use those embeddings and we used sentence Transformers uh for our Trevor Project uh cons Copilot and um coming to data St as it's a vector database and it helped us to manage and R the uh Trevor Project guidelines uh documents uh very seamless in a very seamless way um uh now let's move on to the demo well so um so the demo gods were not really cooperating with us this morning um so we'll also be using a recording of our demo which I did take um actually earlier today but um from a UI perspective we really wanted to I guess nail a couple of things um so Zar could you zoom in really quick just the screen um yeah I think that's a little better um yeah so from a UI perspective we really wanted to nail a couple of things um we wanted the solution to focus on the conversation first and foremost with the AI assisted features off to the side um because like our primary goal was to reduce that cognitive load on counselors that are potentially juggling multiple high pressure conversations at the same time so that's why we designed the UI that way and then um you know also due to the uh potentially like high-risk nature of these conversations we also wanted to design with safety in mind um so since ultimately it should be you know humans vetting um whether what these uh responses um you know uh say before they go out um so we gave intentionally gave um counselors the flexibility to use um a suggested reply or not or modify send a modified version um and uh from a from a longer term perspective we even think like like this design would enable us to improve and fine-tune our Trevor co-pilot model um if we can integrate counselor response choices uh as an input for uh human feedback driven uh reinforcement learning um as an approach um and uh and then lastly um the other thing that we really wanted to nail from a UI perspective was all the so um I think we did show this earlier um but these uh case forms are just um so much work to read and populate um you know in real time so uh this sorry what um we really wanted to make sure that um firstly um they're parsed uh you know in an easy to adjust format which is what happens when um when the chat first opens um and we have this like kind of nice summary um and then at the end um we uh once like a conversation concludes um we try to populate the case form in this sort of structure um like autop populate whatever is possible um and I guess with that um if Amanda if you want to kind of take the take the reain there yeah sounds good so I'll go ahead and just provide some more context on the resource suggestion and the uh finding resources and form autofill feature so as we can see on the right hand side we have some recommended resources that um we have a agent in the background using llama index's agent feature um The Prompt for reply suggestions will provide recommended resources that we saw earlier from the Trevor Project guidelines um we we also have an AI agent running in the background to find local resources so for Trevor Tex counselors are required to provide location-based resources so as we can see in the conversation um about Midway through um the conversation basically asks okay can you help me look for therapists in my area so we use an agent to find therapists based on client area using llama index's Google Search tool and llama index's load and search tool so llama index's Google Search tool queries the Google search engine to receive a list of results and llama index's load and search tool loads this large amount of information so we do a lot of um automation based on chat context and when it makes sense to provide therapist contacts or other local resources we prompt the AI to send an email to share the top results of resources in their area and uh at the very end just to provide some more context of the form autofill feature um at the very end of the conversation we Prov we prompt another AI agent through llama index to automatically populate the CRM data of patient data based on the Trevor case that we saw earlier and with that I'll hand it off to Sharon thank you Amanda um uh Zara can we go back to the slide yeah slide 14 perfect um this being hi everyone my name is Sharon um this being a heckadon project there are definitely a few areas that we wanted to think about extending and doing better um and the first was around guard rails so initially when we as a picture shows when we started to um experiment with like using GPT for for a co-pilot you know it's they uh it's very safe right um and it you know it tries to tell you that like you need to go talk to a human which is not a bad response really um however not not doesn't quite work for our use case so we found that sometimes that could be like overly rigid right the model could be overly rigid over Generation Um and the other set of guard rails that we're considering is if our co-pilot was good enough potentially what we could do is then try to like guide like the human counselors because these are all volunteers they have different levels of experience and training and potentially some feedback could be useful for them as well so some of the the extension here really could be to I think Zara mentioned it like collect data of human chat history or as well as human feedback to provide the chat messages um so that we can do some fine tuning or maybe an open source llm that would be you know also cheaper to operate um and less finicky when it comes to the you know the demo gods that we suffer from today and potentially that could be extended to provide feedback from volunteers um on the right hand side the second point that we were thinking about is um we didn't really have time to d off into like how to optimize our R Performance I think some of the messages in the chat kind of hinted at the same thing um just to give you overview like our whole document set just for this hackathon was not I would say I feel like it's not big enough it was kind of in the T the low tens and that was what we could kind of gather given the time that we had but um absolutely there would have been more resources that I think the the typical travor counselor can actually access that might have been a bit sensitive for us to um to to kind of put in at at that point and we also didn't um start to measure like performance like how well is our the different aspects of retrieval and generation working so a couple extension to this area um the diagram below shows kind of the Trevor Project flow of the conversation has different stages and um I'm not sure you notice just now but some of these cheat sheets and resources and what to do next are targeted at these stages so it could be really helpful for a travor counselor to kind of get a sense of like which phase which conversation phase you're at and then if we divided our Vector collections by those phases then most it would be kind of more likely that we would be retrieving like the most relevant um documents to try to augment uh our generation of potential resources and also we could extend then to think about validation like potentially using fos like ragas or other of the stateoftheart fos out there and that's it for exential potential I will pass it back to um Ria yeah just to just to close this out um it was a lot of fun to work um on this hackathon project uh we are really proud of what we've built and are going to be talking to the Trevor Project and potentially um you know continuing to build it out and seeing um if counselors like myself were able to use it in actual conversations um but with that I would love to pass it over to you know Jerry for for questions for the team or to anyone in the chat um if there are questions that we haven't answered yet great thanks so much for the Fantastic presentation this is a great demo um I think unfortunately we don't have too much time for questions just to keep it um just to make sure we have time for two more presentations um but with that said passing it over to Raymond okay can you hear me yep awesome all right so I have no idea how I'm going to follow up on that one because that project is just so amazing and so inspiring um and yeah really excited for the opportunity to share a little bit about my project uh I it's live I just put a link in the chat for those who are there uh I it's it's spelled weird but it's anything to XYZ and uh hence the C the camel case so let me um I thought it'd be more fun for today to show in the app I do have some slides as kind of a backup in case we need it um but yeah let me give uh a little bit of an overview here um in terms of like the project so this uh this project was in the I think you guys called it the continuous innovation category of the hackathon so uh I had I had already uh built uh kind of a working prototype of this idea of can you take a different approach to 3D generative AI that is using Code um and for context there's a lot of uh people using things called neural Radiance fields or Nerfs um to generate 3D models and the simplest way of uh thinking about Nerfs is it's really first doing a 2D image and and that might be a generated or not but you might have pictures of an object from multiple angles or you might generate those with AI and then the Nerfs um uh is is really trying to figure out like well what is that object in 3D and it's basically interpolating like 2D to 3D and trained on huge sets and I think it's cool I run a 3D printing company I'm I like that's like this idea of like getting to 3D models making ideas real is kind of what I'm excited about and I guess the the pain point that got me thinking about this was like I've tried all the Nerfs I'm friends with like the original Nerf authors of the paper all the companies doing it I've tried everything and I like have never been able to get something I actually want in real life and so the Insight that led to anything was wait a minute like a is really good at writing code and code can generate 3D objects so so basically the the way that the workflow is there's a lot of things that you can change and tabs but when we prompt the AI um oops and my zoom thing is in the way of me seeing my own um prompt I have to get this sorry uh hope I I'm going to type this there we go flange fittings so this takes a little while to run but basically when we prompt the AI it the the AI part is really just a code gen Ai and it's trying to figure out what code it's writing which is using this build one2 3D framework which is a python package that you can think of as kind of a domain specific language um that it's helpful that the AI knows python code but knowing oh man that's not a good one all right demo guides are it's supposed to know that one all right the demo guides are not with us so um you can see we still have work to do on make it smarter but there's a number of different like uh so so I did this I loaded this one before because I kind of had a a sense that that would happen which is like simple text that you can extrude um we've coded in like some packages for doing like threaded objects uh so this is like if you type in like a hex nut it's supposed to find this one um I I found some examples of for Valentine's Day you know I had to get my wife something nice um and so it knows about sort of bezier Curves and and can kind of write the code so what we did at the hackathon that uh involves Rag and I guess first before I go into the Llama index specific part I want to you know thank thank my elders in the in the rag world uh namely Mayo ocean who I originally learned rag from and uh Mayo introduced me to Jerry uh so so shout out to Mayo he's really amazing guy um but the when we went into the hackathon uh I had this basically 10,000 token prompt that was required to get this thing to even sort of work at all right and like this a pretty hard problem happy to talk through some of the challenges of like what's what's really hard um about getting AI to generate 3D models but the um the I knew that we needed rag to make it smarter and so what we did in the hackathon was basically change this back end where now I've got like basically all of these different uh except for that one uh all these different examples are now in the dock store right and I have just a very simple um like working code examples as my documents each one of those its own node I'm not doing code chunking or anything like that the retriever is just giving me the top three examples so now there's a much lighter weight maybe thousand token system prompt that's just reminding it that it's got a output python code and and it's a really good python programmer and it's only going to use this framework and there's some tricks like if it doesn't save like in my in my backend on the server it only outputs the object called save you know little things like that and then the retriever is bringing the top three examples that are the most relevant um so what's so the the main benefit of what we got out of the hackathon which was actually it's interesting like my interest in participating in the hackathon and implementing RAB Rag and implementing LL index was really around making the AI smarter the unintended consequence and what I was actually able to achieve on Saturday was uh I lowered my costs by 5x because now that's not I'm not stuffing every single example I want it to see into the system prompt so for example like if you if you type like a boat you know I'm not paying for two tokens I'm paying for 10,000 and2 tokens in my first version right and now with rag you know it's only fetching those most relevant examples so now each each users's prompt is more like let's say 2,000 tokens and and now there's more variability depending on the the length of the examples so that was like our first real win was being able to uh reduce the costs and again right now this is like it's live you can use it for free and we have some number of like maximum free monthly queries um that folks can use and then maybe to sort of shift it more into the like future Direction the what I'm really excited about now so as I mentioned before like this is a really hard problem right like the a like the AI can't see what it's doing it's just trying to write code that generate something in 3D you could maybe imagine incorporating some kind of multimodal thing but frankly like multimodal the models are very young and they are not good at like this kind of like oh like that curve is the wrong shape or or this object with respect to you know they can like say oh that's a llama in your in your picture but they're they're not very good at engineering kinds of things and so what's really cool now and and maybe to share a little bit more about what we want to do with anything is um make this like it's going to be hard and it's going to take kind of a community effort of like human written examples so like this was an example I wanted to play around with like can you do lates and so you know we generate that example I wrote this and now this is in the training set or like one of the guys that's really involved with build one 123d named jern he like wrote this of like a light bulb right so like now again this is a human written example but like we can we now have a system with llama index uh and the and the Astro ddb uh Vector store to like add these examples build a pipeline for like testing them make sure there's no broken examples in the repo and basically as we get more users who are interested in it uh and building it then it gets smarter and smarter and smarter over time um and so that's where I think llama index kind of uh enables us to have this framework to have this community effort um and yeah it is really hard and like maybe one like one question honestly I have for you Jerry like I've been really struggling with um like what's the eval framework for this right like obviously like I could build something that just like does it render a working object but like I don't even like how how how would you approach finetuning like you can't find tun without evl so like what's the Evo when the output is something that I don't even know how to get the a to comprehend yeah I I you gotta put me on I I don't really know um but I because you know this is all like first you're generating code second it's rendering into a 3D model I'm sure if um we collectively thought about it for you know like an hour or so like there would be both like um kind of ways to use human labels as well as automatic approaches to do it like I think the default is to just collect a big data set and get a bunch of humans and like label it and and use that as training signal but I'm sure there's a bunch of like just um like unsupervised like 3D object metrics that you can use as fine-tuning signals too um and that part I don't really know yet but I like for instance I'm sure you could have heris sixs like whether or not the 3D object is consistent or or have like an AI model detect how good like this 3D object is yeah like kind of like a discriminator versus like the generator type thing um right yeah so anyways just not to not to like quiz you but you you've got a lot more experience with this stuff than I do and it's really stumped me and I agree like shortterm what I want to build is like a user base where like people are like up voting like oh this is the best one and then some of those good ones end up in the training set um and then I also I know we're coming up on time for for my section so I want to be respectful of time but I do want to just show a little bit about where I'm also going with this um which is so one I'd like to be able to get to like something like this right like how do you describe that how do you prompt that like obviously we have the code for this here but like how how would the AI like be able to understand like a much more complex object um and then lastly like I'm starting to play around with like uh a image workflow approach too so I can put some of the links for this in the chat or I've like pre-loaded the QR code for for people on YouTube and we can put these in the description and stuff but like can we now so this is a super simple example if you load this it'll load this poster like on your wall as a poster or this will like load what looks like a llama wherever you are in your physical space but like starting to so like like the end goal in mind is like help people make their ideas real like a natural language prompt to 3D model that like you'd want to use you can preview an AR like see it in your physical space is it the right size of the right shape but I also think I'm kind of interested in like baby steps of like okay like like images are like probably a million times easier than 3D models let's start with images and then can we like extrude images and then so playing around with some of that stuff too but anyways I'll pause if there's any questions or no time for questions just really appreciate the opportunity to to share a little bit about anything. XYZ and it's live uh at yeah at this website for people to try yeah one one thought on the um how do you describe this object I think someone I think H mentioned in the comments you could try feeding it to j4v or a visual model have it output initial text description and then use that as like a week training signal um so that basically at least you have some starter description for that yeah we've been I've been trying to do something like that and there's actually a a decent um there's this YouTuber named to tall Toby and he creates these like CAD competitions where he shows like here's the specs for this actually this part that we're looking at came from two t toi's competitions and so he has like a 2D image that's like you design this in CAD and then the build one2 3D Community figured out like how to write the code for it so there's actually like a not you know there's not hundreds of examples like this but there are at least like dozens of examples where you could imagine kind of riffing on a on a Vision model where you're like hey here's here are the instructions and description that led to this 3D thing and so that's definitely something that um like I've been playing around with and and my initial experiments failed horribly um it it won't be competing in the uh two to Toby CAD tournaments anytime soon well thanks so much for this presentation this is awesome I think I think it's a great uh Showcase of multimodal generation capabilities along with rag and it's definitely Beyond you know the the basic kind of like traps that people think of when they think of rag normally so great example of that project and moving on to last but not least um uh home AI um J are you here yeah I'm here I'm just muted okay sure the screen yeah that's right um so this is our home. a project um yeah so we are we we want to work on this because we want to help people on their home purchasing Journey because um buying a home is probably the most exp expensive transaction you On's life but it's um very frustrating and very time consuming so we are thinking to use LM to help people in this area um yeah uh for example you want to start buying a home and there are quite few steps like you start by searching you want to find a home that um like um follow your criteria for example you want a quiet place or you want to be want it to be um close to a public school or you want it to be far away from a super found so those are kind of the soft criteria which you can't quite um do in zow by clicking on the filter like they you can only choose the for example the four bedrooms or the bathrooms so those are kind of uh very hard filter and also we want to enable the soft filtering uh with language model like you can talk with the model and then it helps you with the soft criteria and the other part is um the disclosure part say you find the home which you think is very good then you want to read the disclosure to see whether there are some common fundamental issues like whether the foundation is good or correct or if the window is good or like how many remodels has it done or like any other uh problems in this um house um however the disclosures are very long there are like um there are there are often more than 100 pages and it's very hard for normal person to do that without pain so you will talk with your with your agent and spend hours on that that's also issue we want to do uh want to resolve uh we actually summarize the disclosure into a very one a very very short one pager so it's very easy for people to understand so the last step is you want to make an offer and we will show that later yeah that's the part for the um discloser part like you do with end paperwork and we want to make it very concise um here in the small demo say we start from here and you describe to the um here with natural language what kind of house do you want to have for example this one you want it to be no termite and no legal issues or there's no environment risk for example super font or it's very prent to earthquake say this um say the model returns you with a few homes and if you click into that you can see this disclosure summary the first summary is about the property details you have the address and also the property tax and all the basic information and the other one is for the major concerns which are the points where you might be most concerned about in this disclosure is it too small um yeah for example in this exterior you see there is a water damage to the roof or you can see there are electrical issues or there are Plumbing issues so if you find that in the disclosure it takes a huge amount of time and now you can see all that in your glance um and for uh what what kind of issues are classified as major concerns we have a real EST real estate agent in our team so we uh borrowed his insights to um to um use the in the prompt to find the major concerns yeah technically we use asra DB and Lama passer and Lama index framework so um so we can um U dump the other disclosures into the database in advance and when we are making the query it uh looks up the it looks up the uh database and uh give us a result so that's one discloser yeah and if you click into another home you can see other disclosures and this house has different issues with the previous one um say you are happy with these two homes and you want to to move forward and make an offer here is somewhere you can um input your own information and then it can help to draft an offer which you can Pro probably send to the uh to the to the CER well if you trust this two enough um ideally um a real estate agent can be replaced with this tool because it covers all the steps from searching and to the investigation and to the final step of making an offer but this is not quite um live yet because you can see this offer is not quite professional like we are still working on for example to make the PDFs more professional to be a real offer or to um make the search more um like more detailed yeah that's the market size with that's our estimation basically is like we have um 200,000 real estate agents so if we have um like we can replace a small portion of that then that will be a great great amount yeah questions great could you talk a little bit more about the stack that you used in building this because you have a bunch of um bunch of components I I'm noticing a little bit of the crate llama you have like this outer page as well where you can like uh select like different homes you have the disclosures curious how you built this whole thing uh yeah sure um I think the um the PEX de is based on the Llama index framework and we are using the drag because um there are not too much public information for the homes we have to find all the information in the disclosure so what we did is we uh frontload uh a few disclosures into the Astra DB um so it has like knowledge a knowledge pull of all the um like information for the homes and when we are doing the search it will SE query like the different homes and find the ones that match our searching criteria and also when we are doing the disclosure summarization um it queries the specific property and then um gave us a summary from the hundreds of pages of disclosure and then I can we send all this as context to uh chat GPT gp4 which is also part of the Lama index framework and then gp4 gives us like the um like the um like natural language response got it and and what part did you use uh llama purse was this the parse of disclosures uh yes exactly and also we notice the using of llama parse is actually quite helpful because there are lots of tables or like unclear uh Parts in the disclosure and I realized like after using Lama index we got a higher quality of the summarization of the homes nice it's good finding um great if there's any questions from the audience uh feel free to jump in as well um and and yeah in terms of like what's next what are the biggest items if you were to you know continue this project what would be the things that you'd be interested in in carrying forward whether it's like a research problem or like a ux uh product ux problem um actually I think the most important part is find the um Market fit because with uh our group is still working on that and we talked with more real estate agents and realized um that might be not their pinpoint because we're trying to solve these problems but we're not sure if they want to pay for that so now we are interviewing with lots of real estate agents to see like what are the most painful part and what are the part they want to pay most money on great well thanks for the great presentation um and to all the projects thank you so much for coming here and presenting during the webinar um I think you know uh I'm sure the audience as well as the YouTube audience is going to really enjoy looking at all these creative applications of ragby on basic trap Bots and so thanks for your time on a Thursday morning uh the video will be up and then if you have any questions uh please leave them in the comments below this for the YouTube listeners all right thank you everyone and have a great rest of the day thank you so much thank you thank you