"How to 10x chatbot UX? 🤖 🖼️ " - Add Image Responses to GPT knowledge retrieval apps

Key Takeaways

one of the core use case of large language model is knowledge retrieval and there are a lot of tutorials about how can you build chatway's PDF or chatway's website already however one problem I noticed is that almost all of those systems are useful but a bit boring for example even though your team spent weeks and days trading really well written documentations with image gifs and videos but when those q a answer the question it normally just answered in plain text which is still useful but a lot less useful and engaging then content like those ones and in some cases text just simply didn't convince message for example for image like this is very hard to communicate with just text so rich media is definitely a very important part of knowledge retrieval and the reason why those q a apps can't respawn in image is because we didn't feed any image data to large Lounge model let's take website as an example normally there are two types of data a scripting service return it's either text or raw HTML file and most of the time we choose text because it's very clean so their large language model didn't have much noise like the row HTML file were half but the problem is it removed all the links including the reference link as well as image URL and for PDF file it's the same case most of PDF data loader just simply extract text and ignore all the image files so in most case we simply didn't feed any image URL data to the large knowledge model so it can't retrieve any but it's totally possible to extract both text and reference link like image as part of context that's why I want to show you a case study of how can you build a large language model q and a bot that can respond with image reference in solution Explorer is basically turn all the content into clean markdown format if you don't know what markdown is it's a live language for creating formatted content for example you can use hashtag with text to Define title and you can also use certain syntax to insert the image so with markdown format we can still keep structure of the documents in a very clean way and on the other side we can also use markdown format to display the data as well and that's exactly how chat GPT is displayed different type of format like table or code and I will show you example of how can we turn the raw HTML file to clean markdown format that has both text and image URL in structured away so that the large language model can use those data to return rage response back like this so let's get it as always let's open a project folder in Visual Studio code and there will be four steps firstly we scripted row HTML file of the website and then we'll convert the HTML to markdown which should include all the image URL and then we will create a vector index from those markdown data and in the end we will let large knowledge model to build a q a chain and the first thing we're going to do is create an EMV file to store the API key if you don't know what browserlas is it is popular service that people use for scraping website so once you create account on both service you can just put this API key here and next let's go back to the app.py and we're first link to import a list of different libraries that we're going to use and load the environment variables that we put in dot EMV file and if you haven't installed those libraries you can click on this top right corner and do pip install HTML to text land chain llama index open Ai python.emv and build for sub4 once you did it we can go back and implement the first function script the row HTML from the website so we will create a function called Squidward site pass on URL create a header as well as Define the body structure so we will pass on URL element selector where we just want the body and once we did that we'll convert this body request into Json format and send a post request and once we get the response from the API endpoint we will need to do some parsing to extract the raw HTML once we did that that should return the HTML stream so let's try this all right they return the HTML file successfully but as you can see the raw HTML is very messy and has a lot of information that we don't really need and so if we just pass this information to the larger management model it probably won't produce any meaningful results that's why we want to do the next step which is convert the HTML to the markdown so our query this function called convert HTML to markdown and we will use a library called HTML to text which is a library that can automatically convert HTML to markdown format so we will Define the converter and also set ignore links to be false because we do want the links to be kept and then run the converter so this function should be able to convert the messy HTML into Clean Master let's try this okay as you can see the results is a lot better after we do this cleanup and it does kept all the image URL like this one and we can use some website like markdown live preview to test it out whether the outcome works so I can try to paste to the markdown preview and you can see like the image has been successfully loaded which means if we pass has this information to larger knowledge model and ask it to generate answer we should be able to display this image as well however there are some caveats for some website converted markdown doesn't have absolute URL for the image assets so they might have something like this for those situations we will need to create new functions that can help turn those URL into proper ones and the way we're going to do that is I will create one function called get based URL it will try to extract the domain from the URL that we try to script and then we'll try to convert those reality URL in the HTML to the absolute URL that we're running and we'll use a library called Beautiful soup which allow us to filter and choose different html text and modify it so we'll run some for Loop for all the image tags in the HTML file we'll try to get the source URL and if the source URL start with HTTP or https then we will Skip and continue but if it is not then we will try to convert that URL to the absolute URL that we need and we'll repeat the same process for the so Source Link in the image tag as well as data source as some website do use data source tag instead and we will do the same thing for the link as well just in case and then we'll return the update HTML so those functions should be able to help us convert all those image URL to the proper one that we need and in the end I will just create one function to bring all those things together so called get markdown from the URL if you try to use this function and you will see all the URL will be passed public so this is just one example for different websites they do have different type of structures so you might need to make some more adjustment to making sure the markdown format is actually clean and nice alright the next step is to create a vector index so we can do similarity search and in here I want to use a llama index the Llama index is a open source library that provides a lot of different data loaders so for example allow you to load data from airtable Asana and many other things very easily and they also provide a list of very useful features for you to manage Vector index for example allow allow me to modify and add new data into existing index without recreating the whole index from scratch so it will be more cost effective and on the other side if I have a lot of different types of documents it can automatically break down a pretty complicated question into sub questions that query different documents and in the end bring them together into one answer and for our purpose I will use the most basic function of llama index which is create a vector index and retrieve information let's create a function called generate answer with two inputs the user query as well as a vector index we create above so the first thing we will Define a data retriever they can get a list of relevant notes and nodes to some extent is like the relevant documents but it also have things like metadata and other stuff but in this case we only need a text so I will do this to just extract the tax data so this should give us all the relevant information about this user query and then we will give those contacts to large language model like GPT 3.5 to General answer so I'll Define the model and create a prompt template you are a helpful assistant above its own context please answer a question with all the rules below answer the question only based on context provided do not make things up and answer questions in a helpful manner that's straight to a point with clear structure and all relevant information that might help users answer the question and answer should be formatted in markdown and if there are any random image video links they are very important reference data please include them as part of answer and I'm using the new launching expression language here it's basically the same as you created llm chain component but this new expression language just has singular syntax and once it's finished I will return the response so this is pretty much it let's try it out so I will try to use this webflow help doc as an example where it has reference link as well as image and GIF so our Lotus URL give a query how can I create a webflow app and use all the functions that we defined above so let's try this I'll do python app.py alright I get this response back and I can copy paste the response here into a markdown preview so the answer here include both the texts as well as a gif here so the content is much more engaging this is how you can extract clean markdown format with image reference data from website and internal PDF file it's basically the same thing we can convert those PDF into structured markdown format and there are libraries actually doing this called as posts which allow us to convert a PDF file like this into a structured markdown format with image reference as well as extract image file however this library is not free it actually costs around one thousand dollars per year to use if you do want to use it I do have example here about how you can extract clean markdown content with this Library I'm also pretty Keen to explore whether we can create an open source version so if you really want this PDF to markdown Library please comment below let me know so this example of how you can create a knowledge retrieval app that returns not just text but also rich media like this and once you have this markdown you can either create a front-end by yourself or use Library like streamlit to quickly create a UI wrapper I'm really Keen to see what kind of interesting apps that UW if you do enjoy this content please consider giving me subscribe and I see you next time