Streamlit for ML #3 - Make Apps Fast with Caching

Key Takeaways

in this video we are going to continue with the app that we've been building um so so far we it's a quick summary we have what you can see over here uh we have this ai q a it's just a you know the very basics of streamlit and we have this uh search bar that we or text input that we've built with streamer over here okay and then we put together sort of the back end of our app so that is initializing a connection to our vector database and we also created all of our context vectors and put those in there and also initializing our retriever model which takes care of encoding the query here and then pinecone over here takes care of finding the most relevant context vectors based on that query vector and then we had a look at how we can [Music] iterate through all the contexts that we return from pinecone and then display them now at the moment it's very ugly and at the same time another thing another really bad thing that we need to solve in this video is this takes forever to do anything right if i if i just um maybe even if i just remove that and press enter i'm not even searching for anything and this is going to take i don't know like a minute i'm going to cut forwards so you don't have to wait as long as i do okay so it's just finished that took way too long so what we want to do or the reason for that is mainly the retriever model uh download over here so every time we rerun or change anything in our app stream the way streaming it works is it re-executes everything in your script and that's really good because it makes developing an app super simple but when you you have something like that you're downloading an ml model you don't want to do redo that every time a tiniest little thing changes in your app you only want to do it once like when the user opens your app the first time then you download it and then you don't download it again that's what you want to happen and we also want the same to happen with our pinecone uh connection okay we just want to initialize that connection once and not every time something changes in the app so we're going to do that uh we're going to figure that out so we can go over to the streamlight dots and we can scroll up to the top or go to this menu and we go to advanced features now and it says advanced but it's not hard to do this okay so we can optimize performance with st catch okay let's have a look at that so we can scroll down it's a caching mechanism that allows you out to stay performant when loading data from the web a lot of data sets or do all performing experience for computations now that sounds pretty much like what we want okay so let's go down and we see basic usage so we have oh this is a good example right so we have this this function here it takes a long time to run every time um and therefore it makes it out pretty slow every time anything changes the whole thing is reloaded so this expensive computation is rerun every time we don't want that to happen what we want to do is okay you can just add this and that means that the output from that expensive computation is just stored okay it's not it's not reloaded every single time okay so let's let's try i'm not saying it's going to work but let's try and do that so we're going to put define init retriever okay and that is just going to return the retrieval model so return that and we do the same for our pinecone stuff here as well so define emit pinecone okay and then obviously we need to actually call those so let's let's do that let's do that here okay so we're going to call those um we want the model is equal to init retriever and we want the index is equal to init pine cone okay let's save that let's have a look at our app okay so it's it's running again let's wait a moment actually stop that because we here we're returning nothing so we do we actually want to return the index that would be that'll be useful um and now we do we have to press this rerun okay now the first time we do this it's going to take a while and first okay we want to make sure is actually is it working like it was before let's see we getting any errors okay now also it always seems to be working fine okay so let's do let's add that st cache that we saw in the documentation let's add that to both of these okay save rerun we get this nice little um spinner running in it retriever it's not very descriptive for our users so later on we'll have a look at making that a little more interesting or descriptive but for now we'll stick with that and this is quite useful because we can see okay you know what are the slow parts of our model to load uh okay so we get this get this error okay why is that so when we are um caching with streamlit what it is doing is well it basically checks if the whatever's being cash changes okay with every rerun uh so it's putting the function or putting some values into your function or rerunning it and having a look at what the the hash code is that comes out of it now in in this case we're calling a an api uh we don't or we cannot actually hash the connection to our pinecone index okay and we shouldn't really do that for our retrieval model either so what we can do is something um which is kind of new from streamer okay so where is st cache is always going to check the hashcode see if anything is changing and there are these new experimental um caches and one of those in particular is this we have experimental memo uh it's fine so we use that to sort expensive computations uh that's fine no no no try that with some things but that's not what we want we want this experimental singleton so basically what that means experiment experimental singleton is whatever you're running should just be run at once and it should not change right we're assuming it's assuming that this will not change right so it's not going to check if it's changed and therefore it is not going to create that um that hash representation of whatever it is you're running so we can write st experimental singleton put it here as well oops let's copy it put it here and okay we've just saved it let's have a look see what happens okay again it's going to take a little while to to rerun everything um hopefully not too long okay there we go so now we have our search um let's say who are the normans okay i'm not going to skip ahead straight away okay so when there's no waiting anymore which is it's really good because before it just took so long so yeah that's that's how we've sort of improved the performance of our streaming app using uh caching and these new experimental caching primitives that uh streamlet have developed so that's incredibly useful and what i want to look at in the in the next video is okay in over into our app yes the performance is there now but it doesn't look so good uh so maybe we can have a look at actually improving uh this look here and to do that we're actually going to not use well we are going to use streamlit but we're going to pull in uh what are called bootstrap card components which are another sort of html css library and using the style from them to display our information it will look a lot nicer than it does now so that's it for this video i hope so useful thank you very much for watching and i'll see you in the next one bye