Llama.cpp for FULL LOCAL Semantic Router

Key Takeaways

today we're going to take a look at the new features in semantic router that allow us to take everything fully local so we'll be using llama CPP for the dynamic routes and the hugging face encoder for our routing decisions now one thing I really like about this is that using a very small model so we're going to be using mistal 7B we add grammars onto that and using this format we seem to be getting much better results for a gentic decision making than I can get with GPT 3.5 now all of this I'm going to be running from my M1 MacBook Pro so it's not like I have anything crazy here and it it will run pretty quick as we'll see so let's jump straight into it now I'm starting off in the semantic router Library coming over to here and I'm going to download that onto my MC once that has been downloaded we should be able to open it and we'll see this and what I'm going to do is just pip install semantic router so I'm just switching across my terminal here if you if you have the local git repo you can install the most recent version like this but I'm going to go ahead and install it from pipy so semantic router and we want 0.0.6 now coming down to here if you are on Mac you want to use this so that's just to speed things up I'm going to use the Myst 7B instruct model it's quantized so you know we can actually run this pretty easily it's not you don't need much to run this and it runs surprising ly quickly while we are waiting for that to download I'm going to come over to here and I'll just point out the pr where we got this implemented so this one from Bugan from aelio super cool and there's one thing in particular I wanted to point out which is that we use these llm grammars here now the LM grammars they are essentially enforcing a particular structured output from your llm which is a big part of why we can get very good performance from a very small model like mol 7B and it is surprisingly good I'm actually seeing better performance with this and M 7B than I am gbt 3.5 which is I think pretty insane now that has been downloaded so the the my model now we'll come down to initial izing our Dynamic route and you might recognize this from the previous example where we demoed a dynamic route I'm using the exact same one here but we're just going to swap out the openingi encoder and the open LM for a hooking face encoder and the mystal 7B LM exact same definitions here so this is our Dynamic route so the get time route and we also have the static routes here as well so they are also in there I'm going to take all of those routes and can drop that time SCH it there and we just put all of our routes in a list here and we're going to use them soon to initialize our route layer but to initialize our rout layer we do need an encoder so we go ahead and we initialize that we're using the Hing face encoder here which by default is the sentence Transformers or Min LM L6 V2 which is a tiny tiny model so you can also run this on pretty much anything as well now we want to come over to here to begin initializing our mystal 7B model there's a little bit of explanation on what we're actually using here we are going to simplify the way that you initialize a llama CPP model but for now this is how you do it and we will still have this option so the idea is we'll probably make it so that if you don't pass in this llm parameter we will use default parameters I when initializing it but for those of you that you know do want to modify your parameters you will be able to so let's run this I'm going to run it on GPU okay so I have that here and then we can initialize our route layer okay so we have our encoder so the H face encoder our routes that we defined before so two static one Dynamic and we have mro 7B okay cool looks good and now let's ask how's the weather today we see that we hit our static route the chat route now let's ask what is the time in New York right now okay and you can see the the grammar coming through here I'm not I'm not actually sure how to stop those from being logged as I'm sure there must be way but we'll figure that out in a future release we have the time and here in the UK it is 1603 so that is correct what is time in Rome right now so I think they're an hour ahead 1703 is correct then I want to try something a little further out the way this so it's I think this is the question where GPT 3.5 actually struggled with quite a lot which is surprising I would kind of expected to be okay with this but it really struggled so what is the time in Bangkok right now I'm going to run this we get 2304 I don't know what the time is in Bangkok right now 2304 so that is correct and then the time in puket as well so I want to you know somewhere that's not a main city because you look at the time zone here and it has Bangkok in the time zone name someone to try okay pck it and then I'm actually not sure why but this this command here or or this question takes way longer to answer than the others and I I yeah I'm not I'm not 100% sure why that is which is kind of interesting but anyway so we're going to be waiting a little moment for this one but I will say that this question that just G gbt 3.5 answering it I I didn't test this exact question but if it couldn't answer bangcock I feel like it would not have been able to answer for bat cool so we come down here we see Asia Bangkok for the time zone and yeah we get the the same time there now let me just double check that they are in the same time I'm pretty sure they are okay yeah cool so it's all good and then we just downloaded the mro model so if you do want to remove that from your computer you can you just run this command the bottom here and yeah that is everything so we have got a fully local semantic router running now it works with LL CPP and it uses llm grammar to just make the performance of small models pretty good as you can see then alongside that we also have the new hugging face encoders which means that any embedding model that is supported by hunging face we most likely support it unless it does some sort of weird pooling mechanism which most of them don't most of them are pretty straightforward so we yeah we can now use semantic router with a ton more models and we can do all that locally which is pretty exciting so that's it for this video I just wanted to show you this very quickly so I will leave it there I hope this has all been interesting and useful but for now thank you very much for watching and I will see you again in the next one bye