AI Decision Making — Optimizing Routes

Key Takeaways

today we're going to be taking a look at how we can improve the accuracy of our routes within semantic router so what we've added to the library is the ability to modify your thresholds on a route by Route basis so previously it was a whole layer now you have individual thresholds per route and of course if you have many routes modifying each one of those thresholds one by one it would probably become quite tedious so we've also added in training methods so that you can optimize your route values with a few lines of code which is great so let's jump straight into it I'm going to go over to the semantic router Library I'm going to go to the dots and then I'm going to go down to this threshold optimization notebook and we'll open that in collab now we're going to be using the local models so just the local encoded in this example because we're using static routes only but that does mean that we can also speed up by using a GPU so in kab I'm going to switch across to a T4 GPU that will speed things up although is very fast anyway like literally a couple of seconds but nonetheless you know if you're doing this for many routes and you have a fairly large training set then you might want to turn GPU on okay so that has installed and we can come down to here where we're just defining our route layer we have a few routes in here to give us you know more to optimize with so they're pretty lightweight one thing that I would recommend okay you can optimize using just the method that we have here but really what you want to be doing as well is you want to be adding utterances that you see you know where it doesn't trigger the route you would expect it to you should be adding those to your routes and you should probably be adding just generally more utterances maybe breaking apart routes into different routes if you're seeing that they don't work so well and the other thing as you'll see in a moment is that we have a training data set and we can modify that to improve the performance as well we we'll get around to that in a moment so with our encoder we are going to initialize our encoder and I'm going to use a different one so the default encoded that uses mini LM so it's a pretty old model to be honest but it's very small and efficient so we just have it as the default but at some point that will probably change maybe to this model so this is kind like a more recent still very small but generally you can you should be able to get better performance with this model so yeah we're going to switch across to that so it's E5 base V2 model that we just download quickly okay now once that has downloaded we can come down and initialize our route layer so to initialize a route layer we just need our routes and our encoder both of which we have and there we go so we have our route layer now let's try with these queries so this one should go to politics this one to um chitchat this one to I think we had a biology question and this one should just be nonone okay and you can see actually we get we almost we get three of four okay this one actually goes to chitchat where it shouldn't so okay let's take that and let's try and improve uh what we have so first I'm just going to show you the evaluation or evaluate method so this is a format that we use for both the evaluate and fit methods so you see that we have these x x y y x refers to the utterances that are the input data for our fit method whereas these so these are the the labels like the intended routes that they should trigger okay and I I just like to keep it like this when I'm going through it so it it feels a bit easier than creating two separate lists so I create this test data set which is just a list of the two PS and then I unpack that so we have our utterances here our labels here and then we're going to evaluate to see what we get now if I run this we actually 75% okay so you know um not that great obviously we can we can improve that and you can see with the actually I think it's with mini LM we do actually get perfect accuracy but I think it's it's better to show with this now what we need to do is create a test data set so what we did here but bigger so that's what I do here when you're creating this one thing that you can do obviously is just use an llm to generate this for you so gp4 ask it to generate a set of these so we have politics chitchat mathematics biology and we have these as well so and we should probably add some more of those so these are the routes that shouldn't be classified as anything and we add those in there because we well if we just kind of have named it's always going to choose like the similarity thresholds can just increase or decrease sorry to capture more area and that means that you know it might work on this test data set but maybe not when we have new queries coming in so one thing that I would also recommend doing is okay we have mathematics politics chitchat let's add some more routes here or more test data that is kind of similar to those other ones but we don't actually want it to be classified as those other ones that will basically just make it harder for the model the training function to get a high accuracy and that's a good thing because we're kind of pushing the model more it needs to try a bit harder to get something something good so I'm going to I'm going to write for you very quickly okay so I've added these five here you can see okay kind of similar to biology these two are similar to The like mathematics routes or or tests that we have this one kind of similar to the chitchat one and this one obviously to politics but at least for me they don't quite fit into those so they're very similar just not quite there so that should be good enough I think to get some you know reasonable performance won't be anything incredible I don't think but we should get something now let's try and calculate the accuracy on the using the default thresholds that we have so you can see mini LM that was actually [Music] 34.8516 point6 there now let's come down and let's see what we have for the default routes so you can see they're all just 0.5 okay that that's coming from the hooking phase encoder it's just a default score threshold that we have set there now we have our our train data so the the X and we have our labels which is y so then we just train okay and by default it will go over 500 iterations of training or steps we can we can try that I'm not sure we need 500 but let's see okay so it's pretty quick you see the accuracy increasing over here and we've got it up to 82 okay so nothing special but I tend to find with these smaller open source models it does tend to be a bit lower so let's try that okay and then we can see the updated route thresholds are these so interestingly mathem itics is incredibly low here which would make me think maybe we need to add some non routes to that I'm not sure but that's something that I would probably consider trying but yeah the rest seem reasonable okay so probably around this 75 down to around 60 is where we have that sort of similar to or not similar up to similar threshold hold for this model and as I mentioned that will vary depending on which model you're using so yeah I mean that that is it really uh we can just have a look at the valuation again we get 81 69 okay so that is it for this very quick introduction to the optimization function here you can also try this with other models so for example all 002 which is not quite the the latest open AI embedding model anymore and also coh here I think as well they would both get up to about 92% after training on the same data set maybe that there's some some differences but not a huge number so the model does matter a lot here but also how we're optimizing so we can obviously evaluate and fit but realistically we also should be adding new utterances to our routes and we should also be adding more data to our our test dator as we as we go along we really want to be iterating on this and sort of improving it over time rather than just hoping that it will fit and then that's it so yeah I mean that's it for now I hope this has been useful and interesting so thank you very much for watching and I will see you again in the next one bye now