I deployed a recommendation model. Testing Models In Production using Interleaving Experiments.

Key Takeaways

deploying machine learning models extremely important topic obviously so over the last few weeks I've been working with a company helping them deploy some machine learning models specifically they have a recommendation model I provide certain recommendations as the output and they want uh the problem that they're trying to solve is not how to deploy that model they know how to do that but how they can test that a new version of the model is actually better than the previous version and this is where I come and this is what I'm I'm helping them to do so today I want to show you the technique that I implemented and hopefully that will give you some ideas for your project it's very sophisticated I think or or at least it's very cool and remember the goal here is how do we know that a new version of a model is better than the existing version and when you talk to people they're going to tell you what you just evaluate the model and compare the performance of the model yes but the problem with a recommendation system with a model like this is that you cannot just explore how good the model is in a vacuum you need user feedback and that is the main component that is going to tell you whether your model works or doesn't work if you provide what look like good recommendations but people don't click on them then your model doesn't work so imagine that what you're trying to do is recommend additional products s based on the purchase history of a user the true test of whether those recommendations are good is whether or not users buy those recommendations they may look amazing on paper but if users don't care then your recommendations are bad that is the main challenge here so let me show you a diagram of the technique that I've been uh working with this company on implementing it's called interleaving expert expent and here the idea just so you follow the diagram is that we have a client that is going to send a request this is a web client that we have and the request is going to be give me recommendation for user ABC and then we have a prediction service this prediction service think of it as the API endpoint where the client application connects to the prediction service when it has only a single model it's going to use that model to just generate five recommendations and send back to the user what we was adding two models now so the prediction service instead of sending the request to the model now it's going to send the same request to two different models I'm identifying those models as Legacy model which is the current version that's deployed some people like to call this the champion model it's the model that's currently running and the candidate model which is this second model here in blue with the dott line the candidate model is the model that we want to test is it better than the Legacy model this is what we want to test right here so the prediction service is going to send 100% requests to both models and it's going to ask both models can you please generate recommendations but before before we had a candidate model all of the recommendations were coming from the Legacy model so the Legacy model generated these three pink recommendations and we will just sending those recommendations back to the client now we're going to be generating recommendations using both both models and interleaving those recommendations in a response so the client will not see recommendations from the Legacy model or the candidate model but it will see recommendations from both models at the same time so we inter leave maybe one recommendation from the Legacy model one from the candidate model one from the Legacy model one from the candidate model then one from the Legacy model to complete the five recommendations and this gives us a couple of good things so number one one we are hedging here and if the candidate model is horrible we are not destroying our application so imagine the candidate model generates good-look recommendations that people don't care about if we just swap the Legacy model by the candidate model well after a month it's just all of our purchases are going to go down the drain because the recommendations are really really bad so we don't want to do that instead we're going to be hedging and monitoring over time how good the candidate recommendations are and we will only switch to the candidate model when we are certain that those recommendations are really good in this particular case here let's say the user sent a request and we send back five different recommendations we can track what the users do with those recommendations are they buying the products that have been recommended by the candidate model or have they not bought anything from those recommendations so obviously it depends on how much traffic your site gets so assuming you get decent traffic you may need to run this for a couple weeks that's our case we run it for two weeks after 2 weeks we collect enough information to aggregate all of the purchases and determine is the candidate model unpair or better than the Legacy model if that is the case then we switch 100% of the traffic goes to the candidate model that becomes the champion at that point and you know we will have to build a new contender later on to just to run the same thing when we have a new version if the candidate model is not working well if people are not clicking on those recommendations then we can just discard that model improve that model come back with a new version later on something else that's also really important is whenever you're presenting a list of recommendations people will tend to favor recommendations at the top like if I give you these are my top five I don't know air conditioner units right people are going to check number one first they're going to tend to favor number one first even though you're going to say in no particular order it doesn't matter number one is going to get favored you need to keep that in mind when you are trying to compare the Legacy model with the candidate model like there are multiple ways to go about this like one particular uh technique is to randomize those recommendations so maybe you trust the candidate Model A little bit more and you are comfortable randomizing recommend the order of recommend ations so that will be one way of doing it the second way will be waiting recommendations based on their position in that list like anything at the top you will know will get more clicks so you will not just favor the model that you're always using to display the top one recommendation because it wouldn't make sense so hopefully that makes sense again this is uh not necessarily fairly sophisticated but it's a very cool way to test anything that requires user feedback it's a good way to test model with production real production data without having to put your entire system at risk by deploying a model that is not good enough you can test this in a back SE little by little you can increase the number of recommendations that come from that candidate model until you build the confidence to deploy that candidate model to make that candidate model your champion model so hopefully this helps and I'll see you later with more tips