Metaflow Tags: Programmatic Tagging

Key Takeaways

in some sense this example is an inverse of the previous one so in the previous example we ran an execution then we inspected the result using the metaflow card and the ui and then if the result was promising enough we assigned a tag to it in this example we assign the tags automatically so here i just added this one snippet of code here in the existing flow that looks at the accuracy of the model automatically here in the code and if the accuracy is high enough it retrieves the currently executing run using the meta flow client api and then like using the programmatic api for tags it adds a tag called promising model to the currently executing run so the tagging happens automatically in this case and now you might wonder that what's the point of assigning tags instead of using artifacts well the simple reason is that artifacts are immutable they are basically in some sense a kind of a immutable representation of the history which can't be changed history is what it is but you can change the interpretation of the words so for instance although the runs and the accuracies and like whatever it might be they are but they are but you can interpret the results in a different way and add and remove tags based on like post talk analysis so let's see how it works so now like we can execute the code like three candidates run and in this case the the accuracy is 0.84 and now imagine that you are a data scientist working on this project so maybe there are many things that you are iterating like maybe maybe you want to try different type of parameters like here we can try something crazy let's change that and see what happens or maybe maybe it was actually the max features that we want to be changing i think that there's an option for log two and maybe as a sanity check we can also just see what happens if we change the random seed maybe the results are just random the point here is that data sciences is fundamentally an iterative process that the development of of these projects is an iterative process and we keep like trying different things we develop code and it's nice that metaflow keeps track of everything automatically so after a while like we have bunch of these iterations and now what we can do is that let's say we can open a notebook and i have this notebook here that i have prepared for this example where we import like the metaflow client objects and now metaflow allows you to filter executions based on a tag so we can say that okay instead of looking at all the runs of this flow let's only look at the ones that have the tag promising model and now we can see that in this case we have four executions that have the tag here so and they are listed right here now of course like it's kind of hard to say like what's going on here so what we can do is that we can actually look at the artifacts now that are stored inside these runs so we can look at the parameters like we can look at the accuracy values we just um collect all that information in a few lists and like create a data frame out of it so we can actually see what's going on so in this case now we can see all those values all the parameterizations that we were testing uh max depth the features the accuracies and and so forth and obviously you can see that this is in effect experiment tracking what we are doing here and now we can visualize the results using standard pandas tools and uh well now looking at these results looking at the accuracy numbers it's kind of obvious that there's that one one experiment like one execution that actually like doesn't look that promising and now thanks to the fact that tags are immutable in contrast to the artifacts we really can't change the the artifacts and the fact that like let's say the max depth didn't yield good results well it's a fact of life like we shouldn't change that but we can actually change the interpretation that this is a promising model so what we can do is that we can instantiate the corresponding run object eight one um two one seven so that's the run right and now what we can do is that we can actually say that this is not that promising after all so we can remove the tag promising model and now if we go back here we can see that the list is shorter like we have actually removed that one execution that like didn't show good results and now a couple of things worth noticing here first we have the programmatic api that we can use to assign and remove tags and also thanks to the fact that we have both the artifacts and the tags available you can use metaflow quite handily for experiment tracking