GitHub Actions & Machine Learning Workflows with Hamel Husain

Key Takeaways

okay so I'm gonna talk about get this video out of the way okay I'm gonna talk about github actions and machine learning workflows there you okay so a little bit about me I want to repeat too much but I'm currently a machine learning engineering github been there for about two years before this I was at Airbnb on the machine learning team and then before that I was at data robot building machine learning tools it's a software-as-a-service company before that was doing consulting some interesting projects that I've done recently so as Lucas mentioned one is called code search net which is it's a project that we ended up open sourcing and what we're trying to do is learn representations of code using deep learning and one of the one of the downstream tasks that we want to accomplish with that is semantic search of code and actually that's a that's a really cool we partnered with weights and biases so they have a product a really cool product called benchmarks I'll actually show that real fast there we go so there's really cool thing the benchmarks if you haven't seen it I highly recommend it and there's a bunch of different benchmarks like if you go to the benchmarks okay we don't have to look at all the benchmarks but so weights and biases can host sort of benchmark for you and it's sort of is you get kind of a leader board just like Kaggle but then also you get to see all of the runs and the details of the runs in a very transparent way and with all the model artifacts and all the files so it's actually really cool so like the code search net project it's there's a github repo associated with it but it's a it's really interesting in two respects so we open sourced a large corpus of code and it's a parallel corpus of code in natural language so it's two million pairs of functions and methods with their associated dock strings are comments and the idea is can we learn a representation of code that aligns with natural language so that we can use that you know and things like search I won't go in too much detail but definitely check it out it's again it's really cool project that we collaborated with weights and biases and then actually another project that worked out recently that doesn't have much to do with machine learning or deep learning but it's still relevant to this conversation is fast pages so fast pages is an open source project that I'll show you a little bit later that allows you to blog with Jupiter notebooks and it's automated completely with github actions and so and so yeah that's those are some things okay so I'm gonna jump right into it so again up actions so the reason I want like I just want to say the reason I want to talk about this as I think a lot of times you know when we're doing machine learning projects there involves a lot of software engineering like DevOps stuff even or you know other like adjacent stuff and I think it's really interesting to hear kind of what are some tools that can help so so like let me first talk about what get up actions is so on the surface give up actions look like another C ICD tool so you may have seen these on github if you haven't used them probably many people have so things like Travis Circle CI Jenkins whatever it might be and like you know you've seen probably these like checks like when you do unit test or integration test you have some tests on your code and they help you maintain quality of your software so get up actions on the surface looks like it can be a replacement for that and and actually you're get affections is really interesting because has some additional features which are important and that kind of make it easier for data scientists as well to to add additional tools to like their workflow so so the main feature that get up actions ass is allow you to trigger events trigger on any github event and then inspect the metadata of those events that's one of the big features so like what does it give up event so before again of actions you could subscribe to web hooks on get up so there's all kinds of events on github you can see like this really long list highlighted in red some of the ones that may be interesting things like basically anything you do on github can trigger an event so commenting on an issue or making a pull request are starring a repo fork a repo whatever it might be will trigger an event and that event will emit a payload that you can subscribe to so this is an example of a payload when you edit in an issue so basically like JSON and it this has a bunch of metadata about what happened on github so this is just like background and so github actions allows you to read this metadata and like respond to it so someone makes a comment on the issue of certain kind for example you can like on automatically respond to that and very easy way you might be wondering like why is this how does this relate to machine learning at all I'm getting to a promise okay so I'm gonna do I'm going to create a get up action really quickly to like give you a taste of what it is hopefully this doesn't go wrong I haven't really ever done live coding before so we'll just pray that nothing happens okay so I'm just going to create a new directory just so you know that I'm not cheating and I'm just gonna you know I'm gonna make this a git repository and I'm gonna make this into a github repo and on github so you'll see here so this repo I have and I'm so to create a get up action so what I want to do with this demo is I'm gonna create action that responds to you if you make if you open an issue and again I promise is all related like somehow to machine learning here's bear ways this sort of like a kind of boo you in into what actions is so so the way you make get a back ship is you you have a github folder and inside the dot github folder you have a workflows folder like that and so I'm gonna open oh yeah and then so like let's call this you have to have a llamo file so yeah there's a llamo file and the doc github workflows folder that so we'll call it demo mo okay and I'll just open this file my tea so demo demo and so okay so with all llamo files they can be kind of scary what I like to do is I'd like to start with like the ammo file that's already out there somewhere and this use it as a template just because you know it's not like programming it's basically either template so I have this so basically I have this thing in my that looked like it was hand wavy but I kind of copy and paste this thing and start from there you can use any like github action you want I highly recommend just starting with something and so I'll tell you like what's going on here so like you name your workflow something and then this on thing is where you like specify the event in this case like I want to like do something on an issue comment so if I go here and I say okay actions Docs and I want to let's say I want to look at events that can trigger workflows I'm kind of going through this fast but you know don't worry I just want to give you a taste for what it looks like so there's this issues of it and and so like I wanted to trigger this when the issue is opened some I'm kind of I could just copy this thing here I'm gonna come back here and I'm going to just use that now the thing with the amyl is you have to the indentation is very finicky it's very specific so you have to be consistent so I'm gonna say this is issue open event and and so let me make my ID a little bit wider here okay and so this is all just like boilerplate stuff I don't really need that this is the name so this job's thing is sort of you don't really it's not really relevant in this example this is just any name you want the name your job this is the runtime the operating system in which the arbitrary code is gonna run in and so this is just boilerplate I just have it here just to kind of help me but doesn't really matter yeah oh sorry yeah sorry okay oops okay okay hopefully that's bigger okay so this is again this is the yellow file I'll try to go through this quickly so I'm just gonna like delete some stuff here cuz again I'm starting with a template and what I want to do is in this case actually let's leave one thing here I'm gonna leave so there's a lot of syntax that is like unique to get up action it takes some time to learn it like these dogs are very like detailed there's a lot of things you can do and things like that one thing that is really interesting is so I mentioned like every time I get up action is triggered this payload is emitted and you can you can so for example there is a variable called github event and that has the full payload which is that json thing and so this basically so this step I'm running a batch command and I'm you know this variable payload and I'm taking this like special variable that's part of github actions and I'm going to show it just for demonstration purposes and then there's this other thing here so this is one of the key features of actions because you can host actions for other people to use so I'm in so there it is so this thing so I don't know any JavaScript but that's okay like there's there's the people well actually this is a official github action that allows me to create a comment and it uses that you know I barely understand like how what is going on here and that's the point so I can use this step here and that's what I'm doing I'm referencing this prepackaged functionality that someone already used and basically I'm you know I have that here and so next thing so when you so these are gonna run in order and so I'm just gonna commit this and I know I'm going through this really fast I just don't I know that it's kind of probably to actually understand what is going on I just want to pique your interest and hopefully like give you a flavor of how it works so this okay so I wanna this this workflow that I saved is gonna be triggered with an issue so let's open an issue it doesn't really matter what I say and let's see if it triggers action probably 50% chance I did something wrong I don't know hopefully it works so you can click in actions and then it'll go through these steps so let's see we like what it bit real fast so so see payload okay so you can see this payload let me zoom in oops going in the wrong direction no but you can see the payload which is kind of that JSON file that I talked about that has all the metadata what triggered this event which is opening in issues you can see the action is opened and all this information about the issue and then you can parse that and kind of go from there and then you see you know this comment event was triggered which hopefully works so let's go back to the issue and see if it worked and yeah get up actions like did that so this is very hello world again I just want to pique your interest so you might be wondering okay like whatever like housing it uses for ML okay so let me show like a quick demo of kind of what it could look like so let's say you have okay so this is a very common problem like you open a PR and you make some change to a model in this case is very simplified like I'm change the size of embedding and just imagine I'm opening a PR and a lot of times you know in code review you want your colleagues to know whether or not this PR should be merged like what's the impact of these changes to the model and that's always not clear a lot of times from PR PR so it would be cool if somehow you could automate this so kind of using the same approach that I talked about before um just like you can make actions respond to comments and issues you can make actions respond to comments and PRS so in this case I've orchestrated something run full test this run full test will trigger a gate of action to run and what that will do is we'll take the doc refer there's a docker file in this repo and I'll take the code in this PR it will build the docker container so and it will push it to the registry which is what's happening here I sped this up so so yeah you can we can see this without getting bored this takes some time to like you know push this thing and also record this because I didn't want to do this live because everything every time I do something goes wrong internet connection or something else but but what's happening here is pushing the dark container and then it's running this machine learning pipeline on the infrastructure of my choice in this case I'm using Argo which is a cloud native workflow system for machine learning and what's happening here is like we're gonna have a pre-processing step and a training set very simple again this is sort of hello world you know this is an external system this is not running in actions accent just deferred to this system and but you can is like okay it's running and then eventually it's gonna complete so yeah pre-processing then there's going to be like I'm training step on that you might have again like there's logs and you know this is not and you need to see ICD I'm just giving you an idea how this works notice that all the context for what is happening is dropped into the pr automatically so your colleagues can can look and see what's happening so like when the model run is complete what we do is we have a weights and biases action which is really cool and what it does is it fetches all the runs for this Shaw and it brings it back into the PR and it and it says okay these are the metrics that we have and then along with the run ID which is really interesting and so you know we can be confident that we tested this for we you know this test relates to the the like code that's not stale in this PR for the shop and then we also can bring back other runs that you might have and weights and biases for like baselines and things like that this is weights and biases sure a lot of people have seen this on this call but this you know again like you can embed whatever links you want and you can expect the runs you want and you know weights and biases obviously is really cool for showing all the context and I'll like the artifacts they're associated with your model and so in this case we have a candidate a model here we have baseline and so and you can verify like that's the Shah there matches the shop that's that's in the table let me actually fast forward a little bit kind of go through this a little bit faster okay so let's say you want to deploy the model so you might you might say okay like I'm ready to deploy one of these models or test the model in another way or do some kind of you know canary you know smoke test and so in this case I have another chat ups chat command deploy and then the name of the run ID here and then I have this another action goes out and fetches that model artifact from weights and biases and then and then pushes it to a model serving like a model serving function just in this case I'm using Google Cloud functions which is like AWS lambda but basically like I'm using it to you know serve like an imprint a container that does inference and then so again like you have and this just shows the dashboard or Google Cloud functions this is running so this is really cool like you with github actions you have in a PR a very transparent display of sort of everything that happens in a machine learning workflow and it's any kind of like a way that you can test models and that's really that's really interesting I see a lot of people struggling with that you know when they open PRS then going back to a PR and trying to figure out why was this merged or where you know where was the problem and this gives you a way to sort of orchestrate some really interesting things okay so I went really fast again there's no way you can learn everything about get up actions or even necessarily understand how it works in only 20 minutes or so I don't expect that I just wanted to give you an idea and pique your interest in some of this so okay I mean let me talk a little bit about I mentioned that there's a good there's a weights and biases action like what does that mean like how does what is that whereas I even like because I look so there's a lot going on behind the scene so when I had that comment in the PR that fetched the runs from weights and biases it went out and it queried weights and biases API and it got all the relevant runs for that PR and it dropped it in as a markdown table so that involves a lot of work like you know there's a fair amount of code involved with doing that and kind of orchestrating that and you might not want to rewrite all that stuff and you don't have to so the cool thing about actions is I can create a prepackaged action and then serve it to the community so that's what we've done for example weights and biases so like this is how it looks in the work flow if you remember that Yama file that quickly went through at the beginning of blazing fast speed this is kind of the idea so you know like so actually PR comments and issue comments are the same thing because in github issues and PRS are the same thing and it was confusing but just trust me that's so this the issue comment will also be triggered from a PR comment and you know you can have a comment like get runs and then this is this is the interface of the weights and biases act prepackaged action so these are the inputs these are some inputs some of them optional that you can specify that will help you get runs that you're looking for and so I won't go into too much detail but you kind of get an idea so like it wastes biases you have tags they also have certain metrics that you might want to see so this helps you specify what information you want to get back there's obviously a weights and biases API key that you might need and then this like sha might help you filter runs that you know for particular shows that's just a high-level idea again it's not you can follow these links and sort of big and more if you're interested okay there's another thing that is enabled by forget of actions is fast pages so oops maybe due to present and that's fast pages so fast pages so if you ever wondered how you could blog with Jupiter notebooks like turn like wouldn't be nice if you could write a Jupiter notebook and then turn it into a blog post I try to do that like before or fast pages I tried to that myself and it's kind of complicated like it was and you know has some convergence scripts convert its HTML but then there's a lot of fiddling around with like hiding cells and formatting and all kinds of stuff you know infant getting in the way and so again of actions was the way that we so we partnered with fast AI and we created fast pages the way to allow people to quickly and easily write blogs with but Jupiter notebooks but also with markdown and Word documents if you really want to so let me kind of demonstrate just because it's relevant how that works so let me go too fast pages see it's my history so this is how you get started let me zoom out so we can see so this is the fast pages repo the way you get started is you use a template some say pages demo you creates a process Tory from this template okay then okay so what happens now is a get up actions well actually and you can do this if you have an open source much like some kind of project you can kind of emulate what I'm doing here and I'm I show you in a second so um so what's gonna happen is a PR is gonna be opened on this repo automatically because it's detected that I forked it via a template and it's gonna we're gonna do like some of the set up so the idea is like this is gonna become like a github pages site but you have to do a lot of configuration to make it work for me because the name of the repo changed obviously the org change is not fast anymore it's my name and like as to like configure it for like everything I'd need and so pull request is open automatically and that was created this is created with a get up action so you could get up actions like made this PR changing a whole bunch of things here getting rid of all kinds of stuff and changing stuff see that's change from fast to my name blah blah blah all that stuff and changing all kinds of stuff in the repo that you know basically it's boilerplate but you need it to work and then it walks you through the setup steps very easy for demo purposes I can show that real fast see if I can work with very small space here okay so I'm not afraid of getting hacked because I'm just gonna delete this after I'm done so not a big deal so so I'm gonna just fall so these are some instructions here I'm gonna follow it real quick just to kind of let you know so um so the instructions say have to call it that and that as a secret and then it says let me go back to the PR and then it says I need to add the deploy key here so that's like the not secret part so I'm gonna put that here and then I'm call it past pages doesn't really matter what I call it okay and then wanting me to attend a gate that's fine and then I've done everything that wants me to in this PR again this is orchestrated of actions I'm gonna go ahead and merge it and then you should see that so you see that now my site should be getting built and so just a little bit about fast pages is built on this project called env dev which for the jupiter fans out there allows you to build entire libraries with jupiter you could do all your coding in jupiter without leaving an ide i know it sounds ridiculous but it's kind of cool like it's actually you should try it just to get the experience just to see like what it's about if you don't if that sounds preposterous but actually it actually adds a lot of features to jupiter such as allowing you to hide cells and add all kinds of cool metadata and all this stuff so fast pages sort of has all that stuff and so this as you can see it this is like building my site it's almost done me refresh it because I see some screen artifacts happening here and when it's done eventually what happens is you know it will be available on this page we can come back to this in a second I don't wanna waste too much time but um but that gives you an idea let me show you like a working version of fast pages I'm just kind of cut cut out the so this is fast pages is what it looks like you know you have no book blog posts again you know how to write any HTML this just converts your your notebooks as they are I'm not gonna go through this this just shows you all the different features and you know you can have like in your blog post you can have your you know interactive blog interactive charts which is cool and then all kinds of other stuff like gifts like animated gifts and you know whatever tweet cars all the other YouTube videos or whatever all kinds of features it's like go back here at some point it should have worked let's see and yeah so this is my site this is the one I just created that's pages demo you see now I have a get a page that's Lee automated and all you have to do is say you save a jupiter notebook into this notebook notebooks folder and you can create a new page or you can delete once if you don't like the ones that are the demo pages okay so that was kind of a lightning tour of in of actions I hope that was interesting like kind of jog some ideas or maybe something you might be wondering like okay like what do I do with actions or like can I do anything so I think there's love opportunities for people for ml people data science people to like dues to like have interesting actions for the community that's around data science so I'll mention a few one is model serving I think there's some interesting work clothes that you could use maybe to make things they may perhaps serve models via like a getup workflow and then also like you can do some more interesting things with notebooks so you know you could have some things that cleaned up notebooks convert them via the NB dev library that I talked about earlier there's actually this really cool project called paper mill that will run notebooks for you programmatically there's this other site I'll just mention real fast so this is actually a fast pages site Koby 19 dashboards it actually updates all of these dashboards I called them dashboard is a really blog post because that's what fast pages is and updates it programmatically with paper mill so that's really interesting and it does that every six hours or so and it's triggered via a cron so in addition to any github event you can trigger get up actions to just run periodically via a cron event anyways also an other things are additional CI CD integration like the one I should you know I showed one flavor of CI CD in that demo that I show that recorded demo but there's a lot of other sort of ways you can do that and maybe other tools that you can integrate with that and other things so that's the end of my kind of lightning talk about actions and how it might be relevant ml I'll stop sharing my screen though thanks thanks a little do we have questions if you have any questions please either use the Q&A button at the bottom of your screen or you can just ask this in the chat first half the talk so Jack M wants to know so when I set up a few actions on one of my repos check spelling in links be ideal for it to run on pull requests from for Tripos however when a peer is submitted github actions fails to fetch because it's unauthorized because it's for any thoughts on workarounds for this only commentary I can find online online is that github doesn't want to allow this due to potential malicious yeah if you did come in like okay there's a full request from a fork because you could have secrets in your github actions and we don't want to leaked those to anybody that you don't random people then we lock it down and so the way that there are ways that you can get around that so I would have to share my screen and tell you and show you but we don't have enough time one way that I could go around that is I actually make a comment on the PR and that way I'm triggering an event that that that is and I if I'm triggering an event then I can then run something that has all the permissions so like I you know in in the fast pages repo I have a special command called backslash preview and that and that uh that kicks off like it builds the website and it drops a link to preview I don't that make sense but hopefully that helps and then there's another question of chunks you didn't get the ones in q8 right yeah so one from money can we build fast pages and post it via github pages is there an end-to-end process or there is some hand-holding or manual interventions yeah that's what fast pages is you just as soon as you use the you do that template that I showed it just it publishes the fat pages for you automatically it's built for fat pages alright is Bill forget on pages cool do people have any other questions we need to be five seconds all right oh thank you so much that was really really good and just put this recording if you miss any of the details we put the recording on YouTube also imal will be is on twitter hama what's your Twitter ID yeah sorry my twitter is my first name last name concatenated together that's my Twitter handle