Reshaping Front End Development with Warehouse.ai

Key Takeaways

[Music] we'd like to thank godaddy our sponsor for our podcasts from no Jas interactive in Austin Texas you may know godaddy as a hosting company but they also have a deep involvement in the know Jas ecosystem you can learn more godaddy.com sgod a TDY com godaddy.com hey it's alex williams the new stack here know Jas interactive and Austin and I'm here at Charlie Robbins of godaddy hey Charlie Alex how you doing great I'm really interested in talking with you we just were having a little chat before you know we went we started recording and I'm really interested in this whole concept of really scale and you know it's an emphasis for us at the new stack and a lot of it comes down to like how do you actually you know how do you actually do it right and how do you decouple you said this very clearly how do you decouple the front end from the back end and you've been working on a project called warehouse day I and it's your now just starting to talk about it you tell us what that is and you know then we can go from there absolutely so warehouse that I I is a front-end operation system that we built at GoDaddy to decouple our front-end deployments from our backend so if you think about your typical nodejs app you are going to have this web server that is also serving all of your static assets so your JavaScript your CSS or HTML is going to come from that same web server that means that any time you want to deploy say a CSS fix that has to also deploy your web server and that process might be much more involved than just say pushing a few files to a CD n so our goal is to make that as easy as possible and as structured as possible based on a developer tool that people are already pretty familiar with which is n p.m. so tell us about what are the drivers that brought you to this point like decoupling the front end to the back end I mean we're talking I mean if you think about the overall kind of workflows that people have there's much quicker delivery right now the deployments are a lot faster right and so this almost like if you're going to have faster deployment you can almost really you're going to need new ways to deploy without having to kind of you know have you know you know with the copy you're not you need to get finding ways to deploy without the computational challenges that you would if you had to do it the old way right I mean there's a lot of upstream dependencies that you might have in your server deploy I mean you might be using Cooper Nettie's you might be using console you might be using some sort of docker swarm thing and that team that's administering that might be kind of different than your front end team you know depending on the scale of your team like the human scale you might end up with a large front end team and a large back end team and oftentimes the front-end devs are dependent on back-end devs two deployments that's not the case at GoDaddy my team actually deploys both but we knew over time that as we grew the front-end team we needed them to own their own deployments and they were already really familiar with NPM and so we're we through the work that we've done and no Jitsu before we were at GoDaddy so building out a deployment system based on NPM specifically for front-end assets made a lot of sense so you your team then you know deploys both right so so tell us then what that allowed you to do by having that but what does it allow you to do you know but the fact that your team does deploy both sure I mean the real motivation for the system at GoDaddy was what my team does so I'm the engineering director for the UX platform team we build out the shared user experience infrastructure for every product team at GoDaddy so we work with I think north of 50 teams and all those teams we want them to use the same version of the asset so they all make integrations with us they all need to say okay at runtime what script tags what CSS link tags should I be including on my page previously that had been sort of a manual configuration process now it's all automated with warehouse so these 50 teams they make a request to us every 15 minutes if we've done a deployment they're going to get their asset right away and at that scale like that makes our lives so much easier because we can just publish something taga this production and it's just out we don't have to worry about any other steps of the deployment process okay so take us through them what actually warehouse actually does it tell us to like do kind of a real explanation of it so it's fronted by a proxy that understands the NPM wire protocol so underneath the covers you know if you've ever run in p.m. with say log level info you'll see it makes it get it makes a put makes a post the publish is actually a put and that put has a JSON payload that contains your actual tarble the thing you're trying to put into the NPM registry we receive that and ship it off to a build service that runs NPM install and runs whatever web builder you might use whether that be web pack or browserify or Babel and then puts that into a CDN something that's s3 compatible and then from there it updates the metadata so that when you say hey warehouse like what should I be including if I want to run in prod it's as though you should use these script tags and these link tags which I've already included how does it know that so when it's actually responsible for building those assets and since it's uploading it to your your s3 compatible place it's going to get a location back from s3 that says oh this is where those URLs are this is where your CDN is so you've got to concede en configured you've put the assets into the thing that's back in your CDN so you know that URL by sort of by definition because you've configured the state of your environment in that way the other nice thing that it does is that let's say that you have to build for different locales at GoDaddy we running over 38 different languages so that means every deployment for us is actually 38 deployments to all different languages and that's all handled automatically by the system okay so why do you guys call it warehouse I just seemed like a good name you know you're you're sort of clearing house for your front-end code so so tell us it now about how are you how you're using this on a day-to-day basis or how you start use it on day-to-day basis so we my team owns among other things the sort of shared control set and also the the shared headers so like if you use any godaddy product you'll see the sort of shared navigation that they all get those teams have to get that from us and when they do that the system that gets that queries warehouses hey well what version of the header is in production what version of the control set is in production and then it just pulls those down ships them back to one of the product teams that ends up in an HTML page that could serve that to a good idea customer so is this going to be available now to really anyone yeah yeah we open sourced it today went out this morning we would love to you know anyone and everyone to run it that sort of our first goal is to make running it sort of a push-button thing so that someone can spin it up and run it say on you know I'll go to Eddie or on Google Cloud or wherever and that's mainly because we're not planning to run this sort of as a publicly available service where it's mainly a piece of software that we put out that other people could run if they want to ok ok so tell us then what are some of the use giving you have your own use case for it what are some other use cases there's guys interesting use cases actually that I thought of that I've never done there's a big movement towards say like AWS lambda right and lambda is JavaScript and if you're going to do anything complicated with lambda you have to run a build you have to actually compile everything sort of like web back into a single file so that today is there's a bunch of deployment tools for that but if you say had warehouse configured to run like your lambda build and then had it shove it into lambda instead of an s3 that could be a really interesting use case for it because lambda would allow what so lambda is server less complaining so it only wants you ship it a file and it runs that file and then it can ship it off to other files so because of that you have this sort of dependency tree of pieces and one of the features of warehouse is that if you depend on a module and that module is published it'll trigger an upstream build so if you have this lambda chain of all these small functions that need to each get compiled if you say compile the base one it could would trigger ploy of all of it so you could see how doing service computing might be a little bit easier that way well my theory is anyway that's interesting because of being the premise behind warehouse is just is to manage the you know is to manage that module really right manage the build process for that module anything that has to take a bunch of JavaScript and turn it into one file and ship that file somewhere that's sort of what we're trying to and then the lambda would allow you then to baby program it really right and then automate that whole process right warehouse could automate your your lambda deployment process for you right and the lambda can then take it in right and then it's just running whatever you wanted it to run like a function right yeah and so that that's pretty interesting you know with you know with this you know you know with this warehouse that I where do you guys want to go with it what are some of the next you know what are your thinking about where you take it next so we've got some interesting features that we want to build out one of the big challenges for the system right now is the like atomic pneus of deploys and so when you actually move something you could promote things through different environments right when you publish something it automatically goes into dev so it doesn't go straight into prod which is one of the nice things and you can say then tag that as prod when you tag it it actually triggers a new build that shouldn't trigger a new build because there are a very tiny chance that something might have changed underneath the covers if you're not using shrink wrap anyway so we want those things to be sort of immutable assets you run a build that is the version that exists forever and then you can promote that thing the automated upstream triggering of builds that I mentioned today actually would put something automatically interpretation for you that works for us we think that we're going to we're going to wait on feedback for it but we have a hypothesis that that's a little bit too scary for most people and so instead of it automatically putting something in production we want it to sort of trigger a canary build like hey I've built this for you because you're downstream modules have changed would you like to promote this sort of like greenkeeper does so I think about security in this whole kind of process and how do youhow what is your context about testing and no and you know making sure that you know that these are modules that you would be you know you that you'd which that you trust right and it when or is that a separate process entirely that you that is predicated on on then using warehouse so there's a really I didn't actually get enough time to talk about this in my session today but one of the things we've built is a check system so because we have the Tarble asset when you publish it actually is your module we unter that thing in memory and you can configure warehouse to run any arbitrary checks so if you want it to say write a check that said hey you have to depend on reacts 15.3 like cool then if your module doesn't depend on rack 15.3 it's actually going to reject that publish sort of like a get post receive hook if you want it to say hey run a check to make sure that all the things in your package.json are safe and all the transient dependencies are safe you could do that there these are things we're not doing today but you could easily configure the system to do so how do you guys do to godaddy how do you think about you know that how do you think about security with you know when you're using when you're using these modules and using javascript and so we run some regression analysis against the no security project we almost read every line of code that we put into production we're a little bit old school like that there's a saying that I like to quote which is that you know by using some open source project you are implicitly saying one day that you are agreeing to maintain it if the author decides that they don't want to maintain it anymore you know most people don't end up there but I've been on that other side of the fence a few times like module author walks away from writing something you're now on the hook to make sure that that thing stays up-to-date so every time you take a dependency you should be looking under the covers seeing what's going on having some sense of confidence if it's a good thing that you should be running or not great so are there new open source projects now that you're going to be you guys are going to be working on I think we're going to keep working on these for a while you know they're they're pretty interesting to us we've got a lot of interesting use cases for them at godaddy and we're working towards having more teams adopt this system internally at godaddy which is pretty exciting excellent Charlie thank you very much for taking that's my pleasure I've seen a great day it was a lot it was a lot of fun thanks absolutely we'd like to thank godaddy our sponsor for our podcasts from no Jas interactive in austin texas you may know godaddy as a hosting company but they also have a deep involvement in the know Jas ecosystem you can learn more at godaddy.com as geo da e dy com godaddy.com