Google Software Engineer Dan Lorenc on Why Now is the Right Time for the CD Foundation

Key Takeaways

alright hey Alex Williams the new stack here joining me today is Dan Lorentz hey Dan it's correct how's it going Alex Dan's an engineer at Google and he's been working on the tech working on tech Tom but I think more specifically the you know the foundations of the new CD foundation that has been announced today at the open source summit I'm curious what is Tecton in perhaps you can then just kind of provide some context for it sure so Tecton is an open source project we announced today that we're donating it as part of the new continuous delivery foundation but Tecton is about setting up a set of shared best practices specifications an API is for expressing continuous delivery systems on top of kubernetes so the idea is that we want to move the control plan of continuous integration and delivery to kubernetes but still support building artifacts that run anywhere so whether it's you know Internet of Things mobile old VMs physical hardware we want to be orchestrating that stuff from kubernetes but deploying it everywhere so when you're saying that about making it almost like its own control plane is that what you're saying I'm sort of yeah so today we talked to tons of customers that are starting their kind of migration to the cloud migration to kubernetes I mean most of them already have some kind of setup for automating their deployments they're almost all using Jenkins or something like that today where it's running on some box that a developer is set up and is usually left running under their desk somewhere or in a closet and that's something they deploy all their applications today so they don't really want to change that earning that a little bit and so we're gonna take that control plan instead of running it on a you know Mac Mini or little machine under somebody's work desk at work that's going to be moved up to the cloud moved up to kubernetes but we still have to support deploying to all those other legacy targets that people are dealing with so that's kind of where then the kind of the the the cooperation comes in right where you're working with other with other vendors exactly so how does a vendor like cloudy's fit into that great question so the continuous delivery space is huge people deploying all software everywhere have to worry about deployment and deployment automation so we can't solve this alone we can't solve this just for kubernetes so we need to work with vendors from all over like lobbies they have some of the most expertise out there for building these complex deployment setups that deploy to any target all over the world mm-hmm and then what about it like what about like a you know like there's you know the companies like get labs to like like weirdoes get laughing it yeah get labs another great example they have a full suite from source code management all the way through an artifact storage system and then a product that can actually orchestrate deployments to kubernetes today so starting to work with them standardize on these building blocks you know the way people express these pipelines the way these artifacts get stored so that we can build out a more secure or repeatable solutions as people migrate to the cloud more secure more repeatable solutions as people migrate to the cloud and through that you're developing this network of API is essentially that provide what what are some of the api's that are provided sure great question we talked to a ton of customers about how they get their software from you know how does it developer build test and debug locally all the way through the getting it running in production call this like the life of the commit you got all these companies and they all show us what they're doing and they all think that they're kind of special snowflakes here and they've got this crazy setup that nobody else has and through the only ones doing something like this but if you start to stand back a little bit and squint nobody here is really that special the stuff they're doing is kind of similar there's always a few tweaks here and there some of you might store their source code in a slightly different format or somebody might require more operators to review the code before it gets applied to production but kind of stepping back a little bit and settling on you know some common you know herbs and nouns that we can start to use first to describe all these things if you have a common language to explain you know this life of a commit then we can start to build up specifications and descriptions for all of those so if we all describe those kind of software delivery pipelines the same way then we can get portability across all these systems so it's really so the portability question so what does it take to then build those api's what are you doing like tell me about kind of the process you know how are you how are you gonna start organizing that are you are you are you acting as maintainer yourself yeah someone maintainer and technical lead myself and honestly the biggest the building's API is is the easy part the hard part is naming and getting everybody to agree on all of this there's an old joke that naming is one of the hardest problems in software engineering and its really true we did a big evaluation of all the you know cognitive CIS systems out there today and get lab is one of them Jenkins has a few and taking the nouns and verbs they use to describe each part of it just from reading their documentation and it's kind of a mess people use the word you know workflow interchangeably with pipeline and some people use those to mean completely different things and so it's incredibly confusing to users that they try to reason about these systems and it's even more confusing to maintain errs so what are the confusions that come from the users basically using the same word to represent different things in every different system so the semantics that become a real issue exactly so that's the hard part here and that's why we require you know cooperation from all the different vendors and that's why we're excited to do this in the continuous delivery foundation what are the what are the issues that are servicing without this fix because when I first started covering the CNC app I was back and when it was announced and then I remember we're going to a first open source summers they had the first segment technical Oversight Committee and Solomon hikes is in there and you know Alexis Richardson and real interesting group of people and and there came out the continuous delivery came up as a topic of discussion they're like oh no I don't think we need to we don't really need to think about one CD because you know environment because people will all use different ones right so you know that's me just kind of like resonates in my mind is like I'm thinking through kind of what we're where we today and like well why why why do you need to do this and if everyone just is using their on CB yeah so continuous delivery is definitely not a one-size-fits-all solution out there people have all sorts of different tastes and different compliance requirements it's something that a bank uses to push something to their production instance it's hopefully a lot more secure and audited than you know something I started might be using just to deploy a pre-alpha app that nobody's actually using it people want to move faster or slower any other different life cycle there at the the big goal here though is that if we can get kind of these building blocks in place it'll be a lot easier for people to build these opinionated CIC these systems on top of them so that everybody can get put together the one they need it's kind of comes up a lot in the kubernetes space too people mistake kubernetes a lot for a pass a platform as a service because it has a lot of the same kind of building blocks but they're often left disappointed because it leaves a lot still to be put together people describe kirino's more as a kit for building a pass rather than a pass itself and that's kind of where we're hoping to go with tech town as well will you be building them the primitives then is that kind of the idea cuz core Nettie's really that's the origination I'm building out those primitives and that's the really the hardest part exactly we want to build the primitives do the hard boring part so that what is hard about it though what is the hardware CD yeah the hard part is just because how it's not it's not hard it's just complex and these are complex business requirements that uh people have to deal with in CD trying to figure out if you're compliant with all the different auditory regulations trying to figure out if your operational best practices that have been set up by your essary team are actually being met by your delivery pipeline mm-hmm these aren't technical challenges these are organizational legal challenges that you have to make sure that your software delivery pipeline meets today most of this lives is bash scripts somewhere which are very hard to do they're not declarative they're very hard to figure out what they do and they're very hard to understand so elevating that up to be a set of declarative primitives it's a lot easier to understand how they fit together if you're actually following these best practices so that's when the nouns for example come become very instrumental in understanding how do you actually describe this exactly ideally you can take a look at one of these Tecton pipeline definitions and tell that the source-code careful in this repository it was reviewed by at least three developers and that was built using this build tool which was also audited and resulted in this artifact there was no chance for somebody to go rogue and insert something in the middle you know exactly where the source code is coming from and where it's going exactly where it's going okay you shouldn't have to read a very long bash scripts to figure that out so you know when the CNC F first started in kubernetes was being first discussed there was a lot of confusion out there because people I think we're just trying to figure out what do you mean you're building out these primitives first of all so what have you learned I'm like what if you guys learn from that and like what are you thinking of that learning when you're going into building out this CD you know Piper pipeline workflow you know whatever you whatever you guys are however you are describing it yeah we learned a lot of lessons for kubernetes the first one is probably just that you know working code wins when kubernetes was first released it was a working system that people could set up it wasn't the easiest to use it still isn't we have a long way to go but it did solve a few problems for a few people we can't just start in the abstract with white papers and API definition so we have to get something out there working and iterate from there all right right well dan I'm just thinking through kind of your role there what is it you're all going to be in the you know with the with the CD Foundation and kind of the Tecton project overall sure yes I'm gonna stay on as a maintainer of the tech on project and spend most of my time contributing to that and I'm also gonna be serving as a member of the technical Oversight Committee on the CD foundation representing Tecton there who else is on the TOC right now yeah so we have a bootstrap TOC now with one representative from each project No so there's somebody from spinnaker somebody from Jenkins X and somebody from Jenkins okay and that's where the discussions will happen about the nouns and the verbs how do you then how will then you organize the underlying organization for you know for contributing and how will you be leveraging what what's what's come of the kubernetes community development yeah great question so the governance of the CDF is still being set up we have our first bootstrap meeting tomorrow and then our first board meeting will be sometime in a few months and hopefully in the location as nice as this one but yes oh look look forward to reading more of that soon will there be an event we do something around that yeah so the first board meeting is probably going to be at the continuous delivery summit which is a day zero event co-located with coop khan asaf khan barcelona yeah so that is right right okay great well then there's a lot to look forward to you're gonna be busy so we I'm sure we'll be keeping in touch and so thank you so much for taking some time today and you know and enjoy your stay here at Half Moon Bay and you know and good luck with the CD foundation thanks a lot great