Platform engineering: How Google Cloud helps ANZ do modern app development

Key Takeaways

foreign [Music] my name is Nick eberts I am a product manager at Google working on fleets and I'm I'm here with Michael frano Michael you want to introduce yourself yeah hi guys uh my name is Michael fanaro I'm one of the platform Engineers for ANZ which is an Australian Bank yeah that's awesome we also have Tim a major contributor to this talk he's sitting right here he's going to answer all the questions at the end all right so what we're going to go over today we're going to talk about platform engineering what is platform engineering and how it's different from devops maybe I don't know and then also uh ANZ is gonna we're gonna talk about their Journey towards building an independent platform we're going to talk about teams tenancy and how we can help solve that with TKE fleets and then we're going to dive into the evolution of AMC's platform like how they're progressing with the Google order over the next couple of years um do a little bit of a deep dive into fungible clusters another demo and have some q a if we have time sound good that no no doesn't sound good all right let's go all right so platform engineering Michael you've heard this term before Shirley but where do they come from like why are we talking about this now yeah so platform engineering has been around for a few years now I think it's going in a lot more popularity a lot of people out here today probably have heard the term devops is dead and all these types of things and I don't think platform engineering is really there to replace devops at all they actually worked well together but what platform really is at its Crux is it's thinking changing the way you think and you build the platform itself thinking about as a as a tool not just a tool but an actual um yeah so one of the things is that we recognize right is that when we're on this path to devops it was oh hey you know ship the org this way I'm going to co-locate these Engineers with these with the with the software teams and that's the way it's going to work and that works for a lot of people too but what we're seeing is that especially with kubernetes there's this proliferation of clusters you've got a lot of infrastructure to manage and there's a good benefit to centralizing it right especially on the infrastructure bin packing side but also on the human being toils side you can you could build this abstraction and then deliver that to the dev Engineers instead of having to kind of ship devops across your org of hundreds of business units so the thing is platform engineering is kind of like a repurpose of the devops philosophy so devops is great but it's a philosophy it's a practice and platform engineering is just one way to sort of implement that with kind of central controls um so I love this model uh shout out to my co-worker Eddie viaba he came up with this and I stole it so thank you Eddie if you're watching um so this this kind of model helps you break down the problem right on the top you have these domains that you have to ship and and sort of think about building right that's the point of the platform you're going to talk about um how you're going to ship code how you're going to build code how you're going to deal with tenancies security observability and then on the bottom you have like Cloud providers and a pile of tooling on top of it the interesting thing here is that the common interaction between these two layers is get container images or oci images and the Google or Google kubernetes resource model right and so that the power of having those common abstractions really helps you have flexibility and choice over what it is and you're going to build and how you're going to build it um so hopefully this model helps you out when you're reasoning about uh platforms but let's go to the next slide I also want to call out that when you build a platform you have to figure out who your users are um maybe in day one is your prototypical um Dev user I don't know what that means anymore but maybe that's your first user right um the beauty of kubernetes is that as you progress and build that interface and build that abstraction for the dev user you can then say Oh wait hang on I think llms are really cool right now so maybe we should build an abstraction for an AIML data scientist human person um and you could build it on top of the same infrastructure so two key things is one you can serve multiple multiple personas from the same platform by building interfaces that make sense for them and then on top of that don't try to do all of them at once don't build a platform for 10 different personas at once you will not have a good time likely fail pick one get a quick win learn some things and then start to add the users as you go along yeah so before we really get into what we've done in ANZ and like how we've been able to deliver our platform we'll start with like what is what is it that ANZ has tried to do over the last few years so anzx was a division in the bank that had this vision of reimagining how banking would be done and so what this led to was what eventually became a new digital platform or a new digital offering to its customers known as Amazon Plus so this is uh this is the new bank that we've been delivering over the last few years and these are just some of the features that the new digital bank has been able to cater for now historically a lot of these sort of customer Journeys or experiences were very fragmented we definitely had mobile apps and we had web apps but a lot of these experiences still meant that customers would have to go to a branch or go see someone physically and that wasn't what we wanted to achieve with this new digital platform that we're building so why was it worth us investing the platform team to create ourselves an IDP inside of anzx it comes down to this every one of those customer experiences or Journeys translates to tens hundreds or even potentially thousands of different micro services so if we want to be able to scale and provide the best customer experiences and Journeys rapidly at a velocity right that continues to grow and meet that demand then we need to be able to have a way to ensure that Engineers are focusing on the right things and not having to deal with low level complexities and the reality is that this ecosystem that we're building continues to grow and this is something that engineers in all organizations face this uh this ability that where they have to create a platform to be able to ensure that your engineers can move without that friction so how did we do it well first of all I think it's important to look about where we actually came from so we definitely didn't get it right the first time and so about two years ago we started building our platform and we thought we knew what our Engineers needed and so we had regulatory requirements and we had security requirements and that led to our first instance of production which eventually led to us going live with our ANZ plus offering in about 22 and two so since then it's been fantastic we've had a rock solid uh production cluster we haven't really had any massive outages or anything like that however thinking of the platform offering as a product to our internal Engineers the feedback was always consistent from the engineers and that was that it was extremely hard to actually get their software all the way through to production especially in a financial organization it has so many um Gates and things that the processes that they have to go through so we wanted to try and address this friction and really simplify their process so what that led to was us partnering up with Google engineers and that's actually last year that's where I met Nick and I met some other great engineers in Google and we tried to reimagine our platform and treat it like a product rather than just a collection of tools that we throw at engineers and try and get them to figure out those complexities and how to use them and so we're actually still building it a lot of the things that I'm going to be able to demo a little bit later are things that we're building and are in Flight right now so this isn't like a perfect thing we're continuing to evolve and the key part is we're always looking at changing our engineering culture so that we get early adopters into this new platform and that way we get these feedback loops very early to make sure we are solving the right problems as a platform team because we don't want to waste effort if no one's going to adopt it in the end of the day and I think that's a really good gauge if you're tackling the right problems so what did we do well at a high level we've tried to build the platform as an ecosystem of our own internal apis so these are contracts that we have between us and our engineers and so what we're able to do with these apis is we're able to use these to create really good Automation and we can create clear interfaces using API V3 schemas and these interfaces are now declarative things that we can store in Source control and Engineers are able to self-service choosing from an array of uh patterns that we've already pre-approved and have prepared for them out of the box and it takes away a lot of that complexity now the other part of what we've been able to do in this journey now that we've got this all declaratively defined is we're leading heavily into get Ops so in our particular case we're using flux CD to be able to sync all these manifests into our management clusters Within our custom built operators and controllers will then be able to instantiate those things now interesting note um for those who haven't heard of KCC that's an open source Google tool it's called kubernetes config connector and what this allows us to do also is also build our infrastructure and life cycle manager with our kubernetes applications so this has been a huge thing typically we're always we're using terraform and the lifecycle management was completely fragmented um and I'll touch on a few of those points a bit later but using things like KCC where it's a declarative representation of your infrastructure flows perfectly with the skid-offs and creating multiple clusters and fleets of infrastructure which we'll touch on soon and obviously the end goal here is that we want to be able to have our Engineers self-serviceable completely autonomous so that us as platform we're not in this process and we get this unified ability where everything that we're doing follows patterns so rather than having 1200 different Engineers doing their own thing everywhere and it's the same thing we can unify that experience and get the same outcome for those engineers so what does that look like well now that everything's an API and what Nick just talked about with this platform is there's sort of different ways that you can interact with the platform so one of the things that we did very early on is a lot of Engineers are familiar with Git they're familiar with CLI is they love just jumping in their terminal and doing things so this CLI is a wrapper so what we can see is happening here is effectively a engineer can request what we call a workspace and this is a higher level construct which effectively means they're going to get a gcp project they're going to get tenancy they're going to get IEM they're going to get workload identity Federation for keyless authentication so that GitHub workflows they're going to get all these things out of the box and all they give us is a little bit of information and this will spit out all the the declarative resources in order to create all that and bootstrap them so historically onboarding teams would take somewhere between three to six months for them to be onboarded and be able to eventually get their code to production which is just crazy it's a long time it's a very long time so with this approach we've able to speed this up so requesting a workspace onboarding new teams has been reduced down to about 15 minutes so a massive increase or decrease I should say in that time that we can deliver them yeah and in this bottom line what they're doing here is they're actually defining that they have an intent of using a particular Cloud runtime they want to use kubernetes and they want to use cloud run and what this does is this then bootstraps them into a kubernetes cluster in the right environments they're defined and again gives them the patterns that they need now it's not just a CLI um ultimately we did build a UI to drive this a lot of Engineers maybe they like click up so they like to just be able to see that visual representation and that's fine so they still have a UI to be able to interface with this as well and then for those superhero yaml stitches they can also just directly go and create the manifests they don't have to use the CLI at all this is just an easy way to bootstrap the template itself but Nick we've got this this far like tell us like what's Google doing to make a lot of this easier for us as well moving forward yeah thank you so Michael's talked a bit about fleets right fleets are organized organizations of clusters you can you can create gke clusters clusters and other places and add them to fleets and manage them together um but on top of that we started to build Concepts to help you with tenancy right so one of the problems you have right now with kubernetes or with GK in general um if you go in the console actually if I show hands who's using gke today Brad pretty much everything yeah so so when you go on the console and you look at a cluster you either have raid access to the whole cluster right access to the whole time the the the the our back boundaries around the cluster right and if you start building a multi-tenant model in which you're slicing that cluster up across multiple teams you want to be able to give a UI that's restricting those teams only to the namespaces that they belong to in right so we built this concept of a scope a team scope that allows you to create um aggregations of namespaces across multiple clusters bind role-based access controls between those those namespaces that get created in the Clusters and your groups and users in Google Cloud identity um and also keep up with uh adding the Clusters that you want to bind to those fleets or to those Scopes rather so that whenever you bind a cluster to a scope it automatically creates that namespace in the cluster so enough talking about it right uh we're here for live demos um this is now the time that you stop using the internet and and so this is familiar right you're all using ke it may be a little bit less familiar because you could see the Enterprise SKU up here um this we we uh released this into preview today um I want to drill in to the team's flow right so I've got teams here and you can see I've got two tenants created right I've got the alpha tenant and the Bravo tenant now I'm going to go into the alpha tenant oops and you're gonna see that I've got the T I have team one group that's bound to it so this means that there's going to be a role base access control that gets installed in all of the namespaces and all the Clusters that's bound to this team scope I've got some some monitoring um this is going to improve as the product improves over the next uh six months to GA and you can see which clusters I've bound this scope to I have two of them USC's newest West and you can see the namespaces um so you can see that I have demo and new demos so I have two namespaces within the scope that are distributed across all those clusters now if I was good at demos new demo wouldn't be here right now because I'm supposed to show you creating it in lieu of that I'm going to create another demo uh namespace so to create a new namespace I'll add namespaces uh we'll call it new demo two um and update the team scope and there's um terraform for all of this stuff that eventually there will be KCC for all this stuff so you could do this with infrastructure as code it's just easier for me to show you the portal and now I've got a new demo um namespace and if I look at the inspect the Clusters you'll see that this namespace is popping up in those clusters um but that's not enough right is it names like do we stop in namespaces does our job end there no we want to actually put some applications in those namespaces so for now what that means is you're gonna you're gonna go out of our flow and go probably into some git flow somewhere and we've got um a PR all queued up um so I'm gonna merge this pull request and when I merge this pull request it's going to install um an application in that namespace create a HTTP route with for a multi-cluster Gateway which allows me to then access that application through a load balancer that's going across multiple clusters in Google Cloud uh so let the suspense build um as that's adding let me show you the route I think it's interesting so this HTTP route for new demo just also with mentioning that um everything that we're demoing here is like we've open source this repo so we'll send out a link or we'll link up later and yeah if you want to have a look at what we've actually done and probably around and see some of the things then feel free all right so um it's just a where am I at this app just gets some metadata about the Pod and know that it's living on and tells you where it lives so when you hit it if you're if you're getting routed to the region that's closest to you it's gonna it should report back say hey I'm in a node on us West here's my IP um and I've appended a value so one interesting thing here is I'm using multi-cluster Gateway I've got a shared Gateway for this team already stood up I add a new namespace I don't have to add another load balancer or Gateway what I need to do is just add another route within that tenant or that namespace so it makes it easy right I'm just adding a route it's going to adjust the the rules that are on that load balancer and within a couple of minutes I should be able to hit that endpoint um so I'm using a path prefix which is weird because it's kind of a suffix but whatever um and this is the old app it's the that was already there you can see that it's hitting where are you I'm here pod namespace is in demo that's not the namespace we've created so if I add that suffix that's new demo it is not yet uh oh there we go sorry and I'm gonna try this two more times and we'll come back to it later if it doesn't route correctly moving on if that worked it would have went to a pod that's in um it would have went to a pod that's in the new namespace that I just created called new demo we'll come back to that though if we have time and I will it will work it just get Ops takes time right you declare your state and then you wait you declare your state and then what do you do wait yeah ultimately it is eventual consistency so what happens after the apply that's going to go through like a whole lot of people's orchestration between controllers and operators okay it's a lot of moving parts to make all the multi-cluster Ingress actually work so yeah and so that that concludes the demo I will prove it to you that it works by by the time we leave here I promise so um moving on let's so we talked about your platform Journey right like the start of it but you I assume you didn't get it perfect from the get-go you talked about iterating so tell me some of the problems that you came up with in the solutions that you had yeah so early on in our platform journey and even just recently as we're sort of building um the platform that we have there was a number of different things that we were facing as platform operators now these aren't all concerns that necessarily the engineering teams had but they were certainly like they're a bit of a mixed mixed case and this is an infinitive list right there's there's a whole heap of other things that we've been trying to solve as we've been progressing the platform and trying to evolve that product so just touching on these really quickly because I know probably a lot of Engineers at some point have hit one or many or all of these issues at some point in time the first one that we've found really difficult uh prior to what we're trying to do with fleets and managing the life cycle of clusters has always been Disaster Recovery now a lot of people including ourselves we definitely have tools that allow us to restore but if you brick a cluster completely your worst like your best case scenario from that point is probably going to be like just to recreate the cluster unless you're ready to devote hours and hours to debugging that cluster and I'll touch a little bit more on that in the future but in our case when you recreate a cluster that's going to change a lot of dynamic or unique values that are between each cluster so even if you do an NCD Dom or you're using like an open source tool that does Backups it's not a trivial task just to what to copy everything across and hope that that apply just gives you this beautiful thing that's recreated that cluster it doesn't simply work so that's one of the things that we were trying to solve with this the other thing that that then leads to that I've seen is configuration drift like you see this all the time Engineers are playing and like I'm guilty of this as well I'm pretty sure I've left a few tests namespace Fosters and what this is this is actually a pretty bad practice because what we have is if we have 1200 Engineers across the ecosystem all playing and manually creating things and backboarding there's no single source of Truth to recreate any of that and it leads to this like drift where a lot of changes aren't being backboarded into our source code which isn't good either and this is something that like Obviously good Ops definitely helps portray this forward some of the other things that then that then Cascades onto is unreproducible deployments I'm pretty sure everyone's probably had that moment where a deployment worked yesterday nothing's changed but tomorrow it doesn't work and you're now trying to figure out what has changed is it the pipeline maybe someone's changed some of the networking for that thing and it's extremely frustrating not being able to diagnose what had changed in your deployment even though nothing seemed to have changed in your pipeline conflict now that leads to inconsistent environments I probably won't Touch Too Much on that it's pretty self-explanatory but the next thing is like these dangling resources because people are not necessarily following best practices or necessarily look following utop's principles you find that in these traditional push-based models people very rarely clean up after themselves so what you get left with is a lot of kubernetes resources that are just dangling their left and these are just orphaned resources now there's some really good open source tools to solve this but when using things like git Ops and using Argo or flux or config sync this just becomes completely eradicated it's just not a concern anymore and then obviously the last two are kind of like just funny favorites we've all had this where you go to production you hit this manual approval gate now you've been writing code or you're deploying some sort of change and now it goes off to a manager of some sort and he has to approve the pipeline the reality is that person probably doesn't actually know anything about the context of that change and yet he has to be the one to approve it so I find that really funny whereas in githubs inside of different everything's been driven now from your source code and actually what you're doing is you're empowering your engineers these are the people that you trust to manage and do their own life cycle management and deployments which is a bit of a mentality and a cultural shift and obviously um some of the things that you get with just traditional push-based models in this case this is pretty specific to good Ops but you you end up always having some disparity between your non-prod or your not production and your production pipelines now there's always going to be something either it's configuration or it's networking or it's firewall rules proxy like it could be a number of things and so it decreases your confidence when going to production and that's not something we want Engineers to face we don't want them to have low confidence in you or in their own Pipelines so what does it look like after we we're using git Ops we're using KCC and we're using sort of this x ecosystem this is an interface or abstraction that we've built while a day Zero experience looks something like this an engineer could use the UI to create their workspace they could also just use the CLI with Git and in these cases they give that high level intent of what they want to use based on the patterns that we've predefined and we'll automatically once it hits this repository and syncing to our cluster our operators and controllers are now instantiating those things and we're creating their entire workspace of things as I mentioned before day two looks something like this Engineers now they interface with their repository we're not just throwing at them tools unnecessarily and saying here you've got five different uis to debug and troubleshoot your tool they deploy to their repository we have a good Ops workflow and at all of these points through the flow we have feedback loops and in those feedback loops we give them deep links to the relevant tools that they can look at to troubleshoot and debug their application because we are rendering their kubernetes resources for them because we don't want them to have to deal with that low level complexity we know exactly what's the environment the cluster the application the namespace and all these things we can give them predefined search queries in Google Cloud monitoring for example we can give them other deep links to other Insight tools and so rather than trailing through a CD pipeline wondering why is my application not deployed well now it's very intuitive they can actually be directed to where to find their application is failing and why it's failing um and so this is a much better experience for our engineers and behind all of this um obviously uh I'll get to that next actually that's fine so there's a lot of benefits to this right um it kind of feels like a sales pitch me saying all this but ultimately I can tell you that we've seen a lot of benefits and in a highly regulated industry like a bank you know we have a duty to be secure and resilient and to build our applications with stability so by doing this we've increased all those things exponentially and on the other side of the personas as Nick was talking about earlier the engineering experience is increasing has become much better and we're getting a lot of positive feedback and Engineers are enjoying the ability to have that autonomy and that trust um so enough about this we're going to talk about fungible clusters and fleets of clusters so I'm going to let Nick introduce what fungible clusters are it's probably a term you haven't really heard of yeah yeah yeah so funny word fungible right um I don't know why we chose that but it's the word that we're using the idea here is that the Clusters really don't have uh personalities think of it as like the you know 10 years ago VMS became um you didn't want pets right you were getting cattle poor cattle anyway uh the same thing is applied to clusters right you don't really want to have special clusters clusters that are long-lived necessarily you want to have shapes of clusters so what do you do you define a shape of a cluster now on with gke that means you define your Fleet config and you define how you want to use the networking things I talked about earlier and how you want to set up your config syncs or your flux or go see these things such that when I add a cluster to the fleet all I do is pass in a label right that label's bound to something that understands how to apply all the right configuration to that cluster and then you just wait like my failing demo earlier which works by the way I'm going to show you at the end anyway so the idea idea of a fungible is that you're pre-baking doing all the work on Day Zero adding a cluster with intent and that cluster is coming up to State um you can go to the next one and so like at day one cluster comes on board um I'm using multi-cluster Gateway or multi-cluster Ingress so my applications get IPS everything's routed I've got uh antho service mesh or managed istio essentially that's stretching networking across multiple clusters so I have failover between clusters and cross clusters service discovery um and what like what's next you the business comes to you and they're like listen there's been a uh an increase in latency that we really can't bear for our end users right um and the sres have figured out oh hang on this is like we've actually see that this is coming from a section of customers that are not really close to the the Clusters that you have right now uh in fact they're all on the East Coast or something where I live um I am my own user okay and um so you add this cluster to this new region with the correct label right the only thing different is that it's it's getting put in a different region it comes up to State it starts serving endpoints up that get added to the load balancer automatically and within and amount of minutes small in depending on how many applications you have to stand up it's getting hit with traffic and you've all of a sudden um increased not only High availability but reduce latency in the process and it's not too costly because the if you have cluster Auto scaling and horizontal Auto sailing setup the load is going to rebalance itself based on the requests that are coming in so you're not really paying a lot more to get that value right so that's the idea of fungible clusters so and better than hearing me talk about it uh Michael's going to show you a demo in which he walks through this yeah I've recorded but yeah this is pre-recorded now the reason I pre-recorded this is because you don't want to sit there and just wait for a cluster to come online um that could be a bit boring but I'll show you everything leading up to it and like cut a section out with the magic of you know editing and then you can see what happens at the end all right so this is going to be a demo in using git Ops for KCC and creating infrastructure in a way that scales so that you don't have to add that complexity onto your users so what you can see here is I'm actually showing that scenario that Nick just went through we've got a management cluster that's close to zero one and we have this other Regional cluster this is the one that's actually hosting our applications the management cluster here is purely just for your platform operators your customers never have to see that and so I'm just hitting at a test application this is the one that we've already deployed that's the where my application you can see it's in Us East one so obviously we want to spin up traffic that's closer for us Australians so we want something in maybe Us West one so what we can see over here is this is the configuration that I've stored declarative in my repository now there's a bit of a hack script here this is just to create the management cluster so don't be alarmed by this there's a bit of a chicken and egg like Inception thing where you need a cluster to create more clusters so this handles creating that initial cluster and that's just your management cluster the kubernetes folder up here there's three folders there's clusters now that's simply a folder that flux uses to lifecycle managers itself and to register the Clusters then there's namespaces now this is like this is all customized so if you're familiar with customize you'll understand this a little bit but this is just where we're creating namespaces for platform components and where we're deploying the platform tooling we don't really want tenants to necessarily have to see this this is all the stuff behind the scenes and then when we onboard tenants we actually have a tenants our tenants are declaratively defined in this bottom folder and what that does is that that all happens as part of their bootstrap when we onboard a team and the last they want to use kubernetes as their Cloud runtime so we create them a flux tenancy which then allows like TMA not to deploy you to team B's things they have that like isolation because we are using multi-tenant costs like these are multi-tenant clusters um so we've tried to simulate that here as well and so you can see like they've defined their application here in the apparepo this is the typical flux resource they're using Helm it could be customized could be home could be anything else in this case it's a Helm chart so they've defined their home chart and this is what they want to deploy now the interesting thing to note here while we're creating this cluster I'm just going to probably yes I'm actually just going to show you the applications show you how it's deployed so we all know that there is replicas but while we're looking at this the interesting thing to note here is the application team never have to Define all of the cost is in production they just defined that they want their application in production so when we're talking about fleets of clusters that complexity of 1 or 500 they should not have to be contextually aware of that complexity at all so that this application can scale Across The Fleets and they only have to know about their application in one cluster they don't have to think about how many are behind the scenes so in this case um where I'm up to so I'm just showing you the uh multi-cluster Ingress and the multi-cluster service resources now these are all kubernetes resources so we can Define them and store them in our repository and have them also in Ops which is awesome and over here you can see that the multi-cluster Ingress object is recognizing that I have a service that is in this cluster it's creating the NEX for me automatically and setting that up to a global load balancer so this is like handling that split of traffic that Nick was talking about earlier that we want to already have but we want to have this automatically happen when we add clusters we don't want to manually be configuring all this and this is sort of the beauty of multi-cluster Ingress or multi-cluster Gateway if you want to use API Gateway or Gateway API so that's fantastic so when we want to add a second cluster now I've just stitched this together and I've copied it myself realistically you'd probably have some sort of template or some sort of engine or thing that spits this out internally we have our own tooling that would do that but for the sake of the demo I haven't actually ported all that over here so you can imagine this is all pre-generated stuff when you want to add a cluster it's all just cookie cutter stuff it's just it's a cluster without a personality it's just another zero two cluster doesn't have a special name we just know that this is another identical cluster of production except it's going to be activating in a new region for us so this is what we're going to deploy now in this case um when we deploy a new cluster I don't really want to have all these manual steps of like bootstrapping flux and then deploying all these tools and all these things so what else is showing there with that um that alert is that flux is able to automatically trigger a workflow that uses workload again Federation which gives us keyless authentication back to our gcp projects so they can dynamically bootstrap not only flux but everything that flux then Cascades and knows about so it automatically gives all of our platform tooling into this cluster and then subsequently after that's successful it'll automatically deploy all of our tenants and their components and I haven't had to add any other configuration I've literally just said give me another cluster and that all happens which is which is good um so in this case I'm just showing those sort of bindings to make workload identity Federation work with GitHub so that's something really cool that we used we do use GitHub workflows there are scenarios and use cases for that so not everything can just be magically done by git Ops you've got to pick and choose what you need to use when you need to use it but in this case this is the thing that's actually giving us that work a lot of data for iteration and this is a KCC resource so you can do it via terraform or you can have it done via githubs and do it with config connector so where I'm up to right now is I'm just going to merge that pull request um at least I know mine will work because it's recorded whereas next didn't work sorry Nick it works now yeah well so similar to next um it does take time um so you can speed this up with you know creating much more fine-grained or reduced the intervals I wish you're reconciling you can do things like put a web hook receiver in here so it speeds up GitHub will actually push the event and trigger a flux event to sync so you can do things to speed this process up for the sake of the demo we didn't go that extra um so you can see over here I am looking over here and I've actually seen that the new cluster while we're waiting for so we've got cluster zero that's here we're waiting for flux to update to sync and apply that config and then what we should see is a new cluster we should see cluster zero two come here and all I'm doing is sort of just showing you what it looks like if you were looking at it through the terminal um you don't have to do this you can just like walk away and come back with your coffee and things will be done but in this case we can see the cluster zero two came up it did take about eight minutes so there's a bit of a you know magic editing there by now we have a cluster zero two now what happens after the cluster's ready well like I said we still need to put everything onto the cluster it's just a Bare Bones cluster at this point so through that alert that we configured earlier it's now automatically triggering a workflow and this workflow will then do everything that I need to dynamically for me and it's automatically detecting that metadata of that cluster so the things like the cluster name anything unique or dynamic it's able to figure those out based on the alert we provided and it's using that payload to then do what it needs here and so now we're just going to change our code context so we can locally see it in the terminal and what we should be able to see is see all those components now coming up and still we haven't actually done anything now why is this important because if you're handling fleets of clusters tens to hundreds of clusters and you're trying to scale your Fleet of clusters you can't be managing this in the traditional sense where you have to go trigger 500 different pipelines that might be in different teams and different repos it's just not going to work and that's why one of the advantages here for us of using git Ops with our API driven platform has been so advantageous here so what we're seeing here is everything's starting to come up to where my application is starting to sync now it didn't come up immediately I was a bit impatient we are showing this using autopilot so this is um sort of like the hybrid between gke standard and maybe Cloud run sort of that sort of serverless experience so it does take a few seconds for the compute to spin up to our needs but in this case if I as you see I did it again the namespace now exists and things are slowly coming up for us and what I'm checking here is that the multi-cluster Ingress is running um and we're pretty much going to see this all automatically create new negs to the load balancer that was already created and then we'll be able to hit the application and hopefully we'll be able to hit it on the terminal and what we're actually going to try is hit that application now live so I think we're gonna okay I'm gonna skip to you Nick sure we're using this now come on is yours working yeah so there's some extra stuff you can just you can ignore that but this is just good stuff to say if you want to look through the UI there's all this information the gcp will provide you when you want to look and manage your fleets so you can feel free to go through that on your own accord um just something good to say all right ah there we go um so can we can we go to my laptop one more time please sorry we're there okay so this is the end point that he was hitting earlier that was going to us East and now you can see it's going to West um repeatedly I just got to say man I didn't do anything for this demo I really appreciate that you did that and I'm gonna slow clap until everyone defense yeah all right [Applause] quickly because no I will never hear the end of this look see it's working it's going to the new demo hey all right hold on Nick they're all done yeah okay we can go back all right so yeah yes I look there's other benefits to writing um fungible clusters I won't go through all this list but when you think of your cost as as more immutable resources and when you start to life cycle them in that way I would say um in our case we've tried to treat our worst case as our best case scenario so what I mean by that is if you're going to Brick the cluster and you have to recreate it that's what we're going to actually do as many times as possible and get really confident and really good at recreating clusters and so in our case that's what we're striving to do and what that does is that exposes all of the Toil and friction in that process so when it actually does occur your engineers know what needs to be done and before that's even occurred you've already safeguarded yourself and fixed a lot of the tech debt that you might have faced um lots of other reasons why you might think about using clusters in this way and using fleets of clusters feel free to check this out in your own time um I'll hand it over to Nick to give us some final wrap up yeah so we are at we're at the end um it's just some quick closing thoughts one please if you're going to build a platform treat it like a product okay it's not a one-time thing you don't just build it and walk away it's it's got a life um also check out fungible clusters it's going to be a little pain up front maybe a little more than a little paint up front but your life will be easier for it in the future um Fleet tendency is GA today so the fleet team stuff um are back and name spaces all GA today uh Engineers are in the front and then um yeah do you want to talk about it reach out reach out to me or any of my team we're happy to tell you about it and get you like a road map session where we really get the rubber hitting the road foreign