Master Kubernetes Operators with Kubebuilder

Hello everyone, my name is Sam Barak and you are watching cube simplified. I hope you already know Kubernetes. No, then go check out my Kubernetes 2025 course that will teach you Kubernetes. Now if you already know Kubernetes, you know that there are concepts and Kubernetes objects like pods, deployments, replica sets, stateful sets, services and so on so forth. But do you know that you can create an object called EC2 instance? No. Well, that's the beauty of Kubernetes because you can extend the current capabilities of Kubernetes and create something which is called an operator. So you can create an operator to control things which are outside of Kubernetes like EC2 instance which we will learn in this particular course. I'm very excited to bring you the Kubernetes operator course from scratch. This 6R plus course is brought to you by Shubhham who has eight plus years of experience and working in Trivago have trained many on open shift holds multiple certifications like GCP cloud professional and DevOps and this course comes as an outcome of his work at Tvago for building custom operators in production. Yes, we'll build a full-fledged working operator end to end from scratch learning why it is even important how to do and everything about cube builder and then building it end to end. I'm really really excited about this course and cannot wait for you to get started. Before that, please make sure to watch this video till the end, subscribe to the channel and share this in your network. Thank you and let's move on and learn about humanities operators from scratch. Okay. So before we can build a custom operator for Kubernetes, we need to know what is an operator, right? And before that there is a term that is called a controller that you really need to be familiar with. Now many of you might not might know already what a controller is. It's you know um you have heard about this which is the cube controller um manager but what does it really do what work is it that the that the controller is is responsible for? So a controller is nothing but it is think of that as a forever running loop right think of this as a piece of software which we will be writing that is a forever running loop and if I want to write a bit of stud code for that it's kind of like this so you always run it and the first thing that it does is it um it observes the state of the resource whichever resource you are writing an operator for you will have a controller for that as well. So if you want to work with pods or deployment you want to work with services you want to work with config map there is a operator for all of those resources. The first thing that it does is it keeps on observing the state of your resource. If the state is updated again for whatever reason you updated the state, maybe in your deployment you change the image, maybe in your config map you edit the data of the config map, whatever reason that happens and if the resource is updated. The second thing that a controller manager or an operator or a controller really does is it compares the current state to the desired state. And this is where you put your business logic. This is actually where you define what to do in case there is a drift that is recognized and most importantly what not to do if there is no drift because it's very important to make your operators or at least your controllers uh ident. They have to be amputent. I cannot stress this enough. We will talk about the reconciled loop just in a minute. But this has to be a deotent in terms of if in case your resource needs no change there should be nothing done on Kubernetes. You should be able to run your controller as many times but it should not result into a change if there was no change needed. And if it finds that there's a drift between the current state and the desired state, it then does an update or you can also say it acts on what logic you have asked it to do what to do in case there was a drift found and then uh we close this. So it's a forever running loop that never stops and it keeps on watching the API server for your resources that you are managing. Now what we are going to build is a cloud uh it's a cloud controller because what we are building will be a piece of software that actually runs on your Kubernetes environment here. Let's say this was your Kubernetes and there you say I want to make kind EC2 instance. Let's put it this way. It goes to Amazon, sees if this instance with this name is already there or not. If it is there, u it does nothing. If it's not there, it creates something. So, it's kind of what we would call a cloud controller. Think about when you run on EKS, when you go to Azure Kubernetes service, it is very easy for you to change the service definition, the SVC for example in in EKS to have a load balancer. You can just say the type of service is load balancer and in your EKS cluster there is a software which is working which is running that abstracts how to create a load balancer how to make your services as you know as backends of that load of that load balancer it hides away the complexity for you and that is what a cloud controller does. There may be many different controllers that cloud providers will give you in their own EC2 or in their own kubernetes distributions to make your life easier so that you do not have to know the the nitty-g gritties of it. You just say I want a resource and then you get one and that is what a cloud controller manager would be. Now when I was talking about uh controller we spoke of this term called ident and this is something I actually want to um and you know um explore a little bit with you. So there are few things that your code should actually be doing when you write a controller when you write u the logic for what to do there are some things that can actually be uh that can actually be done. Um and the first thing is a happy path. So what is a happy path? Um you have a logic your reconcile you know this is actually also called as reconcile loop. It's here this drives the cluster state to your desired state and this is what it reconciles and that is why kubernetes is eventually consistent. And I mean in a way that you make a change eventually which is a very short time again that's why we don't we think it's this but eventually your state is going to match the desired state that you want to do the the cluster state is going to match the desired state. Now let's zoom in in this path a little bit where we have our um you know um case one where you have a logic your resource got updated and your reconcile function is then triggered. This is where you know this is the start of your uh loop. Let's put it this way. This is the beginning of your loop. So the first thing that you do is uh you get your object from the request. The way it works is when you update a resource in Kubernetes and there's a loop that's watching on that, there's a controller that watches on that. The controller gets a request. The controller can actually get the request that you wanted, which is the API request to the API server and it can get the object data. For example, if you updated a config map, your reconciliation loop, your reconciliation loop can actually get the YAML or the JSON of that config map. So you can verify or you can actually you know um what has been changed or what updates has been done, what has been done by the user on that. So you can get the object from that resource and you can then observe the desired state from the spec. What you actually do is you see um you define your uh config map for example or let's say a pod. So you have a pod and then you have a dospec in which you define your containers. So you can see how many or what spec is there for a particular resource and then you can compare uh with that spec what is the actual state of the resource. If they match, you know, if the if the number of containers in your pod are exactly what you wanted, then you have to uh you have to just, you know, skip it. You don't have to do anything. And this is what the happy path is. You do nothing. And this is absolutely important that you realize you don't have to do anything in this case. You don't have to make any API calls. you just ignore that request to your reconciliation loop because the actual state is equal to the desired state and that's what happens when you exit your loop gracefully. Of course, I'm not saying you will stop the loop because you have to keep on listening on the request but you will not make any changes. There's also a second thing that can happen. So uh in this step you have your function triggered. You get the actual object from the request. You see what is the spec of the object. What object is being modified and what is the actual resource of the re of the the actual state of the resource. And this is where it gets interesting. If the desired state is equal to the actual state that you want, you do nothing. We know about this from the previous happy loop. However, if they do not match, for example, in your deployment, the current that you have in HCD is your replica three. Let's take this example. This is nice. So, let's say uh your current uh one that is there is replica equal to three. And this is for a deployment which is stored in HCD. This key is stored in CD. Now you do um a cubectl edit you know you do a deployment and then you give the name of the deployment and then you save that file. The first thing that happens is that your reconciliation loop will get this request that okay because I'm watching deployment this deployment is now updated and that is where you made the change to be replica equal to five what your current is three your desired is five. Now you say okay in your spec you will have replica equal to five. This is what you can do when I say observe the desired state. You get the actual object YAML. You get the actual object YAML and then you observe uh this desired state. So you want five replicas and you observe the actual state which is still replica equal to three. So there is now a drift. the current actually does not match the desired state and this is where your logic would actually come into the picture what to do in case your resources are not matching to what the user has actually asked to do. So there you will calculate some differences. You will probably take some actions. You will do a create, update, delete for the resource. In this case, you will create five more pods. Sorry, you would create two more boards because you wanted five. So 3 + 2 is going to be five pods, which is actual user uh requirement. And then there if your action is succeeded, you update the status field. And then you exit the loop again. This is very important you know uh every every resource in Kubernetes has a dot status. So you have a spec and then you have a status and this is how the reconciliation loop knows if it is actually matching. If for some reason you could not create the pod for whatever reason it may be you can return an error and then you can reue you can retry doing that action. And this is what makes Kubernetes as he appealing. It tries again. It tries again with a you know with a back off. You can configure this that if you were not able to do this right now maybe there was no uh let's say you could not create the pod because you did not have enough memory. Your pod would actually not be created or they actually put in the pending state. This is not a good example. But let's say for whatever reason your pods could not be created. Maybe you were missing the rolebase access control in this name space where the pod should be created. Now it will be recued and the way it goes is it goes back to the beginning of the reconciliation loop here and then it is started again and this is what happens when I say you need to recue. Recue means you retry that action and this is what Kubernetes is about self-healing because if you give the rolebased access control to the you know to the controller it will be able to create resources. It's not like I tried once and I couldn't do it. It keeps on uh you know uh trying again and again. You might have seen this. If you have a pod which needs a persistent volume um it goes into pending if there is no P lab that the pod needs but if you create one the pod automatically gets scheduled it gets started you do not have to do that and this is the beauty of the loop that can reue that you can recue for your um for your cases and this is this is absolutely the brilliance of um self-healing in Kubernetes Now one thing you have to be very careful is this. There's also a sad path and this is something you always always want to avoid when you are writing a custom uh controller. The things are pretty much the same. So what you do is you start your loop. You got a request that somebody updated the deployment. You see what they have made the changes to. You see if there's is actually there or not. The change there is if actually if you go to the desired state you have to absolutely do nothing you have to do nothing absolutely nothing what I mean by that is you do not have to update the resource for anything because when you update the resource let's say here you know um let me talk about that now this is interesting the way it works is I'll go back to the one where you had to make some work, you calculate uh the difference and you update your resource. Now if that action is succeeded, if that action is succeeded, you will actually be triggering because you updated the resource, this will actually trigger the reconcile uh feature again. Kubernetes controllers they do not know what you have updated whether you updated the spec whether you updated the metadata whether you updated the status they don't know about that they just say okay the resource deployment was updated here so I will re retry my actual you know I I would rerun this from the consiliation loop and now because you updated because you created five pods now here replica is actually five and it will now say okay um I get the object the replica is five that the user wanted and now because I'm running this again the replicas have been already created the state matches I don't have to do anything you have to write your reconciliation boobs ident case you know imagine this maybe you uh you got a request and uh your object you get the object you observe them they do not actually make need a diff they don't need any work maybe you have got the same replica was five and then your actual state was also five you do not need anything but by mistake you update the status of last sync you say okay it's just the metadata it does not change my deployment Right? It doesn't change my containers. It doesn't change the image I'm using in the environment variables. It doesn't change that. I'm just putting as a good person, I want to see when this was last synchronized. You say that whenever a request comes, even if I make no changes, um I would update the the status.losync, which actually would then trigger an API call. And you see whenever you update your resource it goes back to the beginning of the reconciliation loop and this is where you would have a forever running loop. Request comes in um okay you got the object you observe the desired state from the spec. I'll zoom in a little bit. uh there was actually no need of any changes on the resource but you by mistake you are updating the status. So Kubernetes says okay the object the controller is looking for has been updated. So it goes back up to the beginning of the loop and then you update the sync uh the last sync again. Kubernetes says I got a new update from the beginning and this loop will continue forever. your resource will keep on updating without having any you know without any stock. So this is very very important that you need to be very careful of um not making any changes if you do not require any changes. Now this is the foundation this is actually the foundation of uh how to write a operator uh how to write an operator the controller is the actual logic that you have to have. Now what uh and there are a couple of things uh when you are writing an operator this is absolutely important I think this is a good thing you should read this the most important question you should be asking or your controller should be asking is if there's anything for me to do that means if the current state is equal to the desired state if not exit immediately do not do anything there should be a golden rule as well that you should follow that You should only write to the API server when the actual state differs from the desired state. Where in this case you see here you you were like okay I know that the actual state is equal to desired state. I make no calls to the API server. I do not update my resource. But by mistake you update the last sync which is again a request to the API server to modify the resource. And then the reconciliation loop sees a there's an update. let me go back and I would restart that uh I I would rerun the loop and then it's a problem. So you always um have to make sure that you only make the changes to the resource when they differ from the desired set. And this is also what a tempotent means that you can run your loop 100 times if the cluster if the machine is already in that state. You should not be doing anything. You know it doesn't break anything. It doesn't change anything if the cluster state is equal to the desired state. That is absolutely important to uh to be to be taken into account. And this is what's interesting. This is what makes these operators uh resilient which is they are stateless. They don't remember what they did with your resource in the last request. They they don't do that. They don't remember. They don't remember if the P replica was three or five or seven. They don't remember if you have the environment variable or not. They always always check the resources. They always their source of truth is they go to the required you know place. Maybe you're writing a a cloud operator. They go to the cloud. Maybe you're writing a database operator which creates a database. It goes to the database always runs the query. And this is why uh these are stateless. So your container your your controller can actually be killed or the you know the node on which it was running it could be deleted it could crash the container uh the controller will go to another node it just starts from there it doesn't have to have a persistent volume to store the state it doesn't know that and this is why it can crash restart and still figure out if it needs to do something uh on a particular resource or not because that's what you have made it to do you have the logic object that it always observes. It always checks the desired state in the current state and if there's anything to be done uh it does it otherwise it says cool the uh resource is already in that u uh in that state which the user wanted me to do. Now if you talk about um uh this is about controllers but what is an operator? I think you guys might already know about operator in in a in a way because you want to write your own operator but let's just go through that quickly. Um, imagine you guys want a house. You know, you you get a house. Let's say you are living in India. And this is a very good example that I like. Let's say you are living in India. You have a house already. Maybe your parents own one. And one day you decide to move to Germany. The place is completely new to you. You have never been to Germany before. You don't speak the language as well. You don't know German. Now, you need a place to stay. you need a house to stick. you call a company uh you know in this case you call a company and the company says hello sir you're moving to Germany we would make we would help you make sure your move is easy and simple we have two options one we can give you a full furnished house we will give you a full furnished house and also we have another option where you can just get a simple uh unfernished I'm saying a simple house but let's say an unfernished house. You can choose whichever you want and we would be happy to give you the key when you land in Germany once you sign the forms and everything. The company also says one thing that sir while we are giving you the furnished house we also give you a helper. Now you say what is this helper? What is it going to help me with? The company says if at any point in time you break a you know a tap, maybe your water filter is broken, maybe the floor is um you know you spill something on the on the carpet. Are you going to fix it? Maybe your bathroom uh tap is broken. Maybe you break um a window. You you don't even know. You don't know anyone in Germany. you will fix it yourself or you can help you can have the helper do these things for you because you don't know the nitty-g gritties of where the hardware store is, how to call someone if I lose my keys for the house. Let the helper do it for you. So the helper is actually someone who has the full knowledge of this house, who has the full knowledge of how to fix things if they goes wrong. You just have to tell the helper, maybe you lost your keys, you know, just tell the helper, go get me a key. He knows where the store is. He has the logic. He has the knowledge of where the store is. He has the knowledge of where to go and in what language, how to speak to the to the person who can make you a key in German and gets you a key. If you have a broken pipe hose, he knows how to fix it. So think of this guy, this helper as the actual operator. Now if you want to port this in um in the terms of software, think about you uh have a database which is called MySQL. Now for you uh installation of things is easy now because you have a container you can simply run it and you would be able to get your app your software. But what about daydo operations? What about maybe you want to do a database migration of your schema, maybe you want to take a backup, maybe you want to take incremental backups on a particular, you know, a schedule. That knowledge needs to be either with you or someone who can do this for you. And this is where MySQL not just gives you the database MySQL but also as an operator for you. This operator is actually a controller running internally. So this controller has all this logic. If the user asks me to uh create a database, I know how to make a database. I know how to log into the DB. I know how to create a database. I know how to do that. It knows about it. So you just have to tell what to do. In this case, this helper was the operator. And in this case, this was the MySQL database product. we were actually looking for and that makes your life a lot easier because you don't have to worry about the lower level details. Now operator has two things. One is a custom resource definition and then the other thing is a custom resource. You know how you can do cubectl get pods? You get a response maybe you have pods or not. It says yes I have pods or it says no pod found in the name space. But if you do cubectl get apple, it doesn't know what this resource called apple is because kubernetes has its own vocabulary. It has the API resources that it has been told to remember and those are the resources the internal ones that are native to Kubernetes like pod deployment secrets uh services you know um all these things these are resources that Kubernetes knows about. But what if you want to create your own resource which is in our case what we will do is going to be an EC2 or uh instance. I could also want to create an S3 bucket. In that case I need to expand Kubernetes's vocabulary that okay this resource called EC2 instance if somebody says uh it gives you a YAML which is kind uh EC2 instance you know how to create or at least you know what that is what to do on that that's a different story you know what that is so that if somebody gives uh on this file cubectl I'll create you can don't just tell me you don't know what is this resource you know about that now I have given you the schema of what an EC2 instance would be I have given you this custom resource definition so whatever the user gives you in this kind accept it because now your vocabulary has been increased and this is going to be a custom resource whenever you create a whenever you instantiate a custom resource definition that is called a custom resource. For example, if you created a custom resource definition uh for EC2 instance, when you create it and then you can do cubectl get uh EC2 instance, what you receive is an instantiation of the of the definition that is a custom resource on which very important on which your operator your controller will be acting upon. on. So your controller knows that on a resource type EC2 instance it has been created. It has been deleted. If you create a resource called EC2 instance, it knows that on this resource there was an update which is to create the resource. The controller will create that resource for you. If you delete that the controller will say okay on this resource which I am watching there is a delete operation performed by the user. So it goes ahead and deletes it for you. So without the controller your your custom resources are nothing. They are just knows about it. It does not react on that. It does not acknowledge that okay I'm going to do what you want me to do because it doesn't have the knowledge. So while the CR and CRD uh you use them to tell what you want the controller with them is actually the how part of it. How do I do that? And this is what we're going to be building. We will be building um a cloud controller which is for building easy to instances on Amazon. And this is what we will be looking for. Um there's also something which you need to know. Kubernetes is not just a platform now. It is a complete operating system for um you know for people. So let's talk about how Kubernetes is actually expandable and how can you use Kubernetes as an SDK. So what's very important with Kubernetes is to look it from not just a platform where you can run your applications but rather how can you expand Kubernetes as a software development kit and what can you do with that on other platforms that's also what you can do. So the first thing that Kubernetes is so widely adopted by cloud providers by onren for other softwares is because of its extensibility. Give me let me get a color different. So it is because of the extensibility because of these custom resources because of these operators and because of the controllers and this is what uh we just talked about. Kubernetes also have API first approach. So everything in Kubernetes has an API. Everything. Your pod is an API. Your service is an API. Your API server has APIs for all of these things. And that makes it very easy to uh write your code for and there are client libraries for this and that makes it very very easy. You have the SDKs that you can build your controllers on uh for Kubernetes. There is Go uh Python, there's Java, there is integration of JavaScript with Kubernetes because there are client libraries for that as well. And you can also uh Kubernetes has backward compatibility because it does not just delete API resources, it deprecates them first. It gives you enough time to move towards a different um uh uh you know to a different API um version and it also versions its API. So maybe you might have seen pods/v1 or you might have seen network um you know um network config /v1 beta one beta 1. So this is the version of cubernetes um API. So it makes it very easy for you to develop new APIs without breaking the existing ones and that makes it really really simple or really helpful not so simple but helpful to expand your APIs and this is a plug-in everything you can have your networking you can bring your own CNI you can choose from different CNIs quite popular ones are stellium um I think yeah is one very popular from uh Isovalent which is a company acquired by Cisco. Uh you also have different options for storage. You have different options for runtimes and web hooks where you can intercept everything as a admission controller which could either validate your request or which can either mutate your request. I think for these web hooks we can have an entirely different course for it. they deserve their own time because I don't do justice if I just talk about there is an admission controller which can validate and uh mutate it doesn't doesn't help so probably something to look for in the future for us and this is why different cloud providers because of this extensibility of kubernetes there are different flavors and thousand plus tools that you can use on top of kubernetes so there is open shift from red hack there is suz from Rancher, Tanzu from WMware. Then there's softwares on top of that which is cubeflow K native um cube which is also quite popular nowadays and that's what makes the developers happy because they say what not how. Now if you are working in a platform engineering team uh you want your let's say you know this is a developer this developer wants a machine in Amazon he wants or she wants an EC2 instance and you manage your cloud let's say you are the cloud uh admin who will give them the EC2 instance they come to you you uh say okay I run some commands blah blah blah and this is the instance and you give them that that's okay. But this is a very old approach. What you can rather let these guys do and this is what um in in internal developer platforms would actually help you with or you can build your own. You can say okay listen what if you want an EC2 instance you don't have to come to me just give me this YAML which is you know you can explain them uh explain it to them. You can have a Helm chart around this that says I want an instance where you can say the number of instances maybe two, the instance uh type which you want and then maybe the you know you can have them give the um the instance uh ID where you can then say the m the AMI ID that you want to use a very simple thing and then maybe also the port number that should be open. They give you this in a YAML format and you pass this from your controller you know after you can have a pull request review. So after they have a pull request, this is stored in GitHub. You have a pull request and then they get an EC2 instance. With this you get to say what they want. They don't care about how to create resources in EC2. They don't care about BPCs. They don't care about anything. And also because you have a GitOps workflow now, you can have Argo CD uh deploying these resources and then the controller takes care of creating the EC2 instance. Everything is as a code. You can have a GitHub very resources um very very simply with this platform uh as as a you know as a as a product which is platform engineering all about. So you can have the declarative options you can use helm to help the lives of developers easy that they can just give you this information. you render the resource and then your controller takes care of that and and this is this is I cannot um stress it enough how how simple it makes our lives easier now because you can run Kubernetes not because the thing is you can run Kubernetes anywhere the reason why you can run Kubernetes anywhere is because of the standardization you can run this in any cloud you can run this on edge you can run this You can run AI workloads on top of that anywhere. You know, Kubernetes is standard because it has one pattern which is a controller pattern that rules them all. Um, I would say DNS just works. Again, it can be problematic, but every pod knows where the where every other pod is. Um, it has its own challenges depending upon how many number of services you have in a cluster, how many pods you have in a cluster. Scalability could be another issue. But for a for a cluster that you have bootstrap, it just works fine. And then you have config management for your developers, which I don't think I need to uh talk about. The single I'm trying to make here is it's not just a container orchestrator. It is a complete operating system. You want networking, it has it. You want memory management, it has that. You want compute management, CPU, uh storage, it has that. It has disk management, it has it. So you can actually build and package and ship your software that runs on top of Kubernetes. Uh any sort of software that you can uh you know you can build and run on top of Kubernetes. It's not like you're just using Kubernetes but you can ex expand it with all of these controllers and these um operator frameworks that we are talking about. And this is why I love Kubernetes a lot. All right. So this was about how do you use Kubernetes as an SDK. Now let's talk about um how do you bootstrap Kubernetes um how do you bootstrap a Kubernetes operator with uh with a software called cube builder and this is where our journey would be beginning. So let's go on and do some hands-on on writing an operator. So before we can build our own Kubernetes operator, we need a place to run this operator on and that is going to be Kubernetes. Now you can build a Kubernetes cluster in GKE. You could probably use Amazon as a managed service. You can build your own clusters with uh QBDM. Whichever way you want to do it is fine because the operator that you are building it will be built into a container image and that container image can be done on any Kubernetes cluster. In our case, we want to keep it simple. So I'm going to be building the operator and I'll be testing this operator which is going to be running on my cluster locally and create instances on Amazon which is external to the cluster just to show that you can manage infrastructure that is external to your Kubernetes environment and this is why Kubernetes is really popular because it lets you uh use it as a SDK as an operating system of the cloud which we will also talk about in the future. So K3D is a Kubernetes distribution by Rancher which has many other distributions like K3S which is also a very simple lightweight Kubernetes distribution. It also has RTE2 which is more hardened in for security if you are working in the governance um and K3D it lets you create containers or rather it lets you create Kubernetes clusters in containers. If you have kind you can use kind. If you have K3D you can use K3D. If you have a sandbox cluster somewhere, you can use that as well. The reason why I'm using this local is because it's very lightweight. It does not cost me lots of resources. It's free of course and it's very fast because it's running on my computer. So for K3D, we can install that very simply. Just go to the installation script and you can download that with either curve or you can download that with wget. I would suggest you go with the latest version. And once you have this downloaded, you can do K3D or K3D version. And I've got the latest version of K3D, which is 5.8.3. And the Kubernetes version that I would be using when I build a cluster with K3D is going to be 1.31.5. But there is a newer version of Kubernetes. What if I want to use that better? We are DevOps engineers. We are cloud engineers. We like to have a single source of truth for all of our applications, which is why we do githops, right? And wouldn't it be nice if you can just version control your clusters as well? Uh that right now I've got one cluster which has two agents. Maybe I want to increase it. Let me put into GitHub. And that is exactly what K3D allows you to do with a very simple cluster config file. And this one has lots of options which you can go to K3D uh and look on the documentation. However, I I've kept it very simple. This one gives me one master. K3D allows you to create multim masteraster multi-node cluster. Again, I'm just going with one because I don't need high availability. And second, I'm going to be using two agents here, which is going to be the worker nodes. And this bit tells me the version of the Kubernetes that I want to use and that's the one which we will be using. You also need Docker because K3D uses Docker because it creates containers in which it runs your Kubernetes cluster which runs containers and that's a whole inception going on out there. But these are the two things that that I would be using. If you have any other distribution of Kubernetes, you can very simply use that. So I've got Docker um running on my machine. I've got OBS stack which is actually giving me Docker in in the background which is giving me a runtime in the background I would say. And to talk about K3D its architecture is fairly fairly simple. So what it does is that this is how it looks like. So you have your laptop or you have your computer on which you want to create multiple Kubernetes clusters. Now as a developer I might need different clusters for different applications. I might want to promote them from dev testing QA just to have a pipeline going for a complete software development life cycle. That's also possible for me too. And that is where K3D shines really well. When you make a cluster in K3D, it creates a separate docker network for all of them. So they are completely isolated from each other and they have their own tider as as you will. They have their own network uh in which they would be talking to. So here you can see I've got one cluster here which is blue and there's one cluster which is green. Cluster A and cluster B. And this is the master node. And these are just robots which is our work is cuz that's where the actual work gets done. And we have these docker networks created. Right now if you do docker network list you see the standard docker networks that are created when you install docker. However when you do k3db cluster create with this config file which is our source of truth. When you do that, there's going to be a new network created which I just showed you. So we will see that just in a moment. Once this is created, when you you know when you ask it to create a cluster, not just it creates your cluster for you, not just it sets up a gateway for you, not just it creates your workers for you, it also updates the cube config or rather it can help you to get the cube config. And here you can see my context is automatically set to cubectl. It says you can use it like cubectl cluster info. And if I do that, that's where my clusters are. That's where my cluster is running. Now if you do docker ps you will see there are a couple of containers that are just started and this is our infrastructure for K3D. We have got two agents which is our worker nodes. We have got one server and we also have this engineext proxy container which is there for some reason and the reason why it is there is for you to talk to your API server because you can use K3D to create multiple masters you need to have a load balancer so you should not be needing to set it that's why K3D does it for you and here it creates a container that is listening on your port on your computer's port which is 5745 five and that's actually uh forwarding the traffic to 6443 of the master or in case you have multiple of the masters and that's why you see the Kubernetes control plane is running on 5745 on all the IP addresses of your computer if you go to this port you will be talking to Kubernetes you will be talking to the cube API server now what can you do every time you have a cluster it's good to do a more testing a very simple one. So we can do cubectl get nodes. There you go. You have got one control plane, one master. You've got two agents which are ready. You can do cubectl get service. There you go. You can do cubectl get pods and some of them are code DNS which is very simple. It comes with a metric server also. It comes with traffic installed which is again uh it allows you to expose your services outside or work as an ingress if you will. Um and it has got a local part provisioner which is for storage. I talked about the metric server already. Now let's try to do some smoke tests. And if you can do cubectl create deployment or kc create deploy it's going to be creating a deployment and it's going to create a pod um k get pods and here you can see it's container creating. If I do k logs and if I can do my deployment, this is a log for engine X. That is fairly fairly simple. If you had used engine X, this should be nothing new. You can also expose your uh deployment. We want to check the network connectivity between our applications. If one service or one pod can talk to other application in the cluster, let's just validate that. So I could do uh I want to expose my service. I want to expose my deployment called my deployment and the port number for that would be 80. Here you can see it's a service resource in Kubernetes and it has got a cluster IP. Now you know if I want one application to talk to another application in my Kubernetes cluster I can use this cluster IP and that's exactly what we would be doing. What we would be doing is here. Oh, okay. Um, so here we have a pod in our new cluster for which we just created a service. I want to test the networking in my K3D cluster. So I would create a new pod. I would try to curl this service and I should get a response from this part and I should be able to curl it because it is HTTP because I know I just ran an enginex server and this should work because it is a single cluster. You know you cannot by default expose your service IP addresses outside the cluster. However, inside it should work fine and that is where we can use our trusty curl image. This lets you just do a curl to any other IP address or host name and that's where we can do k run. I want to use this is my container. I want to create a curl container with the name of curl. This is my image and I want to connect on the IP address of my service. That's look at the pod. This pod is container creating and it's completed already. Crash loop back off. That's fine. Let's check what happened. And if I do logs for curl, it wasn't my crash loop back off. It just started, exited, started, exited, and it's like what is going on? It was not a chron job that runs till completion. Um but you can see here this is the response that you get from the service uh which is engine X and that tells me that my cluster is ready for connection. My cluster is ready for me to build the applications and also uh you can probably go to um you can also check from your cluster if you have external connectivity because we would be talking to Amazon. Might as well check that. So we can do k run curl or let's say Google and I could do httpsw.google.com. Do I have a pod now? Uh, Google container creating and let's see. And that looks Google to me. Um, looks fine, right? So, we have connectivity between our applications and we also have connectivity now uh to external environments and this is going to be the foundation on which we will be building our application. Um you also would be needing to have go on your computer which we talked about. You need docker git the standard developer tools. So um that's it. This will be our uh our setup. Now I think we should talk about what are you going to be really building in this course and what is a reconciliation loop? How does kubernetes know what you want it to do? How does the controller or what is even a controller in the first place? How do they know that I want to do something? The user has asked me to do something and uh I should do that. How do they know that the state of the cluster is not matching the state of the you know desired uh versus current state? How do they know about it? So let's get uh let's let's learn that now. So if you want to know how to build an operator, the best thing to use is an already available framework which is called cube builder. There are also some other frameworks that helps you to build cubernetes operators like operator SDK. However, um cube builder is also one of the very famous operator frameworks that allows you to write your own controllers for kubernetes. This is for people who are using Kubernetes and they want to develop a very indepth uh knowledge of how Kubernetes reacts on certain resources, how the operator loop functions, how is it amputent, what is a finalizer, how does operators would selfheal or if somebody who wants to extend the APIs for Kubernetes, how would you know um you actually compare the state to the desired state. What is a web hook? How does it work? How do you implement versioning with a cubernetes operator that's all which is very very inbuilt and which is very simple with cube builder. So this allows you to have a starting point without spending so much time on what is going to be my project structure. How would I you know uh structure my code? How would I structure my test cases? Um how do I generate my um metrics? How do I add a locking into my soft into my controllers? Am I going to have a leader election? How do I implement a leader election? How do I expose a metrics? on what port do I export the metrics? All of that is taken care by your builder. What it does is is it allows you to have a directory structure in which it has the boilerplate code for building your Kubernetes operators already there thousands of lines uh instead of you to have to write. It allows you to focus on the business logic. It allows you to focus on what is going to be your specification of the custom resources. It allows you to tell what to do in order, you know, how to react in case there is a change in those custom resources. That's what it it allows you to do instead of uh looking at how do I start with an operator in the in the first place. It also lets you generate the role based access control. It lets you generate the cube um um what's it called? It lets you generate the the customize resources as well in case you want to deploy your operator into different places. It also lets you wrap your operator into a Helm chart for its own deployment so that um it can be used in any cluster regardless of whether you are running on cloud, whether you are running on prem on wherever you are running. It allows you to version control your APIs as well. So for us, let's get started with that. And the first thing you can do is you can quickly install um install cube builder. Let's go there and installation and setup or maybe I look on GitHub and there should be some releases um that you can you can download. Um we can also install uh using um the installation book. There are many different ways of installing it. Either you can download it from the releases which uh which one uh is working for you. I'm using a Mac. So I have got an an ARM 64 because I'm using a Mac and that's my architecture. And once it's downloaded, I think you can also use Buu. I'm not sure if you can, but um how can I install that as I show you? There you go. So you can install Cube Builder using a very simple third command. Now first thing that Cube Builder needs or what you do with Cube Builder is you create a project. Now a project, think of that project as a collection of your APIs that you will be building and it's a simple directory structure that allows you to initialize um you know um your your APIs and let's do that now. So first thing we will do I have cube builder cube builder version already uh which is which is available 4.5.1. I think the latest one is 4.7.1. I'm not too far behind but that's okay. So I've got the cube builder and the first thing we will be doing is we will create a project where we will be hosting or we will be you know um building our API. The first thing cube builder uh in it and here is the important thing when you are building your custom operator um let's say you are working in a company called uh example um you want to build your uh custom resources in a certain domain which makes it easy for Kubernetes to know where this operator is coming from. If you do cubectl API resources and if I do less here you can see every resource in Kubernetes is actually its own identifiable API um every resource that we see is an identifiable API c um API resource for example if I uh look at let's say um AI services here for example hub.tra.io io/me1pha 1 we will talk about what the group version kind is but uh just to just for you to know uh you can define the domain in which your API should be uh declared in which your API should be built so for example I could say uh Q builder uh in it I want to be building things related to cloud and let's say I work with um um Netflix for example and my products should be under the domain of netflix.com in this case I'm using cloud.com and the repository in which my um in which my code would be hosted just as a project description what it does is it writes the customized manifests for you so you can have it deployed in different clusters based on your requirements it writes a lot of scaffolding code for you and what it does is is it creates you a directory structure. It writes you a docker file which you can use to build your operator into a uh into a deployable image. It creates you a make file that uh you can use to generate your custom resource definitions. Maybe I open this in VS code. That would make more sense. Um maybe I open this here in cursor. That would make more sense. So it gives you a make file that lets you generate your um your you know uh your RPA lets to generate your custom resources custom resource definitions. It helps you deploy those into a cluster and install them from the cluster. If you are doing a local testing this make file is really really um helpful and this is where is going to be your project. uh this is the project uh information on where uh the what is the name of the project what is the domain under which your project uh is is defined and um and what is the version of uh of of the cube builder project that you are using apart from that and this was the docker file that we were talking about apart from that it gives you this cmd directory now it has already created a lot of files and a lot of folders for you so let's quickly go through that the first TMD main.go is actually the entry point of your operator of your controller. So this already is done for you. You would have to worry about what libraries in Go I want to import in case I want to build a custom operator. Whenever I say operator um when I'm talk I'm talking about controller because that is a loop that actually uh does a job for us. So you would be thinking what library am I um supposed to be uh you know importing. For example, take this the client go and the uh client o package. So this o package is actually the one that allows you to talk to um Kubernetes. It it imports all the Kubernetes client o plugins. In case you were using GCP, Azure, you want to talk to the clusters, it lets you get the cube config. And this is the package that lets you work with. You also have a package for uh importing the Kubernetes API machinery. We will talk about API machinery in a in a bit. Uh this lets you define uh do the runtimes that are needed to define a cubit schema. How do you declare a health endpoint? How do you do logging for your operator? It let you create a lot of codebase. And this is the main.go go which is the main file from which you declare your um your code. This is the entry point for your code. We will talk about that when we um would be writed. You also have a lot of config folders where you define u how are you going to uh be working with your it has some defaults of Kubernetes like your services like your customized files. It has customization that lets you deploy your operator to different uh clusters and name spaces. It has the customization for your manager which lets you create a deployment and the name space in which you want it to be deployed. It's a fairly straightforward customization file. It lets you also create role bases uh access control. It lets you create cluster roles, cluster ro bindings. Um so makes it easier for you to be running your operators. Otherwise if you are managing let's say you write an operator which listens on a resource called um EC2 instance but it doesn't have the permission to uh to to be uh you know um listing EC2 instance in a in a namespace. You will not be able to manage those resources in that name space. So without you worrying about how does my role based access control would look like it lets you create a lot of um boilerplate code along with it lets you create the rolebased access control as well uh for you it also gives you end toend testing so you don't have to write your own testing fees it lets you help uh it helps you with that as well and the one thing that is uh interesting with that which I was looking for is the where did that So where is my cmd config hack? I simply I'm missing Oh yes because yeah so this is just the project resource. This is just the project uh as a boilerplate that cube builder allows you to do. The second thing we can do with cube builder the next thing we can do with cube builder is to actually create an API. And this part is amazing. This is going to be our resource that we this is going to be our custom resource that we will be creating. So what we have just done is what you have just done now is we declared a project called cloud.com. Now with cloud you have many resources to manage. You might have uh things like compute to manage. You might have things like um storage to manage. You might have things like network to manage things in compute could be uh your um let's say EC2 instances you know it could be your AMI in images for example they could be your security groups as well in storage it could be a EBS uh EBS module it could be an S3 bucket that you want to manage uh in network you might want to manage a VPC you want to uh manage is a firewall rule perhaps. So the thing that I'm trying to say is you can create multiple APIs in a single project in a single domain and this is what we are going to be doing. We will be building our own API which is going to be in the compute subdomain and it's going to be our EC2 resource. So that is what cube builder allows us to do uh is to create our own API. So let's do that. I would do cube builder create uh here we go. Cube builder create API. The group is going to be compute. Uh and kind is going to be EC2 instance. I want to create the resource. Yes. So this has created the custom resource and the custom resource definitions for me. Uh it has written them on the disk and yes I want you to create the controller as well. So it downloads um many different go uh go packages. It also creates a directory called API/v1. And this is absolutely uh important. This is the API the version of our API and we are building a file we we are building a resource called EC2 types and that is where we define our EC2 types.code. Um now once we talk about um now once we talk about the uh the EC2 type.go we can take a look at that how does it look like and this is where the actual business logic would go for us. This is where the actual specification of our API would look like. Now before you can build your own Kubernetes cluster um I'm sorry before you can build your own operator for EC2 let's let's see what would this actually look like you know how you going to use the YAML for that so if I give uh EC2 operatory I would probably say um kind is an EC2 operator um meta it would of the metadata. I would give it a name. Name would be u my instance and then uh name space would look like uh default um API version. It's defined in compute.domain.com. Um this is a version one of our EC2 operator API. And then I would have two things. So every resource you have would have a spec or almost all of them uh and then they would have a status field and this is something which is very very important. When you are writing a custom resource you have to define what the resource is going to look like. What is going to be things in the spec of your resource and what is going to be in the status of your resource. And this is what um the the file in API v1 EC2 instances.go helps us to do this. It lets us declare our given um spec for the resource that we are trying. Um, for example, my spec would have um um AMI ID and this is going to be my dummy AMI ID and I would have a key or I would have an SSH key. This is going to be my key pair that I want to use on Amazon. Uh I would have a instance uh let's say I would have a type. So maybe T3 micro I want to have and then you could have a storage and you would have in storage you would then say um I want a standard disk. Maybe you could say I want a a persistence or you could say fast disk which translates to one of the faster block devices in Kubernetes because you want see you all you want to do is you make you're making the developer life easy. you're abstracting the actual details um from the developers. So they can say okay I could go for a standard disk of size maybe 10 gigs and fast would be of size of 50 gigs that is that is the data that I need and this would be one of the minimum things you can use for your cubernetes cluster and with this spec that you're giving every resource has a spec and that is defined for kuberneti it is defined as at a strruct in Golad. So if I uh look at this DC2 operator, I let's say we just keep this simple. We're going to keep these three AMI ID, SSH key, and type. Um this is going to be my things that I want to use and all. Let me just copy there. Um let me just comment this out. Where did that go? There. So I define the spec. Now this is the spec for my uh Kubernetes uh for my operator and I'm going to say my EC2 extend spec will contain an AMI ID. It will contain the SSH key and also the type of the it will contain the type uh of the instance that I want to be using. Now this is where uh it's very important for you to give these JSON tags because when you give a request to Kubernetes about a kind of EC2 instance it needs to marshall your request. It needs to understand what is this key uh and what to do with that is this key is AMI ID, this key is SSH key, this key is type. So these uh JSON tags are absolutely required for serialization so that Kubernetes can know this field relates to a certain um required um key for example. Then you can also have the status for your EC2 instance. Maybe you want to give out uh things like in in this one you might want to give uh the space as probably it's running if your EC2 instance is running or not. Maybe you want to give out things like um public IP and that's going to be a 1.23.4 and this is what you will be putting in the status field. So I would say um if I look in here EC2 operator I want to have phase um I want to have phase which is going to be a string this is the type of string and I want to have uh let's say I want to have the instance ID as well and I can just simply go for a public IP. So these three things are which I want to um be be having. Now this is very important when you are using when you are building resources like this an AI editor would really help you uh like you can see I'm using cursor uh this really helps you to speed up your development again you are the one who's doing the thinking you are the one who is coming up with the spec you are the one who is coming up with um you know what what should you be showing in the status however it helps you as a as a very good helper Now you got the spec, you got the status because these two things are absolutely important to be um to be in a resource. Now how would your overall resource look like? The instance the EC2 instance would have um the type metadata and object metadata. So when you see any cubernetes resource this kind and API version this is actually coming from the type meta. So this meta v1 is actually you can see this is a package in kubernetes. This defines the metadata of any kubernetes resource. This go package defines the metadata of any uh resource and has two type of uh you know it has two strcts there. So the kind and API version that we see on all the Kubernetes resources it is actually defined in a strct in Kubernetes called type meta. And this is what the EC2 instance would look like. It would have some type meta. So you can see here on if I copy this probably this would make more sense. Let me just copy that all the way here. Uh and this would be there you go. So let's comment that out. Now this is a type of EC2 instance which is the kind of a EC2 instance. Sorry about that. There we go. So the first thing this kind has is the API version and the kind. The first thing the resource has is the API version and the kind. And these two things are defined by the type meta. And then we have the metadata of the object itself and that's defined by the object meta which contains the name of the object which contains generated name of the object the name space the UID the resource version the creation timestamp every every object would have these two um you know struts declared inside of that which defines what object it is and second which defines what is the object's metadata and then You have the spec where you have defined this spec and then you have the status which defines the status of the resource and this is how an API is created. This is how you declare what resources are going to be in your API. Now I don't have to tell my developers that guys you need to raise me a ticket so I can create you a resource in Amazon. Oh, you wanted 10 gigs. I probably gave you 15 gigs. Maybe I did not hear that correctly. Let me delete and recreate that or resize it. You do not have to do that. If I just give this to my developer, it is so much easier for them. Maybe I can have them a simple UI that lets them declare the name of the instance, the, you know, the count of the instance, what storage they want. It automatically creates me this manifest. And because I already have a Kubernetes operator and a you know a controller listening on top of that, it is very easy for me to track every request that a developer is making for these uh instances because um they are all they can be put into a version control system. They can be put into GitHub and you can use our code CD that makes developers life so easy. They do not need to know about what is a fast storage. They don't need to worry about what is a standard storage. Of course, they need to know the benchmarking of it but they don't need to know it is a persistent disk. They don't need to know the different type of stoages uh Amazon has to offer. It is offloading on them and that is what it makes it very very simple. Now things that you see here um these ones plus Q builder object root true. So these ones are called cube builder markers and they are there for code generation. They are there for custom resource definition generations for you. For example, this one says this is actually a cubernetes resource. So somebody could say cubectl get EC2 instance. Somebody could say for example here as well where somebody could say cubecdl get instance list and that is going to be uh what is returned this defines it also has a sub resource called status which we are defining here above. So this is what cube builder helps you with and in the end we are registering our EC2 instance and EC2 instance list with the cubernetes schema. this function uh it uses the resources that we just created. It gets the APIs that we just declared and it initi registers that with the Kubernetes schema which is actually this function comes from a file called group version_info.co. Now this one it's a very simple file. It uses the Kubernetes schema runtime package uh from API machinery and the controller runtime. What these packages allow you to do is they let you declare your uh they let you declare your APIs and the kind to Kubernetes and here you are saying that you have a group version. So you're declaring a schema group version. The group is called compute.cloud.com again. So you could say your domain uh domain was actually uh cloud.com and then your group was uh compute uh and then uh your compute.cloud.com cloud.com and then your version is v1 and then your kind is ec2 instance group and this is how every resource in kubernetes think of that as a URL every object on the web has its own unique identifiable um URL for example um think of that as kubernetes every resource is declared in a group it has a version and it has a kind. Every resource does that. Every resource has it. Pod service. If I do that, maybe I could do K uh coopla service. You can see here it kind is called service. Its version is v1. If you do not see the group, that's because it is in the core group of kubernetes, which is uh which is which doesn't have a name, but it's called the core group. So is the same for pod. Uh if you go ahead um here you can see pod is v1. So this is why you now understand when you write kind we are pod API version v1 you are telling kubernetes that this yaml that I'm giving you it is a resource of kind pod which is declared in this group and I want the version v1 for this resource. Every resource have a group version kind and this code is actually adding your declared schema and it is adding your um declared group into Kubernetes. So it's loading your resource y your actual custom resource declaration into Kubernetes. So when you give it a YAML of EC2 instance, it knows what spec this resource has. What is the AMI ID, what is going to be um the phase that is running, what is going to be the uh the public IP that I'm going to be returning. So it knows what is your spec and status. That is what we are doing here. We create a schema builder so that we can add our own schema. And then we have um this this add to schema um it it does add the type in your group version to kubernetes and that's where the magic actually happens. This is where you declare what is going to be in your in in your resources. Um once you have that then you can also uh look into another directory that it has created for you called the internal controller and that is where the reconciliation logic happens. That is where you get the reconciliation logic uh of what to do. So this one is about custom resource but what to do on top of that custom resource what do I do with that that's given in the controller um package in the internal/controller directory and there's a file called your API named controller.go go. What this does is it creates its own package and it then creates your um you know it creates a deconiler. In this deconiler strct it is having two um it imports two uh interfaces. one is the client which gives you the actual cubernetes client that you can use to talk to Kubernetes clusters and then there's a there's a schema that we can then use to convert between the YAML that you are giving and what Kubernetes knows about you know what is declared in Kubernetes um resources then you have some custom markers for for rolebased access control and this is where the actual reconcile dilation loop happens. This is the one uh this was the actual logic that makes sure your cluster state is equal to the desired state. That's the one that makes sure your cluster state would be um it reacts on the cluster state and looks on the desired state and say this is where your logic will go. This is the heart of your controller. This is the heart of your uh of what you are writing what you want to do with that and then you return a result and an error. Now we will talk about um these two things as well. I'm just running you through the code when we write our own as an example then we will uh we will look into this. Once you have the reconciliation logic, it is actually adding um it's adding the controller with the controller manager. So this setup with manager, it uses the controller manager to add your controller too. I think it makes sense if we talk about the architecture a little bit of cube builder and that would be so much helpful. So if I go to architecture, this is the one that will make so much sense. what Cube Builder allows us to do. Oh, wait a minute. Okay, so the when you run, let me go here. When you run, uh, maybe a little bit bigger would help. Let's say here. When you run a Kubernetes um controller, the first thing that it runs is it runs the main.go program. If you remember, this is from the cmd/main.go which is the file here. It starts with the cmd uh main.go file. So the main go file is the one which is responsible when you build your operated into a binary. Here's a main function that's the entry point of of the operator. So let's take a look at its main file from the beginning. It's part of the main package and it does import quite a few of um inbuilt packages from Golan. However, for it to really be working as an operator, there are many more packages that are imported um and that's from the Kubernetes itself. So let's take a look on those packages. The first one that we see here, this is the O package. And this lets your operator uh use the exec entry point plugins or um you know uh talk to your EKS clusters, talk to your GKE uh cluster API server or using the OIDC if in case you're using for authentication. This one's responsible for making sure that your operators can use the cube config or the exec entry points and they can talk to your cubernetes cluster. The runtime package from the API machinery is responsible uh to kind of you know you understand YAML but Kubernetes does not understand YAML. It understands objects which are ghost trucks you know in example. So this one defines schema. This one's define objects that can help you to convert your YAML into Kubernetes understandable constructs. Kubernetes understandable objects. And when you do um cubectl get pods, the YAML that you get is actually converted from the pod object in Kubernetes by using the runtime package. We also have in the API machinery util package and uh this would be looking like it's the same package again but this one's defined in pkg runtime in the API machinery and this one's defined in the util uh as runtime and this one is more like a utility function that helps your operator be stable in case there was a panic which is kind of like a fatal error that your operator got. So instead of completely crashing the process, this lets you log that particular panic and still uh complet still continuing with the with the operator process. So it doesn't just completely trash on to you. We then have uh the client go package which is again uh this is the I think the SDK for go for kubernetes and here we are calling the schema or scheme package and this one lets you register your APIs that you have defined the custom resources. It also lets you define the pod services the core constructs of Kubernetes um with your operator or rather think of it this way that it gives your operator the knowledge of the predefined Kubernetes resources like pod deployment uh secret services and also it lets your operator register the EC2 operator um custom resource that we are creating. We also have the controller runtime package. And this one right here is the secret source which is responsible to have you or to work with a manager that can help you with clients, caches and the leader election. This one is controller runtime is the one that is responsible that gives you the tools to construct the controllers that can listen on changes on your custom resources and then uh you know they can uh handle the caches they can handle the clients to talk to the API server uh and eventually um if in case you want to have a lead election or not uh that also is done by the controller runtime. So if I want to talk about a little bit of the architecture of how this um controller would look like. So we would have the process which is again started from the main.go and this main.go would have a manager. Again you will see this as coming ahead. But here's where a manager is the one that manages two things. one, it has a client and this is used to communicate to the cube API server and it also handles the caches of your requested or um the the the custom resource that was updated. Imagine this, you want to write an operator that reacts on a change uh to the EC2 operator object and that's where the EC2 operator object YAML or the spec will be stored. We're going to talk about the cache much more in the in the future in the video. Not right now. It doesn't make more sense. However, um for me to explain uh the manager, it does have the client which is used to talk to the API server. Then we have the cache. And here's where the interesting thing comes into the picture. This is what we are writing right now. Or rather this green bit. This green bit right here is our user provider logic which is what we are using in the reconcile function. This controller is responsible for reacting on the changes and eventually running the reconiler which is our logic that tells what to do if in case the EC2 operator object was changed or you know um whatever change you made to that this is where it's going to be um this is the logic which is going to be uh running. You can also have in the manager in your controller you can also have a web hook. This is kind of like the similar um validating web hook and mutating web hooks. If in case you want your operator to also uh serve those web hooks, it's possible to do so. Now we also have couple of um we also have couple of packages for the certificate watches. This is the one which is responsible um when you are working with uh let me rather draw it. This will make more sense when you are using things like C or let's say you have um uh admission control admission web hook in your operator. You have a mutating web hook. Here you have a mutating web hook and your Kubernetes API server. You register this web hook with the API server and this can then talk to this mutating web hook. The API server will simply ignore or will not talk to your web hooks. You know, I'm not going to explain the mutating web hooks or validating web hooks because this is not a part of this course. Um, it's something there are very good documentations that you can read about. However, when your API server talks to any of the web hooks, whether it is mutating or whether it is validating, uh it has to have a valid certificate. It does not talk over HTTP. You have to have a valid certificate. And a lot of times you would be using the cert manager to issue your certificates to this uh service your your controller that is hosting the mutating web hook. Now if in case the search manager uh again it's used to uh issue certificates for your web hook and every 90 days I think by default it will be rotating your certificates and in this case if your certificate has changed maybe you are storing that certificate into a secret then it is given into the pod. Um however if this certificate is changed you will need to restart your controller. You will need to restart your controller pod. So eventually the new certificate is loaded and the next HTTP request uses the new certificate which is renewed by the search manager. This offers a downtime and to fix this we have theert watcher um package. This one creates a watcher for the change certificates and it reloads them on the fly without you to have to restart your controller package. So you don't have any downtime uh in case you are updating your certificates in case you updated or search manager did an update for your certificates. We also have the health package what lets you uh expose the the you know the livveness probes and the readiness probes that you can use for your operator. This exposes the health and the readiness endpoint probes which you can use in your deployment when you are deploying this operator and you can say uh check at this endpoint every now and then. Uh it's a similar uh it's a standard Kubernetes livess and readiness probe. We also have the zap package which is mostly used for logging. We then have filters package in the metrics package here. And this one let's uh I think this makes sense for me to first talk about the metrics here and then we talk about the filters. See when you are writing your operator with cube builder it doesn't just let you focus on the reconciler. I mean this is what your business logic is. That's what you are uh supposed to be writing. However, with cube builder, your operator which is running in a pod, it by default exposes an endpoint called matrix. And this might be looking familiar to you. Um because this is something which we use a lot in Prometheus. When you are writing a Prometheus service monitor or when you are writing a scrape config, you give three things to the Prometheus server. the IP or the service name, you give the port number of the scrape config and then you also define uh the you know the path the scraping path. This same you can use uh with your operator cube builder when you are building an operator. Cube builder exposes the metrics endpoint and this it it exposes couple of Prometheus readable metrics like what is the success rate of your operator? How many times the reconciler has executed? How many times it event it resulted into an error? How many times it resulted into a success. So it's not uh it doesn't give you an idea of how many EC2 instances have you created but rather this is more on the metrics of the operator itself and then if in case you want to maybe you you have a requirement that my operator can create EC2 instances but I also want to know how many it has created successfully. So you know you can also expose your metrics you can instrument your code with Prometheus uh go packages and as soon as you were able to create a VM you know uh on on Amazon we'll look into the code in the future uh in the in the further parts of the video uh you can then increment your uh AWS instance count uh to one because you were able to create just one more um instance and then you can expose this to the metric endpoint. The thing that I'm trying to explain here is it's already done for you by cube builder and by default there is no username or password. It is open to everyone and then you can use Prometheus with a scrape config to scrape this metrics the operator related metrics uh into Prometheus and show that onto Grafana. However um you can also then use this filters uh package. This lets you define some sort of authentication that this metrics endpoint is not publicly it it should not be publicly accessible. I only want to um I I only want to allow someone who has this username and password. Uh I want to have some sort of authentication on this matrix endpoints and these are the this filters package provides us these functions where we can use um these authentication gate um gated authentications for our metrics. We then have the web hook. Again, this is the package which is responsible for you to create these validating web hooks, mutating web hooks. There are many many videos available. Uh we also did a live stream on cube simplify of creating your own validating web hook. You can definitely take a look at that. I'll put the link of that in the description. And uh this one helps you declare your validating and mutating web hooks. These are core heart of your operator. You know without these packages it without cube builder using these packages it would be very very difficult for you to build an operator. So cube builder is really good in terms of scaffolding your project. When I say scaffolding it means it is it gives you a very good blueprint. It gives you a lot of boilerplate code which again you can uh refactor but to begin with you only focus on your reconciler logic and that for me it's amazing. Now here's where the repository where my code is going to be in the API v1 and this is where I am uh calling my custom resource definition which I declared. You remember we had API then v1 and then we had the EC2 instance right here. This was our spec of the EC2 instance. That's what we are calling in uh in the the main.go. So I am calling my um my v1 with the name of compute v1 and then I'm also calling my actual controller logic which has the reconiler or this is where my reconiler logic will be in the future. Now coming forward we have a couple of variables. This setup log is fairly simple. This sets up a logger for our um you know for our controller. And the scheme that you see here, think of this as a phone book. This is an instantiation of the new scheme function. The scheme is acting as a phone book. It is acting as a registry where you will write all of your objects that you want Kubernetes to know about or rather your operator to know about. And that's what we do here in the function in it. We use the util runtime which is available here. this runtime package and here's where we have a must function. So what this does is in case this must function returned an error um in case this must function you know was not able to register if there was a panic the program will stop right here because your operator is completely useless um your operator is completely useless if it doesn't know about the core uh API types like pod deployment or rather also your own EC2 instance. So we register the default um core um we register the default API types and you can look at that using um cubectl. Let me increase the font a little bit. We can do cube ctl uh API resources here. You see? So think of the phone book which is our scheme. We are adding all of this um to our phone book. So we are telling our operator this is what we have available. uh all of this is what we have available in our uh AP in our Kubernetes cluster and then we also add our own default uh our own custom resources which is what we are calling from the API B1. So essentially we are telling Kubernetes that the scheme that we declared over here is an empty book and in that empty book using the add to scheme function which is here given uh to us by the client go scheme we add the built-in types so our operator knows what built-in types are available into Kubernetes and also we add our custom type which is the EC2 instance and then our uh registry or the scheme is a complete catalog And that's what our operator will be able to use. Now, now here's the main function. This is where everything starts for any go program. And we are defining a couple of variables. For example, I want to define the metric address um on which IP of my port the metric would be listening to. And once we have defined these variables, we also define some flags on the command line. Uh when you are running your your you know when when you build this with go build and when you run this binary, you can give these uh command lines as metrics address, probe address, you can define leader election and all that. So we define the IP address on which our metrics should be sobbed. We declare some variables which is the path for our metric certificates because um just like web hooks can be served over a certificate we can declare that our metrics also is declare is you know um accessible over HTTP or it needs a TLS config as well and that's what we can define with these variables what is the path of our certificate what is the name of the certificate and the key we want to use for our metrics. The same goes for our web hook. Now there's a very good fun there's a very good um concept that operators can help you with or rather when you are running distributed systems like HCD or especially when you talk about your cube uh controller manager see that is also a controller what we are writing it has many it's a collection of multiple controllers but this runs as three different pods in your cluster or rather it runs each one on the master in your cluster. The thing is when you are writing a controller uh it is very important of how these controllers are running in parallel and do they all make changes or not. For instance uh take if I was running two copies of my EC2 controller. So this is one controller and this is another controller and there was an update um which lets me create uh an instance you know uh I did an update I created an object of EC2 instance kind there was update and this update was seen by both of my uh controllers controller number one and controller number two they both are going to go and create me an EC2 instance and this is not What I want I do want high availability but it should be active passive. There should be one leader. There could be multiple replicas for high availability but only one at one time should be running and this is what leader election u you know um uh is something that you can use and cube builder makes it very easy for you uh that it allows you to uh declare the leader election with a simple boolean. So in that case this is also running. This is also running but this is a leader. So if in case an update statement or an event comes from the API server only this one is seeing it and only one instance is created which is what we want to do. The other one is there but it's not the leader. If the leader is no longer running or or automatically it's going to become the leader and this will be then serving your requests for the EC2 instance custom resource changes. This is what uh leader election means and then you can enable if in case you want to have reader election and you can run your operator into high availability. We define the probe address on which your uh health probes are available. So you remember this this package which is the health where you declare your health's endpoint and the ready endpoint on what port number uh they are exposed. By default the port number I think is 8081 here which is the health probe bind address the command line flag and this is the variable that is going to be responsible for it. Do you want to use secure metrics or not? And this this variable secure metrics and metricsert paths uh name and key they are relating because you can say I want my metrics to be exposed over h over TLS and if you say that you want them to be over TLS then you can define your metric certificate paths the certificate name and the key otherwise there's no need for that. Uh you can also say if your operator does enable HTTP2 or um HTTP you know it does not enable HTTP2 and then we have a list of functions uh that are our TLS options. I'll make it simpler explanation as we go ahead. So we declare a couple of variables we declare a couple of command line flags. We define some options for our logging that this is development true. when you say development, it actually um gives you a stack trace on warnings as well. Um it doesn't give you any sampling. Uh if you go for production, it only gives you a stack trace on um on errors and it does do a sampling for you. So if in case you are deploying this to production, that's something you should always consider um development as false. Now we set up a logger. We uh we passed all of our flags of the CLI that was given by the user. We um you know we define our options for logging. We create a new logger. Essentially what we're doing in this line here is we are setting up a new logger with our zap options or with our logging options. Now with your TLS when you have this TLS config it's kind of like a list of options that you can do. One option here is if you want to disable uh you know uh if you do not enable HTTP2 here in case you are disabling uh HTTP2 you can append that to your TLS option. So we say in this case uh my um you know I did not enable HTTP2. So for me in the TLS options it would be I disable the HTTP2 and I only enable the version 1.1 of my HTTP because I'm disabling the HTTP2. Now here's where you create some watchers for your certificates. You remember we talked about these certificates for the metrics and there could be certificate for the web hook because you can expose both of them um over over TLS. So the certificate could be for your web hooks. The certificate could be for your metrics. And we have a we have a search watcher. So essentially what happens is let's say uh this is what I already explained. You have manager. The search manager renews your certificates on the disk. This watcher will be detecting those changes on the certificates. It will load them into the memory in the current pod in the current operator. It does not restart the operator. It does it on its own. So there's no there's no downtime. There is zero manual intervention. Otherwise, you'll have to um restart your your operator because your um you know your certificate was updated by the search manager. We define our TLS options which is again a list of functions uh that returns us the TLS config and we um instantiate a new variable. So it's kind of like we are creating an alias and this is the one um by by this time the TLS options is the default TLS options um that we would be using and we declare a new variable and we set that as a value. So we can customize um the TLS configuration for our web hook server uh if in case we want to use a watcher or in case you want to even use a certificates or not. So uh it's easier for us to customize. Now if you really gave a web hook certificate path which is here if you did give a certificate path that means you want your web hook to actually be serving over TLS and that's the that's the thing then if the length of your variable uh is greater than zero we will say initializing web hook certificate watcher and I will be then using the TLS as well and I will be using the certificate. So we define a variable error. And here's where we create a new watcher for the certificate uh path and the certificate key. If in case there was an error, you just simply exit one because you wanted a TLS config for your web hook but you couldn't get one. So it makes sense to stop right there. And here we are adding a new option to our web hook TLS uh options. this variable it contains right now till this point it only has one TLS option which is disable HTTP2 that's what we we did here you know uh by this time it only starts with one TLS option which is disabling HTTP uh 2 and if you have given a variable um if you have given a webbook certificate path we then append onto this TLS options that we do want to use um another we do want to use a web hook uh certificate and this is the get certificate function from the TLS config that gives us the name or the information of the certificate we want to use for our web hooks and here's where we are creating a new web hook server with these TLS options similar thing happens when you are working with a metric server options so these are the metric server options uh where we define the bind address on which our metric is going to be exposed This bind address is 80081. Do you want to use secure metrics or not? And what are the TLS options? Again, by this time we are just disabling the you know the HTTP2. Uh we don't have any TLS right now because if you do not do uh if you don't give secure metrics which is as a boolean if you do not give um secured metrics then there would be no TLS options. You only work with HTTP 1.1. you disable HTTP uh you know uh you disable HTTP2 but if in case you did give secure metrics you will be using some sort of authentication um um that your metrics endpoint is not publicly uh it's reachable but not accessible there is some sort of authentication and authorization and only the authorized users and service accounts can access your metrics now u this was the metric service opt that we started with. If in case you did want secure metrics, you give uh some sort of authentication and then this is the same logic that we did for our web hook certificate path that if in case you do give you know your metric certificate path. You create a watcher like we did for our web hook. Uh there's a watcher which is for our metric certificates. Then we append uh the metric certificate option TLS options uh with the certificate uh information. Essentially what this does is if you did give me a certificate path if it's not zero the length of the certificate path is not zero you give me the path of the certificate I'm going to run your metric server with the TLS option that that serves the certificate information. That's essentially what it is doing. So you should not get confused on uh on what this is happening, what this is doing. I just told you if you do give the parts of your metrics certificate, it's just going to expose your metrics endpoint on this certificate that you have given. The same thing happened here. If in case you did give a certificate for your web hook, it's going to expose your validation or mutating web hook over with this certificate information. Uh and here's the one which is quite interesting. This is the from the controller manager from the controller runtime. You see this is the one which I just showed you. This one uh lets us create a manager. Within the manager you can have multiple um controllers. It looks something like this. So here I have in my main.go file um this is my operator. This is the main.go file. In here I have a manager. Oh, wait a second. This is my manager. Let's take it this way. And within my manager, then I will have my controller. And I can have multiple controllers in a manager. If I wanted to uh write something about this, this is my controller. This is my main.go go which is responsible uh for creating a manager using the controller runtime and then the manager is responsible to or it's our responsibility to register our controllers with the manager and that's essentially what we're doing here. So once we declared all the variables, once we gave all the flags, once we defined all of our TLS options, once we have configured if we want to use TLS for our web hooks and metrics and if in case we want to use authentication with our metrics or not. Once all of that is sorted, we start or we use the new manager function that returns us a new manager which is available here. This is our manager. This variable has our manager with all these options. What is the scheme? So our controller knows about all the resources, custom resources or the the core resources available in Kubernetes. What are our metric server options? If in case with the metric server options, do you want secure metrics or not? You know, uh what is the port number for your metrics that you are binding to? What is the IP address for the metrics? What is the endpoint which is usually uh by default/metrics? And then if in case you have given some um certificates the same thing happens for our web hook server. Is it secure in terms of have you given certificates to that or not? Uh and that is our web hook server. Um we then declare the health probe endpoints which is um which is a probe address that is um where did that go? At81 this is what your livveness probe and the readiness probes will be looking into the container when they are doing a probe. And here's where the leader election because when you are creating a manager, the manager should know uh are you looking forward to have a leader election and you should definitely do this when you are building an operator that you want to run in multiple replicas in multiple pods. There should be only one which is both of them are running but only one is active at any time. So this is this is absolutely your responsibility um that you can enable the lead election and then if you could not make the manager because new manager returns you the manager and also an error. So if you could not create a manager, you give the error that I was unable to start the manager and you simply exit because if you don't have a manager, you don't have anything. You don't have a controller. So that's that's the over um that's the one that looks on your controllers. If there's no manager, there's no reason to continue. Just exit right there. And that's why we use the OS package. Now once your manager is created we need to register our controller which is the the custom resource which is uh what we need to do here. So if you were able to create the manager we are using you know um we from the EC2 instance reconiler we define the client and we use the manager.get schema which tells our manager what is the schema of our EC2 instance. Essentially, we use a function called setup with manager. And this one sets up our EC2 instance custom resource with the manager and which is available here in the EC2 instancecontroller.go file. This is this is the one uh which is where our reconil logic is and where our reconider logic will be. So it sets up our controller in here in the main.go go at this point once we started our manager we set up or we add our um you know we add our um custom resource or we add our controller to our manager. So the manager knows that I have this particular controller. This is what I need to listen on to if any changes are done to this custom resources and this is what the logic is what I need to run with uh with the operator. Now it's also interesting here if in case you were having some certificate watches if this was not nil you add the certificates to your um you know you add the certificate watcher to the manager for your metric server for your web hooks again um we don't we don't use certificates right now and I'm also not using any web hooks for mutating or validating so I'm not going to do anything um any certificates for me it's going to be empty otherwise you will be adding the certificate watcher to your Manager. So manager has couple of things. It has controllers. It has another controller. You can have more than one. It will then have the watcher as well for the web hook certificates. It also has a watcher for the metrics certificate and it watches and renews the certificates or reloads the certificate on the fly. So you don't have to restart your your operator. From the manager we also get uh a function called add health check and this is where uh the health check is is being done. Uh we add two health checks or two endpoints. One is a /halth, one is a /ready and this is what you can use like a fairly simple Kubernetes health check that lets you see if your operator is healthy, if your operator is ready or not. And here from the manager which was written by the new manager function. This manager has a another function called start. And this is the one that starts our manager. It's kind of like you got a car which has an engine which has a you know which has um a mechanism for the airbad. As soon as you turn on the key the whole thing starts. So first thing is your engine starts. It starts sending power to other components. This is a similar analogy when you are starting this particular manager. So the manager starts and then it starts the other controllers inside of this process. It starts the watchers and uh and everything comes in into life. Now of course if you were not able to start the manager or create the manager above here. either you were not able to create a manager instance or start the manager, we simply just exit because without the manager there's nothing uh that is available. So I think this was the whole main.go file and uh what I wanted to also show you here is we do import quite a lot of packages. We do import quite a lot of go packages around here. One of them is compute v1 in our API v1 directory. And here in the spec, this spec matches to our um where I go config CRD basis. The CRD around here. See whatever you give in your um custom resources spec that gets reflected into a resource called custom resource definition. And here's the way you're declaring. You're telling Kubernetes this YAML file gets installed into Kubernetes it's a resource that tells Kubernetes about other resources. It's a custom resource that tells Kubernetes about other custom resources. So you tell Kubernetes that I'm telling you about another custom resource who looks something like this. Its version is v1. Its name is EC2 instance. It's a list you know it's the name uh namespaced scope um object it is under this particular group and here's where this is for your EC2 uh instance and you can see here the same one to one mapping we have the AMI ID we have the instance type we have the SSH key and we have the storage which is again given uh given here now at any time when you are writing a spec for the API you might want to change some things. Imagine you could say I want to give a tag. um or you would say I want to give a department and this is going to be a simple string which is what you can use as tagging you know so when you create an instance you use this department value to add that as a tag to your EC2 instance and whenever it's very important at any time when you make changes to your specification you have to run the make command more precisely ly um you need to do the make manifests because your CRD is not aware that you just make changes to your specification. The CRD is still older. Think of this as now it's outdated compared to the spec where we added a new uh value. Do I have a department here? I don't have that. I can't search for it. Okay. So once you make changes we do make manifest and as soon as we do this you see a new department um spec is now added which is type of a string. We can also say um um maybe I want to add project which is going to be another um tag. Uh and then again I will need to use the make manifest because as soon as I do make manifest you can see on the right side it's going to be added here. You see um the project was again added. So at any time you make changes to your spec your CRD needs to be updated uh on the disk which is with make manifest and you also need to update the CRD into Kubernetes because see the flow looks something like this. This is you. This is the spec. And you make changes to the spec. Now, this all is happening on your computer right here. It's all happening. This all is happening on your computer right here. Um, and wait a second. Okay. So this was a spec that you changed and you changed your CRD on the disk. However, um the CRD doesn't just need to be updated on your computer where where you're you're developing. You then have this Kubernetes cluster which is again um where you need to have a CRD and from where you can then create a custom resource. We talked about it the CRD now and from that you create a custom resource. Now you see you made the change and it's updated here. It's version two of the CRD but you are still using an older one. You're still using a version one. So you can use make manifest in the make file which is given to you by uh the cube builder. You can do make manifest. It updates it on the disk. And if you are pointing to the right Kubernetes cluster using your cube config the environment variable, you can then use make manifest and make install. It is then going to apply the same CRD which was generated by the spec change all the way to your Kubernetes cluster as well. So they're always in sync. You're not thinking I made the changes to my spec. My CRD is updated. But when I try to make changes here for this new change, you know, I want to add a new field called project. It says there's no field called project. But I see it here. It's probably because you did not um update your CRD in the cluster. You only updated that on your disk and that's not going to cut it. So um usually if I ever make changes to my spec, I do make manifests. many fifths and I do a make install. So I update this on the disk and I also install this. Make sure you're connected to the right cluster otherwise um if it's a different cluster and the resource the custom resource does not exist it gets installed there or if it's there it gets updated and there could be some breaking changes that you are introducing. So be very very careful when you're doing it. All right I think this was the whole uh explanation of the main.go go which is probably something you will not use a lot you will not make changes to but it's it's absolutely important to know all these options what the web hook watcher does why do we have so many uh packages in gold um you know you can expose your metrics uh endpoint securely when I say securely I mean with an authentication and you can also use TLS or not this is something optional both of them the Same goes for the web hook endpoints. So it is something which is which is absolutely important to know uh that you can also do the leader election and uh this is the main uh function where your operator starts. So now that you have a very good idea of the main uh go file uh which is the one that starts everything. Let's see how the reconiler works. Let's see the reconciler in action. We will make changes to some custom resources. see how our operator gets those changes and then what can we do on top of that. This is what we will be laying as a foundation of creating our operator that reacts on the changes of the EC2 instance object and then we um we will move on ahead from there. Okay. So whenever you want to write your own custom operator the first thing you need to ask yourself is what kind of resource are you going to manage? In our case it is going to be an EC2 instance over on Amazon. We are writing an operator. We are building a custom operator that goes to Amazon based on our behalf and it uh you know creates you an EC2 instance. So something would look like this. You're going to have your Kubernetes cluster in which you have your operator running and there is going to be a human a certain someone that gives you a YAML file because we talked to Kubernetes via YAML. The the interesting thing about this YAML is the kind that you have declared it's going to be um you know the API version that you have declared using cubebuilder which is cloud um which is a compute I think which is compute.cloud.com/ver version one of this API resource and the kind in this case is E32 instance. Now of course in the end what's happening you give this YAML to let's say the Kubernetes API server because it knows about um the EC2 instance which we will deploy our custom resource definitions to Kubernetes. This resource change maybe you say I want to create a resource of this kind. The controller in here will look on this change. It will get the data from the API server and this is the one responsible to go to Amazon and creating you an EC2 instance. This is the one responsible for making the authentication with EC2. This is the one which is responsible to provide the minimal set of instructions you need to give to Amazon when you want to create an instance. This could be uh the instance you know it could be the instance type that you want to give it which is absolutely required. This could be uh a security group you want to give which I think is absolutely required. Um you can you also would definitely need to give some storage on how much your machine would be needing. Some things could be required some things could be optional. For instance, um tags they are completely optional. You can give that, you cannot give that. It is up to you. So when you are writing an operator, you are writing a custom resource like this, it is on you to have some minimum at least most required uh things that you want to send to Amazon. And this is where when you are designing your spec because when you give your a YAML you will have kind API version then you will have a spec and then you will have a status. So this spec here is actually matching what you given your YAML for other resources and that is what you will be having. So in this case your YAML would look something like speci ID. This is going to be the name of the key. Uh then SSH key, instance type and instance subnet. We are using these JSON tags so that Kubernetes can unmarshall uh it knows what is this particular thing that you are giving me called an AMI ID. What to do with this particular object or with this key and in the request that is coming to the API server. We might want to extend this in terms of let's say uh in here for example storage. So I am giving um I also have an option called tags in my YAML which is going to be a map of string and string and you can also create your custom strct types. For example, if I give storage um and then I can have a custom object here. You see we know what is a string. Go knows what's a string. Go knows what's an integer. Go knows what a boolean is. it doesn't know the embedded type of uh storage config and that's the problem. It says it is undefined. So you can define another strct which is going to be uh type storage config. Oh wait a minute storage config uh and there you can give um you know the size of the and that's what I love about these AI editors. So you can give the volume size, you can give the type of the volume that you want. You know, Amazon have different type of volumes there. And then if in case you want to give your device a name, you don't want that. The only thing I would like is a size. And um it's going to be the type of the in uh the device that I want, which is going to be uh one of the Amazon provided ones. And then you can also add additional uh uh storage which is in here in this case one is the root disk and then you can have additional resour devices and this is where this omit empty comes into the picture. It's very very handy. The same thing could be done for our tags as well. The thing is sometimes these resource these options are you know these things are optional. You can give a YAML for Kubernetes that creates an EC2 instance but you may have tags. You may have additional storage. You absolutely need the instance type. You absolutely need the AMI ID that would be wrong. If I do um you know in here if I go and say uh omit empty this is wrong because it has to be a required field in your YAML manifest your resource. So you can choose when you are building your um when you're building your spec as into what things you want. In this case the additional storage is a string or it's a it's a list of storage configs. So you can add additional storage configurations. In my case um I I would keep the additional storage just as you know a type and a string. The same thing will happen for your status. So when you do cubectl get status hyphen o yaml what you see is the status dot you will see the phase in which it is you will see the instance ID you will see the public IP. This is probably the information you get back from Amazon. Imagine this if this developer gives you a YAML. Let's say you are building a internal development platform. You want the developers to query uh the resource that they have created. when they do cubectl create - f which has an EC2 instance and then when this guy says cubectl uh get EC2 instance you need to give him some information or you need to give her some sort of information probably the first thing you want the user to know is the state if the instance was failed if it was running if it was pending whatever state it was and then you probably won't want to give them the public IP of this instance. If in case you allow your organization allows for the instance to have a public IP, you will do that. The only place you can get this information is from Amazon. So when you are creating your instance in this case, you want to pull if it's running in certain time. If it's not, you fail the operation. Otherwise, what you do is you get back some information and this information is going to be the state of the instance and then it's going to be a public uh IP. In our case, this is what we care about. There will be many things that can be given back uh as an operation of creating instance. But that's not what we care about. We want to show the user that in their status they can see um the phase which is going to be a string uh the instance ID which is also a string and the public IP which is also a string in our case. Now whenever we build um whenever we u make changes to our API spec I told you that is absolutely important that you run the make command from cube uh from the root of this um cube builder project so that it generates you the custom resource definition. Um what has happened uh wait a second. All right. So when I do um in my API version one EC2 instance types.com actually it's in config CRB basis and then compute um cloud.com E2 instance this is the actual custom resource definition that you have created when you make changes into your spec like in here when you make changes into your spec what happens is um when you make command cube builder code knows how to write the custom resource definition as a boiler plate and this is where you define the group for your resource. This is where you define the kind for your resource and then you define the version of your resource. So this would tell you that for a for a single kind of resource you can have multiple versions because you see it's uh it's a list of versions that are available. So you can have EC2 in a cloud.compute.com/v1 compute.com/we1 then this schema would apply cloud uh compute.cloud.com/me2 cloud.com/me2 another version of the schema would apply and this is why you probably might have seen that this particular key is only available in a newer version of your YAML there could be some key which is only available in the newer version or uh in the older version it's um it was only available in the older one because in the version two that might have been removed the important thing is um is the spec in here so we have got our properties. We have an AMI ID. We got our instance type. We got our SSH key. And then we got our subnet. And these all are required because we did not get the omit empty. But now because we made some changes into our spec, I absolutely have to regenerate these manifest. And for that I can simply do make um manifests. So what that will do is it will be updating your custom resource definition with a few um more um you know with a few more parameters. For example, one of them is the additional storage. It was not there before but now it is. So then you can have an additional storage and then there's a new option also called storage and there's a new option as well called um tags which is a type of string. So every time uh it also updates the required because some of them do not have per omit empty. They are absolutely needed. This is how Kubernetes knows that this particular key is not available in the YAML. I have to cry about it. I cannot let the user give me this request because the custom resource definition has marked this particular uh you know string as required. this particular key in YAML as required but the user has not given me that. Now this is what you will give to your Kubernetes cluster before you can create an EC2 instance before you can do anything w with the operator. The first thing you need to do is you need to give this to Kubernetes because if you do not then when you create a when the developer creates a YAML of kind EC2 instance and then the API version which is cloud.com Kubernetes has no idea what is this um you know what this resource that the user is talking about what is this group called compute.cloud.com cloud.com in version one I don't have an inst a resource called EC2 instance and this is something you can either uh use a cubectl apply uh with this custom resource definition yaml or you can use make install command with the make file that's something what is actually done for you so you see we use customize to build our custom resource definition and then we apply that to um cq cubectdl apply - fn and on the standard input and this is where we now have our custom resource definition let me first do make uninstall see what happens if I do um cubectl get ec2 instance uh dot um oh wait here see if I do k get ec2 instances compute.cloud.com cloud.com or if I just do tell me how many sitter instances do I have Kubernetes says I don't have that resource but if I do okay I'm going to deploy you or I'm going to give you a custom resource definition at least don't say I don't know what that resource is if you have that give the user if you don't have that just tell them I do not have that resource but don't just say I don't know what resource are you talking about so this is What you do when you uh give your um you know when you do a make install creates your custom vis definition which if I see here you can see this is uh you can do a CRD on you can do a get on your CRDs and this is the custom resource definition that I have which you actually can uh also see like this and this is the same thing that I just showed you on uh on on on cursor which is not on cursor. shipped in terminals. So that's how one would actually um update or create the custom resource definitions. In my case, uh how would the YAML look like? If I would probably ask um you know um my my AI that okay take uh take this spec and give me an updated YAML for this resource. It's going to just spit out how the YAML would look like. And see this is additional to what's what's going to happen. Uh I just going to I'm just going to accept that. Here we go. So this is how your YAML would look like. It's going to be a kind of visto instance. Then you see some specs are there. It tells you um you would have an AMI ID, the SSH key, instance type. Maybe when you give this to your developers, you might want to make it a lot simpler or at least make things like um instance type or VM preset, something like that. if they're more familiar with with those words. Uh maybe you can do SSH key. That makes total sense. Um I think this this does make sense. Um it would have been better for um other other examples. But in this case, the YAML is perfectly fine. So I I would say okay, this was what I wanted to show you as a YAML. In our case, um, the next thing that you would do is once you have your YAML defined, once you have everything defined, now we need to look into the reconcile loop. See, by this time, Kubernetes knows that it has some a custom resource called um EC2.cloud.example doommain.com. In version one, there's a resource called EC2. Now if someone gives it a EC2 instance what to do on that if someone gives it a YAML that please create me an EC2 instance what's going to happen how would it react to that and this is where we will be looking into our reconcile loop so let's get started and let's see how we will build a recon loop. So our reconstru loop would look something like this. It's under internal controller EC2 instance_controller.go. This is where the magic happens. This is where whenever you make changes to your custom resource, that's the place where it comes to and then this is where the logic you would be giving to operate on the resource that has been changed which is your custom resource. It is in the package controller. It is importing quite a few things. One of them uh is controller runtime. This is absolutely important that handles the runtime of your controller. And also you see it is actually getting our own um uh ECI EC2 instance. This is going to go to github.com in here operator repo API v1 and then it's going to call this as computer v1. Essentially what this is doing is your controller needs access to your spec of the uh custom resource. It could also have been very simply done where I could say you know um please go to API v1 because I have that locally available or okay I think it's better because I have it already on GitHub. So what's going to happen is if I show you um if I go to githubhub.com and here is going to be my repository. Let's go to GitHub. I should have just copied otherwise it goes to Golang Populate repo. Here you can see an API v1 and this is where it's looking for the EC2 instance type. This is where our code is. So this is what Kubernetes operator will be using and the controller will be using to map your request to a particular known um data type to a particular known uh spec or status of the custom resource. This is the heart of your um object that you're creating. Um it creates you a reconiler which is used in the client runtime that helps you communicate to the Kubernetes API server. It has a schema object which registers your schema. Couple of cube builder um markers I think yes it's a marker which is creating you the arbback rules so that you can uh work on the custom resources because when you create an operator it would be running it in its own nameace but if a custom resource is created in a different one the operator needs access to see in that name space as well. So this is where the arbback is extremely extremely helpful. Now what can we do with this? This is the reconcile function. This is the one where all of your requests are going to be uh looked into. This is where all of your you know um whenever you do a cubectl get or cubectl apply this is where the changes are going to be looked upon. This function has two return um it returns two things. First it returns the result of the reconciliation and second it returns the error in the in case if there was any if there was no error it will simply return a nil. Now this is the beauty of Kubernetes self-healing. You know how uh if you create a pod which has a persistent volume claim but that PVC is not bound to a PV yet that pod is going to be in a pending state. It keeps on being in a pending state but as soon as you create a PV as soon as you add it to the persistent volume claim the pod is then automatically started because there was a rec going on for that particular pod. the the reconciliation for the the logic for the pod kept seeing if the requests are fulfilled. If they are not, I give VO an error and I start the reconciliation again. It puts it in the queue to reconcile. This is the beauty of selfhealing. It will be done eventually once all the conditions are met and you don't have to trigger that reconcile uh you know uh you don't have to trigger another run of the reconcile loop yourself. Kubernetes does that for you and this is where you will be giving your logic. The first thing that you do the very first thing that you will be doing is you need to operate on that instance. For example, um the basic thing you when you talk to your you know when you say that users can then create an EC2 instance of their type. You want the user to give the name of the instance. You might want them to give the tags of the instance. You might want them to give the storage config. So you need to extract this information. You need to extract this information from the request from the API server request that came to the reconciler loop so that you can use this information to talk to Amazon in in our case because it's a cloud operator. So you need to store or you need to get all these objects that are being given in this YAML by the user in certain variables. So you can iterate on top of that. So this is very important. The user is actually creating a resource of kind EC2 instance. You also need to have a variable of kind EC2 instance. So that Kubernetes you can use the Kubernetes schema to store your actual keys in your variable. It's think like the user is sending a circle. You need a mold that can hold the circle. If the user is sending a triangle, you need a mold that can hold a triangle. If a user is sending an int data, you need a variable of type int. The same thing happens. The user is sending the data of kind EC2 instance. You need a variable that will be of kind EC2 instance. So let's declare that first. The first thing I do is EC2 instance object uh is going uh EC2 instance object is actually here. So I'm using the compute v1. This is the compute v1 and in here I have declared the EC2 instance and there you see this is essentially what we created in our uh types.co. Now we do have a spec then we have a status but essentially this is the root of our Kubernetes um object the EC2 instance will have some metadata it will have some object metadata uh it will have the spec and then the status. So this is what we are calling and creating a variable in our reconciliation object. Then what you can do is because the API server will be sending a request to your controller or rather it's the other way around the controller will listen if there was any changes done on your custom resource and then you can iterate on on top of that. So you create a variable of type E2 instance object. I would rather make it simple just to keep it easy to instance and then we can use the get function. What this get function does is it uses the context which is of your of your request. More importantly, it gets the name space under which this resource was of changed. I'm not saying created. I'm saying it gets the name space of the resource in which the update happened on your custom resource and the actual inflight um YAML the actual context of your YAML is then going to be stored in this particular object in the EC2 instance. So think of this as you take the YAML from the user and you give it to your reconciler. So now it knows the name space in which this object was updated. Uh the name u you know the instance type this YAML had the kind um the storage type this this YAML had the number of tags the user wanted that you can now create on top of this. So I was want to say let's say um log uh there was also I think before this there was a logger that we can also use. So here you can see we have a log function and we can say um I want to log all of my request and using log.info I can do that. So I got I create an object of type EC2 instance and this is the EC2 instance instance type. I get my object which is coming from um the insight request and then I'm saying reconciling EC2 instance and you see I can uh get the name of that particular resource rather than info let's just print this for now. So um I would say I want to have an EC2 instance. EC2 instance and then I can say uh print lm I got a um I got a request for an EC2 instance in the name space and then I could say and the EC2 instance is um EC2 instance just just keep it like that uh you can also probably then say fmt.print print ln and you can print the entire spec you want. I don't want to print the entire spec. I would just say I got a request for an EC2 instance in this name space and the instance is instance name. You see these are all the options that you can where I go. Oh, wait a minute. Here it is. And the instances are here. I can say and the easy to inst Oh my god. Wait, wait. Let's get let's do that again. I want to I want to just uh see the data that has been given to me. And I can say uh I got a request for an EC2 instance in the name space and let's keep new uh uh prints the instance the EC2 instance name is EC2 instance name uh and I would say then the instance FMT.print print lm instance type is uh e2 instance.spec spec and you see this is this is the beauty of uh the AI editors again now you can see I am able to get all the information which was sent which was you know watched by my reconciler by my operator under instance dospec and you can see AMI ID SSH key subnet tag store regation storage this is essentially what you were building in the spec of your custom resource this This is a onetoone mapping. That is why we created a variable of type EC2 instance and then we got the you know the insight request that we received from the API server and then I'm saying I got a request for blah blah blah. The only thing I do not have see I have the instance ID I have the AMI ID SSH key subnet tag everything. I don't have the name for my instance and actually this is the name of the object that I'm giving but maybe I want the user also to give the name of the instance and I would simply say instance name because instance name could be different than the Kubernetes object kind uh metadata the YAML that you give they would be different and here I can say Um my name would be in the spec uh spec dot instance name. It's going to be a capital spec. Now the important thing is I just added another uh object in my spec my custom resource definition that is right now in Kubernetes. It has no idea about this new instance name. So I would have to do my magic again. So I would do make generate make manifests and then I would say make install. So that my Kubernetes is now updated that there's a new resource called there's a new uh there's a new key in the strct for the spec which is called instance name and um that's it. Now once you get the data once you you know uh iterate on top of that in my case I'm just printing it right now but as we move forward we will use this data to talk to Kubernetes and then I will uh create myself an instance that is where you would have your actual business logic what we will then do is once you have used the data in my case I'm not changing anything in the object object there was a resource created I got information about this but I'm not updating that resource so then in that case I will be returning a a result which is going to be you know um is if in case you are spending it contains the result of the deconeller reconciler invocation if you go to the controller runtime on go on on the go uh consiler this way actually contains two things. Whether to recue this or not and this is a default to false. This is very important. When you exit your reconciler function, you need to tell two things. Whether there was an error in the reconciler function and is there a requirement to rerun this reconcile loop. You only remember we talked about this in the previous part of the video that you only reconcile you only uh you know rerun the reconcile loop if you have updated the API object. We are not doing that right now. So we do not need to u you know send any uh reconcile boolean which is vq as boolean by default it is false. So in case in our case we don't have an error and we are also sending a pause for the reconciliation this reconcile loop will not run again. Uh it's kind of like um when you start with the reconcile loop. So this is this is how it looks like. You have the reconcile function and the request came over to this function. you made your change, you made your business logic, whatever you wanted to do. In our case, I'm just printing things. I'm not creating an EC2 instance. I'm not updating my custom resource with the status of uh the EC2 instance creation. I'm just printing this. So, because um here um should be a bit bigger change made. So in my case, did I make a change into the custom resource into the custom resource that request came to me which is in the EC2 instance? And I would say no or you could say a yes. And in in case you have no changes, you would simply return nil for the error and then false for your reconciliation. If you did give a yes, if you made some changes, then you have to return a true for um the reconciliation and then if there was an error, you will return the error. If in case the error was nil, you will return the nil. This part we will talk about uh coming up. But for now, I'm not returning any uh know I'm not uh changing anything in the EC2 instance object. So I'm just returning a false. Now this at this point guys uh let me do uh l.log and then I would say um let's do here recon reconciling EC2 instance. This is the name and I would here say reconst EC2 instance and this is the name of my instance. Now let's get a YAML and then see how this will be um functioning. Now this is the time we run our operator in Kubernetes. Now we can make a container image. We can you know push that container image to a registry and then get it from there. The good thing about using cube builder is when you have a working development environment and you have a cube config which points to a operate on your cubernetes cluster you can just run the main program locally and then it will be as if like it's running in your kubernetes cluster and I will again uh call my trusted AI to use the spec and give me a dummy YAML So we can create that. Uh uh uh this is the spec. Let me quickly get this and then I would say please undo everything. I don't need that change. Cool. So um let's say Kubernetes. Do I have a folder called Kubernetes? No. Let me do an example. Uh instance instance.yaml. And there is our spec. Before spec we have a API version and then we have a kind and then we have a metadata and there you see we then have a spec. The API version is v1. The kind is EC2 instance and the metadata uh wait the API kind is API version v1 but it's compute.cloud.com/v1 cloud.com/v1 EC2 instance metadata would be name of Kubernetes object for uh EC2 and there we go this is simple uh what we have and then I would say let's run our operator now we can do go run cmd main go because in in the cmd folder um wait where is that goes here in the cmd folder. You have your main program. This is the entry point. In any Go code, your entry point is always going to be the main uh go file. This is the one that registers your schema from Kubernetes. This is the one that creates a client so that you can talk to Kubernetes. It registers some u you know um some booleans, some flags if you will. We will clean this up because we don't need a lot of that. We already have gone through this code. The most important thing that it does is it starts the manager here. Uh enable enable enable enable enable. But I think we did see somewhere that it was starting um the manager. Wait a second. This is the new manager function. Just going to give the manager new. Where was that? Uh uh uh uh here. So we're going to have a log of starting manager because we did not work with the web hooks. We don't have any readiness check, livveness check, nothing. So we should just see starting manager and then we should we we will be seeing if we get any requests to our controller. So here we will be exporting the cube config. Let me increase that font a little bit. And here I would run my function. If this is a little a bit small for the font, please bear with me. I hope this is this is you know seeable. But uh essentially what I'm doing is I'm running the main function now. So we will be running our operator. Do I have any EC2 instances? Uh, no. Do I have them in any name space? Uh, no. How does our example look like? So, if I do k - f example. Oh, wait. I need to go to operator folder here. And then I can do k - f example instance create. Let's do a dry run. See if our yaml was good. And there you can see the yaml was fine. Um then I can just simply say first I run my program in here and this is how your go code will be running. So you see this is all what cube builder does for you. You do not have to set up your authentication with the API server. You do not have to set up your um you know um how would you run your your controllers? How would you run your multiple operator loops that you have for different API versions? It does that for you. It starts an event source. So, it's kind of like the listener for your object in Kubernetes. And here it's starting a worker for there's a controller for EC2 instance. This is the group and this is the kind uh which is uh EC2 instance. So, it's kind of like you have one controller for one resource. It is a onetoone mapping. you can have multiple instances of that controller and in this case you would do a leader election uh because if one object uh if one you know instance is managing your request for that custom resource others should not do that but in our case we only have one replica but we have one controller per object if I was if I was creating um more custom resources let's say right now I have an EC2 For instance, this is my custom resource. For that, I have a controller. My controller here, you can see it's called uh also EC2 instance. If I was to create another custom resource which was let's say a storage bucket, maybe I want the users to be able to create buckets in my Amazon account very easily. There would be another controller uh which is going to be then storage bucket. They could be running in the same manager in the same manager or operator pod. I think this is where you can review uh the part of the video before where we talked about what is in the operator. there's a manager within manager then you have multiple controllers but it's a onetoone mapping to the object and um the the controller think of this if anything happens to this resource this code will apply if anything happens to this resource then this particular code would be would be applied now is the uh now is the moment of truth would it would something happen if I uh simply just say please create me an instance.yamel. I should see something in here. That's what I am more concerned about. So let's create that. Um of course it's invalid. I cannot Oh, there you go. So it says the kind is invalid. It must be EC2 instance. Of course in the dry run for client side but not much sample here. my kind was wrong. And then if I do a create again, you see there is my request. I know that the instance name is my EC2 instance. This is Kubernetes now knowing about this. This is our operator knows about it. So it started the worker. Cube builder started the worker and this is our code from here till here. This is our code. We get the log which is reconciling instance and you can see this is the code which is um started here reconciling instance name and then we get all of our U program executed. I got a request for an EC2 instance in the name space you see it gives you the nameace default and then the object name as well um which is request namespace. This is telling you the the name space as well as the name of the object that you have. The EC2 instance name is this is now reading the spec and you can see tags are it's giving you a map of environment dev owner is Alice which is this is what in your YAML looks like uh example and instance. So essentially what the user gave my program our operator our controller most importantly knows about it you see so um my storage would be size 50 and then type GP2 it's actually just printing this as an object map but we can u do that even better for storage let's let's make some changes I want to say storage size is 50 and type is g2. So I could say storage size is um you can say f is size and type is you see storage dot type you can obviously access any sort of object that was there in your spec like this storage dots size because this is how you access yaml so I would say spec dot storage dot size and that's also what's happening spec dots storage dots size and same for the type. Um, you can also do a delete. Now, see this is very very important. This bit executed when we created the resource. When I delete that resource, when I delete that resource, you see my reconciliation loop started again from the very beginning. This is absolutely absolutely important. Whenever you make any changes on your object, the reconiler starts from the very beginning. It does not know whether you created the resource, whether you deleted the resource, whether you um you know um whether you updated some metadata annotation. It has no distinction of what the uh what did you do? It knows about the update that has happened. And this is where it is your duty as someone who is writing the operator, someone who's writing the controller logic that you can make changes. You can run your reconciliation loop many times. But if no change was required, no change is actually made to your object which in this case you can see because the request because the resource was deleted we don't have any EC2 instance name we don't have any instance type nothing but the loop ran completely and here you can it says reconciled EC2 instance blah blah blah something more evident would be when I just uh show you let's say the name of the instance I want to get rid uh all these things because I want to keep it simple. Uh or rather I would say um I would say fmt ln uh got a request simple or update was made to the e uh EC2 instance restores. I'm not saying the name or anything. I'm just saying that there was an update made and this is why I am um called or reconiling that makes no sense. Now I will run this again. I stop my program and this is the beauty of stopping the program when you are building this with cube config uh with the cube builder because it has a graceful shutdown. It doesn't just stop the program abruptly. It is a graceful shutdown and um it it helps you uh cleanly shutting down your manager because I made some logic changes. I'm now starting this again main.go and then I would say um k - f create. So here you can see it says update was made to the EC2 instance resource and this is why I am recon I am reconciling it that's the main uh logic here and then I got the instance type which is E3 medium if I made some changes to this EC2 instance let's say I want to add a metadata I want to add a label here so I want to say labels and I would say Hello Poland world. I save and exit. You see I got another line. It's not like I created the object. It's kind of like I only um updated it. So you see I was not making a change as in I was creating that resource. I just edited that and that was only a simple metadata change which was the labels. But my code ran again from the very beginning. What if I add some annotations to my object? If I do um here, let's go to my annotations and I would say hello again and world. You see the whole reconcile loop runs again. The thing that I'm trying to tell you is whenever you make any changes in your um object in your custom resource, the whole reconciliation loop will run always. What if I maybe remove my label that I had added or remove the annotation? Say that again. You see running it again. Kubernetes does not differentiate whether it was a metadata change, whether it was a spec change. It does not do that. It just simply goes ahead and says okay you change the resource and this is the update. This is why when you make changes to sync like let's say your instance name or instance type the reconiler finds this. This is the beauty how a reconil would work. Whenever you make changes, let's say in your instance type, you make a change from T3 medium to T2 micro, the V consiler has no uh state. First of all, it does not remember that before it was T3 medium and now the user has asked for T2 medium. It does not remember the past request. It knows the status right now. I mean it's in the HCD. But in this case, let's say when you are going and when you are saying that uh my my type for the instance was T3 medium before and you change that to T2 medium. This before is stored in HCD. That is correct. But the reconciler loop that will run, it will have no idea that previously the user asked for a T3 medium. They're completely stateless. What the reconciler loop will now do is it gets your request. This is your logic that you would add that allows the user to change the spec for instance type or maybe you know the user can dynamically change the tags that they want to give. So here in this case if the user has made updates to the type uh key in in the in the YAML of the EC2 uh resource, it is your responsibility that goes to Amazon and sees if the instance of T3 medium was available and if it was you delete that and you create a T2 micro because you can't change the instance type as far as I remember. if you can that's on on Amazon site that's a different story but the reason what I'm telling you is your operator your controller the reconcile loop will not remember the past request it always has to check the current state is T3 medium the desired is T3 medium nothing needs to be done but if the current state in the cluster is T3 medium and the desired is T2 medium it goes to Amazon says okay this needs to go away and this needs to be in action and this is how you do selfhealing or eventually consistent and then you update the object which we will see in the next u sessions. So this is how you will be building an operator that knows how to watch the API server for your custom resource changes that knows how to watch the API server um and update the reconciliation logic in case there was some changes you change the object which we will see um and um yeah that was it. So this will be giving you a very good idea, a very beginner idea. I would not say beginner but it's a good enough idea for you to build your operators and then you run them on Kubernetes. Next thing that we going to be learning is I already have it available. This is going to be how we will be writing an operator which will be actually creating us an EC2 instance. The next parts of this video are going to be more onto how to use Amazon SDK in Golan to create an EC2 um instance on Amazon because we now know from Kubernetes point of view, from the operator point of view, we know how to write an operator, we know how to write the spec, how to install the custom resource definition and how to react on changes into our custom resource in the operator. Now it's about what do you do with that change. In my case, I'm just printing it. In the actual case of the course of this video, we will be building we will be using these changes and then we will be building them on top of Amazon. We will be creating an EC2 instance. So that is what we will be doing next. Till this point you know how to write your operator. You know you can get requests. So you know how you can you know the reconciler loop does it for you. So in the next part we will be using uh the kubernetes SDK in Golang to create us an easyto instance and then we will see if in case a request was successful we don't need to reconcile again we will talk about finalizes but that's all coming in the video. So let's look at how we can create uh EC2 instances with our operator uh using Golac. Okay, before we can actually get started for the code, there is something which is absolutely important for you to understand. We have been working with the reconciler loop and we talked about that the reconciler is the one that takes your request and runs it through a series of you know your logic and that's where you get your changes for the current state to be equal to the desired state. However, this reconciler is expected to return two values. One of them is the result and the other one is actually an error or it's going to be nil. These two return values are actually required by Kubernetes to know what needs to be done for your current reconciler request. So imagine your reconciler got a request here and you made some changes to your environment. You made some changes to your you know resources that you need to change and then you have to tell Kubernetes whether you want to re rerun the reconciler for the same request or you just want to wait for new requests. Uh wait for new requests. In this case, you did not get an any error. You did not return any error. Based on these values of the result and the error, that is when Kubernetes decides, do I need to rerun your existing request with the reconciler again? And this is how we work with things like selfhealing. If you know about this, you can give this an uh you know, give this the give this a try. Get yourself a pod that is in a pending state because of CPU or because of memory. Ask for resources that are not available in your cluster. The pod is going to be in a pending state. Then go ahead and add a new node that will be able to host that particular pod. And once that node is active and available, this pod gets from pending into the running state. You didn't have to do anything. You didn't have to tell Kubernetes that, hey, I got a new node. Please put my pending pod on this new node. It it didn't work that way. It was self-healing because when the first time the first time Kubernetes tries to put your pod to a node, it says, "Okay, there is no node available. I'm going to put this in a pending state." Think of this as a reconciler. So the decision were made that I'm going to put the pod in the pending state and the controller responsible foruling your pod returns a pending which is actually uh it sends an error that for the request that came to me I was not able to properly process it and there was an error and this is where kubernetes knows I have to retry again for that request and this is how self-healing works While Kubernetes was retrying and retrying and retrying with an exponential backoff, you happened to add a new node. And this is when once you added the new node, when the you know when the logic ran again, it was no longer pending. The reconciler said okay, you asked for eight CPUs and I have node now which is 20 CPU available, 20 cores that are available. I'm not sending any error rather I'm going to send a nil for an error that I did not get any error and the pod was scheduled and the pod then eventually went into a running state. This is something that Kubernetes does for you. And as a developer for this reconiler, it is absolutely your responsibility to tell Kubernetes whether your reconciler function was okay or did you get any error and would you like uh Kubernetes to actually retry that particular thing. This could happen for EC2 instances. Imagine when you tried to have your reconcile function and you were calling the AWS uh API to create an EC2 instance and you were not able to do that. You had the right credentials, you had the right access for AM for your user that you are using but maybe uh there was some network timeout happen or anything that could stop your request from processing um happened. you would like to retry again, right? Maybe after like 10 seconds or 20 seconds or whatever your time is, you would like to retry. In this case, you can tell Kubernetes that there was an error. My reconciler function is returning an error that please retry that again. And based on the requests and the error values, Kubernetes decides, do I need to retry this particular request or do I need to wait for new events or new updates to the custom resource for which this reconciler is listening on. So there's a very simple um uh condition that your reconciler can actually uh look into or look for and this is also in the priority order. If your reconciler function, if your reconciler is you know um is returning an error. So your error is present you are returning an error. This result is completely ignored. That means whatever you send in the result is completely ignored and you are then using an exponential backoff. A little bit about the result. What are you actually sending in this result is two things. First you are sending do you want to recue or not? Usually it's a it's it's a boolean where you can say I want to recue or not. And second you're sending a time for the recube. If you are sending an error if there is an error present in your reconciler this result is completely ignored and you will always be retrying. Kubernetes says okay the reconciler function or the reconciler is giving me an error that means it could not prop properly process the the request that came in I will retry this and this is where the self-healing uh loop comes out of the picture if in case you are uh not sending any error so and this is the second thing if you think okay everything is fine I have processed my request I'm not sending any errors and you do send a custom recue after. And this rec is actually this time rather I should have put this as um wait a second I can probably get a better color here. Um this should be rec after. This is the time after which your reconciler should again be uh started and this is like a forever running loop. So imagine this. You create an instance. You create the instance. It's okay. You probably want to check for the instances every 10 seconds or every 20 seconds. Maybe you are doing some sort of drift detection there. And if you were able to look for your instance, everything was fine, that means you are not having any errors with that instance. Um but you re you want to retry that again after 20 seconds and this is what you are sending. You are not sending any error because you did not have any errors. However, you are sending a fixed time. You are telling Kubernetes that there was no error in my request but I want you to rerun this reconiler every 20 seconds. And this is kind of like a forever running loop. It never stops because you don't have any errors but you always want to retry that again and again. You want to rerun this. What could be the reasons for it? I just told you maybe you want to do some sort of a drift detection. The third condition could happen if you are not sending any errors and you want to you know recue and your custom timeout is not set which is kind of similar that you have no errors and you also want to recue but you don't have any re uh recue after set that means you are asking Kubernetes that hey my reconciler was okay I want you to retry again, but I'm not telling you in what frequency do you have to try. It's kind of like similar to level two, which in this case you're also not sending any errors, but you are telling how frequent do you want to try. In this case, you're also not sending any errors, but you're not also telling Kubernetes um how frequent do you want to try. You are letting this with Kubernetes and this is going to be the exponential backoff. This is where cubernetes will say okay the user said there is no error for the reconcile loop. The function was running properly fine but they are not asking me to run this in a forever loop. I would probably uh I'm going to use an exponential backup. So it will run your request and then maybe another time it takes 2 milliseconds. The next time it's going to take 4 milliseconds. The next time it's going to take uh probably 16 milliseconds or so. And this is going to be an exponential backoff until I think the maximum limit is 1,000 milliseconds until then it stops doing it. And the last condition that you can return for your reconil is you do not have any errors and you also did not send any rec flags. Probably you just said result result was empty and then you are sending a nil. You are returning a nil. This is where Kubernetes says okay everything was fine I'm not doing anything I'll just wait for a new update or I will wait for a new event where the custom resource has been updated kind of like for the new requests here this is absolutely critical for you to understand otherwise you might see your reconciler making changes again and again or you might see your reconciler running again and again because you did not send the right set of values you did not put the right return values for the reconcile and for kubernetes to understand what to do now as as I was talking about once this is understood uh I was talking about we will be looking into the go code so let's take a look here and let me get that here so in your screen you can see that I've made some changes to our um our instance spec before this this before u Now it was a very simple one. It was just having an instance type, an AMI ID, probably a key pair and a security group. But when you want to make things more robust and when you want to make things more production ready, you have to think from an overall point of view. When you want to create an EC2 instance, there could be many things that you have to give. You definitely have to give the instance type whether you want to use a T2 micro, T3 micro or any other instance family. Then you absolutely have to give an AMI ID which is going to be the the AMI ID you want to use. You have to give the region as well under which your instance should be created. You need to give the availability zone. You have to give the key pair so that you can log into the instance. You need to give the list of security groups around here. the subnets in which your instance could be running and also when you want to provision the machines as soon as they boot up with your changes we usually use Amazon's user data and uh that also is what you can give you can probably give tags as well you can also give some storage you have to give storage and whether you want the instance to have a public IP or not this is kind of like a boolean that you can give now on the right side you can see this omit empty. This is actually that uh a place where you can control what kind of fields in a YAML when you give your EC2 instance spec are optional or what kind of fields are required. For example, tags could be optional. User data is optional but storage is absolutely needed. AMI ID is absolutely needed. Instance type is required. So this omit empty lets people define the only important or the required fields otherwise the other ones could just be skipped. So here you can see I have a storage which is type of a new strct called storage config and here's a new strct called storage config where we define a root volume and then we can also probably give some additional volumes as well. This is an example where you give your root disk as 100 gigs and maybe you want a VM for a database. You can add a bigger disk in the instance and this could be done by the additional volume and both of them are of type volume config and a list of volume config because additional volume itself is a list of additional disks that you can add to your instance. And this is a very simple volume config where you define the size of the disk. You define the type of that disk, the device name which is going to be available in the instance when you mount it or attach that and the encrypted uh boolean if in case you want the device if in case you want the disk to be encrypted or not because Amazon's uh allows you to encrypt your discs in case you want that. So this think about the EC2 instance as a more holistic approach whether you want to give or you want to allow the users to be able to declare their um set of set of data and the metadata. In this case, you are allowing the developers to create an EC2 instance, not just create one, but also you are letting them login with their key pair and you are also allowing them to use their user data that you can, you know, give to Amazon when you are creating the instance that lets it preconfigure before they can even login and the VMs are exactly how they want it to be. So this was a bit of a change in uh our EC2 instance spec to make it more production ready to make it more not from development but actually to production. I also made some changes to the status where when you do cubectl get EC2 instance you will see the spec and also you will see the status. So in the status I would like to see the instance ID so it is easier for people to see what is available on Amazon and what your Kubernetes knows about the state of that instance if it is running if it's terminated it is unknown it is stopped all those Amazon EC2 instance states and also a very important thing is going to be the public IP because when I do cubectl get EC2 instance I should have enough that lets me log to to this public uh to the to the particular instance and this is a public IP and that's what I want to show when someone does an EC2 uh uh cubectl get EC2 instances and then again we have the standard strct of our EC2 instance which contains the type metadata the object metadata and our spec and status and this is kind of like just when you get a list of instances what's going to happen and this is how Kubernetes knows uh what is a set instance would look like for you. Now I've already made the changes and I told you whenever you make make the changes you have to run the make manifests command and then you have to install that to your Kubernetes cluster. So my Kubernetes cluster already has this custom resource definition. If I do cubectl get EC2 or CRD which is EC2 instances.computee.cloud.com cloud.com - OAML and let's look at this. You can see the name is is easy to instance the list kind the plural the singular it is a namespace scope resource and there you can see I have got couple of things such as the kind and there's my spec I have got the AMI ID the associate boolean a public IP or not it's a boolean the availability zone I want to run my instances on and things like my security group which is type of an array because you can give mult multiple security groups and then I've got my storage configuration where I give one root volume and I have got additional volumes which is type of an object which is then again uh globally it's a type of an array so you can give multiple additional volumes but you can only have one root device you can only have one root um block device now once we have uh defined our spec properly there is going to be Now some uh code that actually uses this and creates you an EC2 instance. So let's see that. Um once I have my instance type I can actually go to my u E2 controller and this is where everything starts. This is where you will be seeing the reconcile loop. We saw this before. We use the reconiler to actually see uh what happens when I get a request. And this is what your to-do list is. my logic starts and I have created a logger for this context that is aware of the context and you can use l.info which is going to just print stuff when you are running your operator. So it makes it more verose and you can see what is going to be uh what's going to happen or what is happening with your controller. It prints out a function uh it prints out uh an info message that the reconsidered loop has started and this is the name space under which you got a request. So rick or rq is the request that comes to your compiler and then you send a result back to kubernetes. So it came from this name space and the name of the request was uh request.name. That's the name of the object that uh we are uh we are working with. Then there are some comments which I was building this. I put some comments for us to be easily understanding this again. But you know what we are doing? We are creating a variable of type EC2 instance so that we can marshall the object which is coming to us in this reconciled loop by Kubernetes into a variable and then we can easily iterate over on top of that. We get the object uh into uh our E2 instance variable from this name space and if you could not get the object and this is absolutely very important. See, you may have any problems uh for getting the object. Maybe you have a wrong YAML. Maybe you probably were supposed to give a string, but you give a boolean to one of the keys. Or you probably did cubectl delete the object. That's correct. Even if you delete the object, it is an update to the custom resource. Then again this reconciler is going to be started and you have to check if the error that you got when you are trying to get the object was is not found. This is one of the errors from Kubernetes. Kubernetes has a package called errors. And let me show you here and here you can see it has all these errors defined for you. So it makes it easier for you to declare what was the error in my case. See sometimes when you create an object it gives you the object already exists. It's an error. But you can actually see what kind of error it was because if I was just set if I was just saying get me the object and if error is not equal to nil I would just say okay there was an error please try again. But the user will never know what the error was. In this case, I can say, "Hey, you know what? I was trying to get your object. I was trying to get it into my variable, but I got an error while trying to get it and the error was is not found." And that is where it returns a true if the condition was that I could not find the object. This is probably when you are deleting the object. Um, it again runs the reconciler. It looks something like this. you have uh the object here. Whatever change you make on this, whatever change you make on this, the reconciler will be running again. So the change could be you added an annotation. That's an update. Then the reconiler the change could be you added a label on top of that to the object again the reconiler would be running. It is your responsibility to write this reconciler in a way that if it surely should not be changing anything. If the change that you did to the object doesn't require a change, it should not be changing the actual resources. For example, your object could be EC2 instance. Maybe on this Kubernetes object, you want it to give a label. That doesn't mean you have to change something external to the Amazon instance. That doesn't that should not happen. So this is something you have to code in your reconciler. Even when you say cubectl delete when you say delete the object is deleted there was a change on the object and then another uh run of the reconciler would happen. So you have to check that when you were trying to get your object you could not get that and there was an error and the error was actually is not found it will simply say um the object does not exist or there's no need to reconcile because the object was deleted and then you just return an empty result and a nil. Remember this is one of the return types that you have to say. What you're telling Kubernetes is everything is fine. There was no error from my side because the object does not exist in in our case and please wait for the new requests which are coming to the reconider. This one request that came in is all good. If you could not get the error for any other reason then it is not found. Maybe uh you did not have write um rback to get that to get the object in that name space. For whatever reason you could not get the object of the request, you will then say uh you will send an error. And you see here the moment you send an error you are telling Kubernetes please retry this object. Uh please retry running this reconiler please retry the operation of the whole reconciler loop. And this is where the self-healing would work. Probably you had some problems where you could not get um you could not get the you know the object but you try again and it if it works then you're happy because you not you don't have any errors anymore otherwise you send an error again and this is going to go ahead with a exponential backoff. So it's like when the first request comes in, it was an error but it was not an is not found error. You send uh you return an error. Okay, it goes back to the reconciler again, runs to the reconciler, there was another um there was an error again which was returned goes back to the reconciler and this happens with an exponential backoff. This is where return values of the reconiler function are absolutely critical. Absolutely critical. Now the next thing is um whenever you delete an object you set a deletion timestamp and I will talk to uh you about this in in the future. We are not going to talk about this right now. This is when you delete the object. We will first learn how to create one and then we will delete one. And this also is about deletion. This is the logic of deletion. We'll talk about that later. And I'll tell you why this is here. This is the logic of checking if the instance is already there because you want to be at the potent. You don't want to create the same instance with the same instance ID if it is already there. And this is also the logic which I probably would talk to you about later. This also is a logic which I will probably talk to you about later. Um so here is where we start in our loop. The first thing you do is you start your reconider. You create an object. You try to get the object into your EC2 instance variable the custom variable type that you have created. And then you say okay I'm starting completely new. I have no ID. I have no instances on my Amazon and I'm going to create a new instance. The first thing you do when you create an instance is or when you create an object, it's a very good idea to add a finalizer. You might have seen this in Kubernetes. When you do cubectl get- or YAML, you might see this finalizer in the metadata texture of your object. What this finalizer actually does is is very it's very interesting. So let's say when you created this object in Kubernetes which was an EC2 instance and there you added a finalizer. Finalizer is nothing but it's a list of key value pairs that you can add. Let's say I add my finalizer as hello colon word. Then this object was created and your reconciler actually went to Amazon and give you a new uh AWS instance. All right, everything is happy. You got the instance. Now the the thing that happens with finalizer is when you say I want to delete this object when you say I don't need this instance anymore I want to delete this particular object you can delete this from Kubernetes however that's not the only thing where you need to delete it from you also have to delete this from Amazon so how do you tell Kubernetes that why I am deleting this from an external resource from an external platform. Do not delete this object from Kubernetes. Only when this instance is completely gone from here can only you delete this particular object. That is the role of a finalizer. Finalizers will hold the deletion will hold the deletion of the object in Kubernetes until the actual cleanup has happened. And once you have you know um once you have deleted the object you then remove the finalizer and then Kubernetes will allow you to delete this particular uh EC2 instance Kubernetes object that we have created as a good practice when you create the object in Kubernetes that's where you should add the finalizer and this is extremely important this finalizer is being added to your Kubernetes object. object which is EC2 instance and this also is an update and this is also going to rerun the reconciled loop any update to the object whether you are adding a label whether you are adding a metadata whether you're adding anything really whether you are updating the status of the object in your code that will recon uh that will start a new reconsidered loop so this is where you have to be very careful of a depotency uh in your code and you see what happens. We we print a message and we say I'm about to add a finalizer and I use this um this append function because it's just a key value pair and adding in my EC2 instance finalizer because I've already mastered this using the R.get get my EC2 instance has actually the YAML of the request that was given to me and I'm creating uh a key in here called finalizer and I'm appending uh my finalizers here in uh called EC2 instance uh EC2 instance.comput.example.com. I'll show you how it looks. It just creates a new key under your object in your object and then it just um adds this uh this as a as as a list uh there as an entry in the list because you can have multiple finalizers uh in your object. Once you have declared that I want to add a finalizer the actual way of updating your object is going to be r.update reconciler.update. There are a couple of functions we get with the reconciler. Get lets us get the object of the uh you know get gets uh the get function lets us get the yaml of the object into our variable and then you can also use r.update. This lets you update the object that the reconider is working on right now. So in this case I'm updating my EC2 instance where I'm actually creating and adding some finalizers and we will see this when you create the instance it will actually give you the finalizers when you uh as soon as you create the the instance and because you made an update on the object it will start a reconciler again. Not right now though this is very important. See it's very important to uh to remember when you have the reconciler let's say when you have the reconciler it starts let's say here you made an update to the object maybe you updated the annotations this is where you will start another reconciler but not right now you will move ahead in your code and you probably make another update in this case you updated the labels of your uh reconciler of your object. Kubernetes also records this as a second time it has to run. Then you do a return and you do a null. What's going to happen is Kubernetes will run this reconiler twice because you made updates to the object twice. It's kind of like it remembers that this is where an update was. I have to rerun the reconciler. This is what an update was. I have to rerun the reconciler. It is a golden rule of reconilers that any update to the custom resource whether it was done by you with cubectl commands or whether your reconil is doing it will start another reconciler loop. It will not stop u you know the current reconciler. It's not like it got an update. it will directly go from here. It's not like that. It will finish the proper execution and then based on how many updates did you make, this is where the reconired loop is going to run again. And this is your responsibility to make sure that um you know when you run this again, this update does not happen. This label does not happen because they're already there. And then you will say, okay, I did not make any changes throughout my reconciler on this object. um I I need to do nothing. I don't need to start the reconider again for this particular uh custom resource. If you make a new custom resource then again yes the reconil will be started and the loop will keep on running. Extremely extremely important to know about the return types of the reconiler. Now once I have uh updated you know um my object I'm telling Kubernetes please add this finalizer to my object and if I got an error that said failed to add finalizer actually if I got any error um I'm printing an error that says hey I was not able to add the finalizer and I say please recue and I'm sending an error. This is where you are returning another uh return type. And you see whenever you get an error whether you are trying to get the object, whether you're trying to delete the object, whether you're trying to update the object, you want to retry again. And this is where you will send uh you will return an error. And whenever you return an error, this whole result is completely ignored. It's absolutely absolutely important to to understand this. when you are returning or when your reconciler you see here this reconciler function is returning two values one is the result the second one is an error if you return an error the result is completely forgotten kubernetes says you know what the reconciler had an error I'm going to retry it again with an exponential backoff and this is where this is how the self-healing works in Kubernetes Now once you have added the finalizer I'm just printing an info message for info log which says finalizer was added. This updates a new reconider loop execution but the current will continue and this is where we create an EC2 instance. I'm just sending a log uh I'm just printing a log continuing with the EC2 instance and the current reconciler. And this is the beauty what we were waiting for. This is what's happening when we want to write an operator that talks to uh that that creates mere Kubernetes cluster guys. This is where it all comes down to. We have our spec of the custom resource. We have the you know the logic that listens on the update of our custom resource. We have the logic to get the manifest or think of this as this way to get the YAML of what the user has given in my EC2 instance. Now I need to create an EC2 instance. This is absolutely important. Now it's going to be so much fun. Now when you want to create an Amazon instance, you know what you need. You absolutely need an Amazon account and you need to have the credentials. You need to give the credentials to your operator or to your controller so it can go on your behalf and work on Amazon. And this is exactly what we were doing. Before we can actually go ahead and create an instance, we need to figure out the authentication then we will use a client that we created using this authentication and we will give it this particular YAML and we're going to ask it to go ahead and give me an instance on Amazon. And this is exactly what's happening now in this create EC2 instance function. So I've created an EC2 instance. I've created a function which is called create EC2 instance and I pass the users requested YAML. I pass the user manifest the EC2 instance and let's see what this function actually does. This function which is the create EC2 instance. It is accepting a value of type EC2 instance which is the whole YAML from the user and it is returning two things. First, it returns another um it is returning a variable. It's returning a type of created instance info. See, when you create an EC2 instance, you get a lot of output. You get a lot of data. But we don't want all of that. We only want to um you know when a user creates an instance when a user creates this instance probably they care about the state whether the state is running or not. They care about um you know uh created or not true or false. They also care about the public IP that was it you know um created of was it there or not do I have a public IP or not and for this information I have created a new strct which looks something like this created instance info and this helps me to send back the data from my create instance function what I'm sending back is I can send an instance ID which is important so people can know what instance ids are there on Amazon. uh using cubectl get I'm also sending a public IP I can send a private IP a public DNS private DNS the state that is all I can send from my function and this is the um this is the return type and my function which is going to create me the EC2 instance it is returning two things first is a strct which is the information of the created instance and second is an error um which is I probably could tell the user that I tried to create the instance but there was an error. Maybe the authentication was a was a problem. Maybe you don't have enough quota in your region in your account. I I want to send I want to uh send them something so they are aware of what has really happened why the request failed to create the instance. So we create a logger which is create AC2 instance. This is good. You can have your logs with custom uh log name and this is easier for you to know which file which function created this particular log entry when you do cubectl logs in your operator. Um so I'm putting up a info which I'm saying I'm starting the EC2 instance creation. This is going to be the AMI ID. I'm going to use what the user has given from the spec. This is the instance type and this is the region under which I'm going to create my instance. So the first thing I have to do is to create an EC2 instance client guys. Now it has nothing to do with Kubernetes. It is completely how you create an Amazon instance in Goland. It has nothing to do with Kubernetes because you already have the instance YAML or rather the instance info because EC2 instance is the whole instance that should be created. Now you uh are doing the generic things on how to create instances on Amazon. The first thing you do is you create the EC2 instance client with this AWS client function. What this is essentially doing is it's reading the AWS access key and the secret key from your OS environment variable. You see because you need to give some sort of authentication on how you would talk to Amazon. I'm using the AWS key and the access key and then I'm using the config from Amazon to load the default config with these credentials that I have given. Um and then if I could not create the config, I return an error. Otherwise, if you have the access key and the secret key, um you are able to create a config and then you are returning a new config. Think of this as this function is just returning an EC2 client. And this is absolutely important. You would use this client to talk to Amazon. So you use your access key and your secret key to create an EC2 client. And then this EC2 client is here. Till this you have the authentication to Amazon. Till this 24 you have the authentication to Amazon. Now you need to say hey Amazon please create me an instance with this key in this subnet. This is the minimum count. This is the maximum number of instances I want of of such. Um this is going to be the instance type that I want you to create. And this is going to be the image AMI ID I want to use. These are the input instance parameters. You are creating an instance. Amazon expects you to give certain instance inputs and these are um one of them. There are many other instance inputs that you can give. If I show you, you can tell Amazon what is the maximum count of instances you want, what is the minimum count of instances you need. Any u block device mappings you have, you probably can say what is the capacity reservation specifications, the CPU options you can give, is it a dry run or not, is it a EBS optimizer or not. Um so there are many many different uh options you can give when you are creating Amazon. This is just something when you create Amazon instance, this is the information you give. You can also give security group ids if in case you want the security group to be used for creating this instance. I'm just keeping it very simple so that we know what's really happening. We are creating our query to create an EC2 instance with these inputs. And once I have my input declared, I'm using my client EC2 client which I created above. And there's a function called run instances. And this is the function from go uh SDK of Kubernet of Amazon that launches the specified number of instances using the AMI for which you have the permissions for and this is absolutely the one that has creating the instance for you. So, so far you created the client, you created the instance uh input and now you have created your actual instance here. Now, for whatever reason, this run instance is going to be returning uh two things. First, it returns the actual output. See, I told you when you create the instance, you get a lot of output. So, this is what's going to be returned. So if you look into this EC2 instance dot uh EC2 run instance output. If I look that on Goland uh here you can see I'm going to look for the run instance output. Run instance output here. And this is what is being returned to you. You are returned. Where did that go? Whoa. Whoa. Whoa. I think I was a bit quick there. Let's wait. Uh run instance output here. So you see you are returned the act the the groups you returned the instances that were created and this is the metadata. There's something wrong with my browser. Quick. Essentially what you are given is the what you are given is uh where did I go? Return instance output. Uh there's a type. So you get the instances and this is where you have the instance metadata. What is the primary uh what is the private IP? What is the instance um ID that you that was created for you? what region it was running in, what zone it was running in. So think of that as a metadata of your instance when you created that and that's what we are um saving in the result. If in case there was any error because this run instance does return an error as well, you will say I failed to create the EC2 instance and then you return the error back to the main program and you say this was the actual error because of which I could not create the instance. There could be many reasons why you could not make one. Um perhaps you did not have the permissions in that region. Perhaps you did not have quotas in that region. Perhaps you used a wrong AMI ID which doesn't exist. um could be a typo or anything. You just uh are returning this to the user. It's a good thing to return them the reason why uh it failed. So that's what you're checking. If the instances returned is zero, you will just say um uh there were no instances returned for me. And till here if we have no error, we have an instance for ourselves. And this is what has happened so far. our code. We had the client and then we use the run instance function to actually create an instance and it gave us some output back. Till this time this output contains things like uh the state that was you know when the instance was created at that time what was the region the metadata the private IP the private DNS um DNS name by this time there is no public IP however there's one thing important second when you make an API call to Amazon with this runin instance function. What you essentially got back in the output is the state of the instance at the time when AWS received that request. It might not be running. You know how when you create an EC2 instance, it goes into uh pending, creating, initializing, then it eventually goes into the running state. At this time you have an instance created for you but it might not be in the running state. It might take some time for the instance to be in the running state. And this is what you want. You want to wait until the VM is running. So when you uh run when you execute the run instance function it creates the instance and gives you back the metadata. What it does not have however is the public IP and the state whether the state is running or not. So it's kind of like you say hey Amazon create me the instance. Amazon says cool I will give you one here's some metadata but you don't get the public IP and you might not be in a running state when you created this actual instance but this is what you say if I got back in my result because you see this one instances it gives you back the list of instances and you are checking if um if there was you know uh an actual instance where I could I did not have any errors and there was an instance created, you will just send an info that says okay, I was able to create the instance successfully and this is the instance ID that was returned to me. You store the result of uh you store the output that was given to you and uh there you can access things like instance ID uh private IP public IP because it's all returned for uh returned by Amazon to you. And now we wait for the instance to be running. See, it's a good idea that you created the instance, but it's not like imagine this. Uh there's this developer and he goes to you and says, "Can you give me an EC2 instance?" And you go to Amazon and you say, "Please create me the instance." And you get back the p the private IP. in your company you guys are using a bastion host which is available and using this bastion you can talk to the uh VM which has a private IP because you might not have a public IP you might have disabled the public IP so essentially what happened the guy asked for a VM you said hey Amazon create me a VM and you got the private IP and you gave it to to him you never waited to see if the instance was actually in the running state or not. Maybe the instance was created but it never reached running. Maybe there were some problems in the region of Amazon or maybe the instance malfunction. Whatever could have happened you you were not waiting for the instance to be running. You gave it to him and he logs in to the bastion only to find out that this instance is not running. So he cannot use it or she cannot use it. And that's where the problem is. As an operator, it is your responsibility that you create that particular resource. You create the instance and you wait for it to be in a certain state that you want. In our case, it is running. So what I could have done is I could have had a for loop. I could have had a for loop that keeps polling Amazon. Hey, is this instance now in running? Is it now in running? Is it now in running? every maybe 5 seconds. That's also doable. Think of this as um a while through and I would say think it like this. So I'm using a while loop check the instance and that's it. This is kind of like my function. It keeps running. I am giving it an input to describe me the instance and I'm describing the instance with this function and I get some responses back. If the state name is running, think of this as a pseudo code. Um, then you break otherwise you keep running. So you keep running every every 5 seconds or 10 seconds. That is a doable option. But it's not a good idea. It's not a good idea because Amazon gives you these waiters that can do this for you more gracefully. A waiter is nothing but it's a construct from the Golang uh from the go package of Amazon that waits for a certain time um for a certain state to be reached of the instance. In my case, there's a new instance running waiter. If I go to that and if I show you or probably even here, um there is going to be a new instance uh running waiter. Now what this does what this does is it defines a waiter for instance one. This one actually has the logic to wait for the instance to be in the ready state. It does the polling more efficiently compared to me having this writing in my logic uh in my code. So you can define a waiter which is going to wait for the instance to be reaching the running state. You can also give the maximum time for which you want to wait because it's not like if Amazon takes forever for your instance to be created. Um you just say um the you know the checking loop will keep running forever. You have to give some feedback to the user and typically you can give the run max time which is going to be time dot minute and three. So you're giving three minutes that you want to wait for the instance to be reaching the running state. This could be depending on your uh requirement. You can you can increase this or decrease this. Every request that you make this waiter will be exponentially uh backing off. So it's it's kind of like it starts from like every 10 seconds and then uh it increases this timeout up to your uh given time. So it does it a lot better. You create a waiter and then you use the wait function to ask it to wait on this instance ID. So you are telling this waiter that please wait for this instance for this maximum time for it to be reaching the running state and by this time if it was not reached the running state there would be an error and if the error was not equal to nil but this time you will just say failed to wait for instance to be running and the instance could not reach the running state in 3 minutes. Now this is important. You do say the maximum time for your instance or that you want to wait is 3 minutes. However, if the instance has reached the running state in the first 30 seconds, then the loop will stop. It's not like you're going to wait for 3 minutes dedicated even if the instance reached the running state before. It's not going to happen like that. This is why the waiters are quite interesting. They have the logic for it. So you don't have to deal with with that. This only comes from uh experience when you are using the SDK. These are the things that you can also Google. How do I make my code more efficient? How do I use waiting? And you will get that. Now by this time when the instance has been created, we get the remaining uh information back. We got the state because we were waiting for it to be running and we only break our loop when the running state is there within 3 minutes of course. Now by this time Amazon has also associated your instance a public IP. three minutes are good enough for Amazon to give your instance a public IP and there then um what you can do so actually um okay this was a bit wrong by this time we are just waiting for the instance to be running we don't have the public IP yet this is where I probably skipped ahead when you are using a veator you were only waiting for this instance to be running and once the instance is running, Amazon will give you a public IP. So you have it running but you don't have the public IP yet because it was not given to you in the output when you created the instance. This is where you will use another Amazon function describe instance. Now you say okay I created the instance I waited for that to be in the running state. Now I am describing this particular instance. Now I'm going to get my public IP as well. Of course given if you have uh the public IP allowed in your Amazon account and this is where another request happens to Amazon. So we waited for it to be in the running state within 3 minutes. And then I'm saying calling Amazon describe instance API to give the instance details. I tell Amazon that I want to describe an instance whose ID is what I got when you created this instance for me and uh I want to store this result which is of the describe instance um function and I want to store this into a describe result variable. If I could not describe the instance again this is a very simple go check you will say I failed to describe the instance whatever reason you are having you just give the instance or if you could describe it your result is going to be in the describe result which is again a type of describe instance output. Now when you describe the instance you get a strct back from Amazon. You get some data in a in a specific strct which we can see it here. You do get some output with describe instances. Wait for that. And this is the describe instance output. What you get is you get the information about your reservations which is the instances on Amazon. And within these reservations you have the instance information. So if you look at the reservation there you have the instances that were described for you. So you can call the reservation because I know I only created one instance. So it's only going to have one element in the list because the reservation is a list of reservations. And for these instances which is only one the public DNS name I uh is going to be describing like this. So I print the public IP and then I say the state is state dot name. Again this is returned by the instance strct of the golab because if I show you here if I go to instance it will have public DNS. Let's look for that. There you go. The instance strct is returning a public DNS. It is returning me a public IP address and also it returns me the state. Um here the state is of type instance state where we also have an instance state name. So it's kind of like they have created packages for all the other one and here you see you have the name. So you have asked for I want to describe my instance and the input is this instance ID. I store the result. I store all the reservations that was returned to me by Amazon and all the instances inside of it. I know I only have one instance. So I can call it with zero index and tell me the public DNS name and tell me the state of the of the VM. Now here's interesting thing. By now you have all the information you need for your EC2 instance. You've got the private IP, you've got the public IP, you've got the instance state, you've got the name of the instance that was created, the key name that was using and now you can actually um um so so by this you have uh all the resources that you need for your Amazon VM uh to be to to be given to the developer who has asked for this instance. The next thing that you can do is you can get the information uh about the instance. But this way this thing is not needed because we already um uh we already requested the the instance information. That's not what we are doing. However, this is what we are returning back. This is extremely important. See when you have all the information about your instance that you have created we want this to be returned back to the actual controller so that this is where we have the instance information. This function is returning me um a type which is created instance info and I just showed you the created instance info here. Where did that go in my API? This one. So here's a strct uh which is created instance info and that is what my function should be returning. This create EC2 instance should be returning and this is what I'm preparing now. So I got all my instance information in a variable called instance from the described result and then I'm preparing my return type because you know this function is returning two things. First is an error if there was any and second is the created instance info which contains the public IP, the private IP uh which contains a public DNS, private DNS, the state and the instance ID and this is what I've prepared now. So once this uh is done we will simply say uh I have created my EC2 instance and there I'm returning my um my return types because I did not have any errors when creating this instance. I'm returning a nil and I'm returning the information of my instance which was created. What might be interesting is uh this function called drefer string uh dreer string. What this does is it is actually just dreferencing my pointer. The reason why is we are dreferencing this pointer is because when you talk to Amazon SDK, it is returning you things like public IP address uh which might not be available at that time. So when you're returning this, it might be a uh it is a pointer type but you are returning a nil pointer and that's a problem. So this is this is important that we are able to distinguish between whether it was an empty string or whether it was a nil value. If it was indeed a nil value and you were trying to dreference a nil pointer, that's going to be a problem. And this is essentially why we waited for so long for the instance to actually have a public IP. This dreference function just dreferences my string to return a string which I can give back to my main function. And by this time and by this time I have an EC2 instance that was created for me. Now the create EC2 instance function doesn't just create me an instance on Amazon. It it does return me two different values as well. one is an error which is a good idea that your function does return an error if there was any or it returns a nil so that you can use that error in the further steps. For example, we are using it here to say if there was an error, we want to put that error as a log output of our reconciler. So when people are looking at the logs of our application which is a reconciler in this case they will know why there was an error which you know which stopped you from creating an EC2 instance and then you can use this error to be sent as a reconcilers's return value because you remember reconciler uh here it is expected to return two things first is the result of the reconciling function which is the current reconciler and then if there was any error with that reconciler. Now it depends how you are creating uh you know um your your reconciler. Maybe you want to retry after waiting some time you want to retry creating that that easy to instance. And this is why uh you can return an error within the return function. What you're essentially telling Kubernetes is I tried to do an an operation which was in my case was to create an EC2 instance and I could not do that. Whatever the problem was, I want you to take some time and retry that process again. Retry that function again. And this is where Kubernetes will say, okay, I'm going to retry running that reconciler loop. So, I'm going to retry to create that EC2 instance for you. And this is kind of like being done in an exponential backoff. It tries, it fails, it waits a little time, it tries again. If it fails again, it waits a bit longer. And this is how the Kubernetes will be doing its uh exponential backoff with your request. So you're not getting rate limited uh by you know um by Amazon that you keep trying and asking for an EC2 instance every 2 minutes or 3 minutes or whatever your uh recon is it waits during the time and it uh it's an exponential backoff. Now once you have the EC2 instance once you were able to create the EC2 instance I'm returning the info as well which is if you remember it's a strct that we created uh probably somewhere around here. This is the strct that we created and this is the information I want from my create EC2 instance function because this is something I want to give to my users when they do a cubectl get EC2 instance. They should know the instance ID and most importa