Automated testing using GitHub Actions... easier than I thought

Key Takeaways

I honestly didn't think it would be that easy I almost didn't listen to you guys like thank goodness I read my comments so if anything just let this video kind of be a reminder to you that you should properly evaluate certain things like actually give it like some time before you decide that that's not an appropriate solution for you all right what's up guys my name is the Cho and I'm building a game engine that's going to be my intro probably from now on so for those of you a little bit out of the loop I have been setting up like some kind of automated testing situation for hazel the game mention that I'm building because I want to reduce some of the pressures of testing from the team and I'll have the previous video linked up there suggest you check it out because it explains a lot more about the context of what's going on but the goal at the moment is that every time someone commits some code anyone on the team commits some code to Hazel pushes some new code we want to like run that code we want to make sure that it compiles on all like build configurations all platforms that kind of thing and then we want to also like run the engine test it and make sure that that code hasn't broken anything it's obviously very useful if that just happens automatically behind the scenes any time new code is pushed and once that kind of infrastructure is in place it can extend a little bit more so for example if someone's proposing code changes in the form of like a pull request that can actually be evaluated first to make sure it integrates nicely and all the tests pass and the code compiles and all of that stuff before a human even needs to like dedicate time and read that code and make sure that it works kind of using their own eyes so in summary it's all about reducing the need for human resources and in that last video I was discussing my thought process of how we would even integrate a system like this and there was something called GitHub actions which I kind of took a brief look at but then ultimately decided that seems too complicated like I got to set up all these like scripts and install stuff on my computer why don't I just write a simple python script that just pulls P the repository sees that there are new commits on a particular Branch pulls the code down you know runs it that kind of stuff and that was basically what I had settled on in fact Peter and I even talked about it and we decided that this was just going to be so much easier why wouldn't we just go down this Roots it it just sounds like setting that up would just be more complex why wouldn't we just do it this way and then the next morning I woke up and what tends to happen sometimes actually probably more often than I'd like to admit I'm lying aake in bed at night and I'm just almost almost subconsciously thinking about stuff and I was thinking about Hazel and all of these thoughts were kind of coming into my head and I was like but you know it would be really nice to use GitHub actions and everyone in the comments like seems to be telling me that it's not that hard like are they really all wrong like wouldn't it be nice to integrate this better with GitHub so that for example anyone on the team can easily view logs and build statuses and and we would get that kind of direct communication between GitHub the repository and the actual like Ci machine this guy over here it's obviously just a more robust system would it really be that difficult to set up no no turns out it's really easy turns out it's really easy I don't know I think sometimes it's easy to kind of be lazy and assume that oh whatever that's too difficult I'll just do it the way that I know how to do it without daring to venture in a little bit deeper and potentially learn something new like I'm definitely guilty of that so if anything just let this video kind of be a reminder to you that you should properly evaluate certain things like actually give it like some time before you decide that that's not an appropriate solution for you so yesterday Friday morning I just moted up the qap and I was like I'm going to give GitHub actions a fgo like I'm going to spend like at least the whole morning going through and trying to set this up because ultimately like I'm aware of all the benefits that it provides it's just that I was a little bit scared of the kind of upfront cost of setting it up I guess but then also deploying it to more and more PCS that we would have to do we'd have to get more devices to test Hazel and I was a little bit worried that it might not be as configurable as easy to spin up a new pc and get it to run all of these things and compile the code but no I think it's going to completely satisfy all of our needs and goals it is in fact very flexible and very configurable and so the rest of this video I'm going to talk about how I set this up yesterday and how it's currently functioning and our plans for it in the future as well so before we jump on this computer and I show you in depth everything the main kind of takeaway here is that this is just a self-hosted runner and the process is really simple your repository just needs some kind of workflow file and that workflow file just defines like I guess what actions need to be taken to complete that workflow and so once you kind of have literally a series of steps of what you want it to do as well as when to do it so for example when there's a push event when someone pushes code into the repository I would like to complete the following steps once you have that you then need to get something like some form of Runner or you know device to actually kind of listen for those events and then execute your steps for you now that can be either a GitHub hosted Runner or it can be a self-hosted runner we went for a self-hosted runner because it made a lot more sense first of all it's completely free because you're the one providing the hardware not GitHub so you're not paying like essentially a price per minute of that kind of Hardware being allocated for you by GitHub there are free minutes included of course but that's kind of besides the point because we know that we want to control the hardware we have the hardware and we need it to be a little bit more customizable than perhaps GitHub can offer us because what we're testing here is a game engine we want it to be able to run with specific gpus and drivers and kind of record frame rates and the environment definitely needs to be controlled by us so of course we went the self-hosted runner route I did by the way mention this in the previous video I I added a little note there being like yeah we could do a self-hosted runner but it'd probably be more trouble than it's worth if this is the computer that needs to run that stuff then uh I guess GitHub actions goes out the window at least as far as I'm aware I don't know maybe I'm missing something that's always possible you guys can always let me know I was wrong it happens we all make mistakes I'm just glad I was able to kind of fix this quickly and early so anyway for a self-hosted runner all you need to do is so step one here we have the Hazel repository this is the one you can get on patreon.com theano if you help support the project this is the full version of Hazel and by the way once you do sign up on patreon all you have to do is go to get. haas.com connect your patreon and GitHub accounts and you will automatically get invited into this repository so it's a really kind of instant process now back in the day you had to message me on patreon wait like up to a week potentially for me to give you access doesn't work like that anymore so inside this repository if you go to settings this is really like step one if you want to do this your set self if you go down here to actions you have to basically enable actions so disable actions was kind of the default for this repository I had to not only allow Studio Cho actions and reusable workflows which is what I thought you had to do because this says that any action or reusable workflow defined in a repository within studio cha can be used but then I was actually getting errors saying that hey you're using stuff that isn't in studio cha which didn't make sense to me because I was like no no the workflow file which I'll show you in a minute here was inside a studio Channel repository but it's because I was using actions that were kind of on the marketplace as it's called basically I was using an action at that time called checkout which just clones the repository you think that was like just a normal thing but that action I guess technically isn't my action I'm kind of importing it as like an external package almost and so for that you need to allow non Studio Cher actions that are either created by GitHub or allow actions by Marketplace verified creators so that needed to be done before it would actually run my kind of build action successfully so that's worth noting once that is done you know if you go back to your repository you'll see that there is an actions tab so over here you just click on actions and here we have kind of the dashboard of all workflows and here's our build workflow we'll get into that in a minute but the first thing that I ended up doing was back here in settings under actions there's this section called Runners so under Runners you can see we can host our own Runners here I had to click new self-hosted Runner and then the set up is really easy for that so this is how we kind of download essentially an application onto our computer that turns it into this kind of self-hosted Runner it's basically just a little app that you would typically run as like a Windows service so that it's kind of persistently running and it will basically communicate with GitHub there's a lot of documentation I might leave it below if you want to read about how it actually does the communication and stuff but the gist is just that it will communicate with GitHub in doing so it will be able to pick up events such as push events and then it will execute whatever work workflows you define right here on this computer so that's kind of what it is it's a runner it will run whatever workflow you define on this computer when certain events happen and it does so by communicating with GitHub so we have Mac OS Linux Windows you can select whatever architecture you want as well these commands are really all you have to execute I made a directory called actions Runner here it is kind of on my local C drive just at the root uh because it said on a drive route so I was like whatever I'll make it there and then you can see this is what downloads the actual Runner so you just like run this in Powershell or just like in the ter app which typically is Powershell or Windows Powershell to be exact you can optionally validate the hash if you like to make sure you downloaded the full file and then this will just extract the installer finally you just run config with a token that I probably shouldn't show you guys I don't actually know if that's secret or not but I'll blur it anyway so yeah just run this config which is you can see over here uh kind of in these extracted files that you've just downloaded and then you just run it and that's kind of it so that's the entire setup you would need to do if you wanted to get like a new node or a new computer being a self-hosted runner uh for Linux and for Mac OS similar setup which is important because I'm definitely going to be doing this on Linux in the future but for now we're just setting it up on windows so you can see also it says here use this yaml in your workflow file for each job because basically when you set up a runner it gets a series of labels and if we actually go back to Runners you can see that this studio Cho QA which is just the name of this computer that has these kind of labels attached to it self-hosted Windows x64 that's nice because you can Define certain workflows that only run on certain labels basically so like only run this workflow on a self-hosted runner or only run this workflow on a Windows Runner so there's all these talk of workflows so what actually are they and how do you set them up because that's kind of the important thing that dictates when to perform what so that also is rather simple if we go to actions we can create a new workflow and then you basically choose a workflow to start with and these workflows are just kind of defined by yaml files which is great because cuz I love the Amel format and I think it's really simple and easy to understand and easy to read so what did I start with well we we don't use make so I kind of just wanted to see in general what C++ workflows do you have and then I found Ms build based projects which is perfect because we use Visual Studio to build Hazel and Ms build is obviously the compiler that Visual Studio uses for you know building stuff so I just used this as a starting point that's basically what it is when you hit configure here you can see it's made a new workflow under your repository under do GitHub workflows here it is you can call it whatever you want it's a gam file and this is kind of the code that it gave you so this was my starting point uh which is great because you can see that it does two kind of critical things or three critical things even first of all it uses this actions checkout now checkout V3 is not the latest version V4 is so I went ahead and updated that actions checkout what this is and where it kind of exists is over here so github.com actions checkout this is the repository check out V4 that's the latest version as of me recording this it tells you how to use this which is great so I definitely consulted this because I needed much more than just a simple checkout and I'll show you my yaml file in a minute I'm just kind of talking about the journey so that's obviously very important because that checks out you know the repository that does a clone of the repository so that you have the repository now on your selfhosted Runner then the next critical thing that it does is it adds Ms build to path and again if you want to look at this you just go to github.com microsofts setup Ms build yeah here's the repository and you can figure out how to use it uh you know new get packages didn't need that we don't have any and then the other critical thing that it does for us is you can see that it runs Ms build in the working directory which is basically where it checked out the code to so this is that directory that you checked the code out to and it will basically build whatever build configuration you specified over here and whatever solution file path you specified over here so this was a really really good starting point you can also see up here that this is what it does these jobs on so on push event for which branch for me I made it Dev because that's how kind of branch that I wanted to test Master is already our release Branch so the only stuff really going into that branch is stuff that's already been tested and it's effectively released so very rarely code is actually pushed there Dev is definitely the branch that you know gets commits almost every day I want to test that now the idea is you set this up however you like I'll kind of move it over here you can commit the changes right from here you just press commit changes you add whatever commit message you want oh by the way if you want to change the branch that it goes to uh you can just hit this files and then this is how I changed it you know to be in the dev branch and so now you can see we're in Dev and we can commit the changes like directly to Dev so that's where I put it but you basically just do that and then like that's it as long as your Runner is running as a Windows service um well actually you don't even have to run it like you could just hit run for example and you can see It'll connect to GitHub it'll give me an error here in a minute saying that it's already running because it is but you can run it from here you don't have to run it as a service but typically it does install it as a service um and that's good because you can see we have this GitHub actions Runner here now I did did have to make one change uh and that was I had to basically change this log on it was running this as like a limited access kind of network Authority account I changed it to be local system account which gave it admin rights and that was necessary for a couple things first of all because I added a step where it downloaded and installed a particular version of python because we use Python to run like all of our kind of setup scripts that requires admin rights so I had to change that but then also Hazel's own setup Scripts also creates symbolic links in order to kind of avoid copying some files and those symbolic links require admin privileges as well so that's what I had to do to the service but otherwise that's kind of it every time you now push it will run this stuff and you'll be able to see every run inside this actions tab so now let's quickly take a look before we explore the actions tab a little bit more deeply let's take a look at what our yaml file actually looks like so this is it uh this is the complete yaml file I called it build pretty self-explanatory you can see it's being triggered on a push to the dev branch and inside the environment all I've specified is the solution file path if you are paying attention you'd notice that I removed the release configuration but we'll see where that's gone later permissions I left on read which basically just allows it to read the contents of this repository I don't need it to write anything to the repository and then here is the job so you can define a series of jobs here but we only have one job it's called build and this kind of defines all of it so if we take a look at this this build job runs on Windows and self-host again what that means is if we go back to the runners you can see this has both the selfhosted and the windows label so that basically means that this workflow can run on this Runner and if we had like some non- self-hosted runners in the future for example this wouldn't run on that because I only really want to run this on like this computer and then we have something called a strategy and that's because I wanted to set up a matrix now there's some great documentation on this stuff which I highly recommend you take a look at basically the reason why I did this was because I knew that this wasn't just as simple as I want to build debug and that's it I wanted to build debug release and dist I wanted to build all of these configurations and so what a matrix lets you do is it lets you basically specify a series of different parameters and it will automatically run every possible combination of these so if I also had some other like defin specified let's say for example actually probably the documentation is probably more useful so this example here where we have two different os's and three different versions it's going to run that six times because it's going to run these three versions for this and then these three versions for this so every kind of possible permutation is going to be run now in this case it's only kind of onedimensional we just got these three but it will basically run these potentially in parallel and show the results to me as like three different kind of runs of this workflow so that's really helpful because of course I need to test all of these so here's our actions checkout 4 so this will check out I don't know if I had to specify this I won't lie it took me like an hour or two and probably about 18 revisions to this to get it to work because I was doing this for the first time and I was reading the documentation trying new things so it's possible that this is just already in the environment variables and if you leave it as default then it will be fine if you actually go to like checkout this is the checkout repository you know that's where I got this from so you can see that it's a repository name with owner but the default is GitHub repository so I'm assuming that I don't need this but nevertheless it's here now sub modules I did have to set to true and lfs I had to set to true because these both default TOA as you can see uh then I set up a specific version of python 3.12 this is what required those admin privileges for some reason for the life of me I couldn't get it to find existing python installations I figured this probably isn't too bad because this will install it if it's not there I assume it probably gets cashed somewhere in the runner so that doesn't have to keep installing in every run but I think this is a a great little thing to have because it ensures that you have a valid working version of python and it's the right version as well that you know is compatible Ms build it will then find this default architecture was x86 so I changed it to x64 this doesn't install M build it just finds the existing installation without much trouble then we have a setup step over here so what this does is from the scripts directory if we actually take a look at the root kind of Hazel repository we have a scripts directory and then we have this kind of setup.bat which just calls python setup.py so this is the kind of setup script that will make sure that you know we have the right version of vcon installed it'll run premake to generate the visual studio project inst solution files it'll pull like sub modules it'll just make sure that everything's kind of set up for you so that's me calling it here with you know inside this working directory which is going to be the checked out rout of the repository but then we're changing into the script directory because it has to be run from there if you use this pipe here you can specify as many lines as you want of different commands so that's what I'm doing here and then so after this step we now have the visual studio project files and the solution file generated so we can now call Ms build so this step is just called build again inside this working directory it will call Ms build with the solution file path which is specified over here Hazels sln that's our solution /m just means that it will build the projects in parallel and then we have an actual property here which is our configuration set to you can see matrix. build configuration so that is this so whatever kind of variation of this Matrix is that is being run at the moment whether that be debug release or dist that is the kind of variable that goes in here and then we're also outputting a log file this isn't strictly necessary because we'll be able to see all of the logs over here in GitHub but it doesn't hurt to just have a text file with your log on the actual computer so that's what this is and that's it that's that's our entire workflow at the moment and so the way this works is when someone pushes to the dev Branch it will automatically start like one of these workflow runs so you can see for example this was me yesterday uh you know trying to work this out this was the first build I deleted the ones between 1 and 18 that's how many attempts it took me to actually get this running uh and then you can see it's going to run all of this and that's how long it took so every time Dev is pushed to it will basically communicate with the self-hosted runner in this case that we've installed on this computer and what it does is a completely kind of clean from scratch execution of this workflow so over here inside this actions run directory there's this underscore work folder which it asks you what you want to call it that's the default name and then inside here you can see we have Hazel Hazel and then this is the checkout of the repository now it nukes this it deletes this every single time that runs oh there was one thing that I forgot to mention and that's this fail fast false strategy so this is important because since there are multiple kind of build configurations here it won't run them in order cuz as I mentioned they can be parallel I didn't actually see it being run in parallel here maybe I have to do something else for that but either way it randomly selects one of these to run first and if it fails it stops all the other actions so it doesn't proceed to test the other configurations I didn't want that happening because sometimes like a debug build will fail but release and dist will work or dist will fail and debug and release will work work so I want to know if all of them are broken or if just one of them is broken or you know something like that so fail fast means that it's not going to bought everything if one of them fails so that was important as well because before I updated it it was it was failing fast now immediately once I kind of enabled this yesterday a real world use case actually just naturally happened which is great like this was useful immediately because you can see that I finally fixed renderer 2D I did a live stream about this uh it'll be on my channel unplug Channel by the way the V for this but I was working on like extending render a 2d in Hazel so it can dynamically resize how much it's able to render and you can see here that the the debug uh configuration failed that's the one that it happened to run first and so you can see it didn't do the other ones but if we look over here we we can look at the logs we can see that it failed why did it fail because check this out we have a compile error it's saying quad vertex buffer base Undeclared identifier so that actually prompted me to take a look at that because I made the rookie mistake of not compiling my code in all configurations I didn't compile it in debug I was just using the release configuration I assumed it would work uh and then that led me to push this which did actually succeed and that was me removing an assert because this assert only gets compiled for debug builds it gets stripped out of release or distribution builds and this was using a variable that no longer existed and in fact this whole thing is kind of obsolete now because it uses a different system anyway but the fact that it was in an assert meant that the code was stripped in the release build so of course it wasn't a compile error but it wasn't debug so well done CI for catching that because I was obviously too lazy to build in all configurations and you can see that you know all three jobs were completed successfully debugged dist and release you can click on them you can look at all the actions so for example here's us uh you know installing python uh here's us finding like Ms build here's us running the kind of python setup scripts to generate all of the project files and you can see there actually is an error that I still have to fix for that as well but it's not overly critical and then here's the actual build so this is the big juicy one that that this is the entire log from it actually you know compiling every single like CPP file and building the actual projects with 41 warnings zero errors and that's kind of it so anyone on the team can obviously access these logs they can see what's going on if they've pushed something they can see what the error is this is going to become a whole new level of useful when we expand this to also run on Linux because a lot of times you might not have like a Linux PC available for testing so you push something you think it's fine but then turns out it builds on windows and not on Linux and the fact that it can obviously get automatically tested is amazing not to mention when we take this a step further and we don't just build it but we run it and we test it and we run all the different automated tests that we have because that's kind of going to be step two we'll improve it with each release so that we can also focus on developing features for Hazel and getting all that running so this is kind of like Mark one of this system and yeah it's amazing uh hopefully this video was helpful if you wanted to set up a system like this for yourselves I hope I hope this was fairly thorough as I mentioned like the only real two issues that I had was initially just making sure that I changed the kind of permissions from this to this so that I could run these external actions as well and then also making sure that the actual Runner application running as a Windows service had the appropriate administrative privileges to do administrative things hope you guys enjoyed the journey if you did please don't forget to hit the like button below you can also get access to Hazel and help support its development by going to patreon.com Theo and I will see you guys next time goodbye y