Securing The Docker Daemon

Key Takeaways

you can register for part two of this series by clicking on the link in the description and after you provide your name and email you should be able to get access it's absolutely free to watch uh part two of this series on demand on the on24 platform what we cover in part 2 of this series is you know controlling container resource consumption with control groups we also talk about implementing access control for containers with app armor and then we talk about limiting container system calls and performing vulnerability scanning for docker containers and we finally end by taking you through the process of building secure docker images so if you want to register for part two of this series all you need to do is click on the link in the description and you should have access immediately hey guys hackersploit here back again with another video welcome back to the docker security essentials series in this video we're going to be taking a look at how to secure the docker daemon now before we move further with this video i just want to make it clear that this particular case or when it comes down to securing the docker daemon this is going to be really really user centric and deployment use case centric in that some of these steps and procedures may not apply to you you know depending on the type of environment you're working in and i just wanted to you know make that as clear as possible so you don't need to follow exactly what we do i'm simply you know going through these steps so that if it is applicable to you you can go ahead and enable or set up a particular security uh protocol or you know implement a security practice um so with that out of the way we can finally begin uh let us get started by taking a look at what we'll be exploring in this video so firstly we'll talk about managing access to the docker daemon that is going to be handled practically we'll also take a look at implementing tls encryption for actually authenticating and connecting to the docker daemon we'll then take a look at implementing user namespaces this is very very very important and finally disabling intercontainer communication which again will depend on whether or not you require that type of functionality but it is a recommended security protocol or practice to put into play so when it comes down to implementing tls uh you know encryption in regards to encrypting traffic between the various docker components this is only going to apply if you have a remote docker host and you want to connect to the docker host or the docker daemon remotely through a client so for example if we take a look at the infrastructure so we have the client here and we have the remote docker host and i'm just going to annotate this uh sorry let me just annotate that to root to mean remote right so that is our remote docker demon and this is typically the case in production and development environments where you have a docker as a host running the docker daemon and then developers uh you know devops engineers can all connect through various other clients so you know we can say uh client two right and they all again connect to the docker host now as i mentioned in the first video in this series that this connection is facilitated through a domain socket and you use a domain socket when it is a local connection so when you're working on the same server and you're authentic and you're actually sending commands from the client to the docker host uh you know locally then this is handled through a domain socket now if it's done remotely as what we're inferring here then it's done through a tcp socket now you you might be asking well what off the connection well are the commands or is the connection itself secured well by default it isn't and that's what we're trying to address so the way we go about securing this particular connection this particular remote connection is to implement tls encryption right so you're encrypting your traffic so you may be asking well why exactly do we need to do that well the reason we need to do that is because if we get a malicious actor here and you know they intercept the traffic that's being sent between the client and the docker host and it is unencrypted they can perform a man in the middle attack right and they can read the commands uh you know that are being sent from the client to the docker daemon and vice versa so that's very very very important that we enable tls encryption if you are you know authenticating to your docker demon remotely and again as i said that's quite common so uh we need to make sure that tls encryption is set up and i'll be taking you through that process uh we then need to move on to an essential aspect which is um you know implementing the actual namespaces so when it comes down to namespaces as i mentioned again in the first video namespaces are a feature of the linux kernel that are used for the isolation or partitioning of an operating system resource so namespaces make it possible to run containers well you may be asking how i don't understand how how exactly does it do this or you know how does this work well if you think of a container as an isolated space and remember we talked about the emulation layer and how emulation takes place where each container utilizes the linux kernel of the host operating system well it uses namespaces to to partition and isolate aspects of an operating system that you not normally think about as important so for example in this particular in the context of containers namespaces are used to isolate resources like users networks and pids so that each container has its own process has its own process tree as its own network has its own file system and it's not using or sharing the same one that the host operating system is using and again it does this for every container so that's why you know namespaces make it possible for us to have containers so again in our particular case we'll be talking about username spaces here so when we run a docker container the process by default is going to be run from the default namespace and as a result we'll be run with the root account so what do i mean when i say this and of course this will make sense when we get into the practical demonstration so what i mean here is that when you start up a container within the container based on the user permissions you've specified it's going to run as it's going to run as the root user right we're not talking about container security we're trying to prevent container uh we're trying to essentially prevent container breakouts so uh if you run a container and you and you leave it running so you know you you detach from the container and you go into your host operating system and you view the process tree you'll see that the actual docker container is running under a process but the key thing to note there is that by default it will run with the root account so you may be asking well what does this mean really in terms of security well it means that in the event of a container breakout and the fact that an attacker is able to you know somehow get out of the container or leverage or find the way an access vector out of the container because they're working within a root process they will have some level of root privileges which again is pretty much one of the worst things that can happen to you on your docker host all right so as a result we need to run containers as an unprivileged user by default that's a really really important aspect here this is not something that is negotiable and remember we're not talking about the container user we're talking about the process under which that container is running and the user that owns that container all right and this can be empty this can be done by implementing user name spaces so we need to reconfigure the docker daemon to use uh user name spaces and uh you know we will take a look at how to do this practically but you can specify your own non-privileged user or use the default doc remap user that's generated by docker which is what i recommend so that you keep everything standardized and you don't deviate from and use users that are not verified or users that may actually be be removed from the system one day so i just wanted to make that as clear as possible so with that out of the way we can finally get started with the practical demonstration all right so i'm back on my linux workstation and i've logged into the docker host and we can begin the process of securing the docker daemon um so we'll get started with one of the most important aspects here and that is controlling who has access to the docker daemon now after you install docker on your system right uh you're going to need to set up another user that's pretty much a a security guideline that everyone needs to implement you can't run your your linux server as the root user you need to add another user and the reason why we create other users with different privileges is to you know it's based on the concept of least privileges so that every user account on the system only has the permissions and the privileges that they require for their particular role or functionality now in the case of docker the way docker implements access control in regards to who has access to the docker daemon they do this through the use of a linux group all right so a group again is just a sorting of users with a specific privilege so in this case the privileges that the members of the docker group will have is the ability to to actually interact with the docker daemon directly or through the docker client so if you've ever used a user without uh without administrative privileges and you have not added that user to the docker group they cannot interact with the docker demon you know directly or through the client they need administrative privileges this is by design now given the fact that we've disabled access to the root account and only have access through the user lexis what we did during the second video of the series is we added the user alexis to the docker group all right so the docker group is is set up and configured during the docker installation now we can actually list out the groups on the system by typing in cat etsy group right and i'll just type that in correctly and you can see we have the docker group here and one of the members is alexis that's the only member this is going to be an important aspect in a few seconds so we can view the actual groups that alexis is part of and in our case you can see he's part of the pseudo group which we added him to because we want him to have administrative privileges but if it was a standard user without sudo or was not a part of the pseudo group we can also add them to the we can add them to the docker group so that they can run docker commands right so alexis can now run docker commands without the sudo prefix right so i can say for example docker images and i can interact with docker with the docker daemon through the docker client or directly right without any issues whatsoever now this is a very very very important privilege because it means that any member who is part of who is a part of the docker group can interact with with docker without anything else without any other permissions without any other privileges because the docker the docker group assigns them these privileges once they remember now the whole idea of access control is very important when it comes down to security because you're if you're working in in a development environment you're going to have a lot of users who are going to need access to docker who are going to be using docker on a daily basis and as a result you'll be adding them to the docker group now what happens during the transition from a development environment into a production environment is that these users are still left as part of the docker group you know and these users could be developers they could be devops engineers and some of them may still need access in the production environment however most of them will not need access and as a result what you're doing here is you're having or is you you have set up a system where you have multiple access vectors through those user accounts where if an attacker gets access to either of those accounts will consequently have access to docker without any you know administrative privileges and they can execute docker commands uh they can take over and modify the docker daemon they can take over containers so on and so forth so it's very important that you remove users from the docker group that do not require that particular privilege so that's the point that i wanted to hit across here and you should only have users within the group who actually require that privilege and you know accounts that are actually secured in our particular case the only way to authenticate to the server using the user alexis is through an ssh key which we set up and you know that essentially means that the only person who has the key can get access there's no password authentication whatsoever and again we've pretty much secured that aspect and locked it down if you want to view the members of the docker group you can type in members and we type in docker and you can see it only says that the user alexis is part of that group so i just wanted to to actually point that out now the next step here is going to be the process of encrypting the communication between the docker client and the docker daemon now this is only going to apply to you if you are using a remote if you are trying to implement you know remote authentication between the docker client and a remote docker host right so in our case we're not going to do that but i'll still take you through the process now i have created a bash script that will automate the process of generating the tls keys right so if you head over to this github repository it will be in the link in the description and it will also be part of the documentation found in the ebook so you can follow along you want to click on docker tls authentication and you want to download the secure docker daemon bash script here which i will just get so i'll just hit raw here and i'll just w get this here or i'll just open this up within the docker host here so i'll say wget and i will get that particular script now once i have it i need to provide it with executable permissions so secure secure docker daemon and i hit enter so what this script is going to do if we can just analyze it slightly is it is going to automatically generate your the required tls keys so the required dls keys are going to be your certificate authority you're you're then going to need your server key and then yours your self-signed certificate for the key which it will generate automatically and then finally it's going to generate the client keys so the client keys are the keys that you're going to copy to the docker client or to your particular client and all of this will be saved in your home directory under the docker under the dot under the directory docker so this is a dot directory here this is where you typically save configuration files all right in our case it's going to save the keys so we can now execute the secure docker daemon bash script and hit enter so it's going to ask you for your certificate password i'm going to provide my mind my password here and you then want to provide your uh you need to provide your server name that you'll use to connect to the docker server so i'll just provide the ip of the docker host right over here i'll just copy it from lynnode and paste it in there and i hit enter and it's going to generate all the keys and the keys as i said will be stored within the actual docker uh the docker directory here so if i list the contents of that you can see we have the certificate authority key the certificate key itself and you then have the key which is the client key private key that is and then you have the server certificate and the server key right so by default i would recommend leaving these keys within this directory however it is recommended that you transfer the certificate key and the actual server cert and server key into the etsy docker directory under a directory called ssl or tls that's what i would recommend but you know in this particular case it's much simpler to keep them in here that's entirely up to you and the the the guidelines and policies that you need to adhere to any organization or company so once you have generated the keys what we need to do now is we need to change the ownership of this docker of this docker directory to root or we need to change the ownership to the root user so to do this we'll say sudo ch own and again you would not need to do this if you you had transferred or if you transferred the actual server keys to the docker configuration directory so ch on r and then we say root right and root sorry that is just root and then we specify the directory which is uh home home alexis and we just call it docker right and we hit enter and it's going to ask us for a password here which will provide and we have changed the ownership right so now in order to make docker or the docker daemon utilize dls verification or tls encryption we need to create our custom system the configuration file that will override the default docker daemon configuration now this can be done by creating a directory we'll just say sudo mkdir and we'll create the directory on this etsy systemd and under system and the the actual directory we will create is docker dot service docker dot service d right so what we're doing here fundamentally is we are going to create a configuration file that will specify how the docker daemon should be run or it will we will use to specify the options and flags and arguments that the docker daemon should run with or should actually use during startup so now that we've done that we need to create the file so it's going to fall under etsy systemd system and under docker service dot dn we will call we will call it override uh dot conf all right so we'll just say override dot conf and i'll hit enter right so within this file we now need to specify uh that this is a service file right so because we're working under systemd however i'm just going to copy in i'm just going to copy the code here and i'll explain what's going on right so i'll just copy that and i'll paste it in here by the way that just copied the ip in this particular case what we can do is i can actually just take you through the process here so service we will say that this is a service and then we want to specify the xx start option so these are the flags that will be run during the start of the process or when this this particular service is executed or initiated so we want to say this is user bin docker d that's the actual daemon right and then we want to specify the actual socket right so we say this is a unix socket here and we have not specified the tcp pro the tcp interface it needs to listen on but we'll just say var run docker dot sock right and then we say tls verify so we're now saying you need to we're enabling the tls verify option or we're using the flag to set tls verified to true so we say tls verify and then we specify the tls set right so the tls set we need to specify the actual directory and the file so this is under home alexis docker and the name of the file is going to be server cert right we already saw that so server cert dot pem right and then we need to specify the tls certificate authority set which is tls ca set right and that is again going to be under the same directory so home alexis um sorry let me just type that in correctly alexis and then we say docker and the name of the file is going to be ca.pem so that's the certificate authority certificate and then we specify the tls key right so tls key is going to be again under the same directory so home alexis and of course you might the the the of the home directory will be different in your case and then we specify the server key [Music] server key dot pem right so i'll just type that in server key dot pem and then we specify the actual uh the actual ip or the host ip that we want to listen on or we want the docker daemon to listen on for remote connections and this is just going to be our local host or on all interfaces and the port that we're going to specify is 2376. now if you're not using tls the actual ipod is going to be 23.75 but because we are using tls and we've generated our certificates we're just going to say 23.76 right and we can then save this file we're just going to hit save and now we need to restart the the docker service so sudo system ctl restart docker and it's going to restart it for us if we list out the ports that are currently open we just say n a n t p here we can see that it's still not loaded the new configuration so we're just going to restart docker again and it tells us that the unit file source configuration file drop-ins of the docker service changed on disk we need to reload the docker daemon so we're going to say sudo system ctl daemon daemon reload hit enter and we then want to restart docker again so it's going to say we have an issue here so we're just going to check the status of the file here so we're just going to say status and it looks like we have an issue with one of the arguments that we provided here so tls verify tls set looks fine there let us just check the errors here all right so it looks like we had a bit of an issue with the configuration here so um the i think i can see the error here we did not provide the actual directory uh the the directory here so i'm just going to what we'll do is we will try and modify this here which i think the directory is for the tls certificate authority set so i did not you know type in home correctly so we will again save that correct that and save that and we're just going to restart the daemon again or reload the daemon and uh we then want to restart docker so we're just going to restart docker again and it looks like it's running fine now we can check the status of the docker service right so status everything looks fine there and i think we if we type in the netstat command and list the ports that we're currently listening on we can see that it is listening on 2376 which means the the docker daemon has been configured to actually authenticate remotely or you can actually authenticate remotely to the docker daemon using tls uh using the tls certificates or tls verification and encryption and it's running on port 2376 on all interfaces and what we can do now is um we you can now go ahead and start configuring the docker client uh the files um that you need to actually copy over to the actual uh to the actual client are going to be the the actual certificate key and then the private key and the tls certificate authority key so again if i just list all the files that you need to copy over to the client i'll just you know list them out for you here so you need to copy over the following files so you need to copy over the key right over here you then need to copy over the cert which i pointed out and the certificate authority dot pem file over to the docker client and these three files again need to be saved under the home directory under the docker container and then you need to configure a few environment variables so the first environment variable that you need to configure on your client is is going to be the following so again i'll just take you through the process even beyond it's the even though it's beyond the scope of this series uh the first environment variable that you need to set is going to be docker tls verify and of course this is being done on the client but because i don't have a client here i don't need to do that it's going to be docker tls verify so we want to set that to one so we're enabling tls verification that's the first environment variable that you want to set the second environment variable that you want to set is going to be the docker host ip so you want to specify the actual host and this is done by providing by saying tcp because you're connecting through tcp and then you provide the ip and the port which in this case is 2376 and that should be configured once you're done just hit enter and that will save that you can then export the docker set path so export docker cert path and you want to save that i'm just going to type that in correctly and this is going to be there the directory where you're going to save the actual certificates that i told you to copy over and that is going to be under home uh your home directory under the docker uh the docker directory that you created on your client and once that is done you should be able to connect remotely to the docker host that being said that is the process of setting up tls you know verification and encryption and ensuring that communication between the the docker client and the docker daemon remotely is secured so now that we've explored that we can move on to the next step which is implementing user namespaces all right now to demonstrate exactly how this works i am going to run a container here we're just going to if i say docker images let me just list all the images that we have so i'm just going to run the docker i'm just going to run the ubuntu image in a container so i'm going to say docker container run and i want an interactive session here and i do i do not want this session well i can actually just leave it as is i can provide it with a name and we'll just call it test and the actual image is ubuntu and we will say we want a bash session here so i'm just going to say bash well i need to i actually need to provide the actual image name here as opposed to the repository so 18.04 i'm going to hit enter and that should create the container for us right so if i we are currently inside the container so if i say ps right i list out the actual um the actual process tree you can see if in fact i can actually list i can actually display this with top you can see that bash is running as the user root which is perfectly fine because we're right this is the the container process uh process tree and the pid is set to one now i'm just going to exit from from from the top utility and i'm going to put my my contr this particular container in the background so there we are i just put in the background and now you know if i can say docker psa you can see that it's currently running in the background right if i now say if i open up the top um if i open up the top command well we can actually just say docker container and we use the top utility and specify the test the test name which is the name given to the container and we hit enter we can just hit enter here you can see if i can just make sure that this is structured correctly that this container is running as the root user so you can see you the use id is set to root but under different process id so this is the concept of namespaces right so namespaces allow isolation of containers and you know it provides them with their own process tree process id but the problem is that this container is running as the user root which means that if if there is a container breakout then this then that particular attacker will be it will get root privileges you know so i just wanted to point that out and that's the issue that we're trying to remedy here so uh i will just remove this particular container now so docker remove and we'll just specify the container name which is test and it's going to remove it so we can just confirm that it's been removed here there we are so it's removed what we need to do is we need to use or we need to remap the actual namespace to the default map or to the default option now when we talk about mapping namespaces and subordinate users if i list out the actual directory for the subordinate users on the system you can see if i type in a sub uid that is sub uid let me just type that in you can see the only user we have on the system that makes sense is going to be the user lexis so we can specify that all containers should run with the privileges of the user alexis however the user alexis has administrative privileges because he's part of the sudo group so i'm going to use the the default user that is created by docker and that is the doc remap user now this can be done by again using the or providing the user namespace remap option and specifying the default user and again we can do this this is why i created the custom override configuration file uh this can be done by modifying the etsy systemd file uh that we created which is uh under um which is under system and docker service dot d and override uh dot conf right so all we need to do is remember these are the arguments that the docker daemon is going to run when it is started up so we can go all the way to the end here and we can specify the flag the uh the username space remap flag so we will just type in right over here i'll just space that out so we can say uh user namespace user our user ns and then we say remap that's going to be equal to the default docker user so that is the default dock remap user and you will see that that particular user will be created um so now that we have provided that uh we now need to reload the docker daemon so we'll reload the docker daemon really simply here by uh if i can just find the command there we are docker or daemon reload that's going to reload the the daemon and then we need to restart docker or the docker service and hopefully there aren't any configuration issues so we'll just restart the docker daemon and it looks like it's running without any issues if we now list out the actual sub user ids or the subordinate user ids you can see that the docker or the doc remap user has been created and now every container is going to have or is going to run under the process uh the the process id of doc remap which is unprivileged by default which means even in the event of a container breakout that particular attacker will not have any administrative or root privileges right so we can actually demonstrate this by creating the actual or running the container one more time so again within the container it's still going to be run as the as the user root and i've already talked about this uh uh or i'm going to talk about it during the the actual video where we cover securing docker containers how to change the default user within the docker container and how to create users and so on and so forth but if i run it you can see that it's going to grab the ubuntu image locally that's because we've changed namespaces before we were running in the default namespace now we've changed namespaces to that of the of the doc remap right so it had to grab a new image because the permissions have changed so again if i say top again you can see that it's still running under the user root within the container so this is the the isolated container and again this is facilitated through namespaces but now if we actually if we actually put this in the background and we again type in docker container and we specify the utility top and the name is just test right which we provided you can now see that it runs under the user id 16536 which is the user id for the doc remap user right over here so that actually confirms it and as i said in the event of a container breakout this particular that particular attacker will not have any additional privileges that they can use to take further control over over the particular host all right so that is uh the namespace uh or actually you know implementing name user namespaces for the docker daemon and of course consequently for the docker containers now let us talk about inter-container communication right so intercontainer communication is something that you may require based on your circumstances but it is required to it is actually recommended to disable it if you do not need your containers to communicate with each other as i said however it's entirely up to you and your configuration whatever you choose to use but again disabling into container communication is fairly simple all we need to do is modify the configuration file the override file that we created so uh we'll just open up that file again and we will head over to the end right and right over here actually i'll just do it from here and i will simply say we will provide the flag and by the way these are all docker daemon commands that i'm using the only thing we're doing is we're specifying that when the docker daemon runs it should run with these particular options or with this particular configuration so to disable inter container communication all we need to do is type in icc and that is equal to false right so we're disabling it and again if you want to enable it you can get rid of this flag or just you can change the value the boolean value to true all right so that is done i can now save the file and we now need to reload uh the daemon so i'm just gonna find that particular command there we are daemon reload and we now need to restart the docker service all right so it restarted without any issues and we now should have inter container communication disabled so the only way to verify what we've implemented is to run the docker bench security utility right and to get a fresh a fresh benchmark so i'm going to open up the directory docker bench security and we are going to run it so docker bench security and hit enter and let's see if our score has improved all right so you can see that it performs 84 checks and all score has improved drastically uh we have we now have a score of 30 which is great so again uh to recap what we've been able to do so far is to uh is to actually secure the host configuration so the with the one exception of the separate partition for containers which is again entirely up to you but under the docker daemon configuration you can see that ensure the network traffic is restricted that is the inter container communication which we actually disabled so is restricted between containers on the default bridge so that is disabled we then also the tls authentication is configured we also enabled user namespace support which we got a pass for um so the only other options here um are to do with the authorization for the docker client commands which we will take a look at when we get started with access control as for centralized and remote logging that's again going to be dependent on your organizational requirements and then ensure containers are restricted from acquiring new privileges again that can be that can be configured again during the container security video we'll take a look at how to configure that during the container security video and ensure live restore is enabled that's not really required but as far as all the other options for the docker daemon configuration files are concerned we pretty much get a pass for all the uh the actual checks that are important um and again for the container images and build file uh we we're still to actually take a look at that so that's pretty much all that i wanted to cover in this video and i'll be seeing you in the next video a huge thank you to all of our patreons your support is greatly appreciated and this is a formal thank you so thank you shamir douglas ryan carr sandor michael busby sids up doozy defean barry dustin umps and michael hubbard your support is greatly appreciated and you keep us making even more high quality content for you guys so thank you [Music] you