Linux Server Course - System Configuration and Operation

Key Takeaways

linux is a popular operating system for server administration because it's secure stable and flexible in this course sean powers from cbt nuggets will detail every part of configuring monitoring and supporting a server setup that runs the linux operating system this course will teach you everything you need to know to configure linux servers including the boot process kernel modules network connection parameters localization groups and more let's get started [Music] bios and uefi are two tools that basically do the same sort of thing but they can be a little bit confusing when it comes to how do you know which one to use on a computer and and do i need to support one or the other or both or the nice thing is you usually either have bios or uefi now bios is an older program it just stands for basic input output system whereas uefi is the new kit on the block and this stands for unified extensible firmware interface which sounds confusing but really it's just the the way that we can interact between the hardware and the operating system now let's say that we have two different vehicles now these are two obviously very very different vehicles one of them is a sports car and one of them is an awesome yellow volkswagen beetle i actually have a volkswagen beetle that's yellow like this and it's awesome but nonetheless you could be a sports car person either way even though they're ridiculously different vehicles they both have some common interfaces right they're both going to have brake pedals they're both going to have steering wheels they're both going to have windshields and those interfaces are fairly common across all vehicles now the brakes in the underlying system are going to be different this sports car probably has really nice disc brakes whereas these old volkswagens have drum brakes now drum brakes aren't as good but the interface itself is very very similar you push the brake pedal and you stop and that's kind of what bios and uefi are they're interfaces between the hardware and the operating system itself now they do work a little bit differently so here we have our hard drive on our system now using bios like the old method of booting a computer you would have like the very first sector on the hard drive would be the boot sector and that's where the mbr the master boot record would live and then that would tell the computer where the partitions are and point it to where to boot now there's a lot of limitations with bile so you could only have four partitions using the bios and mbr combination there are some hacks to get around that you know they would take a partition and do extended partitions inside that but that's a whole nother nugget still there was this limitation also a limitation of size with the drive how much this boot sector or this master boot record can actually reference it can be a small amount like two terabytes instead of exabytes of data and so uefi is a replacement for the bios technology and rather than just have the single boot sector what it does is there's an entire partition on the system and that partition is where all of the boot code is for whatever operating systems might be on the computer so rather than just the boot sector pointing to you know the rest of the hard drive this is an actual specialized partition on the computer and that's where the uefi code is stored also it uses a different partition scheme so you can have tons and tons of partitions and it can address a much larger hard drive there's also other things like secure boot that uefi supports that bios doesn't basically just know that uefi is the replacement for bios it replaces the functionality of connecting the hardware to the software of the operating system now the good news is if you have a computer that's older or even some new ones so come with bios you can still get around a lot of the limitations because there are hacks that will let you use really big hard drives or hacks that will allow you to do some of the things that you can't normally do out of the box but most computers now are coming with uefi and from a from an installer point of view there's very little you have to do because the operating system is going to say okay i was booted using uefi so i'm going to create a partition a uefi partition and i'm going to put all the boot code in there so from your standpoint from the end user or the installer standpoint there's very little difference but under the hood a lot of new cool stuff is going on and that's why uefi is kind of the way of the future remember when you were a kid and in the doctor's waiting room there were these magazines and one of the games inside the magazines was that you had to find the difference in one of a bunch of similar looking figures like for example here are what i have nine different jokers and one of them is a little different now it's not terribly difficult to see which one is different here if you watched while they appeared this one has a tiny little spot right there on his shirt that's different than all the others so we've been able to identify it but you can see they're very very very very similar well the same thing is true about grub and grub 2 which of course is the next iteration of grub now grub stands for grand unified boot loader but really it's just the way that the computer transitions from bios or uefi booting up conditions into the actual operating system itself it's what tells the computer okay where's my partitions and stuff like that it's very easy to confuse which is which on your system which seems silly but it can be very very embarrassing when you sit down at a system you're like okay so is this running grub or grub 2 because they do the same thing right they both boot the computer but they do have some minor nuanced differences grub of course is older you probably guessed that by it not having a 2 at the end of its name but grub is often called grub legacy because it is older but it's still on a few operating systems not many but a few i think slackware still uses grub legacy now the biggest and easiest way to figure out if you're dealing with grub or grub 2 on your system is to look inside the boot grub folder and if you see menu.lst or grub.com that means you're running grub legacy because grub 2 does not have those configuration files grub 2 has this configuration file grub.cfg now it's easy to confuse cfg.com so i always just look for menu.lst if that exists you're on grub legacy now the thing about grub is it's kind of difficult to modify it was uh it's very easy for automated systems like when you install a new kernel to figure out how to put how to you know update the menu and update the boot code but it's really difficult for the end user to modify that stuff and the boot menu usually when it's booting up it says you know press something in 10 seconds in order to change the way it boots so this usually just appears for you and you can see it now grub 2 is a lot more customizable in fact if you go into the etc default grub file this is a configuration file that's easy to read and it's going to allow you to change the way that it looks when it boots up now grub 2 also has a ton of other cool features it can boot from an iso file from a usb disk it can name or it can identify hard drives based on their uuid or their device like dev sda so it's a lot more advanced than grub one but one of the you know the advancements that's also a frustration is that boot menu that i talked about that comes up with grub it's hidden with grub 2. so it just goes right to the login screen you never see anything from grub 2. and if you don't know this little trick it can be a real bugger to get into that menu if you want to change something during boot so be sure to just hold down the shift key when the computer is booting up and then boom all of a sudden you're going to get into the grub 2 interactive menu that you can change boot code things on the fly it's really really cool i'll show you what i mean but it's not difficult once you know if you have grub or grub 2 grub 2 which allows you to do a lot more configuration stuff now i'm here on an ubuntu system this has grub 2 and if we were to look at let's just look at it etc default grub we're going to see this is just a configuration file we can change these things once you make a change you do have to do sudo update grub you do that it's going to update the boot code inside the boot folder and if we look in that boot folder so go into boot grub type ls i look at that grub dot cfg there's no menu.lst so we know that this is grub 2. so that's how you manage things in grub 2. i'll show you really quickly before we end this is a computer that's turned off if i start this computer and hold down the shift key as long as i hold down the shift key boom we get this menu but if i don't hold that we don't see any menu for us to interact with grub at all now figuring out if you have grub or grub 2 can be a little bit challenging if you don't know that simple trick to look for menu.lst but they both do the same thing they tell the computer how to boot up and how to mount its different partitions it's just that grub 2 is definitely an advancement it's more configurable it does more things and it's easier for the end user to manage i hope this has been informative for you and i'd like to thank you for viewing linux is so flexible it can boot from an incredible number of methods or sources now there are several different things we need to understand about the boot processes like is it hardware based or is it software based and i'll tell you what i mean about that in a minute but there are just this multitude of ways that a linux system can boot it can boot over pxe or pxe boot if you haven't seen that it's just the coolest thing ever it's when it boots up completely over the network we can boot from usb from cd ipxc is a more advanced version of pixie i'll talk about that too and then iso images just like the dot iso file you can actually make a grub 2 entry that will boot directly from the iso even though you don't have it burned to a disk or to a usb drive it can live on your system and you can have a grub menu item that boots directly to that iso image it's really flexible what it can do but that hardware software thing i wanted to talk about because where the boot process is taking place is really important some of it is linux specific some of it is not so first of all let's talk about the hardware things now when i say hardware i mean the bios or the uefi the part that takes place before linux is ever introduced and pxe is one of those things pxc which stands for pre-boot execution environment is the way that the hardware says okay i don't see a hard drive or i'm not set up to use a hard drive so i'm going to query the network and it just queries a dhcp server and the dhcp server responds with not only an ip address which is what dhcp servers normally do but also a boot file in a tftp location a tftp server is just a place you can store files on a network and then basically the computer then downloads that image that boot image from the tftp server and that's where linux comes into play that kernel or that boot file is the linux kernel so it downloads it off the internet or off the internet um off of your local network and then it puts it in memory and boots itself from there so pxe starts as just a hardware thing and then turns into a software there's also ipxc which is very similar but instead of using tftp to download that thing it allows you to use http which is faster and usually more reliable than the old-fashioned tftp but it's very similar in concept ipxc if your computer supports it then usb this is also right on the hardware now the hardware determines exactly how to boot from the usb but on the usb itself is where the linux code is right so the hardware knows that it can boot from usb same thing with cd and to be quite honest same thing with a hard drive right the hard drive is booted too because the computer itself understands how to do that but when linux comes into play we'll say the software side of things this is once it loads into linux linux does some things on its own like the iso booting with grub2 that's after linux starts right or after grub starts it says okay i'm in grub now what are we going to do we're going to either mount a partition or we're going to look at this iso file and actually use that as our operating system just like it was burned to a cd so some of this stuff is software this is where you like select a kernel this is where you can have mem test so a lot of the stuff is done in software but most things actually all things have to start in the hardware otherwise you're never going to you know get to the point where the software takes control but it's important to know that pxe ipxc these are not linux specific boot methods these are boot methods the computer supports that linux also supports in that they can provide the boot file so i don't want you to confuse pxe network booting with something specific to linux same with usb and cd we can boot windows from usb or cds and it works you know because the hardware supports it if i'm honest probably the most fun way to boot a computer is using pxe only because there's no media right it just boots directly off the network and for some reason that's just really awesome to to be a part of it's really awesome to see but there are multiple ways that you can boot linux and it's important to know that they're all there it's okay if you don't know exactly how to you know boot from an iso file using grub 2. anybody's going to have to google the specifics of that in order to make it work but knowing all these different processes exist for booting a computer that's vital because that will help you learn and help you troubleshoot when you run into a booting issue when it comes to actually booting the linux kernel there are a lot of problems that need to be solved and it's sometimes sort of like a chicken and an egg scenario you have to do one thing but you can't do that thing until the other thing is done and what do you do first and how do you do it it can be really confusing but basically we want to get the full kernel with all of its modules running so the boot process is very complicated and it might seem over complicated but the issue is to try to get the kernel running and then allow it to access the modules that are stored on the hard drive on your system and it ends up being like i said fairly complicated and if i'm honest i've been a system administrator for over 20 years i've passed lots of certification tests and i have never fully understood every step of the process of booting the kernel so you're going to be like the king of the next nerdy party that you go to because you're going to you're going to know all the trivial pursuit answers when it comes to linux kernels so let's look and see what the boot process actually is and then i'll show you on a system where those files live so i have sort of a little flow chart here now a lot of these steps we probably are already familiar with first the computer either has bios or uefi and that's the hardware on the computer which looks for something that grub or grub too it's one or the other here will provide and then that boot code of you know that grub uses points us to the kernel now this is the stuff where it starts to get a little not confusing so much but complicated it's almost like this fine dance that has to be done now norm what the process is normally is we have the actual linux kernel itself which is a file one file and it's called vm linux or you probably currently see it called vm linus with the z and the only difference here these are the same file the pr the only difference is that this is compressed right it's the z means that it's compressed so it's just a space-saving method uh so usually we use vm linus just to save some room on the system and then this is the the kernel itself with no modules right so there's no modules this is just the base kernel and then once the kernel boots up it will mount all the file systems and then it will have access to all of the modules that it needs to insert to make things work like you know your usb mouse and your keyboard and your monitor and your video card all those things are modules that are loaded into the kernel they're not part of the the static kernel right i mean we could build this huge kernel that includes everything but that's just a waste of resources and so we have this stripped-down kernel that has just enough stuff to make sure that we can mount the file system so that it can have access to all of its modules and then become the full running kernel on our system there's a few problems what if this full kernel and all the modules yeah so basically what if all the modules here are on a file system that this stripped down kernel doesn't know how to mount right it's like well i don't know how to get onto a raid device or i don't have any idea how to mount this fancy new ssd drive that you put into the pcie slot or something well that's where init rd or init ram disk comes into play this is just enough information like module information and driver information to be able to have the linux kernel access the file system so that it can get to its modules so rather than make you know a custom bigger kernel for every specific system what we've done is we have a generic strip down kernel and then this init rd has the stuff we need in order to load the modules by mounting the hard drive or you know whatever we need maybe this is stored in a network drive so we have to have the nfs stuff in order to mount a remote disk on the on the computer so it can access its module so this is just like the the temporary staging area for kernel stuff that we need and this is kind of inserted right into this running stripped down kernel okay now i want to mention this because a lot of people confuse the init ram disk with the init ram file system and they seem similar conceptually they are fairly similar conceptually except that the init ram fs is actually part of the actual kernel itself it's part of this vm linus or vm linux kernel and this is just like it says it's a file system that it creates and mounts in ram and i want to mention this tool even though we're not going to go into it much dracut is a tool that has made this extremely generic using udev and and stuff like that so that it's very flexible but this is basically the tiny little file system that the linux kernel uses in order to do what it needs to do to get to the point where it can load the full kernel by loading modules so init ram fs is a file system that it loads into ram but that's part of the kernel itself this init rd is a ram disk that is mounted alongside of the kernel that allows it to get to the point so it can use the full kernel and knit ram disk is not used after the system is booted up this is just like a temporary staging ground to get to the full kernel wow that was a lot of information but it's really fairly straightforward when you see why it's doing all of those complicated things now i want to show you really quickly some of the files that we talked about so here i am inside ubuntu and this is the boot folder and we can see we have here's the init ram sure enough and this is i have a couple different kernels in here there is an update but the kernel numbers are listed here but here's the init ram disk and here is the actual kernel file itself there's a couple other files the system map actually tells the kernel where on the file system all of its modules live and then config this is the configuration file when the kernel was actually compiled so if you want to see the options that were used there but those are the main files that are on the system one other thing i want to show you is this system.map tells the kernel to look in lib modules and then the name of the actual running kernel and inside there is where you'll see the actual different modules okay and this is where the modules that are going to be loaded in to get that full kernel live so that's the basic uh basics of starting a linux kernel on your system now i know we covered a lot of things and a lot of terms and a lot of concepts but they should make sense once you go through that flowchart of what's happening and why it's happening and in the end which comes first the chicken or the egg well it depends on what you need to do right if you need an egg then the egg comes first if the chicken can handle it on its own no eggs are needed now a kernel panic certainly seems like a really good time to panic but i assure you just because the colonel panics doesn't mean you should soldier but really kernel panic just means that something went wrong with the kernel now there's a couple common causes and there's a couple things we can do depending on what the root cause is now if all of a sudden you're starting to get kernel panics and you haven't done any system updates recently and it's kind of hit or miss it'll happen it won't happen it happens when you're doing something but not another chances are you may have a piece of faulty hardware and this is one of the really common ways that you can experience a kernel panic if you have an overclocked cpu for example that can sometimes cause kernel panics a bad stick of ram is another one that'll really do it and you know some add-on card like video cards are sometimes you know guilty of doing this really anything on your system any hardware if it fails it can cause a kernel panic there's a couple ways to you know try different things especially if you have multiple sticks of ram sometimes you can pull out a stick of ram see if you still get a kernel panic if you do then put that stick back in and keep pulling them out one at a time and see if the kernel panics automatically go away when you remove one certain stick hey chances are that's what's wrong with you know that certain stick is a bad stick of ram it can be frustrating to track down hardware related kernel panics but there's also another very common kind of kernel panic that happens and that's when you upgrade your system so let's say that you just ran a system update on your system and it installed a new kernel you reboot the computer and then boom it says something like failure to init or kernel panic blah blah blah blah if that happens there is something that you can do which is fairly easy to do and you can you know rescue your system so to speak so when it boots up we're just going to go into grub and we're going to pick an older kernel because when you update a system it keeps the older kernel for this exact reason if something goes wrong you can pick a different kernel and boot the system and then fix it so i'll show you what i mean here we are on our ubuntu system and let's say i just recently updated it and during the update it says now i need to restart my system so i'm going to restart the system and everything should go well blah blah blah blah and then boom we get a kernel panic now this doesn't actually give us the words kernel panic because i did something fairly simple to simulate what would happen if there was some corruption in the kernel uh but it says here you know it's unable to find the init rd sometimes you might see kernel panic init or unable to kill init or something like that it says press any key to continue but there's nothing that's going to happen right it's kind of locked up on me because it aired out there's no init rd so what we're going to do is restart the system and when we do that we're going to initiate the grub menu by holding down the shift key so first of all we do need to restart the computer i'm going to hold down the shift key and we should get presented with a grub menu sure enough here's our grub menu advanced options and we're going to see here we have multiple kernels this is the kernel the newest one is the one that it tried to boot by default so i'm just going to go back to the last successful kernel that we had press enter and it should boot this up and our system should be just fine then all we need to do is remove that kernel package with the failed thing using our app system reinstall it again you know do an update so it reinstalls and then it should install that init rd back in place and we should be fine so it's nice that linux keeps a few of the older kernels kind of in line so that we can do exactly this if something goes wrong with one kernel or we've deleted modules or just something doesn't load or we get a kernel panic we can select a previously successful kernel and then here we are on our system working perfectly and now we can go through and remove that package and fix it etcetera etcetera etcetera so if you do get the kernel panic especially on a production system oh it can be really scary but know that there are ways you know to fix it if it's just a kernel corruption you can boot to a previous kernel if it's a hardware issue it's a little harder to troubleshoot but you can also do something like boot from a usb drive or a cd and at the very least you'll be able to access the hard drives so that you can get stuff off of your computer if it is a hardware failure that's causing the problem and then you'll have your data to put onto a new computer but kernel panics are no reason to panic it's just one more thing that we need to understand how is working so we can fix it or troubleshoot the problem and move on from there we know that the linux kernel is modular in that it doesn't load all of the drivers that it could possibly ever need automatically by default it has these modules that it dynamically loads when it discovers that it needs like a driver for a certain sound card on boot now we can configure it to automatically load those modules by setting it up in a config file which we're going to look at now what happens though is it'll load that module but then on behind the scenes there's some incredible dependency checking going on and that dependency checking is really really good at knowing what modules one you know a particular module will depend on there may not actually be a robot called modbot in your system or even in my system i just like to think of a cheesy robot plugging in different modules in my computer but nonetheless the computer is really good at finding dependencies in fact sometimes it's too good and we need to blacklist certain modules so that modbot doesn't say oh this one will work and put it in i'm going to show you why we would do that and also how to do that but we're looking at two concepts today how to automatically load a module on boot and how to make sure that the wrong module isn't loaded on boot automatically by the system detecting a dependency that you don't want it to use now here on our this is an ubuntu system but all linux distributions have kernels we're going to go into the etc folder and there's a file here called modules so we're going to look at that we can see right now it's empty it says kernel modules to load at boot time this contains the names of the kernel modules that should be loaded at boot time one per line lines begin with this are ignored so what we would do is put the name of the module and we would add something like e 1000 which is the name of an intel network card module now normally the system automatically detects the hardware and it knows what to load but we want to i'm going to give you an example of how you can manually load a kernel module and have it done automatically on boot even if the system doesn't you know detect it on boot like oh i have new hardware i need this particular module so this is how you would manually do it right now we're not actually going to save this because i don't want to load that module but let's say that's loaded now if there are dependencies it will also load those dependencies let's say that the intel e1000 network card also requires the pci bus to work well then it would load the pci bus module so it's really smart but let's say that there are two different pci modules there's like uh pci version one and pci version 37 and by default it's going to load pci version one and we don't want that so what we would do is blacklist it so if we go into etc mod probe.d first i want to show you there's a bunch of files in here anything with dot conf is read in right so it doesn't matter which one you put it in there's just some conventions here anything we're going to manually put in we're going to put into blacklist.com so let's actually edit that file and we're going to see there are a bunch of things that are already blacklisted now what this does it just makes it so that the kernel doesn't automatically load it like here's a let's look at this ethernet one okay so this is an e1394 this is a firewire network device all right it's saying here we never want to use a firewire network device so if you detect a firewire device i do not want you to load the e1394 or the ethernet module for a firewire port so if we put blacklist here it's not going to load it even if we put in the firewire module itself so that it activates that port this will make sure that the kernel or modbot or whatever however you like to visualize it doesn't say oh well you're putting that in why don't we also activate the ethernet ability of that port we don't want that so we put blacklist in here same thing like that i've i've done this the most with sound cards because there are like 150 different sound card drivers for the ac 97 model of sound cards so a lot of times you have to blacklist a bunch of different models and see there's a blacklisted sound card right here right this is saying don't load this module that's not what i want and this looks like it's because of an ubuntu bug all right so anyway this is where we tell it what not to load even if it's compiled in and it's part of the module we would put it in here or any of the files in this folder anything that says blacklist space blank is going to be blacklisted honestly like i mentioned the linux kernel is usually really good at automatically detecting hardware and loading all of the modules all on its own by detecting new hardware on the system but if there are times where you want to manually load something but you want it to be done on boot that's where we would put it in the etc modules file and then blacklisting is really important especially if you're having problems with something like uh it just doesn't work with this kernel module let me blacklist that so the kernel will pick another module and try to activate that one one of the things that makes the linux kernel very special is that it's modular so we don't have to install all of the drivers for all the potential hardware in a system we just installed the drivers and the modules for the things that exist it makes the linux kernel very efficient because we don't have all that bloat all those unneeded things sitting in memory where they're never going to be used so the kernel can be very very efficient because of its modular design but in order to take full advantage of that we have to make sure that we're putting the right modules in the right place with all the dependencies so we don't end up with a kernel looking like this right with pieces that don't belong thankfully there are a handful of tools that we can use to properly manipulate the kernel modules and the first two i want to look at are ins mod and mod probe because on the surface they appear to do the same thing insert modules into the running kernel this is after we're booted it'll insert them into the running kernel so let's compare them to scientists right so ins mod is a very basic program you have to give it the full path of the kernel module that you want to install it doesn't do any dependency checking it just kind of slams it into the running kernel and if the kernel doesn't work like it's the wrong kernel version or it doesn't have the proper dependencies it's just going to fail and it's not going to give you an explanation as to why it's just going to say well i crammed it in there and it didn't work boom but mod probe on the other hand is a more advanced program it's a more efficient model and what it does you can give it just the name of the kernel module you don't have to give it the full path you can give it the just the name of the module itself it will look and determine all of the dependencies that that module needs so if one module needs you know depends on another module it will say oh let's load that other one first it does a really good job now it does need us to have a map of all the all the needs and dependencies on a system but there's a program to do this we just make sure we type dep mod when we're done installing a new kernel module and it will recreate that map so modprobe knows where to you know find the dependencies now you might be thinking okay well obviously i'm going to use modprobe when i insert modules and exactly that's what you should do why does insmod even exist well here's the deal mod probe actually knows what modules to install and then behind the scenes it uses ins mod to do the actual inserting so mod probe is really kind of a front end that's very intelligent about what to do on the system but that's why insmod still exists it's still the you know the basic tool that is used to cram modules into the running kernel modprobe just knows which kernels to load and in what order to load them so modprobe is what the humans use and ins mod is what modprobe uses if that makes sense it's really easy to see that in action on our system so i'm root on ubuntu here and let's go into lib modules and we're going to see we have folders for each of the kernels installed on the system we're going to go into the one that we're currently running which is the latest one and inside here we'll see this is where we have a lot of like the map files for the mod probe so it knows where the kernel modules live let's go into the kernel folder where the modules live type ls let's go into the drivers folder type ls again oh there's still a lot of drivers let's go into the net folder all right there we actually see some kernel modules here okay so uh we're gonna play with thunderbolt net but let's say we wanted to load this in so we could use thunderbolt as a network device if we wanted to use ins mod we would say ins mod lib modules generic kernel drivers net thunderbolt net because we need the entire path we're going to press enter and it's going to say well i can't do that unknown symbol in module it doesn't work blah blah blah it has a dependency it doesn't tell us what the dependency is but if we were to use mod probe and we were to say mod probe i want you to put in thunderbolt net and that's all we don't have to say a path we don't have to put ko at the end just press enter boom it's automatically installed how do we know well one we didn't get an error it just installed it but if we type ls mod will show us all the installed modules and if we scroll a bit to the top we're going to see well look at that thunderbolt net is installed into the running kernel and we can see the dependency was thunderbolt but modprobe knew to do that and it installed it now there's another tool for removing them and that's rm mod now there's no super smart version and dumb version of these rm mod is kind of halfway between because if we were to say rm mod thunderbolt it's not going to do it it's going to say i can't do that it's in use but at least it tells us what it's in use by right thunderbolt underscore net which that's a little strange because the name of it doesn't have an underscore right if we want to get rid of this we could say first we have to rm mod thunderbolt dash net okay no errors and now if we do the up arrow twice now we can rm mod the thunderbolt module and again no problems because we we got rid of that dependency first so now if we do ls mod we're going to see if we scroll up that they're not in there anymore they're not installed into the running kernel and that's how you can manipulate kernel modules on your system some of the tools are smarter than others but it's really important to understand why they're smarter and like insmod is important still even though you don't want to use it on the command line for doing very much at all now one more thing i just want to tell you about is if you install a new kernel module like you download a piece of hardware and it has its own kernel module and you compile it and so then you have the kernel module that's put in this folder you know so that it's in our current running modules folder you have to type depmod in order for it to update that database or that system map so that modprobe knows where it lives and what dependencies it might have it takes a while and now we have it updated so now we can install that new kernel module because it knows all the dependencies on the system now i know in the picture here i have these you know wrong puzzle pieces put into the puzzle this is actually a lot harder to accomplish than you would think because the tools just fail right they say oh this won't fit or this is the wrong kernel module some of them are smarter about it than others but it's really difficult to put the wrong kernel module into a running kernel but i want to make sure that you understand all of these tools exist how they work what they look for and the difference between them first off we're talking about network connectivity today and the first thing i want to show you is always check the cable now it seems silly right of course the cable is fine this is plugged in just fine however ah see it wasn't plugged in tight and whether it's a jokester in the office or somebody who ran over a cable with a vacuum cleaner or something sometimes it is the cable and you can spend a ton of time on the command line and be very frustrated to discover that it was just a bad cable so that's just a pro tip check the cable make sure it's plugged in tightly and that the little lights on the back are flashing and doing their thing now we could probably do an entire course on troubleshooting a network connection that's not working or not working right but today i want to go over just a couple quick tools so you can determine very quickly what your issue might be on the network so i want to look at ping and then just to check out our address on our computer and so the first thing i want to do is give you this scenario we're on the computer and we're trying to go somewhere like google and it says unable to connect oh we can't get online so the first thing i would do is open up a command line window and i would say let's try to ping google so ping google.com network is unreachable okay well i know that the google dns server has an ip address of 8.8.8.8 so that's the next thing i would do to eliminate whether it's dns right maybe it's dns that's not working so i want to see if my network itself is working so i'm going to say ping 8.8.8.8 ah network is still unreachable okay so there's an issue with more than dns that means that there's a problem with my network so let's look and see what our network address looks like now there's a couple tools you can try to type ifconfig if this doesn't work yes see it's not on my system it's a newer system so type ipad for address and it will give you the ip addresses on your local computer for your network devices so this is the localhost 127.0.0.1 this is just like a virtual host that says it's it's local so we don't want to use this one but this is our ethernet port right eth0 and it looks like we have an address of 10.10.10.10. so i would say let's ping that so let's make sure that you know our network stack itself is working okay so this is working i can ping my own ip address so the ip stack itself is working i'm going to hit control c but for some reason i'm unable to ping out of the network now let's assume that my gateway address is 10.10.10.1 ping 10.10.10.1 okay i can ping other computers on my network so it's not well first of all we know it's not my computer cable right because i can ping a different computer that's across the network but i'm still unable to ping google so for some reason my internet connection is down well let's look at our ip routing information i'm going to type ipspace route and press enter alright it looks like 10.10.10.0-24 so this network is directly accessible via eth0 okay so that's working that's why we could ping this up here but if you'll notice now you might not notice but there's a line here that's missing we don't have a default route set so that means that the computer while it can access other network or other computers on our local network it doesn't know where to send network or where to send packets that are destined for another network there's no ip routing information here now if this is an ip routing nugget i could say okay let's add a route manually but this is more of a troubleshooting nugget so here's the deal probably when we got our dhcp address we didn't get all the information or something fell apart the first thing you do is either turn it off and turn it back on right that's that's the standard it guy response right well in this case it might work or we could actually go and just go to our wired network interface here turn just the network interface off connect again turn it back on and now let's well let's just look and see what our ip route information looks like hahaha look when we brought our network interface down and back up now we have a default route added so now let's see if we can ping google.com boom we're able to ping it we go over here and we should be able to access google as well and sure enough there's google so we've successfully troubleshooted the problem with our network connection in our case it was a route thing and really just turning off the network interface and turning it back on was the way to solve it but the important part is going through the process of determining where the problem is we knew it wasn't a network line or that network cable because we were able to ping other computers on our network so it's just a matter of troubleshooting and going down the list of where can i connect where can i connect and what might cause those issues figuring out connectivity problems is sometimes more of an art form than it is a science you just have to kind of put yourself in the mindset of a packet and say okay what information do i need to go from point a to point b and what is stopping me from getting in between there tools like ping were are able to make sure that you can or can't connect to remote computers and then using the ipadd you can see what your address is we also used iprout to look at the ip route information and that's where we discovered we didn't have a default route but tools like this are what you can use to troubleshoot your connectivity to figure out what's going on but don't forget always check that cable because sometimes it's just a physical cable unplugged dns or domain name system is the way that your computer converts a url or a domain name like google.com into the numbers the ip address that the actual computer knows how to connect to using ip routing so google.com isn't really helpful for anything except for dns really what the computer needs is that ip address so that it can get there now there's a couple tools that we can use to test dns on our system one of the most common is ping right you just ping and see if it can reach it if it can reach it and get a response hey it's working great but there are three other common tools that i want to cover dig nslookup in host and let's actually look at those individually because they all do about the same thing but they do it in a slightly different way now dig is the program that i use most often only because it has a cool name right there's really no reason that i use it other than the word dig is just kind of cool so dig you type dig and then you can just type the host you want to look up like dig google.com and it will use your default dns server and look it up if you want to specify a dns server for it to use instead of the one that your system is currently using which is a way that you can test your particular dns server you you put the at symbol and then the ip address of the server that you want to query now the other two programs can do the same thing they just do it in a different order you can say nslookup and the host you want to look up and then if you want to specify a server again you don't have to but if you want to then you put the server after that same thing with the host command you put type host the host name you want to look up and then if you want to specify a server you put it there if you leave it off it'll use your default and they all like i said do kind of the same thing so let's look at them really quickly and then i'll talk about how to query different servers and do just a little bit of troubleshooting on your system because testing dns is kind of vital if you want things to work on your system and they don't seem to be so first of all let's look at them all how they function just by default so i'm going to say dig google.com all right and this is the response i get it's queried my default server and we can look my default server is listed right here 8.8.8.8 so that's pretty awesome and this is the response it gets there's a bunch of a records for google.com now let me clear the screen we can do the same thing with ns lookup google.com and this is going to give us a little bit different format right it says our server right here this is the oop i missed i highlighted the wrong line that's our server the same server of course it's our default server and it gives us all the addresses these are all the a records a little less detail but it gives us the same information now if somebody tells you that nslookup is deprecated and it's not used anymore and it's going to be abandoned if there's a weird case with nslookup that that's true until it wasn't they were going to get rid of nslookup and replace it with dig but then they came out with bind 9.3 and for some reason decided that ns lookup was going to stick around so we have nslookup and dig and lastly let's clear the screen again we can just say host google.com and this will give us information as well it actually doesn't tell us what default server it's using but it gives us a lot of information these are all of the a records for it these are all you know the ip addresses this one actually also tells us the mail handlers the mx records for google.com's domain which is kind of interesting as well so just pick the one that you like the best and that's what you can use and the reason that it's important to pick one that you like and use it often is so that you're comfortable with it right like i said i usually use dig and that's weird i know because it's the least like common way to handle specifying a server right the other two you do you know like nslookup what you're looking up and then a server with dig you say dig at and let's say 127.0.0.1 that's localhost right so dig at localhost for google.com and we got a response okay google.com and it gave us this address which is different but it came from our local server now it did not get all the same responses that we got from the 8.8.8.8 domain server you know our default domain server if we were to say dig at 8.8.8.8 google.com we would get a totally different set of ip addresses so that's why it's really important to be able to query different servers so that you can get different responses because on this computer if we try to ping google.com it's going to try to ping that other address and time to live exceeded it's not ever going to work so there's something wrong with the dns server running on localhost now i'll show you what the problem is the hosts file i've actually sabotaged here this is the etc hosts file this is where your computer will look first for any dns lookups and sure enough i've kind of bamboozled it here i put the the wrong ip address for google.com so i put that in there now if i restart our local server now if i try to ping google.com it's going to be just fine because it's going to actually use the real address that it looked up on the internet but that's how you can use the different tools to specify not only what host you want to look up but what server you want to query when you look it up so whichever one of these makes the most sense to you or whichever formatted results you like best just pick one and use that because you can specify a server for each of the different commands but knowing how to use them is important because troubleshooting usually means querying more than one dns server so you can figure out what the heck is going on one of my favorite things about linux is that everything is configured with text files right it's just plain text files it's awesome and the network is no different it's configured with text files now there are a few differences when it comes to distributions if you're on debian or if you're on centos there are going to be some configuration files that are different because they configure their networks different now there's nothing wrong with being unique i really like the differences in the distributions but there are some files that are consistent regardless of what distribution you're using we're going to call those the common files and that's what we're going to look like look at in this nugget these files here are consistent across the board they're just kind of standard linux files so i want to show you what they are and and where they live and how they are configured and what they do and that's again just the common files there's going to be specifics in the different distros but we're just looking at the files that are common to linux in general in this case we're going to look at the files in ubuntu and the first one i want to look at is the etc hosts file so let's actually become root if we want to edit them we have to be root to do so so i'm going to look at etc hosts there's no extension at the end it's just etc hosts and this is a file that acts kind of like the first resort dns lookup so before a system even looks things up via dns it looks in here and there's a couple things here like localhost and ubuntu this is our hostname are set up in here but we can add something we can say what if we wanted to make sure users were never able to go to google we could say okay 127.0.0.1 which is our localhost is now going to be google.com so if somebody tries to go to google.com they're going to try to hit our local machine and we don't have a web server at all so it's just going to error out so if we save that now on our computer if somebody tries to go to google it's going to fail so google.com unable to connect even though our network is working just fine we could go to yahoo if we wanted it's just a matter of that dns entry in the et cetera hosts file that we kind of broke all right so another place we can look is in the etc ns switch.conf file now this configures a bunch of things on our system like group and password files but what i want to show you specifically here is the hosts line so this is saying where does it look for dns lookups to you know find what the host's ip addresses are the first one is called files this points to etc hosts if we have this listed first that means before it queries a dns server it's going to query that file it's going to look in that file first and then in order these are what it's going to do from there it's going to do you know the mdns for local lookups and then it's going to use a dns server so this is the order that it's going in but the first one it looks in is the files specifically that etc hosts file so you can change the order of that i don't recommend changing the order because that's kind of what we wanted to do but one more thing i want to show you is etc resolve.conf and this is a little bit of a confusing file because while this tells the computer what name server to use you'll notice up here it says do not edit this file now back in the day you would just put name server and then the name server you wanted to use on your system but now this is all handled with the systemd resolve d service and and it creates this file on the fly so really we do not edit this at all network manager handles all of the name server entries in here um or whatever your distribution uses for configuring the network so we don't actually edit this but if you want to see what server your your particular computer is using you can look in here and it looks like it's it's querying 127.0.0.53 which is another ip address to our local computer so that means that it has some sort of local caching dns server and it's not querying directly to the internet now you probably noticed that all of these common files have to do with dns and that's fine but it's it's interesting that dns is kind of commonly configured across the distributions when it comes to actually configuring the individual network devices that's drastically different from one distribution to the next but the common files are generally dns files and that's what we looked at in this nugget ubuntu is one of the most common distributions out there but it's based on and built upon debian so when we look at configuring one it's very similar to configuring the other and that's the case with network files now i want to show you where to find the different network files on an ubuntu or debian system but it's important to note that in the last iteration of their long-term support for ubuntu that's when a change has been made to how the network is configured and what files are used and how those files are configured now there is one commonality between them and that's if you use network manager to manage your network instead of editing the configuration files that hasn't changed you can still use that across all the versions that are current with ubuntu and debian but these two ways have changed so first i want to look at the older version of ubuntu and when i say older i mean one that's still valid to use as of the date today you can still use version 1604 let's see what version this is etc os release so this is ubuntu 1604.1 lts long term support and this is still as of today a valid version of ubuntu to use in production but this is configured by going into the etc network directory and there's a single file in here called interfaces so if we look at interfaces we're going to see this is how you configure a network interface with an older version of ubuntu or debian now this is pretty much the stanza that you configure there's usually something in here you can look at and base it off of but this is the format for it we're not going to go through the you know configuring different things it's pretty straightforward you don't have to indent this is just for ease of readability but this is how you configure a network interface via the configuration files now we could still use the um network manager if we were on a gui system like this is a gui system we could use that but i'm actually ssh into a server that doesn't have a gui interface and that's where we use these network files to configure it so let's quit this one this is how you configure an older version but a modern version like 1804 or the most modern version of debian uses a completely different network configuration system now here we're on a newer version of ubuntu this is actually 1804. let's check that out so this is 18.04.1 another long-term support release of ubuntu and this one uses the net plan system for configuring network interface so if you go into etc net plan and you look in here depending on what kind of system you have you're going to have some sort of file in here it's either going to be like zero one networkmanager.yaml or 50 cloudanit.yaml if it's a server the thing about yaml files though let's look at this yaml files do depend on indentation in order to be properly configured so while with the etc network interfaces of the old ubuntu system it didn't matter how things were indented here it really really does and so here this is where we would configure like a static ip address and we you know here's our ip addresses gateway name servers the format's a little different but it's fairly similar conceptually to the old system like i said except for that white space that's really important that it's indented properly so if you make a change to this then you just would do sudo netplan apply and then it will activate the new changes that you made now i do want to point out again this is a gui system so it has network manager but there's also nmtui which is network manager text user interface and then you can edit interfaces using just the text boxes on your screen if you don't have a gui system installed so that's just one more way that you can edit your file in ubuntu or debian if you don't have a gui but you want to use network manager as opposed to just configuring those files on your own whether you have an older system using etc network interfaces or a newer system using configuration files and etc netplan configuring the network on debian and ubuntu is not difficult and you know it's usually just a matter of changing what's existing but if you want to use the network manager to manage all those interfaces either using gui or text tools you can do that as well it's very flexible and even with the change in versions it's not a whole lot more difficult to do one over the other if we've been learning together for a while you know that i prefer debian based distributions and ubuntu is my jam but i have to admit when it comes to network configuration files centos or red hat they really have an elegant solution for how to configure the network and there's no confusion between if you're using the gui network manager or if you're just using the configuration files in the etc config folder let me show you what i mean because centos is just so awesome when it comes to network configuration now here we are on a centos system and it doesn't matter which way you configure the network interface you can go up here and go into the network manager to configure it or we can go directly into etc sysconfig and i just wanted to show you quickly inside the sysconfig folder and sent to us is where all of these configuration things are this is kind of unique to centos and red hat they put most of their configuration inside this etc sysconfig folder but inside here there's also another folder called network scripts and that's where our networking is configured if we look in here we're going to see there is this configuration file for our interface ifconfig dash eth0 so let's look at that if we edit this it's going to change the way our network interface comes up like if we were to change boot proto from dhcp to static and then we were to add a line ipaddr equals let's say 10.10.10.111 which is not currently the ip address if we were to do that and save it and then just do a quick sudo service network restart we would be able to restart our service and if we were to do ifconfig we're going to see scroll up here sure enough that is our ip address now now there are a couple problems we didn't add like netmask or anything so let's go up into network manager wired settings click on the configuration thing so we're to edit this in network manager if we go over to ipv4 we can see it went from it used to be dhcp now it's manual this is the ip address i put in and it just guessed this netmask because the 10 range is a class a ip address so it guessed that i wanted to use a class a netmask but i don't i want to use 255.255.255.0 and i didn't supply a gateway so let's do that now 10.10.10.1 and i didn't supply any dns server so we can do that here 8.8.8.8 and now we'll save this we'll click apply if we go back and edit that script sudo vi we're going to see that some changes have been made now this stuff is still the same boot proto equals static ip address but look down here it added the gateway address that we added it added the dns server that we added and it added the prefix 255.255.255.0 translates to 24-bit netmask so put that in there just use a different form and we can see that it made the changes to the actual configuration files rather than having like two different systems and you have to decide which one you want to use centos allows us to make changes in one place and whether we're using the gui interface or just this text-based interface it's going to allow us to make those changes i love that it has all the change places in a single file as opposed to conflicting and trying to decide which method you're going to use if you use network manager you you're using this configuration file if you don't use network manager you're using this configuration file so it's a really elegant way to handle network interface configuration like i said at the beginning i'm a debian ubuntu man that's what i usually use in my server situations but when it comes to configuring network interfaces man centos has really stolen the show they have an elegant way of doing it either using network manager gui tools or using the scripts inside the network scripts folder of etc sysconfig network bonding in linux is really just a way to utilize computers that have more than one network port and most servers nowadays come with that let's say you have a server and it has three network ports built into it well if you have to connect it to a switch all each of these connected to a switch you would ideally like to use all of that bandwidth and you don't want to have to supply three different ip addresses to the computer you just want to use all of the available bandwidth on all three of those wires and that's where network bonding comes into place now there's basically two different kinds of network bonds that we're going to look at those that require special switch support and those that are generic and don't require the switch to know at all what's going on now linux does provide some pretty cool options when it comes to there but the options can be a little overwhelming i mean this screen is like oh my goodness there's so many choices well that's okay because your choices are going to be fairly limited once you learn what all of these are now the first thing i want you to look at when we're looking at this list is does it require switch support and what i mean by that is a lot of switches especially layer 3 switches or smart switches they're sometimes called will support link aggregation and different vendors call it different things but it's going to be either link aggregation or lacp or ether channel the idea is a smart switch will have built-in code that will allow them to work together so if you have a smart switch chances are that you have the ability to use link aggregation now some of these require a switch that supports that i'm going to start with the confusing one here so balance round robin basically how balanced round robin works is you have multiple ports and it says okay when i transmit packet 1 i'm going to use this port packet 2 is this port packet 3 is this port packet four is this port packet five is this port packet six is this and it just keeps going through packet by packet and transmitting across all of your interfaces now the reason i said it sorta requires switch support is if you're plugging these into a switch it does require a switch that supports link aggregation but what a lot of people do is they'll use this balanced round robin and they will connect two servers together with crossover cables and actually with gigabit we don't need crossover cables anymore but they'll connect two servers together and they'll use balanced route robin and in that case you don't need a switch support or a switch that supports it because there is no switch involved right they're just directly connecting the two computers together and so like if you have a file server that you want to connect to your other server a lot of times this is a way that you can increase throughput without requiring any special switch support so if you're connecting directly computers you don't need to have switch support if you're connecting to a switch you do all right that's mode 0. mode 1 is an active backup and this one's pretty easy to understand basically however many ports you have on your system only one is going to be active and if that one fails then another one is going to turn active and that's just how it works they always have one this is fault tolerance but this doesn't speed anything up it just means if one fails another one's going to come online all right this doesn't require switch support because you know it basically just only uses one port unless the next one fails and then you switch to that port and the switch is like okay so now we're going to use this port great uh balance xor does require special switch support and this is kind of cool how balance xor works basically you have your computer here your linux computer and it does a hash based on your mac address and the client's mac address and so here's another client over here and basically it says okay based on this hash i'm always going to use port 1 to connect to this client and then over here the hash is different so i'm always going to connect to this computer so it's basically a way to just spread out which computers use which port but this computer is always going to use this port and this computer is always going to use this port now this isn't used very often because if you have switch support which is required chances are you're going to use 802.3ad which is the industry standard link aggregation protocol right this means that your switch knows what to do and your client knows what to do and it's just a really smart way of increasing through throughput and availability fault tolerance so if your switch supports link aggregation you should use 802.3ad or mode4 which is the industry standard okay so even though balance xor is cool it's not very often used because if you can use it you might as well be using the one that's even better broadcast i kind of skipped over that broadcast is only in very specific cases that you would use it it just takes all of your ports on the server and spews all of the data out all of the ports at once it's not used very often and it definitely requires your switch to know what on earth is happening now the two i do want to focus on a little bit are these bottom two let's say you have a dumb switch it's not a switch that supports link aggregation it's just one you bought from amazon for 40 bucks it's gigabit but it doesn't have any smarts built in linux is smart enough to be able to utilize the dumb switch and increase bandwidth and throughput and reliability and there's two different ways it does it balance tlb which is balance transmit load balance and the balance of all load balancing and what this does is let's say we have four ports on our switch okay with tlb it's going to transmit from whichever port is currently the least busy so all the transmit is going to be balanced out now incoming is still going to always go to one active portent so that's not as good as alb which is the same concept except it does it for incoming and outgoing so basically the least busy port gets the traffic and this is really brilliant and how it does is it constantly changes the mac address on these ports or on these ethernet cards so the switch is like oh you moved again oh you moved again oh you moved again and from the switches standpoint it doesn't care how many times you switch the mac address so it generally works just fine now some people have issued with this i've used this in production for years and never had a problem so balance alb if you have a dumb switch i highly recommend you use mode six okay if your switch supports link aggregation i highly recommend mode four because that's the industry standard and if you're just connecting two servers together with multiple cables my shirt mode zero works really great to be honest the most difficult part of linux network bonding is figuring out which mode to work but really it's not that tough of a decision if your switch supports it use 802.3ad there's really no reason not to if your switch is a dumb switch and doesn't support it i highly recommend mode 6 or balance alb if you want to take advantage of bonded network interfaces but you don't have the expensive hardware that will support it once you know the type of bond that you want to set up for your two ethernet or two or more ethernet connections on the computer configuring them is pretty straightforward although it's drastically different depending on whether you're on ubuntu or centos so i'm gonna show you how to do it on both and then i'll show you also how to test it to make sure that the bonding modules are working so here i am on ubuntu i've actually already configured it so i've gone into etc netplan and i have my file in here so let's look at this file and what i've done is i've set up the proper yaml format with indentation and everything to set up bonding so i'll go over it really quickly first of all we need to use network d as our renderer we can't use network manager in order to configure bonds because it just doesn't support bonding we do have to define the ethernet cards themselves so ethernet eth0 i set dhcp4 to false because i don't want it to actually assign an address to eth0 after all that is done we don't need to set up anything more for the ethernet ports themselves then we need to set up a bonded interface so a new section here bonds the name of it is bond 0 dhcp 4 is false because i'm going to assign a static ip address and the interfaces is a section that we tell it what interfaces are going to be a part of this bond in our case we only have one which is silly i mean we're only bonding one interface but you can do that one or more it's just silly that we only have one to bond that's just all this computer has i've set up the addresses the gateway the name server is set up just like it would be if we were setting it up for eth0 really the only new difference down here is the parameters and the parameters what i've done is mode active backup that's just one of our modes it happens to be mode one but we actually say active backup we specify by name the mode here and then we save this file and do netplan apply now how you can tell if it's working we can do ipadd to see that you know sure enough it's up here's our bond zero interface here's our ethernet zero interface which is up but doesn't have an ip address and another way you can test is to look at so we're just going to cap the file it's in the virtual file system proc under net bonding bond zero so if we look at this file it's going to tell us the information about it and say bonding mode is fault tolerance active backup primary slave none but the current active slave is eat zero it's our only one that we have it's up and it looks like everything is working good here's our slave interface eth0 down here and that's all working well as well so we have it all set up and it's working now with centos the setup is a little bit different because centos is set up differently in centos in order to configure our ethernet ports we go into etc sysconfig network scripts and in here we have our ifconfig files so i'm going to first look at the changes we have to make to ifconfig eat zero so here we basically we make it pretty simple ethernet type product boot protocol is none uh the name the device is zero on boot we want it to come up but notice i haven't given any ip addressing information and here are the two things the master is bond zero and yes this is going to be a slave to this bond all right so those are the changes we make here if we look at if config bonds zero this is where we set up bond zero we name it bond zero bonding master is yes i did give this one an ip address and a prefix which means the subnet mask on boot we want it to come up a boot protocol is none and then here the bonding options we tell it what we want it to do i actually set this one up to mode 6 mode 6 is balance all load balancing so it does its own version of load balancing for incoming and outgoing traffic again i only have the one interface on this computer but that's okay let's do the same thing now checking to make sure it's working is similar we can do ipadd we can see sure enough here's the bond set up right here and our ethernet is right here and really for configuring the bonds themselves that's all there is to it ubuntu and centos are configured differently but underneath they're both doing the same thing they're using the bonding kernel module and they're allowing you to do some awesome things with multiple ethernet ports on your computer gpt and mbr are two different ways of taking a hard drive and chopping it up into pieces so that those pieces can be recognized and mounted as different drives and stuff on your on your system so they're they do the same thing but gpt is much newer and much more feature-rich than the old-school mbr now there's also a really cool thing called protective mbr i want to talk about but basically we're going to talk about the differences between gpt and mbr here i have two big squares and i'm going to say that these actually they're rectangles i'm going to say that they represent the drives themselves and they're going to be partitioned with two different systems so we'll start over here with the old one mbr which stands for master boot record okay now how this would work is let's say this drive is device sda okay so this is the device on the linux system it recognizes this drive now the mbr is a little tiny bit of reserved space at the very beginning of the disc and this is basically like a table of contents and it describes how this is chopped up okay so this is chopped up into let's say four pieces there's four primary partitions you can also cut each one of the primary partitions into four so we could have you know mult we could have this one be like have four partitions there if we wanted but basically what this means is once the mbr defines in this little first part of the drive right here what is what we come up with other devices on the system we have dev sda1 devsda2 and it goes on down the line and so each of these chunks is referenced on the system as their partition name now this is the raw device dev sda doesn't go anywhere it's still there but the individual partitions get their own device and that's how we reference them on the system so mbr has this you know this little tiny table of contents at the beginning and it says where things are chopped up now gpt or gui guid partition table which i think is interesting because there's an acronym wrapped inside of an acronym this stands for globally unique identifier partition table and basically it does conceptually the same thing right it will chop up this drive and you'll end up with dev sda for the raw device and so on down the line the differences are though gpt does have a spot at the beginning with the table of contents but then it also scatters them copies of them around the drive so that if something were to happen with one of the copies of the gpt it can still figure out what's on the disk it also does some crc correction on the files itself or on the the file system itself so that it knows if there's some corruption and it can fix it it's just very very robust along with the ability to use much bigger drives mbr is limited to two terabytes gpt is limited to i think petabytes i mean there's no practical limit at this point of how much drive space it can talk about rather than being limited by four partitions you can have tons of partitions on here so you know we could chop it up into as many as we wanted and it just goes down the line sda1 sda2 sda3 and one other really interesting thing and i wanted to talk about is the protective mbr so gpt is used on newer systems it's specifically part of uefi that replaces bios but biosystems can still see gpt drives and part of the reason they can do that is at the very beginning there's this section that looks exactly like an mbr basically it's an mbr record here that says okay there is one big partition on this drive the entire drive is a partition there's no room available on this partition and it takes up the entire drive now why would that exist it's basically so systems don't say oh well there's no mbr in this drive so this must be a blank drive with nothing on it well we don't want that to happen so basically what it means is it allows the mbr system or a bio system that is used to seeing mbr it will allow it to realize that this is not an empty drive it's just something it can't read and then there's some really cool stuff where part of this initial mbr can be hints to the operating system itself as to what's happening underneath and so some older bio systems are able to boot from a gpt drive even though it shouldn't be able to because the linux system or you know the operating system itself says okay i see mbr is saying that um you know it's really a gpt drive but i know enough about gpt myself that i'm going to be able to find the software and you know the the drive files and partitions on here myself and so you can actually use it on there but it's because of that protective mbr that all that is possible so when it comes down to it mbr is the old way of doing things gpt is the new kid on the block honestly there's no reason not to use gpt it does everything mbr does it does it more efficiently more reliably it's just the better way to go so i recommend that you use gpt and while there are significant differences a lot of those backwards compatibility issues make it so that you don't even have to worry about them the linux file system is actually really cool because all of the network mounts and different hard drives and usb drives are all on one giant file system it's not there's no a drive b drive c drive d drive there's nothing like that it's just all one big file system hierarchy and we're gonna look at the different things because whether it's a real or virtual file system it's on the same file system whether it's relative or absolute this is just how you traverse the actual file system we're gonna look on the command line about that network mounts they're just you know on a remote system but they just show up as a folder on your local computer it's just it's really cool so first of all let's let's start at the very beginning it's a very good place to start so we have a hard drive right so we're going to say it's dev sda1 and that has the root partition on it you know just forward slash so this is like where everything starts the root partition is the basis of our linux file system it's it's on a hard drive of some sort in our case it's this one now inside there there's tons of files and folders some of the files are directly on that dev sda drive right on the root file system or folders within that root file system they're actually stored on that hard drive some of the folders and the files in there are a virtual file system if you look inside the root directory there's a proc folder an assist folder these are dynamically created file systems that are just a way to interact with the kernel itself so if you want to make a change to the running kernel you can make a change to one of the files in these folders and it's going to affect the running kernel but it's not actually files it's a virtual file system it's just a way that they represent that interface with the kernel is this virtual file system then we also have like a remote nfs server it could be nfs or samba or you know whatever your network protocol of choice is but it's mounted on the folder inside of your root system so this could be like the home folder on your system might not be on the actual hard drive it might actually be on a remote system but it's mounted inside your file system and from you know from a layman's position just looking at the system you don't know if it's a remote system or a local system or a virtual system because they all look the same same thing when you insert a usb drive it doesn't come through as like you know like the e drive or the f drive like on a windows system it's just mounted somewhere on a folder inside that same one solid monolithic file system even if you put in another hard drive so this is we're saying this is dev sda1 like the partition one on the sda drive even if you were to put in a second hard drive in the system it's still going to mount inside a folder or a subfolder of this root drive right it's gonna be like in mount data or wherever you happen to mount it it's just going to appear as a folder inside this root file system so everything is inside that root file system it's really a neat way to handle all of the various things that can be stored in linux and the virtual file system is very unique in that it's not really a file system it's more like an interface designed as a file system so you can interact with the kernel itself alright so let's go to our file system here and i just want to show you kind of how it looks and i want to talk about this whole absolute versus relative thing all right so i'm just going to show you the file system here we have ls and we have a folder called pictures we can do ls picture gotta spell it right ls pictures and i have a folder cbt gold which is this background and then a trips folder uh what if we were to do ls trips okay there's a folder called grocery store mexico orlando now there's a cool program called tree i'm going to use so let's say tree pictures it's going to show us a tree representation of all the files and folders inside here so we can look we have the pictures folder there's a one file in there called cbt gold then there's a folder called trips inside that folder are three different folders grocery store mexico orlando inside each of those there are files that are stored in there this is the hierarchy of that now honestly this mexico folder this could be like a remote nfs share we don't know because it's all on the local file system that is you know mounted everywhere on the system it's just one file system so we don't know where these are we just interact with them as if they're all one local file system so that's just a really cool thing about it but this whole absolute versus relative let's go over to the pictures folder and trips i'm going to clear the screen and if we were to type ls minus a we're going to see all of the things in here now we already saw some of these orlando mexico grocery store but these dot and dot dot now these are special folder entries that mean the dot means the current folder so if i were to say cd to dot i'm in the same folder dot dot means the directory above me so if we are in trips dot dot is going to be pictures so if we were to say cd dot dot all of a sudden you'll see now we're in the pictures folder that's pretty cool right now we can use that as a folder name anywhere so let's go into trips orlando okay and inside here let's do an ls minus la we can see here are all the the pictures that we have and then these two folders dot dot and dot in every single folder you're gonna find a dot and a dot dot because it's just a pointer that means this directory the directory above me okay so we can put that in a string too let's say i want to cd to dot dot dot dot and see where this takes us two pictures now why did it do that well because dot dot if we're in orlando dot dot means trips and then we're in trips the second dot dot means pictures so that's interesting we could do something really complicated and i'll end with this we could say cd trips orlando dot dot grocery store oh okay what happened there and we'll end with this one like i said so we started in pictures and we went to the trips folder orlando back to the trips folder and then into the grocery store folder and sure enough that's where we ended up pictures trips grocery stores so you can use dot dot anywhere in your thing to talk about the relative path of where you're where you're going okay now the absolute path of this is going to be cd home bob pictures trips grocery store and then sure enough we're in the same folder this is the absolute path starting at the root level but we can use relatives paths using things like dot dots one other thing i said i was going to end there and i guess i lied to you this tilde image or this tilde thing is a shortcut for your home directory so if we were to say cd to the tilde which is usually to the left of your one key boom now we are in the tilde which if we type pwd we'll see is home bob all right so those are the relative path tools that you can use to construct where you want to go but that's really how it works and the tree program is just kind of a cool way to look at things you can see everything else spelled out there the big takeaway is that the linux file system is just one big monolithic file system and whether you're mounting network shares or you're mounting usb drives or second hard drives or even a virtual file system to interact with the kernel it's all under the same bunch of file system in this big hierarchy partitions are really just organizational units that are on a hard drive you kind of chop a hard drive up into different partitions and those partitions are used for different things like swap space or you know a particular folder on your system you can use partitions for a lot of things and depending on the system that you're using whether it's going to be an mbr master boot record or gpt gui id partition table you can have a multitude of partitions or just a few and depending on which scheme you use it depends how big your hard drive can be now we're going to look at a handful of tools like parted and g parted and this stands for partition editor and graphical partition editor and the old school f disk which is just a command line tool that didn't used to handle the gpt partition table but now it does so there's this one size fits all tool called fdisk and then i do want to show you a couple tools that will let you see what block devices are available to partition on your system so let's actually go right to the command line so that we can check out what's on our system and i chose to use centos today for no other reason than i thought let's use centos so i want to show you first of all how we can identify the block devices on our system we can type lsblk and it's going to show us this really nice little cool tree thing right so we have fd0 floppy disk which actually is a lie there's not a floppy disk on this system it's a virtual machine but then sda is the main hard drive on the system and it's partitioned into two partitions the sda1 which is the boot partition and then sda2 which is used in lvm which we'll talk about in a different nugget then there are a bunch of other hard drives installed on the system that don't have anything on them at all there are four 10 gigabyte drives sdb sdcsd and sde and we can also see these things if we look at proc partitions this is again the virtual file system that shows us an interaction with the kernel and we can see sure enough here they are sdb c d and e and this is how big they are the number of blocks so they're 10 gigabytes and of course if we just looked in the dev folder and gripped for sd we would see all these devices too right sd b c d and e so what i want to do is just partition one of them we're going to pick sdb and there's a couple ways we can do this we could use g parted now notice i'm root we have to be rude if we're going to do the system level stuff like partition a hard drive but we could say g parted and it'll start up and scan for all the drives available and there's a little drop down here sdb very cool we would have to start by creating a partition table and then we could create a partition with gui tools it's really really easy to use g parted but a lot of times the system doesn't have a gui so i don't want to do it that way now we could also use parted or partition editor this is the cli version of the program that we just used we can type help and it'll show us it does the exact same things it's just used you know use text to tell it what to do all right so i'm not going to use this one either but i want you to know that they exist and if you're more comfortable with them they're a little bit newer and they have a few more features and that sort of a thing but what i want to show you is the old school f disk and i want to show you this because if you're on a server it may not have partition editor you know part ed or especially g parted the graphical one so i want to show you what almost every linux system is going to have and we're going to use that one to actually do our work so we're going to say fdisk dev sd b we have to tell it what device we want to use so it's a good thing we know how to find the available devices on our system type this and type m for help now we just have these single letter commands but that's okay the first thing we're going to need to do is to create a partition table and it can be one of two types we can create a gpt partition table by pressing g or we can create the old mbr or ms-dos partition table using o oh i'm just going to pick since it's not over two terabytes in size either one is going to work so let's just use gpt for the heck of it so i'm going to say g press enter it says okay it built a new gpt disk label this is its id and now we have a partition table but no partitions to create a partition we can actually type p to see what's there there's gonna be nothing there so to create one let's press m again so we get the whole help screen we can say n for add a new partition so new and now it says partition number one through 128 right this is not mbr mbr would only support one through four this is gpt so one through 128 i'm just going to pick the default first sector the default is going to be the first one available last sector the default is going to be the last one available right this is our range and it chose the last one so i want to fill up the entire partition or the entire drive with this one partition so if we type p now we're going to see sure enough we have a partition there it's 10 gigabytes linux file system type that's okay and now to quit let's press m again for help to quit we could just press q and it would quit without saving changes but we actually want to make these changes so we're going to make the change to the disk so we're going to use w to exit which means right so w enter and now our partition table has been done we can say lsblk and look at that now it shows up that we have a partition created on our system it happens to be a gpt partition table and it's a single partition that takes up the entire drive i didn't actually go through the process of using parted or g-parted they're very straightforward and they walk you through it and the help screen is right there and easy to use so if you want to use those by all means go ahead but i wanted to show you the more complicated f-disk because it's going to be on every single system that you run across there's a wide variety of file systems that you can put on an empty partition in linux but the idea of all of them is the same you take a big empty partition like a big field and you divide it up so that you can store data efficiently like if this parking lot were an example of a file system each car would be a piece of data that is stored logically in its place and and some file systems have you know bigger spaces for parking buses some have littler spaces for parking compact cars but conceptually they all do the same thing now like i said there's a bunch of different options available in linux ext is the most common family of file systems that you can use on linux it's very mature it's been around forever the later versions support journals so that if a computer gets powered down before the reads and writes are synced up usually you can salvage the data on there so it's fairly robust as well xfs is a file system that's been around for a long time this used to be what you had to use for really really big drives but now everything supports big drive so that's not really the issue anymore it is still used by centos and red hat though so xfs is still widely used interestingly it has its own set of tools like for file checking fixing things manipulating the the xfs file system itself rather than using standard linux tools it has its own set of xfs tools there's a couple others btrfs it's often called butterfs this was the new kit on the block right this has awesome features like snapshotting and it's just awesome unfortunately it's kind of been abandoned which is weird but nonetheless it's kind of what happened so btrfs is still functional but it's not widely used anymore and then of course i'm just going to mention them dos or the windows world has their own file systems like ntfs vfat fat32 things that you're familiar with if you're in the windows world and linux can usually read and even write to most of these file systems but when we're talking about linux file systems we're generally talking about linux specific ones and ext is awesome i'll be honest i like ext there's generally three there's e xt 2 ext3 and ext4 now this isn't necessarily like one is better than the next they each have their own features but ext4 is the newest and has the most features i'll be honest i almost always pick ext4 as my default file system not because it's necessarily better than any other option but because it's so widely used that means there are a ton of tools and utilities and tutorials online to get data back if you have some corruption so ext4 is my file system of choice mainly because it's used in so many places when it comes to creating a file system basically you need a partitioned hard drive so that you can you know have a partition or that empty field in order to draw the lines for your parking lot or put that file system on now i'm going to say lsblk and we're going to see these are the different block devices on our system i have two partitions created i have sdb1 and sdc1 these are just 10 gigabyte partitions on 10 gigabyte drives so it takes up the whole drive and we're going to format each of them all right so i'm going to say oh here's another really cool thing about linux the file formatting or the hard drive formatting programs all start with mk for make fs file system and then just hit tab a couple times and you're gonna see all the various tools for creating the different types of file systems so let's say we're going to make a butterfs file system so we would say mkfs.btrfs and then what partition we want to create that file system on in our case dev sdb1 all right so it's created it it says we have 10 gigabytes idu's one that's the path okay so it looks like it did that without any problems let's create another uh file system so uh mkfs hit tab a couple times so i can see our options now like i said i usually use ext4 so i'm going to say ext4 dev sdc one which was the other partition that i had created on this system and it went through and it created that file system and now a really cool thing if we use the lsblk but we add the dash f it's going to show us the file systems that are on the particular block devices as well so if we do that we're going to see up here we have sda1 which remember i said red hat and sent to us use xfs well sure enough there it is xfs is used sdb1 is that butterfs file system that we created and then down here is an ext4 file system that we created on our system so that's all there is to creating the various file systems and really you can pick whichever one you want but you do have to have a partition in existence before you can create a file system on it and while yes there are a whole bunch of file systems that you can use on linux and it supports a bunch i really recommend that ext family of file systems if you have a choice and you're just trying to decide what one to use largely because it's used so often there's so much support if something goes wrong you can find a lot of help online if we want to be able to access the data that's on the drives or the partitions of the drives that we put into our system we have to mount them into our local file system now we can do that manually using tools like mount or u-mount or we can use the et cetera fs tab file to do it automatically on boot but conceptually what's going on is that we have a new hard drive or a partitioned hard drive and we want to incorporate that into our file system now it's really important to note that when you mount a partition or a hard drive it goes into a folder and that folder becomes what is in that drive now we couldn't mount it on this folder because this folder already has things inside of it it has to be mounted on an empty folder because it doesn't make sense to have all the things in this hard drive in this folder and then have these other folders alongside of it so it has to be in an empty folder that we mount a partition or a drive so we could actually take this hard drive and mount it into this folder if it's an empty folder and that's what we're going to do on our system is find an empty folder and then this all the files inside this drive will become part of our bigger linux file system starting at this mount point now i'm here on an ubuntu system and i have the regular route mounted hard drive and then i also have an additional 10 gig drive and if we do lsblk we can see that we have well a bunch of things here loop loopback devices but we have down here sdb1 okay this is a 10 gigabyte partition and and it's all set up and ready to mount but it's not currently mounted on our system so in order to mount that i actually want to mount it in a folder if we look inside mnt there's a folder called 10 gig and inside that folder is nothing it's an empty folder so i want to mount it on there now if we do blk id we can see a little bit more information about the device itself so here is dev sdb1 now here is the universally unique identifier for this partition uh keep note that this is something that's specified about the partition we'll look at that in a second but we know that it's type ext4 okay that's good to know and we know that this is where the actual partition lives so we can say mount now i'm going to say dash t for type ext4 usually mount can automatically figure out what kind of file system it is but if we know there's no reason not to do that so i'm going to say mount dev sdb1 and where do i want it to mount on mnt 10 gig press enter and now if we go into mnt 10 gig we're gonna see sure enough now there's a lost and found thing this is the root level of that hard drive but now it actually lives in our file system right here we can type mount alone on the on the line it will show us where it's mounted right so right here it's mounted on mnt 10 gig so let me type cd and then we can unmount it by typing umount mnt 10 gig it's not unmount it's umount all right so there now that that's how we manually mounted and unmounted if we wanted to have it mount automatically on boot we would edit the file etc fs tab all right and i'm actually going to stretch this out so we can see everything on here we can see there's already an entry in here for the root file system sda or s yeah sda1 but notice it doesn't specify it by its device it specifies it by its universally unique identifier which we could get by using that blk id command it'll get us that uuid for the particular partition now we could use the uuid for a dev sdb1 and we could put that in here to mount it but i'm actually going to use let me make another entry here i'm actually going to say that the file system is on dev sd b1 we can specify it by device or by uuid or we could even use the drive label if we want but in this case i'm just going to use the the device itself and then tab over the next field here is the mount point so i want this mounted on mnt 10 gig tab over the next thing is type it's an ext4 options i'm just going to say defaults and then the last two fields are dump and pass now dump is an old school backup program it used to dump the files to a backup this is really deprecated it's not really used anymore at all so dump you're going to want to put zero for dump the last line though pass this means do you want it to run a file system check zero means never run a file system check one means run the file system check first so you put a one on the root partition any other partition that you wanna have checked when the system boots up you're gonna put a two so i'm going to put dump of 0 you always put 0 for dump and then pass is 2. it's the second most important because the root is the first one you want to have scanned and then everything else is going to be 2. so you can have like five different partitions mounted they could all have pass of 2. now there's more to it to get it to scan automatically on boot but when you're setting up the fstab file this is where you do it so save this and now we could just type mount minus a and it's going to mount everything that is specified in fstab and we can look by saying mount and it'll show us that sure enough it remounted that because we defined it in fstab and if we reboot the computer it's going to automatically boot it or it's going to automatically mount it as well it's really easy to mount partitions using the manual tools it's also pretty easy to use the fstab file to specify it and you can either specify it by device name or the uuid that we can find out using the blk id program to scan a linux file system generally you use the tool fsck or just fisk as it's often referred to now the real key though is to have it scan automatically periodically on boots so you don't have to manually do it because here's the deal in order to run fisk the file system itself has to be unmounted that's not really a problem for secondary or tertiary drive mounts like the home directory or something like that but the root directory it's pretty difficult to unmount the root directory and scan it unless you're in the boot up process or you've booted from a cd or something so i want to talk specifically about how to set up the system to scan the the file system on boot including the root file system so that you can you know have it automatically maintain itself now when it comes to scanning automatically there's a few different flowchart things that go on the very first thing the kernel looks for is inside your fs tab file if the pass setting is set up if it's a zero you know if you have your your dump and pass and if the pass is set to zero then it won't scan it just absolutely will refuse to scan it doesn't even look any further it just stops right there and continues booting up the system if however you have that partition set up with either a one or a two a one for the root partition a two for any other kind of partition if you have it set up then it looks at the drive itself and it says okay has the maximum number of allowed mounts been reached and if that threshold has been reached it will scan the drive before it boots up if not if it hasn't met this maximum yet then it's not going to scan it's going to increment it's going to say okay i'm going to mount one more time and add it to the number of times i've been mounted but i'm not going to scan it even if the pass is set up to scan it's not going to scan it if it hasn't met the max now the max by default is negative one which means it's never going to scan because that's just the way of saying like don't ever automatically mount so by default you're never going to get an automatic scanning which is a little bit frustrating because you do want to have your system automatically scanned so on ubuntu here i have dev sdb1 mounted on mnt 10 gig okay so this is another partition this is not my root partition but it's mounted on mnt 10 gig all right if i look inside of the fs tab file we're going to see that on this partition that i have automatically mount on boot it's set up with a pass of 2 which means that it is going to check to see if it should scan automatically okay it's not the root partition so i don't have it set up with a pass of one but since it's set up with two it's going to check and say okay if it's time i'm gonna scan this so let's get out of here and how we can see what the maximum number of allowable mounts before it will scan is is to use the tune 2fs we're going to do dash l for a listing of dev sdb1 and it's going to show us this now what you want to look for in here is this maximum amount count remember i said by default it doesn't ever scan that's because this is set to negative one and you're never going to reach that because that's just the way of saying disable it if we want to have it automatically scan every so often we need to change this so we would say tune 2fs c for count and i'm going to say every 10 months i want it to scan in dev sd b1 okay so now the maximum amount count is 10. we can look at that by doing the same command over and we can see now the maximum amount count is 10. okay so what does that mean well every time the system boots it mounts the partition well we could speed that up we could say u-mount dev sd b1 mount dev sdb one we do that and now if we look it's going to increment it by one because we unmounted it and remounted it so if we do that a bunch of times and now we look back and see okay our mount count is now 16 and our maximum count that is allowed is 10. well why didn't it automatically scan well it only does that on boot so if we were to reboot the system it would go through the flowchart and it would say okay you have your pass set to two so that means i need to check the drive and say okay drive is your mount count higher than the maximum amount count allowed before a scan and it will be so if we do a reboot and once it's booted back up we look and run that tune 2fs again so sudo tune2fs-l dev sdb1 now we're going to see that the mount count is down to 1 which means that it scanned it it ran fisk on boot before it was mounted and it reset the mount count to zero and then of course it mounted it so now it's one so now it's not going to rescan that on boot until the mount count gets above this maximum mount count and then it will do it automatically and figuring out what number to set here can be kind of tricky because here's the deal if you're on a laptop you might reboot fairly often so you're going to want to have that number kind of high so it doesn't scan every single time but if it's a big file server maybe you rarely ever reboot it maybe once a year well in that case you might want the mount count to be very low so that every time it gets rebooted it does scan for consistency's sake and then one more thing if it's a huge partition with just millions and millions of files it's going to take hours or days to scan so maybe you don't want to scan it every time it mounts it really depends on your situation so plan out how often you want it to scan and the type of situation that you're in to make the best choice for your particular partition and honestly the best choice for you might actually be just to run fisk manually just unmount the partition or boot from a cd or usb drive if you need to scan the root partition maybe you never want to automate the process you only want to do it manually for some cases that's fine too regardless it's important to know how the system works so that if you think you're automatically scanning you really are lvm or logical volume manager is basically like a software version of a storage area network it allows you to take a whole bunch of physical devices and lump them into one big group that allows you to kind of carve out slices of storage for the use in your local system it consists of a bunch of parts physical volumes volume groups and logical volumes they use the word volumes a lot in there but nonetheless it's a way of taking raw storage and combining it into a thing that you can just slice up and expand and contract and add more things to it without disrupting the existing services here's how it actually works or looks in practice so we have we start with physical volumes i'm just calling them pv and we're going to say that we have four 10 gigabyte drives now in the standard practice these could be actual hard drives they could be raid devices or they could be partitions on a hard drive it doesn't really matter what they are they're just chunks of storage and you create physical volumes out of them and then you combine those physical volumes into volume groups and so then all of these are combined into a volume group and if you add 10 times 4 you get 40 gigs so then you have this volume group which is like just a big bucket of storage and that bucket of storage has no protection now i know that's not like a feature right but i really want you to know that if you just have 10 gigabyte drives all in a bucket so you have 40 gigabytes of storage and you're just gonna take and carve out chunks of that that doesn't offer you any protection so if one of these drives fails all of a sudden you could have your end result be completely corrupt and useless so physical volumes being raw devices and not having any redundancy is a little bit scary so anyway just wanted to throw that in there once you have this volume group you carve out a slice of it and it can be a small slice like here i just said this is about seven gigs i just kind of like spatially guessed how much of 40 gigs that would be um but you can carve out a big chunk so you could have like a 30 gig slice or you could have like a 32 gig slice that would use like two plus a little of another drive basically you don't know underneath what's going on the volume group is just a big chunk of storage that you carve out a slice of and then this slice is called a logical volume that logical volume is what you format with a file system and mount on your local hard drive or on your local file system and what you install linux on so it's a long step process but it allows for a lot of flexibility and so i want to show you what it looks like in practice even though it's not this robust system with multiple physical volumes if you have centos installed they actually use lvm even if you only have one drive so i'm going to show you here if we look at the etc fs tab file what we have here is our device is called dev mapper centos dash root okay now this is like devmapper what is that well this is where logical volume manager creates those logical volumes for us to use and you'll see here this devmapper sent to us root is mounted on the root of the hard drive or root of the file system it has xfs as a file system and it's installed same thing down here they have another carved out slice called centos swap and that's mounted as swap space on the system so if we look inside dev mapper we're going to see we have sent to us root and sent to us swap now i want to show you really quickly a handful of commands so you can see what's going on if we look at pv display this is physical volume display it's going to show us what's going on behind the scenes right like what makes up the bucket of data that we're going to use we only have one physical volume i know it seems weird right why am i making a bucket out of one device well you can expand it if you want so they're giving you the room to expand later if you want to do it after it's already installed so we have one physical volume and it's a partition it's dev sda2 all right it's in a volume group named centos and that gives us it looks like about 19 gigabytes of storage okay so this physical volume is inside a volume group called centos so our our bucket with all the data is called sent to us and it looks like we carved out root and swap from that bucket so let's look at that really quick we're just going to look at lv display logical volume display and we should see two and sure enough we have two logical volumes i'm going to scroll up a little bit here our first one is named swap it's in the centos volume group because that's the only volume group we have it happens to be two gigabytes in size and it lives in dev centos swap this is an interesting thing you can you can use dev mapper and the name of it or you could use dev the name of the volume group the name of the logical volume as like in folders here so that's another way that you can reference it and find the actual logical volumes and then the same thing down here dev sent to us root the logical volume name is root it's carved out of the centos volume group and this one looks like it's about 17 gigabytes so if we look inside dev sent to us we're going to see sure enough there we have root and swap as our two different logical volumes that we could one is formatted right this is formatted with xfs and this is just swap space i know we didn't go through the process of actually creating all of the different parts but hopefully you understand exactly what lvm is doing taking physical volumes whether partitions or hard drives or raid devices combining them into a volume group and then carving out logical volumes that you can use as regular devices on your system and it just allows for flexibility kind of like a software-based san in your own computer building an lvm is actually one of the most straightforward things that you can do when it comes to block storage devices on linux it's surprisingly consistent all the way through the process for you know the names of the tools and it really is kind of fun and once it's built you can expand it by you know adding more drives to the system or you know stretching your existing volumes but let's go ahead and actually create from start to finish an lvm system on our linux device now here i am on ubuntu and if we do lsblk we're gonna see that well we have a bunch of stuff but down here this is the drive that our system is installed on so we're not gonna touch this one we're going to use these four devices so sdbcd and e which are 10 gigabyte devices notice i don't have partitions created on these now you can create partitions some people prefer to use partitions for their physical volumes in an lvm some people prefer to use the raw devices either one works fine they work the same the advantage of setting up a partition is that if somebody else comes to your system they're gonna see that there's partitions on the system and they're gonna know that something is already done there whereas if you leave them raw devices they might think oh look empty drives now it's kind of far-fetched and you're not just going to like start formatting drives in somebody's system but that's the reason some people like to use partitions i'm just going to use the raw devices and to turn these raw devices into physical volumes we use pv create and we just make a list of the devices we want to use so dev sdb devs dc dev sdd and dev sde so it created them now we can do pv display if we want and it's going to show all the devices that we have now each of them is 10 gigabytes and the name of it is just the device itself dev sdc so the next step is to create a volume group now to do that we just do vg create which is very very nice the first flag or the first command argument here is the name of the volume group so i'm going to call this bucket because it's just a big bucket of our hard drives right of our data that's available so i'm going to call the volume group bucket and then we just make a list of the physical volumes that we want to add to it so those same volumes we just did before and then press enter our volume group bucket was created we can look at that by doing vg display there's four metadata areas meaning we have four devices our current active pv are four we have 40 gigabytes of storage just about all together because each one was 10 and now we have this thing called bucket that we can carve a slice out of if we want and use that slice and put a file system on it so to do that we're going to use i'm sure you guessed it lv create i love the consistency of these tools it's really nice here we do dash capital l and how big of a slice we want in this case let's do something that's going to span all four disks so i'm going to say 32 capital g for 32 gigabytes and then dash n the name that i want to call it i'm going to call this big underscore slice and then we have to tell it where we want to get the data from well in our case it's in bucket the name of our volume group right there is bucket so what we're doing is creating a 32 gigabyte slice called big slice out of the volume group bucket press enter logical volume big slice is created so we can do lv display and sure enough lv size is 32 gigabytes so we know if that worked the name is big slice it's in bucket the lv path is dev bucket big slice which is exactly what we would expect because we've created this volume group called bucket and now this logical volume lives inside there so the last step is that we would actually use this as an actual block device or as a hard drive in and of itself and then we could just do something like mkfs ext4 dev bucket big slice and there we go so now we've done that we could mount it somewhere and every time the system starts it's going to be available in dev bucket big slice and then if we put it in fstab it's going to be mounted on boot and then there are other tools that we can go and like lv extend if we wanted to make it bigger so we could do something like lv extend dash l i'm going to say plus 5 gigs dev bucket big slice and now it says the size has gone from 32 gigabytes to 37 gigabytes it was resized and we did that without adding anything to the system we just used the tools to change the size of our logical volume the thing to remember again about lvm is that it provides flexibility in your system it doesn't provide any redundancy so if you have one physical volume fail it's going to crash the entire volume group and logical volumes are going to get messed up so you want those pvs to be like a raid device if you're worried about something going wrong underneath and losing data but setting up lvm as you can tell is very simple and honestly even kind of fun raid is a redundant array of independent disks or drives and basically what it means is that it you take a bunch of drives and put them together and you end up with a larger pool of storage now the cool thing about raid is that it doesn't just pull things together like lvm and give you a bunch of data it allows you to do some pretty neat things with performance or redundancy so i want to talk about the different raid levels that we can offer using raid specifically linux raid linux has a software version of raid which is very powerful very robust and surprisingly efficient so basically i want to make sure we cover all the different raid types and to do that i want to show you this big scary dragon now here's the idea i'm going to i'm going to talk about raid but instead of a redundant array of independent drives we're going to say they're a redundant array of independent knights of the round table okay and in true 80s fashion our knights are going to be just squares like from atari's adventure anyway here's the deal with raid 0 our drives are set up in a stripe which means that they work together reads and writes happen across two drives very very quickly and so with raid 0 or a striped array it's very fast right you can get dual writes and dual reads at the same time the problem is let's say that the knights are attacking the dragon and the dragon is attacking the knights if we lose one of the knights well then the dragon can get right through to the king because if you lose a single drive in a striped array or in a raid 0 all of a sudden your data is gone because half of your data is written on that other drive so while it's very fast they better kill the dragon quick because if one of them fails all of their date is gone and the king gets destroyed now raid 1 is kind of the opposite it still has multiple drives but they're set up in a mirror which means every time you write something to the first drive you write it to the second drive so you have a complete copy of both now because they're in a line like this they can only attack the dragon one at a time right they can't both attack the dragon like over here in the striped setup but the advantage here is if one of them dies there's still another full night there protecting the king same thing with your data right if one of these dies you still have a full set of your data because you've been writing it to two drives the entire time so a mirror doesn't give you any speed increase because you're not spreading the writes across here you're actually writing all of your data two times once to each drive so you don't get any advantage over a single drive when it comes to speed but you do get that advantage of either drive can die and you still have full protection for your data now raid 5 this is a little bit different raid 5 uses a parity disk and how that actually works is like some digital magic that is beyond the scope of this nugget and to be quite honest it still stumps me a little bit but conceptually how it works is you have multiple drives three minimum and any of these drives can die any one of them it doesn't matter which one and all of your data is still in place so you get the advantage of being able to lose any drive in your array the disadvantage is you lose one drive's worth of storage now what do i mean by that let's say these are all five gigabyte drives okay if these are all five gigabyte drives together that would be 15 gigabytes but you lose one drive's worth of storage for that parody magic and so if you have three five gigabyte drives you're only going to have 10 gigabytes of usable space but the advantage is it's writing to multiple drives as it's going along and if one of the drives dies you still have all of your data represented in the remaining drives now if you lose two of course then your date is corrupt and and you're done but you can lose any drive and it's not just like this is the drive you can lose no you can lose any one of these and all of your data is still there it really is magical and so that means that you can lose a knight and the dragon is still going to be stopped by whatever two of the knights are still protecting the king hopefully that makes sense with the different raid levels and just briefly i want to talk about there are some hybrid levels as well so raid 0 could be raid 0 1 where you have like four drives and what happens here is you have a stripe of mirrors your drives are mirrored and then striped across the mirror or raid 1-0 which is a mirror of stripes or vice versa but basically four drives and you're mirroring two two of them and then striping those two mirrors and vice versa you're gonna stripe them and then mirror those two stripes with raid five there's actually a raid six which is cool but it requires another drive and then you can lose up to two drives and still have your data there's two of those parity drives in place now the downside is you lose two drives worth of storage on your full array but it's awesome because you can lose more than one drive now even if you don't follow along with my awesome dragons and knights kind of scenario hopefully the raid levels make sense now my trick my little mental trick to remember what it is i look at the number after raid and i say how many disks can i lose because for years i would confuse raid 0 and raid 1. but here's the deal with raid 0 you can lose zero drives right because if you lose one you lose your data with raid 1 you can lose one drive and you still have another drive raid five i guess you could lose one of five i don't know it kind of falls apart but rate zero and raid one are the ones that i would always struggle with so i think of that number as how many drives i could lose and still have my data you can go to the store and buy a raid card like a hardware raid device and then you can use that on your system and you'll be able to have hardware raid but linux has a really awesome and powerful software raid program that will use kernel level tools to allow you to create your own raid devices without needing any specialty hardware at all now there's a couple things we need to discuss like partitions versus using raw devices i want to make sure we cover all the configuration stuff but conceptually it's really easy rather than having a hardware based card we just use our regular sata controllers and then our hard drives can all work together in a raid array that we choose now when i talk about partitions versus raw devices let's say we have 200 gigabyte hard drives but they're from different manufacturers now they both say that they're 100 gigabytes however if you look close they might have slightly different number of sectors and slightly different size so this one says it's 100 gigabytes but it might be actually 1028 megabytes and this one says 100 gigabytes but it might actually be 1022 megabytes now they round for marketing purposes and that's perfectly fine and usually on a system it doesn't matter but if you have like this drive fail in a raid array and you need to replace it with another drive and you try to use this drive and all of a sudden oh you created a raw disk device raid array unit and it has 1028 megabytes of space and you try to replace it with another hundred gigabit drive but this one only has a thousand and twenty two megabytes you're not going to be able to work it because this isn't big enough so generally what people do is you take and make a partition inside of your drive that is slightly smaller than the hard drive itself so it might be 99.9 gigabytes and then the same thing when you have a new drive you're going to have enough room to create a 99.9 gigabyte partition and so even though this drive itself is slightly different the underlying partition is going to be the exact same size so you're going to be able to use it to replace a failed drive in an array so that's why we generally use partitions even though using a raw device would work until you need to replace it with a smaller drive that is your replacement now here on our ubuntu system if we do lsb lk we're going to see we have four 10 gigabyte drives that are installed in here we're going to make a right array with those this sda1 this is our root partition this is where our system is installed but these drives down here are the ones that we're going to use to create our raid array now i've already partitioned the first three you'll notice it's a 10 gigabyte drive and the partitions are 9.9 gigabytes awesome we're going to do the sde the last one together and so fdisk dev sd e and the first thing well we can press m to see all of our different options but i'm going to go kind of quickly i'm going to say o to create a new partition type and it's going to be a dos one it doesn't have to be dos but i just decided dos and then i'm going to say n for a new partition and i want the partition to be primary so default number one default the start point on the drive default and here's where rather if i choose default it's going to be 10 gigabytes in size and that would be fine as long as our replacement was the exact same kind but i want to do it slightly smaller than the drive itself so here i'm going to say plus 9.9 g and press enter and now we have it 9.9 gigabytes in size so any 10 gigabyte drive we'll be able to replace it with because we'll just create a 9.9 gigabyte partition inside now the one last thing if we do t for type and press enter it's going to say okay what partition type do you want here it says it created a new partition with type of linux but if we type capital l we're going to see all of the available codes here now this is not a format this is just like a flag to give the kernel a hint as to what sort of partition this is supposed to be so the one that we want to put on here is actually f d for linux auto raid so i'm going to type f d and now it says change type of partition to linux auto raid or linux right auto detect i'm going to press w to write this change to the disk and now if we do lsblk we're going to see we have all of them here now that partition type is just to give the kernel a hint if you put these drives in a new system it's going to say oh look at those partitions those are part of a linux array or of a raid array so we're going to treat it as such so it's just a hint but it works even if you don't change that partition type all right so now it's pretty easy to create the actual raid device we're going to create a raid 5 device with four 10 gigabyte devices well about 10 gigabytes and so we should end up with about a 30 gigabyte usable space with our raid 5 array now the tool we use is mdadm and we're going to say dash dash create because we want to create a brand new one i'm going to say verbose just so we can see it do things as we type it in and now what device do i want to create well the devices are dev md and then the number of the raid device you're creating so we're going to start with md0 because that's our first device and we don't have any raid devices on here yet i'm going to do dash dash level equals 5 i want it to be a raid 5 device and then dash dash raid devices equals 4 because we have 4 devices and now we need to list those devices out and we're going to list the actual partition so dev sdb1 dev sdc1 dev sdd1 and dev sd e1 we'll press enter boom it created it that quickly now we can see the details of it if we were to look at md stat this is the virtual file system proc and this is going to show us the md stat which is the current rate arrays in our system so here we have it's currently a raid 5 array it shows our devices here lots of information it says recovery that's because it's building the array but we can use it while it's currently using the array which is really awesome okay so we look into dev grip for md we're going to see there we have md0 so we have a device all created and we can now use this as a hard drive in our system before we do that though i want to save this configuration of this raid5 array into our system so that on boot it knows exactly what sort of array to build to do that we just do md adm detail scan and if we do that it's going to show us the configuration so that's the configuration for our current array what we want to do is save that so i'm going to redirect it into etc mdadm md mdadm.conf so we're going to save that result into this file and now every time we boot the system md0 is going to be created and then we just treat it like any other hard drive on our system so nkfs dev md0 and boom it created it and now it's part of our system and it's going to be about 30 gigs in size let's see lsblk and we look down here md0 shows up as 29.6 gigabytes about 30 gigabytes of raid 5 storage on our system if you're thinking that was a little bit too easy well you're right linux raid is awesome it's super simple to set up it allows you to use the regular drives in your system and set them up as a raid device so as long as you can save that file into mdadm.com that detail scan that we did and you can check for the progress or the status of your current rate array in mdstat you are really set that's all it takes to use raid on a system using nothing but software provided with the linux kernel installing tarballs sounds like some sort of prank you might pull on somebody in high school but really this is the way that we would install software on linux for years before package management systems came out now you can still download tarball files of source code for programs and install them although it's not terribly common anymore now we used to refer to this as the three step we would extract compile and install and this is the process to convert source code into an executable program now i'll show you how to do that really quick i've downloaded already a very simple program as a tarball file it's called sun weight it's just a simple program that waits until the sun goes down and then executes so if you have a script that you want to run at sunset that's kind of a cool tool to use to do it but the first step is to extract it so we're going to say tar minus the xvf sun wait we'll go into the folder that's created if we type ls we'll see there's a couple files in here this is the source code and also a make file now sometimes there's more complicated things like dependencies sometimes there's going to be a config file so we'll run config and go through that process and it'll tell you if you need other dependencies since we don't have a config file i'm just going to type make because make will compile it into an executable program now you'll notice we got some warnings but we didn't get any errors a lot of times if it's a big program it'll say oh i need this dependency or oh you forgot this library and you'll have to download and install those dependencies before you can compile it but this one is very very simple if we type ls again we're going to see now we have a result here we have the sun weight program now we can execute it right here by saying dot forward slash sun weight and we'll see sure enough there it runs and we could use this to wait until sunset to execute a program if we want to install it though we have to either copy it to our user local bin folder or sometimes they'll include in the make file an installer so we could say something like make install this one doesn't actually have that ability to install it it's a very simple program so if we wanted to install it if you get an error like this you just simply say okay i'm gonna move sun weight to user local bin and now if we type sunway it's going to execute because it's in our path now i told you that three-step process is really really simple and it is you just basically type make and it compiles and then you have a binary that you can install either using a script or just putting it in your user local bin file there is a big disadvantage though if you compile things from source and that is there are no update mechanisms for getting a newer version if you use a package manager it'll update old software but if you just compile it yourself and install it manually there's no way to update it and that can be a real problem especially when security concerns come up so while it's important to understand how to use tarballs to compile and install programs it's not really the best way to go about it if you have any other options deb files are the way that programs are packaged up in the debian and ubuntu world now there's a couple different ways that we can manage the subsystem and there's a little bit of confusion as far as what tool to use now behind the scenes they all use d package which is like the the lowest level of interaction with dev files and i'll show you why this is not what you use on a regular basis in order to actually install packages on debian or ubuntu there are three different options though for installing packages using the proper system apt apt-get and aptitude now i want to talk about them because they do the same thing but it's just a matter of being replaced by something better so apt just apt all on its own is the newest program to interact with the app system it's new it's simple this is the one to use so just use apps i'm going to try to use apt if i can aptitude you may find online instructions people telling you how to install packages with aptitude this is older but it still works i don't recommend you use it though it's a little bit strange when it comes to dependencies but it's been outmoded and now apt is the way to go now here's the other thing apt get has been around for a very long time it's the oldest of the three it still works but i don't recommend you use it because again apt is by far the best way that you can go about installing packages here's the real problem though i've been installing packages on debian and ubuntu for so long that sometimes if you're watching me in a nugget i might use apt-get by mistake just because it's a habit it still works there's nothing wrong with doing it but the proper way to go about installing packages is to use apt that's what i'm going to try to do and that's what i'm going to show you now now on our ubuntu system if we type ls we're going to see i have a deb file so it's kate kate is a text editor that works in the kde environment and this is the installer the dev file now remember i said that d package is the program that is used behind the scenes that's how you interact with dev files here's the problem so i'm going to say sudo d package minus i for install kate.deb it's going to try to do it but here's the problem it doesn't resolve any dependencies so if i want to use d package to install it i'm going to have to find every one of these deb files on the internet download them install them one by one find out if they have dependencies and it can be a real mess so thankfully the apt system takes care of all the dependencies for us so i have to erase this so i'm going to say pseudo d package minus r kate and it's going to undo the mess that i made and now we're back to square one rather than downloading the deb file we can use the app package management system and just say sudo apt install kate and it's going to look for the latest version and get all of the dependencies and you can see there are a ton it would have taken me a week to come up with all these dependencies so if i say yes it's going to install them all and it's finally finished that took like over three minutes i sped it up so i didn't have to sit here with me the whole time but now all we have to do is run kate because it's installed with all of its dependencies on the system and here it is our little text editor kate now another nice thing about using a package management system is we can keep things updated so we could say sudo apt update which is going to download the latest repository information to find out all the updates that are out there for us to install and then once we have the freshest versions of what's out there we can use that same program apt and say sudo apt upgrade and looks like we have a couple things we could upgrade hit enter and it's going to keep our system up to date that easy not worrying about dependencies it does all that on its own so while all three of these will technically work for installing and updating packages you really want to use just the simplest one which is apt it's the newest and it's the easiest to remember rpm is the red hat package manager and it does just that it manages packages on a red hat based meaning red hat or sent to us or anything else that uses the rpm system that's how it manages their dependencies and their installation and their programs and updates etc now there's a few tools we need to know how to use in order to really utilize rpm yum is kind of the de facto standard but there's a new kit on the block called dnf i want to talk about that and then of course rpm itself is not only a package management system but it's also the tool the low-level tool that we use to actually handle packages now the cool thing about rpm is there's no two steps required when it comes to installing now what do i mean by the two-step process well here we have our multiple programs i just mentioned now yum is yellow dog updater modified which may seem silly but just for a brief second yellowdog was a linux distribution that ran on powerpc or old apple hardware and its claim to fame was actually that it started this yum program for managing packages now the operating system itself didn't do well after the power pc platform kind of faded out but yum is still around today and in fact that's the program that we use on almost every rpm distribution it handles dependencies um it is it updates the repo information as it's installing and upgrading so unlike app you don't have to say like yum update and then yum upgrade when you upgrade yum updates before it does anything else now dnf which and i'm not kidding here it stands for dandified yum but dnf is the new program it's in fedora right now it's not in centos but it's going to replace yum it's just a rewrite it has some features that are that it worked better but similarly structured and how it works and then of course rpm is what happens behind the scene there's a program called rpm and this is the low level tool but it doesn't handle dependencies so we generally don't use the rpm tool on its own let me show you what i mean i'm on centos here and we're going to use yum to install packages but first i want to show you rpm now if we look i have downloaded this program called kate this is just a text editor that works in the kde environment but if we were to install it with rpm we would have a problem if we say rpm dash i for install the name of the package it's going to say i can't do that because you have 110 billion different dependencies that aren't installed so what you'd have to do is find every dependency every rpm install them one by one these probably have dependencies of their own that we'd have to track down but thankfully that's where the package management system yum comes into play so we could simply just say yum install kate and it's going to go to our repositories it's going to update the cached information like show us the latest information from those repositories and then it's going to search for kate it's going to find all of the dependencies right here say it's looking for all the dependencies and the dependencies of the dependencies and now it says would you like to install it along with the 77 dependent packages okay i'm going to say yes actually i'm going to say no if we install it then it would just download install all of those packages and life would be good and we would have kate installed but what i'm going to do is now show you that's how we would install a package but if we want to update the system all we need to do is type yum upgrade and it's going to query all of our repositories it's going to download all the package information dependency information that we need and now it's just telling us if we want to get our system updated we're going to have to download 116 packages install one new package and it's going to take 308 megabytes of space do we want to do that i'm just going to say yes and it's going to go through the whole process of downloading all those rpms it's going to use the rpm tool in the background install each one of them one by one until they're all updated and all installed to the latest version so yum is very simple to use and knowing that it comes from you know a distribution that's no longer even a valid distribution i don't know that's just cool for me that you know yum is still around even though yellow dog hasn't been a distro for many years now i don't have a fedora system to show you dnf first hand but it's a very similar program to yum it's going to work very very similarly so if you find yourself on fedora just use it very similarly to how you'd use yum and you'd be fine one of the big things i want to point out though is that there's no two-step process like there is with the apt environment meaning you don't have to update your repositories before you install it you just run yum and it's going to update and upgrade all the information before it does any installing or upgrading one of the nicest things about the app package management system is that you can add repositories which are just different groups of software packages so if something's not in the standard ubuntu or debian system and somebody else has written something you can add a whole repository of new software and then the app system can use it just like any other package on your system it's pretty cool now we're going to look how to add something and there are a few gotchas that we need to look at but thankfully it's a fairly simple system when it comes to configuring new repositories so i'm here in the etc apt folder on an ubuntu system if we do an ls we're going to see there are two things i want to show you there's sources.list this is a folder and there's actually nothing inside there right now but we could create a new file.list in here and it would be read just like this systemdefaultsources.list so it doesn't matter where things go i'll just add something to the existing sources.list so we're going to look and you'll see there's already a bunch of repositories added each one of these lines is a repository containing software that the apt system can install now some of them are commented so we would just uncomment the existing ones like here this is a partner archive we could uncomment this and then this would be an active repository but if we want to add a third party repository we can like i said either add it at the end of this file or create a new file in that sources.list.d folder and call it like new program dot list but i'm just gonna add it in here and i'm going to add the opera browser repository in here let's say we wanted to add the opera browser it looks like this it's okay if every field here doesn't make perfect sense to you this is just the format it says what kind of a package it is where it's stored what folder in there and then what version of the software we want to actually add the stable and then it's non-free as far as like what what type of repository it is so this is the line right this is where the opera browser is stored on the opera website so if we save this we would be able to do apt update no dash apt space update and if you look it is updating them all but i do want to show you back at the top here it gave us a little warning slash error oh and it repeated the error down here so i'm going to show you here it says this repository is not signed so what that means is we don't have their key because we don't want there to be a man man-in-the-middle attack where somebody you know takes over opera.com and then starts sending us bogus packages so they do signing key signing in order to make sure that we get the right software so if we want to use the repository we have to add their gpg key to our system now it's not difficult to do we just do kind of a two step here we have to download the key itself so wget so we're going to download the file and then pipe that into apt key add all right so we'll do that it's going to download the key and it's going to add it to our system and it just says ok we can see if it's there by typing apt key list and it's going to list all of the keys in our system and if we look we should be able to find sure enough here's the opera one so there's the upper key that we installed and now if we do apt update notice there's no errors at all it updated and now we could just install it using our app system app install opera stable and it would install our needed packages you know it would resolve dependencies even if they're in other distributions and it would install it for us just fine i'm going to say no because i want to show you another way you can add a repository and this is kind of a cool thing that ubuntu added it's called a ppa or a personal package archive there's a particular text editor that i really like it's called adam atom and they have a ppa which is a repository and to add it you just say add apt repository ppa colon and then where it lives or what user it is web team and the name of the repository is atom press enter it's going to do a couple things it's going to add it to our sources.list but it's also going to download that key for us so all we have to do is say press enter and it's going to install the key it even does an apt update for us and then we would just do apt install adam and sure enough it would be able to do it from that package or from that repository that we just added it can be a bit overwhelming once you have to start adding gpg keys for the repositories that you put into your sources.list but ppas make it really really really simple they do everything for you i really like ppas i think it's been a wonderful addition to the way we handle apt packages repositories in an rpm system using yum are very similar to that of an app system adding them is maybe even easier and editing the config is about the same just with different files let me show you what i mean now the main configuration file in centos is going to be the yum.com file so let's look at that it's in etc and it's yum.com here's where we can do a couple things one we set the main configuration things like where the cache is stored and things like this one thing i want to point out in here is this gpg check now you can actually turn off yum's ability to verify using signed key pairs and this will just allow it to install any repo or install from any repo that you install in your system it's not a good idea to turn this off though because again this is a safety check to make sure you're not getting a man in the middle attack but the main section is at top you can put new repos in the bottom or like it says right here you can put them in etc yum.repose.d as individual files with the format file dot repo that's usually what's done but we could add them here let's look inside the yum.repos.d folder so let's get out of here go into etc yum.repost.d and here's where all of the repos that are currently installed on our system live so let's look at one really quick look at the base file here and we're gonna see how it is set up so there's the base configuration up here the name of it the release where the actual files are stored gpg check you can turn this on and off for individual repos as well so if you set up a repo and you don't have signed key pair you could turn it off for just one repo and then this tells us where the actual key file if it's turned on lives so we can put our put it in there manually so that yum knows where to look to find that key now normally these aren't added manually but you could type all of these in these are all individual repos that are defined in here in this bracketed section usually what you do though it's really elegant is let's get out of here you just install the package so one of the really popular packages is called apple which stands for extra packages for enterprise linux and to do that we just do yum install apple release i'm gonna say yes to install it and now if we look in here we're gonna see here we go the apple.repo has been added so now if we were to install packages apple would also be one of the places that we could pull packages from we can just cap this to look at it and we can see it's enabled the gpg check is on it tells us to use this key file for checking the signatures and that's all there is to installing a repository in yum and since yum automatically updates we don't have to like update the cache when we install a package it's going to automatically fill in the blanks for us yum is an incredibly awesome and powerful package manager and we can tell just by how easy it is to add a repo and edit that config either using the yum.com file or the individual files inside etc yum.repo.d apt and yum are certainly the most common package managers out there but there are a few less common package managers that you should still be aware of now arch linux uses a program called pacman for managing their packages and open souza uses zipper now i'm not saying that the mascot for zipper is a purple horse but i'm just saying maybe it should be nevertheless it's not too difficult to use them even if you're not familiar here i have two terminal windows to two different linux distributions the first one is arch linux which uses the pacman package manager so if we just do pacman minus h we're gonna see here are a list of the commands now it's not immediately clear how you go about installing a package unfortunately it's not just install it's actually capital s for sync we kind of want to sync the system into a state that we wanted so to install a package it isn't difficult once you know what to do just pac-man minus capital s and let's install vim my favorite text editor you have to actually spell pacman correctly pac-man minus capital s vim it'll say do you want to install it and say yes it's installed it and now if we type vim you can see sure enough there's vim my favorite text editor now over here in open souza it's a little bit different here we use a program called zipper zypper we're gonna do zipper minus h and it'll show us all of the help commands that are available this one it is pretty easy you just do install or you can shorten it to just in so we could say zipper in vim we'll install vim again press enter it's going to retrieve the repositories online just like yum or apt would do and then it's going to install the vim package for us and we'll be able to use it on open souza it'll ask us so we want to continue i'm going to say yes and it installs all of the packages and the dependencies so now same thing here we get startup vim and sure enough there's vim this time on open souza pac-man and zipper are really two of the more popular alternative package managers but there are some others if you're on slack where you're gonna have to install things by hand using tar.gz files but these two along with apt and yum will get you through most systems managing local users on your server is really easy and there's a bunch of tools that make it even easier now there's the standard command line tools that allow you to add modify and delete accounts and there's this super cool script that i really like which makes adding a user very very easy there are a whole bunch of different facts about a user that is stored in the system though full name username password all of these things plus some others that aren't even listed are on there and it's important to know that the tools will manipulate all of these but you don't have to specify every single thing every time especially things like office number those aren't even really used anymore but there are possibilities that you can add specific information in the local group now i just want to go right to the command line so we can actually start adding users and at first we're going to use the tools that you know kind of come with it like the low level tools user add user dell and user mod so first of all i want to say user add because we don't have any extra users on our thing i'm going to do dash h and that'll give us the help screen now you can see there are tons of options for adding users but the format is pretty much the same user add whatever options you want to add and then the login or the username for the new user now we're going to use just a couple and i want to show you the problem with using user ad as opposed to that fancy script add user we have to determine like a home directory and a user shell and all sorts of things like that so let's say we wanted to make a user we're just going to say now notice i'm root we have to be root to make a system user but we would say user add and i want to do minus d home susie minus s for shell i'm going to use the bin bash shell and her username is going to be susie press enter it's all done now there is no password for susie we'd have to actually do that with the password command uh we'll actually say p-a-s-s-w-d suzy all right so now she has a new password and we can here let's start a new window ssh suzy at localhost susie and we're logged in as susie oh but see it says unable to change to directory home susie there's no such file or directory well dog on it we said that that's our home directory but by default it doesn't make that directory we actually have to make that directory or there's another command dash m which will create it as we're adding the user so it's possible again to use this tool and set all of those different flags up as we create the user if we go through all of these things but it's much easier to use the add user script and i'll show you what i mean add user frank and now it says okay i'm creating the frank user added a new group for frank created frank's home directory copying all the files from etsy scale and now it's asking me put to give frank a password so i'm going to do that now the full name of frank his room number which again we don't use room number work phone all those things anymore really but full name is nice to have in there and is this information correct i'm going to say yes and so now we have a really nicely set up user so if we were to open a new tab again and do ssh frank at localhost log in with frank i have to type frank's password correctly and now you'll see sure enough if we do pwd we're gonna see we're in frank's home directory he has a perfectly usable account because that add user script goes through and remembers all the various things that we need to do so let's get out of here now there is another tool we can use user dell so we can say user dell let's do minus h so we can see the options there's really only one option that's ever really important that is minus r which removes the user's files so we could say user dell [Music] r for remove frank and it removed all those files that said there wasn't any mail form so i couldn't remove that but now frank's home directory is gone now we could modify suzy's account if we use user mod so let's see user mod minus h and we're gonna see same thing we have a whole bunch of tools and this is just to modify an existing user so we could do something like change her shell like right now it's been bashed but we could change her shell so we could say user mod minus s let's say bin false susie what this is going to do it's going to change her shell so if she tries to log in it's going to fail so let's open up a new tab and try to log in as her ssh susie at localhost it logged in and then immediately logged out see it says it still doesn't have a user account we are a user directory we didn't make her a home directory but then it says connection to localhost closed that's because as soon as it logged in her shell is been false which immediately exits and then we're back to being logged out we're logged in as bob here so the ability to modify a user account is really important but if you're going to add a user i highly recommend you use that script you can certainly use the manual tools with all the flags to add that account but add user just makes it so easy by remembering all the steps and i'll be honest i often have a difficult time remembering is it add user or user ad add user user ad for my own sake i like to think okay alphabetically add user is first and that's the first tool i want to use for adding a user so that's just the trick i use regardless of what tools you use to add users it's important to understand that modifying them on the command line is fairly simple and not that difficult to learn local groups on a linux system are fairly straightforward to handle but it's important to understand the difference between primary groups and secondary or supplementary groups on each individual user now every user on the system is going to have their own personal primary group now usually that's going to be the same as their username so bob is going to have a username of bob and he's also going to have a primary group of bob and that's usually how it goes it could be a it could be a different group that says primary group but almost always that's how the primary group is going to be on a linux system and then there are all of the supplementary groups which happen to be like things that he belongs to like maybe he's in the admin group maybe he's in the sales group marketing third floor public all of these groups are in addition to his primary group and it will give him access to certain folders on the system that he might not have access to if he didn't have these different group memberships now in order to actually create groups the tools are very straightforward just like adding users with much fewer options so let's actually look at that and i want to show you how to manipulate users in groups on a system now here we are on an ubuntu system and we can simply say group add public it's going to add the group public to our system we could say group ad sales and it's going to do that now we could look at the different options group add minus h you see there aren't too many options we could specify a group id number if we wanted and we could do some things with group mod to change some of those features like the group id and then of course we could delete them with group dell and the same sort of thing you know we just delete them as we would use user dell on a system to get rid of a user so this is pretty straightforward but the part about primary and secondary can be confusing so right now if we type groups bob we're going to see that bob is in all of these groups the first one listed is his primary group so bob's primary group is bob and then these are all of his supplementary or secondary groups now i want to show you user mod minus h because it shows us how we can manipulate groups right here so lowercase g is how we force a user to a new primary group so we could change bob's primary group using the lowercase g flag on user mod in order to change his secondary or supplementary groups we use a capital g but here's where the gotcha comes into play if we do like minus capital g public it's going to delete all of his other supplementary groups and only make him part of the public group and that's not what you almost ever want so there's this nice dash a which means append the user to another supplementary group without getting rid of the additional groups that he already belongs to so let's actually do that in practice so we're going to say user mod minus a for append minus capital g public bob okay and now if we do groups bob we're going to say that bob still belongs to all those secondary groups and also to the public group which is exactly what we wanted to have happen because if we wouldn't have used that a this is what would have happened if we would have done user mod minus g let's do sales for bob this seems like it's going to do the right thing no errors but then if we do groups bob we're going to see uh he still has his primary group of bob and then just sales and so we'd have to manually go back through and add him to each individual group which is really a pain in the butt so you don't want to do that you want to always remember to use the dash a now in function the difference between a primary group and a secondary group the primary group is what is used if bob were to create a file so let's open up a new tab here we are bob say this is in our home folder if bob were to touch a file and we do ls minus l we're going to see the file that we just created is owned by bob and the group membership is bob's primary group of bob so that's really the difference between primary and secondary is when you have a primary group that's what the group membership of a new file you create is going to belong to now creating groups is really easy with the group ad group mod group dell tools so i didn't even go into that very much in depth the real important takeaway is the idea of primary versus secondary or supplementary and how to take individual users and put them into groups without deleting all of their other supplementary groups that they already belong to i hope this has been informative for you and i'd like to thank you for viewing figuring out what users are on your system and what accounts they're using and what they're doing is really an important part of forensics but also it's an important part of a sanity check like why is my computer running slow or what user am i logged in as and there is a handful of simple command line tools that we're going to go over that will just help you figure out the users on your system now the first scenario this happens a lot if you log into embedded systems like routers and stuff where you don't get a prompt that tells you who you are you just get like a hashtag here well what you can do is say who am i which seems a little bit silly but it will just give you the user that you're currently logged in as if you're starting to use sudo and su and you're sshing from one computer to another sometimes just figuring out what user you're logged in as is really important so that's it seems silly but it's something that i use in nuggets even you'll see me use that quite a bit another one if we just do who it'll show us who is logged into our current system it actually gives us quite a bit of information we have bob frank it looks like susie is logged in twice and it tells us when that person logged in and where they're logged in from so bob is our current user and i'm logged in on the display zero meaning i'm using an x windows session here and then frank and susie are both logged into localhost probably ssh and then suzy is logged in from a remote computer probably over ssh as well so who tells you that w which is like what although you don't have to type the rest of it it's just a w gives you the same information with a little bit more so these are the users where they're coming from when they logged in how long they've been idle and then what again that's where that what came from what they're actually doing so here i'm using a gdmx session uh looks like frank is sleeping on the job frank judging you buddy and then susie's just logged into a bash terminal so these are tools that give you information another one is pinky which seems like a silly tool but it replaces the older tool finger that has kind of been abandoned about 10 years ago but this gives you even more information but not the what's going on this gives you the login name full name when they logged in how long they've been idle where they're coming from and a combination of these really usually any one of them will give you the information that you want but these are all here to give you information on who's logged into the system now if you want further information about it you could also do id susie and it will give you information about susie's id including her user id her primary group id what groups she belongs to looks like she only belongs to the susie group so let's do an id on bob and yeah bob belongs to a whole bunch of other groups so it'll give you all of that and then lastly lastly there's the command last hahaha if you type last it gives you a history of the people that have been logged into the system recently so it looks like we go back all the way to march 7th here today's march 13th and i logged in a bunch of times just so we would get some results here and we can see who logged in where they logged in from and when they logged out or if they're still logged in so you can see i logged in at 1001 logged out at 1001 up here logged in at 1003 still logged in from the local host computer and that is susie so there's a lot of tools that you can use to figure out who's logged into your system how long they've been there where they're coming from what they're doing and these really might seem like throwaway commands like why would i need to figure out who's logged into the system but knowing the simple little tools like who and w and last can really be convenient when you're tracking down what's going on on your server who's logged in who was logged in did somebody log in bob said he logged in did he really log in well check out last and it'll tell you the point is there's a bunch of tools that are available that'll give you information on users on your system whether they're logged in not logged in or have been logged in in the past passwords and group memberships are things in linux that are stored in text files just like everything else but the problem is we don't want people to have access to our passwords and so there's kind of this elegant system of shadow files that has been invented this is fairly recent in the world of technology it used to be everything was stored in a single file called etc password but things have been changed now so that everybody doesn't have access to seeing the encrypted passwords so how it works now is this let's say this is our user bob now bob when he logs into a system has to be able to see what his home directory is so he needs to have access to a lot of information about his account however we don't want bob to have access to everybody's encrypted passwords so that's kept over in another file called a shadow file so what we have in the password file is we have bob's username and then literally just an x that takes the place of where the password used to be stored and if this x is here then the system knows okay i need to go over and look in the shadow file for the actual encrypted password of bob and then it does system authentication with root access as opposed to bob's access which is to read the password file but he doesn't have access to everybody's encrypted password the reason we don't want that is if he were to do like a brute force attack if he had access to everybody's encrypted password he could just keep hammering away at it until he finally figured out what the password was we don't want that so we don't want every user to have access we just want root or the system to be able to authenticate and check out you know the encrypted password file so anyway there's a neat and elegant system of how this works now notice obviously the etc password file is readable by all the etc shadow file is only accessible by the root user but when it comes to editing those there are special tools to make sure that we do it properly now first let's actually look at the files so if we do an ls minus all of etc pass sswd we're going to see that sure enough it's owned by root and everybody on the system can read it right only root can write to it but everybody can read it but if we look at the etc shadow file we're gonna see that only root can write to it and only people in the shadow group can read to it everybody else in the system can't even read it so our encrypted passwords are protected now there's also if we do ls minus l etc group we'll see the same scenario where this is the group definitions for users on the system and the same settings permission wise as etc password has and if we do ls minus l etc g shadow we're gonna see it's the exact same thing as the shadow file now groups don't normally have passwords associated with them but they can so the shadow system does the same with the group file as it does with the password file now if we wanted to edit one of these files we could do something like sudo vi et cetera pass wd and it's going to let us edit this file using our text editor but this is not the ideal way to go about it because we want to do it the shadowy way right we want to be able to edit it and then be sure that the shadow file matches so let's get out of here let's not make any changes to properly edit these files what you do is say sudo vipw this is part of the shadow package and it's going to ask us what editor we want to use if you want to use nano this is the easiest one like it says right here easiest you can use this i prefer to use vim so i'm just going to choose selection number two but number one is perfectly fine and it opens up and looks exactly the same and we make changes here and we could like go down here you'll notice all the things about bob are listed in this line right here's his username his password is just a placeholder here as an x because the actual encrypted password is in the shadow file but his user id is group id his full name his home directory his shell we could make any changes we wanted here and then we would just go and save the changes and this is what it would tell us it would say okay you've modified the etc password file you may want to modify the shadow file too and to do that you do vipw dash s for shadow so the same thing we would say sudo vi pw dash s and now this is the actual shadow file that we're editing and we can go through and if we wanted to make changes here this is bob's encrypted password now this is obviously not bob's plain text password his plain text password is just the word bob we know that but this is what it's like encrypted but since this is only accessible to root there's nobody who's going to be able to do a brute force attack to try to decrypt this because they don't have access to the passwords encrypted themselves so we'll get out of here and now the exact same thing with the group password or the group file in the group shadow file is done too we can say sudo vigr and this is going to edit the group file and sudo vi gr s and that's going to edit the g shadow file all right so that's the proper way to go about editing those files manually if you man if you edit these files it's going to do the same thing as if we did like user mod and changed somebody's home directory it just does it by editing the underlying configuration files so yes group and password files are still just text files but there is this elegant shadow system that allows us to make sure that the right people have access to the encrypted passwords and not everyone on the system can see everyone else's encrypted passwords quotas are the way that we make sure users or groups don't overuse the hard drive we don't want a particular user a particular group to use up too much of a hard drive and stop other people from saving files now there's soft quota limits and hard quota limits the difference is with a soft limit you're warned every day hey you've gone over your limit hey you've gone over your limit whereas if you reach the hard limit you're no longer able to save files at all now there's a couple things that we have to do to get our system ready for using quotas and keeping track of things but the first thing is we have to make sure that our partition is mounted correctly so on our system here i have a disk mounted or a 10 gigabyte drive mounted on mnt disk okay now if we look into our etc fs tab file i'm going to show you how you mount it so that quotas are enabled so here's our drive it's dev sdb1 and it's mounted on mnt disk it's ext4 i've used default mounting options and then i've added a comma and usr quota now we could also put grp quota we could put another comma and grp quota i'm just going to do user quotas group quotas work the same way so we'll learn one and we'll know how to use both but we have to make sure that it's mounted this way so if you're making this change you want going to want to reboot your system to make sure that it actually takes effect and then if we type mount we're going to see that sure enough mount disk or mnt disk is mounted with quotas enabled specifically user quota management okay so we know that the drive is able to support it but out of the box quotas are not turned on so what we need to do is first of all scan the existing drive for files owned by a particular user so we need to actually do sudo quota check dash a for all partitions that support quotas dash u for user owned files press enter it's going to go through it's going to check our drive and now if we look in mnt disk we're going to see sure enough now there's a quota file that has been created and it shows just you know all the files on the drive who owns them which is none right now but we're going to change that in a minute now the other thing is so that prepares the drive but we actually have to turn quotas on so we're going to say sudo quota on a for all supported partitions so now quota quota ing is actually turned on but if we want to set a particular quota for a user we're going to have to use ed quota so sudo ed quota and i want to do this for the user bob on our system so we're going to do it for bob and then we get this list now it's going to show us all the file systems that support quotas in our case that's just this one dev sdb1 now i have to explain really quickly there's two kinds of quotas we can do we can set up quotas for inodes or we can set up quotas for block usage now an inode means a file so we could say how many files a person can store on a particular partition but this isn't all that useful right i mean what if they have two files but those two files are like 27 gigabytes each so rather than set limits on inodes i tend to like to set them based on blocks now by default these are one kilobyte blocks so this zero means how many are currently in use and there's nothing on the drive owned by bob so that's set to zero right now but i'm going to make some changes here i'm going to say i want the soft limit to be 500 kilobytes and i want the hard limit to be 1000 kilobytes okay or like one megabyte this is not practical number you'd probably do something bigger in real life but we're going to save this say yes and now as bob if we go over to mount disk ls we're going to see that's in there now there's no usage currently but let's say i were to create a file now to do that i'm going to use dd it's okay if you don't use the dd command basically we're going to say an input file of dev zero output file of file one block size equals one kilobyte so that we can know exactly how many kilobytes we're using up and count equals let's say 400 so this should make a 400 kilobyte file if i press enter and do ls minus l we're going to see sure enough we have a 400 kilobyte file on here and this is fine this isn't meeting our quota at all we haven't done anything bad we're not even up to our soft quota but if we do this again i'm just going to push the up arrow and change this to file 2 and press enter okay it's done the same thing we do ls minus l now there's 800 kilobytes stored on this particular disk what this means is it hasn't stopped us from creating it but every day we're going to get an email from the system that says hey you've gone over your soft limit you really need to delete some files and we'll maybe do that maybe we won't do that but here's what happens if we try to create another 400 kilobyte file which will take us over the limit right because 400 plus 400 plus 400 would be 1200 but we only have a thousand kilobyte limit so what's gonna happen if we press enter it says that there's an error writing file three the disk quota is exceeded so let's do ls minus l and see what happened so it looks like it went along and it was creating fine creating fine it got to 200 kilobytes and all of a sudden it couldn't write anymore and that makes sense because 800 or 400 plus 400 is 800 plus 200 it is a thousand so we hit our hard limit and that's exactly how quotas work on the system once quotas are turned on it's really a hands-off kind of thing they take care of themselves the emails go out automatically every day if the user goes over their soft quota and at the hard quota it stops them no matter what so quotas are easy to set up once you remember to use quota on to turn it on make sure that it's mounted with the proper options and run that initial quota check so that it knows what files are on there so it knows when it is or isn't getting close to the actual quota that you set user profiles are where initial settings are set for a particular user like if if they're going to set up like aliases or they need their path variables those are the sort of things that profiles will do and there are system-wide profiles and also individual profiles and it can be a little bit overwhelming because not every linux system is the same now there are some commonalities usually there's an etc environment file and that file will often but not always set up the path variables so that the users who log in get a particular path set up now there's almost always an etc profile in the system-wide file in a profile is something that is run on a login shell so like the very first time you log into a system like if you're logging into a gui that first time you log in you will execute the profile these are settings that only need to be executed one time like it doesn't matter if you're going to open a new terminal you only need to set these settings one time and then there's also going to be one of these not both of these it's usually either one or the other either the etc bash rc or etc bash dot bash rc uh this one is pretty common in ubuntu this one is pretty common in centos but nonetheless these are they serve the same function so you're going to have one or the other and these are profile settings that need to be set every time you open a shell so let's say you're already logged in to x windows like in a gui session you're in there and you click on an icon to open up a new terminal window well that's not considered a login shell this is just considered a sub shell of your main system login so you will not execute a profile you will only execute your bashrc now this is again systemwide so that happens to everybody when you log in you get the systemid profiles applied and then every individual has the possibility to have these individual files in their home folders now they all start with a dot so they're all hidden you have to do ls minus a if you want to see them but these are the same things up here it's just if you have any changes or additions you want to make to the system-wide settings you put them in your own personal folder and you put it in like bash rc and that'll run every time you open a sub shell everything in here will be set up and then the dot profile or bash underscore profile depending on which system you have you're going to have one or the other one of these but this is executed the very first time you log in just like the system-wide profile your personal profile only gets executed that initial time when you log into the system any sub shells will only apply the bashrc files but that's the way it works i'll show you really quick how it's set up on a system but the hierarchy is really the important thing to understand now this is bob's home folder i did an ls minus la so you can see all of the things in here he has a dot bash rc file and he also has a dot profile that means that these are going to be applied after the system-wide settings because the system-wide settings are given to everybody and then any personalizations like if you have an alias that you want to set on your own you would put it in your own personal bash rc file now inside etc there are those common files like ls minus l grep4 profile we're going to see we have etc profile and also etc profile.d this is a folder if we go in there we're going to see there's a bunch of sh scripts all of these are included in the dot profile in fact i'll show you what i mean vi dot dot let's look at that system-wide profile file and if we look all the way down in the bottom here it's going to call in all of those files inside profile.d so when i say that the profile is executed by everybody not only etc profile but also etc profile.d everything in here is going to be executed as well to every user on the system now there's a couple important things to remember don't worry so much about what exactly is the name of the file when you look in the etc folder you're either going to find an etc bash rc file or an etc dot bash rc file don't worry too much about which system has which whichever one is there is the one that you need to use now as far as the hierarchy goes it's important to remember that the system-wide stuff is executed for everybody so everything in etc is executed for all users and then if you have changes or additions you put them in your personal dot profile or dot bash rc user profiles are pretty easy to track down and once you understand how the system-wide and individual settings work it's a snap to figure out which comes first if you're working on the command line a text editor is going to be an invaluable tool because pretty much everything in linux is text based now you should use nano nano is the editor you should use it's a wonderfully simplistic straightforward intuitive text editor that works just fine and then there's vi vi is clunky it's hard to use it doesn't make a whole lot of sense and it's the editor that i use almost exclusively now i know that doesn't make any sense but here's the deal vi has been around for a very very long time like since the beginning so even though it's difficult to use i've managed to learn to use it and it's just what my fingers do with muscle memory so i encourage you use nano unless you've been using vi long enough that it's the only option that seems to make sense now there is one scenario that you may want to learn at least the basics of vi sometimes you're going to come across a system that doesn't have nano installed most systems do but if you end up on a system that only has vi these couple commands are going to kind of save your bacon so here's the deal this is what makes vi so confusing there are two modes there's command mode which is what it starts in an insert mode which is what you use when you actually type text now the way that i can kind of describe this is if you're sitting down and typing you're going to be in insert mode because insert mode is where you insert text and delete text and use your arrow keys to you know go around and change text but then if you need to do some command like save or quit or anything like that you're going to go into command mode and the way i think about it is let's say you are on a standard word processor if you're typing you're in insert mode if all of a sudden you need to reach for the mouse and click on something you're going to be in command mode so while it's not a perfect analogy if you want to do something like save you're going to want to go into command mode so that you can save and quit now since there's no mouse it's all still text things that you're doing but think mentally okay i'm in the mode where i'd be using my mouse to save things instead of just typing out text now to go back and forth that can be confusing too so you start out when you open vi you're in command mode if you want to start typing something you press either i for insert or a for append meaning like do you want to insert right where the cursor is or to the right of where the cursor is but either one is going to work fine so either i or a and then if you want to get back into command mode you press escape so those are your magic keys to go back and forth right i or a i usually use i to go into your typing text mode or insert mode escape to go back so that back and forth that's how it works now the actual commands to save or quit or save and quit are right here and they may not make sense but if you're in command mode you're going to press colon and then type w and then q and press enter that's going to save your document and quit a lot of times people get stuck in vi and have no idea how to get out it can be so frustrating so this will get you out also if you want to quit without saving like you've accidentally made changes and you didn't mean to you press escape again to get into command mode and then you press colon q exclamation point and press enter and that will that will exit without saving and then if you just want to save halfway through a document you can just do colon w enter and it's going to save but you'll stay in then you'll press i to go back into insert mode and continue making edits i'll show you really quickly well i'll show you what nano is and then i'll show you vi just so you can see it in practice like i said almost every distribution is going to have both nano and vi so i'll show you nano first if we do an ls we'll see i have this text file dot so if we just type nanotextfile.txt it's going to open the editor with this and you can just use arrow keys and you can start editing right away this is like you would expect any text editor to work okay so press enter it's going to insert blank lines and then if you want to save you can look right down here we have control x to exit and you can do other things too there's all sorts of commands but i'm going to show you the basics here control x to exit and then it says would you like to save your changes and you can say see the options here y for yes and for no control c for cancel i'm going to say yes and then it says what file name would you like to write well it'll default to the current text file but if you wanted to save it as like copy 2 you could i'm just going to hit enter and boom we're done the text file has been edited it's very simple very easy to use and again i recommend you use it now vi i'm going to look at the same text file with vi so vi text file and here we have we're in what mode are we in we're in command mode now we can still use the arrow keys to get around but we can't edit any text or insert any text if we want to add text we press i and then look down here it says insert so this is a little cheat it tells you that you're in insert mode if we're going to go back into command mode press escape i'm going to go into insert mode press i escape to get out insert mode and once you're in insert mode you can type text and then if you want to save it again you have to press escape and then colon w q enter and then boom we've saved the file we can look and see the text file has been changed all this changes were saved and that's how you use vi it's confusing but that's how it works so again use nano it just makes sense it's easy that's what i recommend you use but if you have to use vi at least now you know the couple shortcuts that are going to get you through so that you can actually use it to edit text now remember i said i use vi all the time and it's true but the funny thing is that it's followed me into things like word processors so sometimes in my microsoft word documents even on the very bottom you'll see colon wq because my head i just automatically do that when i'm done editing text anyway use nano but vi is fun and it's a good skill to have i hope this has been informative for you and i'd like to thank you for viewing viewing text files is an extremely common thing for a system administrator to do on a linux system so we're going to look at a bunch of tools that allow us to examine text files in a way that allows us to view but also to search and i'm just going to go right to the command line so that we can see these things work in real time now i've created in my folder here a file called two cities now this is just the public domain tale of two cities this is the first chapter i'll just type cat so we can look at it two cities see it's just a tale of two cities the first chapter all right so let's clear the screen now the first thing i'm going to show you is the head command and what it does it'll show you the first lines of a text file so the head or the beginning of it so we can just say head two cities and it's going to show us the first 10 lines of the story so the best of times the worst times that's part that we're familiar with now we can change that how many lines it shows us if we were to do head dash n 20 it's going to show us 20 lines the first 20 lines of the file so see it's a little bit longer now and it's shown us all 20 lines now head isn't usually as commonly used as its companion which is tail so let me clear the screen oop and it spell clear right clear the screen now if we were to do tail two cities this shows us you've probably already guessed it the last ten lines and we could do the same thing with the dash n and a number we could decide how many lines of the file we want to see now this is really useful if you're looking at log files and that's almost exclusively where i use the tail command if i'm looking at a log file i just want to see the last things that were written to a log file so i'll do a tail of the log file in question and i'll see what was added to the very end so i don't have to look like 27 megabytes of text for all of the logs just the last little bit of it and so that's a very useful command and again you can use n20 if you want to see 20 or whatever number you want to see 10 is the default now the other ones i want to show you are less and more we'll start with more this is the older command so let me clear the screen if we were to type more two cities this is going to show us the entire file and if we want to scroll through it we press the space bar and it'll go to the next page space bar go to the next page the enter key will go line by line but this is that's it so we're all the way to the end of the into the first chapter but that's how more works you just kind of go through it like that you can also search but it only searches down and it's not one that i use very often anymore because it's been outmoded by the much more powerful although it has a more diminutive name less so actually let me clear the screen so if we were to say less two cities now we have what looks like a similar kind of interface but we can use our arrow keys to scroll up and down page up and page down work so we don't have to worry about like hitting the spacebar to go down a page we can spacebar will go down a page but then we can scroll back up with the up key now the other really nice thing about less and more does this to an extent but less is even more powerful if you type forward slash in a term so let's search for france press enter it's going to take us to the first entry of the word that we searched for it's going to highlight it and it's highlighted all of them so if we were to press forward slash and enter again it's going to repeat the same search and here we are francis found again france is down here again forward slash it'll take us to that one put it right to the top of the screen so we can search through an entire text file as well so it's very very powerful to use the command that seems like it would be less powerful because it's named less but really it's a lot better okay so to get out of here and this actually confused me for a long time if you just press q just the letter q it'll exit the less command and get you out of it so less more head tail they're very commonly used i usually use less and tail more commonly than the other two but that's just because i want to see the end of a log file and i want to be able to go up and down when i'm scrolling through a text file and search really powerfully it's not a really tough nugget because these are pretty straightforward tools they're all useful and you'll probably find yourself using them fairly frequently on the command line sometimes when i'm at the grocery store i wish i could search for where things are like i think pizza sauce should be right next to spaghetti sauce but it almost never is well thankfully when it comes to searching for text in a linux system there is an awesome tool that allows you to do just that narrow down what you're looking for with tool called grep now greb does use regular expressions or regex and if you're interested in the you know very fine-tuned filters you can get with regex i cover that really great in the linux foundations course but today i want to talk about searching for strings of text using grep because it can be a real powerful way to get the information you want really quickly now i said that grep uses regular expressions so if you want to make sure that it's absolutely searching just for strings of text you can use the dash capital f flag and that means just fixed strings usually you don't have to do that because if you just search for a string it's going to generally find it in the file but occasionally your string might be regular expression characters and you can cause yourself some headaches so if you want to be safe use dash capital f i usually don't because it's usually not an issue but i just want you to be aware that if you do grep minus capital f it's going to just search for strings now there's two different ways that we can use grep we can say graph the string from a file and it's going to search the file for the string that we specified and that works really well but there's also another way you can do it you can do this cat file or anything that has a text output like ls or anything that's going to output text and then you can use the pipe symbol and kind of push it through grep and search for a string i'll show you why this is a really powerful way to use grep because it seems a little backwards like why wouldn't we just say you know grep this string from this file i'm on an ubuntu system here and i'm just going to search a log file okay so i'm going to say grep now we can say dash capital f or we can leave that off i'm just searching for strings i just again want you to know that capital f is going to force it to just use strings but i want to grep for dhcp from the var log syslog file press enter and it's going to find all of the lines in that text file that have dhcp in it actually even highlights the dhcp which is really convenient now there's another place we could get some system log information and that is using the dmesg command but here's the problem that's a command that has output but it's not a file we can't grep the d message command so that's where the pipe symbol comes into play and it works really really well we could just say d message and instead of just having it print to the screen we can use the pipe symbol which is usually above the enter key in a us keyboard grep i'm going to use minus f this time we don't have to necessarily dhcp and then it's going to take all of that output from d message and grep for dhcp and sure enough there's two lines that have dhcp now another really useful way that we can use the pipe symbol let's clear the screen because i want to show you that first one that we did right we we actually grew up for dhcp from that file and we got these results let's say we had just pages and pages of results and we wanted just to look for things that mentioned init init what we could do is kind of like chain grips along we could say grep dhcp from var syslog and then pipe those results into grep init and press enter and now we're just going to get the lines that were in this result that also contain the word init and so here now we've filtered all the way down to these two lines of text from the log files so even if you're not getting super fancy with regular expressions you can do some really powerful searching of strings using the grep tool for things like log files or any kind of text that you want to search for and remember you can chain those grep commands together so that you get a really fine filter looking for exactly what you're looking for every application in the linux system has three sort of like pipes it has standard input standard output and standard error and basically it's just a way to get information in and out it's an io type situation for every individual app now i have just an application here drawn out and i want to show you the difference between the three so standard input is pretty easy to understand right this is like if you're putting something into a program we use the pipe symbol if we're going to pipe something into it or we can use less than if we want to just assign a file to the standard input now a lot of programs don't accept things on standard input but some of them do so if you've ever seen me pipe a command into another command what i'm do