Linux/Mac Terminal Tutorial: How To Use The rsync Command - Sync Files Locally and Remotely

Key Takeaways

Hey everybody, how's it going? In this video, we're going to be learning how to use the rsync command. Now, the rsync command allows you to transfer and sync files efficiently across different directories and machines. So, if you ever want to set up a backup for your files or sync files across different machines, then rsync would be a great way to do this. So, it's a lot more efficient than just copying files because arsync only copies what's different. So, for example, if you were backing up thousands of pictures, then copying all of these every time would be way too inefficient. So if you did this with rs sync then it would only need to sync what's different since the last backup. And also if you were copying data and lost a network connection or something like that then when you reran the rsync command it would know where it left off and wouldn't have to start from scratch. So this is an extremely beneficial command to know how to use properly. So let's go ahead and get started and look at some examples. Now rsync might already be installed on your machine. If you're on a Mac then it should already be available to you. And if you're on a Debianbased Linux machine and don't have arsync installed, then you can install that with apt get install and install arsync. And if you are on an RPMbased Linux machine, then you can install this with yum install arsync. Okay, so when you have arsync installed and ready to go, let's go ahead and look at a very simple example. So I have two directories that I created here on my desktop. And this top directory here is called original and the one below is called backup. Now the original has some images of my dog and the backup is currently empty. So let's run an rsync command to sync these two directories and copy the images from the original to the backup folder. So in the terminal I'm just going to navigate to where these directories are located and that is on my desktop. And now to use the rsync command I'll just type in rsync. And here we could add some options for rsync. And we'll look at that in just a minute. But for now, we're not going to give any options. So now we just want to put the source directory that we want to sync. So in this case, we want to sync the files in the original directory. And since we're not using any options, we actually have to specify the files and not just a directory. So we can say original and then a star to sync all of the files. And now we want the destination. So in our case, this is going to be the backup directory. So if we run that, then we can see that it synced all of the files from our original directory here into our backup directory. Now, there's no harm in running this again. So it won't do anything since there's no files different or any files that need to be reincted. So you can just rerun this again at no harm. So we can see that I ran that again and it didn't recopy the files or anything like that. Now the way this is running right now, it won't actually copy directories. And that is why we needed to add the asterisk to specify that we wanted all of the files within the original. But a more common usage for rs sync is to sync everything in a directory. So for example, if we were to create a subdirectory within original. So I will just say make dur original and we will create a new directory called more pix. And now if we reran our previous rsync command, then we can see in our output here that it said skipping directory uh of more picss, the one that we just created. And when we look in our directories, we can see that that was created in the original here, but in the backup, it was not copied over. So if we want to sync the entire directory then we can use the -ash r option to say that we want to recurse into directories and copy the total contents of the directory and subdirectories. So if we just rerun this and we give rsync a - r option now we no longer need this uh asterisk here. So if we run this then if we come over here to our directories then we can see that we have our moreepix subdirectory within the original and also that was copied over to our backup. So that worked uh as planned. Now when you do it this way notice that I have the trailing slash here after the original directory. Now that's actually important. Um so that means that we want to sync the contents of the original directory. If we leave it off then it will sync the directory itself. Um, so really just let me show you the difference here. So I will sync only the directory and I'll leave that trailing slash off. And now if I run this then without that trailing slash, if we look in our backup directory here, then we can see that instead of just syncing the contents of the original directory, it actually synced the original directory itself. So, if we go in here, then all of the contents and the subdirectories are still here, but it actually placed that folder within uh the backup directory here. And these are just left over from the last sync. So, if I to illustrate this a little further, let me delete everything and then rerun that same command again. And now we can see that it just copied that original folder over. So if you want to sync the actual directory, then you can leave that trailing slash off. But most of the time, my intention is to sync the contents of these directories. So you want that trailing slash. And when you include that and run it, then you can see that it copies everything and syncs everything as it should. And I'll go ahead and delete that original directory. Okay. So this R option is great for getting all of the contents. But I'd say a more common option for doing this is the - a option. So the - A option is actually a combination of many different options. So it stands for archive and not only does it recurse into directories, but it also copies sim links, preserves the permission, uh modification times, groups, owners, and things like that. So I like to use the - a option most of the time to copy everything. Okay, so now let's look at a few more useful options. And I'm going to delete the contents in this backup directory again. Um, now if you are about to run a large rsync job that syncs a lot of files and folders, then you probably want to do a check on what files and folders are going to be synced so that you can make sure that it's what you expect. Because checking what's going to be synced before you actually run the command is a lot better than realizing afterward that you spent a lot of time moving gigabytes of data to the wrong place. So to check what files are going to be synced without actually syncing the files, we can use the dry run option. So we can specify that we want to perform the dry run option with d-d dryun or we can use dash in for short. So just to be more clear for the video I'll write out the longer option there. Um so let's say that we wanted to do a dry run of syncing our original folder contents with our backup directory. So to do that we could say arsync and then instead of the -ash r I said that I like to use the dash a for archive that includes those permissions and everything like that. And now to do a dry run we can do d- dryun. So now if we run this then we can see that it doesn't actually show us anything. And if we look in the backup directory then we can see that it didn't actually sync those files or directories. So, it did do a dry run, but the point of this is for us to know what files are going to be synced so that we can make sure that it looks correct. So, to display this, we need to pair dry run with the verbose output. And to get the verbose output, we can include the -v option. So, let's add the -v option to our last uh run here. And I'll just include that after the a. So, now we can see that it displays the files that it would have copied to the backup directory. But since we had that dry run option as well, it didn't actually make those changes. So this allows us to make sure that everything looks correct. And if it does look good, then you can run the same command without the dry run included. So now if we take the dry run away and run that, then we can see that with that verbose option, it told us exactly what it did. And without the dry run, it actually did perform that action of syncing those files and subdirectories. Okay, so just to show you why arync is so much more beneficial than just copying files over, let me delete a couple of these files here from my backup directory. So I'll just delete two pictures here. And now if we rerun that command with the dry run option from earlier and run that, then we can see that it knows that it only needs to copy these two files and not any of the others. And these two files that it says that it will copy over are the ones that we just deleted. So you can imagine that this would save an incredible amount of time. If you had a lot of files or a lot of images, say you had thousands of pictures, then copying all of those every time you want to do a backup would be extremely uh you know inefficient and almost impossible to do. And the rsync command makes this a lot easier. Okay. So what happens if our backup directory contains some files that our original directory doesn't have? So for example, let me add a simple file to our backup directory. So I'll just do a touch and backup and I'll just create a t a file here called test.txt. And if I create that, we can see that it was created in our backup folder. So now our backup directory contains this test test test.txt file, but our original directory does not. So let's run another dry run and see if this matters at all. So, we can see that it still wants to sync those two files that we deleted from earlier, but other than that, it doesn't seem to care that there are more files in the destination. Now, it's just going to move over the files that exist in the original directory. Now, if we want our destination to mirror our source and get rid of any files that don't exist from our source directory, then we can add the d-delete option to do this. So, we could say before our dry run here, I'm going to add a dash dash delete. And I'm going to make my screen a little larger here so that that all fits. So now, if we rerun this with the dry run, then we can see that it still wants to sync the two files that exist in the source directory, but it also says that it wants to delete test.txt. Now, it also says that it wants to delete this uh uh DS store. Don't worry about that. That's just a hidden Mac file that gets created sometimes, but apparently that is another file that exists in the backup directory that does not exist in the original directory. Now, you have to be really careful with this because there are some horror stories online where people have accidentally set this delete flag on an empty directory which resulted in their entire backup folder being wiped out. because if their source directory is empty and you say that you want to delete everything in your backup that doesn't exist in the source directory, then that would be everything. So, you do want to be extremely careful with that. And I don't use it a whole lot. Now, I could see this being useful if you had a website or something that you wanted to keep your local and remote files synced perfectly together. So for example, if you deleted an HTML page on your local machine and then ran arsync, then you would also want that to be removed from the remote machine. So you definitely have that option, but again, just be careful when using that option. Okay, so now let's actually see what it looks like to sync files between a local and a remote machine. Now, I have a test machine that I have SSH access to, and I have its IP address. Now, if you'd like to have some test machines that you have SSH access to so that you can follow along, then I do have a video on how I set mine up, and I'll put a link to that video in the description section below. So, I have a test website in my projects folder with some HTML, CSS, and JavaScript files. So, if I uh CD back to my home folder and do an ls on this my site directory, then you can see that we just have an HTML file, a CSS file, and a JavaScript file. So let's say that I want to push these out to a web server and we'll pretend that my web server is the virtual machine that I'm currently running. Now if you're transferring data to a machine over the network then arsync does have the option to compress data while transferring. And to do that we can use the - z option and also a useful option for transferring over the network is the dash capital p which will show the progress. So to sync this project I can say rync and then dash z to compress a for archive that we saw earlier capital p to show the progress and then we want to set the projects and my site as our source. Now remember what I said earlier about that trailing slash. Now in this case I actually want to transfer the entire directory and not just the contents. So I'll leave that trailing slash off for now. And now we can put our destination. And the destination is going to be our remote machine. So first I'm going to put my username. And I'm going to make this a little larger here so that we can fit all of this in. So this is going to be almost like sshing into the machine. So our username at and then the IP address. The IP address for my test machine here is 1 192.168.5600. And then we want a colon to specify the location. And I'm going to put this in the home directory and inside this public folder. And then I'm going to put a trailing slash there. Now I do have keybased authentication set up for my test machine there. So it didn't ask me for a password. Now if you don't have that set up then it might ask you for a password to fill in. And then after you insert that then you should see output similar to this here. So we can see that it shows us what it sent over and what the progress was. Um, so now let's SSH into that machine and make sure that everything got there. So I'll just say SSH and the username. Then the address is 192.168.5600. Now let me navigate to the directory where I synced those files. And there was not a folder in here before. And now we can see that it exists. And if I cd into that directory and list all of the files, then we can see that the files were synced between our local machine and this remote machine. And again, this is better than just copying the files because it'll only sync the differences. Now, we can do this in reverse also. So, let's say that we had a cron job on this test web server here that created a database backup or something like that. So, let me create a directory within our website. And I'm just going to call this backups. So I'll do a make dur and call this backups. Let me clear my screen here. So I created that backups directory. And now within that backup directory, I'm just going to create a test file called backup.sql. And we can just pretend that this is a SQL dump into a backup file. So now that that file exists on our remote machine, I'm going to exit out of that machine. And now we're back on my local machine. So now let's say that we want to sync that backup from our web server to our local machine. So we can just use the servers folder for our source and say so I'll just do an rsync and I'll do those same options. The Z is for compression, A for archive, capital P to show the progress and then corems at 192.168.56100. And now we want to specify the location of that backups directory on our remote machine. So we'll put in a colon and that was within public my sitebackups and now we'll put in a space and now we want our destination and our destination is going to be on our local machine and I will just put this within our projects uh my site directory. Now you can put this anywhere but I'm just going to leave it within that my site directory. So if we run this then we can see that it looks like we received our backups directory and our backup.sql SQL file. And if we look in our projects directory and my site, then we can see that it does have a backups directory in there. And if we look within that directory, then we can see that we do have that backup.sql file that was pulled down from our remote machine. Now, a very common use case for arsync is to pair it with cron jobs. Now, if you don't know what cron jobs are, they're a way to schedule commands to run on regular intervals. And I have a separate video on setting up cron jobs and I'll leave a link to that in the description section below. Um, so pairing cron jobs and arsync together allow us to run regular syncs and make sure that backups and things like that remain up to date. So it's extremely useful just having that working in the background without really needing to think about it. So if those are the kind of things that you've always done manually as far as you know backing up a lot of pictures or pulling down backups from web servers or managing large projects then you know pairing up some cron jobs with some arsync commands uh just makes it to where you really don't have to worry about those things and it just does it all on its own automatically. Okay so I think that is going to do it for this video. I hope that now you have a good understanding of how rsync works and you have some ideas for how you can use this within your workflow. But if anyone does have any questions about what we covered in this video, then feel free to ask in the comments section below, and I'll do my best to answer those. Now, if you enjoy these tutorials and would like to support them, then there are several ways you can do that. The easiest way is to simply like the video and give it a thumbs up. And also, it's a huge help to share these videos with anyone who you think would find them useful. And if you have the means, you can contribute through Patreon, and there's a link to that page in the description section below. Be sure to subscribe for future videos, and thank you all for watching.