Python Tutorial: Calling External Commands Using the Subprocess Module

Key Takeaways

hey there how's it going everybody in this video we're going to be learning how to call external commands using Python with the sub process module so there are many scenarios where you might want to call an external program using Python and if you need to then you can also capture the output of those commands or even pipe the output from one command into another so that's what we're going to be learning to do in this video using the sub process module so let's go ahead and get started so I have a blank Python module open here and first we're going to want to import the sub process module so let's just say import sub process okay so now in order to run an external command this is extremely simple we'll look at how to do some more complicated stuff in a bit but to simply run a command we can just say sub process dot run and then run the command that we run a run so I'll do LS now just that heads up I'm on a Mac so I'm going to be using Linux commands in this video but you can use these same processes to run Windows commands as well so LS the command that I just ran here it will list the files and folders in the current directory and again that's a Linux command on Windows you have to use Windows commands so the LS equivalent on Windows is derp so di are whoops sorry dir so if I save this and run it then we can see that the LS command printed out everything in the current directory now if you're on Windows and tried to run the dir command or something similar then you might have got an error at this point that the system could not find the file specified so the reason is because the dir command is built into the shell so we'd have to pass in an argument of shell equals true so if you're on Windows then you could say pass in an extra argument here of shell is equal to true and if we run that you can we can see here at that on Mac we get the same result but on Windows that should prevent you from getting that error and also if we set the show argument to true then we can just pass in an entire command as a string so if I wanted to add more arguments to my external command then I could say so for example some arguments I could add to LS would be - L a so if I run this then now we can see that runs that LS command with those additional arguments and those additional arguments just provide more information here now if you are using this shell equals true then that can be a security hazard if you're using untrusted input so only use that if you're passing in the arguments yourself and be sure you're not running it with user input or anything like that but if you're not using the shell argument then we actually can't pass in the entire command as a string like we did here instead we need to pass in everything as a list of arguments so if we wanted to run this same command without the shell arguments set to true then we would have to say so let me get rid of the shell argument here so I could just pass this in as a list and I'm gonna have to do this as a list so our first item here is going to be the command LS the second item here is the arguments so now if I run that then we can see that we get the same thing okay so we can see that all we're doing right now is running this sub process dot run method and it's printing out the results just like we'd print something in a Python script now the reason is that we're not capturing that standard out the standard out just goes wherever the standard out of the script normally goes which is in the console here so let's try to capture this into a variable instead now you might try to do this just by saying something like p1 is equal to sub process dot run but if I run that then we can see that it's still printing the standard out here to the console but also let's see if we get anything within that p1 variable so if I print out p1 and run that then we can see that after our LS output here down here where we printed that p1 variable that comes out as a completed process object now if you want to see what all we can do with a completed process object then you can run help on that but let's just go over a few of those really quick so first we can check the arguments that were passed into the original command and we can do that by printing out p1 dot orgs so if I save that and run it then we can see that we have the arguments that we passed into that original command here we can also check the return code and the return code will show us whether we got any errors or not so if I print out P 1 dot return code if I run that then we can see that we got a return code of 0 and that means that it ran successfully now a way that I like to remember that is to think of that as meaning zero errors now we can also grab the standard out as well so let's grab that if I print out p1 dot STD out and run that okay so when we print it out that standard out we can see that we got none and that might be a little surprising to some people so we got a result of none but really what we want is the output from the external command that we ran so the reason that it's printing none is because like I said before it's just sending the standard out to the console all of this stuff here so we want to capture that instead so to capture that we can just pass in another argument here and we can pass in an argument of capture underscore output is equal to true and if I spell that correctly if I save that and run it and actually first we can see that we did get something here first let me comment out the print statement we're printing the p1 standard out and if I save that and run it then you can see that now we're not getting anything in the Python console here so it's no longer outputting that just by running this sub process dot run command so now it actually is capturing it in this p1 variable so now when we actually print out p1 dot standard out if I run that then we can see now it's captured there now one thing to notice here is that the standard out was captured as bytes so it doesn't look like it did before with all those new lines so if we want the new lines to actually be spaced out then we'll have to convert this to a string now there are a couple different ways that we could do this first we could just decode those bytes by saying p1 dot standard out dot decode run the decode method on that if I run that then we can see that it looks like it did before now I've never done a video on Unicode and bytes versus strings and encoding and decoding and all of that that's a tutorial that I'd like to put together in the near future but in this case when we decode those bytes it's converting it into a string now if we don't want to use decode then we can just get rid of that and pass in an argument to the run method saying that we want text instead so I'm going to remove the decode method there and I'm just going to pass in an additional argument of text equals true so now if I run that we can see that we're not decoding the standard out but it comes in as a string anyway so that's good okay so when we set that argument of capture output equal to true what that's actually doing in the background is setting the standard out and the standard error to the sub process pipe and that allows us to capture those values so let me show you what it would look like to set that standard out argument directly so instead of saying capture output is equal to true here instead I'm going to say STD out is equal to and that is sub process dot pipe and again that's actually what setting capture output equal to true does in the background but it also redirects the standard error to that pipe as well but just setting the standard out equal to that sub process pipe we should be able to run what we have here and get the same result so if I run this then we can see we got the same results that we did before now we can also redirect the standardout to other places as well so for example let's say that we wanted to redirect that to a file instead that could be used for logging or anything like that now in order to do that we could simply just open up a file and redirect it to there so I can say with open and I will just open a file here called output dot txt and I want to open that in write mode and I'll just open that as F now let me indent this so that's within our context manager there and now for our standard route instead of redirecting to this sub process pipe and said I'm just going to redirect that to F our file and I'm going to remove our print statement there because we don't need that because we're not capturing that using that sub process pipe anymore so now if I run this it doesn't look like we got anything but if I open up my sidebar here we can see that now I have this output txt if I open that up then we can see that we got the results from that command in our output txt file so that did work okay so I wanted to show how we could redirect that to a file but now let's go back to what we had before where we're just capturing that output in a variable so I'm just going to get rid of all that file stuff and instead I'm just going to set this back equal to capture underscore output and set this equal to true okay so now let's take a look at a couple of different things that can happen if our command isn't successful so let's see what happens when we get an error so I'm going to change my command here to instead try to list the contents of a directory that doesn't exist so I'm just going to add a another argument here and I'm just going to add this to a bonus directory I'll just say dne for does not exist and this argument that LS command is going to try to list the contents of this that doesn't exist so if I run this then we can see that we don't get any output and that's because we captured it here now one thing that might surprise some people is that we don't get an error within Python some people might expect Python to throw an exception if the external command fails but it doesn't do that instead it just returns a nonzero error code so again we can check that just by printing out p1 dot return code and we saw this before so if I run that then before that was 0 but now it is 1 which means we got an error and if we want to see that error then we can print that out as well by saying stderr for standard error if I run that we can see that the error that that command got was LS DNA no such file or directory because that directory doesn't exist so within your script if you only wanted to proceed if that command was successful then you could put in a conditional if you wanted to so something like if p1 dot return code is not equal to zero then that would be your error case or equal equals zero for it not having any errors now if you did want Python to throw an exception if the external command fails then we can just pass in an argument of check equals to true so if I add in an additional argument here I'm going to say check is equal to true and now if I run this then we can see that now Python throws an exception here and we got a trace back and it says that our command returned a non zero exit status of one so that might be useful depending on whether or not you actually want Python to throw an exception if your command actually fails or returns a non zero status code now another common thing to do with errors is to just ignore them by redirecting them to something called dev null and redirecting them to dev null just means that you are ignoring those so we can do this with the sub process similar to how we've redirected standard out using the pipe so we can replace our capture output here and instead before we did STD out this time I'm going to do stderr for standard error and we're going to redirect that to sub process dot de Vie in you ll Dev null and that's all caps so now if we save that and run it oops and I'm actually getting a trace back here still because we still have check is equal to true so I'm just gonna get rid of check also I don't need text either so I'm just going to delete both of those okay so now we're redirecting that to dev null if I save that and run it we can see that it just redirected our error so we don't have a standard error here it just says none okay so so far we've been looking at how to capture output and errors but we can actually change the input that different commands receive as well so for example let's say that we wanted to take the output from one command and have that be the input to another so let's look at an example of this so for the first command I'm just gonna run a cat on a file which is a linux command that will just print the contents of a file if it just has one file as the input and for the second command I'm going to use the output of that first command to grab that file and grep can be used to search the file for certain contents and again these are Linux commands since I'm on a Mac but you should be able to do something similar to this using different windows commands as well okay so first I'm going to run the cat command on a file that I have in the same directory here called test dot txt and I actually have test exe open here and sublime so let's take a quick look at this so we can see that this is just a very simple file with seven lines that says this is a test with seven lines okay so in order to run the cat command on that file I'm going to change this up a bit so instead of LS I'm going to run cat and I want to run cat on that test dot txt file in the same directory I don't have any more arguments after that and I don't want to redirect standard error to Dev null instead I'm just going to set capture output equal to true so that we can get that output and any errors okay so to see if that worked then I'm just going to print out our standard out so if I save that and run it then we can see that we got the contents of that file it's in bytes right now so I forgot to add text is equal to true to make that a string so I'll save that and run it and now we can see that we got the same contents of that file okay so now in order to use that output as the input of the grep command I'm going to copy what we have here previously and I'm just going to change this a little bit so let me paste this and now I'm going to change this to be p2 as our variable and now I want to run grep and I am going to do a couple of different arguments here so I'm going to also pass in an argument of - in which gives us the line number that it finds a match just so we can test this and let's look back at our test txt I will search for the text of test and that's on line four so I will pass in a test for the word that we are searching for okay so I'm also going to keep capture output equal to true and text equal to true but I'm also going to add one last argument here and I'm going to say input is equal to and I want this input to be equal to p1 dot standard out so if I save that I'm going to comment out the print statement on the p1 standard out there so we're just passing that in as the input for that grep command and if I print out P 2 dot standard out and run this then we can see that grep says that it found test on line four of that output from the first command now that's a bit of a contrived example because you can just grab files directly but it might be useful if you want to capture the output of this command first and do some type of processing on it and then pass that into the grep command so it's definitely useful to know how to do this at times because it's nice being able to grab that output from step to step but if you're not doing anything with the output between the steps then you can definitely just use that shell argument that we saw earlier and just write out the whole command using pipes and Linux if you're comfortable doing that instead so what I mean by that is that you could you know instead of doing this whole process here we could just say shell is equal to true and then instead of passing this in as a list of arguments we could just say cat test oops I got to get rid of that square bracket as well we could just pass all of this in as a string and then do a grep - in test and if I print out the standard out of that then we can see that we got the same result but like I was saying it's definitely useful to know how to pass in the input to different external commands because you know if you are doing step-by-step processing and doing some string parsing or something like that using Python then you can pass in those results into a different command so that's definitely useful okay so I think that's going to do it for this video I hope that you found it useful learning how you can call external commands using Python and also how to redirect those outputs and inputs and things like that it's definitely useful especially when I'm writing scripts on a web server or something like that it makes it possible to run some background commands within the Python script itself and process that that output further if need be but if anyone has any questions about what we covered in this video then feel free to ask in the comment section below and I'll do my best to answer those and if you enjoy these tutorials and would like to support them then there are several ways you can do that the easiest way is to simply like the video and give it a thumbs up and also it's a huge help to share these videos with anyone who you think would find them useful and if you have the means you can contribute through patreon and there's a link to that page in a description section below be sure to subscribe for future videos and thank you all for watching