Linux Essentials: Curl Fundamentals

Key Takeaways

hey guys hackersploit here back again with another video i do apologize for not uploading last week uh as you already know if you follow me on twitter that i wasn't feeling all that well and i was pretty sick but i have recovered now and the normal uploads will resume so this video is going to be focused on essentially getting you to fully learn and of course utilize curl to its maximum potential and i'm going to be covering as many commands as i possibly can now that being said curl has a lot of functionality that you need to explore on your own as well but i'll be covering some of the most important bits of functionality that it does offer to a normal user or a power user as well so uh without any further ado let's get started so for those of you who have never heard of curl curl is essentially a utility that allows you to transfer data to or from a network server using one of the supported protocols now the protocols that it does support are http https ftp ftps you also have your sftp uh you have tftp telnet et cetera now there are various other protocols that it does support in addition to the ones i've just mentioned uh but i will get into that later now the great thing about curl is that it is a multi-platform tool which means uh it works on linux uh mac os and windows now i'm gonna be covering how to use it on linux but of course the commands are pretty much the same now in regards to installing uh installing curl it as far as i know it comes pre-installed with most linux distributions but if you want to install it whether or not using the aptitude package manager you can use the sudo apt-get install curl command and if you are if you're using an archbase distribution you can use the pacman s command to install it and the the name of the package is curl all right if you are using a mac os installation or mac os as your operating system you can use the brew install command so that is brew install and curl all right and that's pretty much uh simple right over there with windows you need to download an executable or a bin file if i'm not wrong and you need to save it in your windows directory and of course from that you can directly launch it from your command prompt that being said let's get started now curl as i mentioned comes pre-installed with most linux distributions so let me just clear the terminal here and i'll maximize it so we can see what's going on all right so the basic syntax is as follows but before we do that let's open up the help menu you can explore the help menu if you are if you're really looking for a bit more functionality or you're looking for specific functions as you can see it's a very comprehensive tool and i'm primarily going to be covering uh the basic commands that you that allow you to utilize it quite well uh that being said many people ask me how i personally learn about a command that or a tool that i'm not familiar with i personally use the man or the manual for each of these tools so the manual contains a very very good description of of all the bits of functionality in regards to a tool uh and all linux tools pretty much uh come with a manual and the manual is extremely useful so if you you want to check them out go ahead and do it i really really recommend it when learning how to uh how to use a tool all right so let's talk about the basic queries you can perform with curl all right so when i talk about interacting with the various protocols this particular video is going to be focused on http in https i know there is ftp functionality but i'm not going to be getting into that because that's more that's where things sort of get a bit different but for now let's focus on those two protocols all right so when performing a basic query of a url uh you essentially type in curl and then the url so for example if i type in hsploit.com here and hit enter what will happen is this will essentially fetch the content of the specified url so for example i've hit curl https hsplit.com and it will essentially return to me the content of the entire web page now you might be wondering well how can we verify this and that's where we get into essentially saving your output into a file or downloading files so i can say for example curl and if i want to output the uh the output of this command or the contents of this query i can say i can use the lowercase o and i can then say i specify the directory i want to save it in so home alexis desktop and i'll just uh sorry that is desk that is desktop and i'll just call it hsploit dot html and after that i'll type in hsploit.com right over here that is the url now it's very important to specify uh the protocol that you're using so for example if a website is http you won't need to specify to get it if it is using the https protocol it is very important but i'll get into redirection in a second so https hsp.com and we hit enter and it's going to give you this little status right over here essentially giving you an update as to what is the current status of the entire process here now for some reason it's taking quite a while now but we'll wait for it to complete and there we are so let me explain a bit of the uh the the structure of this little status menu or table here so for example you have the total amount to be downloaded uh the amount received the average download speed the upload speed the the total time are the time spent and the time left and uh right now if we cut uh the the hs flight dot um well actually if we need to go into our desktop here uh so if we capped hsploit.html here you can see the entire contents now if we open that up here if i open that up with uh with chrome for example let me just open this up where is google chrome um where is google chrome here there we are google chrome if i just open that up here in a web page you can see that pretty much it has all the content of my website so each exploit.com and it's pretty cool that it does save the entire contents of the html file here now of course this is going to um this is going to leave out extra bits of files in regards to the the web page but this is a great way of saving the contents of a query now that could be for any other protocol it not it does not necessarily need to be for the http or the https protocols all right now when we talk about downloading files that is pretty much how it goes for a web page or for a query so let me just remove the hsp.html mind you i could have saved it as a txt file i simply i'm saving it so that i can use it at a later time now you can also use curl to download files from the web which is really really awesome and that's primarily why i have the ubuntu page open here because i want an example to use so for example if i wanted to download the ubuntu iso i go to the download page and i select the a version 18.4 uh 18.04 lts and it's going to start the download so if i just going to show all downloads here and let me just copy the link here the download link so i can go into curl and i can say for example i can give the file name a new name or i can just download it directly so let me explain the two ways of doing it so for example i can say curl i can output the uh i cannot i can say download this file as ubuntu iso dot iso sorry iso and then i i paste in the the download link and what that is going to do is that it's going to download the file and save it as ubuntu iso dot iso now the extension is very important because if you get that wrong it's going to save it as a different file type all right so if i enter you can see it's going to start the download and there we are we can see that the total amount to be downloaded is one uh 194 megabytes which is about 1.9 gigabytes and you have the received the download speed the average download speed and the current download speed right over here the time left etc etc now you can see that my internet speed is pretty bad right now but that's primarily because i am updating one of my systems so i do apologize for that so if we check the desktop you can see we have the ubuntu iso over here which does prove that it does work all right now the other way of doing it let me just remove the ubuntu iso.iso file there the other way of doing it is by downloading it with its original file name which is specified on the on the web server so to do that i simply type in curl and i use the capital o and i paste in the download link and hit enter and that is going to download the file with its original download name so you can see ubuntu 18.04.2 uh desktop amd 64-bit dot iso or md64.iso all right so that is pretty much how to download files and now that i've explained that we cannot talk about redirects because redirects are extremely important so if i just get rid of ubuntu here so if a website is redirecting to another url it's very important that you know how to specify this to curl and why is that important so in many cases redirects may be set up and of course that is specified by the http 300 request or the it could be any 300 requests mostly theo one or three or three and it's very important that you know how to specify this with call now the ht the http protocol syntax is very important in curl and you can do this by using the l command or the capital l command sorry so for example if my website was uh if the urlhsflight.com was uh so if i say hsploy.com was redirecting to another url to another url and i wanted to specify this to curl i would use the l command now i know that my website is not and a great way of demonstrating this is by using uh the so for example if i say curl http hslight.com right over here and hit enter and we'll give that a few seconds here and i'll explain what's happening in a second so that you can see the importance of specifying uh the l parameter if there is a redirect so you can see i got no result and that's because curl is going to http and that's the importance of protocols with curl you need to specify the correct protocol irregardless of whether or not you're using ftp http https so my website is https now my website by default will redirect however you need to specify that with curl i do apologize for that and let me just clear the notifications here so again i i as i mentioned you need to specify this with girls so i can say curl and i use the l command over here and then i specify http uh and then i say h exploit.com here right over here and hit enter and that will essentially redirect me to the appropriate webpage and as you can see right over here we get the entire contents of the hsp.com website or html file rather so that is how to do essentially how to work around um how to work around redirects all right now another bit of important another important command sorry is the querying response headers so you can also view and analyze the response headers being sent by a particular web server and this can help in web assessment so for example if i wanted to query the response uh the response headers i'm being sent back by the web server i can say curl capital i and i can say https hs plate dot com and i can just hit enter and what's going to happen here is it will give it a few seconds this is going to depend on uh varyingly depending on your on your particular web server so you can see it gives us the protocol we have cookies here uh we have the php version so we're getting important information now uh looks like we have some wordpress json files here if i delivered by the cdn or the content delivery network uh let's look at some important cookies here that might give us a bit of information in regards to the website so an important bit of information here of course i'm using this as an example is the dwqa and this is my question and answer plugin that i use and you can see that it has an anonymous cookie that's being set here and that can give you information as to a bit more of how the web application is working so that is how to query the response headers being sent by the server and of course you can see right over here this is a cloudflare server and we have the cfra which is the cloudflare array that essentially uh uh is telling cloudflare uh what uh it's giving information to cloudflare about the particular client that is accessing the web server all right now that is very interesting however it really gets interesting because girl can also do a really cool thing it can also allow you to view the request headers and the connection details so you can view the tls handshake et cetera et cetera so if i say curl v and i say curl v https and they say exploit.com here and hit enter you can see that we can view the entire tls and shake which is really really unique and awesome so let me just see if i can figure uh which is wait which is right over here so there we are we can see the the initial connection here uh we i'm particularly looking for the tls uncheck and right over here we can see the tls handshake taking place so you can see that it is successful and this is a great way of identifying um problems with the tls uncheck if the certificate has an issue so for example we have the client hello the server hello we then have the encrypted extensions being exchanged the certificate is being exchanged and then verified and then finally the tls uncheck is completed you can see that we have the version tls version 1.3 and the ciphers that are being used or the encryption being used all right so you also have information about these certificates uh expiry date when it was actually registered uh the issuer all that good stuff so if you do want to view uh if you do want to view the connection details and a bit of an advanced information or additional information in regards to the connection you can go ahead and use the uh the v command now the last thing that i want to cover in regards to the http and http hts protocols is the fact that you can cross post requests and many people have seen me using this in ctfs particularly when i'm essentially brute forcing um login pages so i can give you an example so if i wanted to essentially test credentials on a website and this can really be in any uh i can be using any parameters i could be spoofing various cookies all that good stuff so for example i can say if i wanted to log in to wordpress site and i wanted to essentially test various credentials i can say for example curl data and that is the sorry the data that we're specifying and then in here i would specify log and this is the particular syntax for wordpress and the password would be equal to i can say password here and then i close the uh the uh the quotation marks and then i specify uh the wordpress page here so i can say https and you can say wordpress.com and i can say wordpress login dot php and that is the login page right over here and if i hit enter and i know this you can see that admin is not a valid username or wordpress.com etc you get the response so this is a great way of testing various pieces of data so you can also play around with cookies all that good stuff and in this case we're simply playing around with the login and the password parameters here and we specified our own values now if you do need to to encode the particular parameters you can also use burp suite for that so for example um let's see if i wanted to go um let me just uh turn intercept on here if i just went into hsploy.com here so i just say it exploit.com and it's a wordpress login.php sorry.php here and i just hit enter and uh for some reason we're getting stack uh stack we're just getting a stack protect here in any case uh the what i was getting to if you do want to encrypt your url or any other parameters you can use the control plus u key so that is pretty much all that i wanted to cover in this video uh let me know if you enjoyed this video if you have any questions or suggestions let me know in the comments section on my social networks or at the forum at hackersplay.org and i'll be seeing you in the next video peace guys [Music] you