Robots.txt - Search Engine Optimization Tutorial part 5

Key Takeaways

hello and welcome to part 5 of my seo tutorial here we're going to be discussing the editing of your robots.txt file and some use of regular expressions to make this a little bit easier as we discussed in the previous video it's really important to specify what urls to not crawl in order to keep the indexed content of your website updated frequently or even at all so you want to keep in mind um as well that even if you suggest it's more of a suggestion than a forceful action that you'll be taking in this robot dex text file so what i mean by that is it only suggests a google or a robot not to crawl a page it's not going to prevent it from doing it if it wants to nor is there like any legal obligation tied to it right so there's something you really don't want crawled by google um you need to put it behind some sort of paywall or something like this that google like literally won't be able to get to so if you just say hey please don't visit this page that's not good enough so anyway with that let's go ahead and dive into the robots.txt file now if you just make a website yourself totally by scratch you you're not going to have one of these i mean some hosting plans i think will just automatically slap one in there for you but it really just depends so this robot.txt file if you don't have one you just straight up make it it goes in the same it's like in the same directory as your index.html or index.php or whatever your index page is you want to have that in that same directory so as you'll see we can go to robots.txt and visit myrobots.txt file so what this means is you've got the user agent what this says is what user agents are we gonna allow to actually like view this this um this website and this star here is probably what's gonna be your little favorite little tool here it basically just means anything right so it's i'm not even sure i would say it's a regular expression necessarily it just means zero or more repetitions for regular expressions as far as that's concerned um you can't really use any fancy regular expressions in here so just keep that in mind then really the only thing you need to understand is that the star or the asterisk means anything so for example user agent we're allowing any user agent to view this website a good example would be to disallow various user agents or you could actually just specify all the user agents you would want and if they don't match that they would be thrown out so in some of my tutorials i show you guys like how to use python to visit websites well when python visits a website its user agent most likely is going to be url url lib2 and maybe sometimes your python version right so so when websites see that they can just be like nope i don't want you visiting my website and they give you like a 404 or something like that um so i might even be like a 500 and i don't i forget what it is i don't see it anymore because you can edit your user agent so it's not a big deal but that's just an example of what you can do anyway so this one this robot.txt file says any user agent is able to visit my website so this is including all the robots now you use disallow to not allow um something to view various pages so so obviously this wp-admin it's a link that exists but we don't we don't want the robot to visit it because it won't even be able to visit it wp-includes same thing it's not going to be able to visit that stuff just keep that in mind you can use that that way like for example this is a search function right here so basically it's disallowing any uses of the search bar right because that would result in an infinite number of searches possible so this will block all of that uh from happening at least with the google bot right or actually well any user agent and any robot that comes through here trying to go through all of our links isn't going to waste its time with that so just keep that in mind also if you don't you know if you don't have that again all you have to do is if you have ftp access or any sort of hosting access at all you just literally write up a robots.txt file and you know put all this stuff in it you don't need this right here it's it can be suggested to put sitemap colon and this just tells it where the sitemap is this is not required here but you can use this and just literally just type this stuff out put it up and that's all you have to do it's very basic but again very highly recommended especially if you have any sort of any sort of dynamic content if your whole web page is static it's not as big of a deal i would still put user agent and the asterisk there but obviously don't disallow anything so anyway that's it with robots.txt what you need to edit and why you would want to edit it as always thanks for watching