Automating Web Scrapping Using AutoScraper Library

Key Takeaways

[Music] hello all my name is krishnak akam welcome to my youtube channel so guys today in this particular video we are going to automate web scrapping and for that we are going to use this library which is called as auto scrapper please make sure that you watch this video till the end guys because you will find an amazing functionality from this particular library we usually know web scrapping is a very tedious process because probably we want different types of elements from a web page and to scrap it and why it is used it is used basically for the data collection stage itself and while we are scrapping we face a lot of difficulties based on nested list probably nested html elements will be there and so many problems will be there but today by using this auto scrapper i'll just show you an amazing way to scrap scrap any web page that you actually want so over here this is uh what we are going to do the problem statement is that i'll just go to my github and here what i'll do i'll just go to my repositories okay now over here you can see suppose i want to scrap this entire page okay and let let us just consider that okay uh from my repositories i really want to find out my title name of the repository and probably the star okay i want to i want to take out this two important information just let's take an example over here okay i want to take the title i want i want to take the star how many stars are basically there and here you can see that i have so many projects where i have so many stars over here and we'll try to see whether we'll be able to get this information or not okay and probably you also want to know how many followers are there you can also get this it is up to you okay so let's try this so first of all what we do is that we install this particular library by just writing pip install auto scrapper so i'm just going to paste it over here and i'll just write exclamation once i do this you can see the requirement is already satisfied because i have already done the installation now coming to here the next thing what i have to do i have to just write from auto scrapper import auto scraper okay i'm just going to import this particular information this is the library that this is the class that we are going to use now the url i'm just going to take this entire url from the repositories this entire url you have to take take it uh from your page if you have a github profile it is fine i'm just going to put this specific url now what we want to do is that when we take this particular url we also need to take some of the important information which we really need to scrap now in this particular case i need to scrap this title and i need to scrap this particular star usually in the legacy system what we used to do we used to say okay go and inspect this particular element and probably i used to say okay what is this particular element we used to go and find out this h3 tag this class and based on that we used to scrap it but in auto scrapper nothing you just have to write some information you want to write just listed in a list you really need to write some information like whatever information you want to capture now in this particular case you'll be able to see that if i take an example i'll go over here and see there is a title called as car prediction so i'm just going to copy this i'm going to paste this as my one important information okay so this one important information that is called as candidates that that basically means what all information i really need to select from the entire page and here i'm just giving one title i'm not giving any tag or html tag here you can also give a regular expression okay and the second information i'm just going to give and you can give any stars you want but from here i'm giving 85 stars because probably in from my entire repositories at least this will have an 85 85 stars okay you can give other stars you can also give this 359 stars probably this particular title or any other title or any other stars from this entire repository but make sure that that particular star is unique the count is basically unique if you give it probably here you can see that there are two star away and probably i give two star from somewhere here this will get confused you need to provide the right information now in this particular case 85 stars i'm just going to provide over here so this is my information so this two information leads to this is basically my title of the repository and this is basically my number of repository number of stars so i really want to take this two important information initially from all my web page and then i need to get all the generic information now here i've just given one example of title and star when i when i scrap from that particular page what it does is that it tries to find out okay where this cross car price prediction is there so it will go and see okay here it is present star is also present over here now what it will do from this entire page all the unique titles and star will be scrapped not only unique if there are repeated also it will just scrap because it knows we want the title and the star from this entire web page see guys how amazing it is okay so let me just do it then after this we will just say auto scrapper we'll just initialize this and then we'll say scrapper.build and here i'll be giving my url the url is nothing but this entire tab repositories and this is my wanted list here you can put any number of elements probably you want followers also you can actually put followers so let me just write like this 8.4 k okay first of all let us try this and then we'll also try to see with respect to followers okay so here is my url and then wanted list now once i will execute the print result what it will do it will take out all the information you see this guys now see it is taking out 3 185 2 from where it is starting 3 1 85 2 right then we have 6 then we have 359 26 6 all these information are there and then i have all the information about my repositories that is the title name like type of transformation pi web heroku car price prediction all this information is done now just see this guys so amazing it is you are able to take out this entire information like that right you did not have to put any html tags and all now the next thing is that can i apply the same scrapper to some others repository because it needs to apply this entire technique in other urls also right so let me just consider i neuron repository so i neuron github profile okay i'll go to i neuron here i'll go and click on i neuron okay and this is basically my repository this is my company github repositories guys and from this particular repository now i should be able to get it now this will be my new url and i really need to apply this entire techniques on the new url also but before that what we need to do we need to do something called as grouping over here two main information is we are trying to take out one is the stars uh and the other one is basically called as title right so we'll try to group these two information in order to group we have something called a scrapper dot get underscore result underscore similar okay now what we are going to do let me just come to this book guys before this okay what we are going to do i am going to just say that okay and over here instead of giving i need on url i can give my own url okay so i'll just an entire github repository will be given to you guys okay so i'm just going to copy and paste it over here so here we will be using something called as scraper dot get underscore result underscore similar and i'll give my entire repository url and then i will say grouped is equal to true now we know that two information we are grouping one is star and one is title so here you will be able to see two unique id will be coming see this is one id rule underscore bx31 this has all the star information all the star information and the next rule is rule underscore mx 9a this has all the information regarding the title from my page okay so two groupings are there okay so what we will do is that we will now assign this groups okay we need to assign these particular groups and we can put our custom title or alias okay now we know that this particular group is basically star group so i'll just go copy this group and then paste it over here and i'm going to assign it to a new alias which is called a stars because this basically refers to the stars because afterwards whenever i try to take out any any information you know i'll be able to get that entire information directly okay like if i just want to from this particular repository i want to find out all the stars so i will be using this last name okay then coming to the next stage which i will say that okay over here you'll be able to see that i'll be using another repository another group id so another group id is nothing but this one rule underscore mx 9a why this was there guys probably i was just practicing some of the things previously then i'll just put away a title because these are all my title information okay all my title information then i will say scrapper dot keep underscore rules i need to keep these two rules so i'll copy and paste it over here okay oops yeah i'll save this i'll keep these two rules and i'll save this as something called as github repository search now i will be able to use this particular scrapper anywhere i want guys because i have been able to save it now in order to load it just see this example i'll just delete this okay now in order to load it i'll say scrapper.load okay and then here i can basically use github to search repository like how we save the model and load the model right similarly we can do this now what i'll do i'll write scrapper dot get result similar similar result i want a similar result from the i neuron repositories now see this this will be pretty much amazing and then i will try to save this in my result i'll save it now this has got executed now if i write result of title then i will be able to sorry result of title on list indices must be integers or slices let's see what is this okay okay just let me just see one more parameter that i really need to put which is called as grouped is equal to true okay we need to i just missed out one parameter which is called as let me just say shift tab and i'll say group by alliance is equal to true okay comma group by lis is equal to true because i have put an alias right now if i go and see the result here you will be able to get stars and title because i miss this particular parameter if i don't write this particular parameter group by alice that basically means i will not be able to see what uh what what are my allies that are used over here now i can go and write result of title and here you'll be able to see all the title names all the title of the repositories from the i neuron github okay it is pretty much simple guys see what are the steps that we did i'm just going to write it first step we are we have first of all installed right installed auto scrapper okay done then we took a specific url that is my repositories i've told that what all information that i'm going to get from this one is basically from my repository i told that okay let's give me a unique star and title these two information i really want to get it right so that is the reason why i gave given these two information these are called as candidates and then i'll be building based on this particular url and wanted list after that what i did i can also find out results similar with some kind of grouping right so over here you have to make sure that you write group is equal to true so here a unique id will be assigned to each and every candidate right because i've just given a generic number and it was able to scrap the entire information from this entire page right isn't it amazing then after that i had assigned one alias that is the rule allies then we kept the rules and then i have loaded this or saved this in the form of github repository search and now i can apply the same thing to any website with the same url like this it can be any github profile and there i have to just use group by alias is equal to two and i'm able to get this particular information now let's see that if we want one more information which i told that which is about 8.4 k followers now i want this particular information also let's see so here i will write one more information which is like 8.4 k okay so this is the information now let me just execute it probably i think it should be working uh let's see over here we were okay here we have got 8.4 k is it 8.4 k or 8.40 let's see inspect okay it is somewhere around 8.4 k okay now we have we have got that particular information now what i'll do i'll again group it okay again this id will change guys always remember this id will change uh as we try to because it assigns a new id itself right so i'm just going to try it out this one this one i hope it should work okay and then the third one is something like this one 8.4 k okay some information this 0 and 2 some more information it has got scrapped i think uh this is the information from uh this one zero following and star two okay this information is also picked up entirely okay i can also write zero following and this star also that information will also come up see how amazing it is it is getting automated now i'll write this particular rule as my third parameter and here i'll say this is my followers followers okay so this is my entire followers okay now inside this i'll just try to put this one more information which is called as uh this one right and i'm just adding that particular information guys you can definitely do this it's pretty much simple and this is very very strong all together so i'm just going to save it load it and now let's see now if i write result of followers here you'll be able to see 9 you run 1.4 k followers is definitely there right isn't it amazing right i'm able to get that particular information now this shows how strong this scraping tool is okay and one more thing that i'm probably going to show in my next video is that how do you scrap the ecommerce website you know by using this auto scrapper okay that will be probably my next video but i hope you like this particular video regarding auto scrapper please do make sure that you subscribe the channel press the bell notification icon and i'll see you on the next video again all the github link will be given in the description of this particular video so i'll see you in the next video have a great day thank you bye