Step by Step: Scrape UNLIMITED Emails for FREE with n8n

Key Takeaways

all right what we have here is a trigger workflow that's going to take all 97 of these search queries which are looking for sub regions in California and looking for dentists and it's going to run them through this workflow which is going in and scraping them in Google getting their emails and then putting it into a Google sheet for us so let's test out this workflow and we can see it take place I will also pull up the Google sheet that it is populating this information into and we can see this happen in real time and this is all going to happen for free and it's going to get you hundreds and even thousands of emails depending on how many queries you put in right here but it's going to be free and you're not going to have to use any you know external apis or use credits as you can see that first workflow just finished up that first query so we've got 12 in here we can also go to our executions and if this will refresh in a second we'll be able to see that it's it's triggering multiple as you can see the trigger went off and then it's going to scrape emls this one just succeeded and now it's doing the other one in the trigger you can see that we wait for I have it set for 6 seconds to wait in between each execution but this is just going to keep going and it's going to keep scraping emails so I'm going to let this run I'm going to let this finish up and I will just sort of speed this up so you guys can see that it's actually happening um in this workflow and it's not me pacing in any information so let's let this finish [Music] up okay as you can see the first 10 have finished up we've got 159 total items have come back and in this spreadsheet you can see that we have 159 emails in here and we're going to continue to let this run but before I do I just wanted to say this build was 100% inspired by acam I'll tag his LinkedIn right up here and a a link to his LinkedIn down in the description so I found his build on LinkedIn and I and I wanted to play around with it so thank you for for the build and let's get back to the video all right just checking back in you can see 58 items out of the 97 have finished and we are at 814 total items here 814 total emails as you can see in this Excel sheet and so this will get us clear over 1,000 if we let this finish up so let's just let it keep going real quick all right so went ahead and stopped it because we almost got to 1,000 easily would have cleared 1,000 emails as you can see we got 909 so let's um let's quickly look into the executions here and we can see um just every time that it ran which would have been about 60 times if we remember correctly so looking at all these executions we're able to see exactly what happened and what was going through as you can see each time it was an individual different query so each one was looking for um a different sort of sub region and scraping through so if you you want to understand how to you know set up this workflow and get it going then stay tuned cuz I'm about to walk through step by step how to do this if you're looking to download this NN workflow that I'm about to show in this video you can get the template in the free school community so the link for that will be down in the description you'll join this click on YouTube resources and there'll be a post for each video I make you'll just click into the post for this video and you'll see the um workflow template to download as well as there's a few code Snippets later in this video and I'll place the code Snippets in here too so I'll see you guys in there all right now to actually build this workflow flow we need to understand a few things so first of all we're scraping these business emails from Google Maps so in this example right here we're scraping for the query of Bay Area dentist and that search is going within Google Maps so right here you can see in the HTTP request it's going to Google Map or sorry google.com/ Maps search and then we're using the Json query which is Bay Area dentist as you can see from over here so real quick let me just make this um let's throw this onto a new sheet sheet two so I can just show you guys how this is going to work hit test workflow it's going to scrape in here from Bay Area dentist we're just doing some logic here to filter the data or sorry organize the data how we want it um optimize the data how we want it and then it's going to run through here and this is where it's actually going to scrape out sort of those emails from each of those pages on Google Maps the businesses on Google Maps and then once again getting the data how we want it and then it gets put into the sheet in this case we got 10 emails so if we go to sheet two we can see those 10 emails right here so let's just get rid of these for testing purposes and now we'll get into actually how do we build this so the first thing to understand is we're we're searching Google Maps obviously so when you go to Google Maps this is sort of the URL that you get um and this is what it looks like but then when you want to Define down and search for example in the United States where you want to look for you search something like California real estate and it's going to pull up a ton of different ones sort of the the highest you know SEO businesses that are here and it's not going to give you as many so in this case our request would have have been like California plus real plus estate because in an HTTP request you can't have blank space you need to have the pluses to separate um where space would be so that is like the search that would be taking place but what we're going to do is Define down within California different um like sub regions or like sub neighborhoods so that's why here we went with Bay Area because now we're getting more hits in this area and then we're just going to iterate over um multiple different sub regions so that's why in the demo you saw there were 97 queries that we were feeding in which these were all just different sort of you know sub regions right here so once you understand that now we will hop into Ann here is a new workflow I created for the purpose of building out this workflow and then we'll also build out that trigger workflow as well but we're going to start here so we're going to obviously this one this workflow is going to be called by another workflow so that's how we're going to start this one off and for testing purposes once again we want to use Bay Area as an example so you're going to come in here you're going to type in query in quotes colon and then Bay plus area plus um real plus estate and so you could do this obviously with different sub regions or different um Industries it could be um in aram's video he did the example of dentists in New York so he did you know different regions in New York but we're doing California real estate so now we have this data to test as it goes through the next thing to do here is that we want to actually make our http request so within here we're going to keep it as a get because we're asking for information back from a specific URL which once again we will go back here and we can actually just grab this part of the URL https ww.google.com Maps search so that's where we're going to go um I'll put this in here and so just to show you guys if we copy and pasted this into Google we would get you know Google Maps so that's how that's going to work and then we just want to specify what we're searching for within this search so we're change this to an expression that way we can drag in this query right here and obviously this is going to be dynamic based on what we put in the um the final test when we want to use that trigger workflow to send in boom bay area boom different query boom different query boom different query and that's how it was looping over this workflow and so that's really all we need to do we'll hit test step to make sure it's working as you can see it got our ugly chunk of HTML data back so that is why we we need to add a code node for this next one in order to parse out the information we want so I'm going to paste in a snippa of code here all right now I put in this code here once again this will be in my fre School Community just look for YouTube resources look for this video and it will be in there so now we can hit test and we will see what we get back out of this huge massive chunk of HTML now we're getting ton of these sort of um business domains and as you can see a lot of these are duplicates these are kind of like the not the information we want at all so that is why our next note is is going to be to sort of filter out this information so here we're grabbing a filter node and within filter all we want to do is we're going to be filtering this value which is all the URLs coming back and then we want to make sure that it does not match a particular reg X so regular expression we're going to throw in this um query I guess you could call it or this information and this is just saying that everything within these brackets or sorry within the parentheses is what is like in this list of things that we don't want the URL to match and then everything is being separated here by a um I forget the what they call them but the vertical bar and so if we test the step we will see that it's filtering out a ton of items it only kept 32 which are going to be actual um business domains that we want to email although you can see there's some duplicates so we're going to remove those in a sec but as of now in the discarded pile you can see all of these emails that we don't want so um Google Google Google user content gstatus um ggh htb schema all this kind of stuff and just make sure you have this put in right if you put like spaces it's not going to work so we'll test this again and you can see now we we kept some of those items that we didn't want because like the schema didn't work because we had a space in these GG phts so make sure you don't have spaces in between um the vertical bars and then if this isn't working for you for some reason what you could do and it's just a little more tedious is you could do does not contain contain and that way you could type in in each individual one and then you could go again and say okay it also can't contain um you know this information but this is probably just the easiest quickest way to do it if you really wanted to you could throw in like the the Dooms although if you did it like this I think you'd have to add this um you'd have to add this this backslash in order to s in order to um signify to regular expression that you know that's like how it would work but you can pretty much just get away with doing it like this so hopefully I didn't spend too much time there but let's test this step again and we will move on to the next part of this which is wanting to get rid of these duplicate values so we'll add remove duplicates and you want to do remove items repeated within current input not these other two actions so this is just going to take anything from URL and you could just leave it as all but I'm just going to drag in url so anything in this field it's going to get rid of that's a duplicate so if you remember we had 32 items right here and now we have 20 items coming back because it removes those duplicates okay now we want to do another HTTP request because right now we have these 20 um different sort of domains coming back and we need to search through them in order to go get some more emails so we're going to do an HTP request and we're going to be doing it from this URL so it's super easy we don't have to type anything out all we're doing is grabbing this URL but there's 20 coming back and we want to Loop so we want to add a loop and then we're going to have to get rid of this thing and we we don't want this to be the done field we want to um hook this up here so it would have been smarter to just add a loop first and then HTTP request so do that um unless you're following this exactly step by step then you probably did that what I just did but anyways the important thing here to do is as we know it's going to Loop through all 20 of these items and what we can do within this loopover items node is make sure that it doesn't stop if there's an error in one of those URLs because currently all those 20 right here are going to parse through and when it's being searched through the HTP request if any of them error the whole workflow will stop and so in the case of you're running like 97 queries like in the example the whole workflow would fail or it would stop when it hit that failure so we want to make sure that this is continuing and we want to continue using an error output so as you can see anything that errors will just go down this branch and nothing will happen we'll basically just skip it and then we'll set up the logic for the ones that are successful what do they do so as of now we are grabbing these URLs I think that's all we need to do we'll hit save and we'll test the step to make sure that everything's happening properly so it just went through here now it's going to be searching all um of those items okay so now we got an error and it did stop so that's not what we want we want it to be able to keep going because it only did five so the resource you were requesting could not be found maybe this is the one as well we need to make sure that this continues um so maybe we can try this test that again see it coming through and it was number five that failed okay there we go so now it would have just passed over the one that was erroring and now we're able to keep going so that's exactly what we wanted and this one should finish up and then we'll continue with the logic of these 17 runs that did work as you can see two of them actually aired so that's fine because now the workflow is going to keep going because we have a done branch and now we can set up this done Branch workflow or logic sorry okay we're going to drag this out and we are going to to do another loop and another essential thing is we want to make sure this do the same thing continue if it's an err and it's going to use an error output so now we have an error Branch here and so we're going to get rid of this replace me and we're going to add a code node and let me paste in some code real quick all right here's our code this one's basically just parsing out the actual emails into the format that we want you'll see that in a sec but another thing we need to do the same thing here is we if if one of these errors which they likely will every once in a while out of like you know th URLs come back at least probably two or three are going to err so we want to make sure it doesn't stop the whole workflow we just want to make sure that it continues on throughout the logic so I don't think we can just test the step directly because it's a loop we have to run the whole thing again so we will just test the whole workflow and then we can see everything going on um once this finishes up it'll come down to this Loop and it will Loop through this code node and then I'll show you guys what I was talking about with the whole parsing emails and once again both these code Snippets will be in the school so join the link in the description but let's let this one finish up and then it will come down here and this Loop will be really really quick this is always the longest step is you'll be waiting for these to finish up as you can see that was super quick so sometimes when you're running this at the end some runs might take a minute or two some will be 20 30 seconds and it's going to be waiting on this step right here so let's look at this code node see how this information is coming through so it's getting this huge chunk again the chunk that comes through pretty much whenever you do HTTP request and so usually need a code note after to parse the information so the 16th run nothing came back here we got all these emails so as you can see it took this information and it got just the emails we want number two nothing number three that that one AED um for blah blah blah so let's just see so a lot of times this will happen like this but anyways you will get some emails so now we need to set up the logic of once we get those emails from this done Branch right here which are all the emails what are we actually going to do with these emails okay so we're going to drag out the done and we're going to grab an aggregate node and this is going to be taking the emails that come back from the loop after it's been put through the code and parsed through and then we want to actually just aggregate them together but we want to make sure that we turn on merge list and I don't think we can test we're going to have to test the whole workflow y we're going to have to test the whole workflow um okay so we'll test the whole workflow and see how this information is coming back we want to make sure it's in a list so that we can um successfully split them back out and it's hard to it's hard to visually understand when I'm just explaining it so we'll actually be able to pin this data and it will be nice because then I can show you but yeah just keep in mind when you're kind of testing these Loops you it's it's tough to pin the data at least unless I'm doing it wrong but now we have all these emails that are coming through in the from the loop and then we have them in one nice um sort of field right here so that's why we wanted to do it as merge list otherwise it would still be sort of individual objects and that's not how want it so let me pin this data and now that it's pinned here this is going to stay so that's good for testing and we can add our um split out so it's going to turn a list and it's going to put them into separate items and obviously we're just going to drag the list of emails don't drag one individual one and then we should just be able to test it and it should be how we want it perfect so we have these emails now because we can put them into an individual row within our sheets and you might have noticed now that we have a couple duplicates or more than a couple we've got some duplicates so pretty much the final step here is another remove duplicates we're going to Cho the first one remove items repeated within current input and then we want to select the field which will be from the split out node we're grabbing emails so we'll test this we have 23 actual emails coming out out of 246 so in this case tons of duplicates so it's a good thing that we did that otherwise um we'd be sending emails to the same same inbox over and over and over and then we just need to add them to our Google Sheets node so we remember you want to append a row in a sheet that's how you actually get them in there we have signed in with with our account grab the right document grab the right sheet and then it's going to map them we have a column in there called emails when you're setting up your sheet to put stuff into if you don't put a sort of like a column header it's not going to work so make sure you just throw something in there that way you have a column to match on and all we're going to do is send emails to the column called emails so this is what's coming through we have our first example of an email and if we hit test step as you can see it's going to send those items into our Google sheet you can CL click into here and there we go 23 items just came through sorry here's the one we're working on so this is the one we just did and 23 items came through and so that's pretty much how this one works um now we need to set up the logic of building that trigger workflow I'll talk about why we want to do it like that in order to parse a specific query through so you know this one's going to Bay Area real estate and then that one will run and then the trigger will send a different query to the next workflow and so on so let's get into that workflow okay so this one is being triggered manually and I'll talk about some cool use cases as once you see how this works I'll talk about some ways you can make this even more automated but the first thing that we want to do is we want to grab a loop because we we're going to obviously for if we put in 100 queries to search through in California for Real Estate we put them in here and then we want it to just send off bam first one bam second one bam third one so let's remove this and we will grab um workflow oh yeah it was right there workflow um execute workflow so this is the same way that a an AI agent can call another workflow you can also have a workflow call a workflow it doesn't have to specifically be an agent calling a workflow as a tool so we're searching from our database we're going to grab the scrape demo that we just built and that pretty much all we need to do here we might have to change the way that it uh we should probably change on error we don't if if one of the workflows errors we wanted to make sure it's again continuing so make sure you set up that otherwise it won't run through all 100 items however however many you put in there and then finally we just want to wait and so I did six I think you could five you could do probably even like three we'll try 3 um but the whole point is that you you just don't want it to you don't want the emails to overwrite each other in the um you know in the sheet otherwise you'd just be putting information on top of each other if it was just sending out looping super fast so that's why we do the weight here and now we're pretty much ready we want to talk about how you actually send off different queries into the workflow so in here I came in chat gbt and I said give me a list of search queries for the keyword real estate for 100 sub areas micro neighborhoods of California and then you had to you know just show it how it's going to be structured so this is the one that I did in the demo so we'll do a different one um we will ask for for the keyword let's try something like um Finance let's just do 20 and let's do Chicago and then an example we will just say South plus Loop Plus Finance all right so we'll send that off and this should give us one to do and so my my thought was we're triggering this manually right so once this finishes up I'll show you exactly what I'm talking about but so we've got our 20 we're going to come in here and put in this mock data so should just be able to grab this paste that in there perfect no errors and now we have these 20 search queries that it's going to send to Google Maps and scrape for emails so let's just test this and see if I made any mistakes and if if I did we'll we'll live figure them out together but um another way you could do this that could be pretty cool is obviously I use chat gbt to generate this stuff you could map this up where you know you have logic over here where you can send the query or you can chat to some sort of agent right in here and just have it do it in here so instead of going to okay so it only went through two items um let's check our sheet okay so we got a couple emails let's see what happened so we'll go to our executions of the um thingy the scraper so in here okay wait what was the error see the first one came through how many did this give us this gave us 10 emails which is I think is how many we have right here okay we don't even have 10 um so what happened here maybe Lincoln Park Finance that's the one that went through okay let me just look into this real quick and I'll be back with what happened okay I figured out what happened the executions just didn't refresh so we weren't looking at real information but this is the first run it gave us our seven email addresses which as you can see we had these seven and then the second run when it tried to go after it scraped through what was this Lincoln Park Finance it only had two items and then from those two items we only actually got one set and then it just stopped because there was no more information so maybe we need to make this where in here on air it would also continue and on here it would also also continue so you want to make sure everything's still continuing that way it will um will always go through if that makes sense so let's also just unpin some of this data actually that that shouldn't make a difference but we want to make sure every step is continuing that way if for instance one scrape like Bay real estate which in this case that one's fine but the one that we just shot I think it was Lincoln Park Finance nothing actually passed through here but we want the workflow to continue because that's the only reason to stopped is not because it errored but because it um just didn't have any data coming through so we'll try this again we'll see if that change worked um and then if not we'll get in there and just look again what happened and then if we still can't figure it out we'll just search up or switch up the search term and go with something that works maybe maybe Finance isn't the best way the best thing to look for on Google Maps but there we go we got our first seven ons again now we're in the second one and it stopped again so we will look into that so exact same thing as what happened the first time it gave us our seven but then when we come to the second execution of um Lincoln Park Finance once again right here it ends so maybe always output data okay that's what we need to do so much smarter than continuing okay so always outut data so even if nothing comes through we want it to still output so that it at least signifies to the trigger where workflow that we finished that way um it'll continue going for the rest of the queries because we've only gotten we'd only gotten these two done we still have 18 more to go so um hopefully you guys like this sort of live thing where errors happen and we try to look into what happened and how to fix them but um once again this is going to scrape the first one give us seven emails hopefully the second one it will give us nothing but it will still continue to pass through the process so now r on number two really hope this one continues to pass through please please please we got our seven emails from that first workflow that first query you would think that if no data is coming through it would be quicker okay perfect so that seemed to be the issue I'm glad we got that figured out live because I actually didn't um I hadn't run into a situation where that happened so good takeaway here making sure that if you're looping stuff through where you want the workflow to trigger multiple times that you have um the always output data turned on and maybe that's like a super simple thing but it passed my mind in this case and now we're getting through the rest of our actual um queries that we put in here which we put in 20 queries so let's let this one finish up and I'll be back Okay I lied I'm back already so we're on run six 43 items so far I wanted to show you guys more about like if you clicked into down here all executions then we could obviously see the trigger is still running because we triggered it so it's been running for 2 minutes and then these have been all the iterations of the past queries that it's been feeding through so this one's running for it's been running for 30 seconds now as you can see they're kind of going to vary so um what was the first one of this run we trigger this at 1309 so 1309 right here the first one was 13 seconds then 20 seconds 20 seconds 9 seconds 30 seconds um I've had a few that will take multiple minutes some are as quick as you know like this one 9 seconds this one's going for um this one will probably push a minute but it's cool because you can look in here and see um how they're sort of staggered and where's the oh I I closed out of the trigger workflow so we can't see that live anymore but obviously this one is running so we'll just come into here real quick which is just one of the scraping examples and we can look at the the executions so this one is is we can see that one's going right now this one was for Gold Coast Finance for this one we got 13 emails back as you can see and they were put into our Google sheet which is um right here with our items so yeah I hope you guys found this one interesting um that one just finished up it took a minute and 26 and we only got like maybe three or four emails from that but I hope you guys found this one interesting once again huge shout out to um acram uh his profile is in the description but I thought this was a super cool use case of you know having a workflow and I thought this this idea was super neat the way that you iterate through items that you put here and once again I think it could be really cool to you know you have some sort of open AI messageer model over here because that's essentially what we did in chat BTA to get this stuff in the first place you would have this this over here and you would want it to you know have an interface where you could chat to it and and then you could just ask it right in here to find um you know say I want to find places in or I want to find dentists in um Texas and then this model would figure out okay well based on how we prompted it would figure out to find sub regions in Texas and then format them into this sort of query format with the sub regions plus whatever the keyword was which in this case I just said I think I said dentis in Texas so that would be really cool and it could be an expansion on that um so I might play around with that and see what I can do there but that would even automate the process even more which could be super cool and yeah any questions um please reach out leave me a comment or hop in the community and I'd love to talk so thanks for watching this one guys I'll see you in the next one