How to Build a Google Scraping AI Agent with n8n (Step By Step Tutorial)

Key Takeaways

so today I'm going to be walking through step byep how to build this Google scraping AI agent in NN in this specific video I'll be using the agent to scrape Google for LinkedIn profile URLs so let's not waste any time by the end of this video you guys all have a great idea of how to get this up and running and how to give your agent access to different tools like the ability to scrape Google so let's hop into a quick demo real quick so we can see how the agent Works before we actually see how to build this thing so here's the agent it's super simple we've only got these three tools here and if this doesn't make sense yet by the end of this video it all will so don't worry about that but what we're going to do is we're going to chat with this agent and we're going to ask it to get CEOs in real estate in Chicago and so we're asking it here and it's going to go through these tools and this is the worksheet that we're actually having the agent populate these profiles into so as you can see we just got 10 of them and if we click here we go up and look at Jeff we will see that he is indeed a CEO of real estate in Chicago so we know that this tool is working correctly and that is basically how this thing works and you you'll be able to ask for you know describe a type of person that you're looking for and it will go search Google and scrape Google to get those profiles into a sheet like this now that we've seen a demo of how we chat with the agent and how it actually works let's move into what we need in order to build this thing so first nadn that is the website that we're building the workflows on for the tool and for the agent then we've got open AI API if you don't have an open AI account you'll need to make one but super quick and you'll just need to be able to go in and get your API key there and then finally Google Sheets um everyone sort of knows what Google Sheets is but that's like I said the database where we're storing the URLs into so finally we're building two workflows today the first one is going to be the tool that scrapes Google for the URLs and then the second one is the actual agent that we can interact with so that's all for the prep stage let's hop back into NN and let's start a new workflow and start building this thing okay so this is the first workflow that we're going to be building this is going to be the one that grabs the LinkedIn URLs from Google so the first step here is going to be a when called by another workflow trigger this is just saying that this workflow will execute when another workflow calls it which in this case the AI agent is going to be thinking about how do I actually scrape Google and it's going to call this workflow so we need to fill out this node with with our data as you can see this is just sort of the basics but we want to fill it in with actual parameters so when we are talking to the agent we'll give it these parameters to search Google for which are going to be job title company industry and location and we're going to fill this in with the information that we used in the demo just because we can actually test the workflow to make sure that it's it's scraping correctly so I'm going to fill in this information here so that we can test this workflow as we're building it okay so here's our information the next step is to connect an open AI node so we're going to go grab an open AI and message a model so at this point if you haven't set up an open AI account you'll need to do so and once you have that set up you're going to create a new credential and as you can see we need an API key so if you go to open AI on the left hand side go to API Keys you can create a new key up here name it whatever you want and then it will give you this value to copy into nadn which is the API key when you hit save it'll turn green if you're good to go and we're good to go so we're going to leave it as text leave it as message a model and then we need to choose a GPT model to use I'm going to use 40 it just seems to be the most consistent but then in here we need to give this not a system prompt so we are going to say I'm just copying this in so you guys don't have to watch me type but we're going to say can you parse the Json query which is this information over here and then output it as the following parameters separately so we want job title company industry and location output over here and we're going to change this to a system prompt because we're telling the node you know we're giving instructions on how to act and then we're going to add another message because it needs the actual data so all we're going to do is drag in query and this will be you know this will be variable based on what we say to the agent so as you can see in this case we've got job title CEO real estate and Chicago so then finally we just want to Output the content as Json and then we can test this step so we can see that the information is coming through over here CEO real estate and Chicago and one thing I just thought of is we want to make sure that this information is always coming through as a string so in front of this dollar sign if we type in json.stringify and then open parenthesis and we close it it'll go green so we know it works and then we'll just test that step one more time and that just ensures that all the information coming out on the right side of this node is going to be a string now the next thing we need to do is actually set up the node that's going to call Google so we're going to go in here and we're going to grab an HTTP request node so these nodes at least when I first started playing with them they just seem super um intimidating because there's so much to fill out and there's it looks like you're going to need to know how to code but we'll make it as simple as possible here so the first thing is the method of the request basically get just means that we are wanting to have information returned to us we're not updating information we're not putting information anywhere we're just getting information back so we're getting the UR URL we're going to use is just a simple Google search so copying this in here Google search so this is as simple as in here if you typed in this exact thing that we're copying it would bring you to the Google search page so that is what this is doing authentication we can leave this as none for now but we do need to set up parameters and headers so first thing is a parameter um this is just saying once we're in Google search what are we actually looking for so the first thing is going to be um we're calling this one q and the first part of this parameter is we only want to see LinkedIn sites so sitecon linkedin.com slin and let me just show you guys if we were to put this in this Google Search we are seeing only LinkedIn results so that is what is happening right here and now we need to fill in the parameter with the variables that we are telling the agent so we've got LinkedIn and then we are going to say we're going to be also searching for you know CEOs or the job title then we're going to also drag in we want their industry and then we also want the location so now we have a variable sort of search criteria and let's see if we paste this in oops if we paste this exact thing into Google search we will be getting there's Jeff that's the first um result that showed up for us but now we're getting CEOs in real estate industry in Chicago and only LinkedIn results so that's what's happening down here and finally which I think is always the most confusing part is we need to set up the headers so let me just paste in this information here and then I can explain sort of how I got it this information I did not know myself right like this looks crazy Who would know to type this in pretty much all I did was um I came into chat gbt here and on the left hand side you've got explore gbt so if if you go in here you can see different ones that people have made and I just typed in NN and it brought me to um basically an n8n assistant GPT which I don't know this is loading forever but you can search it right here Ann assistant so I clicked on this one and I just started chatting with it and I basically asked like can you help me set up header for HTTP request [Music] node um I'm searching Google for LinkedIn URLs okay so as you can see it's really helpful it's going to show you how to set up this node it's going to tell you about the git method it's going to tell you about how to set up that URL but right here is what we're looking for which is the header so the header is named user agent and then here's that Mozilla all the stuff that I put in there and it says that some websites like Google May block requests if no user agent is specified so you need to add one so that's exactly how I got this information down here simple okay so we're going to test the step to see if we're actually getting information back from Google and you know this might look like it didn't work but it did work so this is just a ton of HTML huge chunk of nonsense that no one you know can really interpret but if we come in here and we search for LinkedIn um you can see that it's like pulling back data on LinkedIn and if we click down far enough we will start to see our actual search results so here's linkedin.com slpa we go up a little bit we should see Jeff so here is Jeff's um LinkedIn profile so we're getting information back from Google now we just need to actually make it digestible so the next node is going to be a code node and this is going to parse the information that we're getting this nasty chunk of HTML and it's going to parse it to only return what we want which are the LinkedIn profile URLs so once again I don't really know how to code enough to get that out of here especially in Java so what we're going to do is we are going to grab some of this let's actually go down to the Jeff example which is right here okay so I'm just going to grab a chunk of this right here and say that we're going to go back into the chat assistant and we're going to say can you write code to parse out the LinkedIn information we're looking for here is the here is an example chunk of HTML being returned okay so let's see what we get after that prompt okay so after a couple tries I think we've gotten to a good spot I gave it a screenshot it updated the code a little bit it wasn't working it gave me an error so I told it what the error was and then we landed on some more code and we did it again but now we have this snippet and I think that this one should work so let's test it out as you can see we're getting 10 LinkedIn profiles back from this super ugly chunk of HTML over here so we know that this one is working now we need to get these URLs into our Google sheet so we're going to do that with a Google Sheets node you'll type in Google you can see sheets and we want to append row within a sheet so this is assuming that we already have a sheet setup like as you can see mine's right here it's called Pro Prospects or proex um LinkedIn so this is the one that we're populating and so you should already have that set up and then if you don't have um your credentials already set up it's super simple you'll go here create new credential and you just have to sign in with this node and this is kind of assuming that you've already set up your Google cloud service so if you haven't anden has great docs on how to do this it'll walk you through step by step how to set up something like your ooth consent screen and your different enabled apis and services but it's super simple you'll come in here and basically just set up this screen that is giving access to naden to access your information so you can do that by adding your email as a test user or publishing the app but um if that doesn't make enough sense then please go to the nadn documents that walk you through how to do that step by step so once we have that set up you'll be able to access your sheets so if we come in here or your documents you can see all my other documents and then we've got um the prospects one and then from sheet we want to choose just the first one because that's all we have so now we see that we want to match each column manually so the only column that we have is this one Prospect LinkedIn URL and what information are we wanting to put in that column that is going to be over here from the code node we're grabbing the LinkedIn URLs that we just scraped from that HTML so it's going to go through this step until all 10 of those items are put into our Google sheet so we can go here and actually hit test step and if we pop over to our Google sheet we should see those 10 filling in right here as they did so there we go and it output those items correctly and this is basically all we want the tool to do so these are basically the five nodes that we're running through and then the last thing we need to do is we need to add a set node which we're going to name this field response with the value of done so all this is doing just to keep it as simple as possible is we talked about how this workflow is being called by the agent and then it's going to go through this process and then once it gets here it's done but the agent doesn't know that it's done and it doesn't know to tell us okay we did that for you so this this field is called response and it has the value of done and so this is going to come back and give this information right here to the agent and then it's going to know okay I can respond to the human because we've done what it wanted us to do so last thing before I save this is I'm going to take out these hard coded variables so that we can actually test this thing dynamically so so we've got that saved there now it's just a query and we'll save this okay now that we've built the tool we're in the workflow that is going to be the actual agent that will be calling the tool and the good news is the agent workflow is much simpler than the tool and a lot quicker to set up so first thing again is going to be a trigger we're triggering this one by a chat message so pretty self-explanatory this is just when we talk to the agent with this chat button down here so next we want to add the actual agent so we type in AI agent here we will get this guy it's going to be a tools agent because it's going to be accessing the tools that we give it and we can just take the prompt from the previous node automatically this is also where you could add a system message down here to really give a a detailed prompt to the agent if you have a lot of tools or if you have you know very specific use case but right here we can just leave this as a helpful assistant okay so once we have the agent we need to give him the brain which is a chat model which we always use open AI chat model so you'll have this set up already and I'm going to give this guy 40 okay so the brain set up let's set up the memory real quick super simple all we got to do is click on window buffer memory up here it's it's the easiest one and we can leave the context Windows 5 for this specific case but this memory is just giving the agent context for what we're talking about so it's not just like a question and answer and then his brain is resetting and he's he woke up again and he has another another question answer but if you ask him a question and he responds and then you want to refer back to a previous question or refer back back to what you guys are talking about then he'll be able to do so based on this memory and then we got to add the tool so there's a lot of different tools here here's a Google search API which we can get into in future videos but right now we're just going to be using a workflow that we already built which is super cool about nnn so this is going to be the one that grabs profiles so we will just say grab profiles and up here do the same thing and then you need to give this tool a description of when to be called so we just going to say call this tool to get um yeah LinkedIn profiles okay keep that simple where is the workflow coming from it's coming from our database and it's really nice that NN gives you the ability to just choose from a list so you can see all the different tools I have and we are going to be using grab profiles demo and then this is the response field that we talked about earlier with that that set field that we set response to done basically telling the agent you know once this tool is done it's going to return the response and then the agent is going to know that we did it okay so that really should be it um there's a chance I may be forgetting something and it might air but why are we already oh yeah this can't have a space in it apparently which is super crucial um so we'll save that and let's just test this thing out to make sure it's working so let's reset those and we will come in here and we'll just try the CEO one for now and then we'll try another one but can you get CEOs in real estate in Chicago okay so I've obtained LinkedIn profiles for CEOs in the real estate sector in Chicago if you need anything else let me know so as you can see we got that back we got Jeff we got all these guys let's um just make sure that we are actually you know it's correct and that we're not just using you know hard-coded variables so let's do um Founders in um Tech in San Francisco maybe I'll say technology though technology you Founders and technology in San Francisco I've obtained LinkedIn profiles okay let's see we got these new people in here so let's go to this guy see what him he his profile is looking like Tech comedy with hint of Truth San Francisco and um he seems to be a Founder let's just check one more guy we'll go here founder and CEO at motive San Francisco okay so looks like that's working and you can always go to your executions over here and we can check sort of how the data is actually moving through based on each request that we've done so this is the most recent request and naturally it's going to take a couple seconds to load up but it's coming through as Founders and and Technology coming through here the title is Founders Industries technology location San Francisco and then again you know grabbing this huge ugly chunk of HTML we're getting the profiles through here and then we're putting it into the Google sheet so that is kind of how this one works um now if you guys are wondering sort of why we're only returning 10 responses and if you wanted to do more how could that work so my understanding is that we would need to be using a an actual Google search API like Sur API in order to do that and I'll show you guys exactly why I think that is but let's go back into this workflow to grab um grab profiles so here's our request and like I said we had parameters of what to do so if we came into here and we added another parameter and we called this one start and we gave it the value of 10 so now we're seeing for example we come into this LinkedIn search um at the end of this query if we add start equals 10 it's going to give us page two of these um results so this is page two and this is what we'd want to be scraping but if you come into here and we see we're getting items back here we're getting like the HTML back but then when we go to actually get the profile we're only getting two um so sometimes it's I think that there's just an issue with scraping Google based on HTTP request that when you're doing this and you're going through that Google's may be providing a capka to to find out if it's a human or if it's really a robot scraping information but if anyone else has more insight into that please let me know because I I was looking into it and that's sort of like what my conclusion was but the purpose of this video was more to you know how do I set up a workflow to scrape information and two parts information especially if I don't know how to code and that was with you know the NAD and assistant on chat which is super super helpful and also just showing you how to you know update rows and sheets and then combine an agent with a tool so um yeah so that's it for this one please leave a like if you've enjoyed and definitely let me know in the comments what else you guys want to see so I can continue to add to my list of videos that I want to make and stuff to learn about so that I can sort of help all you guys out so thanks guys