Learn Javascript 101 for SEO's

Key Takeaways

hello and welcome to the search engine journal e-summit i'm super excited and glad to be the one who can open up the day so to speak and uh to be welcoming you to the event today and um the title sounds a little more technical than you might have expected for an opening keynote but don't worry i'll gently introduce the topic so my name is martin split i am a developer applicant in the search relations team at google and i'm here to talk a little bit about javascript for seos more or less the basics so today we'll talk about a few things very first topic that we will address is why is this entire thing so hard why is there such a chasm between developers and seos and what can we do about this and why do we even have to deal with javascript and why does it even matter and then once we figured out why we should look into these things and why they matter we look into how websites actually work i mean the high level concept is probably familiar to everyone you open your browser you type in a web address and then you see your website but what does actually happen behind this and what can go wrong when javascript is involved and last but not least we talk a little bit about how to fix the friction between seos and developers and what you can do to get better results and to actually have actionable advice to developers that will make a difference in the end and will also be falling onto open ears or meeting open ears with your developers right so why is this so hard what's what's going on here why is javascript everywhere and why is everyone like freaking out about it um i think there is a bit of a disconnect between seos and developers oftentimes and many teams it seems to be like oh the developers are not listening to what i'm telling them or the developers are building things that that are broken in terms of seo and how does this happen and then there's pushback from developers so like the seos come in with ridiculous things or require us to break otherwise good solutions or require us to make changes that take so long and are so complicated to make now uh that we just can't prioritize it high enough and i would like to take a step back and think about what we're trying to achieve seos are trying to build websites that can be found by potential customers and users and show up when these customers and users need what you are offering the services or the product you're offering and developers want to build a product that works great is fast and actually makes the users happy and actually implements and does the things that it needs to do and does so correctly without failing and without falling over that's the goal and i think these goals are compatible i think developers build for the users and if there are no users because no one can find it on the web they won't be happy about this they want users to use the application they build so seo is important to them even if they might not fully understand or see it that way and on the other hand seos want the product to be good what's what's the point of ranking high when then in the end no one converts and no money is being made because the product just doesn't work and so we are building towards a shared goal we're just looking it from different looking at it from different perspectives and we need to to figure out how we can converge our journeys and actually make this trip on the same road so to speak instead of arriving at the goal from different perspectives and like clashing in the middle of the road so why would you care about javascript as an seo why would you care about you know the things that developers deal with and javascript to begin with and how does it matter for seo to begin with if you're using javascript and how you're using javascript why is this important well first things first javascript has historically been the duct tape of the web so to speak right whenever we wanted something that the web couldn't do we could emulate it we could simulate it with javascript we could build some javascript that does what we need to do classical examples are lazy loading before browsers implemented that we could use javascript to make lazy loading happen if we wanted offline access then we could sometimes trick around a little bit if we wanted uh web components that allowed us to build more interesting user interface elements that didn't exist in browser we could use javascript for that and so on and so forth there has been many different things that weren't possible without javascript many of them are now possible without it but a bunch of stuff is still only possible with javascript it also brings powerful features to web experiences so for instance service workers are basically allowing websites to work offline or independent of the network that's done by using javascript a service worker as a piece of javascript that sits between the website and the network and makes decisions so it behaves like apps because you don't have to be online anymore the website can work without network or with slow network um but that is powered by javascript without javascript that wouldn't exist progressive web apps wouldn't be progressive web apps without javascript it also allows you to respond to things happening and the obvious things that come to mind are things like clicks but also things like what if a network resource isn't available what if the user changes between dark mode and bright mode on their device and stuff like that you can respond to this using javascript and then change the page change the layout change the color schema if you click on a button you can do something like load an additional piece of content like an additional image or an additional product you can do a bunch of stuff that javascript enables us to do especially when it comes to interactive websites where it's not just showing some information but users want to interact with it like a web shop for instance you can do a lot of cool stuff with javascript that you can't really do without it it does matter for you as an seo though because even if you're not building a super interactive website if there is javascript involved it has full access to the page content so javascript can make decisions on what content to show or content not to show what content to load in what content to hide there's a lot of possibilities in terms of what the content will look like that come from javascript javascript could for instance detect oh this is a google bot so i'll just like show or hide a piece of information which can be cloaking which can be problematic right it can inject things it can inject a different canonical it can inject a different title it can inject a different meta description it has a lot of power over the website which is why you definitely want to understand what the javascript on the pages you're working with does it also is a very core part of the web if we talk about web development you definitely talk about html you should definitely talk about html at least you will talk about css because that's how everything looks like and javascript is the third part of the core web technologies which has a lot of influence over the way of interacting with the page as well as how the page looks like and how it works so it's quite a crucial critical part of the web and as with every tool javascript is just another tool it can hurt your website it can also benefit your website a service worker that makes your website work offline is great for users who are on bad connectivity a website that loads so much javascript that it takes minutes of downloading javascript over the network until you see anything is not great so javascript isn't inherently good or bad it is a tool and the way that it's being used by you and your developers determines how good or bad your page will come out on the end of the user so it is quite important that you understand what the javascript on your page does also developers are building according to requirements and these requirements are usually like how fast is the page load on which browsers does it work how does it work like if i click on this that thing happens um but seo is not necessarily a requirement for them or they're not aware of this it's a speciality i mean you are an expert in seo you know a lot of stuff that developers probably don't know and developers are experts on their field they know a lot of stuff that you don't know so we are both incomplete and we need each other to complete each other's skill set so if you know that developers need to be aware of certain things like oh there's a no index meta robot that needs to be removed from that page or hey there's a software for that we can tackle like this that's stuff that developers might not even have on their list they might not be even aware of any of that stuff so definitely talk to your developers and it's important that you do because they might not know what to do in terms of seo when they build websites using javascript so how do websites then actually work well it starts off with a recipe right so this is basically an ingredient list the ingredient list is for a page for an html page it has a title of some sort it has a headline it has an image it has a few paragraphs of text maybe but that's not what we actually see this is the recipe when you are baking something or when you are buying something from a bakery you want the actual product you don't want the recipe but this is what we are creating we create the recipe that is then being sent to a browser and the browser now actually makes the dish so as the browser is making the dish it looks at the recipe and it basically creates a document so if i would be baking i would probably have some sort of pot or pan or something where i actually bake on and this in this case is the document and the document in the browser is not anything weird or abstract it's basically what we call a tree in development at the top is the document and then there's the head there's the body and then in the body there's in this case there was a headline an image and two paragraphs and so on and so forth inside the paragraph there might be a link and you know it goes down quite deep so as the recipe arrives at the browser it fills in this tree by creating each of the bits and pieces each of these elements so it creates a headline it creates an image it creates a paragraph it creates that link from the html it then puts them together so basically this is more or less the assembly part of things where um it looks like okay so this is a headline so it takes all the space here and then there's below it is an image and next to the image we have some space still left so we can put a paragraph in here and then we have a line break and then we have the next paragraph and so on and so forth so that it all fits on the page it has to do that again if we resize the window because then obviously everything changes so if you are on a phone if you switch between portrait and landscape it has to relay out these things but once it has figured out the layout it can actually start baking it can actually start rendering which is putting the pixels into place so that you can see it on the screen that's more or less how a website is being displayed and this is also what googlebot does when it renders crawling gets the recipe and then rendering bakes the actual cake or renders the actual website so javascript is very powerful and has multiple points of interaction with the way that the browser renders a page when the browser hits a javascript that has some content in the javascript as text it interrupts the browser parsing html and actually runs the javascript first because the javascript could at this point create additional elements it can remove elements that are already on the page it can change them it can change their content but the browser has to wait until the javascript is finished because before it can continue to create more elements and do the rest of the page it also can even later on interact at any point in time with a dom tree so if i click a button and some javascript runs it can create new elements it can remove existing elements it can change elements move them around in the tree it has full flexibility on the page it can also respond to events such as clicks but also such as failed downloads or api calls or if your website goes sorry if the device goes into dark mode or bright mode it has a lot of power and it basically responds to events that can happen either by the user or from coming from the browser or the device that you're on it also interacts with the network a lot so javascript cannot just work with what is already in the document tree or on the website it can make calls to other websites it can make calls to services and grab information and then do something with this information for instance again creating elements or changing elements on the page but what does that look like when you actually have javascript on your page that doesn't exactly do what you want well let's have a look let's let's debug this together so here we have a website that shows a bunch of news so we see that there's a few news items already here and as i scroll it seems to be loading more items so i scroll down i hit the ground of the or the bottom of the the page and then it loads more okay fantastic the problem with this seems to be that on this page while we are scrolling more content appears but this additional content doesn't show up in google search now the question is why is that a good first part of call is the mobile friendly test or the rich results test or the url inspection test depending on if you have access to the url in google search console or not i'll use the mobile friendly test we see the the data shows up so the javascript gets executed that's great but then when we check here we see there is one two three four five six seven eight nine ten there are ten stories on the page and here we definitely have more than 10 stories this i don't even have to count this is more than 10 stories so googlebot is only seeing the first 10 stories we can also have a look at the loading what was loaded here so we see one two three four five six seven eight nine ten stories have been loaded and then here is a call to the top stories jason which loads all the different stories to take in hmm why is it different in this browser where it works and not working in googlebot here so let's debug this one great way of debugging this is right built into your browser you don't have to buy any tools for this or anything here we see everything that is in the documentary in the dom tree that i discussed earlier we can see that there are many many list items if i reload the page this looks different though if i reload the page we see 1 2 3 4 5 6 7 8 9 10 that's it and if we look in the source of the page to start with we actually see none of the new stories we only see an empty list this is an empty template and the list is empty entirely so javascript is being used and we've seen this part this javascript execution initially is definitely part of what is indexed in google so we know that this is what the mobile friendly test sees so the javascript executes fine but what actually happens here on the network if we are loading this page again and look at the network we do see 1 2 3 4 5 6 7 8 9 10 of these stories are being loaded but when are the other stories loaded let's try it so i go to the bottom aha we see this loads additional stories but we still don't know why this gets loaded or where this is coming from so we see that this is going fine we get a positive response we do get a bunch of stories here but there's also this initiator tab the initiator tab tells us where in the code this is actually being triggered and we can see there's some anonymous function this is the function that actually fetches it from the network okay so this is making this is the javascript that makes the network request and this is part of the show page function well that's not really that helpful show page does show up here as well but who calls show page well the first one here is a window dot on over scroll and we see that here that whenever this is an event whenever the scroll bar reaches the bottom of the page it triggers an on over scroll event or an overscore when that happens we increment the current page counter and then we load the page for that page count so when we go down here we can actually also set what's called a breakpoint and then when we hit it the javascript execution stops here and then we're seeing this happening now what we know is this doesn't seem to work in googlebot why does it not work in googlebot because the implementation of this is wrong infinite scroll is not a problem but this one was implemented incorrectly maybe because developers didn't really care about seo and didn't know that googlebot doesn't scroll googlebot doesn't scroll which is why this function will never be called this implementation has other problems as well what if i don't scroll but whether i start with a small window and then extend the window that isn't a scroll event so it would also not load additional stories so the implementation needs fixing but now you know what to tell your developers you can tell them oh i've noticed this page is actually using scroll events but googlebot doesn't scroll and here's the documentation that shows why that's not working and maybe there's even documentation that shows how to implement it better so that's how you can troubleshoot websites using the browser as well as google tools the benefit of doing this debugging together is actually you understand what the page does what the javascript on the page does and you can give more detailed and helpful feedback to your developers because you understand what's happening if you don't understand that's perfectly fine there are many many real world websites that are a lot more complicated than this to debug maybe just talk to your developers maybe ask them can we debug this together because there is a mismatch between what google sees and what users are seeing can we find out where that's coming from and fix it but understanding what tools you have to do a first investigation is very important so that you can give developers the right feeling for where the problem might come from and help them identify the problem quicker that's going to be much appreciated so but the friction usually is larger and more fundamental how can we fix the friction between developers and seos well there's a few things that i think would might help with this it might help to understand how developers more or less work roughly the work flows like this developers become a problem that needs solving like we want to load more stories when the user reaches the end of a page not a problem let's define a few requirements it should be fast it should load 10 more stories and so on and so forth so we define a few requirements and then we start implementing what's happening so the very first thing before we actually start implementing and writing code is we need to make decisions so we say for instance well it needs to be uh when the user hits the bottom of the page so we can do it with scroll handling that's a decision that needs to be made we can say we want to load it like this we want to load it like that we want to maybe change the url not change the url there's a lot of decision making that probably happens depending on the situation that people are trying to implement things and developers make decisions once every couple of minutes i would say maybe even once every couple of seconds decisions happen all the time and many of them are very very small but lots of small decisions usually end up being accumulation of decisions that sometimes lead into a dead end or lead into problems so the earlier you can influence the decision making process the better it is once the decisions are made code is being written so in this case the code that we saw was written and then it's validated that it solves the problem the code is then being written and implemented and so that it actually solves the problem but to check if it solves the problem or not it is validated which means the developer now sits down and actually tests what they have implemented against the criteria that was decided when i scroll down more stories are being loaded 10 stories are being loaded they are loading fast they are working fine they're displaying correctly done this task is done we have succeeded and built the website successfully now you can influence each of these steps i would say optimal is when you are part of the requirements engineering when you can help your developers define what is important what is required for a solution to work in this case you should say no we cannot work with scrolling because googlebot doesn't actually scroll so if you would have told them at the requirements stage you could have potentially prevented the issue to begin with also as developers work on things they usually think quite often on the decisions they make and as they go along these decisions might be revited and revised to actually make different decisions be there for them give them guidance how can they make the right decision in this case and if they're like we are not sure if we should rely on on an intersection observer or a scroll handler you could have said well the google documentation says googlebot doesn't scroll so whatever the implementation observer is it's probably a better choice than the scrolling you could again could have avoided problems further down the road as developers are implementing this they usually also test if it does what it's supposed to be doing help them by showing them the documentation that is available on developers.google.com search also check out if you can help them by telling them what kind of tools to use to make sure that their implementation actually works the way that they intended there's plenty of tools available such as angrog local tunnel to actually get a public url that you can then plug into tools like the mobile friendly test or the rich results test last but not least as they are testing things and validating their solution works do the same do have a look at if there's anything on the page that is a no-no or a red flag for you to give them feedback right away and say like hey there's a problem with the thing that you think works it's actually not working quite right for for search or for google bot or for seo purposes and give them again then guidance and requirements to keep working that way you can participate in every part of the cycle you don't have to sit with them every day you don't have to be there multiple hours every day sometimes it's enough to show them the guidance and to teach them how to get seo requirements right and then help them test it it depends a little bit on your team and on the structure that you're working within and on the organization that you're working in but i would say try to help your developers and guide them rather than only be jumping in when things are burning to put out the fire so i think there are three simple things you can do offer your support often and early you are experienced you have the knowledge that they might not have and that's okay that's fine they are developers they have other things to do you are an seo you have other things to do as well but if you can help them and establish a connection with them where they feel comfortable asking you for guidance and help as well as offering that to them more frequently than like once a project or once every month it helps to see issues before they actually manifest in code which makes them cheaper and easier to resp to fix as well do hang out if they are having a stand-up meeting maybe join maybe listen in on that if they do sprint demos or sprint reviews maybe listen into that one as well um if you can just like have a lunch with them every now and then and ask them about what they're working on and what the challenges are or if you saw something on the internet where you're not sure if it has the seo implications or not ask them about it what is the service worker i don't know explain to me they are happily explaining that one to you i'm sure also prioritize your input if you come running with every small detail that isn't perfect and pretend that it's the showstopper number one eventually that desensitizes your developers if it is a show stopper call it out as such but if it's just a non-optimal meta description or a title tag that needs to be rewritten that's something that is relatively minor i would say for a developer and it doesn't really stop the show it's just something that you need to take care of but be reasonable just try to be reasonable don't cry wolf when there isn't more than a sleep in front of the development team and you so do make sure that you are reasonable in requesting attention from your development team and the product owner if you want to learn more we have a bunch of resources for you available there is the javascript seo basics guide that is our guide with pretty much everything that you need to know to understand javascript seo in google search there are videos the video series is currently um eight videos and season one but we will be filming the season two as well and there will be more videos coming uh we have a mailing list where you can discuss javascript site specific problems and get guidance and help from others in the community as well as googlers and we also have a bunch of documentation under developers.google.comsearch both for seos as well as developers and last but not least we do have regular webmaster office hours where you can ask your questions on youtube and we'll try to answer them there as well both for regular sites and for javascript uh we do publish a lot of different videos uh throughout like we have technical content we have non-technical content um so definitely subscribe and check out our youtube channel as well and you can find me on twitter at uh geckonaut with the three three instead of ee for geek um yeah thank you very much for listening i hope this was helpful and interesting and i'm looking forward to our q a session