Search Console, JavaScript, serving, and more!

Key Takeaways

[Music] welcome everyone to another special episode of search off the record podcast our plan with this series is to talk a bit about what's happening at google search how things work behind the scenes and who knows maybe have some fun along the way this episode is a bit special so you'll soon find out more my name is john mueller i'm a search advocate on the search relations team here at google in switzerland i'm joined for this episode by martin and gary who are also on the search relations team and we have a special guest today daniel yay daniel oh yeah hey ah so good to have you on the podcast daniel for those who don't know you yet would you be so kind and tell us what you are doing on the search central team and who you're working with of course but first of all i want to say that every time that there is a new episode of the podcast i'll wait all the day until like after work i'll pour myself a glass of wine and i'll listen to it very carefully to learn a lot from you guys so thank you very much for the podcast it's really a pleasure to be here thanks so cool they're here thank you yeah so i joined google about eight years ago i was part of the google analytics team for six years as an analytics advocate and so doing stuff related to data analysis best practices and data visualization and how to use data in a way that can you know make your business succeed so this kind of stuff and doing presentations as we do in our team and writing stuff and working with the engineering team to make our products better so i joined the search relations team about two years ago and it's actually a funny story that i found this position playing ping pong what so i was looking for a new challenge and i was playing with someone from the developer relations team so i actually like playing ping pong and i was telling him the story and he was like you should talk to john do you know john and i've heard about john and so i reached out to john and we had a really nice chat and he put me in contact with the search console team and we had a really nice chat so and because the search console team is based in israel it also made a lot of sense to have someone that you know speaks hebrew and could relocate back to israel and i've been in the team since then i'm kind of embedded in the search console team in a way so i work a lot with the engineers and but also with the search relations teams and it's a great place to be i'm really enjoying the ride and we enjoy that you are here it's so cool that you work with a search console team because i think a lot of stuff has happened and is happening in the search console product space is there anything from the bunch of recent announcements and releases that you would like to highlight so one of the recent launches that i was really excited about and also because it came mainly from the users it was a top request for a while is we added the regex futures to the performance report and regex regex okay got it rejects okay so yeah regex so performance report so basically this feature just to explain to those that didn't see is the new capability to perform more advanced features using regular expressions and it landed really well so people are really happy because well it was a top user request and they've been asking and i think that really shows how we are kind of trying to developing new features and you know listening to what people have to say and developing new capabilities that help people perform their jobs that's interesting so you said that this feature is what you would consider a small feature but has big impact and that it was picked because it was the most requested by users so how would i as a search console user request a feature then so there are a few ways that we try to kick off new features and of course users so when we know that users need the feature to do their jobs we try to make it happen so it could be either through twitter we are over there and we're hearing what our users have to say it could be the user feedback button on a search console and it could also be for example the search community the forums where we're always also monitoring and trying to see what people are asking and what kind of problems they're seeing as well so that's one way that's how users can influence the tool and apart from that we also of course get requests from up above you know from our leadership and sometimes they have ideas they want to cross-functional features things like that and we try to do our best to incorporate those features into search console but also i see a lot and usually i'm very excited about those features our engineers find ways to make it either more accurate or more interesting or more user-friendly and they kind of bring new ideas and make them happen one example also very small addition which i think was really cool is that on the tables on search console you have all kinds of icons besides the pages and there is one to copy the url and this is a feature that one of our engineers he every time that he was checking something he had to copy the url and it really annoyed him so he just built a feature and it's super cool because it's really really helpful and so i think it shows one of those examples so basically again three ways to start a new feature one of them is through user feedback through leadership requests and also our engineers initiative that's really cool that's nice and i guess we have like lots of cool other things coming up in the future and i'm really looking forward to see where we're taking search console some of the things that i'm really excited about and i think that we'll have more stuff coming up so my background is in data analytics and obviously i'm i'm biased towards data and how can we make more data available in a better way and so that's one of things that i'm very excited about how we can make more data available to users so that they can build their own solution so that they can innovate with our data and and i think that would be super cool that is super cool indeed yeah so you're kind of saying analytics is helping search console did i so we use analytics in search [Laughter] no we don't we don't use analytics and search that's the first thing that i learned in the team john from you you said we don't use analytics in search that's true cool i love seeing all of the new stuff that the search console team is bringing out and i really like also that mix of some things coming from users some things coming from the team and some things coming from leadership where you can really tell that the people who are involved are passionate about the project and i think it would be kind of awkward if we didn't have people from leadership saying he's like oh you should do this because this is fantastic because if they come with these kind of requests that means they're aware of the product they know where it fits in and they can connect the other products around google to kind of make sure that search console plays a good role in the whole ecosystem so i think that's all pretty cool [Music] one of the things that i've been working on on the side in my own time is trying to figure out a solution for my blog where when i look at it i think i last published something almost 10 years ago so it's almost like a static site and i keep having to update the cms all the time so that's one of those things where i'm like i need to find a better way to deal with all of this i don't know martin how do you deal with your cms or do you still use your own oh so i have long stopped using my own cms and i have also i have to admit that i have kind of long stopped using cmss at all because most of the things that i built on the site are actually just static websites where i control the content and like there isn't that much interactivity on it so i usually use a static site generator for the sake of simplicity i usually use hugo but i know there's so many others there's like 11 t and i don't know like there's so many different static site generators that are fantastic and then you don't have all of these like weird issues with like having to update things and installing security patches because you basically just search static html files wow what a coincidence i don't know i also just started using hugo oh look at that wow okay i didn't even realize i don't know i didn't even check but it's one of those things where i thought like well my blog is mostly static anyway it might as well make it completely static so i looked around for different static site generators to move to something like a more i don't know static html setup and i ended up with hugo which uses markdown pretty much for the content so you just create text files and then you run some script and it generates a website for you so i thought that was pretty cool kind of surprising to see like independently we came to a similar solution so cool yeah i don't know it's one of those things where in the beginning i thought well i'll just use a static site generator because it'll be easier and it'll be completely hassle-free because like what is easier than creating an html page but there's just so much involved i ended up spending so much time digging into all of the different settings and i don't know tweaking the themes and setting things up so that they actually work i'm like at this time i could have i don't know written something like wordpress on my own pretty much there's just so much of all but i really enjoy making pages again which is kind of weird and i ended up writing a whole bunch of blog posts that are not published because of course my site is not actually live yet which is kind of awkward because i kind of switched to the static site generator because my site is static and then because of that i ended up making my site a little bit more dynamic so everything is kind of weird in that regard irony is everywhere but holy moly themes themes are so tricky to pick how did you decide which theme you go with i just clicked on some random ones and it's like oh this one is pretty simple looking i'll just use this one i mean the thing is once you start tweaking the theme yourself you're pretty much committed you can't just like tweak around and then switch to something different again so that's true yeah that was kind of tricky and then trying to implement all of the things that i kind of wanted to have that took a little bit of figuring out how all of these different aspects of this static site generator work together so one simple example is i wanted to have nofollow links or have it so that i can put nofollow link somewhere and in markdown there's no notion of nofollow links so essentially i could just say well i don't care about nofollow links it's like what is google going to do penalize my blog it's like we will see but i figured out some options and i can make no follow links now so if you want to sell links on your site martin with nofollow of course then i can help you out there nah nah thanks i'm good i think i'm okay oh well okay and the other thing that took a while to figure out is everything around redirects where once you have everything in static html you essentially have to use javascript redirects to make all of that work and i don't know that was a little bit of a hurdle because i was like oh static site generator makes clean html and then everything for search will just work and then it's like well actually redirects you have to do with javascript and then like oh it's like oh my gosh i've created this weird mix of javascript and static html site which is probably like every website out there i guess but i don't know you end up running into these things how do you see javascript working out nowadays martin should i be worried no you don't have to be worried about that i actually have a question for gary i think because i was wondering like static html sites generated by a static site generator probably have two ways of making redirects if you don't consider the server being an option one way is the javascript redirect and you can potentially also have like a meta refresh tag on the page but gary how do we treat meta refresh tags is that a good thing or should we avoid them so when we were creating the redirects for the blog for the webmasters blog to the search central blog then my preference would have been the meta redirect because that we can understand without executing any javascript right but failing that javascript should work tm love it a question that i often also get with javascript is like if we treat javascript content differently and i mean we do have annotations for content like what we think is the centerpiece of an article or what we think is like content on the side and stuff but as far as i know and as far as i can see we crawl a page and then put the content into the document in our index and then we render the page and then we complete the content from the dom and then that's it like there's nothing that is fundamentally different between javascript generated content and static content except for when there's edge cases and we can't see content that is generated by javascript and i think links right links are also immediately extracted from the crawled html and then we do that again when we have the rendered html i think there's like a logic that gets us linked discovery as early as possible in the process too but besides that i think the content is pretty much treated the same way so you should not be too worried about javascript john okay sounds good i mean like first i have to get my site finished anyway so we will see and maybe it'll be penalized by google anyway because of my bad nofollow links so we'll see oh you just reminded me i need to actually verify so i kinda with a friend i founded a foundation to foster first aid training in switzerland and i made a website for that and i don't have a google search console account for it yet i should probably get that no martin no i just set up analytics daniel calm down no come on i put my static site on google cloud and in order to do that you verify the domain and because of that it's like automatically in search console it's so fantastic well that's actually nice yeah number one ranking guarantee oh no i still don't know which keywords but like i'm sure it'll be somewhere how to hack ftp oh my gosh all of those old blog posts i went through all of those and it was like should i just delete everything that's older than like 10 years and i ended up going the other way and i found some other old sites of mine that i had and i put all of that content in there as well so it'll be like a time warp wow it's like your own version of the internet archive i was surprised that i found all of this stuff and like they were weird cms's but i don't know the mysql dumps i still had there and i was able to pull it out of there directly so it's it's super crazy oh i thought you were like googling for it and finding it which would hint at how well optimized your stuff was no i mean the domains that i had back then they're like mostly expired so it's like it's gone i mean it's probably in the internet archive somewhere but oh well and i thought like all of this stuff is like super embarrassing like old test sites where i realized that actually having a link to a page helps to get it indexed i was like at the time that was pretty mind-blowing but nowadays i'm like well i mean like people have to learn somehow right so yeah i i figured i might as well put all of this old stuff online as well and show people that it's like you can do weird stuff and have i don't know strange questions and try to figure it out on your own and still i don't know make it somewhere so at least i made it to a podcast yeah no i also think like an iterative process is a good idea like i literally was like okay so i know that there's people who want to book the courses and i need to have like the booking system and stuff online so i basically just build a first version of it if you look at the copy it is definitely not great i'll get someone to write proper copy for it to actually optimize it properly but the first version i'm not sharing that anywhere because i know that people are like oh look at this how badly done this is i'm like it doesn't matter it currently serves our goal and that's what it needs to do and if you had a question and you found out the answer and then you publish something about it that serves its own goal so that's fine i think that's perfectly legitimate yeah i think we're kind of probably the exceptions when you look online because we're like oh we don't care if we get a lot of traffic or not and the other people that we talk to of course because they come to us with seo questions or like how do i get as much traffic as possible yeah and i think interesting for us to try things out and to get some hands-on experience with these kind of sites but the background is kind of still different we kind of have to i think keep that in mind too do you know john that the first fondue that i made that was based on a recipe that you put up on google frugal oh my gosh or whatever its name was google base right or google base yes and i actually found that recipe without looking what actually i didn't even know who you are like i didn't know that there's a john mueller who works at google or maybe he wasn't even working at google at that point i don't know and i found this recipe on apparently google base and i made it and it was great and when i actually moved to switzerland then i was looking for that recipe and you couldn't find it anymore and it was very sad yeah i think google base moved on to something else what did it end up being is that like shopping i think probably shopping yeah yeah i'll have to dig up that recipe and put it on my side too just for old times sake sounded like you were saying like it moved on to greener pastures well it kind of did very indirectly [Music] oh my god so gary tell us a bit more about what's happening in search we've been kind of like following along with all of these cool stories around indexing and serving like what's the next step what are we missing so in one of the last episodes we talked about serving a little bit and we had a very high level overview of what's happening there and maybe we should start taking it apart and focus on the different components that we have in serving that make the magic happen and one of the first things that are interesting enough that i think we should talk about is query parsing and query understanding of course before that there's other systems like the load balancer and whatnot but i don't think that they are all that interesting from our perspective because they are just network components that route the traffic and try their best to prevent ddos attacks so let's talk about query parsing and i know for certain that there is a video from paul har from a webmaster conference in mountain view where he talks a little bit about query parsing and understanding but it covers two of the bigger elements of query parsing and understanding but there are other elements or other aspects to it that are also very interesting and externally very visible so when we query our index then before we can actually send the query to the index we have to transform it into something that our index will actually understand and that would be the parsed query and the parsed query compared to the user's query at least in most cases is an absolute monstrosity basically if you search for something like purchase auto or purchase car for example that becomes a massive protocol buffer that protocol buffer look it up on your favorite search engine it's open source but it becomes this massive protocol buffer with many many many many words and that's essentially what query understanding and parsing does expanding the query into something that will not only return the results that are relevant for that query that exact query that the user searched for but also very closely related queries and sometimes it also makes mistakes that you can see in the video that we probably will link to in our description so query parsing first you have to decide what to do with a query right like if you have purchased car for example then there's not all that much to parse about that there's two words and you probably want to synonymize them but otherwise it's fairly straightforward what to do with it but if you have stuff like the lord of the rings for example then that's a more complex query and you have to decide what to do with that string do you treat each word separately or you treat the whole thing as a basically a compound and as an essentially an entity on itself then there are other languages besides english i know this is a shocker and some languages don't have spaces in them for example or traditionally don't have spaces in them and then you have to decide where to insert spaces this is especially important for languages like tai where a sentence well a string can go on and on and on and on and on and if you translate it then there would be multiple sentences in that one string because they seldom use punctuation for reasons i don't know why chinese for example that's also a good example for lack of space and punctuation and that's just how the language is there's nothing wrong with it users of those languages can understand them but for machines it's more complicated so we have a system called a segmenter that will try to split up the strings that the user supplied to us and basically create words that we can look up in our index now now that we have our parsed i guess user query we move on to understanding and in this phase we do a lot of magic one would be synonyms for example and also anti-synonyms i guess if you search for purchase car then you would also expand it or we would also expand that query to let's say purchase auto or buy car or i forgot what the last one is but you get the point then we would also inject in the parse query the plural forms of the noun i guess in this case so it would be purchase cars and then we do other things ah i forgot to mention anti-synonyms this is paul hars example where we this was back in 2005 i think 2004 2005 where we learned for some reason automatically our synonym system learned that united the airlines is a synonym of continental airlines which was pretty bad and of course i mean if you know about airlines history then you know that later they merged we knew it which was great but it was several years later and paul said that he hopes that it wasn't our fault that they merged we don't know what else do we do so yeah in that case we want to break that basically we don't want to associate united for example with continental airlines what else do we do stop words we have to remove stop words if you think about it the internet is full of words like off the a and in most cases you don't need those words in the query you don't have to search for them because well they are just not relevant so you can safely drop them most of the time because there are cases where you can't actually drop them like the lord of the rings there you actually need all of them to understand the query and to detect the entity what else would you add if you designed a query parsing system do we do the same thing with understanding the content on the page as well or are these kind of different systems that evolve in parallel they are different systems so when we parse a page then we just tokenize it basically we do the splitting if the language needs it but otherwise we just well put everything in the index that we find on the page okay so like synonyms it's not that we would store the synonyms with the index we would pick them up when someone searches for them that's right so for example when i search for purchase car for example then if marketing split.they has a page for that or a page for purchase auto then we would pick potentially that page if it doesn't suck and would return it because we expanded the query to purchase auto as well so basically the query would become purchase car or auto okay cool would we also try to understand the language of the query or is that something where we basically say if the words are on the page then the language is probably okay so the language that becomes more interesting in a lower level but usually we pick up the language from the browser usually it's not always basically if the browser sends accept language header for example then we would pick that up and use that or if you set your google search language to i don't know thai for example then we would detect that and use that unless it's obvious that you are not searching in that language like for example if your browser is set to japanese and you are using latin characters searching for banana then it's more likely that you want english results so cool so many things are happening in search and essentially all hidden behind that tiny text box it's so pretty cool wait wait wait you are not off the hook yet oh no like there are way more things that happen in query parsing and understanding oh my gosh there's stemming for example i knew it okay tell us more where you search for auto trading what do you do with trading you have to do something with trading because you want to expand that word to other words what do you do with it you need to find the base form the infinitive kinda yeah that's stemming yeah basically it becomes auto trading or trade so is that something where we have a team of linguists that put together a big spreadsheet with all of the different versions of the words or how does that happen great question but no no language this is learned automatically from the web corpora so basically we have all the words on the internet well not all the words but a large portion of the words of the internet and we learn from there we learn how to spell things and how to do spelling correction for example and sometimes that also does funny things to our spell correction system where we say that no you spell this wrong and you are like dude i learned this when i was two years old i know that this is how it's spelled and google insists insists that you spelled it wrong anyway what else do we do also we remove you said we remove stop words like a and yeah and stuff but what if a stop word is important for instance if i search for the rock i probably mean the rock well then we don't remove it yeah okay so we put them in when we or we leave them in when we think it's important how do we know when it's important so there's a concept detection going on and we try to match the query against concepts that we learned from well the internet okay and then if it matches a concept then it's likely that it could be that but of course context also matters and the rock might mean something different in different cultures or countries like if you go to the us or hawaii then the rock that is a completely different thing as in entity than searching for the same thing in malta but essentially yes like we match it against concepts and if there is a concept then we will keep the the stop word there okay so we would take into account the user's location when it comes to understanding which words we added as a context yes so cool okay so that's important when people search for i don't know what the good example is do groceries that's a predominantly american thing to say and in the uk they would say differently in australia they would say differently so that's why you need a context so cool i guess we also use that for understanding when things are more local like if you're searching for pizza like probably web search will try to say oh well john does not want a random blog with a pizza recipe um that gets way more complicated because there we will have the whatever we call it i don't remember what we call it the external is called the map pack that will pop up and to that thing we will just pass on the well the parsed query plus the location that was detected in a separate thread and we will try to give something for the user that is very close to their location but that is unrelated to what we have in the parse query what about numbers i think we also might want to have a thing that if i type the numeric words like 200 or seventy four or four nine swansea which is particularly tricky because germans would like flip the digits so that we understand that it's a numeric quantity and with the parser also rewrite that to an actual numeric i actually don't know try it i would expect that at least in english it would just match up against synonyms but i don't know how it would work in weird languages like hungarian and german and just is this also where we pick up things like the umlauts where in german you have those dots on the u for example and you can also write them as u e like my last name like mueller like theoretically i have this umlaut thing on my u and to keep it simple i just use the english version which is ue would we just see those as synonyms and learn that over time that is a great question and you will get a covet cookie for that because now i have lots of cookie recipes that is something called diacritical expansion and we will try to expand the query with the different ways people might write those funny characters so like for example the o with two dots that appears in many languages hungarian german tweeted finnish dansk i think what else that might be it actually that also becomes uh o just simple o like what you do with your name or o e which is the more traditional way to express those umlaut characters basically we call umlaut character any character that has two dots on it but they would essentially become synonyms in the parse query so cool yeah most of the time that works it's the results are slightly different but i imagine that just because the parse query is like slightly different in the end that is another excellent point for another copied cookie so all the synonyms that we add will have a weight so basically your original query i don't know milling john will have a weight of one basically as you wrote those words they will count in full weight any synonym that we add will have a lower weight and the less confident we are in the synonym the less weight it will have so basically if you search for milling john then that would be the 100 weight but if we add to that a synonym let's say grinding john okay then that will have much less weight of course yeah it will just grind off the weight i i see it makes very much sense i mean in german it kind of does oh my gosh thanks a lot super insightful stuff it's like i always find it fascinating with these things where i'm like oh i know how this works more or less and then i say something and you're like well actually it's very different so that's always cool [Music] anyway so slowly things are opening up around here how do you all feel about making it back to the office at some point no no no why no look around it's dark it's warm i'm alone it's great but what about the kale gary you're missing kale yeah exactly and that's a great thing but what about your tiny sofa oh i threw it out you did actually i didn't i just left it on the desk good good what about eating oh two meals a day oh that's cute gary cooks so damn well by the way i should probably pop by at some time point again well you're always welcome you can just pop over and thank you yes yeah we are back to the office in tel aviv what like wow a few days a week or what well in theory we are 50 back but the 50 is not calculated per person it's calculated per office oh which means that some people well there are still people that they don't want to come at all so in the end you can come more or less every day and you just have to book before and well the restaurants they're back and the tel aviv office this has a very nice food which i haven't eaten for a long time and i want to come i didn't get even one coffee cookie from gary so i'm kind of i'm still maybe because i didn't answer any right questions well or you know maybe because you're in israel maybe but that cafe that we have well cafes that we have in the tel aviv office that is in my top three yeah very great basically it's like tokyo office and tel aviv obviously are like competing for the first spot yeah so you would go back to the office if you could go back to tel aviv i would probably want to go to tel aviv just to eat the food is so amazing yeah yes but i think even more than the food like being back to the office i think over the last few weeks since we started like coming back more often it's been really nice i think what changes it is that there is a certain threshold that it's becoming really interesting and it really helps also when it comes to work so what i'm trying to say is that people you know there are many people in the team that you know i used to have a day-to-day interaction and we would sometimes eat together lunch or go for walk in the middle of the day or just you know meet in the micro kitchen and i would hear about new projects in the team and and what's going on and this kind of stuff i think at least or maybe it's me but i didn't keep this kind of social interaction with people they're not really good friends like i have some good friends in the team that we kept you know in touch and meeting during the last year but i think now that we have more people back to the office i can see how like having like an office and a micro kitchen and like how it encourages creativity and you know interaction between people and so i think that part is really nice and i'm really happy to be back partially for now but i think i'll also come back fully when everything opens as normal yeah i really kind of miss those social interactions where you just see different faces during the day i'm kind of curious to see how it'll it'll end up where it's like you've been working at home for such a long time and then kind of suddenly transitioning back to having to interact with real people and not just like being in front of your own computer kind of thing but john usually we were standing at our desks with our headphones on what are you talking about what that's not true gary come on like sometimes you put off the headphones and then it's like martin i have a question and we talked and then we went into the cafe around the corner to actually have i never did that you did literally oh yes it happened oh yes it happened or i brought you a little sofa which you were very excited and happy about because that's exactly what you expected me to get you you know okay we will need to beep out the following five seconds and sometimes anna our video producer just came by with like tons of equipment because she had her desk near hours i don't think we have desks anymore right it's like all is up for grabs when you book to go into the office right now but we used to have our own desks and then anna our video producer was nearby and then she like logged around all this equipment i was like all right ladies and gents we're recording a podcast today we're like yeah we totally knew that and we totally prepped for it too yeah so you know office days i'm sure we'll just have like bouncy balls all over and then everyone gets a bouncy ball no more desks ball pits ball pits that's we'll see anyway maybe we should take a break here before we get too far off tangent and figure out what all we're missing oh man so many things anyway thank you for joining us daniel it was great having you here uh where can folks find you online if they have more questions about search console they can't question i'm not answering any questions well i'm usually no question i'm usually on twitter and i kind of i read the questions i'm not as good as you guys entering them but i'm usually at least reading the questions and when there is something that you know well we find bugs or well not bugs because search console does not have bugs but things behaving in a different way let's say so i'm mostly on twitter and of course search central youtube channel so well we all have videos there i have the search console training which is over for now we'll see how it goes but it's over there so so cool yeah yeah so should we stay tuned yeah stay tuned martin oh my gosh well i don't know martin is about to fall from his chair no it's been fun doing these podcasts and i hope you the listener have been finding them both entertaining and insightful as well at any rate let us know how you're liking these and if there are any topics that you think we should be covering in one of the future episodes and of course don't forget to like and subscribe bye everyone bye bye stay tuned [Music]