Cheese, Web Workers, universal search, and more!

Key Takeaways

[Music] hi everyone and welcome to another episode of the search off the record podcast our plan for this series is to talk a bit about what's happening at google search how things work behind the scenes and who knows maybe have some fun along the way my name is john mueller i'm a search advocate on the search relations team here at google in switzerland and i'm joined for this episode by martin and gary who are also on this search relations team hi guys hi john quite attack my diamond so cool to have you two here again which is kind of like every podcast i guess but anyway i looked at the notes and i saw we've done a bit over 20 of these episodes so far which is pretty cool what do you think yeah i'm very happy to see that this is working out well and that people enjoy listening to these as much as we enjoy recording these i think except gary i think gary doesn't enjoy this no yeah that's what i thought okay fine he doesn't like nice things that's the problem no so how should we celebrate the 21st episode of the podcast what do people do with a big gathering we could cut this short cut it short no never why oh gary come on don't be such a party pooper so what were your favorite parts so far in the series yeah gary what were they i really like the endings of the recordings because you speak in a different language every time no because they end yeah i actually really enjoy your journey taking us through you know search from beginning to end that's kind of cool that is cool yeah okay i mean the endings are nice because then you had like a nice episode until then it's true it's it's kind of like dessert right it's like you've had this really nice meal all of this good stuff you learn something along the way and then you have i don't know some cheese and then it's done and you're like this ending was good hey john do you like cheese i like cheese i have it on twitter right it's like it's it's recorded i mean it's not that i put it on twitter but it's it's on my twitter it's on the record so to say you want to say it's on the record on the record well this is the after record podcast so nobody can count it against me i guess i don't know how does that actually work well but you have it on twitter that you love cheese well yeah i don't know how that got there so weird you remember the strange cheese we had at the cheese master class thing that we did a couple years back the bell park nola i have it now it's amazing wow okay i finally found a place that sells it nearby so that's pretty cool and actually it's literally across the street so i didn't expect it to you know that's really nice so you like cheese i do like cheese after that uh masterclass i got uh what is called the blue cheese it also wants some award jersey blue yeah the blue cheese and it's so amazing and you can deep fry it and then it's even more amazing what you deep fried cheese yeah you bread it and then you deep fry it it's amazing i'm telling you it's amazing you hit it with some pepper it's fantastic we should have a cheese podcast this seems to be bringing your your attention in quite well it is relevant well it's cooking cooking so talking about cooking it's like should we have more guests yeah i think it makes sense what do you guys think about getting someone external on the podcast as well not just like internal folks i have candidates no would you mind spilling the beans or nah surprise i mean i know a bunch of people i really really like to talk to not you martin sorry um and i meant external people and i would love to chat with them again on a podcast like for example i don't know aleda soldiers for example i forgot her company's name but she's one of the most prominent seos externally i think jennifer slag she's brilliant and our interviews at those in-person conferences were one of the few ones where i was actually sweating on stage because the questions were so hard to get out of eric anger i really like talking to him very smart guy eric is great mark super interesting dude yeah i have a long list nice okay that sounds pretty interesting or should we add more people internally as well no like from different parts of google or yeah like how do you see it i would love to have david price one of our lawyers or legal counsel i don't know like what's the proper term uh on the podcast because i'm pretty sure he sees interesting things happening no okay that he can't talk about i mean he can talk about a few things i'm sure and i'm pretty sure that the things that he can talk about is probably interesting okay or like we could add someone or invite someone who's more local someone like aurora perhaps yeah from the i don't know she's on the trust and safety team i think right i think it is i think it's ads policies or publisher policies or something right no i mean she's been doing the sustainable monetized website series on her channel which is pretty cool so it's like it's almost like maybe we should figure that out first yeah yeah i mean she's in reach we had maria already who else is local yeah someone from the assistant team maybe i don't know it's like voice search still a thing people haven't been asking me about it anymore for a while now what's voice search oh that's a surprise do you do you search with voice no no no i mean like now that you're in a room by yourself you can talk to your phone and you don't get any awkward looks right so is it easier nah nah i'm a fast typer i type okay i mean i'm a developer what do you expect me to say like i type quickly and it comes natural to me at this point okay you're like a machine interface martin i'm like a statically typed machine oh my gosh [Laughter] gary what about you do you ever do voice searches uh the short answer is no the long answer is that i tried and still no i don't have voice enabled like for example on my phone i disable siri right away as soon as i set up the phone i just disable it on the tv i have i think it's the google assistant that's enabled and i couldn't figure out how to turn it off unless i haven't spent all that much time trying but the few things i tried didn't work out all too well probably because of my accent perhaps you an accent no but your german is so great yeah about death did you know that the google assistant doesn't necessarily like swiss german i can imagine that it's like who really likes swiss german oh it's a beautiful language come on i love swiss german i love switzerland way more than hulk touch like high german okay swiss germany is so um melodious yeah but could you imagine if more people started searching by voice and all the people in switzerland started using swiss german like how that would change the seo world aren't there like 40 different sub dialects in swiss german because i think like every village speaks slightly varied so you'd have to set up ahreflang for all of those right could you imagine try to imagine like like even if you go down to burn for example like understanding their swiss german is tough to say the least yeah no yeah versus the turkey one where you go down to grobinden for example and try to understand that when you are coming from berlin or humble or whatever it's tough even for locals and german speakers let's not not talk about like what if you go up on top of a mountain and you find a shepherd for example and you try to talk to that person you will fail like even if you are born in zurich okay so you're saying like voice search in the mountains is a bad idea i mean you have 4g reception in switzerland on the mountain so you could try sure but it's the mountain so you have echo echo echo i think we have echo canceling that should be okay yeah on this episode of swiss languages yeah we should switch more to kind of the web again maybe invite some web workers what do you think did you just yes yes you did oh my yeah this is a fun one back in 2018 when i joined i did a bunch of tests around our javascript capabilities and one of the things that i tested was web workers and web workers are basically just a way to have like multi-threaded code on the web and offload some stuff from the main thread i'll explain all of these lovely words in a moment i see big eyes here and i noticed that in rendering web workers don't really work the way that you would expect them to work and that's normally fine because very few people are using them and even fewer people are using them for content related things but giacomo zechini i hope i didn't butcher your name giacomo if you're listening to this recently tweeted out that specific question like since 2018 no one has caught up on this like no one has noticed that behavior and he ran a few tests apparently and he noticed that the behavior is different from what you would expect and ask me a question about it and i'm like oh here we go this is interesting because i talked to the rendering team and the rendering is like yeah but is anyone really using it for anything where googlebot would need to run the web worker properly and i'm like i don't think so but i would love to hear from the community if they have considered using web workers and then not done it because they wanted to do something that involves the content and they found out that it doesn't work super well in rendering because i wonder if it really is a problem i don't know to be honest so i guess web workers is something you would do more on an interactive level like you click a button and it does some fancy calculation and then brings the results back yeah yeah that's pretty much the traditional use where the problem or the challenge on the web is that javascript is a single thread of execution so if you have like a website with lots of cool things on it and then a button that does some long running task whatever that is it could be some sort of calculation that just takes a long time for instance or it needs to do something on the network network is kind of fine because that kind of happens in the background naturally but when you do calculations that kinda is running on the same process or on the same thread of execution if you want to with everything else that happens on the page when you interact with it so scrolling the code that runs the calculations any buttons any link clicks anything that happens on the page is kind of blocked once this long running task starts so if you are having some very heavy calculation like i don't know you have an input box and you can type in give me the hundredth prime number prime number calculations take a while so if you click that it might take 10 seconds to get to the 100th prime number during these 10 seconds you can't click on anything you can't scroll the page you can't do anything on the page that's a challenge so why would you do this with a web worker and not just send it to the server and have the server do the calculation because it's like a big server with lots of cpu that is true but sometimes you might not want to do that or sometimes you might have to make it work offline as well which we can do now thanks to a specific kind of web worker called the service worker if you can push it to the server by all means do because that's the easiest thing you can potentially do but if you can't because you can't rely on the server to be there on the server to be available on the network to be available on the network to be fast or if it's just like a lot of data needs to go into the calculation and uploading the data to the server can take really really long let's say you want to do some sort of image processing right there's like scrooge.app where you can upload an image and it doesn't really upload it to the server it keeps it in your browser locally and then you can try out different image encodings and compare them if you were doing that with a server a would have to have a pretty powerful server to do that for all the users that you may have on the on the application at the same time and the other thing is you need to upload the image and then download the image and images are big usually right so it's nice to do that on your device so that the network does not really play a role here but again image data conversion takes a while so instead of running this conversion on the same thread blocking the app and making it unresponsive for a couple of seconds they are doing it in the background with a web worker okay cool and the service worker is essentially the same thing or is it like how does that fit in that's like a pwa thing right yeah a service worker is basically a very specific kind of web worker so web workers are just scripts that can run in a separate environment basically as if they were running on a separate computer kind of inside the browser but they only run when the website that starts them is actually active whereas the service worker is the same thing it is also a script that runs in its own environment but it can be triggered by the browser in certain conditions so for instance whenever the website makes a network request the service worker is triggered and asked what should happen with that another thing is if your phone receives a push notification and the service worker has subscribed to them then the service worker would be woken up to process this push notification so it is a specific kind of web worker but it uses the same mechanisms as web workers okay okay cool yeah the network request side that kind of makes sense because that way you could make it so that it kind of works offline and that it caches a lot of these things right i guess that's where it fits into the pwa side of things yeah yeah because it allows us to be like offline capable and push notifications and background fetches and all that kind of cool stuff and where would this come into play with like pages for indexing like if i imagine most of these web workers would be kind of like these interactive things where like you mentioned the squash app where you upload an image and then it does some processing and brings the results back how might you play in with normal indexing because you probably i guess that would be something that would have to work while the page is initially loading or yeah that's a really good question and the one thing i learned in my career as a developer now as a developer advocate is that developers creativity is infinite so people come up with fantastic uses of technology that you do not anticipate so for instance you could use web workers to to basically your server would give us a skeleton and placeholder version of the content that kind of just takes up space and then the web worker would do some whatever heavy lifting needs to be done in the background with like lots of api calls and stuff happening and and then eventually once it has the content already it passes it over to the main thread that's one way of reducing main thread work and i think it's one of the things that does come up for time to interact or first input delay which might become more important because of the corvette vitals but as i said like i can only guess what developers out there have built i could imagine things like oh the content is somehow encrypted and needs to be decrypted on the client's device or something like that but then i would say like yeah but if the key has to be sent to the client why encrypt in the first place nonetheless i've seen people do this kind of stuff where crypto was kind of just like the black magic that they needed to keep people calm i'm intrigued i would actually really like to hear out there if people are struggling with this if people for seo reasons had to not use web workers for something and if so what that was because i'm just intrigued what kind of creativity is out there yeah that might need our help i guess it's tricky to recognize if we ignore it for rendering then we don't really have a log of people who are trying to do it that's true yeah cool i guess you could just write your content in swiss drum and then you wouldn't have to encrypt it and decrypt it it's like the simple solution is like why do all of this magic javascript it's so complicated yeah that's true why javascript why because javascript that's why does any of this play in with like promises that's like another one of those javascript constructs where it's like you you wait for something to be finished is that like a connection to these web workers or is that totally unrelated it's somewhat related because both of it are ways of tackling asynchronous code so normally code is synchronous so you say like instruction one and then instruction two and then instruction three and maybe in if some condition is met we do instruction two and three again until it's not met then we do instruction four and five whatever with javascript the way it works is that with a lot of stuff you are kind of basically scheduling a task and that task is eventually run when there's space available to run that task so for instance a click isn't immediately executed if i have like an on click handler for a button that doesn't run asynchronously so it's not that my code stops until the button is clicked and then it does something it basically says like when the button is clicked run this function so what happens in the browser the button is clicked and the browser goes like okay so i'll add the code that is on this unclick handler to a queue where all the scrolling all the other clicking is happening as well and then eventually the time has come to run that code for the on click handler so that's asynchronous and then there's different ways of doing this there's promises where you say like hey when that happens run this code there's also callbacks it's another way of saying hey when that happens run this code but all of this still happens in the same thread so the same like all this is sharing the same execution environment the same processor if you want whereas a web worker that is a completely separate environment that is completely detached from the rest of the code and it communicates with your main thread with the main line of execution through messages so you send messages to and from so it's somewhat related but it's not the same thing okay promises run in the same environment workers run in a separate environment okay so it's like with javascript you're always waiting for things i i got it mm-hmm how does that pretty much wasm fit in here that's something i heard with squoosh like they made this with wasm apparently yeah what's wasm so wasm is the short version of web assembly and webassembly is an interesting project and it kind of ties into web workers as well because the idea there is with javascript you write javascript which is human readable at least developer readable let's say so this text then needs to be translated into machine readable instructions and then these machine readable instructions are being executed that's great for normal amounts of javascript but for a bunch of stuff it would be cool to kind of have like a way to skip this first part to have the human written code and the room and readable code translated straight into this machine readable format and then executed and that's what webassembly is that's where it came from the original idea was hey we can actually speed things up by skipping the first part or making it as quick as possible so browsers have now implemented a way to basically give us a pre-interpreted a pre-translated version of the human readable javascript it's a machine readable version of that and then that can execute faster and that brought us a really really cool thing which is the idea of taking something that is in a programming language that traditionally does not run on the web like c plus plus or c code or rust code and then basically make it into this machine readable javascript form and then run that and that's exactly what squoosh did so for instance the code to transform an image into jpeg that is c code normally that you would run on your machine it doesn't really run in the browser so they made a webassembly version of it and in order to not have the website block entirely because if you have like an image let's say like 4096 by 4096 pixels that's a whole lot of pixels it sounds large but it actually isn't actually make a different example if you look at a full screen image for a 1080p screen then we have and i'm actually typing this into a calculator because my math isn't that great we have to process 2 million pixels each pixel being at least 3 bytes large that's a lot of data that you need to process and that would mean that if i would do that in a web assembly or javascript environment just as part of the page it would take a few maybe like two seconds but in these two seconds it wouldn't respond to any scrolls any clicks anything that i do with the page so they are running their web assembly in a web worker so they basically say like hi here is the data here the i don't know three megabytes of image data that we have or two megabytes of image data that we have please make that into a jpeg and then they throw that over and the website is now available to like respond to clicks and everything the web worker does the work does the webassembly execution for the c code that has been compiled into webassembly to actually make it jpeg data and then it sends that jpeg data back and then we can display that in the image tag tada wow okay it all fits together seems like a lot of works and you're saying like web workers currently don't work for rendering they kind of do i mean we execute them and we register them it's just that because they are in a different environment we are not very good at picking up the events that happen in them so for instance if you have a web worker that gets sent a message and then immediately responds back that works generally but if you have a web worker that fetches something from the network then we don't really wait for the web worker to actually get the network response and send the response back to the main threads so you see really funny things happening in web workers when you execute them in rendering yeah so what about promises and wasm do they work like if you have them in your normal thread they do work yeah i think we are waiting for the task queue to be empty and promises add something to the task queue of the main thread and so as long as it happens in the main thread you will be fine so cool okay well i didn't understand anything out of that what come on it's that can't be true it's javascript gary it's the language of the future apart from swiss german i don't like the future oh swiss german is the encryption of the future i think someone should make a programming language using swiss german commands i i think that would be super cool do you remember the time when uh someone had this brilliant idea that i don't remember which programming language should have german instructions oh that's a vb script visual basic script yeah that was like that as well right yeah or was i just like in excel or something like if you ran it there i think it was vbscript and that went well all right well i mean the thing that always annoyed me there is you could never like copy an example from somewhere on the web and just make it work because you had to translate all of the commands as well but i'm thinking of a programming language that is swiss german by default and doesn't have local versions hmm interesting i know that there was this if php were british article ones that was pretty hilarious instead of like if else it's per chance and otherwise i think and instead of switch it's what about yeah and try catch becomes would you mind and actually i do mind and it's hilarious okay see something like that just with swiss german i think that would be perfect it's like all of these computer people that just speak english and it's like they should see what it's like to actually have to learn something new again oh wonderful i was writing some c plus bus code internally and i wanted an example for something very specific and i found this blog on the internet which had well it promised that i would find the example there but it looked like someone ran uh perhaps grammarly or something like that on the code as well so a bunch of things were corrected like i don't even know like the if statements for example they were expanded to sound more human-like like the parentheses for example were dropped and then if whatever blah blah blah this then it was like this what yeah i just noticed that sometimes when you have quotes in code oh yeah depending on how people put the code samples together it'll be like those weird smart quotes and you copy and paste it and nothing works it's like ah and then your compiler is like nope nope nope nope nope i have never seen this symbol before in my life it's like come on it's a quote it's a quote too smart for me sorry ah if only there were universal compilers double quotes or single quotes yeah that's confusing too it's like in some programming languages they're the same and others they're not the same it's like why can't they make up their mind oh what is it shell script i i love javascript for that reason oh oh it's like you used a double quote here i can't interpret whatever you put inside the double quote okay here's a single quote okay now i will execute it but you put this other character inside which you forgot to escape put it in double quotes yeah if only the error messages were that clear usually it's just like oh i've deleted all your files that's fair [Music] so in the previous episodes we were talking about serving and i think we are finished with ranking things more or less there is one interesting thing in most search engines which is universal search it's not google specific other search engines do it as well and because most search engines rank results roughly the same way with different flavors this is applicable probably to a free search engine that i can think of so during the ranking process we assigned a score to every result that we retrieved from our index we talked about the web index but the reality is that when we created our rewritten query the parsed query then we sent that to every index that we have so for example if we have an index for images if we have an index for videos if we have an index for news and so on they would also get that parsed query and then they would do the same thing that we explained in case of the web index which is basically console posting list for example and then based on that retrieve some documents that is then ranked or assigned a score now once they've found stuff then they will return that to something called let's make up a name blender or mixer i don't know something universal mixer and that will take all the results that came up from all the different indexes and try to mix them together so the reality is that the result page is actually very similar to a an auction hole where results are actually bidding for their desired positions so for example the featured snippet might want the first position so it will submit a score that i can bid this much for this position it can also say that i don't want second position or third position or fourth position or so on i only want the first position and then we look at the web results for instance and i'm gesticulating with my hand again even though i'm on a podcast i don't know why i do that i always do that so we have the feature snippet which bid let's say 1000 for the first position then we have the first or the next web result which has a score of 1500 let's say then that means that that web result outbid the feature snippet but because the feature snippet said that i can bid 1000 and 1000 only and i only want the first position it will just fall out the result set because it didn't want the second place it'd be for the first place and first place only the next web results that have scores assigned to them they will naturally follow another thing that happens in universal mixer is that as i said we take the results from the other indexes and then we will try to mix them into the result set now we have preferred positions for some things like for example the video results they are just highlighted by assigning a thumbnail nowadays to the web result where the video is found in let's say johnwheeler.com will get a video because john will put up a video of how to make fondue for example and then the way that's highlighted is by assigning the screenshot or thumbnail from the video to the web result but in case of images we actually get a universal result which is nowadays is more like a carousel i think especially on mobile where you can scroll horizontally through the images i think that might not be completely true um i use my own search engine i developed one yesterday it's called steve anyway so we have preferred positions for those and we will try to match them into the or among the web results what else do we have there then we have certain search features that always appear roughly at the bottom of search result page like for example the related searches which is a feature that also gets the parse query or the interpreted query and then based on that it will return a set of queries that are related to the original query in some sense but that feature doesn't want to take up space in the browsing area or where you are you would expect the search is also it's usually positioned at the bottom what else do we have news so the related searches on the bottom i guess would need to bid something really low or would they kind of also say like my preferred position is 10 or whatever i think it's more like that like my preferred position is this and i can beat this much for that perhaps related searches is not a great example because you are not competing for a spot like it's not like you are taking up the space of a result but it's more like you are extending the search result page with a new visual element perhaps it's more similar to knowledge panel i guess okay so it's almost like all of these different indexes or kind of content have their own search engine and basically they're saying like my result is like super relevant or kind of relevant and then there's like a super search engine on top of all of these search engines that mixes them all together technically yes i mean many of our systems are shared like for example google bot is not used only for web trolling it's used for fetching ads landing pages as well it's using a different user agent for when it's doing some other job but it's still a shared system it's used by many teams and the same way the retrieval systems might be used for different things as well so for example if i need a featured snippet for example then i can technically use the same retrieval system that we use for retrieving web results okay because essentially it's doing the same thing but perhaps differently like something has to be different about it and these systems or many of these systems but especially the retrieval system is modular enough that you can just plug in your own module that you wrote for ranking stuff differently like for example with images you have to rank the landing pages slightly differently because you are focusing on the image not the landing page itself necessarily okay so cool and like how do we recognize if we should show images or videos or that is is it just like video search saying it's like oh it's like this cat video you have to see this cat right that's a fantastic question we learn it so like when you search for something something that normally doesn't have images or videos and you tap the images tab on the result page then you are essentially teaching google that there was this random person who wanted images for this particular query and if there are enough users doing that then you are essentially teaching google that that query might deserve images or videos or whatever so cool so cool and you mentioned they can bid for individual positions how do those positions map to things like search console uh webmaster tools webmaster tools yeah yeah so every position has a number assigned to it like for example we have position one zero whatever and it doesn't matter all that much what we put in that position right we can just say that there was something in that position usually when we log i imagine that you are talking about impressions and clicks i haven't been to search console for for a long time so i imagine no yeah i yeah i only use steve console anyway so uh since you're talking about impressions and clicks we can record that there was like this was the query that is obvious and then there was an impression a disposition basically you showed up at this position we don't necessarily have to say what you showed up with like for example you could have been with in a featured snippet which is just an expanded result essentially with the smarter snippeting i guess but it was just a result and then we record that impression happened at this web result position and then when someone taps on it then we can just say that but that web result there was a tap on or click for this page for this query and then we aggregate all that and present it in search console okay so here's a controversial question oh i'm glad that you found me for this okay so featured snippets you mentioned them a few times externally people say this is ranking at position zero okay is it zero i don't know never looked it up i mean if you ask a developer then whoever worked with arrays for example then it will be zero it will be zero but like if you have a normal web result as the first one is that also zero no that's one unless you're a developer because then it's zero because it's an array okay am i confusing enough i i um okay so the non-funny answer is that i actually have no idea what position that would be i could probably look it up but i really don't want to okay but it's like the general notion that is sometimes seen externally of like if your web result ranks first that's position one but if you have a featured snippet that's position zero that doesn't make that much sense or does it might make sense it's like well if you want my interpretation then i think it doesn't make sense okay i like externally i did say position zero a few times just to use the same language as seos oh it comes from you what no no no why would you say that this is how rumors have started well it's good thing that we are off the record but i would argue that it is a position one not a position zero okay yeah i also noticed like depending on the the programming languages they also have this either zero or one for the first item in the arrays so it's like super confusing like someone should figure that out it's like all of these off by one errors i bet it's like lots of things break do they do do they i don't know i mean everything that has software in their brakes it's i think that's part of the universal law it's not a bug it's a random feature it's like a fun surprise waiting for you to be discovered okay that's one way to kill a discussion just bring up carries just bring up a raise no okay fine would you prefer linked lists gary oh god how about some graphs graphs i actually don't mind i actually like working with graphs cool unless they're represented as an adjacency matrix that's i don't like that representation too much to be honest okay this is getting like way too complicated technical for me but but if we keep going on for one more minute and 45 seconds then we passed one hour recording oh my gosh okay you should see our producer facepalming very very hard in the background right now well luckily this is a podcast nobody can see anything so we're all safe that's true that's why i'm saying it we are off the record after all we're all off the record cool all right i'm sure there will be zero cuts we'll just like upload this uncut straight to the internet yeah okay well in that case then there's nothing left for me to say other than that's it for this episode of the search after record podcast we've been having fun with these podcast episodes and i hope you the listener have been finding them both entertaining and insightful as well and at any rate let us know if there's anything that you'd like to have covered differently if you'd like to have some web workers featured as guests along the way or if there are other topics that we should be covering as well and of course don't forget to like and subscribe and see you all next time bye cheers