Figuring out finicky 404s

Key Takeaways

laughs [Laughter] [Music] hello and welcome everybody to another episode of SEO fairy tales with me today is my king founder and CEO of ipull rank where you help bigger Brands to be more successful in Google search results and the search engines probably as well and you are objectively the best rapper at least that's what it says on your website is that I mean if it's on the Internet it's true obviously obviously absolutely everything on the internet is true I think that's the takeaway for the episode but before we make that our takeaway you brought me a little story From The Trenches so to speak uh what happened there [Music] so I was in a room with probably like 20 people at one of my clients they're like one of the car comparison sites and uh their c-level executive like the sponsor for the SEO program was like hey Mike we're seeing a big spike in 404s and this guy is awesome he's always been super supportive of everything that we're doing and he was like I'm not sure why that is can we figure that out it might not even be a problem who knows right if I just deleted a bunch of stuff and yeah so I'm I'm on the spot at this point I'm like okay let's figure this out and you know first they're they're showing the screenshot of of GSC and it's like okay who's fighting software for us and I'm like all right well let's see what's going on right you see that in the coverage reported in Google search console right Juicy yeah all right yep [Music] foreign because you know I love the platform in that it allows you to combine crawl data with log file data so we get a sense of how googlebot is actually crawling and so I looked I looked to rule out some of the obvious things because you know typically client is like hey was it an algorithm update very rarely is it an algorithm update um and so we're looking through trying to see if there's any like rendering issues going on because that was my first instinct true if it's like a soft 404 that might just be rendering so was it rendering to be blamed technically we'll get there we'll get there so okay all right so uh we looked through all the crawl data we looked at the log files and there was no spikes in in log file activity which I typically see if it was an algorithm update right [Music] and so um you know from there I was just looking through like okay uh was there any changes to the pages because you can do the comparison and line always says no when you ask that like this did you change anything I didn't touch it it was like that when I found it exactly and so having a tool like modify you can do the actual gifts and so on so nothing there and at that point I was like okay well let's look to see if there were any like environmental conditions right because it may not have actually been this software for issue maybe search volume changed around those queries yeah and um look at that nothing there thank you [Music] at some point someone in the room was saying like hey you know googlebot can crawl from different locations and I was like that's true let's just take a look another look at the logs and I noticed at one point the IPS changed oh so what we ended up doing is we did some like reverse lookups and things like that so you can see like okay ah right and it did so suddenly you guys were crawling from Traverse City Michigan and first of all I didn't know where that was and so um we looked at the site itself put in Traverse City and this is specifically on their inventory pages so once you put in Traverse City there's no inventory [Music] so to you guys it was a soft 404 because there's nothing on those pages and so um you know because we we even looked at the faction render I mean I don't know that's that's what that was you call anymore URL inspection tools and GSE yeah no problems because they're crawling from presumably Mountain View or whatever it is and so we're like okay well again no rendering issues here what's going on but once we looked at the site from Traverse City then we saw there was nothing oh so it's like a regional content situation where we just came from somewhere where you don't have content and that's why okay right so it was like a geographical issue that when we crawled from uh where was it some reverse City Michigan First City I've never heard that okay maybe we have a data center there who knows apparently we have a data center uh you didn't have any inventory for that [Music] so how how did you fix that though so we did is we just default to National like showing National inventory um and we were very mindful that it could be considered cloaking so we don't do it just for Google Map we do it for all for everyone yeah and you know I personally don't believe you guys care about cloaking that much anymore because there's so many reasons for different content to show up for different contacts but a client is going to be super scared of that yeah all right boom let's make sure we show that to all users [Music] foreign [Music] it's interesting that you say that we do care about cloaking but cloaking has a very specific definition it's when you mislead the user so as long as you're like we don't exactly know where you are so we're just going to show you the national content or we don't have inventory for your location so we're just going to show you National content on our homepage I think that's fine that if I come to a car comparison website I see car comparisons so not really cloaking what I would be worried about is if I come to a car comparison website so now I see ads for drugs sure that is not great and then that is that is considered cloaking we do care about that but not really like yeah and it's actually quite rare for me to see that in the wild anymore but a lot of people take that definition very literally I know and so a client is like oh is that cloaking we can't do that and I'm like it's a better user experience exactly what are you talking about damn generally like the the user experience should be good and consistent with user expectations and as long as it's that [Music] we had that case with a new side where they're like oh you're not seeing anything from us and it's because of geolocation so they basically just load the local news and when you don't have geolocation available then they just show nothing right which you have to admit is a bad user experience I am like no you don't get my location and then I see a blank page not great right so and they're like oh but isn't it cloaking if I'm like no you're not doing something specifically for for Google you're doing it for users who choose not to give you the locators and that's not cloaking so um but I see where you're going with this and it's interesting that that that is continuously a problem where people are like is this cloaking because our documentation as far as I know says that it's about misleading the user if you broke so right you could argue that maybe misleading the user is a bit vague and we should if we are open we're open for suggestions if you have better suggestions let us know we're happy to to take them feedback is always appreciated [Music] foreign so to recap they came to you with a rise in 404s but software force uh well software for specifically but they didn't change the sides right and they were like it's an algorithm update and you're like no they said is it an algorithm update they weren't sure okay so they weren't sure and you're like no it's not I mean that's almost always my answer like my default is like it's not an algorithm update or client is like well I heard there was an algorithm update what has changed for everybody I'm like I can't tell you that the day after one is announced like because the reality is you guys are a software company I would imagine when things are rolled out there are there may be components that get rolled back in that two week window and it doesn't roll out one or nothing like it's it's in between somewhere exactly and so people just assume like they read an article like oh these are all the things that change there's an algorithm update yesterday and I'm just like okay well what happens if they change it back you know so I typically refuse to answer those questions at least until there's like two to three weeks out of the past and you guys have like kind of settled into what it's gonna be and then we do our reviews [Music] um I don't know I have a lot of issues with how algorithm updates are handled by our community because I feel like a lot of it is like okay well let's look at one of these tools that measures visibility and we saw visibility drop for all these sites all these sites have commonalities here's what must have been the problem that's not necessarily true no and you're also not comparing what those pages look like before and after that's true like there could have been things that they changed exactly so that's why it's it's so difficult for me to like engage in that yeah I I don't engage in that because the the web is such a living breathing organism and I also find the discussions of all other things being equal dot dot right very very tricky because they never are right because all things being equal means you copy the website and that that has its very own problems um so yeah that's uh that's a fun one [Music] and I like that you said that you use the log file analysis because I haven't heard that being mentioned very often previously I know that a few people are doing it and leveraging it but I don't feel like it's talked about enough because I think it's such a useful tool it is but it's so difficult to get that data you know it's like you have to find some person that lives in a back room in devops somewhere who's like which logs do you need it's like quite obviously I need the access logs guys come on like but nevertheless um you know just getting them is difficult and then if you're not using a tool like Spotify or screaming frog log file analyzer or there's a numerous tools out there a lot of you don't know what to do with them yeah or like if you don't know how to like parse them yourself then you're kind of like okay big uh text files yeah so um I think that's the difficulties just kind of like that lack of understanding and also access to them but I find them incredibly valuable especially for sites that are like millions of pages because understanding what Google is crawling and what's not and where they're seeing issues that you know you're not seeing in crawl tools it's one of the most valuable things that you can get and it's it's so interesting that that is so undervalued or so still perceived or still is hard for seos to get into because I know that in the developer Community logging has been an ongoing discussion for a long time and like there's a solution such as the Alex stack and then uh elasticsearch does a lot of vlogging things Splunk um gray log there's like so many different things that are basically sort of settling down to be the standard solutions for logging log4j being an abstraction layer above all of that from the code perspective foreign [Music] why don't more people work together to get the most out of the logs because we we collect all this data about what happens on our systems um as as any company does like you have all these logs and not that many people are actually actively looking into them and if it's for like outage monitoring or something but you can you can drag so much more out of it including performance metrics sometimes absolutely I mean the thing is like I've been using the internet in some form since like 92. and so all those types of things are just native to me like yeah command line native to me so um I'm I'm like before we had JavaScript driven analytics analytics one yeah exactly so you know it's it's really easy for me to look at those and know what they mean but not every SEO has that experience has that background yeah I wonder if there do you know any resources for people who want to learn more about that [Music] I know the built visible guys have a resource on that I'm pretty sure the one Lee guys have one uh we should write one but but yeah those are good places to start also like the Cabana tutorials will tell you a lot about it true yeah kibana see I completely forgot about that one but yeah that exists as well sweet awesome [Music] thank you so so much that was an interesting ride into uh geospecific content and software for us and kind of rendering but not really rendering so that was that was fun I think we're gonna link all the resources for the log file analysis that you mentioned in the description below and I would love to hear from you all if you use some sort of log file analysis and if so what and if not why not and why is it hard for you to get into that all right thank you so much for being here with me today Mike thanks for having me and uh for all of you out there stay safe take care and bye bye all right [Music]