This is why Salesforce services went down

Key Takeaways

on may 11 2021 salesforce services went down for five hours and uh the cause of this outage was as always dns we've seen this many many times how dns essentially affects your entire infrastructure and can you down but the question you might have is who's saying if venus goes down and you bring it back up does it really take five hours to for the services to pick app back up again that's the question i try to answer in this video so in this video i'm gonna go through the executive summary that uh the salesforce team have provided how did what exactly services went down and the cause obviously the dance and then i'm gonna give my opinion about why does it take so long when dns goes down how about jump into it all right guys how about we read the executive summary of why and what happened with salesforce back on may 11 2021 and uh then we're gonna discuss exactly how the dns essentially affect your services right all right on may 11 2021 at approximately nine o'clock the evening universal coordinated time utc the salesforce technology team became aware of service disruption across salesforce production instances the disruption impacted the ability for users to log into their salesforce environment within the core salesforce services marketing cloud commerce cloud government cloud experience cloud ruku pardot i don't know what part it is uh velocity in addition the status to salesforce.com trust site was also unavailable we've seen this many many times guys every time there's an outage the means to inform the customers about an outage is out is in outage too so customers have no idea that what really happened they just seen an outage and you can't really as as a company you can even inform them because the status website relies on a piece of infrastructure that goes down with the outage we've seen this many many times microsoft google amazon now salesforce every time so that's right now i think people realize it and companies realize that they need to separate the status infrastructure from the actual core infrastructure this is very very important right because you need to let your users know i mean the easiest ways to go off let's try to everybody i don't know how that feels really if i'm microsoft and i tell people about an outage in twitter i don't know about that it's like it's like everybody goes to twitter when things goes uh the outage and that that might be okay but i don't know how i feel about that let me know what you think guys trust site was also unavailable and customers were unable to log support cases so even they want to log support cases they couldn't some customers may have also experienced issues with multi-factor authentication during the incident as as we became aware of the disruptions and investigation began across the sales force technology organizers and resolve the disruption the technology team confirmed the impact of the production host which was impairing diagnostics and analytics as he was high confidence that the issue was triggered by an internal emergency domain name system change so the salesforce team made a change to the dns right that's where we know that triggered out but how let's read through the technical details a little bit i just need to know what exactly that and then we can talk about it we can we can kind of puzzle things together technical details the salesforce change the salesforce changed to the dns impacted multiple data centers affecting the sales for service for customers the status.for salesforce.com and salesforce authenticator i think this is the app that does the multi-factor authentication over the following hours the team worked to restore the service blah blah blah we're going through that all this jazz took took five hours to do exactly i'm interested in more a little bit when we do an update to a dns service like you fix the bug or you improve introduce a feature in the dns right what really happened let's read more root cause analysis the preliminary post-incident analysis by the source sales force technology team determined that a dns update triggered dns services to stop and not restart immediately there you go because when you do an update like i mean update i'm going to restart the service immediately but it it stopped but did not restart immediately that means it was only out for few a moment i don't know how long they didn't mention but that moment caused all this cascading effect so let's speculate like we always do in this channel and let's go back and because i think we're ready enough so let's go back all right guys so obviously kudos we start always with coolest all the engineers that work in salesforce and work very hard to bring the service back up and running i always say that because i don't want to be in their shoes because i was once upon a time a few few years back and that's not fun right getting cold midnight things go wrong you stay over hours just to fix the outage not fun so kudos good job guys now let's analyze why dns going out for a few like that's what we that's what we read we read it's like it's only a few moments right it didn't understand immediately that's what i can deduce from that statement right so we made an update to the service obviously to update an update you stop the service to take effect right unless you're using elixir or erlang so you can do hot swapping in the code that's what whatsapp does right they can change process while it's running that's just fantastic man right but uh the rest of us have to stop the process and restart it or do a rolling restart to pick up the change that's what we do all the time right but when we restart the service and it didn't restart immediately what will happen i mean yeah dns goes down for a few moments like what happens well in a micro services architecture or you don't have to call it micro services architect just a service oriented architecture where services are relying on each other to talk and you have a load balancing in place that tag these services with the health checks right if you try if a service tries to communicate to another service and it couldn't in this particular case because of a dns not available you queried the dns is not available and it says as a result the service will think that hey the other server is actually down so i'm going to try again and again obviously i can't find an ip address because maybe there's a service discovery that is not telling me what this ip address is and then you try again you try again and at the end of the day the service will say that hey by the way that the server is just down and the load balancers as a result will have the same effect if you make a request to a load balancer and load balancers tries to communicate to the back end service it will try multiple times if it's down if it couldn't communicate no matter what what error it gets which we talked about in another video we load balancers and proxies are nefarious for throwing this generic error that's called bad gateway i talked about it right here if you want to learn more about it but bad gateway means anything can go wrong from dns from tls handshake couldn't be established from a bad certificate on the back end of from a service taking too long to respond from a service actually giving you an invalid response like you're connecting connecting to the back end but giving you a malform response all of this is bad gateway and as a result bad gateway errors result into the back end being marked as unhealthy so future requests will never go to that so you will try another service that load balancer will right another seven and obviously it's going to be down because the dns is down well the load balancer is not smart enough i guess in this particular case to know that oh it's just the dns it's not really the back end you cannot possibly figure that out okay and and and the proxy proxies and the services are not smart enough to to look behind it obviously right because if you if you can't look up dns all bets are off right so for that period of time let's say one minute took one minute or two minutes all of these services are starting to just go down i don't by any when i mean go down they are marked as down marked as bad so services won't get any requests and as a result authentication will fail everything will fail and even if your dns is back up and running hmm it won't help you because guess what the load manager receiver request yeah it will it will start connecting to one service right again or because all of them are bad so maybe the auto scaling will kick in and say we'll spin up a new services and those right will pick up the new dns update and you all of a sudden there are hungry requests flood of waiting cued up requests just hitting that poor new service and obviously it's gonna go down and and we've seen this many many times when an outage happens right once a service goes up and that immediately goes down because the service cannot handle that potential law that's why what i like about envoy in particular as a proxy is there there's a feature called i talked about envoy guys check it out here invoice they thought about this they said okay we need to introduce panic mode panic mode means if all my back-ends are suspiciously unhealthy really i really don't care i'm just gonna forward the request anyway if those load balancers continue to request and forward the request to the back end right despite this dns that at least try then it would have maybe succeeded it will not have been removed from the backend fleet as a healthy as an unhealthy service right that's one of the advantages of one feature that can be easily added to all products and i believe all proxies should have this feature at least disabled by default give me an option to enable it panic mode is really really powerful again guys all of this here is a speculation we don't know the actual root cause analysis they are still working on it against kudos for the sales force team but that's why it takes time when dns goes down microsoft we've seen it like a few few weeks back i think microsoft went down because of dns once the dns has gone tough luck unless you have great resilience software it's gonna really be hard to bring it back up all right guys that's it for me today i'm gonna see you in the next one if you like this content subscribe i talk about back and stuff that's what i do and i'll see you in the next one guys stay awesome goodbye