Auth0 MongoDB Gets Overloaded Causes an Major Outage

On April 20th, 2021, Auth0 US environments had a full-service disruption. The outage was due purely to performance degradation on their internal backend systems. Auth0 has released a root cause analysis report and In this video, I will summarize their backend architecture and the cause of the outage. Let us get started Just an FYI I’m only going through the backend pieces that auth0 explained in their RCA which isn’t their entire backend infrastructure. The Auth0 feature flag service returns configuration values for tenants and subscribers which is stored in MongoDB. (Although Auth0 doesn’t mention this on the RCA we know their core database MongoDB from a previous blog that I’ll reference below ) The feature flag service also has a caching layer which the frontend apis can query for performance. If cache doesn’t have the value or takes too long to answer, the frontend api queries the flag feature directly which hits the MongoDB directly. Auth0 also execute multiple queries running on the backend for routine maintenance. These queries where identified as inefficient and consume lots of resources primarily because they didn’t have proper indexes. As a result of a sudden increase in traffic, the flag service caching layer was saturated which caused latency in response time, eventually causing the frontend apis to massively query the feature flag service directly putting even more load on the MongoDB. Combined with the backend inefficient queries, MongoDB received even more queries and consumed more IO resources which caused latency to increase even further The auto-scaling triggered in the feature service as a result of latency in response time to scale the service adding even more load to MongoDB this eventually overloaded the database until it exceeded the available disk I/O which caused the full outage. If you love this content I think you will enjoy this video where I discuss the Slack outage, I learned so much about Slack backend architecture f