Observability at Snap: Using tools and telemetry data for troubleshooting

Key Takeaways

foreign [Music] good afternoon everybody it's 1pm on a Wednesday afternoon and a beautiful day in San Francisco and you're here to talk about observability at snap using tools and Telemetry data for troubleshooting isn't that exciting you know troubleshooting is not exciting but the fact that you are here is exciting so welcome welcome to Google next 2023 and welcome to this session so first off let's start with a quick introduction my name is afrina and I'm a product manager here at Google Cloud essentially my role is to be the gap between you and the product so over the years of working in Cloud I've spoken to many many customers right I can put customers in three big buckets number one customers that are in the app modernization Journey right here in Tech we call it modernization meaning you're moving from your on-prem to a public cloud or private cloud number two you've moved to cloud or you were born in Cloud not you know you'll figure out how do you sustain and scale number three you've kind of done with number one and number two and you've figured out how to scale you've gone from millions of users and you have a different set of problem to solve but then from a troubleshooting and observability angle you've kind of reached the plateau of productivity so for today's session we have invited one such customer that would fall into the third bucket as far as the agenda goes we'll have snap to talk about how do we start in Google Cloud how did the journey look like and they would also be giving us a sneak peek into how does day-to-day troubleshooting look at snap and then I'll be back on stage and I'll talk about common pitfalls what stops the rest of us from not being able to get there what makes the journey complex and hard and we'll talk about what are we doing here at Google Cloud to make it a lot more easier for you towards the end I'll talk to you about what are we building next and we'll go from there so without any further Ado let's welcome even on the stage thank you thanks hi hello everyone my name is Evan I'm a tech leader from snap I hope you guys snap for almost seven years now for the last few years I have been mostly focused on building our obserability stacks uh for the first section I'll quickly talk about a little bit of history between snap and Google Cloud let's start with a quick poll three questions first one but silverhands who had heard about Snapchat before this talk wow that's not actually it's much more second question who actually used Snapchat maybe a few times was decent third question who may have a kids or young siblings that I also use Snapchat great SD thank you all right oh I guess more thoughts has already been familiar with Snapchat but what you might not know is that snap is actually turning 12 in two weeks so this is very exciting for us it has around 400 million daily activity users quarter and still continue to grow when snap was initially launched this backend was built on top of a Google Cloud app engine it's a fully managed serverless application platform it follows a monolithic like architecture meaning that all the service is running a single backend it has been worked well it's the part of the fast growth of Snapchat from zero user to 100 million detective users it's been amazing but as you can imagine right for a single gigantic cluster and the group speaking of bigger manual problems.certainty like big blast readers is very low cost efficiency and Etc now to solve those problems snap over the last few years has spent a lot of efforts trying to migrate over to a modern Cloud Model region microservices architectures it works quite well now with that there's new set of problems Services because now we have a thousands of services right how can we make sure each of the services are healthy so we need a good observability solution for that now that's my kind of a job comes in as Tech leader for the team I want to make sure we build a platform that I can first improve reliability second improve developer productivity and certainly improve quality efficiency we have five principles I first want user-centric this might not be very upwards for internal tools right but if you have internet very intuitive very easy to use it helps a lot with adoption the second and third principle a kind of no-brainer here because to support Excellence of Engineers and the southern Services you want to make sure your solution is highly scalable and also robust first one cost optimized right you always want to pay a very close attention to your cost because it can easily explode right especially for observability platforms lastly model Cloud given now most of our services are run in across different clouds clouds you always want to match to that so here's a high level architecture diagram maybe let's you know spend a quick two or three seconds take a look so this is the very three former diagram right but there's the Hue there are the few key decisions we made behind the scenes like first of all we query we we provide a unified set of tools for of services across different clouds so that we can provide a consistent troubleshooting experiences second thing is for Medics we have been evolving our Matrix back end to our site offering from chronosphere which supports the standard premises format with the parameter format we are able to create a unified data sources basically for automatics right for Medics from within snap services or metrics from cloud windows and automatics from third-party software's chronosphere also provides a very powerful control plane to help us take control of our quota and cost spending we do also use cloud monitoring for some of our use cases externally logging like Cloud logging has been evolving a lot for the last few years it added tons of new features and also improved their performance we had partnered with Cloud logging we're successfully trying to resolve some issues and we are happy to still use cloud logging as well number one choice for logging uh number four last one all the tools we choose here can work at snap scale just give you some some data here to get a sense here informatics right we ingest like billions of data points per minutes for logs we ingest more than 1.5 petabyte of data each week all right for the next section I'll quickly walk through two examples to Showcase you how we do troubleshooting at Snapchat So Stories is one of our most liked features on Snapchat let's say suddenly the stump chatter is unable to load stories right from the phone as engineer who is on call for this storage team was a typical incident handling life cycle looks like right let's say the uncle's name is Stella instead will first receive alerts and she acknowledged alerts she started to troubleshooting following some drum book she first saw that there is huge increase of 500 paracetic codes for the API that handles story quests this is what I mean right there is back-end internet survivors but what exactly is error right what caused it for that seller continue to investigate she first checked the department and configuration chain histories but I found nothing changed now this becomes more interesting she wants to use logs to find more informations foreign so let's take a look kind of a zoom in look let's see what's what's on Stellar screen right so this is probably working yep on the left side standard first uses the left field to try to narrow down to the logs to specifically to error logs from the storage services there's also one thing may not be very obvious here but logs field Dynamic group of the logs for each of those labels this kind of system labels and so it can't okay whether this error is coming from a specific resource or not let's say for this case it's not let's continue so for next probably is that I want to know okay now one exactly does the incidence that's happening when exactly that's the first error starts showing that's this histogram at the top helps because it shows the very clear trends which are based on the number of errors or number of logs for that period of time so Stella can click on the first part in there and it takes to throw the first set of our logs but sometimes this might still not working because there might be just too many duplicate logs there's just too much noise how do we solve that problem so Stella look at the top of the results and see there is a high dissimilar entrance feature in there if you click on the button trying to preview the suggestions and if it makes sense she just applied it remove that deeply logs and then she maybe do this a few more times until she has some clear signals so with those clear signals right so you may be able to record the issues but I guess really the root clock here is not really that important for this demo but it was really important that how can you go from there what is you know how can you do better if this happens again right you always want to do the follow-ups follow-ups such as maybe just adding a very specific Medics right for this which can better capture these specific issues you know add it to a dashboard setup alarm and also make sure you also update your runbook as sure summary with all those night features from cloud login status was able to resolve Vision much faster right let me show you a another example which are kind of typical for platform owners let's see Sam is from our data processing team she owns the ingestion pipeline which ingesting data from of course a lot of different Services through Google Cloud Pub sub during a weekly review Sam founder that okay suddenly the panel is expect for one of the days and who caused that right so you want to figure out that Sam first check the overall like boundaries traffic Medics he funds like and he can confirm the start in time but that's not enough there is indeed actually a breakdown medics but there's a problem is that that this Medics has around 300 000. now let me quickly post here and take another poll who has again patchable hands who has ever tried to load a Medics with this much cardinality well at least there's a few it's good now what happens right someone's doubling it's like someone's doing this right nothing happens exactly so the thing is that for such cardinality right this probably matters become not very true because a lot of times the graph is very it's maybe taking very long time to load or just not load at all even though it's loaded it may not be ready it becomes super hard to interpret anything from the graphs right there's just too much noises so we have to think differently how do we address this problem right let's try logs this is very easy right you can easily add a One log line and contain the breakdown information it's just done right there but also there is one kind of a stream I want to call out here that you definitely don't want to log online for each of the messages you processed you always want to you know first do some local applications and then you meet the log line in some sort of a reasonable frequencies and let's say you know One log line per part per minute something like that because if you don't do that those executive logs can potentially impact your application performance or you know also uh cost a ton of money as well all right now we have the logs time to do some analysis here it enters the kind of the latest two which is big big query powered from cloud logging it's called log Analytics so it's very easy to enable actually uh with no extra cost under normal conditions so at snap we have this enabled for all of our services to do the analysis right you just need to do build requirements here the query building experience is very much the same like bigquery if you have a user before you can essentially just you know keep iterating on it and you get the results you want and for every time you can also try to you know show the result in a table format for here at the bottom here uh one helpful tips here is that you may want to try to use one of these black large language model based and tools to help you build queries if you are not a SQL expert it actually works another nice thing here is that for those aggregated data you can actually source as a graph in here like we have been we just recently started using this uh it's been amazing it has a similar dashboard experiences like uh from cloud monitoring highly recommend to check it out as another source summary here is like with those powerful tools right from cloud logging we can now actually solve some complex problems in one place you can turn these unstructed data into structured data and then into graphs which is much easier to visualize that concludes my two examples hopefully it's useful for you with that I'm gonna hand it over to afrina to talk about some of the common troubleshooting pitifuls Ena thank you [Applause] thank you so much you know what really stood out with SNAP is it was an amazing presentation where they talked about the guiding principles the architecture and how they do troubleshooting right they're very agile they're super fast and and a lot of us have been trying to essentially do the same thing but we don't realize the pitfalls that we might fall into let's talk a little bit about the common pitfalls that we see in troubleshooting observability is essential for Effective troubleshooting I mean it goes without telling but if you're closing a large number of tickets with root cause unknown as a resolution that you know that your observability is not correct like you're not collecting the right signal you're not bringing them to the right place so let's talk about it a little bit more what are the common pitfalls the number one and essentially the biggest fragmented solution this means is if you're trying to DIY a lot of things or you have too many solutions in your ecosystem you have one underlying let's say for example you have one underlying logs database right your devops team is using it to resolve issues in the local computer you're exporting that logs and your security team is Now using another good looking tool with different visualization to analyze the same logs and now your business analyst team is taking the same logs and Performing and creating different dashboards you see it's the same underneath data why do we need so many applications right the overhead of managing applications the avoid of trying to scale which is number two here and then trying to figure out how do you manage cost which is number four like Ops is not cheap it is expensive and if you do not know how to control it the cost can get skyrocketing if I can throw a coin into my pocket for every time a customer tells me that I do not know how to figure out my cost I'll I'll be a multi-millionaire by now I won't have to stand here itself fragmented solution will lead to number two number three and number four right right you cannot figure out how to scale that's going to be a bottleneck then lack of cohesive experience if you notice when even was showing how troubleshooting looks and snap they were able to go flawlessly from metrics into logs so when you're thinking about troubleshooting it's essential that you that you have signals that talk to each other you should be able to go from metrics to your logs metrics to your traces and then preserve that end context experience and if you don't have that you're actually making it difficult for your engineers to work on troubleshooting it's going to be very challenging so to summarize the common pitfalls that we see fragmented solution complexity to scale lack of cohesive experience and skyrocketing costs so another thing to add here in cost is we cannot tell you how many petabytes of logs you're going to generate right but what we can tell you is if you generate this much this is likely the amount that you're going to be paying predictability is important and across industry we don't have a standardized way to charge logs metrics or traces we all have so many different ways and I could probably write a book on how to figure out costs right it's that complex our recommendation is for you to pick a suite of tools essentially in one ecosystem and when you're doing that think about this right number one operate at scale right Google went from zero to billion users on the same platform snap just went from zero to 400 million users on the same platform so Ops should never be a bottleneck when you're trying to grow your business you should know that Ops should be able to scale right number two ability to meet your reliability Target like if you have a well-established principle error budget you know your SLO then you should be able to meet your reliability targets number four is provide the versatility what this means is you essentially have the same signals across your organization so you should be able to bring in users to use that right it's just not your devops team your secops team your Biz Ops teams so many different users should be able to make the best out of the same underlying data set and that would save you considerably on cost and overhead and you'd be able to reduce your toil number four remove fragmentation we just spoke about it number five is cost effective in Google Cloud Google Cloud's Cloud operation Suite is our observability solution and what this does is we have several tools within the ecosystem to collect all your signals metrics locks trays audit logs from both gcp services and outside of gcp so what does it mean you have on-prem hybrid Cloud structured logs unstructured logs metrics from other ecosystems you can collect all of it and bring it into Google Cloud and our goal is to give you the same experience regardless of where your data was born if your data was born in gcp yes you'd have a higher experience but if your data was not born in gcp you still need to have the same experience So within Google Cloud operation Suite we have Cloud monitoring that collects time series metrics for 60 plus gcp Services it would allow you to measure SLO you could create in context dashboards across then we have Cloud logging which is the home for logs you can centrally collect your organization's logs route them to flexible locations where you want perform analysis we saw several examples which even just shared with us then we also have Cloud trace alerting and much more so let's talk about troubleshooting in Google Cloud what I want to do in this section is essentially highlight some of the features that will help you in your troubleshooting Journey a typical troubleshooting journey is going to look like this right you're getting your alerts and notification then you want to go explore what's happening in your system and then you're saying okay can I get to my root cause then you get to remediation where at some point you get to auto remediation you don't want you want to reduce your toil so just mapping that to the product development about how we think here at Google it's we want to help you do four important things number one collect collect signals from everywhere because observability and troubleshooting is a lot more faster thing to do if you have a single pane of glass number two configure that's what monitoring is for you need to know what's happening in your ecosystem number three detect so when we are thinking about detecting of course you can configure certain things but hey life always throws up with random random new things right you cannot always predict things that's going to happen so we think about okay if you can detect the nouns what can we do to help you detect the unknowns then lastly troubleshoot and make it easier for you to get to the root cause now what I'm going to do is essentially group two sections into one so I'm going to first talk about features that you could use for collect and configure then I'll talk about features that you can use for detecting and troubleshooting all right so Step One is collect bringing data into Google Cloud so if your data is already in Google Cloud but if it's not the step the first step in troubleshooting is centralizing your data collection right so Step One is Ops agent so if we have to talk about Ops agent I'll just put it in two categories it's automatic comes pre-installed if you're in gke or Cloud run you do not have to do anything it's day Zero observability if you have GCE VMS your virtual machines then you would have to install your Ops agents it's our opinionated way to collect metrics logs and traces number two and number three would help you if you want to collect signals from your multi-cloud or on-prem environment number number two is our open Telemetry collector we support that and we and we enable you to use an open Telemetry collector number three is bind plane observability this is our third party vendor and they can help you remotely manage all your configurations once you've collected all your signals this is a number one recommendation when it comes to logs we want to I mean we recommend that you collect your logs in a single bucket sorry like centrally stops okay so we want you to centrally collect your logs in one single bucket and you can applied guard rails by managing access using something called log views see you so when you do this you're essentially creating that single pane of glass that's going to make troubleshooting a lot more easier and this is exactly what snap just shared with us in the architecture right they were able to troubleshoot issues faster because all their logs as in one centralized location this is going to make it easy let's say you work for a financial services your security team is always going to ask you hey are you collecting all the logs are all your logs in one place they're going to be always behind you and centrally storing your logs is essentially the key to it that you could configure something called an what uptime check does is if you have a business critical application and before your users report that hey some an application is down we can configure it we can ping it periodically to see if its responds to our HTTP https or TCP request and let you know that hey your request is probably done somewhere that configure alerts and notification of course we all know about alerts but I just wanted to highlight new features that you could add to your troubleshooting or start so you could forecast an alert what this means is let's say your CPU utilization is climbing up and before some things become an actual fire drill you want to get notified two or three days in advance you could configure something called forecast alerts and you will get notified so you know there's a difference between a page and a ticket this is where you can create a ticket then you could configure a log Space Alert again there are two ways you can alert right you could alert on a log or you could alert on a metric so when would you use a log base alert let's say a very very high priority security incident happened and you want to know immediately right if something happens once and if something shows up once in your logs database and you want to get alerted on it logspace alert is the best way to do it of course we have log space metric we have been using this for many years metric absence are secops friends love this because if you're not collecting metrics from some place how would you know like if you configure an alert for metric absence you would know that there's something as broken in your pipeline and you should go take a look at it and Google Cloud mobile is another easier application to have in your phone and if you have a log Space Alert the fastest way to find out if is by getting a notification on your phone and I believe about 29 30 of our users start acknowledging and troubleshooting from their phone because they have access to the console next through our gcp you should be able to create an end context dashboard this creates a single pane of glass view that we were talking about right when you create an end context dashboard you could pull in your metrics logs and traces all in one view for your team to look at now they're not scrambling they're not looking into 20 different Windows to go find the same information everything is available in context in single pair cloudtrace when troubleshooting gaining sufficient proof to support your hypothesis is critical Cloud Trace new UI helps visualizing your request flows from distributed traces it includes a more responsive and interactive detail section next let's talk about detect and troubleshoot so error reporting is another amazing amazing tool and it comes in free with Cloud logging so what error reporting does is it will keep combing through your logs and try to look for errors in your logs database and when it sees an error it would automatically send you a notification let's say here I've configured a slack channel so you get an automatic notification next when you click on that notification that you got in your slack Channel you're not going into the error reporting page what I love about this it consolidates gives you that single pane of blast for everything that you need to know with respect to one error right so here you can see the description of the service a clear description of the error a histogram and then it also tells you the root cause of where it found the trace and the logs from here you can go into logs too now we recently integrated duet AI a generative AI capability into error reporting so next time you don't know what to troubleshoot where do I go from next that's often the biggest problem in troubleshooting Journey you were like hey what can I do to troubleshoot so it will give you three to five ways for you to go explore furthermore now from error reporting you can go into logs Explorer by clicking on view logs now what this does is it gives you a starting point to go explore your log furthermore you don't have to reduce noise you don't have to figure out filter all that right so error reporting gives you a starting point to explore your log and even just showed us all the amazing features that he uses in logs Explorer to make troubleshooting easier my idea of fun is coming through logs says no one ever somebody going to disagree okay I get paid to love logs per living and I don't do that myself so we recently integrated duet AI in logs Explorer so this has been a huge Advantage like if you go in and say can you explain this log entry to me in simple language natural language terms it will tell you what happened this is amazing I don't have to strain through combing through logs and try to understand and summarize myself versus duet AI can do this for you okay this part was deleted in this service at this time in this cluster isn't that everything that you need to know at that point right so now logs Explorer can help you troubleshoot when your cardinality is not so high versus if your cardinality is high and you want to bring in more users into your ecosystem then log analytics comes in play log analytics okay so to explain log analytics in very simple words you send us your log data unstructured semi-structured whatever format you have and we in the back end will figure out how to store it in a SQL database you don't have to create an ETL pipeline you don't have to figure out how to store it we will do it for you in the back end then it's schema on read so there is no pre-processing your team is not spending hours and hours trying to figure out how to build a database you don't have to DIY we will do it in the back end and put scheme on read and it allows you to use C Corp right SQL is I guess the third most popular language in the whole world and generative AI can help you write SQL queries so schema and read and SQL and with instant charting and dashboarding capabilities bigquery lets you come to petabytes of data in seconds right an excellent use case of this would be say for example you want to find out devops Trend help me understand my API performance by looking at the top count of request Group by response type and severity it would take you three to five steps to do this in another tool with SQL it's just one query and you're able to get to root cause this faster likewise if you have sec Ops help me find all the audit logs associated with a specific user over past month when you generate your compliance report this is just one query that you need to run and you have your answers instantly with networking troubleshooting help me troubleshoot network issues for GK instance using VP VPC flows and firewall okay SQL makes it easy it brings in more users into the ecosystem if you think from a long-term perspective it saves you on costs it makes it easy it reduces a steep learning curve that you usually get with any observability solution we also have something called Community security analytics it's a GitHub repository let's say you want to get answers to all these questions and you're already already ingesting a ton of security security um logs you could go into SQL just copy paste the SQL query from there and execute it it makes life so much easier no log analytics is the first step I want to talk to you about what do we think is going to come next and Ops and what are we building here in Google Cloud our vision is to bring in logs metrics traces metadata and a managed data Lake and enable you to create use cases such as log analytics which is the step one that we just saw and build more on it such as gcp utilization phenops and much more an example of how the data Lake would look like say for example you could build a report like this which pretty much combines a metrics from your Cloud monitoring and you're building information from cloud billing and generates insights into cost and utilization when knobs has been a problem for many many years this data Lake and bringing all your signals into one ecosystem is going to make troubleshooting a lot more easier that being said you know it's truly an exciting time to be in cloud and a furthermore exciting time to be in apps right with building a data Lake and having generative AI on top of it troubleshooting is going to get a lot more easier and faster so for the final wrap-up I'd like to invite Evan for this stage if we have to give you three suggestions advice or recommendation I'd say number one is think about scale right Google was able to go from zero to billion users on the same platform snap was able to go from zero to 400 million users on the same platform and Ops was never a bottleneck we never stopped the business growth we enabled the business to grow faster so when you're thinking about Ops decisions think about okay will this stop us from scaling the only time you shouldn't be thinking about scale is if you don't want your business to grow or you're going to deprecate your solution I hope that's not the case right number two remove fragmentation if you're going to add a step in your Ops Journey think about what would it take for me to do it 10 times more or 20 times more if you feel like it's complex then you're adding a fragmentation layer for your Absolution so removing fragmentation will greatly greatly help you with observability and troubleshooting and lastly partnership goes a long way like Google has been helping snap you know help with their operation system for a very long time at snap has been helping Google develop a product life cycle for the op solution so with Partnerships we can go a very long way that being said thank you so much for being here today and we truly appreciate if you can take a quick minute to give your feedback and the learning does not have to end here these are the other sessions where you can learn much more about observability Solutions if they're not live it'll be recorded on YouTube and I'll be stopping the booth at the innovators Hive down if you have any questions I'm happy to take it there being said thank you so much for being here this afternoon truly appreciate it thank you foreign