How Lightstep is Illuminating the Case for Distributed Tracing

Key Takeaways

we'd like to thank the cloud data computing foundation for sponsoring our podcast from cloud nadir Khan in Seattle inspired by internet scale computing the cloud native computing foundation advances the development of cloud native technology and services by creating a new set of common container technologies informed by technical merit and then user value you can learn more about the cloud native computing foundation at CN CF I oh [Music] hey it's Alex Williams the new stack here AG Kubik on cloud native Day in Seattle Washington and we're just getting started with the conference and this place is packed I mean it is like wall-to-wall people and I'm here now at Ben's single minute of light step and great to have you here thanks it's great to be here this is a quite an event yeah it is uh it is packed it's really packed I actually was talking to some that I kind of like that it's a little over subscribe to makes it feel more energetic you know probably a few less people then you have your meetups right yeah usually we have you know we try to get those in piano stick part or that Stadium show or something but occasionally we're we have to go for a ballroom or something so so Ben I was just talking with you a little bit about your own background and and I really a theme for our discussion here I like to talk about open tracing and how that relates to kubernetes and your own company light step so first of all tell us a little bit about your background and like you know you worked at Google for 2003 to 2012 and you know and tell us about your kind of your your history with tracing and what is open tracing sure so yeah I was a cool for a while more than I might recommend to those listeners out there I was traveling around the country and and just couldn't couldn't you know kick the habit of staying there but but for the first couple of years I was just getting my feet wet I was you know frankly very young and I eventually realized there is a ton of pain at the company around visibility into distributed systems which at the time were quite similar to what you see deployed now in these containerized environments lots of small services with very narrow focused charters and it was that major the huge pain points that you would get to when you were developing like any distributed infrastructure back in those days yeah I mean the biggest one then is the same one I think it's the biggest one now in micro service architectures and it's the one that people I think are frankly embarrassed to talk about which is that they just literally do not know what's happening at a very basic level and transactions like what's happening like like IME events so it's it's fusible to say okay let's look at how many requests this processes processing per second or whatever that's no problem but if you want to actually tell the story of how an actual transaction goes from use an end user on a mobile or web client through a distributed system and back again to that user that is really really difficult to do using off-the-shelf time series monitoring and logging creation so difficult that it just doesn't happen so most the company is that I'm aware of that have expressed interest in tracing which is frankly a lot of companies at this point they're trying to address that basic need that they literally do not know what's happening in their own systems which is a profound problem and and one that is almost inevitable if you try and apply single process oriented tools to distributed systems with with you know heavily decoupled micro services so that's that's the main thing we were dealing with we like tot we wrote a paper a dapper which is a system I built at Google that emphasized a bunch of really interesting critical path analysis and aggregate analysis of traces and you know trying to come up with equivalence classes of distributed call graphs and bla bla bla bla bla and that was definitely real and was valuable but actually the most most valuable thing was you look at a trace that was something that took you long you'd say oh gosh it's doing these ten operations in serial which don't depend on each other they should be done in parallel which is an extremely easy thing to see if you actually have the data if you don't it's impossible and so that was the the fundamental thing that most people needed and frankly the fundamental thing most people need now is just basic basic coverage of a distributed system now that those problems would manifest themselves in what kinds of ways with that manifested an end user issue would it manifest itself in terms of just inefficiencies you know with you know with you on the distributed you know environment itself the answer is yes yeah and and many other things as well we found that it was also quite challenging to contextualize failures at the lowest level of infrastructure you often see in modern systems you'll see these diamond-shaped graphs where a front end will call some you know middle tier service which then calls some database layer or Cassandra or something like that where you have many different consumers of this common thing at the bottom of the stack and the bottom thing that the stack will have some error and you have no idea where that request actually came from or what contextualizes it and so that's a really pernicious thing often times that the where errors are most important to you at the sort of workhorse layers databases that sort of thing it's difficult to contextualize things without some sense of where the data came from and so that that was in addition to the bread and butter of user facing latency and just general you know system awareness those are the things that that we would often see people so you so you did all dapper at Google yeah and you've now are you now you're behind the open tracing diagnostic tool what's the difference between those two yeah well I think the biggest one is that well maybe the biggest ones that dapper was never open sourced or anything like that Deborah's proprietary technology developed inside Google that we happen to write a white paper about Brian the closest parallel in the open-source world is certainly Zipkin which was I've designed very carefully to model the the description of the system we had in the paper and so and that was open sourced by Twitter and is now maintained by a very healthy open-source project run by Adrian Cole is at pivotal now and then fried chicken yes so Zipkin and dapper are you know kissing cousins in terms of their design open tracing is in some ways a different type of project open tracing in Zipkin are actually completely compatible and I would say complementary open tracing is only trying to address the abstraction of instrumenting source code my experience at Google developing dapper was was quite different than my experience working on similar technology in the you know in the larger ecosystem Google had a monolithic source repository still has a monolithic source repository so if you made instrumentation changes to Google's RPC system and the Google's control flow system you were able to get coverage of their entire distributed system outside of Google it's not like that at all if you add instrumentation to I don't know pick your favorite open-source project all you've done isn't spend that one project you haven't instrumented the rest of someone's stack and so it's this incredible death by a thousand cuts you're trying to get tracing deployed at a modern organization that's using many different frameworks from many different open-source projects and and so on and so forth so that the point of open tracing was to come up with a lingua franca for describing the the flow of control throughout distributed systems this is actually the pain point for deploying distributed tracing it's it's not standing up a tracing system it's not the UI is it's not it's not any of that stuff it's just getting instrumentation established and open tracing is a standard that aids in the process of building the instrumentation so you can't run open tracing it's not end of itself a system it's a vendor neutral API that plugs in the Zipkin it plugs into Jaeger which uber just announced last week which is they're tracing system it plugs into a number of commercial solutions and a number of smaller open-source tracing systems as well but it's it's a it you can think of it more like you think of HTTP or something like that where it's a it's a common standard that you can program against and get interoperability for this type of observability system and that's really the heart of it is like that standard is a yes and constituencies that care about it are certainly commercial entities that have big distributed systems they have this pain point of instrumentation tracing systems like sip can or Jaeger or what have you that want to have as much coverage instrumentation layers they can get and by supporting open tracing they can get a lot just in one fell swoop and the maintainer is open source libraries like the maintainer of G RPC or the maintainer zuv drop wizard or whatever micro service framework you're interested in those sorts of project owners are interested in open tracing because it allows them to interoperate with every supported tracing system with one piece of with one commit essentially by instrumenting one time with open tracing and so it's it's a standard that benefits those three constituencies so open tracing has been adopted by the cloud native computing foundation and what is its goal what is its role in that context well I think that I mean to a certain extent you should probably ask the cloud native folks but my understanding from being on the TOC calls and so on I think the excitement they had about open tracing was really that they aim and all the keynotes this morning really underlined this I think they aim for vendor neutrality and portability and extensibility and those are three things that open tracing gives you it's also operating in a space that's very very tightly coupled to frankly the the downside of moving to cloud native if you're using a monolithic approach to building your application you don't have this problem not describing but as soon as you move to using kubernetes in earnest or similar you know orchestration systems in earnest you are blindsided by this disability problem and so it it's complimentary and that it it helps people move more quickly into that sort of deployment model and and of course in as much as open tracing needs users open tracing benefits from that same migration so it's a mutualistic relationship between open tracing the other things that CN CF is onboarding into into their into their foundation so tell us about light step and that's the company you found it yeah so I'm happy to talk about it light step is well it supports I've been tracing it's in many ways something that is informed by our experiences that much of the court team came from Google it's informed by our experiences building systems like dapper and and other monitoring systems and the pain points that we were addressing at Google I think that we've attempted to do our best to to not over fit to Google's experience things like that monolithic source repository at Google are really not representative of the larger ecosystem and one of the things that is troublesome for me about most racing systems is that they actually often directly cite dapper as an influencer honestly dapper is was not the right thing for the rest of the world it was debatably not the right thing for google and it was built in 2005 and and so I think with light step were attempting to build something that we think addresses the same type of pain point but in a way that's more representative of modern stacks the way that modern companies prioritize things and so on now I'm looking at the story that I wrote about lights happening here quoted in it and you're saying you know and I'm thinking about you know where does light step go where you know what is this direction and you'd say I'd like to see a story for open tracing in that environment which which will require standardization beyond just a level of api's but involve some standardization about the way the tracing data looks on the wire and it's formatted on the wire I'm eager to pursue conversations of people from those communities what is it about the needing to go beyond the level of the API is so fundamental that's actually quite relevant to this conference actually in kubernetes and so on I think one of the one of the dreams evarin has they're trying to play something like this is to make no source code modification at all and to just integrate via something like you know something that runs within a kubernetes pod or they have this concept of a daemon set which allows you to have things like fluent D or link or D or Envoy or something on those lines running on on on every VM that has pots installed and so on if it's possible to integrate open tracing at that layer you can deploy distributed tracing without source code modification which is a really important thing especially when you want to support software in environments where recompiles and redeploys are not happening every day I think a lot of the bleeding-edge adopters do actually have a continuous deployment model where you know most the software they run production is actually written in-house and so on but if you move into more established enterprise environments much of the software that they run is is is from vendors and and they you know read apply it infrequently and recompile it never and if you want to operate in that environment it's necessary to have some kind of shim that comes in at the container VM level and open tracing to date has focused on the API which has been the the most urgent priority but as open tracing gets pushed up market which I think we're starting to have those conversations with people and an inbound basis I think there's a lot of desire to have a standard the standardization of wire formats that would allow something that intercepts HTTP to automatically trace an arbitrary applications so it's it's a these sorts of wire format [Music] standards would would allow a large enterprise to adopt tracing without tying themselves to a specific vendor or to a specific even tracing methodology by integrating via a container proxies and so on it seems like what we're also moving to is when you're talking about the wire is just being able to understand the data a lot better right yep absolutely being able to almost kind of form develop that metadata so you can do the analyses you know that would allow you to move beyond than the so yeah that's exactly right and I mean I would underline that by saying the the technology that runs within the application itself that we developed for dapper its largest commercial application at Google and I hope I'm allowed to say this but I'm I don't know it's been many years since I weren't there is probably actually not having anything to do with monitoring in the traditional sense of reliability it's that code path is used to to pass contextual information about about the the top-level owner of of a request whether that be Gmail or calendar or web search or spreadsheets or whatever Google product you're using that information is passed literally fifteen to twenty-five layers of distributed infrastructure down until it gets down to a desk or something like that and the individual disk rights are apportioned by that user gmail web search calendar etc and that apportioning allows those underlying storage systems to do real-time push back and throttling of global quotas for these for these various groups than Google where Google can manage this multibillion-dollar spend on that sort of technology with very fine granularity despite the fact that those lowest layers are totally shared infrastructures they have no wastes by over provisioning every individual user they have one big pool and it's it's you know throttled at an extremely fine grain layer a via a feedback loop that's powered by this sort of tracing technology not for latency analysis but actually reusing that code path to do propagation of information and open tracing supports that via this mechanism called baggage which is the idea that there's data that flows along with the request transparently and this is like a really power well I think a paradigm shift in the sense where you have data that's transparently flowing through an application and being used to tie you know entity you know keep all the information from the top of the sack to semantics and the bottom of the stack and that's a really powerful shift I think that we're gonna see a lot more of as people move to to more shared infrastructures so kubernetes is still very nascent and then opened raising is very amazing yes water what are the you know the projects in the immediate future that you're going to be working on kind of in that context well I think we already talked about one of them and that I'm having conversations the number of people about developing better standards for interoperability you know serve ability tools including tracing tools in a containerized environment that's a big piece of it we're also working on a number of integrations with technologies that that work well in that type of environment like G RPC which I mentioned earlier yeah in order to get benefit beyond the exactly yeah which is an G RPC and open tracing already integrate well and I think we're we're looking to proliferate that sort of thing into similar projects and similar technologies for our listeners out there who may not be familiar GRP see what you see yeah yeah ER PC is and is an open sourced RPC framework so an RPC framework is is something that allows a process to make a remote method call it's not dissimilar from you know things like thrift or Avro or a number of other projects that have been spun out in the last decade I think G RPC has a ton of momentum if you just look at commit logs and and github stars and so on it's just on this incredible trajectory it's also Google is migrating all this internal RPC is they make 10 billion RPGs a second right now they're migrating all that to G RPC over the course of the next couple of years as well so it's an important project the benefit of it is is mainly not around actually RPC it's around tooling so it allows you to do things like service discovery client-side load balancing tracing monitoring in a way that's completely transparent to the user and that's that uniformity as an API layer 2 plug-in is what makes RPC systems so appealing it's not actually the RPC that part's actually pretty easy you just make an HTTP call but it's all the tool you can build around in integrations so you know to end the interview what's next for light step for a light step well we don't really talk a lot about light step so far it's not so much that were stealth I think it's just that we've frankly been somewhat surprised and overwhelmed with the amount of inbound interest but I think we'll have some success stories to talk about in the next couple of quarters and and we'll be GA next year sometime which will be exciting and I think our our goal is to is is to change the way people think about monitoring of distributed systems and we'll see how far we get with that great well congratulations on starting the company and and you know and getting the open tracing project is getting a lot of interest we see a lot of interested on RSI at the news tag so so let's keep in touch yeah thank you thank you yeah we'd like to thank the cloud data computing foundation for sponsoring our podcast from cloud nado Khan in Seattle inspired by internet scale computing the cloud native computing foundation advances the development of cloud native technology and services by creating a new set of common container technologies and formed by technical merit and then user value you can learn more about the cloud native computing foundation at CN CF al [Music]