How FluentD Can Help Monitor Microservice Architectures Through Unified Logging

Key Takeaways

[Music] Thank You Cisco for sponsoring our day of podcasting at KU bukan we had some great conversations thanks again to Cisco you can learn more about Cisco and their micro-services platform at mantel dot IO that's ma n TL dot IO hello this is job Jackson from the new stack and we're here another episode of the new stack makers podcast and today we have a couple of originators of the fluent D project and which was just recently adopted by the cloud native computing foundation as the 4th project so we want to learn a bit more about fluid D we want to learn a bit more about treasure data the company that's currently I guess the steward of fluent date and we have well first of all my co-host Lee Calculon you will be asking the hard technical questions and we're joined by Kaz and Eduardo first of all let me let me start don't both your full names and your titles at Treasure data hi I'm Kazuki Oda I'm a CTO and co-founder treasurer data my name is Alana engineer at receive data all right good to have you guys and I think well first thing I first came across fluent D I kept hearing about it but I had a interview with the one of the people behind the Microsoft operations management suite and they said always flatten drag maybe yeah yes yeah and I was like well that's surprising that Microsoft is looking outside of there right boundary says must be a good technology and then you know we found that she actually being used in quite a number of different places so let's first how did it how did fluid D first come about but what problem was it uh attempting to solve yeah so five or six years ago we were looking at the data industry landscape and media and everyone is excited about the boja tube and big data and all these like cool technologies but they focused solely on how to store and then process the data but what we have realized is so my background is I organize the Hadoop community in Tokyo where I was born and seven years ago eight years ago it was three people fight people Nord community but now we have more about more than 2000 people so I talked with a lot of data guys and one of the biggest pain point they had was they actually said 70% of any big data or data warehouse or data related project it spent for how to move the data or collect the data from various data sources to one single place so once the data is stored into either let's say like Hadoop elasticsearch a MongoDB it's relatively easier to handle the data but collecting the data at scale reliably and easily it's really really hard so that's why we started this project called full and D so there are technologies to address that yeah but for various reasons yeah so I think at that time there's a technology like syslog D which focusing on syslog collection there's also a part of log stash at the same time there's a multiple projects available but what we found another is we made full and D as really lightweight and also flexible agent where this agent has a lot of plugging mechanism so that community can contribute a lot of plugin for a lot of sources and also outputs and after five years of project we have now around 600 plugins available huh for all the sources and outputs and this is the nature of data analytics so any data related project but you're not only collecting from single source where you're collecting to map from multiple sources and you may want to put the data into let's say like a Hadoop and also elastic search and also Splunk or some of other places depending on the data type on where they come from so fluidy really enables you to route the data from multiple sources to multiple outputs in a really easy way no no very good 600 integrations um that's a mouthful yep yeah you see mentioned part of the genesis and the reason for the project initially to help simplify that and even though there are other you know capabilities out there how did you guys simplify that you know just technically what did you do to to make it lightweight or to make it yeah so let's say you have you're running the service on kubernetes and you also have other systems right and you might want to collect the data from nginx logs or load balancer logs or application logs or kubernetes log itself or some other stuff like external sources or cloud services can generate the data right so what usually develop or ended up do is writing a lot of custom script to extract the data or do our sync to the log server will sit at the crown scrape and what happened in a lot of large companies people who set it up korone left the company and you didn't realize that current script is running the critical part of your production system and when it fails no one figured out what's happening right so full and the really becomes the central hub or hold the data collection and also do buffering encryption compression for the reliable delivery from this multiple sources to the outputs so you have if you have a blendy you're guaranteed to have old any new sources can be collected and any new output can so the message can be routed to the new output by changing configuration rather than writing a lot of these maintained ad-hoc scripts I still have grown not jobs that are running I have curls still sending logs somewhere some write nice but then in order to make that kind of lightweight and you know talk about if you will a little bit of the the languages that are used within the project how it is that you're guaranteeing delivery or transmission of logs that are ingested so we'll start from the language perspective so first of all majority part of full Andes written in C language so like network handling we're using the gem called cool dot IO which is a wrapper of live event and also we invented another message of another serialization format called message pack which is a binary serialization format or JSON compatible types right so we use these to see base libraries of core and we wrapped around the Ruby layer so that a lot of people can write the plugins by Ruby okay and publish the plugins as a ruby gem so that everyone can download ruby gem install into your own lundi and then you can start using that plugin as a community so that's the language perspective and another differentiation fool Andy has compared to the other logging layer is full Andy has the really advanced buffering mechanism so let's say say and then for buffering the user can use either memory or file as a buffer and a lot of people are using buffer so if you're using file buffer all the data coming in from multiple sources force goes into buffer so that it gets persistent and then for it to the destination at a certain interval or like sighs right and but for example if the destination Hadoop server or relax seek search Goes Down you're not losing the data rather all the incoming data gets buffered on the disk of the flu and Dino's as long as you have enough disk size and if the destination recovers then we have an automatic retry to send the data from buffer to the destination so by having this buffering mechanism you have less chance to lose the data because other software does only memory based buffering or no buffering like syslog D so you started losing a lot of data right said yes you know on that as as fluent D was proposed for you know as an incubation and a project to be adopted into the scenes yeah yeah I understand you know there's a certain level of technical diligence that's done on those projects and and one of those I think is probably you know a near and dear kind of challenge to Eduardo's heart around configuration of how that buffer is used and you know sort of what happens when you overflow and you know at what I don't know I don't know if you can talk about that a little bit in terms of the right configurations there yeah when we started to try to apply to C and C if I'm going to talk about about the process first you have to meet certain criterias so one of things that the project is widely used you know there are many people contributing to the project for example different companies not just us and the other is that it's really made for cloud native environments okay but happened that also the same time flow and D was being used in kubernetes is used inside kubernetes as a user with docker but there were some configuration problems on which was people were trying to generate a lot of logs inside kubernetes sometimes wendy was crashing because it did not have enough memory to run in the container but was most of the problem happens because the whole bathroom staff was working in memory right and sometime the destinations were down or they were very slow so of course if you process cannot flush the data out at some point you need to storage that information somewhere and by configuration was in memory and right now we were working with the Google team for a few weeks so we managed to have a real configuration with buffering in the file system and our different flags in terms of how many chunks we are going to store in memory how many in the file system so we did some kind of the optimization for that ok and then everything did everything much move to move flow and D to C n CF and B better into the kubernetes ecosystem makes perfect sense is that what was the default configuration change there you know in terms of others avoiding buffer overflow and or having the right settings such that buffer is in fact persisted to disk you know yeah well they are not too many options is either a memory or the file system and of course when you start a container you have some restrictions you say ok with this content is going to use this amount of CPU this amount of memory if you reach memory the kernel is going to kill that container right away now so do we make sure that the setup is align it with a container set up flags okay I mean the default is actually a file buffer okay but the kubernetes initially for some reason used memory buffer which identified if we identify the problem quickly so we changed it right and by stats we were using a lot of memory with like 500 blog lines per second now we can handle 5,000 log messages per second which is a huge improvement and I think the jury of application should be okay with this conflict new config perfect so tell us about this display ecosystem here what what are people writing plug-ins for and what's the process I guess what's the interface for the plug-in and also is this are the plugins things that you yourself kind of manage and oversee or is it just like a marketplace and yeah so what's the first question what are people using what are people creating buffer beverages done yeah so for planes there's four types of plugins available so the input plugins output plugins parse or plugins and also filter plugins so people can customize and then publish plug-ins for each type and therefore inputs it's a lot a lot of protocols like net flow protocol receiving the data from switches or TCP or HTTP or even Twitter receiving the data output typically is a lot of database systems like MongoDB elasticsearch Kafka parsers some known format including from JSON to custom binary format and also filter is how you filter data so typically more like developer oriented ops person like devops person right the plugin for a specific system which is newly released or existing or they just publish it and the way plugin works is it's packaged as a ruby gem so once you write the plug-in ruby code and you can publish it as a separate ruby gem and then after that flew and the user can install that gem into the local full and deep environment and you can immediately start using it so it's like gem install full and deep plug in X Y Z and you're ready to go and full and gem push full and D X Y Z you're ready to publish to the community very convenient yeah you know I just had a had an epiphany I just came to understand why the naming of fluently the project itself it was you'd mentioned earlier there were no already's you know capabilities and solutions out there in general syslog D being one and then there was it's already a syslog-ng you know next generation and then so you know since that name was taken there couldn't be this is like not next next generation I know I would say they were not fluently transferred from XYZ this log they usually use UDP previously now they're switching to TCP by using UDP you might lose the data there's no buffering capability available in syslog I don't know about syslog and they might have it and then configuration is not easy it's a sea based system no plugging so it's really hard to extend so to talk about that per minute if you would just the the notion that you know among other things that fluent D brings is you know guaranteed transmission also you mentioned an encryption earlier are there but there's some amount of you there's that that takes that takes a little bit of CPU cycles or a little bit of performance away from you know to the extent that you're not guaranteeing transmission you can just send it and forget it so maybe you can talk about the events per second and count translations per second or how do you measure that yeah so we have a benchmark available on the github so the typical like a four gigabytes of memory two cores machine on Amazon I forgot which instance type it is but the one core can handle 18,000 messages per second okay it fills up the CPU hundred percent CPU for the one core Nessus is per second or like what type of message do it's the test messages generated so if it's a syslog we have an overhead to parse the syslog protocol right which decreases the throughput usually but you know if you look at typical application you're if you're generating thousands of messages per second from your app something is wrong already unless it's not intentional just gameserver advertisement server right really high traffic so I think in a lot of cases throughput is more than enough so you mentioned kubernetes earlier and I'm just I'm expecting that fluid D as a daemon or as a process is oftentimes deployed maybe as a daemon said or maybe as a cluster add-on but I and so I kind of get that but back to the encryption piece um how are is that is there a requirement around a PKI certificates and or is there pre shared key or just how does encryption yeah so fluently you can afford locks from full and D to another fool indeed and then we call it is about for water and also aggregator for other usually runs within each like container or edge systems while aggregator runs on specific nodes so that you can aggregate all the data coming from thousands of containers and then for this somewhere right so the encryption comes in between flu and D so you can use your own certificates to encrypt the data on the wire and also we're planning to add encryption on the disks for the file buffer so that no one can see what you're collecting as well right but also you can disable the encryption between flu and D so usually people who are sending the data between the data center they usually do encryption because the network goes into the internet public Internet well if it's within then the same data center they don't encrypt because they rely on the internal network security usually makes a lot of sense [Music] well you guess what we were talking about them CN CF you know it's a bit a decent big announcement of a part of the keynote that Dan Khan gave earlier yep just the the voting in of the project itself I guess maybe some early reflections on that from you guys so you know what does it bring I mean what how does CN CF how's the support help you guys a particular okay a well I had to question from the reflections I think that it was a really good process it was really difficult to fluently to join CN CF it not was like okay you have a bunch of users you are in because we got a lot of a technical concerns from the from the committee it's a and they were for real so I think that we always took a very positive attitude for the comments for the concerns about okay what are they this is really good in some areas but sometimes there's some concerns for some specific use cases so and of course we need to work on that and and that's the idea if we join CN CF it because we want to support you know CN CF and the companies that it relies on our technology you know so a what this was different if a second question was about sorry what advantages does joining up of what support does CN CF offer you guys how does it help you guys advance the technology well I think the first one is that the first gift that you can get as a software developer is to free documentation now if the talking series that we get like a 20k per year for documentation and I think that is really good we will get access to the cluster so of course we can start measuring influen D in different use cases in scenarios and of course improve on that areas and from the community side we can of course started working with like kubernetes core committers working with the Google or other companies who joined CNC if like Samson and others so I'm sure we already have a lot of developer ecosystem by ourselves but combining with other CN CF project with fluent D community we can do a lot better and then we can be the default log collector for the cloud native error or a cloud native environment so I think you guys hit on like you know four different aspects of like you know the benefits of you know there really any project joining up and it sounds like you guys are seeing those benefits already at some plans towards you're using up part of those nodes on the cluster that that's fantastic maybe the test that you were talking about earlier the the four gig into the before gig to note or Kia because some of them well our user typically has thousands of servers right because the moment you need to Lindy is the moment you have a lots of servers so people really assume fluently works with lots of servers so we need the environment like that to test it out it's an impressive environment wonder if you could tell us a bit about treasure day know at what point would I your services yeah so um this is really interesting part so tragic data is not directly monetizing full and deep until recently so our original idea is providing the analytics cloud service and what we how we grow is were around a hundred people company right now after we founded this company five years ago so same time pull Andy gets born and how we grow is interesting we leverage open-source a lot so we invented full in the five thousand companies adopting flu Andy and they use on-premise and then sending the data to like elasticsearch I had to purse plunk or whatever data outputs they have but some of the people maybe three percent five percent they don't want to manage the parkin and because they know tragic data is behind fool indeed they naturally come to treasure data and they by changing one line of configuration in fluently you can start sending the data so that's how we establish the business that's why we can spend full-time developer and the marketing efforts to flu indeed because the more full and deconstruction the more our business will grow you know which were you talking about I'm fluent dear pair bed and and but Atwater I've also heard about I'm fluent bit salmon and just kind of some work I think that you and maybe some others that put towards that what but what it what is fluent it okay what is fooling bit affluent Deccan be consider it as an aggregator when you wanted to collect the data or with its centralized the whole data before to a final destination like Kafka of readies Amazon s3 but sometimes you would like to have some lightweight for water in the H note so fluent beat means to be like the client for flu indeed for various special cases where you need really high performance for a very special use case so Fulham beat a it uses has like the similar architectural affluent D but the difference that is reading full in C and the good news that I was just another thing that we are extending flu and be to support goal and plugins people from we were talking today people from Microsoft people from Google from kubernetes in they're pretty happy with this kind of move because they can extend one of part of a log in architecture continue using fluently in the other so fluent bit means to be like the final component to to close this cycle of login stuff okay yeah no very good we started the release year that the one like year and a half ago uh-huh with full documentation there's the first version very proud of it and right now with inversion 0.9 we those releases last Friday unsere that 10th started with such a work on that Monday so we expect to have media and what that 0 next year but I thing at this point is pretty stable it's not a great to one that 0 to to feel that it is stable I think that numbers means something but not all excellent actually what definitely let us know on you guys to pick 100 do follow up learn even more about that sure all right fantastic well I think that I think we should wrap it up at this point but gentlemen thank you so much for taking time to get us up to speed thank you very much thanks for the invitation and whoa let's keep talking thank you guys thanks to Cisco for sponsoring our day of podcasting at KU bukan we had some great conversations thanks again to Cisco you can learn more about cisco and their micro services platform at mantled io that's ma n TL dot I [Music]