Streamlio Cloud Demo: A Messaging Platform to Ease Development Around Streaming Data

Key Takeaways

hi it's be Cameron game with the new stack I'm here today with John Bock vice-president of marketing with stream Leo cloud which provides a streaming messaging solution that runs on the cloud thanks for joining us today John thank you so in minds with what we were talking about before could you please explain why should developers be interested yeah so one thing that's been the case is that most developers who are writing applications that process data have dealt with data that was coming from batch repositories in the past and yet now there's an increasing emphasis on having developers do to write applications that can handle data as it arrives whether those developers are data engineers for developing code to process data or whether they're actually developers running applications that embed data and analytics and so one of the challenges in the past has been that for a developer to work with data as quickly as it's arriving used to require complicated stream processing engines and complicated functional programming paradigms and different SDKs and api's and such that we're pretty unfamiliar to most developers and so it become a pretty big barrier for them to develop these applications that could process data as quickly as they arrived and so what we've actually done here at stream léo is put together the technology needed to make it easy for developers to have a simple experience for developing applications that can operate on data immediately but also to make it easier for those developers when those applications and that code are pushed in production now we certainly talked a lot about what it takes for developer to create code but a lot of the developers time is actually spent dealing with how this the code that they wrote is operated in production and spending a lot of time with DevOps and with the production ops team helping them to figure out how to make sure that that code runs effectively production and so that's one other thing that we really spend a lot of time at it's Truglio is really thinking about how do we make an environment where when a developer deploys their code it's very easy for the developer and that the ops team to work together and in many cases the ops team can breathe independently addressing some key challenges that they might see in performance or in throughput as they monitor the application without really need to put a big burden on the developer and so that's really what you know our platform for messaging and streaming is all about is making that something that is much easier for a developer to be able to work with is this something that might go with version control software or how would that work or would there be any overlap yeah so this definitely complements what people are using today for everything from you know their IDs to their CI CD tooling and pipelines fundamentally this is about where you actually deploy your code and what the SDK API is that goes into developing that code and so in this case you know you can use whatever ID you want you know you can put that new code that you've created then into a CA C D pipeline that pipeline then can interact with our platform to deploy that code and to be able to update that code and then on the operational side we provide tools that help both developers and the DevOps people to really kind of understand how that works together so there's definitely complementary aspects to this that don't require people to change their development tooling and those tool chains great so could you just walk me through it how does it work yeah so what we have up here is we're going to start with the scenario of a developer writing you know a simple application to do some processing of a stream of data so in this case you know we're adding some some functionality to do some processing of a true data coming off of Twitter you know it's a very simple source to kind of illustrate to you a stream of data that's constantly arriving constantly changing and so the developer is going to you know be asked to run an application that basically does is some simple classification of that so it's going to look at the data coming through and it's going to use some simple machine learning to decide whether those tweets that are coming through our positive or negative and based on that it's going to create new streams of data he's going to create a stream of data that's going to have all the positive the positive tweets about it the given keyword that they're looking for and it's going to create another stream of data that's going to have all the negative tweets so this could be more like a data engineering type developer where they're focusing on creating some additional data streams that are going to then be used by applications you can imagine all different types of applications that this could feed into so in the past if you had wanted to create this type of processing you probably would have used some complicated stream processing engine it would have had a complicated SDK you know what you can see here is actually a pretty straightforward set of code that allows people to leverage existing libraries in this case you know our code is pulling in some existing NLP libraries that are doing processing of that data and incorporate that in a very straightforward way you know the code itself basically just needs to understand that there are inputs fundamental you know those tweets coming through the inputs in this case and then there are outputs in this case we have an output for that stream of negative tweets about our keyword and an output for the stream of positive sweets but our keyword and so that is basically all that there is to you know the SDK and the API and you know this code here which you probably many developers have written similar code already just kind of illustrates like it's relatively straightforward to write this code you know you won't see anything here this related to functional programming you know maps or flat maps or concepts like that you won't see anything in here related to the need to understand DAGs or things like that in the environment what you see here is basically just you know something that focuses on the logic that the developer is creating and so you could imagine you know there's a parallel here that we really wanted to create that was about the way that people are really kind of adopting the serverless and lo code environments we wanted to bring that into this world of streaming and messaging and so rather than having to you know have some complex coding happen you know developer focuses on the logic that they want to write and then they can deploy that to an environment where they don't need to understand the details of the infrastructure to actually make that code operational in this case you know we're showing you a Java example obviously but you know these could be then things like Python as well and fundamentally you know all this needs to be all that it needs to be done to make this useful now just take a pilot into it and a function so Java has that notion of functions you can compile that doesn't require bringing in a bunch of other libraries things like that the Python has you know this notion of functions as well so fundamentally what we've done is taken this code you know put it into a function we can then check this into our you know version control system for example that could then kick off you know a build and the CI CD pipeline and that C sed pipeline could eventually then deploy this into our messaging environment now what we've really focused on a stream leo is like you know how do we make that experience seamless for developer in this case as you can see you know this is an IDE and we don't provide the IDE because we know that developers already have their preferred IDs we make it very easy for people to integrate with this because we can be you know the destination for the CSE pipeline to push that and that's how some of our customers are already using this capability so that was kind of talking about a little bit the development side but as I mentioned before you know one of the big challenges for developers is often that they get pulled into dealing with a lot of the production and operation side of things certainly the ops team isn't going to know the internals of the code that developers wrote and so often they actually bring the developers in and have developers actually spend a lot of time helping them to understand what's happening in the production environment and the saris also play a role absolutely yeah yeah and so those are critical teams but they also put a pretty big burden on the developers who have to like work with them constantly you know if for example like there's a performance issue in the environments you know it's very common for the SOE team or the dev ops team to have to go back to the developer and as to developer to spend some time helping to troubleshoot that and giving them recommendations but you know we really want to kind of D couple that a little bit more so that the development team didn't get burdens as much by those operation concerns and so that's we're going to switch over to you now and we're gonna show it the you know the way that this looks for someone who is actually operating the environment and how that can help to facilitate you know the communication between developers and that SOE or DevOps team so we're gonna switch over now to a UI that is designed primarily for the ops and SRE team to look at but also is something that developers can look at as well because you know certainly want to develop our deploys code one of their first things they're gonna ask is well you know how is it working you know are things working as expected so this is basically kind of a the front end dashboard that you can see as you log in and one thing that's important here is that you know in many cases you know developers have typically been deploying to kind of isolated environments so you know you develop your code you develop it and then you deploy it to a cluster and that cluster is limited to you know a single team for example or maybe a single application but increasingly as we especially as we get into micro services and such you know the notion that every application or every service is going to have its own cluster for processing it just doesn't make sense anymore and so as a result you know people are going to be operating in these multi-tenant environments and so as a developer like the first thing I start to worry about is well does that mean that I'm gonna be colliding with other developers you know I'm not gonna have namespace conflicts or collisions and things like that so that's something that we did definitely build in this platform is this notion of a hierarchy of view entities that allow us to isolate different groups and different applications so we have this you know hierarchy you can see it here come to the top of our dashboard of tenants name spaces you know topics that basically is a structure that allows us to separate different groups on different applications and even different specific you know services and sub components so you can think of for example a tenant might be sales or might be marketing or might be you know business analytics a namespace might be you know customer you know satisfaction or you know sentiment or things like that and then a topic is gonna be an individual data stream so for example you know in this case we could click into the topics and you know you'll see you'll see there's a bunch of different topics here just a couple of mention you know that Twitter one topic that you see there that's basically that stream of data that's coming in off of Twitter so that's kind of the input data that we're using for the purposes of this demo and then you know as you scroll a little bit further down we can also see that there's a couple of other topics and that's related to the output of that function that I just mentioned so that function that we just showed you earlier is basically creating these two new data streams one called positive tweets the other called neutral tweets that are basically the results of doing that NLP processing on that incoming stream of tweets and then there you see other other topics here as well you know so these are basically different streams of data that you know potentially an application developer might want to use now as an application developer you know I have access only to the ones that I've been given access to I don't even see the other ones so there could be a completely different namespace or completely different tenant and if I haven't been given access to that I don't see that so from a developer's point of view I can operate and think about just being kind of in my sandbox so to speak and not have to spend a lot of time worrying about you know coordinating with other people to make sure that we know the main space collisions or our identifiers you know don't clash or things like that you know I basically have my own separated environment and that also applies when the SRA and ops team is trying to figure out how to allocate resources you know in a multi-tenant environment you know I as a developer don't want to have to like basically put a lot of resource management into my code I want that to be something that the operations team handles and so what this actually does is you know it allows me to as a developer to give the operations team the ability to you know see how to manage resources so the operations team can set quotas you know and do other resource management and that can operate on these different levels like these topics for example to really control like how many resources are used so that you know a runaway application doesn't cause all of their applications on the system to fall behind or to be starved now that's another key aspect of making it possible to deploy this in a multi-tenant environment without a developer having to you know understand all those detail about what's underneath or even what other applications and groups might be using the system is it possible to maybe maybe it's at the previous slide but you know on the as far as the production pipeline goes for example before we get to the operation side you know where are we see in this interface or how's this showing data for example about the Jenkins Jenkins pipeline or a yes so the the monitoring of the CI CD pipeline is going to be upstream of the stream low platform so we're not going to show you that that directly you know that's going to be the tools that you already probably use to monitor your pipelines you know whether it's repairs Jenkins step dashboards or you know other tooling type dashboards from the various CSE sort of you know tooling components we're basically you know where you're deploying that and so we're kind of the output you know we get to the end of the the build process where do the artifacts get deposited you know that's fundamentally where we are so that code that you saw earlier you know that would have gone into some compile process and then as the final step of that compile process it would have generated you know a Java Java function and then that function you know basically through simple scripting integration would be something you could easily easily have deployed then into this environment and now it's you've given over to the ops team to help you to your QA it to stage it and then to move it into production I'm assuming this is in multi cloud we can incorporate multiclad environments into this interface as well or without how does that work for example on premise you have a few maybe lumps is you have Google Cloud is that it's able to pull all that together this is entirely based on an open source software called Apache pulsar and so that software can run on any environment you can rent hybrid cloud environments so you could be running an instance of this you know in your local data center you could be running this you know managed by yourself in any of the public clouds and we also do offer a stream Leo managed version of this which is what we call stream you apply and so it is definitely very common that people are using this to help move data among different environments I may have you know a production application running in a data center but I also may have data sources that I want to process in the cloud because that's you know the first place that I'm able to aggregate those data sources and so there's definitely you know some great capabilities that this allows you to unlock in terms of being able to move to you know that's your resilient it guarantees you won't lose any data there's replication that can be used across different sites from edge to cloud your data center and so there's a lot of flexibility capabilities there that adult or doesn't have to worry about a developer just needs to think about streams of data on what they're doing the process that data they don't need to worry about thinking about how does that get replicated because there's something that the operator and the sree team can handle directly themselves let's drill in a little bit so let's drill into you know one of these topics so let's go into that Twitter that yeah we're gonna take a look at you just an example of you know some of the things that are happening here so I as a developer I might be monitoring this a little bit because I want to know when I'm deployed my application that nothing's gone wrong but I'm gonna take a look at this and I'm gonna see like okay interesting I see data is flowing through um you know I can see that but I do see that you know the storage rate is growing over time and I do see that you know maybe there's going to be a needs to you know expand the parallelism of how this is being processed now I might as a developer not go any further at that or I might not even have to go there because you know this is something that the ops team can also see so my DevOps team or in my oh sorry team they can look at this and say like okay interesting like I see there's a little you know there's a little bit of kind of a growing backlog you know I can see you who's consuming that maybe I actually need to add a little bit more horsepower to be able to like help this to you process a little bit faster so you know what I can do as my SRE team I can go in and say like you know initially that processing that was done by a single worker well what if I want to actually now have that done by significant greater number of workers I can actually just go in expand the number of workers in the pool I don't even have to go to get the developer involved and so the developer you know they just didn't wrote a simple function they didn't have to understand distributed programming they didn't have to think about you know complex you know ways to coordinate across different different elements you know that's handled by the platform here and so that allows the DevOps team the SRA team to say like okay I can go ahead and I can increase that parallelism directly and that actually will help to keep the system stable and make sure to bring a processing data as quickly as it's coming in because you could imagine for example there might be some spike in the incoming data in the case of Twitter there could be some spike because there's a trending hashtag that happens to be related to my keyword and so I see a big spike in data I don't want to fall way behind in processing that data and so I can simply increase the journalism no never have to bring in the developer at the same time the developer can kind of see what's happening as well and so they can see like okay good it looks like you know the the team is being able to keep up with the data I don't have to get involved as a developer I don't have to do lots of performance troubleshooting because that's kind of something that you know the team cannot take care of directly themselves so this is definitely you know very different than what you often see in many application environments you know many application environments the developer has to go in they have to write a detailed code to emit specific metrics they have to provide you know different ways to like you to be able to like you troubleshoot and trace different things going through the system in this case what we've done is we've basically built that into the platform so that the SRE team and the DevOps team can handle most of that self service so we've shown you the example of I can increase you know the number of different parallel instances of that code that are executing to process that data coming in there's also cases where we can actually add brokers so because this is fundamentally a messaging system you know one key aspect of it is brokering so adding brokers is a very simple thing we can do as well and so would you increase the amount of throughput that this system can handle in that aspect and then the other aspect is the storage of this you know there are many cases in stream processing where actually do want to be able to look at you know more than just the current event in the stream but I want to look at older events in stream as well and so you know the technology that we built based on Apache pulsar has the ability to have this effectively infinite stream storage in the cloud because it can leverage things like Amazon s3 or Azure blob storage so that I actually have full access to that so that's another thing that I can as a you know the SR team the DevOps team can go ahead and handle scaling up that without the developer having to be involved the developer you know they just know that they have access to like even potentially the whole history of the stream of data in one system without needing to know exactly how that data is stored and the SRA Maps team you know they can go ahead and they can basically scale that up as needed non-destructively for the application so you know if you think about you know what a developer would have to do in the past to write and set of application they typically have to be very conscious of exactly where that it is you know some of the data in traditional environments would be in a stream processing engine some of it might be in a data Lake and some of it might just be in a key value store or some other type of database and their application would actually have to have all this logic to understand which data to ask for from where and to deal with the fact that sometimes you know the data is aged out of some of those systems and so as a result like yeah the application has to first check is the data here okay if it's not here I need to go check this other system now what we've done with this storage capability that's part of Apache pulsar and offered through you know the streamlet solution is we basically remove the developers need to think about that you know the developer just knows that if I want to get access to any data in that stream I have at that access in one system and so that is another aspect of what makes this a lot simpler for developers than what they would have had to deal with in a lot of traditional stream processing or message processing type environments are there any plans to maybe add to expand on the metrics or is that consider sufficient right now yeah so on the metric side you know there's a core set of metrics that we have they're just built into the platform so metrics everything from you know how many times is the function processing to how much data is flowing through that function to what's the backlog into that and so on so those are the standard metrics that are fully available you know we actually feed them you know I collect them via Prometheus and because of that you know there's a lot of different integrations that people can integrate into their various tools to actually then visualize those metrics so it's everything from you know you could put a graph on a dashboard in front of this it's gonna create your own - you could use one of those you know commercial packages out there for metrics viewing and drill down so you know that's fundamental framework is there and then we also allow you to make custom metrics as well and so those custom metrics can come in a number of forms you know you can for example there's there's a facility to allow you to emit log messages without needing something to write your own logger or bring your own loggers in and then those could be fed through that system but there's also custom metrics that can be created and fed through that same framework so that they're available through Prometheus as well - any - lets connect its Prometheus to be able to scrape those and visualize those and help people to understand okay it looks good any kind of you mentioned Prometheus anything maybe any kind of overlap with gravano for example or any other kind of interface is there yes you know Prometheus for us is it's great because it can help us to collect those metrics and then those metric can be surfaced in any number of ways like Griffin is certainly one thing you know we've worked with a number of customers to give them dashboards using Griffin that actually you know can be customized and then use those to you view both you know these standard metrics that are available in the system as well as custom metrics that they might have created for their application so that's certainly one possibility we also worked with other tools you know we've worked with you know companies where you know the SRA is for example you know they use data dog or systems like that and so that's allowed allowing us to go ahead and integrate with those as well so that's actually a good point in that you know this gives you the framework that you know is both an SRE a devops person or a developer contains other tools you know certainly like the developers probably don't spend much time in you know things like data dog or things like that but there might they might want a simple dashboard that can allow them to monitor like that new code that I deployed you know is that performing well you know are things working well now this is just an example of a graph on a dashboard so you know this is something that you know we've created for a number of different customers that basically ties into that Metris collection system to show them just some different statistics so this is kind of an ops dashboard but you can imagine you know a subset of this obviously could be made available to a developer as well you know so that they could actually take a look at some of the things that matter most of them you can see here for example you know there's function metrics and topic metrics you know I as a developer I actually might care about this topic metrics because that really shows you know how much data is flowing into my processing code my stream processing code and you know it allows me to look for you know anything that might be anomalous there like maybe I know that you know I pushed new code that was deployed and I look you know that these charts to see was there's no is there some sort of change in that because that could be indicative that there was a problem in the code I pushed I'm doing my QA and testing so there's definitely a lot of different ways to see the data in this environment that you know allows the DevOps team yes our team and the developer to all have different levels of visibility based on what they need to see great yeah so stepping back for a second talking about what's available to the developer and what's handled in this platform so one of the key things in messaging is you know delivery guarantees and so that's something that is definitely you know something that you know in many systems a developer has to consider how to write those into their application but what we've done here is take that notion of delivery guarantees as many people in messaging are familiar with that notion and basically have the platform take care of that so things like exactly once at most once and you know at least once delivery all of those are something that you can be set and you get it can be set in the platform so that the developer doesn't have to write extra codes to take care of those so that's another aspect there's something that you know you could have been a burden on the developer that's not taking here by the platform similar examples are you know people who have written messaging applications before you know they're probably familiar with the concept of you know keeping track of offsets or cursors and things like that you know the idea being you know if I am an application I need to keep track of what messages I have seen and what messages I've told the messaging system I've seen to make sure that the messaging system knows what it needs to redeliver messages or where I am if I am in the stream if I've disconnected in reconnect in the case of stream leo and of apache pulsar here you know we actually have that handled on the server side so that the application developer doesn't need to understand those things the application developer you know just needs to write the code that processes the data and the system will take care of keeping track of where in the stream if that specific application consumer is so that's another key aspect that's something that's handled by the system rather than developer in terms of getting started with this you know typically we're finding a lot of people start with the cloud version and so we actually have a managed service for that so that people don't need to even worry about setting up software they can basically just you know let us know and we can set them up with an account to go ahead and start using this in the cloud but certainly if they wanted to deploy you know we're very friendly to kubernetes type environments and so this is it's an application that fits really well into those type of environments so people who are running in the cloud and kubernetes are on Prem in kubernetes you know that's a very common way for people to deploy but certainly you know we also do provide packages and things like ansible scripts and such for people who might be working on just you know bare metal in virtual instances in the cloud or even you know installing it on premise and so there's a lot of different ways that this software can get installed for the developer most of that they won't have to worry about that'll be taken care of by the ops or the you know the production infrastructure team you know they just basically need to work with their you know DevOps team to make sure that the integration happens with the CI CD and the build pipelines so that they know how to be able to make sure that the code that they check in gets deployed to that environment so from a delivery point of view you know there's not a lot that they have to worry about or learn new here and the fact that it's available in cloud editions makes it even easier for a company to try it out you can start it is it possible to provide maybe any pricing info yeah so finally the pricing and there's a couple different models you know if you're using it in the cloud version there's prepackaged configurations you can take a look at those in our website that basically you know are designed for certain throughput levels and so you choose which is appropriate for you and then on the stream there side you know we manage ensuring that you know we're delivering the right resources to make those throughput levels possible you know we don't have the the the pricing numbers obviously we work with people directly on then if you want to use this you know in an environment where you're managing yourself in on-premise type environment you know we actually you also provide licensing that's based on the amount of resources you know in the underlying messaging cluster that's running the system so because it is kind of messaging in data processing infrastructure that's the approach that we take in the on Provost environment now we do have you know a trial to get stories e to get started we don't charge you for that trial we just deploy it you know in your Amazon account someone's like but what you might have seen with like a data bricks or things like that what an easy way to get started with that as well okay well John that was great I think this can definitely be in helpful for developers and thanks very much for taking us through this thank you appreciate it