Cloud Native Short Take - Druid Architecture

hello everyone welcome to the latest rxm cloud native short take my name is christian luxena and today we'll short take the rxm druid architecture module in this module we discuss each of the constituent components of a druid deployment we also demonstrate what occurs in a druid deployment when a task is submitted we'll then explore the interaction between druid and zookeeper and talk about the infrastructure considerations that affect druid and finally in this module we explore how druid scales in today's short take we'll walk through druid's architecture by launching several containers for each of druid's components we'll see what functionality we receive with each new container building up to a complete druid cluster to begin we'll look at the groundwork for our druid deployment its external dependencies i've got these dependencies running on docker druid depends on zookeeper for cluster component coordination it also relies on some kind of relational database like postgres for example to store metadata on its various jobs and other identifiers and lastly druid relies on some kind of file storage which is usually an object store like s3 however for today's demo i'll be using local disk space with my external dependencies covered i now need to configure my cluster many druid configurations can be provided using an environment variable so for this demo i'll be using a file full of them to configure my cluster environment variables make druid very easy to configure for containerized environments like what i'm trying to achieve here the first druid component i'll bring up is the coordinator a druid coordinator is responsible for ensuring appropriate data distribution to each of the data processes that are part of a druid cluster additionally a task distributor called the overlord runs alongside the coordinator to trigger data ingestion movement and management tasks across the cluster it also gives us a gui data in druid is handled by a couple of other components one of these components is the middle manager the middle manager is responsible for creating druid formatted events by parsing live data streams like those provided by kafka or pi parsing files in batches which can be from local disk or even a hadoop integration i'll start a middle manager now now that i have a middle manager process running i can actually load data into my cluster at this point even though i don't have all of it up and running to load data into the cluster i first have to submit a task specification to my coordinator i'll do this by providing a json file that describes where druid needs to retrieve my data and how events produced from those data should look like this file is called an ingestion spec i'll send this ingestion spec to my coordinator's overlord port to initiate a data ingestion task this task reads a short json file that i've provided to the containers and produces queriable druid events during the ingestion task the middle manager parses data from the specified source converts that data into events stored in a columnar format and then produces files called segments which contain parts of that data the segments are separated into time-based partitions called chunks those segments are then sent to the file storage known as the deep store which i have running on my local disk despite the fact that my middle manager has these data they're not queryable yet at this point i don't have any querying functionality and the batch data that i fed into my cluster aren't actually queryable from the middle managers the middle managers at will however make the streaming data from kafka for example available immediately for querying now let's see why i can't query these data by checking the log as we can see the coordinator is saying that it has no servers that can actually serve these batch data for querying now batch data like what i've ingested is only make queriable once druid takes these segments and makes them available through a historical process historical processes are responsible for holding batch data for user queries i'll launch a historical process right now now if we look at the coordinator logs it should have found a place to store that data we can see here that a new historical process has been detected now my data handling capabilities with druid are handled exclusively by the middle manager and the historical in order to increase or otherwise scale my data handling capacity of druid i just need to add additional middle managers or historical processes to my cluster so at this point my cluster has successfully recorded the data now it's time to query it query functionality is provided by the broker process brokers parse user queries which will then reach out to the middle managers and the historical processes to collate results based on that user's request i'm going to go ahead and launch a broker now multiple brokers can handle user queries thanks to a process called a router a router can effectively load balance queries between different brokers and it can also perform different kinds of routing for brokers so let's say you had a broker that had more data that was live versus brokers that would almost exclusively deal in historical data you could do that with a druid router which i'll actually launch right now in addition to broker handling routers can provide additional uis for data handling so if we head over to our router port on our node you can see that we have a new gui which is slightly more expanded than the previous gui from here i can go to query and within this query view i can actually look for data based on the druid version of sql called well druid sql which is actually not quite sql however it gives a sequel like interface for those of you who are more suited to using sql in your daily lives so by adding additional brokers and routers to my cluster i can scale my query capacity without affecting my data storage capabilities and that's it you now see how druid's deployment architecture can suit a cloud native deployment discovering what makes a druid cluster tick is just one of the many things you will learn in the druid architecture module and the druid classes from rxm rxm has many other classes covering databases which you can browse at our course catalog at rxm.com under courses click on course catalog and if you scroll down you'll see the various categories of courses that we have let's go click on databases and you can see that we have druid among many other offerings for database focused classes in addition to data science and other types of courses and that's our cloud native short take on the rxm druid architecture module