Elixir Tutorial Part 4 (GenStage)

Key Takeaways

hi guys welcome to the lipstick tutorial part 4 my name is Ken sir from the tensor programming block and today we're going to take a look at gem stage a short summary what gem stage basically does is it allows us to handle a real-time stream of data in a distributed way so we can assign multiple processes to basically pipe this data into our file or program or whatever we want to put the data into so we'll go more into detail for this when we actually build our project here so let's start building our project we're going to use mix and it's a new gem counter and we're going to add an extra argument that we haven't added before and this is - that's up and what this will do is it will generate a boilerplate supervisor file for us let's CD into that folder and then open the gen counter in our sublime text so here's our gen counter program and as you can see there is a folder in here called gen counter and under Lib with an application da/dx and then here is our supervisor boilerplate that was generated by mix as you can see the supervisor is basically complete except for the fact that it doesn't have any children of course because there are no children processes for it to watch also if you look at our mixed exs file you can see that it's set up the application to actually run with this mod and then the application name okay so gem stage is not currently a core library in elixir so we are going to add it as a dependency to do that we're going to put it inside of this list down here we're going to say with an atom gem stage comma and then the version number so this will be inside of a tuple and then we'll put a comma after it and save this then we can go back to our terminal and write mix do guest I get comma compile and this will allow us to perform two operations one of them will be getting the dependencies and then the other one will be compiling those is it compiled our gem stage dependency and then it compiled our gem counter application even though there's nothing in it currently so there are three different pieces to a gem stage project typically and those pieces are called a producer a producer consumer and a consumer so let's start with the producer first so we're going to create a new file and it's going to be appropriately named producer da/dx so we want to create a producer that emits a stream of numbers because this is going to be a counter is just going to be simulating a stream of data so for example you can think of this instead of numbers you can think of this like if you are getting JSON from HTTP source or if you are reading a very large file it would be very similar to what we're doing here let's of course to find the module first and this will be gen counter dot producer we want to use John's gauge we want to make a start length function inside of this function we will put an init variable and we can actually define what that init variable this value will be with two backslashes so we're going to define it with zero so we can call this function without actually declaring in it and if we don't declare in it then it will default to zero we will link this to gem stage the start link inside of it it will be very similar to our gem server in that we will want our module atom so the name of the module so we were basically wrapping our module with this gem stage behavior and then we want our init sort of like our state in our gem server and then we want the name in this case our name is just going to be the module name as well then of course we want our init callback this will have our counter in it and we can make this a single line function because all it's going to do is call back with producer and so it's going to set the state inside of our gem stage and it's going to call back and set up the process we also want a function called handle demand this function will take in demand and state and we will say events equals enum dot q lists what this will do is it will allow us to create a range of numbers and then put it into a list in a lazy method we're going to have a range from state to state plus demand minus one so this will allow us to create a very large amount of numbers actually infinite numbers if we let the program run for as long as we want and then we will return a tuple of no reply event and then state plus demand so the init function of course is setting the initial state and this is very similar to our gem servers but it's also labeling this particular module as a producer the response of saying producer this atom is telling the author's that it is a producer basically this function is what gem stage is relying upon to decide what exactly this module is supposed to do then the handle demand function is where the majority of our producer is being implemented so what we're doing is we're returning a set of numbers which we are demanding we can choose how much we demand at a given time and then we will get that set of amount of numbers of course our consumer part will actually make that demand so the demand from our consumers represented by the demand here is an integer that is corresponding to the number of events that this producer can handle at one given time this one in particular this defaults to a thousand so now let's make our producer consumer and this will be produced or understood consumer da/dx and of course this module will declare it as Gen Con or dot producer consumer this is kind of the middleware in our gem stage so we're also going to use the gem stage module inside of this module and we also want to require the interviewer module and this will allow us to do a few things that are sort of like type checking and I'll show you what I mean in this module we also want a star link function this function will take no arguments and it will call gen stage that's startling I'll pass it in the module atom and we'll pass in an atom called state and we'll pass in the name call it module again to this atom called state is just a filler it doesn't really matter in this case because state doesn't matter inside of our producer consumer our Start link is going to have an init function this will take in a state variable and it will return a tube rule of producers consumer state and then we want to fill in this field of subscribe to with the producer that we want this to subscribe to in this case it's our gen counter producer this will go inside of a list next we need a function to handle the events so that's handle events will pass in the event from and we'll also put an underscore next to from because we're not going to explicitly use it in the function and the state we're going to define a variable called numbers and patter match it with events typed it into enum filter so enum filter allows us to lazily apply a function to a list in this case we're going to use an and sign and then we're going to call integer dot is even what this will do is it will let us basically call this is even function on our list that we're piping through here and we're going to just put in the RT of the function this is kind of a rolling saying it's nothing you really need to worry about we could write our own is even function if we wanted to to make this a little bit more clear instead of using this syntax so then our tuple at the end will be no reply numbers which is this here and then the state so basically what we're doing is we're going to be counting up infinitely with infinite numbers and then we're only going to show the even numbers so even though we're going to deal with all the numbers you only want to really sewn the even numbers to our consumer so we're filtering out the numbers that we don't need so again our init function is like what we did in our producer we are defining this process as a producer consumer and then our subscribe to is telling this particular module that it needs to be in communication with this other module with our producer module our handle events function is where all the work is happening so we receive our incoming events in this events variable process them and then return them as a transformed list so as I said before we will just be returning the even numbers in this take from this particular function and then pushing them through to our consumer file so this is kind of similar to how our consumer will be defined though there is one minor difference when we label a process as a producer consumer the second argument of our tuple this number is used to basically meet the demand of our consumers downstream if we were dealing with the consumers than this value itself gets discarded so let's create the consumer and show you what I'm talking about our next file of course is appropriately named consumer dot e X and our module will be Jen counter that consumer and we're going to use Jen stage and again we want a start link function this function again will take no arguments and it will call gauge n stage start link function we'll put in our module name again and then we'll just put in a atom that just says state and we won't have a name for this consumer we also want an init function again and remember these init's are callback functions so these are functions that are deployed by the process itself as a response to the start link function happening in a way Jen stage is basically an abstraction above a Jen server it's just made specifically to deal with big types of data so we want this init function to subscribe to our producer consumer instead of our producer so we will put in here subscribe to Jim counter dot producer consumer and then we will have a handle of that function like you had in our producer consumer in here we will have a for loop so elixir has four loops they are a little bit different than four loops in other languages so as you'll see here we will say events and we will have a backwards arrow and we will say events will do this sort of like a for each loop in JavaScript or like a for loop in Python where you can say for I in you know our list for example we're saying for a single event inside of our list of events then do all this stuff and what we're going to do is we're just going to say i/o inspect so we're basically going to print to the screen and we're gonna say self comma and then comma okay finally we're going to return a tuple of no reply then a empty list and then the state so the reason why we're using this I'll inspect is so that we can actually see all the data coming through this consumer so our producer is going to produce an endless string of numbers these numbers are going to go into the producer consumer and then they're going to be filtered out into even numbers and then our consumer is going to print them all to the screen it's Annelle it's actually wire our application up with our supervisor we will have to define our worker for each of our processes we have three so we have our producer producer consumer and our consumer process and we can of course define more if we wanted to in multiple different processes for our first one we want to say Jen's counter producer and then we want to list this euro inside of it that's because we want this to start at zero so the initial value will be zero next one will be the producer consumer which doesn't have any state so we just pass an empty list to it and finally our consumer which again will be passed an empty one we want to keep everything else the same for what we had before so we're keeping this as one-to-one and we're just going to use this supervisor here we can jump into our terminal and we can actually run this and take a look at what's happening we're going to say mix run no halt and go alright so here we are as you can see it's counting up there's one process and I just kind of positive by doing that and it keeps going and it's going pretty quickly so this is a very simple gen stage example now we can do some pretty cool things with this so let's stop this for now let's jump back into our application and the main use of gem stage is to actually use multiple processes to handle the data at the end so we want multiple consumers we will define a second consumer in here and of course we need to differentiate these two so we're going to put in an ID here so for a first one we'll just call it ID of a and then for a second one we'll call it ID a B now if we run this I'm a positive as you can see the data is being processed by two different processes and as you can see the numbers are actually being processed in a weird order so if you remember in our producer I mentioned that our demand defaults to a thousand so each of the processes took a demand of a thousand so this process here the PID of 154 took zero all the way up to 999 as its demand do you think that as a set of data then that's the set of data that it's decided that it was going to go through our other PID 155 took the demand of 1,000 up to 900 our 1999 so if we continue to let this run and let it go past 1000 up to 2000 as you can see it reassigns how it distributes the data we got up to 196 198 processes 155 immediately started at 2,500 and process 154 started at 2,000 so it looks like the demand changed so in this case this one's only taking 500 demand and I'm not sure what this is taking until it finishes and this is probably because the CPU couldn't allocate more to these processes let's now assign three processes so if we create another consumer process we're going to call this one C and then we rerun our program PID one is taking from a thousand all the way up pide two is taking zero to a thousand PID three is taking 1500 all the way up to I am a semi 2,000 and because these processes are a very lightweight the more that we actually add the faster this will actually go so the thing to really keep in mind when dealing with streams of data like this is first of all when you want to use a distributed way of dealing with data you have to remember that the data will not necessarily come out in any particular order the other thing to keep in mind is that Gem stage uses a lazy method of evaluation so there are two main methods of evaluation for data one of them is called eager and the other one is called lazy eager means that you evaluate all the data all at once and this can obviously cause problems when you're dealing with really large amounts of data because you're basically saying okay just take all the data and do it at once so for example the way I like to think of this is like if you're playing a video file on your computer a video file doesn't need to buffer it doesn't stream anything it's just reading the file that's on your computer but if you're streaming something you're going to have a little bit of latency because you're streaming bits of data on your computer and that's a lazy method of evaluation so you're basically saying okay we're going to read the entire movie but you don't necessarily have to have the entire movie saved on your computer to view it so you can choose which piece of data you get to see at what given time that's basically what Jen stage is doing we're defining how much data we want to see through our demand and then we're allocating that to our processes all right guys well I hope you enjoyed this quick tutorial on Jen stage in the next tutorial I won't be trying to look at the register as well as some of the other cool features in Alexa that we have in gonorrhea and then we will be transitioning into Phoenix framework she enjoyed this video feel free to like and subscribe this video if you have any questions or comments feel free to comment and if you just liked it and of course by all means go ahead and download us as much as you'd like alright guys well I hope you have a good night