TamNguyen- Handling Custom Jupyter Data Sources | JupyterCon 2020

Key Takeaways

hello jupitercon my name is tom also known as data engineer one and today i'm going to be showing you guys how you can use kedro and jupiter notebooks pedro is a framework built by quantum black geared towards data scientists and data engineers to help them write better data pipelines oftentimes what happens is a lot of data sciencing and data engineering work is collated inside of notebooks which are very useful for those experimentation and iterations but can fall away when you want to try to reuse that pipeline you want to try to move that pipeline or you want to try to understand that pipeline a month down the line in that process of finding out how best to collaborate between data engineers and data scientists pedro was born pedro is a scalable modular pipeline building framework in today's video we're just going to be talking about a single jupiter notebook and turn it into a kedro pipeline this notebook in particular comes from the british natural history museum we're going to be taking the apis that are available and downloading the data and then creating the transformation nodes that we would like to reuse inside of ketro and we can go ahead and get started by cloning example now that that's done let's go ahead and install kedrow now to get kendro you just type in pip install dash u and drone android comes with an awesome cli that we use in order to interact with our quetro project to create a new project we type in kedrow new let's go ahead and just name this on example for now we can go ahead and hit enter through all of these we can open up that project i'm going to use pycharm pedro comes with a standard template for data pipeline projects the most important folders for our exercise today are going to be the conf folder the source folder and the data folder the conf folder holds configurations for your data pipeline and i think that's something interesting because not many people realize or think about how they hard code configurations inside of their jupyter notebooks often it's just simple parameters but it also can include paths to data this creates a barrier that prevents an individual from copying that jupyter notebook and pasting it somewhere else because that means you also have to paste a lot of these data paths exactly in the same way petro abstracts this with its catalog which we'll be investigating in just a little bit the next folder on this list is the data folder now this is an obvious folder it's where we keep some of the data metro comes with a simple structure for designating your different tiers of data we're going to be using this folder to store our data but it is not required to store your data in this folder to use kedro finally the source folder is where all of the kedra pipeline code is going to be sitting as you can see here the con example project name has been populated inside of the source folder and this is where we're going to be putting a bunch of our pipes very quickly i'm going to explain what i mean by data pipeline in kedrow now if you'll take a moment to take a look at this graphic here i have an example circles to triangles pipeline this pipeline consists of four elements the circles data set is an example data set which contains a bunch of circles right you can consider it as a row of circles next we have in here the center the circles to triangle node transformation function this is a transformation function that will turn the nodes into triangles next we have a triangles data set and this captures the output of our circles to triangle's node transformation function finally the fourth element here is the pipeline itself so the pipeline is what contains each of these elements the two data sets on our transformation node and is responsible for extracting data from the data set and then passing it into the circles to triangle node then it is also responsible for extracting the data from the circles to triangle node transformation function and then inputting it into the triangles data set so these four elements are the basic elements of a ketro pipeline and these are going to be the elements that we'll be using today for our notebook conversion now that we have created a catcher's project and we have a copy of the notebook in question into the notebooks folder now making sure to source the virtual environment i'm going to cd into our con example project and type in pedro jupiter notebook pedro comes with a jupiter notebook integration it makes it very easy to operate pedro pipelines as well as jupiter notebooks simultaneously navigating into our notebooks folder i'm going to go ahead and open this notebook and we'll see here that there's actually a lot of libraries that we also need to install sourcing bin activate we're going to do pip install dash you oleum shapely numpy pandas matplotlib requests next we're going to install a kedro library because of how many different ways you can connect to different data sources in kedrow the ketchup team has split up the kedro library into several distinct chunks and one of those chunks is the kedro pandas chunk this will allow us to read and write data using pandas all right let's go ahead with the very first portion here there's a web api for the global biodiversity information facility that allows us to gather data for particular years let's go ahead and gather data and copy the url very simply i've just imported requests and i'm using the request.get function now that we have access to this object i'm going to go ahead and just write it directly to disk into our data directory at zero one raw and we're gonna name this field data.json now that we have that initial data let's go ahead and use ketro to open it up in order for kedro to be aware of this data we have to create a data abstraction this is known as creating a data set inside of the catalog so here i'm going to create a new key inside of this yaml file called field data the type of which is going to be a json.json dataset and the file path is going to be the relative path to the o1 raw folder field data.json file so now we've added this data set to the catalog this means that kejrow has access to it there's a few special objects that the jupiter notebook ketchup comes with we have the context object which just gives us the context of our project and the catalog which is the object that points to our ketchup catalog if we do catalog.list we're going to see nothing right now because we just added this but using some jupiter line magic reload kedro we're going to reload all the ketchup configurations and when we run catalog that list one more time we're going to see field data appears calling catalog dot load for the field data data set we'll find our json file the reason why this data abstraction is important is because if you want to move your jupyter notebook you no longer have to worry about making sure that all the file paths inside of that jupyter notebook are correct you can simply use the ketchup catalog's data abstraction to refer to the file that you need and then in the target machine or the remote machine just change the catalog.yml file to point those data sets to the correct file paths and you can start to imagine how portability can be achieved using catro in this way now scrolling down a little bit more we see that there's actually a cleaning function here for every result that comes back in those pages we're filtering those results to make sure that decimal latitude exists and turn it into a pandas data frame this is going to be the first function that we create it's going to be a cleaning function let's go ahead and name this clean raw field data and now we have a data frame from the clean raw field data function and here we see our new data frame with the clean json data but what about kendra where does kedra come in now well here's where it gets really interesting going into view i want you to go to cell toolbar and i want you to allow tags on this first cell with the function i want you to type in node and add that as a tag then save now going back to our terminal i want you to type in kendra jupiter it has three commands the first command convert what this is going to do is it's going to go through all of our notebooks that are inside of our notebook folder and convert any cells that we have tagged with node into a kedrow node we're going to go ahead and type in kedro jupiter invert dash dash all and you should see this converting notebook message if you don't see this message make sure that you've saved your changes with your tags in your notebook now what this has done is created a brand new python file located inside of the nodes folder going back to parcharm here is our python file created from our notebook let's go ahead and just rename this guy bnhm now that we've created a function to transform our json data into a pandas data frame let's create a function that'll plot our pandas data frame the function i'm going to write is very simple it's going to be called plot field data it's going to take in the field data data frame and it's going to return a pi plot figure and in true pandas fashion we've written a very long line let's go ahead and try this guy out and here we get the plotted field data now let's go ahead and tag this as a node and then rerun our conversion function and thankfully we're using pycharm because here as you can see i've made a typo great now we have a field cleaner as well as a field plotter let's go ahead and create a brand new pipeline now by default the pipelines are registered inside of the register pipelines function inside of the hook stock pi in the root of your project module in order to write our pipeline it's actually very trivial we're going to go ahead and take advantage of the current default pipeline that's available to us and in here we're going to add a node and we're going to import it from kendrick pipeline we're going to go ahead and use the clean raw field data function as our first argument now the next two arguments which are required by the node are the inputs and the outputs now notice here that we're taking advantage of the abstraction and this field data is pointing to a json file finally for outputs we're actually going to create something arbitrary intro is going to automatically turn that into an in-memory data set now the next node in the pipeline that we're going to create is going to be our plot field data function and then the inputs is going to be the output of the previous one and for the output field data plot going back into our catalog yaml file we're going to call this field data plot the type of which is going to be the matplotlib writer the file path is going to be data08reportingfielddata.png now with these two nodes we've actually successfully created our entire kedro pipeline opening up the terminal again we're going to type in kedro run and here we have a very convenient log output showing us kedra went through when we open up our data folder opening up this png file will give us exactly the png that we saw inside of our jupiter notebook thank you for joining me on this jupiter con video if you'd like to see more videos like this make sure you check me out on youtube at youtube.com dataengineer1 and if you have any questions feel free to hit me up on twitter i hope you guys learned something today and i hope you guys have a wonderful rest of your jupiter conference take care and i'll see you guys next time bye