Yiwen Li - Intro to Elyra - an AI centric extension for JupyterLab | JupyterCon 2020

Key Takeaways

hello everyone today i want to bring you intro to envira which is ai centric extensions to jupiter my name is even and today i have my colleague here and edward and stachoozi i'm a data scientist at ibm codex team hey everybody my name is ed and i'm also a data scientist on the kuday team hello everyone i'm sashathi i'm a developer advocate and data scientist in kodai team and first of all thank you so much for joining us today and i hope you are having a great time in the conference so let's quickly go over today's agenda starting with our team's introduction we will introduce you to two of our open source projects elira and data asset exchange and end with a demo showing how to use them together with reference link that you can take back from the talk so kodaite center for open source data and a technologies we are 30 plus developers and data scientists working on contributing to open source projects that covers entire data science life cycle as you can see on the screen on the right side of the slide this is the projects that we are contributing our goal is to make ai systems easy to create deploy and maintain today we are going to focus on two of our in-house open source projects first is elira our nexus data asset exchange and they both are marked on the slide here and now over to ed and he will introduce you all to elira so what exactly is a lyra well at its core allyra is simply a set of lab extensions catered to people who like to work with artificial intelligence and machine learning using notebooks our goal is to help data scientists machine learning engineers and software developers through the most common model development life cycle complexities alara was officially announced as an open source project by ibm this past april if you'd like to access the github repo or the docs you can follow the links here we've shared on the screen the lyra boasts a host of features to achieve its goal which we will discuss shortly for now notice from the preview of a lyra's ui here on the right that a alara doesn't radically reinvent jupiter lab to provide you with all of its functionalities it simply extends the distinct jupiter lab environment that we've all come to know and love over the past few years added tabs toolbars and launchers so what can actually do for you well you can really break down a lyra's core features into five items the first alara provides an easy to use notebook pipeline visual editor why build your next ai model using a pipeline editor well it's because a lyra provides a visual way to convert multiple notebooks covering pre-processing steps experimentation optimization and deployment onto batch jobs or workflows second allyra supports rounding notebooks as batch jobs directly in the ui making model training easier the iris supports easy creation and insertion of reusable code snippets fourth allyra pipelines support git version control allowing rollbacks to working versions of the code backups and most importantly sharing amongst the members fifth and finally are exposes python scripts as first-class citizens this allows users to locally edit their scripts and execute them against local or cloud-based resource seamlessly even will demonstrate all these features for you shortly in our demo but for now i'll pass it back to shruti to talk about ibm's data asset exchange that's it so let's have a quick overview of data asset exchange we called as tax so tax offers high quality wetted data by wetted i mean we start by tracing origin and merit of the data set learn about the usability rights and ownership from here we take all the information and we create a standard metadata and perform internal legal review before releasing the data sets to data asset exchange data sets here have clearly defined open data license we provide exclusive access to ibm research datasets that have played crucial role in building popular ai systems like ai debater entity recognition and so on so what do we offer do we offer only data sets no not really we offer much more than just raw data set these data sets comes with tutorials that demonstrates the usage of data and can be directly exported as watson studio project or a watson studio compatible notebook we also have data glossary that that will be that you can use for learning more about the attributes of the data set we also have an option to preview the data that can help you understand your data before using them in your project so this is our watson studio project so you can directly use all the notebooks in your watching studio project industrial use cases have been created using dax datasets and they are available as industrial accelerators which you can directly use with cloudback for data we have resources listed in slides that you can refer to know more about it but now over to even for the demo thanks sashusi so before i show you the demo i want to introduce you how the data science process look like so majorly a data science pipelines generally include five steps from data extraction to result interpretation as a data scientist i have to run multiple notebooks be one project especially if i change anything in one notebook then i have to run the other notebooks one by one it i will save the efforts by constructing them as a pipeline and run them all with one click so now let me walk you through the demo so this demo will show you by running ford notebooks using a lyra i will introduce you two ways to run your pipeline both on local machine and on coop floor pipelines to build a pipeline you can simply drag and drop the notebooks onto the canvas and connect them just as easy as drawing a graph you can arrange the notebooks in sequential or parallel order need to configure the notebook properties for several informations we put the environment variable dataset url as a dataset download link from dax we copy paste the link into the section this notebook produces output files i specify the file name as jfk weather.csv which is download through the first notebook into the directory the output file uploaded to cloud storage were saved in local directory after notebook processing complete we also need to select a docker image that will be used to run a notebook you can bring your own image or choose from the predefined public images such as a panda image a tensorflow image or pytorch image we choose pandas as a data image here because we mainly use pandas and numpy packages in the notebook we could only select one docker image each time however if you're running on your local machine the image you choose does not matter also you can declare the file dependencies as jfkweatherclaim.csv but this is not necessary since the files are already in the same folder i redefine the name of the pipeline by right-click my trackpad change the name to even dot pipeline also you can add comments to provide short descriptions this helps your colleague knows the function of each node before going into it now everything is all set let's save the pipeline and submit it to run locally the running logs are shown in terminal if everything completes successfully a message will pop up the run outputs will be shown in the notebook cells see on the left side the notebooks are just updated a few seconds ago now let me quickly walk you through the content of each notebook firstly the load data notebook downloads extract the zip file through the dataset url link and save it as jfkweather.csv file the part 1 notebook loads the data set downloaded from the previous notebook replace wrong values such as non t 0.02.01 second filter out everything out of the range reverse the column into numerical type clears out the missing values rename the columns and then save the clean data set as jfkweatherclaim.csv file the part 2 notebook selects five columns from the clean data and visualize the trend it also explores the dependencies between those columns then visualize the trend of rolling average in 2017. the parts3 notebook let's explore the approaches to predict future temperature by using time series data set creates the training validation and test splits train the data set and then compare the performances between different baseline models using mean squared air and it build arima model predictions for the first 48 hours of validation set you can get this notebook on the asset exchange page by clicking on the preview notebooks you can see we have street notebooks very similar to what we have in the demo pipeline the notebooks were designed to run on whatsapp studio and ibm cloud i made minor changes to those notebooks and to fit this into the jupiter environment so now let's submit this pipeline again on cool flow before running you need to configure your runtime by putting quidditch into each section i already pre-configured my runtime so we can just submit the pipeline on group flow in the run process eliro will generate gather and package the required artifacts upload them to cloud storage and triggers the pipeline execution in a selected crypto pipeline environment pipeline runs are listed in the experiments panel the graph panel displays the execution status of each node you can see the part 1 notebook is completed and has green check mark at the right top after all run is complete you can access the pipeline's output artifacts using the supported s3 client clicking on data folder which includes the jfk jfk data file and also the clean version and also on this page we have all of the inputs and output files listed all terribles are input files by clicking one of the output files and download you can see the result of the notebook is saved in the output cells is the demo i i would like to show you today and you're if you're interested and want to try this by yourself please feel free to install your lyra and clone my library examples repo to get the notebooks you can also get the notebooks on data asset exchange and run from there elira is an open source project so please feel free to open issues or enhancement requests on the elia repo there are also a few other talks related to elira from ibm code team please also feel free to check this out and this is everything i want to show you today and thanks for coming