Create No-Code Predictive Dashboards using Amazon QuickSight and SageMaker - AWS Online Tech Talks

Key Takeaways

foreign [Music] Market specialist with AWS and I'm joined by Raji sivasu brahmanyam a Senior quicksite Solutions architect thank you for joining the AWS team to learn how you can create no code predictive dashboards using Amazon quick site and canvas all of that done with no code local development mode the idea is however raw your data might be you'll learn how to curate cleanse and prep your data using AWS data brew and then create a machine learning model on top of that data set using canvas before you can visualize the data in Amazon quicksite finally you will learn about deploying natural language query searches using quick set queue which will Empower your users to ask business questions in plain English and get instant insights we have an exciting agenda for today we're going to talk about the challenges that customers face in data prep analytics and we'll discuss the no code low code future View and we're going to look at how you can utilize local no cut approach with AWS Services we will also discuss key features of queue quicksite canvas and data proof finally we will have an interactive demo of these four services just to start um there's a recent study by Gartner that states that with the rise of digital transformation which has only been accelerated by the pandemic 88 of it leaders say workloads have increased in the past 12 months menu report an increase in demand for new applications and say that they're mostly concerned about the workloads and how this might stifle uh their ability to innovate and the same study predicts that by 2024 80 of tech products and services will be built by people who are not technology professionals so let's think of a common use case uh we see where you are tasked to empower marketing and sales teams and Telecom organization to be able to get predictive insights on customer churn and come up with marketing campaigns to proactively retain customers so you're pressed with the time and don't have the technical staff in-house who is skilled with all the technical skills set to achieve this um you also know that data is not clean and your business users don't have the technical knowledge to create the data you don't have a data scientist available to create ml model um and now let's see how various AWS services can help you achieve that goal here is the reference architecture that we'll be using first we're going to have a data set in Amazon S3 we're going to utilize we're going to connect to that through data Brew to S3 and we're going to utilize data Brew to curate that data we're going to clean prep and transform the data without writing any code and then we're going to export it back to S3 um from uh as a sagemaker canvas we're going to import that data set from S3 and in canvas we're going to train the ml model we're going to generate batch interferences download the data and Export test3 again then we're going to import that data into spice which is part of quicksite and it create a dashboard in quicksite before we create a topic on that data set uh utilizing Q let's start with talking about Q we'll take a deeper look into queue first IQ is a natural language querying capability which is part of quicksite and will let end users ask ad-hoc questions of their data it allows them to make sense of their data interpreting questions and then answering them correctly uh and that is probably the most challenging problem in this space Q is a machine learning powered natural language query capability built into quicksite it empowers any user to ask any question about their ba data in plain English the key customer problem Q solves is when a business user wants to ask a new business questions that are not powered by uh or answered by existing bi dashboard they rely on BI teams to create and update the data models and dashboards which really can take several weeks to complete this long lead time to address ad hoc bi questioning causes frustration for the business users and often it does put additional overhead on the thinly staffed bi teams so here's an example users is asking a simple question Q what is the week over week Revenue difference Q interprets that user question and the intent of that question it retrieves the data from the source and generates a quick site visualization with Q users can get answers in seconds because it uses the machine learning to automate the data modeling process when and when it's answering ad hoc business questions also users can ask any question on all their data Q is providing answers to question on all the data unlike conventional nlq based bi tools Q is not limited to answer questions only from a limit limited set of data in which data models and dashboards have been pre-built also the Q users are not limited to asking questions that are confined to a predefined dashboard and can ask any question relevant to their business um finally sticking to the no code local theme queue is easy to deploy and manage with a few clicks you can enable queue to begin answering questions for your teams and organizations now unlike conventional lq based guide tools that would need months to prepare and model the data queue automatically infers the semantic meaning of the data Maps out the relationship across tables and builds the required indices to enable accurate and consistent consistently fast responses all right moving to quicksite which is the first Cloud native serverless fully managed business intelligence and reporting tool with the with machine learning and AI capabilities what if you didn't have to have this sort of trade-off between ubiquitous access and prohibitive cost and to do that uh you really need to rethink to be our architecture from the ground up and which is exactly what we did with Amazon quicksite as a cloud native bi solution which is natively um so what does that mean to say uh Cloud native Cloud native is no servers or software to manage you can start with tens of users and scale up to hundreds or thousands of millions of users up and down with zero service to manage furthermore we are the only solution with pay-as-you-go pricing for readers so we think that's a huge advantage of a cloud native architecture but with ubiquitous access in mind the other part of the architecture is being first part uh first party native AWS solution this means that we're fully integrated with the rest of AWS we're going to have AWS great security and compliance and end-to-end encryption furthermore you're going to have the centralized IAM permissions with fine-grained access control and cloudtrail logging for audits yes with quicksite you can create an interactive dashboards and visuals that you can embed anywhere internally or externally you can add Rich interactive filters and drill Downs with no code you can access it from any device you can automate the data refresh and you will always have blazing fast navigation the third piece of the architecture is the visual machine learning service sagemaker canvas it's a serverless visual no quote machine learning service that will allow business user to automatically clean and combine your data create hundreds of models under the hood select the best performing one and generate new individual or batch predictions all done with no code sagemaker canvas was created to empower the non-technical business users it allows business users to build ml models and generate accurate predictions with no code required users can access and prepare data for machine learning they can also utilize built-in automl to build models and generate accurate predictions they can share ml models and collaborate with data science teams uh sagemaker canvas also utilizes usage-based pricing to avoid licensing fees and reduce the TCO but sagemaker canvas utilizing no code at all you can combine data sets from various sources like local disk Amazon S3 Amazon redshift and snowflake and we're constantly adding different sources data sources to add as well users can quickly understand and prepare their data via visual interface which is extremely easy to use and within second within minutes they are able to get their first machine learning model they can review Advanced metrics and feature importance to understand and explain those predictions now for the final piece of the architecture is the we have data Brew which is the visual data preparation service it's a server serverless visual data preparation service that will allow business users analysts and data scientists to visually explore and experiment with data independent independently without writing any code with AWS data Brew end users can easily access and Visually explore any amount of data across their organization directly from their Amazon S3 data Lake Amazon restrift data warehouse Amazon RDS databases to name a few sources only customers can choose from over 300 built-in functions to combine pivot and transpose that data without writing any code data Brew recommends data cleaning and normalization steps like filtering anomalies normalizing data to standard date and time values generally generating Aggregates for analysis and correcting invalid misclassified or duplicative data for example tasks like converting merge to a common base or root word it could be an example of converting yearly and year long to year users can save the cleaning and normalization steps into a workflow which is called recipe and they can apply them automatically to Future incoming data if changes need to be made to the workflow data analyst and data scientists simply update the cleaning and normalization steps in the recipe and they can be able to automatically apply those to new data that arrives as well users can visualize the data lineage and integrate with data Pipelines and summarize data Brew is a serverless and fully managed service so customers never need to configure provision or manage any compute resources most importantly data Brew is for users of all technical level and you can apply all different steps with using no code at all I will pass to Raji who's going to lead you through a demo of all four services thank you Sam now let's consider the use case for the demo let's say that you are working for ABC telecommunications that provide services to its mobile customers now your sales team observes that there is a problem with the sales number and they would like to run analytics and see whether there is any problem that exists among the customers and marketing team also wants to collaborate to see whether they could run insights and come up with a campaign to Target the customers who might be at risk with that let's say that you have this data which has a customer information as well as the customer usage stats and now you you are planning on coming up with the insights as well as an ml prediction model to know which customers are going to churn out so that you could Target them you know that you have shorter time to develop and you don't have enough technical staff in-house to do all the activities you need and the data you have requires some curation and then you know that you don't have resources who have ml experience with this let's see how AWS Services can help you build a solution for your users for this demo we have the data in the S3 bucket and you know that this data needs to be cleaned and we are going to use blue data Brew to curate this data and the output of the data Brew will then be used within sagemaker canvas to come up with the customer prediction model and then the result of the sagemaker canvas will be used to provide a visual dashboards to your customers then on top of the visual dashboard you would like to provide a natural language search capability that your raw data is in this folder now let's go to data brew and see how you could curate the data first step in data Brew is creating a data set pointing to your source our source here is S3 so let's create the data set and let's name it as churn prediction source is S3 let's point to the S3 raw data and this is the data file we know it's a CSV and we also know that there is a first row header and now let's create the data set and you see that data Brew created a data set here with data Brew you could do two things one you could come up with a data profile to see how clean your data is and then you could do data transformation to curate the data I have already created a data profile for the data we are using so just to have a quick look at a high level this will give you the total rows total columns in your data and different column types as well as the missing columns duplicate rows and correlation between different columns as well as the value distribution across column and then column level summary and then the column stats is the tab we are interested in for this exercise which will give a little more detailed information on column values in your data file here let's pick a phone number column you'd see that the sample phone number is like this you know this is a pii information and for this task you would like to mask so if you want to create a transformation to mask this data and the other one you'd like to do is clean up some of the data for example churn has special characters like dot in the value which you would like to clean up so in order to do this Transformations within data Brew we have to create a project with the data set so let's go and create a project let's call this as John prediction and in data Brew which transformation is called recipe and we are going to create new recipe and I am going to use the churn prediction data that we created a couple of minutes ago and now we have to pick the role to run this particular job and I'm going to use the same role I used to run the data profile and now I'm ready to create my project at this point data Brew is provisioning the compute capacity for doing the data transformation and this is going to take a minute or so now our project is ready to create the transformation let's start with the first recipe click add step and you can create a recipe here or you could directly go into any of these functions and create a recipe we are going to redact the phone number so we are going to use this function and redact value and for the source column we are going to pick the phone number and I'm going to use the found sign as a redox symbol and then I'm going to keep the data format as is but I'm going to redact the whole string in all rows if you want to preview the changes you could click this and data Brew will show the original column and the new column side by side once you confirm that your changes are good you can hit apply and at this point data Brew creates what we call it as a recipe for doing this transformation and you see that here let's add one more recipe to clean the churn data do it here special characters then for churn column we want to remove dot so I'm going to use custom special character and Dot over here okay I'm going to apply it all rows and then say apply this transformation you see how easy it is to apply different transformations in data Brew without writing even a single line of code now you can use all these different functions we have over 300 built-in functions that you could use to transform and curate your data now this particular transformation has been applied to the sample data in order to apply to the entire data set we have to create a job and this is a recipe job we are going to use S3 location for output and I'm going to use a different bucket for this I'm not going to change any of the default setting for our demo and but I'm going to pick the role that we used before and now I'm ready to create and run this job at this point data Brew is applying the transformation we created for the entire data set and this is going to take a minute or so now the job has finished and we can see the output of the data Brew in the output folder here so this data is now curated for the two columns we were fixing and now this data is ready for sagemaker canvas to use intern prediction now let's move to Stage maker karvas and if you are logging into sagemaker canvas for the first time you need to set up some initial domain and that can be done by using the Quick Step In This stage maker Studio and you have to set up a domain the second one is you have to set up an IAM trust policy to allow Sage maker and then you have to apply course for the S3 bucket I have already done so and that's what you are seeing here and now I'm ready to create a model in sagemaker canvas for that I have come here I have to apply open canvas and the first step within the sagemaker canvas to do ml model is create a data set and we are going to import the data set which was created by the data brew as an output in S3 you can see that you also have an option to upload a file from your local computer or use data from redshift or snowflake so for our exercise let's use the data Brew output and it's in this particular S3 packet and I have it in the transformed data folder and this is the output of the data once I hit the import data you see that is being added and it's still processing and now it has completed processing it the data has about 21 columns and 5000 rows now we are ready to create a model with this data within sagemaker canvas in order to create the model you can go here and say new model and name the model let's do jump prediction create you see that there are four steps involved in it and the first one is picking up the data set and we are going to pick up this data set say select data set the moment you say select data set you can see that sagemaker canvas automatically moves to the next stage which is built at this stage we have to provide the target column for the sagemaker canvas on our Target column is going to be churn basically we are going to see whether a customer is risky or not the moment I picked the target column sagemaker canvas recognize that there are two values true and false and it also recognized that it has to apply to category prediction or binary classification model to this particular data set in case you want to change the model type or if you think that the sagemaker canvas did not pick the right model you can change the type by clicking here and picking the model you want sagemaker canvas to use ours is going to be a binary classification we are going to leave it as two category prediction and now over here also you could do a minimal data transformation but we are not going to do that for this demo and here you can do preview model and at that point you get the model accuracy it will take a minute or so but it will give you the model accuracy or you could go directly to build the model there are two options to build the model one is quick build and then the standard build standard build would take anywhere between two to four hours because it's going to use extensive models behind the scene and also more hyper parameters to come up with a better accuracy over speed on the other hand quick build will use a limited number of models as well as few hyper parameters to come up with the prediction model and with quick build you choose speed over accuracy and for our demo we are going to use the quick build option and now you see that preview model has already returned the estimated accuracy to be 95.8 percentage now let's do the quick build over here I'm not going to validate the data but start the quickly right away the moment I start the quick build you see that stage maker canvas move to the third stage which is the analyze this is going to take anywhere between 2 to 15 minutes to generate the model for us now the build is complete and we have the estimated accuracy of 98 95.8 and also the column impact for each of the fields within the data we used if you see here a different column have a different kind of impact and you want to choose the hyper parameters for the columns that are not being impacted heavily so that you are not losing the target predictions when you are running the model on the real data okay and now we can move to prediction and in the prediction you could either do batch prediction or single prediction and if you do bad single prediction it will do row by row whereas the batch prediction will apply on a bunch of rows at the same time and for our demo we are going to do batch prediction and then again we have to select the data set in our case we are going to use the same data set but in Practical scenario this will be your test data set and when we say generate prediction this part will be quicker for us because we use the same data set to train the model as well as to predict the model now the model result is ready for us to view and we can click here and then the data can be downloaded I'm going to download it to my local computer and then I'm going to use this particular file in my next step which is building the visualization with quick site now if you see with sagemaker canvas I am able to generate a prediction model for customer churn without writing a single line of code now let's move on to creating visualization based on this data set using quicksite I am in a quick save console and the first step again here is create the data set using the data source you have multiple different options to connect to the data sources and in this case I am going to use upload a file option and upload the data file that has been downloaded from the sagemaker canvas just now foreign what we see here is the preview of the data from our data file now I am going to edit and prepare some data before I go into the dashboard I'm going to use a calculated field and create one for potential I'm going to say that if a churn is false count it as one otherwise no the moment I hit save this particular potential churn column will be added to the list of fields in the data set you can also see that quicksite automatically recognizes the data type that is there in your data file now another option you see here is augmented sagemaker this is used if you want to connect quicksite with the sagemaker model in real time for this to work in you have to create the sagemaker model with the standard mode and then you have to publish that model and create an endpoint to that model once that endpoint is available to Quick site you can use augment with sagemaker within quicksite to connect to the sagemaker endpoint for that model and then you can connect with the predictive data in real time and now uh I'm not going to do any more changes I'm just going to save and visualize this now the data is being loaded into quicksite spice this should be very quick but in the meantime what you see here is all the fields in the data set that we were using and different types of visuals that you could do with the quick site and then options for creating visualization within quick psych over here and then there are drop down menu option to create different visual items within your dashboard now you can see that the data is being loaded into spice and we can start building our dashboard I'm going to create a very basic dashboard with maybe three to four visuals the first one I'm going to do with the churn and the cast call and I will create a donut chart for it and then let me add one more visual since we are analyzing the customer churn I'm going to have a horizontal bar for churn by state let's say that I want the state to be in the y-axis and chart to be in the group or the color so that I get two bars for each stick and now since we are talking about State let's add a visual for a map I'm going to use the field map and I'm going to use State for the location and then we added a calculated field called potential churn I am going to use that for this one okay now I'm going to format it a little bit different than the default one because I'm going to use this in my next step with a natural language query search I'm going to conditionally format it based on the value in the particular potential churn field I'm going to say if the value is greater than 50 I'm worried that the customers fall in that range would be churning off so I want to put it in red color and then for customers between 40 and 50 let me say that I want to pay attention to them so I will make it as Orange and then for anybody below 40 let's say that uh we will leave it green this would give a visual representation for the marketing team to Target their campaign to retain their customers and now let's add one more object before we proceed to the next step let's do a data grid and I'm going to have a basic table for customer information churn as well as the calls they make during the day evening and night all right now we have a decent uh information in the dashboard this will give a quick insight into the customers that are churning and then by state how how the churn is looking and then what are the risky states that uh marketing teams should aim their campaign for and then detailed data about divisions itself now let's say that this dashboard is ready to be published we'll publish it we'll call it as churn prediction they publish the dashboard you could share this dashboard with the user base and let them use it but what if you wanted to give them up platform where they can go and query on any insight not just the visuals that you have in your dashboard and that's where the natural language query search capability of quicksite comes into picture for natural language query search we call it as a topic and each topic within quicksite Q is going to focus on the subject area you want your customers to get answers from and to create a new topic you just click over here and enter the topic name for this one as a customer description is optional I'm going to leave it blank now I'm going to use the data that we used for our prediction and as well as in the quickside dashboard then create the queue topic so currently Q is indexing the data fields and um it is going to create the index to make it more searchable when the user is used querying the data using the natural language while that is doing it let's have a quick look at what is happening in here this is where you could find the statistics on the Queue topic usage itself uh and then here is where you would get the distribution of feedback whether for a particular questions you got a positive feedback or negative feedback or no feedback at all or whether any question was not answered by queue itself and then you can look into this one to get into the details of the data that was being used this data set is the quick side data set that we point to the queue topic to and this is an important step in the quick set Q topic to curate and fine tune the topic so that the user has best experience using this in the first step is to exclude any columns you don't want queue to use in answering your user questions for example here we redacted the phone number we know that it's not going to help in any of the answer so we can say exclude and then we can also um see whether these names are business friendly if not we can change it right away for example here there is account length you know this is nothing but the account duration so you can change that by simply editing here changing the length to duration and sometimes your business users might call one field in a different way and that can be achieved by adding synonyms to this particular uh any particular field for example churn could be customer churn and let's see what we could change this could be a customer saved and so on and then other one you other thing you could do with this is you can add a calculator field similar to how we added calculated field in quicksite itself and then you could also add multiple filters if you wished within the queue topic uh with this is also similar to a quick set dashboard creation and named entity is something that is different in queue uh this will help you to group multiple fields to answer for a single question and we will create one for our topic you say this is customer info and we can add a synonym to be customer details and then we can add the fields to be state area code and then maybe one more let's say a conjuration save this so once you do add any named entity you'll see that up here at the bottom of your field list and whenever you use this particular named entity in the answers you would get the information on all the columns that's added to this named entity now let's say that we are ready to do a few questions on this particular topic and see how your user experience would be so let's start with uh show me the chart by state because we are analyzing by jio you see that um Q is taking the user type requesting and restating it to the field here I have made a typo and churn but Q is smart enough to map that to the turn field in the data and its mapping state to the State field and now you can change it to different uh Dimension we did add account duration let's see how who is responding for account duration again I made a typo but you recognized it and it is using the account duration field from this list we also added a synonym for customer sake so let's do um count by return count by customer served now you will see that is mapped to Custom Calls field the reason is that in the queue topic here we have made a synonym for custom Services uh cast save calls to be customer served and now let's go back to analyzing the churn count by state and now if what if your user didn't like this particular visual by state um instead they want a map they can change the visual type by simply clicking over here and picking the visual they like to choose now you see that a quick set you was able to take that value and plot it in a different states and now what if your user don't like this but wanted to use the visual that they created in the dashboard a while back they can easily link a particular answer to a visual in a dashboard by clicking here and selecting the dashboard sheet and then the individual visual itself this is what was the map we created in the dashboard with color coding thresholds let's use this one for this particular answer now as an author What hap what user did is they took a visual created in the dashboard and linked it to a question within queue so next time when a reader comes in or anybody questions um cue with the with this same type they will get the map visual that's been linked and now you can add more variant to the same question if you wish we also added a named entity so let's uh go back to the named entity now let's use the named entity called customer details by um potential turn you see that we created in customer details with three different columns and Q is able to map that customer details from the question that user asked to the named entity within the queue topic and brought the uh data grid with those three columns now you can also provide feedback to queue as whether a particular question was answered correctly or Not by giving thumbs up now if you if um you want to monitor what users are doing with the queue topic you can use the user Activity part of queue and you can see how many questions were asked and how many questions were answered by q and what was the feedback positive negative and if there is any question that's not answerable by queue and you can also see if there is any question that's been reviewed by the author or not as well so this actually concludes our demo where you saw we started with the raw data and we used data Brew to clean up the uh couple of fields in the Raw data and then we used sagemaker canvas to run the ml model to predict customer churn and then we use the output of the sagemaker canvas in quick site to create the visual dashboard we publish the dashboard and then on top of the dashboard data set we created a natural language query search capability called quicksite queue with this you can see that you can Empower your end user with the natural language search capability right from your raw data without writing even a single line of code and thank you for your time [Music]