Serverless Data Integration for a Modern Data Infrastructure with AWS Glue - AWS Online Tech Talks

Key Takeaways

[Music] welcome everybody to our discussion on serverless data integration my name is zach mitchell i'm a big data architect with aws glue in aws lake formation and today we're going to be discussing glue specifically our learning objectives today are getting a brief overview of what glue is just to refresh everybody's memory discussing glue and how it applies to different users discussing glue for different workloads and how you can work with glue for varying workflow types and then discussing how glue can meet the requirements of your scaling needs we understand workloads aren't always the same size and so let's discuss that and finally we'll discuss glue centralized usage of a data catalog so what is aws glue as as an overview glue is a fast open scalable integration service as it says um we are a fully serverless data integration service built on the foundations of apache spark in python there is no servers or compute to manage it scales as you need it and you pay for what you use the key capabilities of glue include our scalable data integration engine uh with our built-in transforms designed for simplifying uh data integration tasks um again it's our serverless execution engine that is only charging you for what you need and it's the ability to monitor all of those jobs in a centralized system and easily identify and troubleshoot issues we have a centralized and unified data governance strategy that centers around the glue data catalog that we'll get to in a little while we have glued crawlers which simplify the ability to get data into the catalog and to have all of your data represented therein and we have fine grained security and access control by aws lake formation glue allows you to connect to and ingest data from hundreds of different data sources we have native connectors from the most popular systems like redshift over on mongodb and we have a connector marketplace for things such as sas providers like salesforce we also expose an api to allow you to easily build your own connectors through your own custom interfaces glue is not persona specific and we're going to get into this today ideally quite deeply glue is designed to be used by a variety of people to explore data to help generate code when necessary and to allow you to really dive in when you want to get your hands dirty it allows you to work together and collaborate together even when you have a variety of skill sets within a team so let's get into the data integration overview first because we we can describe what glue is but let's take a step back and describe what data integration is and why it's necessary data integration is a loop it's the process of discovering your data and combining that data transforming it and cleaning it such that it makes sense for your business so you can get quality you can run useful analytics you can build your machine learning models so on and so forth it involves centralizing that data and that data centralization usually revolves around the catalog i need to know what data i have i need to be able to move that data from its original source and its original shape i need to be able to change it to fit a different source in a different shape and that of course requires me to be able to connect to those sources and those destinations however data integration is extremely hard so if you don't get it right you're going to get incorrect results the problem is you want those results because those results provide meaningful insights they provide increased collaboration once you realize that hey two teams are likely doing the same thing and these all lead to faster decisions as you increase that collaboration and you gain faster and more meaningful insights so why is it hard the first reason it's hard of course is because data is growing and changing every ever more is growing rapidly and exponentially data sources seem to crop up all over the place when you think you finally understand when you're where your data is somebody brings in a new data source and you have to deal with it and with increased diversity of sources comes a diversity in the style and type of data no longer is all data sitting in a relational table in a database now you've got data in flat files you've got data in semi-structured stores like open search and even graph stores like neptune all of these have to be dealt with typically in different ways you've got different personas who are now dealing with data no longer is it simply one data engineering team that produces reports about the data you now have personas at the business level that simply want to get into the data with no code or very little coding experience you have developers again that original central team that still need to do their job and still need to do the deep dive and tinkering with the data they have and then you increasingly have purpose-built users of data analysts and scientists whose sole job it is to go and look beyond the data that currently exists and derive again increasingly diverse insights from that data you have an increased number of applications interacting with your data as well more and more of those applications are becoming real time or near real-time and therefore are very sla sensitive and waiting a day or two or even an hour for data is no longer acceptable your applications are scaling ever more rapidly and of course as you're trying to balance all of these these demands of time sensitivity and highly scalable nature of your data you come up across budget constraints because these things cost money and so how do you balance the needs of your applications with your budget well the problem is traditional solutions aren't suited for this you don't have a scalable infrastructure your infrastructure back in the day was complex to install and to maintain and was often very very rigid that rigidity came with a high cost both the people cost and having to deal with it as as well as a cost for you know extra licenses for advanced functionality for other transforms centralized cataloging you seem to need a different piece of software for each piece of your of your data infrastructure and a lot of times that was all based off proprietary tooling that locked you in and you were stuck your data was in and you can't go anywhere because your data is sitting in that engine it becomes a critical component of your business so against all this how do you get all of this data into the hands of the right people well that requires the right tooling so let's get into the tooling that glue provides typically in one of these presentations i would actually go the simplest tooling to the most advanced but today i want to work a little backwards to show you the simplification that can occur when we go from the advanced tooling that's built at a really low level to the simpler tooling that is built in in toward no code personalities so our advanced tool and our most modern tool is called glue interactive sessions with interactive sessions we provide an interactive data integration experience we provide interactive development include jobs and interactive data exploration interactive sessions replaces glue development endpoints if you've used those in the past interactive sessions provides the rapid creation of serverless spark you are able to launch an interactive session within about 30 seconds from the time you decide you want to go work with your data you can configure and install any packages you need inside glue's serverless environment right from the same place and every time you go to work with interactive sessions you get dedicated resources you're not sharing a set of compute with your neighbor or your colleague you're not sharing a set of compute with anyone else it is dedicated to this specific task of your session it's important to note that interactive sessions is extraordinarily cost effective you literally only pay for what you use interactive sessions aggressively clean themselves up and time out so you're not wasting idle compute and all the billing is per second the main interface into interactive sessions is a jupiter kernel it's easy to install on your local macbook or pc so you can run it locally you can run it in the cloud either on sagemaker or in glue studio and because you can run it locally you can connect it to your favorite ide so let's go ahead and dive in real quick and let me show you a demo of this kernel now this particular demo i recorded a while back uh but i recorded it on an airplane at 40 000 feet with no bandwidth because i was on an airplane but i wanted to see if i could use interactive sessions from anywhere and specifically in the use case i wanted to go pull up a data set in the glue data catalog but first i need to configure my session sessions are configured via jupiter magics magics are simple at commands that are prefixed with a percentage sign and allow you to specify the configuration of glue and any setting you want in glue if you need to remember what magics are you simply call help to see what you can do now back to what i was doing so in sessions i was looking for a table in the coven 19 data lake when i was doing this particular demo that's right i was looking for the county populations table now that i remember the table name i can go look at the data so i'm going to use an sql magic to select from the county populations table and look look at the data sure enough there it is so now i want to see if i can find the largest counties by population so i'm going to go again run that query again this time sorting by population estimate once i've sorted by the population estimate you can see that the sorting is not correct um in this case it looks like the population estimate was a string and so we're going to import glue and we're going to use glues transformation engine to figure them out now you'll notice i made a mistake the beautiful thing about interactive sessions is if i make a mistake i can simply correct it i don't have to wait minutes in between mistakes to go fix them like i used to so sure enough i can see that county populations was a string and i could go back and fix that string by casting it over to a to an integer using applied mapping so that's what we're going to do real quick we'll go ahead and clean up the entire dynamic frame we are going to rename the column to get rid of the spacing because nothing likes the spacing and so now we're going to keep the two columns we had now that we're happy with columns we can go ahead and create a temporary view in a data frame to see if we can query on top of them and sure enough now county populations are properly sorted and we can see the top 10 counties by population in the coven 19 data set it's worth noting that this entire demo was done sitting on an airplane it was done sitting in visual studio code and so this is showing interactive sessions running in an ide local to my macbook essentially being able to connect to glue's distributed compute service from anywhere i have an internet connection worth calling out again is the configuration and why jupiter magics were used to configure interactive sessions jupiter magics are again these snippets of code that state this is how i want my serve my my session configured when they're run as the first cell we take that's that configuration we apply it to your to your glue session and environment these are the same parameters as you would use in glue job if you're familiar with those you can pass any configuration you need to glue including custom spark config in this case enabling the cryo serializer as long as you do that before running your first cell of code your config stays and and goes if you forget something say you forget the connection to your database or you discover you need another database connection no problem simply add it to your magic list restart the kernel and rerun your code and within seconds you have a brand new cluster with a brand new configuration and off you go the other thing worth calling out here is this particular configuration assumes i'm you i'm wanting to use glue streaming i can use interactive sessions with glue structured streaming now is is really really great if i'm a developer and if i love working in code however there are many customers who want to get the power of glue and the flexibility of it without getting into the code so that's what glue studio comes comes in glue studio is our visual job authoring and monitoring tool you don't have to write code you simply have to drag and drop nodes in blue studio you can preview data at each step of the way you get real-time schema inference and just like the rest of glue it supports hundreds of connectors and yes you can still use transforms in sql or custom code as needed if you have the capability and the desire to do so well not required they do come in handy when you want to do something that we don't support directly out of the box glue studio also offer offers a monitoring environment for all of your jobs and glue through a single pane of glass okay let's get into a demo of glue studio let's go create a job with a visual source and target this will pre-populate for us three nodes a source a destination and apply mapping so let's go ahead and select the source from s3 we'll go to the catalog table that we created out of the previous demos data so we called that demo populations now before i can start a data preview i have to fill in the rest of my my information for my nodes in this case my s3 bucket needs a location to write data to once we've filled that out we can go back to my source bucket and we click data preview and once we select a role preview will start and might take a few seconds but within a few seconds we will start to get a data preview data previews are powered by interactive sessions the same things we've been discussing in the last few demos data previews are reasonably quick the first one often takes a few more seconds to spin up the compute but afterward everything is very very quick we can see here that it's the same data we were working with earlier excellent so let's go ahead now and you know let's add a transform let's go add a pii detection transform just for fun and let's look for a person's name because we can select which ones we want no no let's just go ahead and do all of them let's include all detection types let me just see everything let's go preview the data let's see if we can figure out what has pii after giving it a second to load again we see the same preview you'll notice that we're only previewing five of the seven fields so let's go ahead and select the rest of the fields and you can see that detected entities says there there are counties the county column has a person's name in it okay is that really a problem let's go scroll over and find out you can see that most of the county names are names mccracken pope grant they're counting names while they're detected by pii as proper names they're not but let's assume they were let's go ahead and redact them with a with a text field so we're just going to asterisk them out let's preview the data again to make sure it was taken and sure enough there go the there go the proper names any county that was identified as a proper name is gone awesome now let's continue on and let's assume we want to do something with this and write this out to our data catalog let's move that apply mapping back to our detect pii you'll see the output schema gets rid of detect the detect struct because we don't want that in our output we just want the clean data without the personally identifiable information in this case the county names that you know are our proper names and now that we're we're done with our data we can select a bucket we can select a database in a table we want to write to in this case we'll call it populations cleansed and let's go ahead and save say our transfer ah that's right we need to name the job let's go ahead and call the job clean populations sounds good to me and you can go look at the script this script is generated dynamically every time you change a field or property in the visual editor once we're satisfied with it we can go to job details all i'm doing today is enabling auto scaling we'll save the job and once we're happy with it we can click run now once we've run it we can go to the run details page to get information on this particular run and how it's going and if we wanted to keep going we could go to schedules and we can click on schedules worth calling out in addition to the visual editor glue studio offers glue studio notebooks they're the same notebooks as interactive sessions locally running in jupiter they're free serverless and they offer one-click job execution scheduling it's the same thing you saw me running in visual studio code except it's running in jupiter notebooks are sitting in glue studio nothing for you to host nothing for you to manage the same magics that i was showing you earlier apply here and run in here and of course there are built-in monitoring support just like there are for blue jobs now it's all nice and good that you can integrate your data in in a batch mode as we've been seeing but what about your other data glue has several execution modes we've got badge which is your standard job that runs at a set schedule you've got streaming modes that allow glue to run continually ingesting data as it hits a stream you have event based execution for glue that that will kick off a glue job based off of an event trigger and then you have an interactive api that allows you to integrate glue into interactive applications so let's talk about batch for a minute why batch simply batch is there for scale and for reuse it's the thing we typically think about when we think data integration or etl but batch goes beyond just a single job often it goes into workflows and pipelines as well so let's talk about workflows glue workflows allow you to orchestrate jobs with glue and with other aws services you can use glue spark jobs as well as glue python shell you can monitor the execution of the entire workflow in one place you can use triggers either schedule based on-demand or event-based triggers inside your workflows and of course you have easy access to monitoring logs the whole reason for workflows within glue is to encapsulate the entirety of your data integration pipeline in one place now workflows are great but oftentimes you're repeating the same process you might be repeating the same workflow for sales for marketing for another department now these come with their own challenges if you're repeating these continually you're coding them manually you're making mistakes because you have to change the same thing over and over again you have some poor data engineer who has to do this 50 times in this job and at some point it fails to scale so how do we avoid that developer spending that valuable time and how do we avoid the errors and the pain glue gives us custom blueprints these are workflows that are templatized you simply give glue a script a configuration file essentially how you want us to prepare and launch the environment for your for your workflow and a layout file to say hey what should the inputs and outputs to my workflow and we do the rest you can then instantiate that workflow as necessary so your end users can just say cool i need my daily report workflow for a new report and i can fill in the boxes click submit and your glue workflow creates a copy of itself with the correct parameters and does its thing the next mode we talked about is streaming glue streaming is built on spark structured streaming it is fully serverless and you can build jobs visually interactively with interactive sessions uh or in a more traditional manner like you have always done with uh with an ide and spark you can easily connect from kinesis or kafka directly in the visual editor so streaming is a great way to move your your spark jobs from badge into a micro batch and streaming method because really the difference between streaming and batch is only a few lines of code glue makes it fairly easy to migrate from one to the other either visually or in a traditional manner the next method we discussed was event driven integration so how do we let your data drive your work glue etl integrates now with that amazon event bridge amazon event bridge has hundreds of built-in sources allows connections to custom applications and your sas applications they provide all of these triggers that allow you to kick off a workflow that includes glue jobs crawlers and other things and the whole purpose of this is that you let your your data do the work you might have data that comes in daily you know once a day it's sometime roughly in the evening my clients upload their data to my s3 bucket and i have to go get it so every night i kick it off and i go get the data and i hope it's all there with eventbridge the second the data lands you can go ahead and just get the data you don't have to schedule it on a schedule you can do it in in response to the event of your data landing in your s3 bucket this allows you to move closer to that more that near real time goal that everybody seems to be seeking without the great expense and and hassle of maintaining a continually streaming environment when you don't likely need one for every use case and finally the last the last version we'd like to talk about the last method for integrating glue is interactivity using interactive apps using glue interactive sessions you can integrate glue directly into your own applications using the glue apis this includes glue streaming and it allows you to extend glue into any application that has access to aws on-premise or in the cloud so let's go ahead and see how this might be done so here i'm going to have a simple bash application called detect pii and this is going to run what amounts to the same code we ran earlier again on the same table i'm not trying to do anything fancy with data and this thing's going to go out and it's going to run the detect pii systems against interactive sessions and it's going to ask hey is this actually pii well in this table no queen anne's maryland is a county it's not pii so we're just gonna put no what what about castro texas is is that a name no again it's counting so no and so on and so forth and one south dakota now then there's no pii in here at this point i don't think there is pii so i'm going to exit now if there was pii i could have said yes and something else in my application could have happened such as locking it down inside leg formation or notifying the data owner that they have potential pii within their data while automatic detection is great and extremely useful there are times that auditing is is appropriate so this is all nice and great but how does glue work how does blue do all of these things and scale from all of these personas and all of these workflows and use cases glue runs a scalable execution model where a job is kicked off by a job manager and the job is divided into stages and the data is divided into partitions and the job manager takes those stages and pairs them with partitions to schedule tasks on a worker or a node of compute and glue serverlessly scales to thousands of workers and parallely executes these tasks on the workers and of course you only pay for the compute that's used glue announced at uh reinvent and we launched it the san francisco summit auto scaling and so now in addition to just setting a serverless compute you check the box as i showed earlier and we will only provision the compute you need for any given task so let's dive deeper real quick into integrating at scale because scaling challenges are hard you've got different business events schema and size variants source variants and this again all impacts your cost and capacity you've got resource prediction problems tuning's hard and yeah everyone typically over provisions because the last thing you want is for it to fail i'd rather pay 20 more than have to be woken up in the middle of the night so that's where auto scaling comes in as discussed auto scaling is a check box it reduces cost simply put you check the box and you don't have to think about the capacity planning you enable auto scaling you set the maximum number of workers you want to allocate to your auto scaling job and and you let it go so let's go ahead and look at how auto scaling works within a multi-stage job if we have a jdbc source and we want to read from it apply a custom transform a simple mapping and write it out you're going to have a few connections when you start the jdbc job then you're going to have high parallelization after the data is read into memory and spark now in a traditional cluster without auto scaling you would have to spin up all the workers first then you would have to do the small read and the big writes and you'd have a bunch of idle compute with auto scaling we only have to provision the workers that need to do the read when it's time to do the read when it's time to do the advanced transforms we can scale up as needed not before and that way you're saving yourself a significant compute and a good potential cost savings now all of this is nice and good but if i've got so much more data that i can interact with in my company how can i unify that how can i centralize all of my data and centrally govern it glue data catalog is our mechanism for central data organization it is a meta store for data lakes it's highly scalable and durable it's extremely cost effective it offers security compliance and auditing capabilities and it is hive meta store compliant uh with several open with an open source connector to attach all of your favorite hive compatible systems now easiest way to get data into your catalog is to use a blue crawler group glue crawlers are little serverless applications that connect to your data sources automatically discover your schema and extract that schema and then they write that scheme into the data catalog every time they re-read your data source they will update your schema as needed inside the catalog you're able to specify your own classifiers for custom files this is a big deal because you can even use glue crawler for those old school flat files with classifiers and grok this can allow you to remove some of your legacy systems that currently exist because it's really hard to deal with this old data and of course crawlers can run on demand as part of a schedule or part of a workflow or in response to events event based crawlers on s3 for example only crawl when a new object hits remember there's no real such thing as an update in s3 it's a new object or not so you have you can have a broader coverage with crawlers because you can say hey monitor all of these these folders these prefixes and s3 let me know when anything changes and because we're not having to list the buckets crawlers become faster and more cost effective making it easier to maintain your data lake at every stage in your data journey now we also offer lake formation for fine grain access control and while i'm not going to go too deep into lake formation i did want to call out govern tables lake formation government tables are a data ty or a table type on top of s3 that provides atomic transactions on top of your tables so that you can atomically maintain and update your objects you can automatically compact your data as necessary to make sure it's performant for these distributed analytics applications such as amazon athena it allows you time travel in your table so you can look at the table and the data as it existed at any given point in time this allows you to do cleaner testing of ml models and training it allows you a better understanding of when your data has come in and how it's come in and be able to compare what's changed when there are a lot of advantages to automatic time travel within your tables thank you all for coming i hope you've learned something i look forward to seeing what you guys build on aws glue thank you very much [Music] you