Building High-Performance Geospatial Applications with Amazon DynamoDB and Amazon ElastiCache

Key Takeaways

[Music] hi i'm jeff duffy i'm a senior technical advocate for dynamo db i'm lee hannigan i'm a daniel v specialist sa at aws and we're going to talk to you today about how to build some modern applications with dynamodb and elasticash for redis and today what we're going to cover is a use case that's going to help guide our application architecture we're going to talk about the choices that we're going to make for the data architecture for the user interface and the application itself um we're going to talk a little bit about the alternatives to that architecture that we selected and why if our requirements were different we might choose different services we're going to give you a demo of the application itself just to give you a sense of what it's supposed to do and then we're going to do a detailed walkthrough of our implementation including you know our cloudformation template and some of the code that we're going to use to implement it and we're going to save a little time at the end for q a so our use case today is is that uh we've been engaged by a startup who wants to get into the rental business and they want to build a rental property system that stores increased data for rental properties and they want to let their agents update and change data about rental properties when they need to but they also want to let their customers search for rental properties that are around them to identify you know where the options are and then dig deeper to understand you know the details of those properties so here are some characteristics about this customer's workload that are going to help us choose uh our architecture so they're really concerned about things being cost efficient during periods of low utilization their startup they don't have a lot of customers yet and even once the the traffic increases property updates tend to be pretty infrequent uh so they want to make sure they're not paying for something they're not using they need reads to be low latency but also high performance because that property search traffic tends to be extremely bursty you know you're you're going to get people who mostly tend to look for rental properties on the weekends or after work and that traffic is going to spike our system needs to be able to handle that they want to be able to do fast geospatial searches of that property data um you know they want their customers to be able to find properties around them within a given area it's very of course important where the locations are and so that's got to be easy and performant and then finally they need this to scale very easily so they have some aggressive expansion plans for their coverage areas and you know in the event that they get lucky and their their system goes viral they want to make sure that it's very easy for the system to scale up to handle the additional traffic so for their system of record we're going to choose amazon dynamodb because dynamodb is a fast flexible nosql database um it offers a pricing mode capacity mode called on demand that lets you just pay for what you use which means it's going to be extremely cost efficient during those periods of low utilization but on demand mode also allows you to just use more of the database and it scales very easily to handle that increased scale so it's going to fit those customer needs very very well of course dynamodb is going to offer all of the other features that this customer will want in the future maybe transactions of course all the security features like encryption and rest transit it's very performant single digit millisecond latency and then when they want to do that aggressive expansion in the future and maybe expand to other countries global replication is built into the system we're also going to add uh amazon elasticash for redis as an in-memory data store in order to serve the geospatial query and the low latency read requirements uh for the application and so elastic cache is a managed cache system we're going to today be talking about elastic cache for redis which is compatible with open source redis and offers extremely low sub millisecond latency for data retrieval and also it's very easy to scale as well and so we're going to meet that low latency high performance read requirement we're going to support those geospatial queries that we need and we're going to support the scale requirements for this customer so our data architecture for this application is going to look like this we're going to use amazon dynamic db as the system of record and we're going to use amazon elastic for varedis as our read cache and our geospatial search engine so how are we going to get data from dynamodb to redis well the answer is that we're going to use a lambda function so dynamodb has a facility built into it called streams that allows you to do change data capture anytime something is added or changed to dynamodb it creates an entry in a stream and we can create an aws lambda function that will be triggered to then go update elastic cache for redis and reflect that data change there so we can use it for those geospatial searches or those point reads now we could choose some alternatives to this if we needed to so for instance um you know for a system of record we could use a relational database like amazon rds or aurora so why didn't we choose that in this example well because this customer was really concerned about the database being inexpensive if they weren't using it all in the early days rds are both instance-based systems and so they won't quite meet that requirement as well as dynamodb so dynamo offers uh its own read cache system called dynamodb accelerator dax so why didn't we choose that over redis and the answer is because we need that geospatial search facility which dax doesn't support um also we could have chosen to use amazon memory db for redis which is a kind of a persistent version of redis a managed service as well but again uh so memory db isn't quite as on demand in pay-per-use as dynamodb is to meet that particular customer need finally for the geospatial search requirements we could have used something like amazon document db with mongodb compatibility it actually has much more robust geospatial search functions but again not quite serving the on-demand need and that's going to be true as well for the amazon open search service and finally dynamodb has a number of geospatial libraries for a bunch of different languages out there that you can use for dynamodb itself in order to you know serve geospatial queries but a lot of those libraries haven't been maintained in some time and a lot of customers are not terribly comfortable with using something that's not in active maintenance so when we look at the user interface architecture for our property search we figured out how we're going to store our data we're going to use dynamo for the system of record and redis for our geospatial searching and recache how do we get the data to the customer into their user interface and so starting at the left we're going to have a simple website in today's example it's a javascript client and it's going to call apis as you do with modern architectures in order to store and retrieve data and we're going to use amazon api gateway as the interface for that api an api gateway is a managed service that lets you front data or systems in this case we're going to use aws lambda as the thing that it fronts in order to provide security and functionality for apis and then api gateway will communicate and sort of proxy that request through to aws lambda functions which themselves are going to do the required amount of storing or retrieving of data for the customer and returning in the api response so here's what our complete architecture looks like and while this is a pretty complicated diagram we're going to take it in chunks to sort of talk through the individual details of the system we'll go through today so up top in the green you're going to see the property update flow and starting at the left under web client we have an agent who wants to update a property and maybe change the price or maybe add a new property to the system so they're going to call that property status api up top they're going to call through api gateway which will trigger a lambda function that's going to update properties and that lambda function will write the changes to our dynamodb table on the right you're going to see in that that white box are cache population flow and so what happens is that every time a change gets made to our dynamodb table it creates an update event in a dynamo stream and that stream will trigger a lambda function who will go and populate the cache and elastic dash for redis so if they update or a change gets made to uh a property in the dynamodb table this will then flow that change and populate the change in elastic cache for redis you'll see above the redis cluster uh something called secrets manager that i haven't talked about before one of the things we're going to show you today is that newer versions of redis and elasticash for redis support role-based access control so it used to be you just kind of use the shared key and that was the extent of security and redis well today we support more sophisticated our back and so we're going to make use of that and one of the things we're going to need to do is to securely store the credentials used by the lambda functions to access redis and we're going to do that using secrets manager which can securely store data that lambda can retrieve and use to access the system so finally on the bottom left in the orange we're going to see the property query flow and this is the flow for users who want to be able to search for properties around them or get information about a specific property and so their client is going to call this api gateway endpoint for the property search that again is going to trigger a lambda function which is going to go and retrieve the credentials it needs to talk to elastic cache for redis and then that function's going to query the redis cluster get the information it needs and return it back through the api to the customer so one of the things we want to call out is that the network architecture when you're using lambda is something you need to think about a little bit so services like amazon api gateway aws lambda dynamodb and even secrets manager don't need to operate inside a bpc um in order to to work right they can operate outside and use public apis to communicate but amazon elastic needs to run into vpc and so we're going to need to configure our system in such a way that our lambda functions can both talk to elastic cache and in dynamodb in secrets manager so when you want to use a lambda function to talk to a vpc you have to configure it for vpc access which creates an elastic network interface and that means that all the traffic is going to flow through the vpc that's great very easy to talk to our elastic cluster nodes but how do we talk to dynamodb secrets manager then the answer is bpc endpoints which are endpoints that you can configure in your vpc and set them up to allow lambda as you can see to communicate through those endpoints to the services that it needs to talk to and we're going to show you in more detail how to configure this later so high level it's important for us to kind of talk for a second about the schemas in dynamodb and redis so you're going to understand how they relate if you're not familiar with dynamic db tables contain items which are kind of like rows in a relational database um and when you query data in dynamodb when you when you do gets and fetches you have to provide the primary key which has to be unique for every item and so you have to provide this at least what's called a partition key which is what we use to sort of divide your data behind the scenes to make that number perform it and then you can provide what's called an optional stroke key so for today's design our partition key is going to be the agency id of the agency updating the property and the id of the agent who is updating the property we're going to join them with the pound sign um so you can see an example below there where those two numbers are um and then we're also going to have a sort key which is going to include some locality information like our city and our dress also put together with the pound sign as well and the combination of these two things will form our primary key which will be unique for every item or property that we put in dynamodb and this is what that schema will look like if we put it in dynamo json one of the things that we need to point out early to understand so dynamodb while it's sort of a key value style access pattern um for a database um it supports what are called non-scalar values what that means is like you can have an attribute in your item that actually has a complex thing like a list or a map underneath it um so you can get some pretty complex schema designs but redis doesn't support that redis is more of a true key value store where every value has to be just one thing we can't go multiple levels and so knowing that we have designed today our schema and dynamic db to accommodate it if you hadn't and you wanted to use this system that would be perfectly fine you just have to do some extra work in your lambda functions to sort of flatten your dynamo schema so then when we take a look at what it looks like to store this data in redis so redis is more of a simple key value store and so what we're going to need to think about is that every every property and all the details for that property are going to have to have a key associated that we need to store in redis and that key needs to be unique but the good news is we already have a unique key it's the primary key for dynamodb that partition key and short key put together in this case with another pound sign so you can see an example here of where we've created a property key by combining those that's how we're going to store the details of a property in redis um and below is just an example command that you can use this is python and to store that data in redis we're going to set it with that key and then map all the details to it for the property like the price and the location and things like that we're also going to though use redis not just to store this data as a cache but to index for geospatial queries um so redis offers this by uh calling a command called geo add and what we need to do then is to give it a key and then the property longitude and latitude so that'll let us then do a two-step process of storing the key in the cache and then adding it to the geospatial index for customer use then when we want to get the data out of redis we're going to do geospatial searches and we simply do that by saying hey um that first part the properties that's just the key name that we're picking for a geospatial index could be anything um you know and then we're going to pass latitude to longitude a radius of distance and then a couple other options that say hey i want to understand how far this property was from my search center and what the coordinates of it are and you'll see an example there that if you call reds for this you're going to get back a list of lists each of which has the data for a particular property its location is distance from center and so forth one of the things we'll point out that we did in this design was to make sure we had enough data encoded in the property key that we didn't immediately have to go back to redis again to get more details to display the property in the ui because round tripping for every property if there are a lot of them would generate a lot of additional traffic instead we intentionally encoded enough bits in here that we could show the properties and then if you need to dive down and get all the details for the property we can do what we show at the bottom here which is to retrieve the details by that property key we call redis and say h get all which hash git i'll get me all the details for this hash and it'll return all of those details to you so you can populate them in your ui so we're going to have all of the code and samples that we're showing you today in a github repo that's available to you we're going to post the link for you in the chat as well but you can find it at github.com aws samples slash dynamodb elastic geospatial workshop and so we have to make a couple comments about this first this is not production-ready code it's intended to be a sort of teaching device so there's lots of comments lots of logs in there and so forth to help you understand how the system works if you want to try it yourself the other thing that we'll mention is that it's not 100 plug and play because of dependencies that the code needs and some of the packaging that needs to happen for lambda we've given you instructions in the readme file for things that you'll need to do in order to get this deployed but it should be pretty simple and not more than a five or ten minute job to get up and running so i'm just going to take you through the application um and just show you how we interact with redis and how we get the locations for each of the properties um so what we use is the center of this map as our latitude and longitude to center redis so every time we move around the map uh we'll we get a new center which uh sends our request to redis so what happens every time we move we send the locations to api gateway uh that then sends the request onto a lambda function which uh does the geo search on redis returning us the list of properties um it shows it returns just only the latitude and longitude of the property as well as the property key so you can see in these little boxes at the bottom that we we parse parts of the key just to make it the application look a little bit more complete when we click on an individual property we do another request to redis so let's say we click on this property that does another request to redis with the key that we got returned in the previous request um so that does it api gateway again to lambda function again to redis and back and that gives us all the the uh the details for the that exact property from there we can edit the property we can change anything which again populates the cash so let's imagine we want to change the price of this property to a thousand dollars so we click on edit property uh we scroll down to the price and we change that down to one thousand dollars um and we hit save changes so what that does in the back end is it goes again the api gateway it updates it goes to lambda function that lambda function updates our dynamodb table um our dynamodb table then exposes that change in the change data to capture screen streams sorry uh and from streams it goes again to our lambda function which populates our cache so if we were to go back to that property now we can see that the price has been updated to uh one thousand dollars per month um and that is all done um three api gateway uh indoor lambda functions and uh updates whatever is needed um so to understand this a little bit better uh we're gonna go through the the cloud formation template associated with all our resources we'll go through piece by piece so that you understand exactly what's going on in the back end so now we're going to go through our cloud formation template where we um initiate all of our resources and everything's handled by a single stack so we're just going to go through it from top to bottom first of all we create our vpc our vpc then is used to containerize all of our subnets and our resources um so the first this is one subnet it gives us available ips from 0 to 63 and we we assigned that to a single lazy so for high availability we created three subnets here you can see subnet two and subnet three so we have three subnets for high availability for both our lambda function and for our redis cluster and there's they're created across three availability zones if you see here this is available availability zone 2 which is our third availability zone for that region then we create our route table and this just allows our vpc to interact with their subnets and where the flow of the requests go then we associate each of our subnets to the root table and that goes through here for the three subnets then we allow a security group so the security group is used to access anything that's inside the vpc uh be it the redis cluster or the lambda functions um and it's a self-referring uh security group uh which means it basically allows access from itself so if you're saying that anything you assign this security group to it'll allow access from anything else that is also assigned that same security group this is where we define our security group ingress rule so this is allowing tcp traffic from port zero to port 65535 then we declare our vpc endpoints as jeff previously mentioned to interact with some services which are outside of apc we need a vpc endpoint to do so this prevents us from having to try and go across the public internet um so here we have our secrets manager endpoint uh which holds our secrets which we use for access to our redis cluster here we have our dynamodb vpc endpoint this allows access to our dynamodb table to store our record of state [Music] and then we have our dynodb cable so as the requirements as jeff said in the beginning our requirements was that our dynamo db table or or whichever data store that we use um handles periods of low traffic well um and for that reason we use the billing mode paper request which is a then would be on demand mode you pay for every time you write or read if you don't write a read you don't pay so that suits our use case where the customer is making very infrequent rights and sometimes long periods of no activity which the customer will not need to pay for that then we we initiate our lambda functions so the first function we come across here is the function that we use to update our lambda function or update our dynamodb table sorry um so we just gave it a name you have to define your lambda function handler for each of the lambda functions you send it a role so it has the necessary permissions to interact in this case with the dynamodb table um you sent out a security group id and you tell it then what subnets so to be highly available we give it two subnets here is where we say our code where we store code so we zip up our code because our code uses some external packages such as redispie we package those up and we store them in s3 and then we define our s3 so this is my bucket for the workshop and this is the path to the key of this zipped up lambda function that we keep there uh then we have our lambda function runtime our timeout our tracing config and our environmental our environment variables so from here we grab the environment variable which is table name so that the lambda function knows which table to update when we send requests this is our permissions so this is a line api gateway to invoke so every time we hit the api gateway we want to pass the request to this lambda function so this is allowing the the api gateway permission to execute that lambda function and that's the same for all three of our lambda functions this lambda function then is a little bit different that it consumes from the stream and not from api gateway so what it is it's handling any changes that are made on the dynamodb table um so you'll see that it'll have a little extra configuration at the end so again we have a depends on um which means that uh the iam role and the replication group must be created before uh we can create this lambda function um we give it a security group again again we give it two subnets the code comes from comes from s3 as before uh same settings here but this time for environment variables we have the endpoint for our redis cluster we have the secret ar arn for our elastic secret and then we have uh just a string for our geo index key um and then we assign the event source mapping so a lambda event source mapping is how we map the uh dynamodb stream to our lambda function uh we're taking it in batches of 10 which is probably a little bit unnecessary since the the number of writes to our dynamo dv till we're going to be low byte size could probably be one for this use case so that it's as close to real time as possible uh enabled is true so if you admit enabled it's default false which means that for our changes in your dynamodb table the lambda function won't actually be invoked um and then we're saying that the this is the stream which we want to tie this event source map into uh belonging to our dynamodb table and then the function name the number of retries which is five this again also defaults to minus one which means that your lambda will continue to retry for the period of that item uh remains in then would be stream which is 24 hours um you wouldn't want something retrying for 24 hours um so i would give it a retry attempt of five um if if you were to to do this you know in a more um you you'd want to actually have a dlq there so after the the five attempts you would want to store that item which you can't persist in a dlq which is a dead letter q and then maybe have some other process which is um retrieving from the the dlq and you can further action that may be stored on s3 for further insight um then we have our third uh lambda function which is our query function that allows us our geospatial queries uh much the same as before we just assign it the role uh the lambda function handler security group are two subnets codes coming from s3 um and the environment variables are the same as the other one um so that's pretty much the same uh again this one's been invoked from api gateway so we allow apa gateway um to invoke that then we assign a role to each of our lambda functions so we just for the sake of this workshop we just assigned a single role we allowed that role administrator access which you definitely wouldn't do in the real world um so you would want to make sure that this access is only allowed things that you need such as update the dynamodb table um you know search on the redis cluster access your secrets and secret manager those sort of things and then we have a log group so the log group then is just somewhere where we're going to log or all of our you know and anything that happens in the lambda function that we log out we want to log that in in cloudwatch logs that brings us down to api gateway so this is where we define our apis um so first of all we create a rest api we give it a name and a description it's pretty straightforward then we create a deployment so each each of your apis is associated with a deployment um so we just give the deployment a stage name everything else is uh is pretty standard um to access api gateway there's certain ways you can access you can use cognito you can use an iam role attached to perhaps an ec2 instance but for our use case because we're accessing from a local machine we use an api key so that api key is unique to you which it's the next step of authorization when you want to invoke your api gateway so associated with the key then is the workshop uh the name and the api which is the rest api we created previously um in order to use your key you need to use a usage plan an api gateway usage usage plan um again that's just um some standard uh setup you you this is your uh rest api and the rest is names um api gateway usage plan key so this is where you associate your key with your usage plan so again this is our usage plan and this is the key that we use and we tell it the type of key that we're using which in this case is an api key so this is our resource for our property search api and this is the resource for our property detail api which gives us the property details so for each one of those resources we're going to have an associated method so this is the update property method api gateway method authorization type is none http method api key required is true because we need an api key to allow us access we're using post um the type is aws and we're showing that we want to invoke this lambda function so when we call this method we want to invoke this lambda function which is lambda function table update dot arn so that's the arn of our update table lambda function then we we create our response types what we want in the headers um so here we have for headers content type x.amz date authorization api key etc the load method so we're allowing post plus options and access control allow origin here you probably want to further refine that to your domain name of your application for us we're going to use star as we're running from local and then we create a response template for us we're not going to do that but we're just going to return an application json we assign the resource id which is the update property resource we created previously and the rest api we want to associate it with which is the rest api we created previously also and then we have the method responses again we're just going to say that these responses aren't required we set them to false but um for cars to work in our in our use case we will need to return those headers from our lambda function so then we have an api options method so when you run api giveaway requests from a browser you first have to send a preflight request which does some checks for headers and etc so we're going to have an options method which is just a mock method for each one of our uh each one of our apis so um it's it's pretty straightforward um it's just tasked with so api gateway just returns what headers are acceptable and then your request continues on to the post which is the part that you're interested in so again we have http method options this time status code we're going to return 200 we have the same headers again same methods in the same alloy origin as before uh same response template uh pass through behavior is when no match which we won't even need that uh request templates so we're going to turn the status code 200 and this time instead of type aws we're going to it's type mock um everything else is um the same as before we're just going to center 200 back and this is the headers that we can send um so that goes through the same for the other uh there's no difference in and so this is uh the method for the get properties it goes through the same logic we're using type aws proxy which means we proxy to the lambda function so any requests coming to api gateway 2 that hits this method passes the complete request to our lambda function and in our lambda function then we parse the request and to understand what we want to do with it everything else is the same we created options for the pre-flight request uh we set all the same parameters and that goes the same through all of our lambda functions uh so we'll just skip to the bottom of those that's the options for our last one and then we come to secrets manager i'm going to let jeff go through the stuff that he's initiated here for secrets manager it's quite interesting in how to interact with redis um using the auth that we've defined here and i'll let jeff talk you through it so in order to set up our redis cluster and use role-based access control we're first going to need credentials that is a username and a password that our lambda functions can use to access redis and so we're going to use secrets manager to store that information uh so here you can see that we're setting up those credentials that we're going to use to access redis and we're doing so by defining a uh aws secrets manager secrets uh which we're calling the elastic credential secrets and what we're asking here is to generate the password the secret string for the secret uh so we're gonna give it a template um so secrets manager store secrets is basically like a binary save string um but it has some functionality built in cloud formation that you can create what is more or less like a json string that you can parse with with whatever in this case our lambda functions so we're going to start by having a template with a username and we've hard-coded the username here to workshop user we're asking it to generate um a string key of password and then we're telling it to generate a password of link 30 to not include these characters you know so if you want to do this for another system that had characters you couldn't use you can say hey don't use these and then we've given it a name to refer to later so what this will do is tell secrets manager to create a secret to give it that form to create the actual password and then distort all as a string which is sort of this json object that we can then later decode in order to use for our redis cluster so in order to use role based access control for elastic for redis you have to set up a user of course first and we're going to let that user depend on that secret that we created now the way that it works in elastic cache red is for role-based access control is that you have an access string that sort of masks what a user can do so for here for simplicity we're letting this user read or write any key in the redis cluster of course in production you're probably going to want to limit this to something that's a little more restrictive we're going to then resolve the credentials that secret manager created with these uh resolved stances here and so um users in redis can have more than one password which is why this is a list of course we're only going to have one right now and we're telling it hey go to secrets manager and resolve the secret called cash cred secret um pull the secret string from it and then from that parse it and extract the password and we're going to do the same for the user name that we're going to assign to both the user id and the username for our redis user so this is taking the credentials we made in secrets manager and it's creating a user in redis for us and then we're going to go ahead and create a user group um users and elastic aspheritis have to be members of user groups to make them useful so we're defining that user group we're going to give it an id that we're going to later use to refer to what we create the the cluster itself and then we're going to define which users are part of this group it is a requirement of the elastic cache for redis system that you always have a username default as part of the user group but this workshop user here that we've defined and created credentials for earlier is the one that our limited functions will actually use to to access redis so then of course because our redis cluster needs to operate in our vpc we're going to have to define which subnet groups it's part of and they just refer to the ones that we've talked about earlier that we created and now we get to the meat of creating our redis cluster you're going to notice that this is called a replication group and cloud formation that is because what it's doing is uh there is a a way to deploy elasticash for redis that is called cluster mode enabled and rather than just have a number of nodes that have sort of a failover system set up among them they have multiple sets of nodes that are shards where the data is sharded between them and this is a as a more advanced way of deploying elastic cache redis that's going to let you do some more interesting scaling options with respect to adding and removing shards on demand it lets you create much bigger sets of cache memory we don't need that for our example today and so we're going to create a simple replication group which is just a number of nodes that have an automatic failover system within them but all of the data has to be able to fit within the memory of a single node so we're defining that there as a replication group and we're going to define some properties for our elastic cluster um of course we're going to turn on at rest encryption we're going to say that we want that automatic failover enabled so we're going to have a primary and some replicas if our primary node fails one of those replicas will promote itself to primary and take over the job of uh reading you know servicing the reads and writes um for the keys um our cache note type so so elastic cache for redis is uh an instance based service which means that you have to choose your node types or really instance types in order to define how much memory they're going to have to store your data as an in-memory store we picked an r6g large here of course if you go up to an extra large you'll get twice the amount of memory and so forth um and so the parameter group name that you'll see here is important because it also will define whether or not this replication group ends up getting deployed uh as cluster mode enabled setting it to the default redis 6x says that i want it to be a replication group there's another parameter group name that has cluster mode in it that will make it a cluster mode enabled group if you choose that instead so here of course we're just telling it which subnets we want it to live in for that subnet group that we defined above the engine is defined as redis because elastic hash also supports an engine called memcache d that we're not using today so engine version has to be uh six hour higher because redis 6 is when we started to support the acl system that allows us to do that role-based access control that we're using today multi-az enabled lets us say hey we're going to deploy this across multiple subnets let's let's make sure we're deploying our nodes and multiple acs for the highest level of availability this simply defines that in addition to the primary that always has to be part of a replication group i'm going to add two extra replicas for three nodes total for my replication group to give me that 3az deployment and then finally we're going to make it part of that refer to itself security group in order to make sure that the lambda functions that need to talk to our redis cluster are able to do so um you must by the way if you want to use role-based access control have tls enabled transit encryption right which you want to anyway is the best practice and then finally we're going to say these are the user group ids we're only going to have one if you remember we created that user that users are using the credentials for security manager we put it into a user group well now we're going to refer to the user group id that user is a part of here so when we create this replication group it's going to automatically let that user read and write from the redis cluster so i'm just going to walk you through the lambda function which is tasked with updating the dynamodb table um we just take our normal imports bottle 3 is what we're using so we create a resource which is our dynamodb table and then we create a table resource which is a bottle tree table type of dynamodb and we pass in the name which we've saved in our lambda function environment variable which is here uh we create that outside of the handler so if you have frequent invocations that this is only created once so you're not creating a new resource every time your lambda function is invoked that the your dynamodb client is passed across in vocations um then we move into the handler we created some dummy response and then we start making our statement to update our items we initiate our maps for holding our expression attribute names and our expression attribute values then we pull our keys from the event so our event is what's coming from api gateway and the keys is the keys of our item so event keys primary key event keys and sort key so we assign those both to variables then you'll see us have some heavy comments here to to sort of highlight what's going on in the next step of code so first we assign our expression attribute values which is basically ascend a variable to each one of the values that's coming through um so in this case we're going to say we have an agent and this is the value of the agent again we do the same thing with our expression attribute names so basically what we're doing is we're assigning a variable name uh to the value the reason we do it for this is because then modb has a list of reserved words which you can see a link for here we don't want to clash uh with those reserved words so we assign each of our keys to a variable name which means that we won't even if if the word does clash uh it it won't because it's assigned to a variable it then would be you'll be happy with that and then the the outcome of our of our expression is going to look like this so it's going to be set uh your variable name equals your variable value and then they'll be um with a comment between each of them so if we were to say another agent there or agent c maybe it would look like this on the output so this is what this loop does um so we're taking the for each key value pair in our data which is sent from api gateway we loop through it and we assign those so here we have expression attribute values here we have expression attribute names and then we assign both to our statement and our statements used as our update expression so then we come in once we have those maps we come into our update item so we assign the keys that we gathered from before from the event uh we call the update expression uh we pass in our statement uh this we call a minus one here so we cut the the trailing um comma off the end that's why that is minus one expression attribute values we pass in that map expression attribute names we pass in that map and then we build a response code so we're just saying the status quo is 200 because in in this instance it was successful uh we pass a message of success and then we uh we print some logs just so you can debug in cloudwatch uh again then if we have an exception we're going to say the status quo is 400 and that something went wrong and then importantly we passed back the headers so as we previously mentioned api gateway we use that to pass a response so everything is is passed to the lambda function so that ultimately ultimately means that your lambda function is responsible with returning headers and everything else so your headers here must exactly match what you've defined in your api gateway which we've done in uh cloudformation so you can see the the headers are all the same for that we've defined our methods are the same and that we've accessed control our origin star so you would want to further refine that then we return the response so that's basically there's not very much in here that's how we update our items um if the item exists we update it if the item doesn't exist then we add a new item and and that's how uh denmark interprets the update item api so now jeff's going to go through the two lambda functions which we use for redis and we'll see how they work so first i'm going to take a look at the function uh that's responsible for for populating our property cache and so this is the one that's invoked from the trigger from the dynamodb stream that has those events with changes it's going to it's going to take that information and put it into the redis cache so i'm importing the uh the libraries that i'll need boto3 of course which is the the aws sdk for python um you'll notice this interesting one here which is type to serializer so if you ever get a chance to go and look at the dynamodb json that that represents a dynamic db item it has multiple levels because the type of that item whether it's a string or a number or whatever is encoded in that as you'll see in a minute um redis will not accept that that multiple level and we're going to have to do something about that in order to make it uh types that we can store in redis so uh what we're going to need next is our environment variables in order to make sure that we're connecting to the things that we need um and so of course we're going to need to figure out what region we're in to connect the photo but next is going to be our cluster endpoint so this is the end point we're going to use for our lambda function to connect to our elastic redis cluster these environment variables were populated remember when we defined this lambda function in our cloudformation template so the next one we have is the rn of the secrets manager secret that contains those user credentials to to authenticate to elastic cache for our user so this is not the credentials it's simply a resource identifier that lets us go to secrets manager and ask for the secret content to get the credentials and then finally we have the index key this is whatever name that we picked for the key that holds the geospatial index in redis so the next thing that we're going to do in our code is to go and make a connection to sequence manager and we're going to ask secrets manager for the secret value for that arn this remember is is more or less a binary safe string so we're going to have to decode that string in order to get the bits of information that we want we're going to call the python json load s or load string function which is going to take that string and make it a python dictionary and then we can go ahead and just get the username and password out of the dictionary that's created in order to connect to redis and authenticate then we're going to actually make the connection to the redis cluster using the redis client this is one of those dependencies that we talked about before that's going to have to be bundled in the zip file to deploy these lambdas and we're going to make a connection to our redis cluster using the endpoint that we got from the environment variable we're going to use the credentials that we got from secrets manager we're going to make sure of course that we have ssl turned on and then once our connection to the redis cluster is made we're going to be ready to go ahead and do some useful stuff and then finally i'm going to instantiate that to serializer i talked about before which is going to let us turn that dynamodb format into a python one that we can use to insert data into redis as lee noted before all of the stuff that we've just talked about happens before the lambda function is defined and what that means is that the lambda system itself can then sort of memoize or cache this so it doesn't necessarily have to you know go to secrets manager to get the secret info or make a new connection to redis every time it invokes the lambda function which will hopefully save us some startup time so in the main handler function what we're going to do is we're triggered by the dynamo stream and we're going to get hinted this a bit um so this event will typically contain one or more records and we're going to iterate then over each of these records and we're going to take a look at that record structure and get something called the event name which is the operation that happened within dynamodb did i insert something new or did i update it um so if it was a put item which is an insert or an update item which is a modify we're going to have to look in that record for a field called new image because that's the way that the new data that's inserted into dynamo is emitted in the event if it was a delete which shows up as a remove operation um it's actually called old image in the event we're going to need to pick that out instead what did this look like before it was deleted we need to know that because we're going to need the key in order to delete that key from the redis cache to get rid of it so it doesn't live in the cache anymore if it's been removed from dynamodb so here's a bit of fanciness using a python thing called a a dictionary function that lets us say i want to use that to serializer and i want to take all of my wdb structure and i want to turn it into a python dictionary instead that's all this is doing there and i end up with a dictionary called property details so one of the things is that even though we've converted this to python types now it turns out that so redis has some limitations on what types that you can store in redis as well and one of the things that redis doesn't allow is boolean types it doesn't have a boolean type in its system so that's not too hard to deal with all we're going to do is we're going to iterate through all of the property details and if we happen to see any booleans we're just going to convert them to strings because that's good enough for our system and what we need in order to store redis if you were making this into more of a robust production system you would probably want to go a step further and instead of just having a for loop look for only booleans look at all of the possible types in dynamo and how they're stored in redis and make sure that you're doing the right casting and conversions in order to be able to store your data so if you remember we need a unique key to store these property details in redis and we decided to use the partition key and the sort key joined with the pound sign which is what we're doing right here to create that key and then it comes time to actually you know go ahead and update redis itself and so if it was an insert or modification we're going to call a redis function that just says hey listen i want you to take that detail key and i want you to make the value of that key all of the details for that property then we also want to go and we want to add the property to the geospatial index right so we want to say okay if we're adding a new property or we're updating it we're going to call and say i want you to for that property key add it to the geo index for this longitude and this latitude and then redis will add to that geo index all of that data so then when our clients come and they do a geo search for a radius we're going to get this key back if it's within the radius that they wanted if it was a deletion from dyn