Digital Payments Architecture and Implementation with AWS Open Source Databases - AWS

Key Takeaways

[Music] hi everyone my name is jared persinger i'm a database specialist solutions architect with amazon web services and i'm joined today by my colleague dustin brown also a database specialist solutions architect with amazon web services and today we're going to talk to you about a digital payments architecture that we created our digital payments architecture falls under a program known as db galvanized db stands for database and db galvanize was created to help customers migrate their financial services industry workloads over to open source databases like amazon aurora and amazon rds historically there's been a misconception that commercial databases were safer more secure than open source databases and that's really what this program is designed to do is help kind of demystify some of those misconceptions and we have a couple of ways that we can help here so we have uh we can provide use cases and studies best practices we can also put you in touch with a database specific uh account team with people like dustin and i and we can provide proof of concept credits that's something that's particularly interesting or interesting to customers so why digital payments on aws first and foremost security right we're talking about the financial services industry so security always needs to be top of mind and security is always top of mind with us at aws so our aws architecture supports some of the most stringent security standards in the financial services industry then we move on to compliance compliance is something that of course is a cornerstone of financial services as well when we support the highest standards of financial regulatory compliance with our aws services next differentiation and i think this is really cool gives you an opportunity to stand out among different financial services industry uh competitors by building modern architectures on open source databases like aurora and rds and you know that's a great point jared the ability to differentiate with aws services specifically aws database managed uh services is is a great way to build modern architectures that lend well to microservices model decoupling your different aspects of your tables out into the appropriate uh database managed instance such as aurora postgres or mysql and into the nosql space such as a dynamodb given the opportunity to auto scale uh to uh scale to to meet the the needs of your uh of all of our customers so differentiations are a really important part it's a great point dustin and and that really kind of leads us into modernization right i mean um if you can differentiate then you're probably modernizing as well we give you the option to free up people like database administrators to really work with your app developer teams to kind of modernize and create new and innovative applications that speak to your customers so we had some goals when we set out to create this digital payments architecture um and and they're kind of here highlighted uh security speed scalability availability and reliability so first and foremost again security you're going to continue to hear this throughout our presentation today but we wanted to focus on a lot of the tools that are available in aws to be able to safeguard payments and financial services industry um data and then speed you need to be able to do things quickly right the faster you can take transactions the faster the faster you can make money so uh that's an important thing with with all of the databases that we're going to be talking about today and the architecture as a whole and then massive scalability we need to be able to grow uh with our customers if we get more customers we want to be able to grow with them some of the ways that we can do that is storage auto scaling read replica auto scaling or even some of the new serverless instance offerings that we have available finally we want to achieve availability and reliability those are of course always going to be important if your customers can't access your application and your database then they are not able to do the payments that we're trying to achieve on this platform that's this is a great point a great great point about availability and reliability and typically as a as a database administrator myself for for for many years uh being able to have your your systems be highly available highly reliable um and that's one thing that amazon aurora can definitely provide uh spanning multiple availability zones spanning multiple regions even having your automatic automated backups taken care of for you being able to achieve that high availability in your financial systems is something that's uh that's pretty paramount absolutely so let's go on to the next slide here we're going to talk about uh i want to talk a little bit about the architecture before we show you the diagram um so some of the key components here one of them is going to be this payment processor api what that's going to allow us to do is really offload a lot of the risk of storing sensitive credit card data in our application we we put that responsibility on the payment processor and we connect to an api that payment processor then returns a token that represents that credit card information so again keeping uh security of this credit card data is very important and that's one way that we do that by offloading that to the payment processor uh another piece that's important is data isolation we're trying to move away from these monolithic architectures and we're trying to get into a a more decoupled architecture and you'll see that with the use of virtual private clouds we'll see that in the diagram here shortly but we separate the data layer in the application layer and then really what are we trying to do here we're trying to store business data really we want to be able to store these transactions so that we can go back and see what we did what customers purchase what if there's a something like a return or something like that we need to have that data available somewhere so we know that that transaction actually occurred and we can verify that so what we're going to use for that is amazon aurora and aurora is our fast commercial grade database uh built natively in the cloud and it's very secure and very scalable as you'll see when we go into the deeper portions there and then lastly we want to cache the session information in amazon elastic cache elastic cache is going to allow us to move at very high speed it's going to provide very low latency and it's going to be able to uh and it's going to be a great feature to help us with these financial transactions so let's go into the architecture slide next you can see we have this diagram here uh we'll start at the top we have as you can see we have our users here at the top um represented that's that's really the users the the company that is uh that is selling or the the purchaser that they're all kind of encompassed in that icon there and then over here to the right you have the payment processor api as well um what we're going to do is have the users uh log on through amazon route 53 that's our domain name service and they are going to be able to authenticate in uh something called amazon cognito cognito is a way that we enable third-party authentication to your application you don't have to use this this is just an example of what we used for the sample architecture of course you could use username and password something like that that's really up to the user to decide how they want to do that then moving over to the left of that we want to highlight a couple of security features since we're since we're honing in on security so much with this architecture we want to point out uh aws shield so aws shield is our distributed denial of service security measure that we take and then we also have the web application firewall so the web application firewall is going to do all the things that apple applicate that firewalls do preventing unauthorized access on certain ports and things like that but again just a couple of extra security features that we have have added here to this architecture and then as you can see the line kind of goes down from there and it goes to into something called the customer vpc so customer vpc is our virtual private cloud it's a logically isolated portion of the aws cloud uh the customer can have several of these uh on on one account but that it's a way to isolate your data and create some privacy between other portions of your workload so we're in this customer vpc and then within this customer vpc you'll also see these these blue boxes here called availability zones availability zones allow us to have greater availability when you talk about availability zones we're talking about a one or many data centers that are separated by different flood plains and electrical grids so you can imagine that gives us a degree of of security and reliability there as well so we move down into the into this vpc and we hit our application load balancer our application load balancer just balances the load so that we're not hitting one or more instances too many times and causing a poor experience for our customers so that application load balancer hits the public subnet then the public subnet uh is we at some point we have to have a customer-facing portion of this right at some point support point it has to reach the public in order to reach our customers but we don't necessarily want all of our other layers like our application and our daily data layer exposed to the general public so you can see within this public subnet we have a couple of ec2 instances here they're in an auto scaling group which i'll get to in a bit but those ec2 instances are really just routing the requests down to the private subnets and you can also see the nat gateways there as well those are going to serve outbound requests which you'll see in a moment so as we move down into this private subnet you can see we have a few more ec2s or a couple more ec2s and this could be many ec2 instances they're in an auto scaling group and what that auto scaling group is going to allow us to do is kind of expand and contract the number of ec2 instances that are available for this application depending upon the amount of need at the time so think something like black friday you're probably going to need a significantly larger amount of of ec2 instances than you would on a regular day those ec2 instances are going to take that request and forward that up to the payment processor so you can see there in step five we have the payment processor up outside of the uh the aws cloud and the way that we're able to forward that outbound request is through that nat gateway that you see there in the public subnet but what that payment processor is going to do is it's going to confirm the credit card information that's available and supply our application with a token that represents that credit card information so again we're never storing open credit card information on our application or in our database anywhere we're offloading that responsibility to the payment processor api and there are a multitude of different payment processors up there out there we just want to leave that option open to the uh to the greater to the user so once the payment processor confirms that it goes back down to the application layer and then the application the the frontend application forwards that out of the data out of the vpc through the vpc endpoint that moves into something called amazon private link which connects these two these two vpcs and that allows us to connect into that other vpc so you can have one account with multiple vpcs as you can see here again this is another logically isolated portion of the aws cloud that allows us to create greater security for our data layer so within the data layer we have the amazon api gateway within the amazon api gateway we've created a rest api that we connect to the api then connects over to several want two or more availability zones to increase availability and then you can see within that first private subnet we have uh lambda doing our application logic we chose lambda for this one you could really choose anything you want you could choose an ec2 instance to run this application really any any it's up to the individual user to decide that but we thought lambda was cool because it's a serverless offering and allows us to run uh our code in seconds and only pay for the amount of time that we're using it so what that's doing is it's taking it could be doing extraction transformation and loading and forwarding all of this into our database really storing it in the database in the structure that we want and as you can see in the bottom layer there really the base of this entire application is amazon aurora uh that is our single source of truth here and then within that also we have amazon elastic cache so elastic cache is going to be providing that sub millisecond latency it's going to be doing some some session caching and allow us to have very high speed transactions it is not a required element of this but it's certainly nice to have especially if you plan on doing a lot of different transactions and now the last couple of things i'm going to draw your attention to here are in the bottom right corner so we have our uh security and compliance features here these are just a couple of more tools that we wanted to call out for you so we have our um secrets manager and aws secrets manager allows you to store things like your database credentials in secrets manager and even rotate those credentials on a scheduled basis to comply with whatever your your needs are then we also have aws identity and access management what identity and access management is going to allow you to do is control all of the things in your your database and or in your aws environment so it's going to allow your engineers and your administrators to log on and tinker around with all of the proportions of your architecture in a secure fashion and then lastly we have amazon key management service aws key management service so the aws key management service is going to store all of the cryptographic keys for your data at rest so whether those are crypt customer managed keys or aws managed keys you could save them all securely in aws key management service now finally we'll move over to the right here and we'll talk about some of the monitoring options we have here so we have amazon cloudtrail that is going to allow us to take a look at anything that's happening administratively in our database so if we have our admit our users going in there and maybe perhaps someone's accidentally deleted a table or dropped a table we can use something like cloudtrail to be able to go back in there take a look at that and and find out uh who who did it when it happened uh it gives us great great options to look back and monitor compliance and then lastly for uh monitoring we have amazon cloud watch cloudwatch is great because it's going to provide metrics about our databases our instances all sorts of different os level metrics and display those in graphs for for you to use later on yeah thanks jared for that for that great overview and that diagram of of architecting and migrating over your financial systems to uh to amazon aurora what that might look like and as you can tell we've we've kind of did an expansive expansive view integrating other security components such as cognito the application layer and an entire ecosystem available not just at the database but now focusing a little bit more on the database amazon aurora for my sql or postgresql as you may know is a fully managed and commercial grade database service it's ideal for the financial services industry it's highly available durable highly compliant and caught and it can be cost optimized uh used using the open source engines that you are familiar with such as mysql and postgres the other the other side of this coin is the amazon elastic cache which offers a fully managed caching service designed to meet the needs for your ultra low latency which often times the sub millisecond for you in your trying in your transactional financial service applications these two database platforms coupled together can create a low latency highly available relational environment that will securely store your your credit card transactions and your financial transactions that you would be migrating over so moving on to security in the database layer it's important to understand the different security tools and resources that are available to our aws customers specifically in the aurora postgres and mysql arenas security is is more important than ever being able to securely store your transaction information customer data is vital especially in this day and age uh to protect data in transit uh we're gonna start we're gonna start with data in transit as aws encourages our customers to leverage a multi-level approach all network between aws data centers is transparently encrypted at the physical layer all traffic within a vpc and between peered vpcs across regions is transparently encrypted at the network layer when using so when using supported instance types the following connections can be combined with aws private link to provide connectivity between your private network and your aws environment so you could use aws site-to-site vpn connection or aws direct connect connection the second aspect of security would be database isolation and with aurora and oftentimes many customers ask me this question are is my database truly isolated can can other customers access my data can aws access my data and the answer is that your database is is is isolated each database server has a vm enforced isolation boundary and it does not share the underlying kernel the cpu resources the memory resources or the elastic network interface with any other server which offers a true isolation of your data from from from from outside influences at the application layer you can also have a choice about whether and how to use additional encryption using a protocol like transport layer security or tls which many of you are no doubt familiar with you also need to be able to access your resources securely and efficiently one recommended approach would be to use aws identity and access management tools for database administration we also got you covered uh for storing and rotating your database credentials with aws secrets manager using aws secrets manager allows you to easily rotate manage and retrieve database credentials api keys and any other secrets through their through their user-defined life cycle so any passwords any database passwords any other secret information that need to be stored encrypted and passed to the to the database layer at the appropriate time can be stored in aws secrets many customers use this as a trusted tool to uh securely store their their uh their secrets next we want to mention compliance validation you want to ensure that your database meets your compliance requirements you will have the option to audit your environment with third-party auditors to assess the security and the compliance of of your aws services as as part of a multiple aws compliance programs uh with an example of being sock pci fedramp and hipaa next we'll we'll mention logging and monitoring to aws cloudtrail and aws cloud watch you can think of cloudwatch as a central repository for all of your aws log logging needs aurora integrates very well with the rest of the aws ecosystem and specif and has been designed specifically to do so you can send all of your logging to cloudwatch and also using cloudtrail in conjunction with these two services you'll have a full audit of all of all of your logs that can be used later on for reporting purposes for audit purposes etc and lastly on our list is the ability to audit your database using another purpose-built tool known as database activity streams uh or other you all and you also have the opportunity uh to implement other native auditing tools as well uh first for a list of supported tools you can uh talk to your account team and see what might work with with what's already working within your organization or you can uh adopt database activity streams so using amazon aurora in financial systems storing transaction information from the application is a very sensitive area storing a token as a representation of the credit card number instead of the credit card number increases the security for each transaction this token may be used as a reference for each purchase or even for refunds that would need to be processed days or even weeks later you'll able to you you'll be able to use this token as a reference point for any previous transaction that has occurred for any auditing purposes and uh it is considered to be secure as it is not the actual credit card number that is actually handled at the payment processor layer which is outside of aurora moving on to a different aspect of being highly scalable for both reads and write functionality it's really important to understand that aurora will scale for for your needs you have push button or even automated scaling with amazon aurora serverless will allow you to plan for plan big for your future database performance needs uh in the future will allow you to plan and even for those moments where you can't plan it is uh the push of a button you can have a larger instant size then you can stay you can scale up up to many magnitudes higher and then the last uh what we mentioned here earlier was encryption at rest and in transit um and you'll you'll be able to use those uh networking uh tools and uh encryption uh to to keep your environment uh very secure as the sister component of this of this architecture we wanted to also again mention amazon elasticast for redis using amazon elasticash as a distributed cache for session management as a very popular uh and and friendly way of of increasing uh low latency transactions and connections and and increase user uh your user experience storage needed to track whether transactions are authorized and authenticated is also a good use case for amazon elastic cache it also allows you to capture sessions to allow retries for transactions of failure occurs so it's a a very good counterpart to amazon aurora inside of this architecture for these reasons so sometimes it's really helpful to to see a a model of the of your schema of what this might look like going forward so this is a sample database model that we put together to show how the payment processing database might look like you'll notice the three distinct tables the payments authorization the transaction info and the payments tables turning your attention to the payments table you'll notice the card token column notice we are not storing the actual credit card numbers but simply the token of of the of the card itself so no sensitive information from a credit card perspective is stored in in the actual database and you can see the payments authorization tables allows for the approval codes for the transaction id and then the transaction info allows you to have the product code where the terminal where that where the transaction came from which terminal and the the payment method that they're actually using so this is an example of the of a database model that you might uh adopt on on one of your financial systems in amazon aurora yeah i really like this slide because it's although it is simple it kind of gives users a place to start from you know uh their their tables and their schemas will likely be much larger and more complex than this but it kind of gets the wheels spinning if you haven't used a payment processing service before great point next let's move our attention a little bit more to the performance in the database layer we talked about security we talked about high availability let's talk about performance so as many of you already familiar with amazon aurora you understand that amazon aurora can scale up to 15 separate read replicas and these read replicas can be scaled across multiple availability zones for for high availability purposes being able to have these read replicas allows you to port over your read traffic and to to scale up to 15 different instances up to a large instance such as a 16x large instance not all of the replicas need to be the same size as the primary you may have very low right traffic you may have very high read traffic vice versa depending on what your needs are you can you can scale to the appropriate level you also have the opportunity to [Music] implement your read replicas in an auto scaling fashion uh which would allow the read replicas to be created as traffic increases and then and then those read replicas be taken away as traffic decreases um in addition to this you can also scale to a globally with amazon aurora global database this allows you to have cross region replication uh to any part of where any database aws region exists in the world this allows for a better customer experience being able to host your databases wherever they actually live you can track your performance metrics through performance insights and enhanced monitoring being able to enable performance insights insights and enhanced monitoring inside of your database is a great way for database administrators and for those who are looking to understand their database performance you're able to drill down to the actual query level of what is occurring on the database tie it to particular sql weight types such as network issues cpu issues memory issues uh and other certain weight types and you can understand exactly what's going on with your database uh being able to take back to your developers and or your fellow database administrators and saying hey we need to optimize this query this is taking 80 of our resources in addition to that you have something called enhanced monitoring which allows you to monitor metrics at the os level this gives you a a more of a particular insight into the operating system um even though we have these safeguards built around aurora and you can't actually access the operating system we've enabled enhanced monitoring so that you can see and monitor exactly what's going on at a process level i really love performance insights it's something i encourage all my customers to use and i've gotten great feedback from them on performance insights just the ability to look at like a simple dashboard and see the what's going on with your sql queries and what the impact is on your actual instance is just is great being able to to see it kind of in a a way that that allows for a graphic experience it shows you exactly where your weights are and what query they're tied to and it makes it extremely efficient get in there understand your query try to optimize it and move on from that issue we also have the opportunity to scale vertically to handle new transactions and increased app usage you can scale up to the 16x large instances in amazon aurora which is a fairly large instance for more information on that please see those instant sizes being able to scale up to that size both on the on the primary and also on the reed replicas makes for a massive scaling experience probably a good time to talk about serverless instances as well they give you the ability to scale instantly up to 128 aurora capacity units which basically equates to 256 gigabytes of ram and that's a new feature that's been really great a lot has a lot of customers interested that's a great point and with aurora serverless v2 you have a lot of the same features that you have with aurora such as cloning the automated backups the uh replication across different availability zones as well as up to the 15 read replicas with aurora serverless so great new addition and then amazon elastic cache is the last thing we'll mention here it allows you to choose the numbers of shards and replica based on your caching needs so depending on the mode that you choose you can you can replicate and and scale as well and anytime we're talking about migrating to aurora it's really important to understand uh the need to appropriately right-size the environment um so you want to create a good poc that will address your success criteria for the viability of moving to the platform we really want you to understand uh what how your how your database is going to [Music] to perform on the aurora platform you want to identify the number of transactions per second for both read and writes you want to be able to test your workload based on expected users and the the amount of transactions you want to look into oversizing the database initially to set up and then reduce the database once traffic is stable so oftentimes we'll see a customer they'll uh double the instant size of what they feel that they need and then they they right size they monitor they see exactly the type of traffic that's going on there and then that's a great thing about amazon aurora and the cloud in general is that you can scale back down to only what you need this will help you identify the the appropriate instance size for your workload and then also uh heavily monitor for cpu and memory utilization both from a cost savings and a performance perspective is is really important when you're migrating to the cloud one aspect of right sizing that we traditionally talk about is is right sizing your storage right i mean in aurora you no longer really have to worry about that you have up to 128 terabytes of storage and that's going to grow and shrink as your data grows and shrinks that's a great point the storage does automatically does automatically scale whether or not you're on aurora serverless or aurora standard continuing our discussion for right sizing the database environment we want to talk about a little bit more about uh monitoring your database workload being able to monitor the the queries that are coming through where whether or not they're hitting the cache whether or not they're going to the database disk layer is very important to understand in amazon aurora with aurora you're charged for i o operations that occur when it goes down to the disk layer so monitoring that workload to determine if you have if your cache is the size appropriately will help you get the most cost savings and most cost efficiency possible when you're when you're moving over to aurora from your on-premise environment a question i often get asked is why should i trust aws and amazon aurora with my databases especially in the financial services department it's a great question i get asked this often as i meet with with with many different customers and what usually turns into a great discussion that we can talk about going back to that poc let's build out that poc i don't want you just to adopt this we don't want you to have blind faith that this all works we want you to test the process uh as an example you can test your database failovers does multi-az failover actually work does it give me the response times that the aws documentation says it does rmi databases being able to be backed up and restored in the way that the documentation says as part of that poc we encourage you highly encourage you to to test off to check off those boxes yes the automation is working as promised um start with a small database workload or smart with maybe a start with a a less risky application build that muscle build that migration muscle build the faith build the confidence that aws can be a truly trusted partner and as you right size for for cost for performance and for security uh that will allow you to build that trust and be able to successfully migrate your workloads going back to the cash workload discussion it's very important to allocate enough ram to your instance as we talked about before and these two metrics down here volume read iops and buffer cash hit ratio will help you understand how much of your queries are going to the query uh they're going to the cache versus the disk subsystem you want to keep your transaction short in aurora you're going to get the most bang for your buck doing that and then you want to paralyze your workload in your queries the disk the database subsystem uh is is going to uh highly be highly efficient when you paralyze your actually your your actual workload your queries on the topic of proofs of concept it's important to know that you're you're never alone throughout this process uh you know we have great account teams with a lot of different technical experts on all these two include database specialist solutions architects like dustin and i dustin's given a lot of great corners to help people look around but you're never going to be alone throughout this process so that's something important to keep in mind and let's go through a few custom just a couple customer examples it may be helpful to see how other organizations have actually uh have actually migrated their financial systems to amazon aurora after pay is a good is a great example after pay is a payments platform that helps customers particularly uh generation z and millennials make payments in accordance with their lifestyle allowing them to purchase a product immediately and pay for it later um after page chose aws to create a centralized data platform that would allow afterpay to bring together disparate internal sources of data store the data and enable the querying of that data with aws after pay can now query data in 45 seconds down from 45 minutes on premises and execute its etl processes and these processes are down to about 15 minutes reduced reduced from about 12 hours of retail processing time before another great example is is ppro ppro moved the implementation of alternative payment connectors to aws using cutting edge technologies to attract developers and accelerate their onboarding after that experience all applications are in the migra are in the migration process to aws and they're continuing to migrate their infrastructure with aws ppro is able to scale seamlessly and increase developer productivity this fintech also benefits from high availability in the cloud using amazon aurora and also amazon dynamodb ppro's multi-sided network of global payments companies use ppro's apis for a variety of functions including a localized gateway processing and merchant acquire services i really like this example because it calls out the fact that they used amazon aurora and amazon dynamodb customers don't really ever have to choose to just go move their database from one database to another aws service oftentimes a combination of different services is the right answer using the right purpose-built database for the right workload is what we're all about so how can aws help right we've talked we talked about the the dbgalvanize program we've talked about uh trusting us with your with your financial systems on running on our infrastructure and our purpose-built uh managed database services uh how can we help as a solutions architect uh architects jared and i are our uh our goal is to is to and our primary function is to help customers succeed in their in their pocs and their use cases in understanding uh the best path forward to to migrating their database infrastructure we can give you uh financial systems use cases and case studies that way you can see and read up on how other financial industry industries have migrated to the to amazon aurora we can give you uh the most modern data strategy best practices uh we have the state of the art information uh at our fingertips and uh access to uh a lot of great people that can that can help uh give lots of prescriptive guidance for best practices we have access to webinars executive roundtable discussions and and data monetization week sessions that can uh that can allow you to adopt and and and migrate even faster with these database migration workshops you also have access to uh particular partners that can help accelerate and actually help you do the work uh which we have access to to to make those introductions uh we can give you some proof of concept credits which are very nice uh when you're doing your testing so that during that time when you're trying to build that trust and build those uh migration muscles and uh the knowledge base that you need uh you you can do that on our dime and then uh database specific account teams uh you have access to uh specifically your account team uh they have access to a database solutions architect please reach out to them we can get you uh some help for for uh for successfully migrating your your financial your financial systems well dustin i think that covers it what do you say we go into some question and answer session yeah i i think we're ready for that we have uh justin and sudar here to help us answer these questions as well so looking forward to that [Music]