Introduction to Payment Systems | System Design

hi guys welcome back to the channel hope you all are doing well today we're going to be talking about payment systems this video is going to be an introduction to payment systems we are going to look at an end to end flow of a user landing on a purchase page and how the back end and front end works together with a third-party payment service provider to complete the purchase successfully later in the week i'm gonna have another video related to payment systems but that is going to be in context of an e-commerce application where you have both a buyer and a seller okay so you can see in this screenshot over here we are gonna make the assumption that we are using an existing payment service provider such as stripe or paypal that gives you a checkout page to render to the client however you still need multiple different ways to integrate both your front end and your back end with the payment service provider that's what we're gonna be talking about the most okay so before getting uh into it let's talk about some of the things i will not be talking about in the video the first thing is going to be storing credit card information usually this involves a lot of different security and lots of data privacy laws that you will have to comply with so we are not going to store any credit card information in our back end instead we're gonna rely on the third party payment service provider to do it for us we're also not gonna talk about things like fraud checks connecting to bank accounts charging credit cards directly we are gonna let a payment service provider like stripe or paypal handle the actual transaction all we're gonna talk about is the integration that we need to do with one of these payment service providers now that we talked about what we are not going to be mentioning in the video let's talk about what are going to be included in the video the first thing is we are going to be talking about a third-party payment service it can either be paypal or stripe but the main thing we're going to be focusing on is how our application can integrate with one of these payment services instead of building or reinventing the wheel when it comes to payment services we're also going to be talking about what kind of database we want to use in our back end for applications like these we're going to talk about what kind of data types are more commonly used to store information like currency and purchase amount we're also going to talk about synchronous or asynchronous communication between our backend systems usually in a payment system you will have multiple different microservices talking to each other so we're gonna talk about how you wanna design the system so that the communication between the services can be as efficient as android burst as possible finally we're going to mention something related to the most typical issue when it comes to payment system which has to do with double charge where you charge your client twice for the same transaction we're going to talk about what it is and how you can avoid it in your system even though you are going to be using a an external payment service provider you will still need to design certain things in your application to make sure that all the features are working correctly all right so let's begin with talking about why do we want to use a payment service as opposed to building our own the first thing i already mentioned before has to do with credit card information of course these kind of information it's very it has to be stored in a very secure way you have to comply with a lot of different data privacy laws and usually it gets very expensive to maintain a system that can do all these for you similarly when it comes to credit card systems you have multiple different credit card uh systems such as like visa discover mastercard amex and if you were to build your own payment service as part of your application you'd have to integrate with all of these in a very unique way which can get very expensive and difficult to maintain and uh i would say the most important reason is you want your application and the engineers working on the application to focus on the core logic of the application rather than reinventing the wheel for things like payments where you already have a solution that you can use now of course if you have an application that is very large in scale you might run into cost issues when using some of these third-party solutions and of course that's going to be something you want to reevaluate but for most cases where you're not working on too large of a scale you should opt for using one of these payment services rather than building one from scratch all right so we are gonna break down the purchase flow of a user into two distinct sections one is gonna be what happens when the user is about to check out up and they land on the purchase page and the other section is gonna be related to what happens when they hit the buy now button after filling all their details so let's start with the first section which is going to be what happens when the user is ready to buy something so i'm going to zoom in a few levels so that the picture is clear that should be good okay so that's what we're going to take a look at yeah let's see what happens when a user is about to check out so you can see we have our system over here and the steps i have mentioned i have numbered them so that should be easy to follow the first thing to note is we are using the purchase page that the purchase service provider is giving us so paypal stripe these are the more common purchase service provider and they give you a purchase page that you can use in either your mobile application or your web applications so we're going to go with the assumption that we will be reusing one of those uh checkout pages so uh let's say the user is ready to check out and we wanna show them the purchase page now let's see what's the steps involved in doing that the first thing is you can see in the yellow box over here this is your back-end payment system of course to simplify things over here i have the whole back-end in one box but usually or more typically you are gonna have multiple micro services or multiple systems working together to do this whole flow but to keep things simple and make it easy to understand i'm going to treat the whole back end of ours in this one yellow box the first step is going to be the backend system so your backend system is going to call an api of paypal or stripe and register the payment session with a unique checkout id so every time a user is about to check out some items or buy a certain thing you need to generate a unique checkout id related to that purchase once you send that checkout id to register the purchase with paypal or stripe they are going to give you a unique token back you're going to store this token along with the checkout id in your database we'll talk more about what database to use but for now you just imagine it as whatever database that you want to all right so the database is essentially going to have a mapping between your uniquely generated checkout id and the uniquely generated token that the paypal or stripe is giving back to you this way whenever you're looking at the purchase status of a certain checkout you can easily use the checkout id to find the token and then use that token to call paypal or stripe to give you the purchase status all right so step one is going to be registering the payment session with a uniquely generated checkout id step 2 is going to be stripe or paypal giving you the unique token that maps to that checkout id and then step 3 is going to be storing this mapping between your checkout id and the payment systems token into a database now that you have that token and you have registered the purchase that's about to happen you can go ahead and render the purchase page with the token that the payment service has given you over here you see i believe that's the checkout page of or the purchase page of stripe but stripe paypal all gives you customizable purchase pages like these that you can easily use rather than rebuilding the whole thing one of the key information that you do need when rendering this page is going to be that token that you're getting in step two back from a paypal or stripe so at this point your purchase page is being shown to the user who's about to buy something and it does have the token on the client end which you do need to actually go ahead and complete the successful payment now let's move on to the next bit where it's going to be related to what happens when the user clicks the buy now or purchase button after they fill in all their details okay this is where it gets a tad bit more complex so i'm gonna just make sure it's zoomed in enough there you go okay hopefully you can see the whole picture once again all the steps are numbered i am gonna have the all the diagrams and notes that you see i'll have all of them linked in the description if you want to take a look uh but yeah let's get get into it now that you have the purchase page showing to the user let's say the user goes ahead and like fills in all their details and hits the buy now or purchase now button now what happens the first thing is of course your client is going to make a request most probably it's going to be a post request to your backend system with the purchase information and the token right so purchase information can be like the user id of the user that is about to buy something and maybe the item ids or something like that right so information that's not super sensitive so the client is going to make that request to your backend with those information along with the token that you that was generated by paypal or stripe when you registered that session if you remember over here in step one was to register the payment session and in step two you were getting the unique token so you're back and your client needs to send that token back to your payment system when the user hits the button once you have that of course you can in step two you can go ahead and store the status of the purchase that's about to start the status can be an enumerated value so you can have things like started completed failed or like succeeded stuff like that so once your backend receives the request the first thing it can do is store the fact that or store the information that a certain purchase has started and you can have information like the user id the items the amount the token and all those things in that database once you have a record of the purchase that has been initiated you can go ahead and call some api endpoint that paypal and stripe is going to give you in this api request you need to pass an information similar to the purchase information which can include the total amount the currency and all the stuff you need to send send them the token this is the token that was generated by paypal or stripe in the first place and then you give them a redirect url this is the url that you're giving paypal and telling them once the payment is done or the request is successful redirect the client to this url so it can be like a purchase success page or if the purchase was not completed synchronously it can be some kind of a dashboard that the user lands into where they can see the status of their purchase so that's going to be step three your backend actually making the request to paypal or stripe to complete the transaction for you now let's say of course in an ideal world the purchase is going to be successful immediately let's say you get the response back the credit card of the user has been charged and your back end can go ahead and once again store let's say step four step five yeah so your back end if you see we have step five right here uh your back end on a successful response can go ahead and update the purchase status in your database table just to make sure that you have a local version or local copy of the state of the world and then in step six as you see over here the user is redirected to the purchase success page given we are assuming that the purchase was successful okay now one other information you do have to give paypal uh i'm not sure if it's optional or they always need it but one thing that uh you do like one of the thing you need to know about is something related to the web hooks endpoint this is going to be an endpoint that you register with paypal or stripe what that endpoint does is in step 7 let's say in step 3 when the backend is actually sending the information to paypal let's say for whatever reason the payment is taking longer right at that point in us even though it looks synchronous but in a system like this paypal won't block your request until the payment is done instead what it can do is give you a response back like tell you something like hey the response is successful but we're working on it and then what it's going gonna do is in step seven over here whenever this purchase is successful let's say they had to do a bunch of fraud checking and a bunch of other things and it took like a few hours for the purchase to go through what paypal is going to do is it knows that webhooks endpoint that you registered with it when registering your application so whenever the payment is successful paypal is gonna or stripe is gonna make a request to that web hooks endpoint with the token id or some id which uniquely identifies the purchase that is still in progress and if you have some logic in that webhooks endpoint what you can do is based on the request coming from paypal you can go ahead and update the status of the purchase in this database table let's think of an example where the purchase was initiated so right now you store it in your database with the status set to start it right and now your backend system talks to paypal people tells you okay we're processing it's going to take us some time and you go back and update the status of that purchase to pending and now let's say four to five hours later paypal actually goes ahead and completes the transaction and it knows the webhooks endpoint which you own so it's going to make a post request to that webhooks endpoint with information that uniquely identifies the purchase that was just completed and then in that endpoint you can have logic to go ahead and update this table once again setting the status from pending to successful in in the case where the purchase was a failure you can do something very similar where paypal still calls your web hooks and point when the when the transaction is complete and tells your endpoint that hey uh this purchase with the token of xyz field for whatever reason and then you can go ahead and set the status of that purchase to failed okay and whatever your purchase status page is so the user usually can check the status of their purchase in a certain page you can pull the information about the status of that purchase from the table that we talked about here okay so hopefully that gives you an idea about the end-to-end flow of what happens both in terms of rendering the purchase page and then going ahead and actually executing a purchase end-to-end okay now we'll dive a bit deeper into the database that we were talking about of course i'm keeping things simple here we are not talking about uh too much or we're not going into too much details about each of these system but the idea is to give you a more general picture of how a purchase system can work in practice so let's talk about what does the database look like that we're talking about here so it can be pretty straightforward so you can have something like this where you have the checkout id of checkout id which has to be unique because every single checkout should be unique then you have the psp token which is the payment service provider token that is mapped to that checkout id if you remember every single time you are registering a checkout with paypal or stripe they give you a unique token back so you're storing that mapping over here and then you can store information like the amount of that transaction the currency and the status which is gonna start with pending and then it can go to complete it's gonna be complete ted one second yes it can be compending completed or failed all right a few things to note about the schema of the database of course over here i just have this for example purposes for but some important things to consider one is you can add a unique constraint on your checkout id this is gonna make sure that every single checkout is unique so whenever or for whatever reason you're trying to insert a row with the same checkout id it should give you an error the next thing is the psp token you see over here that's generated by paypal or stripe and it directly maps one to one with a checkout id now uh the tricky thing is going to be storing the amount or the money you're charging the user in your database if you look at the data over here you might think this is like a floating point number or some kind of like a real number in your database but that is not the best thing to do when it comes to money because of the way different databases and different systems deals with integers and also floating point numbers so what you want to do is when storing the amount so as long as it's any kind of money related amount if you're storing it in a database or if you're transmitting it over the network it's always safer to do that as a string right you want to store it as a string as opposed to either an integer real number or floating point number the reason you want to do that is different uh technologies like whether you're using postgres cassandra or even like the when transmitting as json or parquet or whatever format you're transmitting the data in if you store it as a number different technologies can uh think of that number differently based on the way the deal with floating point numbers so the safest thing you can do is just store the whole amount as a string so you can let's say you have 56.00 so this can be like literally like a string okay what this gives this gives you the ability to both store and transmit the data as a string and then whenever you need to do some calculations you can go ahead and convert the string to a floating point number right because at that point you can be guaranteed that during storage or uh while transmitting the information that string did not really change whereas if we're storing it as a floating point number depending on the technology you could lose some accuracy which would of course mean charging your user more money or less money all right so not safe to store amount as number we talked about that and even when transmitting typically people use json but you can have other standards that you use when trans when transferring information from point a to point b so it's always safe to do the all the transmission using string in the amount rather than a floating point number or integer and if you take a look over here the status completed so of course ideally if the purchase goes through immediately your back-end is gonna just update it to complete but on the off chance that the payment is taking too long of course paypal is gonna call that webhooks endpoint that you register with paypal with the purchase success information and of course in that webhooks endpoint you can make that update the status in the table here all right so i think one thing you've noticed from the table over here is it is a pretty typical my sequel or postgres like a typical relational database as opposed to the more non-relational solutions like a dynamodb or cassandra there are a few reasons for that uh usually for most applications when it comes to these purchase operations your queries per second or qps will be relatively low so from a performance point of view most probably you won't have any issues just because of the lower qps and a relational database should be enough for you to handle handle the load of purchases coming through now of course you can have a very very popular application and uh if you niche down into applications like amazon which primarily deals with like people buying things you might run into a different use case but more often than not in app in whatever applications the qps when it comes to purchase and points should be relatively low uh making it very easy for you to use a very simple relational database there are some cool things you get by using a basic relational database like mysql and postgres one thing is you get transactional guarantees you also get proper consistency for both these properties that you see here these properties are very important to have in any kind of purchase or any kind of checkout application so just getting these by default from your database is a very very good thing to have if you used some kind of non-relational database like cassandra which is not really consistent and does not give you transactional guarantees then you might have to implement both of these as part of your application code which can get very difficult uh to maintain one thing to keep in mind is even if you are using a relational database if you have a database system where you have one leader node and three or four replica or follower node and let's say for whatever reason your purchase flow or your purchase system is being bombar is being bombarded with requests then what can happen is when writing or reading the data you can start running into issues because of replication lag if you remember replication lag it is essentially the the delay between a data being written into master or leader and the same data being reflected in the follower or replica nodes so uh let's say if you in a very very high traffic time if you're writing all your data to the leader node and then during this purchase flow you're reading the data from your follower node you can get inconsistent data with kind of which can affect your payment system if you do expect something like that to happen uh depends on the kind of setup you have going on it might be worth thinking about uh both reading and writing from the master or leader node for critical paths of the purchase flow this is gonna make sure that you're always reading and you're always writing and also reading the latest data in this critical purchase flow all right uh we have two more things to talk about so first thing is going to be the microservice communication so if you remember from the diagram here of course we had our whole back end illustrated in this one yellow box we did this to keep things simple usually or more realistically you are going to have multiple different systems or microservices working together to form this one yellow box and of course that makes you think if you have a system with different microservices all uh all of them lie in this path of a user hitting the pay now button and the purchase being successful how do you communicate between these microservices okay uh so yeah uh there are fundamentally two different kinds of communication one is synchronous another is asynchronous so let's talk about both these in the context of microservice communications in a purchase system or a payment system so of course in asynchronous communication you rely on synchronous http request this means if you have two services a and b a is going to make a http call to b wait for b to respond let's say if b as part of that request has to call c now a has to wait until b and c both are done and it gets the response from b that's essentially what a synchronous communication is the pro of using synchronous communication is it's easy to set up and it's very predictable you know how many services you have you know who's calling whom and what order so it's always easy to set up and also very easy to debug however in the context of a payment system having your whole system synchronously designed can have quite a few problems the first thing is going to be that any component in the critical part of the critical part of the purchase flow can fail let's say you have three or four microservices in between the client and talking to the paypal or stripe if you make the whole thing synchronous then any one of these microservices can fail at any point at which uh at which point you are gonna kind of like fail the whole purchase of course with synchronous communication you will potentially run into timeouts let's say you have a b and c a talks to b and b talks to c uh if for whatever reason when b is trying to talk to c it's taking too long the request that initiated from a might just time out right so if you have synchronized communication and you have too many dependencies along the way it's very typical for you to run into timeouts which in a in the in the case of a payment system that's implemented with a synchronous design will mean that your purchase is going to fail both the two things i talked about makes the whole system very brittle and when it comes to critical paths like a purchase system it's not really nice to have a brittle system when we talked about timeouts that can lead to a snowballing effect so just because one of your services is very slow it can affect all the other services that are actually calling that slow service and overall due to all these timeouts all these failures and snowballing effect you are gonna potentially run into a lot of inconsistency between different databases related to purchase information now on the other hand you can use uh asynchronous communication in my opinion that's the best thing to use for a payment system because of a few things that we're going to talk about now so essentially uh asynchronous communication relies on some kind of a messaging system like kafka rabbitmq or whatever technology you can think of there are a few pros of using an asynchronous communication system when it comes to the purchase flow the first thing is better failure handling so for better failure handling what that means is when you have messages in a message queue you can easily keep the message to be retried later on so for whatever reason if when consuming the message your service is running into any errors you don't have to fail the whole purchase instead you can just try again later on this makes the whole system significantly more decoupled now at any point in the system or any point in the flow if a certain failure happens you don't just feel the whole purchase instead you just keep the status set depending and try it again a few seconds or a few minutes later having such a decoupled design in such a critical flow gives you the benefit of trying or retrying a purchase multiple times instead of having everything set up synchronously and then failing the whole purchase when just one of the component is failing of course it's not a silver bullet it does come with a few things that you need to worry about two things i have over here is asynchronous systems are always difficult to debug and set up when you're setting up a system with messaging queues it's not as straightforward as two services just talking through http uh requests so yeah you have to keep that in mind even for debugging it can get a bit more confusing of course you can set up different infrastructure and tooling to help you with the debug process but compared to a synchronous process this can get more difficult to both set up and debug all right so that brings us to the last thing which is going to be related to the double spending problem this is a pretty typical problem in payment systems we don't really run into this anymore right now but i have read a lot of blogs and articles on how in the early days of the internet this was a pretty common thing people ran into so if you remember from here we did talk about retrying when our purchase fails that's one of the big benefits you get out of asynchronous communication and the retries even though it's amazing because you're not failing the purchase immediately retries can lead to double spend uh if you are not careful because if you think about it if a purchase uh is failing for whatever reason and you're trying trying it three to four times then you're literally trying the same purchase three to four times and if you don't have things set up that might mean that you're charging the same user three to four times for the exact same purchase so uh the most common way that these payment systems get around the double spent problem is through what i highlight what i highlighted over here every purchase page will have a uniquely generated checkout id and a corresponding token generated by the psp what that means is if you remember over here when rendering this purchase page you are generating a unique checkout id for this page and stripe or paypal is giving you a unique token that maps exactly to this checkout id so those two things your generated checkout id and paypal or stripes token these two things in tandem is going to make sure that the double spend problem does not really happen okay uh all right so uh how does double spin usually occur so we already talked about it uh when you have multiple systems not only within your own back end but also when you're talking to third party systems like a paypal or stripe you can have network errors at any step especially when you have the whole thing set up asynchronously and different components are going to retry different number of times you can easily run into network errors similarly for whatever reason your client's browser not might re might not refresh properly or might not be redirected redirected correctly and both of those can lead to your user clicking the buy now button twice so let's say they hit by now nothing happens and then they hit by now again that is one of the main source of double spent problems because now the user initiated two three four five purchase requests by spamming the button now uh the most common way that payment system providers or like paypal or stripe the way they handle uh these kind of double spend problems is through that psp generated token so whenever you are making a purchase request to that psp so i'm talking about this request over here your backend calling paypal to actually complete the transaction when whenever you are making this request you are sending the token which uniquely maps to your checkout id right and this token is coming from paypal or stripe so they are aware of this token okay so yeah you are always sending this psp generated token to the psp for every request you're making what the psp does is it recognizes the token right and if it has seen that token before it's just gonna treat that request as a retry rather than treating it as a whole new purchase it's going to treat it as a retry what this means is if the purchase was so let's say you make a purchase request to the stripe or paypal api they see the token they look at at their database or whatever system they have and then they see that oh we have seen this token before and for this token the purchase was uh not successful for whatever reason then the psp is going to retry that request for the same token instead of creating a whole new request now of course if it failed and you're retrying that's amazing there is a chance that it's going to be successful now and everyone's going to be happy but on the off chance that the request was already successful right let's say you make your first request with a given token and the request was successful everything worked out but for whatever reason that client's browser did not or the user's browser did not really update so they hit the pay now button again and now your backend is making the same request to the psp with the exact same token the psp is going to look at the token and look at the status and see oh this token has already been completed right so the request with this token that purchase was already successful and it's just going to throw away that request rather than charging the same user the same amount of money yet again if for whatever reason the user is literally spamming the button and the psp is getting 10 20 requests every second coming with the same token instead of just discarding it it can start throwing a rate limit or i think it's a four to nine uh just to tell the back end hey this yeah you shouldn't keep on sending me the request with the same token given i know that it's a successful purchase so yeah so hopefully that gives you a good idea of how things work this was more from an introduction point of view i did not go into a lot of details of course you can easily zoom into any one of the components we talked about and talk about different algorithms or technologies you can use but my goal was to give you a more overall high level overview of how a purchase system works or a payment system works when you're integrating with a psp like like uh paypal or stripe even though it might be very easy to think that you can just call an api and just forget about things but in a more real life system you will need to design certain things in your backend to integrate in a robust way with this psp all right so hopefully that was helpful if you have any questions just leave them in the comments below and i'll try to get back to you as soon as i can do look forward to the video i'm gonna upload on friday i believe that's gonna be related to what we talked about payment systems but we're gonna talk about that in the context of uh e-commerce application where you have both buyers and sellers in your application this example was more related to you are the seller and the buyer is the user who just bought something from you so things are more simple that way but if you have a marketplace product where you have both a buyer and a seller you have to keep a lot of other things in mind like a ledger a wallet and then you have people paying you and then you paying the seller so yeah on friday i believe i am gonna upload that video but i would highly recommend checking this one out first because otherwise that might be more confusing all right with that being said i hope you guys have a good rest of the day and i'll catch you in the next one bye