Python Web3 Development #7 - Advanced Architecture and Security

Key Takeaways

hello everybody and welcome to part seven of this tutorial Series where we'll be getting into the architecture of cartei and talking about this more in depth compared to what we've gone over in previous videos now I'm going to start by giving you a general overview and summary of the main components that we're already familiar with just to make sure everyone is up to speed and remembers it then we're going to dive into some of the more complex components for example we're going to talk about consensus and we're going to talk about how cartez rollups are different than other types of Roll-Ups and why that's important for you to understand so let's begin by going through a highlevel overview of the main components of a caresia application so we have cares rollups caresi machine the DAP backend and the DAP front end now as we've seen here and we've looked at this diagram before what we're responsible for building is the front end and the back end the rest of the components are already built for us and they make up kind of the caresi rollups framework now as we saw in the previous video caresi rollups itself is a set of onchain and offchain components that implement ment the optimistic rollup solution now the onchain components include smart contracts right so for example input box part of caresia rollups our dap Factory again we need to deploy that onto the target Network then we have the caresan machine now this as we know is a virtual machine running an entire Linux operating system in which the dap's backend is executed now there's some more details related to this for example it runs a risk 5 architecture which is actually able to be hashed allows us full trans parency and gives us some features of the Cartesian machine which are important and actually kind of go into the security and details implementation Etc which we can talk about later regardless the caresan machine is where we run the daps backend now this is where we have the state and verifiable logic which corresponds to the smart contract this is what we wrote in Python and what we can write in pretty much any programming language that runs as a Linux application then we have the front end works the same as any other front end for a decentralized application we built this in react and JavaScript so here is the general architecture that we're already familiar with all right so now let's dive a little bit deeper into cesi nodes now the caresi node itself is the L2 component that consists of the combination of the cartez machine that's what runs our backend and some specific middleware now what this middleware does is actually manage and controls the communication between the onchain components and the offchain components so this middleware what it does is read data from the L1 smart contract this is what's running onchain and this would be the inputs for example that are sent to the cazan machine and it's then sending them to the machine to be processed then whatever the caresan machine spits out so a voucher a notice report Etc it can then publish that result back to the blockchain so that's kind of the basic idea on how the Cartesian node works you have the middleware which is actually communicating with the layer one smart contracts and then you have the caresan machine itself which is running the back end in an isolated environment now as we see here in practice there two different types of our caresi notes user and validator notes now each of these interact with the onchain rollups in a different way and thus run different types of Cartesian notes so the user or reader notes now these are the most basic and what these are doing is just advancing the state of the offchain machine and making that state publicly available now anyone can run a user or reader node whoever is interested in the state of the decentralized application is able to run one and that way they can always have up information about the cartei DAP they also could use one that was made publicly available then we have a validator node now the validator node is doing something a little bit differently what this is doing is not only uh keeping track of and advancing the state of the offchain machine it's instance of the offchain machine it's also trying to fight dishonest validators so pretty much what it will be doing is checking at every single Epoch if the state that was submitted to the blockchain matches the state that it has so so a bunch of these different validator nodes will run they'll all come up with what they think the state of the Cartesian machine currently is or what it should be next and then if there's some kind of dispute they will resolve that dispute by taking part in some kind of verification game we'll talk about that later on but the idea is just like we have nodes or we have miners that are running on ethereum we can have the same thing but specifically for our cartez dap so we can run as many validator nodes as we want anyone else can run a validator node and that means we have our own set of infrastructure for our specific decentralized application that still is decentralized Right anyone can run a usern node anyone can run a validator node and these validator nodes are going to be checkpointing State updates on the blockchain and ensuring that all of the states that are being receive from the other validator nodes are indeed correct so very similar to how a normal blockchain network works in this case it's just for our specific decentralized app which we'll talk about a bit later now this leads us a little bit into EPO so pretty much much what we want to avoid doing here is constantly interacting with the blockchain if we don't have to so what we do is we have these validator nodes that do not checkpoint every new state update what they do is they're updating or checkpointing batches of State updates at the end of a specific Epoch so as you can see here it says in order to end an Epoch all validators need to reach a consensus about the state of the machine after processing all of the batched inputs so validator 1 takes all of the batched inputs in this Epoch and determines that at the end of all of these inputs the state of the machine is X then you have validator 2 if they come to the same agreement okay we're going to go ahead and checkpoint what the state was at this point in time however if they were to disagree then they have to take part in some kind of verification game and they have to dispute this by actually running some computations on the blockchain which we're going to get into in a new section so that's kind of an overview of the main things we already know now let's also just talk a little bit about sending requests so the the advance and the inspect now we already know how these work because we've used these quite a bit but there's two ways we interact with the cartei machine we have advance and we have inspect now when we advance what we're doing is we are sending an input using the blockchain network to our decentralized application the important thing here is that all of our inputs are logged on the blockchain so they are managed securely by layer one now these will be batched and they'll be processed by our validators and by the caresan nodes but the idea is anytime we want to advance or modify the state we do need to send a transaction and we do that by sending an input now if you're interested in exactly how this works we're using a Json RPC call to the input box contract we don't need to get into that too much but again the advantage here is that we don't need to actually execute all of the logic on the blockchain in a smart contract like we normally would we can send whatever data or whatever um Advance request we want through this input box contract and then all of the processing which can be quite large as we have seen can be handled offchain and then logged on chain then we have the inspect request now this is an external HTTP API very simple it's a call to the Cartesian node to Simply read what the state of that machine is now we can see that there is a video here it kind of goes through how this works so feel free to watch that and it kind of talks you through how you actually add an input how you do an inspect Etc so if we want to advance the state what we need to do is actually send a request using our blockchain Network so in this case the ethereum node we trigger the advanced State request and then that hits the cartez node so we have this in between step where we need to actually go to layer one however if we want to send an inspect we can go directly to the caresan node because no State update can be invoked with an inspect request only with an advance request so inspect very fast you go directly to the node advancing you still need to go through layer one and you need to have that input stored on layer one it will then get batched into an Epoch and it will be validated by the valid validator nodes that are running on the cartei network or for your cartei dap sorry all right so now let's dive into the cartei machine itself and talk about this a little bit in more detail so if we were running a normal decentralized application that wasn't using caresia and wasn't running on layer 2 then what we would be doing is relying exclusively on Smart contracts that are on layer one now these face severe constraints on the amount of data they can manipulate and on the complexity of the computations they can perform now these limitations manifest themselves as exuberant trans action costs and even if such costs could someh be overcome it's extremely long to run different computations if you don't believe me go and try to write a simple smart contract that does something like 100,000 Loops okay pretty trivial to do that on a standard computer try to do that on the blockchain and see what happens anyways the idea here is that in comparison our decentralized apps running inside the caresan machine can process relatively unlimited amounts of data at a pace of over four orders of magnitude faster now this is possible because they're running offchain so they are free of the overhead imposed by the consensus mechanisms used by blockchains which is really what slows them down now the double-edged sword here is that the consensus mechanism is also what keeps them secure when it comes to doing computations and running applications not just logging transaction data it causes things to be very slow and very expensive now in a typical scenario what's going to happen is our decentralized application will execute the Cartesian machine offchain and report it results to the blockchain different parties do not need to trust each other because the cartei platform includes an automatic dispute mechanism for the cartei machines any interested party can read the computations that are happening offchain and if they don't agree on the result of those computations so essentially the state update of the machine they can enter into a dispute and this mechanism guarantees that it will always be won by an honest party obviously there's a lot of math that goes into this but the reason why this is able to work is because of the three main properties of cartei machines first first of all they are self-contained second of all they are reproducible and they are transparent so that means given the same set of inputs the cartisian machine will always reach the same state or the same output so that's important because that means it doesn't matter when you run your cartisian machine who's running it in what environment Etc it's always going to reach the exact same output and it's verifiable that given a certain uh set of inputs this output was reached and we're going to talk about that by going into something thing known as the state hash which is kind of the main idea here behind the caresan machine now obviously there's a lot more information that we could dive into here and you can see that there's some pretty in-depth documentation about the cartisian machine but I think the thing that will uh kind of bring this full circle for you is looking at something known as the hash view or the state hash okay so hash view of state so one of the goals of moving computations offchain is to allow them to manipulate vast amounts of data so much data that it becomes economically prohibitive to explicitly store them in the blockchain nevertheless for smart contracts to delegate computations offchain they must be able to specify the computations their inputs and then reason over their outputs so this is key to solving the contradictory goals in the clever use of cryptographic hashes now the basic idea here is that the cartez machine itself is transparent so that means everything from the ROM the ram The Flash driv general purpose registers all of this complex computer architecture stuff that I don't know that much about is available for anyone to see now what that also means is that at any point in time we can take all of this information we can run it through a hash function and we can generate a unique cryptographic value that essentially encrypts or encodes the entire state of the machine now obviously I am oversimplifying this and I highly recommend you read this for more information on how it works but the idea is at any point in time I can go to my caresan machine and I can ask it to give me a hash of the state now only this specific state will generate the hash that it gives me and this hash is the main security feature that we use so for example if we have a certain set of inputs those inputs will lead us to a specific State we can then hash that state and we have essentially an encoding or a representation of what it is with very little amount of information that we need to store be transmitting we don't need to actually tell someone what all the ROM is the ram is the flash drives what could be multiple gigabytes of data we just take a hash of all of this information and that encodes this state so now another cartez machine or another cartez node can actually check the state of its machine and it can see if it matches the state that we have it can do that without having to have all of this information it simply needs the hash of the state so let me hop over to the Whiteboard here and I'll explain more about what I'm talking about all right so I'm on the drawing tablet here and I'm going to go through a bit more of how this hash view of the state works and then talk to you very briefly about the verific AED game now keep in mind this is very very high level grossly oversimplified so I just want to make sure that's clear if you are really interested in this please read through the white paper and some more of the documentation that I will link in the description so first of all let's say that we have a blockchain okay so this will be B this is our blockchain Network then we may have say two cartez nodes which are running on Layer Two and are keeping track of the state of our decentralized application now this blockchain itself is tracking all of the inputs that are sent to our decentralized app so it might have input one input two input three these are all actually logged on the blockchain now what will happen is these inputs will get sent over to our cartez nodes and they'll be executed by the cartez machine so these machines here will be keeping track of the current state of the machine and they'll be updating the state after each input comes in so input one gets sent over to machine one and then updates it state to represent the change after input one same thing happens here input one gets sent to machine number two and machine number two updates its state now after an Epoch so a certain batch of these inputs has been sent what's going to happen is both of these nodes assuming their validator nodes are going to report what their current state is back to the blockchain now they can do this efficiently because they can take whatever the state is so let's imagine all these lines here are the state they can effectively bundle it up and they can take a hash of this state so it can hash the entire State and then they can actually submit the hash of that state to the blockchain now this is valid because only one given state will generate one given hash and it's infeasible to find what the hash of the correct state would be I'm not going to dive into this too much this has to do with hash functions and cryptography but the basic idea is they're going to log after an Epoch what they believe the current state of the caresan machine is and they can do that using the hash view of the state now what will happen let's say that this guy logs you know the hash 1 2 3 and this guy logs the hash 456 in practice it be much larger than this I'm just using simple numbers is at this point they've disagreed on what the state of the carian machine should be after this batch of inputs so what needs to happen is we need to have a dispute resolution where we figure out what the real State should actually be now what would have been happening is the application beforehand so the smart contracts running our cartez dap would be logging what the state is after every single Epoch so it would say something like Epoch 1 and the state is equal to you know XYZ right Epoch 2 same thing you get the idea and it's keeping track of all of these states that have been agreed upon that had a consensus from these caresi notes so what we do is we don't need to process every single transaction that's ever been inv involved in our cartez dap we simply go to where we know we had finality so where we know the state was agreed upon and there was consensus and then we start processing the inputs that are not agreed upon so the state update that was not agreed upon on the blockchain so let me just really clarify this again because I know it's a little bit confusing pretty much these caresan notes here are constantly keeping track of the state of our decentralized application they're logging that back to the blockchain and when they both agree they say okay well that's good the state is final now we can move on and we can process the next set of inputs now the blockchain itself will just assume that everything that's sent from these nodes is correct so long as they reach consensus on that it doesn't need to actually process any of the computations unless there is a dispute so at this point where there is a dispute where these two hashes or these two State updates don't match what we do is rather than processing the computation offchain we actually process part of the computation on chain now this is more expensive and it takes longer but we utilize the security components of layer one to ensure that the state update that we're about to log onto the blockchain is indeed correct so again basic idea is when these two things do not agree so when our nodes do not have a consensus on what the state update should be we actually push the computation onto the blockchain and we compute the new state of update directly on the blockchain so we ensure that since it's computed there on the blockchain that it's going to be correct and valid and then we log what the next state update actually should be so that is the very very very high level overview of how the kind of uh verification or dispute process is done whenever we disagree on what the state of the caresan machine should be these two parties will play something known as a verification game they're going to interact with the blockchain the blockchain is is going to determine where the last known state that valid was it's going to start from there pretty much and it's going to take whatever the new inputs were that have not been agreed upon so the state after those inputs and it's going to process a partial computation on the blockchain to ensure it gets the correct State it's then going to log that on the blockchain and then we're going to continue moving on to the next Epoch so one of the main advantages of this system that you're seeing here is that we're only performing computations on the blockchain when they need to be performed to essentially resolve a dispute and we're only doing a partial computation which is much quicker and faster to execute rather than rerunning the entire caresan machine for example directly on the blockchain again this is a bit more complicated to really understand how this works at a deeper level but the idea is we already know what a final state was at a certain point in time we can essentially start from there just compute the part of the computation that was dis disputed and then we can log that result directly on the blockchain and we can have that immutability component where the blockchain says Hey at this point after these inputs this is the state and we keep going and we move to the next Epoch now another Advantage here is that we only need to have consensus among the involved parties so rather than the entire network verifying every computation the consensus is just reached among the parties that are directly involved in the transaction for example if two users are interacting in a cartez based application only those two need to reach an agreement on the outcome of their interaction it doesn't concern the entire ethereum network for example so that's a little bit about how the verification actually works here on carteia and the dispute mechanism again this all has to do with those main properties of the cartez machine and the fact that we can actually hash the entire State we then can use the blockchain when we need to actually resolve a dispute and we can have the finality there of a batch of transactions which is our Epoch okay now let's head back over to the computer and talk about the last few components so the last thing I want to go over here is something known as application specific rollups which is what caresia implements but in order to understand that we'll quickly just recap a blockchain rollup now this is a scalability solution that pushes complex computations offchain this means they run in a separate Computing environment outside of the main or base layer such as ethereum now when we use rollups the blockchain's role is simply to receive receive transactions sorry and to log them and as we saw on occasions where there is a dispute or a disagreement then we can actually execute the computation onchain but we don't need to do the entire thing it can be a partial computation where we can verify on the blockchain network with the correct State should be now doing this uh really drastically improves the scalability of our applications because we're able to compress data and we can increase the number of transactions that the blockchain can process because we're taking those heavy Compu mutations and moving those off chain when we're just using the blockchain for the sole purpose of logging transactions it can handle much more data much faster than if we were executing all of the computations directly on chain okay so how does a rollup work users interact with the rollup through transactions on the base layer they send messages in this case they're called inputs to the rollup onchain smart contract to define a computation that they want to be processed in this case it could be an advanced request of the Computing in environment then interested parties run an offchain component in our case a node that watches the blockchain for inputs understanding and executing their State updates once in a while the state of the machine is checkpointed on the blockchain at which point the state is considered to be finalized so once the state is finalized that's when we can actually say that a notice is correct and we can execute something like a voucher from caresan now there's two main types of rollups that we use zero knowledge Roll-Ups ZK rollups or optimistic rollups now now a zero knowledge rollup uses something known as a validity proof what this pretty much means is that with every single transaction it submits to the blockchain so every rollup it's going to actually produce a validity proof that can be used to verify that what information is submitting is correct this takes a little bit more time it's a bit more complex but the benefit here is that we have instant finality so as soon as something is submitted to the blockchain we can immediately say that it is indeed correct however it can be a bit expensive to actually verify these uh rollups because you have to do those on chain and it can be difficult to produce validity proofs then we have optimistic rollups now optimistic rollups use something known as a fraud proof and really the general idea behind an optimistic rollup is that we're going to assume that everything single state update that is sent to the blockchain so those kind of batched checkpoints are valid unless they are challenged so rather than requiring people to always kind of submit a proof and verify that what they're submitting is correct we just optimistically assume that whatever is submitted to us is indeed correct and then we only challenge that or we only verify that story if it is challenged so this actually removes a lot of the load from the blockchain because we're only utilizing the chain for computations when we have some kind of challenge or when parties disagree which ideally will not be too often now the way that this actually reinforce is by using Financial incentives that reward honest Behavior we don't need to get into that in this video video but that's the basic idea so moving into cte's version of optimistic rollups this uses something known as an interactive fraud proof now the idea here is that we are hardly ever actually having to do computations on the blockchain because either there are no disputes and no computation takes place or if a dispute occurs it only needs to compute a single instruction to distinguish between misbehavior and honesty in either case the results themselves are always computed offchain and as a consequence the computation involved can be massive that's that's kind of what I was talking about we only need to do a partial computation I don't want to get into that too much because it's a little bit complicated on how that works but the idea is what we're doing on chain is very limited and if a dispute does occur it's not that expensive to resolve so now we can touch on something known as application specific rollups which is exactly what cartez implements so normal rollups say like optimism arbitrum Etc at least the ones you might be familiar with are kind of a one siiz fits-all solution and they still actually suffer from a lot of the traditional computation issues with a normal blockchain Network now they actually are really limiting in terms of how you can build your applications and they're good generally speaking for a lot of different types of applications but they're not great for a specific type now caresia itself has application specific rollups which means that you can create your rollup however you want for the specific type of application that you are building so this allows you to optimize performance functionality Etc and just gives you more efficiency and flexibility than a standard RPS framework now a lot of the reason why this works is because of the caresan machine the cont consensus mechanism Etc but cartez has this integration with Linux right as we know that's why we're able to write our smart contract in python or really any language that we want and it gives us the ability to be very Dynamic and flexible with how we design our application as you saw we can design our own protocol we can specify what we want to do with inputs how we're going to handle inspect requests and we can do this in a very familiar development environment now part of the reason why this works is because we have off-chain computations with onchain security so as you saw caresia enables the complex computations to be performed offchain while still maintaining the security guarantee of the blockchain now this balance is achieved through the dispute resolution mechanism which we've talked about which differs from the general approach of other rollups where computations are simplified to fit the blockchain constraints so in other types of rollups you have to write your computations in a way where they could be executed on the blockchain with cartei you don't need to do that because of the way that the computations are verified we can write them however we want as you've seen using our traditional program languages and then they can still be verified on the blockchain using a partial computation that is executed on there so with all that said it just makes web 3 development a lot more accessible because again you're designing something for your specific application needs you're not trying to fit into the constraint of a blockchain network or of a pre-built solution you can do really whatever you want so with that said that's kind of going to wrap up this video I know there's a lot of stuff here I went through a lot of complex topics and I didn't make it 100% clear but I wanted to give you something to chew on something to think about and kind of encourage you to go to the caresia documentation if you're interested read more go through the white paper and really understand how all of this stuff works I think it's really interesting I'm still really just getting familiar with some of the more complex mathematical Concepts but I think it's really interesting how cartei differs from some of the other rollups Frameworks and the type of applications you can build with it as you've seen in this series and as we've kind of gone through this documentation it's really unlimited the type of apps you can build because of this Linux environment and the unique way that carteia handles rollups in the interaction with the blockchain so with that said I'm going to wrap it up here I hope you found this helpful and I look forward to seeing you in the next [Music] video oh