Blockchain Development: Querying with Open APIs Course

Key Takeaways

hey everyone my name is anu kubo and i will be your instructor for this blockchain development course all about querying data on web3 decentralized applications in this course you will get an introduction into indexing and querying data on the ethereum blockchain and understand why querying data on decentralized networks is so different to what we are used to on the traditional web goodbye http requests hello subgraphs we will be doing this with some explainer sections as well as some hands-on experience by building your own querying technology by the end of this course you will have the knowledge to query data from the ethereum mainnet as well as the variety of chains on the hosted service in order to build your own decentralized apps this course is made possible with a grant from the graph creators of the tool we will be using in this video but first let's cover just exactly what we'll be looking at in this course first off we are going to start off by looking at the graph a decentralized protocol for indexing and querying data from blockchains this will include why it was built in the first place and how to use it next we will move on to looking at sub-graphs which are open apis we'll discuss how subgraphs help you get the data you need to create and power your own decentralized applications or d apps we will then move on to the more hands-on approach and cover how to build a sub-graph with a step-by-step tutorial then move on to how to query a subgraph from a front-end application and how to manage your api keys once we have built a sub-graph it will then be time to put it to some good use and make some queries with it using tools such as the graph explorer which we will go into how to use together you will be glad to know that this course will take a very beginner approach so if this is your first time working with blockchain related topics you are in safe hands the only prerequisites i ask of you before starting this course is to have a basic understanding of javascript node.js and graphql however if you are not super familiar with both please have a go at following along anyway as there is still plenty to learn without coding i think okay so a very packed course for you today i'm excited let's get to it as always please do give us that sub if you like this video as this is the way this channel can grow and we can continue to create free content for everyone in the freako camp community so we have already covered what the graph is a decentralized protocol for indexing and querying data from blockchains but why do we need a protocol for doing so well as some of you may or may not know most of our activity online today happens on an internet referred to as web 2. web 2 is widely considered to be an internet dominated by companies that provide services in exchange for your personal data this was considered the norm until the introduction of web 3. web 3 bypasses all the large corporations and companies that web 2 relies on by communicating in a peer-to-peer way with web3 we are no longer relying on the companies as our central hub for our services but rather rely on peer-to-peer protocols in web 2 users are able to utilize free services in exchange for their personal data and in web 3 users pay protocols to work for them in exchange for self sovereignty and privacy in other words web 3 is decentralized and looks a lot like this this meant that whatever we built on web 3 is less reliant on centralized companies whose servers may go down causing up to crash as one of the many examples and more reliant on a distributed network of lots and lots of individuals two great examples of this peer-to-peer networks or in other words decentralized networks are the ethereum network an ipfs or the interplanetary file system network let's start with ethereum now what you might not know is that it is actually very difficult to query data on the ethereum blockchain directly this is because so much of what is stored on the blockchain is stored in a very complex way so currently when you want to find a file on web2 you tell the browser exactly where to look for the file you are searching for this location is essentially your url and this method for accessing files is called location based addressing this works fine until you guessed it the server where your files are stored goes offline or hackers manage to take something down and you can't access your files anymore web 3 overcomes this issue with content-based addressing this means that with web 3 when requesting a specific resource you do not request its location you request what you are looking for instead and it is most likely spread all over web 3 and not just in one location so super safe but hard to query so while you are able to find a specific thing that you are looking for more advanced real-world queries and operations like aggregation search relationships non-trivial filtering or other things you are used to doing with simple structures are just not possible to get this data you would have to process every event ever emitted by a smart contract two phrases we will be looking at next read the metadata from ipfs using the token id and ipfs hash and then aggregate it even for these types of relatively simple queries it will take hours or even days for a decentralized application running in a browser to get a response you could also build out your own server process the transactions there save them to a database and build an api endpoint on top of it all in order to query the data however this option is resource intensive needs maintenance presents a single point of failure and breaks important security properties required for decentralization this is where the graph comes in the graph solves this problem by indexing blockchain data once blockchain data is indexed the graph creates a sub graph that can be queried with your standard graphql api some graphs are just open apis the graph knows which data to index for your sub graph based on something called your sub graph manifest that you define so now thanks to the sub graph manifests and the graph you are able to query data on the blockchain and our problem is solved and not only that we can view the results of our queries in this neat ui right here let's have a deeper look at sub graphs and writing subgraph manifests in the next section as we mentioned a subgraph manifest also known as a sub graph description is something that the graph uses to index the data that you need on the blockchain we also mentioned that this is something that you define yourself for example you can define a subgraph manifest to query the smart contracts you are interested in for a sub graph the events in those contracts to pay attention to and how to map event data to data that the graph will store in its database smart contracts on the blockchain for those of you coming across the town for the first time in a nutshell are pieces of repeatable executable code they are stored on the blockchain and run when certain conditions are met so think if this happens then that happens you can even think of them as a sort of function if you wish it could be to do with releasing money to stakeholders sending notifications or anything really all in an encrypted way when conditions are met smart contracts can emit events in fact events are the way you can communicate with your smart contract in general so when something happens events are emitted and loads are written to the blockchain that then your front-end applications can pick up so we know what is going on once you have defined your subgraph manifest you can use the graph cli or command line interface to store the definitions on the blockchain and tell the indexer to start indexing data for that subgraph okay so think what you want to query define it then run the indexer and that's it once you are happy with what you have built and tested it out to see you are getting the correct data back you can publish your sub craft to the decentralized network this now means that by using your subgraph and your definition of it applications can query data served by what is called a query market comprised of a network of indexes or competing to provide the best indexing service at the best price this is opposed to how we query things today on web 2 which involves going through a centralized server that just offers one service at one price building apps or in this case decentralized apps also known as the apps is very different from what we are used to today when building on the blockchain as we touched on briefly already data is not stored in an organized way it is scattered all over the place this is done by design and in layman terms is the reason it is so secure as it is decentralized and not reliant on a centralized source on the traditional centralized web databases and apis query filter sort paginate group and join data before returning it back to your front-end applications usually thanks to some sort of http request this is possible as data is organized and indexed on the blockchain we are missing both these things the organization and the indexing layer this would mean that if someone wanted to build a decentralized app they would have to build a way to index data by building and operating proprietary indexing servers this would require a lot of time and energy and each company would probably be doing the same thing the demand to build a tool that everyone could use that would also standardize the way data was queried was that enter the graph the graph was essentially born out of this core market need today the graph is one of the most used blockchain protocols and has saved a number of companies and developers time and money by building a data querying tool that they no longer have to build themselves by now we know why the graph was invented as well as what we as developers need to do in order to query data on the blockchain but how do we actually do it we have now reached the practical part of this course so let's get up our terminals because i'm going to show you how first we are going to have to install the graph cli as this is the main thing we will need in order to create our first subgraph so let's jump right into it and get going okay so as mentioned before as a developer we can choose to either use a sub graph that was already developed by another developer or we can make our own so we're going to make our own in this tutorial we are going to build a sub graph for querying nft data from the foundation smart contract which you can see here we are going to write a query for fetching nfts as well as their owners and build relationships between them so that by the end when we essentially look all over the ethereum network our data will come back like this foundation also has a developer portal where they have information about their smart contract code as well as links to their own sub graphs if you want to check this out later so before we get going with this tutorial please make sure to have node.js installed on your machine here you can find ways to do this and for this tutorial we are going to be using nvm to manage node.js versions so if you don't have those installed please just go ahead and do that now gravitate towards this page right here and just download this onto your machine as well as nvm like so so all i'm going to do is just take this script right here and just put it in my terminal and run it okay so once we have these two things done let's carry on first off i'm just going to gravitate to the graph.com and on here i'm going to select the drop-down hosted services once here you have to essentially make an account i have signed up using my github account this has already been done so it's just going to take me straight to the platform and i'm going to choose to go onto my dashboard where you can see some sub graphs i have already made as testers as well as my own now to create our own just go ahead and click add sub graph right here we're going to have to name our subgraph something so because we are working from the foundation app i'm just gonna call this foundation app sub graph and give it a quick subtitle so this can of course be whatever you wish i'm just going to fill it out for best practice and then let's go ahead and leave these two blank and then we can fill out a description and we can give it a github url these two things are optional please feel free to fill them out if you wish and just go ahead and click create graph and great now you should be taken to this page right here with some more steps the first thing we need to do is initialize a new subgraph using the graph cli so all i'm going to do is view the cli commands here so all i'm going to do is view the commands right here to be able to install the graph cli now gravitate to a directory that you want to work in for me it's going to be called webstorm projects once again i'm using npm so i'm going to choose npm install and i'm going to install it globally g is for globally and i'm going to install graph protocol forward slash graph cli now i'm going to leave that to run once that has been installed we can essentially initialize a new sub graph with the init command okay so just go ahead and wait for that to finish running i might as well just zoom this right here so and once that is done let's just gravitate to here and i'm just going to copy the first command that i see so this command is to initialize a new sub graph so all i'm going to do is grab the command and replace the github name with my github name and replace this sub graph name with foundation app subcraft which is what i called my sub graph and once again i'm just going to hit enter on that now there will be some prompts for you so just go ahead and make sure to select the ethereum option and then on the sub graph name i'm just going to hit enter i'm happy with that i'm also happy with the directory that this wants to be created in i'm also happy with the directory it's going to create my subgraph in so i can view the code i'm currently once again in my webstorm projects directory on my computer and i'm just gonna hit enter on that now here please uh select mainnet of course you can choose whatever you wish in the future but just for this tutorial we're gonna be doing on the ethereum main net now when i ask for a contract address this essentially is the contact address taken from the foundation nft contract okay so i have taken this just take the same one exact same one that you are seeing here uh pause if you need to take longer on this section i'm just gonna go ahead and click enter for now and when it next asks you for a contract name just go ahead and write token now if we hit enter this command will essentially generate a basic sub graph based of the contract address passed into the argument so all the little numbers that the contract address it's going to generate a sub graph for us based on this okay so if we now go ahead and look in our directory we should see a directory called foundation subgraph that has been generated for us thanks to the command above all with the template code that we need to get started okay so there is a project for those of you using vs code you can just open it up using the command code dot if you have that installed or you're going to open up with whatever code edit you wish i'm going to choose webstorm so i'm just going to go ahead and find that now and open it up so there we go and as you will see in here there we go there are all our files that have been generated for us now i'm going to just go ahead and talk you through some of these as we mentioned in the explainer the subgraph manifest or in other words how we define our subgraph manifest is this subgraph.yaml file okay so we've talked that one through once again this is how we define our sub graph and next we also have the schema graphql file which is a graphql schema that defines what data is stored for your subgraph and how to query it via graphql so at the moment you just see a example entity that has an id account an owner and approved so while we are here actually i'm going to ask you to change the content of this file to essentially let us query the nfts in a way that we want so let's go ahead and do that so all i'm going to do is paste this piece of code that i have pre-written here now let's talk it through a little bit this is essentially graphql with graphql we can define entity types just as we have done here with the type token and type user the graph will essentially take this and generate top level fields for querying single instances and collections of that entity type now this is important each type that should be an entity is required to be annotated with an entity directive so what we are saying is that we want to index these two entities okay these two entities that we are indexing are going to be the token and the user this way we can index the tokens created by the users as well as the users themselves what is also interesting is this derived from right here by adding this derived from field we can now do reverse lookups so essentially what is happening is we are creating a virtual field on the entity that may be queried but cannot be set manually through the mappings api rather it is derived from the relationship defined on the other entity for such relationships it rarely makes sense to store both sides of the relationship and both indexing and query performance will be better when only one site is stored and the other is derived okay great so now that we have defined our schema essentially for the entities we can generate the entities locally with one simple command we're going to use the command graph cogent in order to spin up some code so let's just go ahead and do that and now if we look in the ts file some code would have been generated for us based on the two schema that we wrote if you haven't come across this command before this command essentially generates assembly script types from a combination of the subgraphs graphql schema and the contract abas included in the data sources by using this command we are essentially making everything type safe now let's move on to the subgraph jam of file we're going to have to update some things based on what we have just done so the first thing i'm going to ask you to do is just head over to the data sources mapping entities and just add the user and the token entities just like so next up let's move on to the data sources mapping event handlers and i'm gonna actually get rid of all these event handlers as we don't need them anymore that was just the template code and i'm just gonna add these two events right here along with these two handlers make sure to keep the indentation the exact same as this will cause issues for you if you don't and finally we need to update the configuration to add the start block and change the contract address to the main proxy contract address so once again i am just going to use this right here make sure to use the exact same as me now go ahead and open up the source mappings ts file where we're going to write the mappings that we defined in our sub graph sub graph event handlers so once again i have pre-done this for you and we're just going to get rid of all this code all of this template code and make it reflect what we want which is the token and the user schemas and here is the code so once again if you need this i will be putting this in the description below this is a great template for you to have now these mappings will essentially handle events for when a new token is created transferred or updated when these events fire the mappings will save the data into the subgraph and there we go we are all done now let's run a build so we're going to do this using the build command like so making sure that we are definitely in the project that we need to be and next we are going to have to go back here and run the following command of the deploy section of the tutorial so all i'm going to do is grab this command graph off and once again i'm just going to replace the necessary fields so this time i'm going to need the access token which is going to be unique to you and you can find it right here and i'm just going to run that command now we're just going to have to wait for that to do its thing and finally i'm just going to grab this command to deploy our graph making sure to once again replace the github name and the sub graph name so this is my github name please go ahead and use your own and whatever you called your sub graph just make sure that is the same and run the command and there we have it we have deployed our sub graph you can see it here and then if you go back to your dashboard you will see these subgraphs show up right there wonderful so once we have that we have deployed our subgraph it is now time to start querying the foundation app nft space to see if we can return back nfts as well as their owners ids and so on so let's do it so this should spin up a playground similar to probably ones you've seen before if you have used graphql before as you will see there are some schemas on the right you will see the other schemes that we defined so the token schema and the user schema that we can use now to write our queries a query has been generated for us also the example query so if i click this button right here it will run and you will see the uris for the content as well as the token ids and the token ipfs paths so wonderful this is looking good now let's try write our own query so i'm going to do this from scratch and i can also use the schema to help me do this so for example if i want to query tokens these are the things that i can get returned back to me so i'm just going to go ahead and start doing this now this is the syntax for querying things using graphql so i just want to return back tokens like so and the tokens that i want to return well i want to get back each token's id as you can see here as well as each tokens token id which is a different type of id and i also want to pick out the token ipfs path so as you can see we don't need to pick all the fields these are the only three that i want to return so i can literally pick and choose what i want returned i could have picked the owner i could have picked the creator i could have put the timestamp i could have put the name but i have just chosen these three and if we run that there we go another neat thing that can be done is that i can order the order in which i want these tokens to come back to me in so if i want to do this all i have to do is write the order by and order it by the id as well as add an order direction so making sure that is in between two parenthesis i'm going to choose the order direction to be descending and if i run this you will see the token id starts at 99999 and if i go ahead and put ascending so asc for ascending it will start at 1. so there we go let's just change that back to descending maybe for now i can also skip the first 100 so for example if i put skip 100 just like so and run this then the first 100 tokens will be skipped great now let's perhaps look at the users so once again i'm just going to use this syntax to return back the users and if we have a look at the schema we can see that we can choose the id the tokens and because we have derived this i can now use the token schema to get back tokens okay so hopefully you can see what we have done there with the derived from tag that you saw earlier i am essentially using the user token schema and i've now sort of made a relationship between the token schema and the user schema so now once again i can choose to order the tokens just like we did before and this time perhaps let's do it by the timestamp so i could just choose create it at timestamp option and wonderful please do have a go around our playing here before moving on to the next section in this part of the tutorial i'm going to be showing you how to build a front end using react quickly in order to display the data in our browsers so all the data that you see here i'm going to be showing you how to get it to here okay so let's get to it super simple let's do it if you do not know react or would like to use some sort of other framework for the front end please feel free but please do have a go at watching long anyway to get the general concepts the two packages that we are going to need for this part of the tutorial are oracle and graphql you will see them right here aqua allows us to rapidly use graphql in our applications without complex configuration or large api overhead and the graphql package will allow us to write queries in this format right here in our front end so to get going we're going to have to grab our query http path in fact if you would like to actually use any of the sub graphs you can go ahead and do so so i'm just going to show you how to do that before moving on so right on here just select the graph explorer and we can choose any of these really so perhaps let's go ahead and choose the art blocks one and then you will be able to view the query the url query that you will need so for example to use art blocks i would need to get this query url okay and that's really it so now that we know that let's go ahead and get our own query url from the subgraph we have made so i'm just going to gravitate back to my dashboard where we made the foundation app sub graph and i'm just going to go ahead and click on it and here we go here is my query url that is the url that i need to make the queries so just make a mental note of that now let's get to creating our frontend app so all i'm going to do is once again gravitate to my websome projects and in this directory i'm going to use the command npx create react app and then whatever i want to call my app so i'm just going to call it my graph studio in order to spin up i react out fast with configuration with files so everything will essentially be done for me if you want to read more about this command please go ahead and check it out here it is one that i use often my project so that we can get to the coding part quickly so once this has finished running at the moment it's just getting all the necessary files i need and all the configuration then we should get ready to start our project and there we go we are now done so now i just need to go into the project i made so i called it my graph project so i'm going to go into there and i now need to install the packages that i discussed so erco and graphql and i'm just going to wait for those to install and i'm just going to open up my project so once again i am using webstorm so i'm just going to open up webstorm again and there you will see foundation up subgraph that we made in the last section and i'm just going to open up the newest directory that we have made which is called my graph project and in here ta-da you will see all the files necessary to get going with create react app so i'm not going to talk you through these hopefully you know this command already and now to start the app i'm just going to use the command npm run start like so and that will spin up my app on localhost 3000 there we go this is what it should look like now if we look back in our projects and check out the package json you will see the oracle package and you will also see the graph your package along with their version so if anything is not working for you right now it could be because you're using a newer version if that is so just revert back to this version and run npmi so npm install again to reinstall all these packages okay let's move on now the first thing i'm going to do is just in the app.js file i'm going to import create client from arco so the package that we just installed and i'm also going to import use effect and use state from react okay so those are literally the only three things we need now even though we are not importing graphql here we still do need it so don't think that that was installed without a reason next i'm going to put my url in here so the query url i'm going to save it in this project let's save it as const url and let's just go ahead and grab it so i'm just going to get that here and just paste it as a string like so the next thing we need to do is use our query so i'm going to define this as query and i am going to put some backticks in here because we need the backsticks in order for this to work and i'm just going to get this query the example query that we have in our playground okay if i run it it should return this data so that's the data that i want returned so let's grab it and let's just paste it in our project like so so super simple there we go the next thing that i want to do is actually start a client so let's define our client and i'm going to use create client that we imported from the package aqua and then we're just going to pass through the url maybe let's call this something different i'm going to call this api url like so and the next thing that i'm going to do is just fetch the data so for this i'm going to write a function it's called fetch data and it's an async function and in here i'm going to use the client that i defined and i'm going to use query which is a method that comes with it and pass through the query and then use the method to promise on it so again this is just the query that we have defined and i'm going to await this and then catch the response once it resolves so once the promise resolves and i'm just going to console.log it out so we can see it in our browser now this function i'm going to put in a use effect so just go ahead and do that in order for this to work and let's check it out so now if i inspect the browser you should see a response with the object which has the data and the tokens and the users coming back to me okay so exactly the same as we saw on our playground we are now seeing this in our browser wonderful okay so this is pretty much it if you want to actually see in the browser as well i'm just going to quickly show you how to do that maybe let's just show the content uri and the token id so all i'm going to do is just save the tokens to state we can do this using use state or the use state react hook so let's go ahead and do that now const tokens and then set tokens and i'm going to start off with state being null so at the moment tokens is null and we're going to use set tokens in order to change the value of tokens to something else so i am now going to get the response data tokens because we just want the tokens really and i'm going to use set tokens in order to change the value of tokens to the tokens that are coming back to us with the response and once again we want to display this in the browser so i'm just going to get rid of all this template code and if tokens exist i want to map each token once again we need the index as well because we're going to have to add a key to the wrapping div so let's make a wrapping div and then just pass through the key of index and we're not using it so i'm just gonna put in a little underscore like so and now we are gonna have to i'm gonna make these uh a tags and if we click on them it's gonna take us to the token content uri it's gonna open up a new new tab and then perhaps let's also have the let's have the token id so ta-da now if we click on the token uri it will take us to all of this information here this is pretty neat and then we can also see our token id next to it wonderful so we're getting the data back we're displaying of course please be more inventive feel free to display your data however you wish this is essentially the steps into getting that data showing up on your front end if you want to use perhaps a different sub graph that might come back with different data you can use in different ways as well so the world is really your oyster with this okay so we've covered a lot so far now the final part is going to be all about managing your api keys and billing for your studio so we're just going to head over to the sub graph studio by navigating here now the first thing that you are going to need is to have a wallet set up i'm going to be using metamask for this tutorial so i'm just going to show you quickly how to get set up so i'm just going to show you quickly how to get set up with metamask just in case that you haven't before first off just gravitate towards metamaster io and simply install the chrome extension once that is done you should see a little fox show up here and if you click on it this browser should show up now let's go ahead and click get started as we are new i'm going to go ahead and create a new wallet and just go through these steps go ahead and create your password making sure that is unique to you finally after clicking next you're going to get some secret recovery phrases these are super important don't disclose them to anyone i'm going to be sharing them with you here as my wallet is not going to be active so here are my secret words and i'm just going gonna have to confirm my secret recovery phrases by dragging and dropping them in the correct order so once again do not save this with anyone keep it super safe as if anyone has this they will be able to access your wallet and there we go so that is it we are now set up i have a metamask wallet let's continue so now it's going to prompt me to connect with my wallet i am going to go ahead and do that and my wallet should pop up in the corner like so and there we go super simple just go ahead and click next through these and once we are done go ahead and click on the billing tab here you will see your billing balance as well as any costs that you have incurred in the billing period now to enable billing this is important you must have ether and grt in your metamask wallet so these two things are important in order to get set up this is what your dashboard should look like if they are it should give you the option to deposit grt to your billing balance billing is covered on the matic side chain in order to decrease transaction times as well as transaction costs in this step we are going to move grt to the matic network so i'm just going to put in the value of 25 and approve and confirm once the grt is successfully moved to the matic network we can now switch to thematic network to view more details on the matic network please visit docs.medic.network forward slash docs develop network details network as you can see in the url right here here you'll see the details that we need to import into our metamask wallet including the network name the chain id and the rpc address so now let's go ahead and add that to our metamask wallet so i'm just going to click on settings and gravitate towards networks and now we're going to manually add the network so once again i'm just going to add the network name the chain id and the rpc once the network is configured in your metamask wallet we can now move on to the next step we can now move our grt from our wallet balance into our billing balance so let's just go ahead and confirm this add grt and confirm and great you will see our billing balance has been updated after all these steps have been taken we are now ready to create our api key so all i'm going to do is gravitate towards the api key drop drop-down and i'm simply going to create an api key so we can obviously rename it here and let's go ahead once our new api key is created we can now scope it down to authorized subgraphs and authorized domains to scope an api key to a subgraph we first need the sub graph id subgrave ids can be found here so just go ahead and find the subgraph id you need so this one right here and paste it like so and there we go so the whole reason we did this is so that we are now enabled to use the pull together sub graph okay we are now authorized to use it thanks to this api key and there we have it we have now finished the tutorial all about querying on the blockchain so in conclusion what we have learnt is how to build sub graphs how to hook up a front end to use our sub graph and then how to manage our api keys and billing thanks so much for watching and i'll see you again soon