Introduction of TensorRT-LLM Engineering Baseline Work making TensorRT-LLM developer more efficient

Key Takeaways

Hi everyone. Thank you for joining us for today's Tensor RTLM live stream. Good morning or good evening depending on where you're tuning in from. Today we have two wonderful presenters, Freddy and Anway, who are going to talk about making Tensor RLM developers more efficient with baseline work. Before I hand it off to them, just a couple of resources to share with you all. uh the Tensor TLM GitHub repo. If you'd like to connect with us on Discord, you can join our Discord server. And then lastly, we'll be answering questions throughout the live stream. So, please feel free to drop your comments and questions. Um but if you have any questions to ask us afterwards, you can develop us or you can reach us at the developer community email address as well. Yeah. Okay. So, so, uh, can I begin? Yep. Go ahead. Yeah. Yeah. Thank you. Yeah. Today, um, I'm happy to have a a presentation about the engineering baseline work of Tensor RM. And, uh, here is today's agenda. There will be five sections. The the first section is about the O part. The second section will introduce the CI infrastructure of Tensor RM. In the third section, we will show how to optimize the CI time. And in the fourth section, we will introduce the accuracy test suit of Tensor RM. The last section is about the future work. Uh first is the over wheel and the uh tensor RM is open source library that accelerates and optimizes the inference performance of large language models. It accelerates the the RM performance on Nvidia GPUs and the Tensor RM source code has been available on the GitHub since the October 2023 and uh in this year the development is fully transitioned to the GitHub in March. the the fully transition means that the first the GitHub history is now visible and all the GitHub pull request are checked directly into the GitHub assisted by the Nvida GPA engineers and it it also means that uh the we we provide the full CI/CD support across uh different GPU architectures and the the the the fully Open source uh brings many uh CI uh challenges to our CR infrastructure. The first challenge is is the high velocity development at scale and uh after migrating to the GitHub we have the fast paces past development cycle uh which create a massive CI workload. In each week there are more than 200 active pull requests uh which generates over 700 CI jobs. The second challenge is uh is that we heavily rely on the GPU resources. There there are many u many GPUs in our CI cluster. For example, we have Atom S30, L40S, H100 and H200. As you know the GPUs are are very expensive and limited in availability which leads to complex scheduling and the Q management problem. The third challenge is uh is the combination explosion of test configurations. The AR inference system is very complex with many automation features. uh for example we have different quantization algorithms we have different parallel strategies and we have different batching strategies. So uh all these uh it has many many uh automation features and uh and and and thus uh we will get uh it's potentially uh growing test matrix which is the the the number of models multiply the number of hardware multiplies the number of bashion features and the uh the the first challenge is is about how how to validate the accuracy in your non did mean detministic words the the uh yeah it means that the first the the GPU kernels and the runtime heristics can introduce slight nondeterministic variations in the output the second is is about the performance trade-offs uh since there are many opinions intentionally trade the minor accuracy for significant performance gains. So how to establish a automatic and a reli reliable threshold to validate this trade-offs is a complex task. Yeah. So so is any questions now? Okay. If no question I I'll go to the second section. Yeah, the second section I I introduce the current CR infrastructure of Tensor RM. Uh first is about the uh testing tools. Uh there are two two kind of test in tens. The first is the passent test. The second is the CPP test. And in in the passent test there are also two kind of test cases. The first is the unit test which uh looks at the test unit test directory. The second is the integration test. The integration test means the model level test which is usually our uh end to end test. Uh for our uh evaluation task it will take more time. For example, it it might take five minutes. Yeah. All the in integration tests located at the test integration defense directory and uh both of them can use the p test to that uh to to launch the test. Uh the second is the CPP test. The CV test they are uh Google test based implementation and wrapped as our aggregated pi test and uh after we have the uh test cases we will uh organize them uh by different GPU types. So, so uh we we will have our uh test DB in the inside the test DB there will be uh many YAML files. Each ammo file corresponds to a GPU type. For example, uh since we have aton GPUs for the for testing we will have a we have a YAML file uh to organize all the test cases that will be run uh on the ATON GPU. And we will have R4S YAML file uh H100 Y file etc. Yeah. And uh it requires the developers to update the YAML files when new new test cases are added. And here are two links uh uh to to show how to run test and and how how to add a new test to the test DB. And uh and then it's it's about the the CI infra our all the GPUs are are in the blossom cluster. It's it's our CI cluster and we use the Jenkins to launch CI pipelines for the main branch, the release branch and the feature branch. There are different uh pipelines. Uh the first is the uh R0 premerge pipeline. The premerge pipeline means that when a developer submit a pull request to the GitHub, it has to uh pass the premerge pipelines and and then it can be merged to the GitHub man branch. And uh the second pipeline is the postmerge pipeline. The post merge pipeline is not peromit wrong. uh uh usually uh it it's about at at least up eight hours between two runs. The post merge pipeline uh has more test cases compared to the premerge pipeline. For example, it it has more H100 and also the DJX H100 multiGPU test cases. And the uh third pipeline is the postmer docker pipeline. It it will uh push the tilt dev and the release stock image to the remote repo. It it it's it's also not per commit wrong. It it it has the at least eight hours to uh uh between between two runs and and also we uh and and we also provide uh some some some B to help developers to trigger a new pipeline. uh the the CI/CD bot provide uh capabilities including the wrong command the the run uh subcommand the skip the reuse pipeline etc. So so you can you can see the uh full command details by the command app. Yeah I will introduce uh some of them. The first is the uh reuse. It it means that the if if if you want if you don't want to trigger a new pipeline, you can reuse a previous successful um pipelines. If if no pipeline ID is is indicated, it it will use the last pipeline. The the second subcomand is the skip test. Some sometimes you if if you want to skip all the test badges, you can you you can use it. It will only run the build and the package stages. And uh the second the third subcomand is the stage list because some sometimes you only want to run uh some uh specified test stages. For example, I I just want to run the aton and to end the first stage. Yeah. And the the first subcomand is the GPU type. For example, if you want want to only run the test stages on the specific GPU types. It says say I I I only want to run the test cases in the H100, then I will use this subcomand. Yeah. And uh the next is the multiGPU test subcomand. And if if if you want only run the multiGP tests and the third uh and the last one is the postmerge subcomand it means that I I want to trigger an a broad a broader uh test cases. I I want to run all the test cases in the postmerge. Yeah, that's our uh current CR infer. Yeah. Is is there any uh questions now? Okay. So if you if there are no question I will go to the next section. Yeah. Then next section is is about how to uh optimize the CR time. uh the since since we have uh so many test cases and we have uh so how how to reduce the C time means uh will will uh bring uh will improve the the developer uh the development of efficiency and the the first the first trick is to re reorganize the test cases by hardware. It it the motivation of it is to balance the text workload among different hardware. And in the left table it list uh our current uh CI uh GPUs in our CI cost. We have the ATM the S30 FDs etc. And each of them they have uh different numbers. For example, the we we have 100 150 item but but we only have uh seven 17 B200 GPUs. So so we need to carefully balance the test test workloads among different hardware and uh uh and and in the previous slide I I I show that all the test cases are organized by different GPU types. So, so we have uh different YAML file for different GPUs and then we we will put some important test cases uh to the to the H100 and B200. And for the test cases uh that need uh less GPU resources, we will put them to item or 30 GPUs. And uh and and the each each GPU uh will have uh one one or more u test pipelines. And in the in the in the right picture, you can see that we we we we launch different pipelines. for example, the the DJX H100 multiGP pipelines, the ATM the ATM tens pipelines and and by uh by re organizing the test cases we can make sure the the the different kind of GPU can be fully utilized. And the the second uh the the the second trick is is to reduce and refine the test cases. And as you know the ARM the the large language model is a fast uh fast uh improvement um uh uh area. So in every months there will be new models come out. Yeah. So so uh we we will remove the test cases of the out of that models. We we we split the models in into two groups. One group is a core group. The other group is a contrib group but we'll move the out of that models to the contrip group and the deprecated the test cases of uh contract models. The second is that uh we found that there there are some outstanding test cases that takes too long time. The too long time means that it more than two minutes. uh uh for example there there is one test the testp.py pi this test cases takes more than more than u one hour to finish. So, so we so so we split them uh split it into several small test cases and then we also uh replace the big model with with a smaller one in our test cases. For example, we we we don't need to use the llama uh 70B in the test cases. Instead instead we we're using the llama 8B in our test cases. So, so it it it will uh take uh less time and uh and then we also remove the unreasonable test cases. For example, uh there is a test cases uh which use a model more than 100 billion. So it's it will take too long time. So, so we we just remove it and and we also u move more test cases in from the premerge to the postmerge since the uh postmerge uh it's it's not a per commit wrong. Yeah. So so it's it has more more buffer to to uh contain more test cases. And uh the third trick is is to uh maintain some uh conditional uh triggering rules. Yeah, by by default all all the all the single GPU test cases will be triggered in in every in every uh commit and and uh and we we have some uh text to group several test cases. For example, in the uh red picture, it it shows that we we we tag uh the the following test cases with our tag the the py it means that all the test cases all the test cases are in the py back end. Yeah. So we have uh and and then if if if uh we we we have some uh some rules to conditionally trigger some groups of the test cases. Uh for example, we have the multiGPU rule the the it means that if if if your GitHub PR uh changes changes some file in the multiGP file list and then it will uh trigger the multiGP row it will launch the multiGP test. Yeah. So, so uh if you are uh if your change does not does not uh trigger the multiple rule means that you can you can skip the multi test. Yeah. And also we have different rules. For example, the pyouch only rule it means that if your change does not change any pych back end related code uh sorry if if you are change only if if you change only include the the py back end uh code then it will trigger the page only uh tests. Yeah. And we also also have another rule for the uh DC visor model. Yeah. And by by all uh using all these tricks we can uh uh reduce our CI time. Uh yeah. So yeah. So is is is there any any questions now? No, I I think no questions so far. Okay. Okay. So, so if if no question I I I will uh uh hand to only for the accuracy test suit section. Yeah. Okay. Thanks Freddy and hello everyone. I'm happy to introduce the accuracy test suit of tens. Um and uh this accuracy test suit is uh a part of the engineering baseline work and it was introduced about just before we migrated to the GitHub development and uh this suit addressed the some of the 10 points of our daily development life. So and before I introduce the details, I would like to uh introduce the background. Why do we need this test suit? And first of all, why do we need the accuracy test? And if you if you work in the ALM inference, uh you you would know accuracy is a basic and crucial requirement. And uh for in a uh LM inference framework development for a model uh initially implemented with good accuracy future code changes may easily cause accuracy regression. For example, the kernel code, the modeling code, runtime code, all of them could cause accuracy regression. And so we need accuracy test to run in the CI workflows and or QA cycles. Uh they can detect and reject bad code changes. This protect our models from accuracy regression. So what is an ideal accuracy test? Uh the requirements include uh three parts. The first is the test should be effective and reliable. Um by effective and reliable I mean a passing test run should indicate no accuracy regression and the failing test run should indicate accuracy regression. Um the second uh part uh point is that the should be efficient and low cost as Freddy uh mentioned that our uh or CI is very uh resource demanding. So we should make the test affordable on our CI/CD workflows. And the last point is that the test should be easy to run and debug locally. If we found find some accuracy backs, we want to reproduce it uh easily and so we can uh uh is start to debug. Okay, let's move to next slide. Yes. Um so, uh how to set up an actress test? This process could be very intuitive. Um so let's go through the steps. The first is to decide the data set and task. Um if the data set is too large, you may optionally decide a sample volume of a subset. So then we evaluated the model on the subset and obtain a a reference accuracy. Then we can set up a threshold which should be slightly lower than the reference accuracy. Um then we can start implementing the testing code and the testing code is basically uh running the uh evaluation and compare the resulting accuracy to the uh threshold. If the evaluated accuracy is higher than the threshold then uh we let the test pass. If the evaluated accuracy is lower than the threshold uh the test should fail. uh everything sound should sound very straightforward and uh once implemented the test can be run in the CI/CD workflows and or QA cycles. Um but there there is a one seemingly trivial question uh how do we decide the sample volume and the threshold? Um previous to the accuracy test suit uh the sample volume and the threshold are decided in an ad hoc way which turned out to be very problematic. For example, for the threshold, it can be set too close to the reference accuracy, which could cause intermittent failures due to normal accuracy fluctuations. And the threshold could also be too far from the reference accuracy. And so that the um test always pass even if some accuracy regression happens. Yes. Uh this slide shows some uh I would call bad practices for the for accuracy test. Uh for example, the the ad hoc sample volume and the threshold. uh in the uh left code snape show in the slide we can see uh we start an ARM in instance and we choose to uh run the inference on 20 samples and we get the um inference results and we evaluate the results against the references and we have the accuracy evaluated accuracy and We uh last we make a assertion that the accuracy should be larger than uh higher than uh 60. So we can see the sample volume 20 and the threshold 60 are chosen head hook and uh in the right part right code snip we have some some I would call variance of the uh some bad practices for example the literal checking with uh the similarity checking and the keyword checking uh this this practice are uh are even worse because they are effectively using sample volume of one and as I have just mentioned uh this ad hoc sample volume and threshold uh causes intermittent failures because some reasonable uh accuracy fluctuations. Um I believe many of our tensor engineers have had fixed some accuracy threshold for the intermittent failures. By fixed I mean we just lower the accuracy threshold to let the test pass and apparently this is not a good practice. Yeah. And uh uh during the past engineering we have some learnings and uh during the uh the fight fight against the intermittent failures. The first learning is that a model's accuracy can slightly slightly vary because it reasonably executes on different kernels. For example, uh the um instance can run on different batch sizes and we may use different fusion patterns or we upgrade kernel implementations and we run the test on different hardware. they can uh all these factors may introduce some accuracy uh accuracy variance and uh so uh but the this doesn't mean the model implementation or or kernel implementation has some issues. So we can conclude that some slight accuracy drop doesn't necessarily mean accuracy regression. Um a second learning is that if we increase the sample volume it can reduce the evaluated accuracy variance. And this may may also sound very straightforward because uh the evaluated accuracy is typically averaged over sample sample scores. And uh according to the central limit theorem the average the sample scores variance is inversely proportional to the sample volume. Um let's revisit the question the simul trivial question. How do we decide the sample volume and the threshold? So you may find it becomes uh a bit tricky. Uh uh we we would like a higher sample volume so it can reduce the evaluated accuracy variance but we also would like uh lower sample volume so it reduces the test cost cost and uh we would like to set up higher uh accuracy threshold so it makes the test more strict uh any uh accuracy regression can be easily detected. And last, we also hope the threshold to be lower so that the test would be more robust and so the reasonable accuracy variance can be ignored. From a statistics view, uh we are actually balancing some uh some objectives. Uh the first is to minimize the sample volume. Uh is it is to minimize the uh test cost. The second is to minimize the minimum detectable effect SATA. The minimum detectable effect means the minimum accuracy difference which should be regarded as the regression. The third is to minimize the force positive rate alpha. The false positive uh rate alpha means the probability that the test fails when accuracy does not regress. And the last one is to minimize the force negative rate beta which is the probability that the test passes when accuracy actually regresses. So it becomes a bit complicated. Uh so actually uh the objective objectives are conflicting against each other. For example, uh given the sample volume n and uh uh given the first force positive rate and the force negative rate alpha and beta, uh increasing the sample volume n allows a lower uh minimum detect theta. And uh if we have fixed uh fixed sample volume n and minimum detect set theta, the th threshold setting actually involves a tradeoff between alpha and beta. And uh uh fortunately hypothesis testing provides a very uh good solution to this uh balancing problem. To introduce the hypothesis testing uh we need to reformulate the accuracy evaluation as a hypothesis testing problem. Um given the data set and the model. the evaluated scores can be viewed as a population um or or I would call it a a distribution. So when we set up the test set, we we effectively evaluate the model on n samples which is effectively drawing n scores let's say uh x1 x2 to xn from the population and the reference accuracy is sample average uh xbar. When we test if the there is a accuracy regression, we once again evaluated the model on n samples which is effectively drawing uh n scores uh x prime one x prime 2 to x prime n from the population again and the evaluated accuracy is the sample average x prime bar. The so uh so here we have two hypothesis. The first is non hypothesis which is the x prime one to x prime n are draw from a distribution with a mean equal to or higher than x1 to xn. And the alternative hypothesis is that X1 X prime one to X prime N are draw from a distribution with a mean lower than X1 to Xn. Yes. And uh once we formulated uh reformulated the accuracy test uh evaluation to the hypothesis testing framework we can give the practical steps to set up accuracy uh test. The first three steps are uh choosing the three parameters alpha, beta and theta. And uh what uh I would mention that we we usually choose uh you just use the default values uh we use uh the uh first positive rate alpha to be uh 0.05 and the negative rate beta to be 0.2. two. And for a typical data set which has a uh score ranging between zero and uh 100, we would choose uh minimum dete or three percentages. And then once we we decided the three parameters we can go to the fourth and the fifth steps uh which is uh calculating the minimum uh minimum required sample volume n and the threshold. We list the formulas on the slide. And if you are interested in the statistical derivations, I mean how how does this formulas come, you can visit this uh this link in our GitHub repo. And uh please note that uh a a very crucial difference is that uh now the sample volume n and the threshold gamma are computed instead of being chosen at all. Yes. Um so you may have noticed that uh originally we needed two parameters the sample n and the threshold gamma. Now we need three parameters alpha beta and sa. So did this uh does this introduce even more complexity? The answer is actually no. uh because as I just mentioned that we have very decent default values for alpha, beta and theta and these default values are in most of the cases they are universally applicable to different data sets and models. However, the sample n and the sold gamma are data specific and model specific respectively. Which means that if you decide to uh determine the sample value n and the threshold gamma, you need to decide for every data set and every model. Um here is some visualization results for for for the uh for the m storage. Uh the first is we visualizing the minimum detect seta by sample volume and and force positive rate alpha. Uh in this slide we have the a fixed uh negative rate beta to be uh 0.2 too. And from the results you can see that if we have fixed the false for positive rate alpha increase the sample volume allows a lower uh minimum type effect and if given uh a fixed sample volume and a lower uh alpha causes increased. So there there are indeed some trade-offs and in the uh on the slide the left part is the MMU results and the right part is the CN DML results and we can see uh if we decide to uh mix the uh minimum detect SATA to be lower than three we should choose sample volume and to be 4K for the MMU uh but the sample volume could be less to uh 512 for CML. So why why is the ZML requires a lower sample holding because uh this is because that DML's accuracy metric is root score which has a lower valance. Yes. Um another uh another key point is that in the statistical derivations uh we have a key assumption which which I have just also mentioned uh just just before uh that uh the here we have a t statistic and uh this statistic should uh from should be from a distribution that converges to the standard normal distribution. So the numerator which is x prime bar minus x prime uh the numerator should also follows the standard uh normal distribution uh follows the normal distribution. So how to verify this point? uh we we performed a cross SM uh experiment. The experiment is basically uh that we set up the accuracy reference on H100 and evaluated this uh accuracies on H uh A100 and A48. Uh these three uh GPUs are most popular GPUs on our CI CI infra and um as as you know that if if if we can assume that the code uh the kernel code the modern code and the runtime code are all all right. Uh so uh the they they should also uh they uh the accuracies on different GPUs should have uh uh all right uh accuracies but uh since uh different GPUs have different kernels. So uh the accuracies can uh have some variance and these variance are reasonable because it it is just uh introduced by different cuts and so the question becomes that does the reasonable accuracy difference follow a normal distribution and we uh show brought the empirical distribution on the slide. You can see that the uh accuracy the cross SM accuracy difference is following basically following a normal distribution. Uh so so we can see that the core assumption of our methodology is largely correct and uh we also noticed that some there are some outliers in the distribution and uh some of them turned out to be some accuracy issues. Um I would also like to share share the effectiveness of the hypothesis testing methodology. I mean how does it improve our daily development life? Um according to the methodology uh it helps us to determine a sufficient sample volume n and a proper threshold gamma which com which are combined and ensures that we have low force positive rate alpha and low force for false negative rate beta. And the low false positive rate alpha uh have a implications that uh when we migrated or most accurate test to the accuracy test suit uh very few uh intermittent accuracy failures are reported against the new test suit. And uh for for the from the low uh for the negative rate beta uh it helps us to identify some subtle accuracy issues which may not be identified in a uh simple way. Um for example some some accuracy issues uh include uh accuracy corruption for a few uh samples like tens of samples among thousands of samples. Without the framework we we we can just observe the accuracy uh just drop slightly and we cannot uh assert whether there is accuracy regression. And another example that is that some accuracy issues uh happens on specific SMG TPUs. Uh yes. Okay. Uh this is the uh the part for the hypothesis testing and uh this is also the most most core part of the accuracy test suit and we have also uh we have some some additional improvement uh for example the test efficiency improvement and the core idea is to accelerate the accuracy test with tender RTM's own optimizations. Here is a great example that uh in one period we thought that uh the MMU evaluation is not affordable on our CI/CD workflows and uh because uh in that period uh most of our accuracy test eval are using were using single prompt inference that is inference uh one by uh prompts one by And uh so we optimize this in uh several steps. First we introduced the batch inference. We increased the uh uh batch batch size to be 32 and it it improved a lot. And then we introduced the the incin batch inference which is uh you know the inflight infrared field batching. And this means that we upgraded the API from model runner to air map API. And the last is to intro uh reducing the reduce the sample volume based on the hypothesis testing framework for uh for example the MMU. uh the four data set is for uh 14K and we is a hypothesis testing framework tells us that we can use a subset of 4K samples. You can see from the last column of the table we evaluated the llama 3.18 billion on MMU. Uh the end to end evaluation time is 14 seconds on H100. Uh and uh for a relatively low-end GPU A30 uh so the end to end evaluation time is just just about one minute. So it becomes affordable on our CI/CD. Okay. The last part of improvement is that uh we want to make the test suit uh is uh easy to debug and and run locally. uh in uh previous to the accuracy test suit, most of our accuracy test uh start a sub process that cause an an evaluation script and this caused the complicated tracing and loing and some additional overheads. And the the most problematic part is that the scripts are task specific and the script arguments are not well designed. If you switch back to the tender RTM version 0.19, you may find uh you may find in the under the examples folder there are summariz.py and GPQA.py Pi and all the evation scripts and every uh scripts have a long argument list and some arguments are turn out to be a bit confusing. Now in our accuracy test suit uh we run the um API in the native page process which simpl simplifies the debugging workflow because we can easily insert any PDB break points and uh we choose you we choose to use the offline uh API which is AM API instead of the online API because it provides clearer uh error messages and PC for debug and uh we we also uh the new accuracy test also avoids repeated model loading for multitask evaluation. You can see from the right code snips that we start a M instance and evaluated it um two task tasks the MMU and the JSMK and we have a bonus. Uh the bonus is that we unifies uh we unifies the different evaluate uh script to a single entry point tium yar and the tmium yar also unifies the arguments and align arguments to tatium ser you can see from the right code snips that there are two commands very simple uh they evaluated the llama 3.18 billion on MMU and JSM case separated. Okay. Okay. Then we can revisit the requirements. Uh the first is the test should be effective and reliable and we achieve this by hypothesis testing methodology. The second is that the test should be efficient and low cost. And we achieved this by async batch inference and reduce samples. and the reduced samples is also from the hypothesis hypothesis testing storage. The last is that the test should be easy to run and debug locally. Uh and we achieved this by using offline API. Okay. Uh that's all from the accuracy test suit part. Uh so do we have any questions? The Hulk stomping is okay. Yeah. Yeah. Um we we uh we have considered that uh to use some adaptive uh sample sizing for uh actually hypothesis uh our hypothesising framework is using uh is uh statistically equivalent to some the competence intervals and I'm not uh 100% sure about adaptive some sample size. We have considered that we should use different sample sizes for uh different models with uh you know uh with different priorities. We use more samples for high priority models and less samples for uh uh less important models. But it turned out that it uh would be easier if we use a uh unified sample volume for all models and it makes our test code more simple and straightforward and easy to maintain. Okay, thanks. Okay. Uh so I'm handing over to Freddy uh to introduce the future work. Yeah. Yeah. Thanks Emily. Yeah. So the last section is about the future work. Yeah. Uh the uh the first is to enhance our our fail fast mechanisms for the multiGPU failures. uh currently we rely on a standard timeout period for the multi test failures which is uh waste a lot of multi GPU resources. So uh we want to accelerate the federal detection and the the second item is uh is to reduce the flag tests. Uh sometimes we met the network issues because of uh sometimes we we download uh hug infested set then it's it's time out and sometimes the the the test cases just come out without any clear reason in the log. Yeah. So uh we want to reduce the flag test and the third item is is is to enhance the performance tests. We want to uh continuously monitor the performance of the key models to automatically detect and prevent any uh regression over time. Yeah. So, so, so I think uh that's a fish rock uh section. Yeah. So, is there any uh questions? No. Looks like we have no more questions. Um, at the very beginning of the presentation, I shared ways that you guys can connect with us either on Discord or reaching out to us via email. So, if you have any further questions after the fact, please reach us there. Otherwise, thank you so much to Freddy and Onway for being excellent presenters today and thank you everyone for joining us.