Large language models with Keras

Key Takeaways

hello everyone Hi so welcome to this last session of the day large language models with kasas I'm marttin I'm Gabriel and I'm s and we are from the K team and today we would like to uh take you on a journey through various workflows involving large language models and kasas kasas is a high level modeling framework it's basically what helps you assemble your neural network models out of neural network layers uh and uh with our team is also working on two additional libraries called Keras CV and kasas NLP which are two collections of state-of-the-art pre-trained computer vision and natural language processing models so we want to show you how to handle large language models with those we'll start with very easy workflows how to load a model how to implement a chatbot and then since kasas is a full-fledged modeling framework we will also turn up the complexity a little bit towards the end of the the the second half of the session and show you how to implement more demanding workflows in kasas let's start with Gabriel so so uh we're going to be implementing some chat experiences as Mar was saying with Gemma and to set the stage Gemma is Google's family of these top performing open language models these are language models that are able to take text in and text out so they're what you would use to create poetry write stories uh even talk like a pirate as we'll show later on our release uh is through this kisl P package our natural language processing version of Caris LP that has all these utilities that we'll explore throughout the session one important thing about kis 3 our newest release that came in just a few months ago is that it is fully multi backin so that means that everything we're going to be doing today is going to be available in both pytorch Jacks and tensorflow uh all with just a single line of code so here we see our Jam implementation for Caris uh NLP uh just to inside on this point that everything's implemented in very small uh modular layers uh all these are available through the Caris NLP Library all this code is open and a ailable we'll dive into it very soon uh throughout this session we're going to be looking at how to load in the 2 billion and 7 billion instruction tuned we'll look at what that means and the uh normal pre-trained model we're going to uh make some chat Bots uh make it talk like a pirate we're going to be diving into some pretty cutting edged uh research and seeing how to implement all of this inside of Caris so let's try it out let's start with Basics we import Caris NLP our NLP package for Caris and then we're going to just do from preset uh for our gem instruct 7B English so this is loading up our 7 billion parameter uh instruction tuned model what do I mean by instruction tuned uh we'll get into how it's uh trained but essentially what it does is it takes in full conversations it's able to take in a model prompt then a user talking then a model then a user and all this within his context window we're going to start by just saying hi hi Gemma so how does this prompting work when you want to add in the full conversation what we want to do is use this conversational markup so in this case you want to say start of turn user then ENT turn the start of turn model and you interlace each of these so first one then the other and then every time you call Gemma you're going to want to add in that whole context so that it is able to appropriately answer what it needs to Martin is now going to talk us through the tap bot demo all right let's switch to the demo please so all of the demos we'll be showing you today are available uh this is in collab you see the URL you can actually run this at home in this demo I'm running just for the technicalities I will be using the 7 billion parameter model which is pretty large uh I'm running it on a TPU device tensor Processing Unit which has eight cores so here are my eight cores that loaded uh and as Gabriel said Keras is a high level library that works on top of a lower level mathematical Library which today can be either Jax or tensorflow or pytorch so this line I'm selecting the Jax back end so I've run this beforehand up to this point where I'm loading the model this is the line that loads the model from preset and so on the model is loaded it takes a minute I didn't want you to wait but from there on I'm going to run this myself this class here is the Little Helper that implements the turn by turn tagging it's just 15 lines of code that's what helps you tag who is speaking and pass that context back to the model at every question you can copy paste it from here into your own applications and so let's start chatting what could we ask this model to do well why not chat about prime numbers so uh tell me in a few words how to compute all prime numbers up to a th and Jimma answers use the SE of Artin algorithm all right that's good it's very short so maybe Gemma could generate the code for us let's try and anoun that so now in Python and here is my python since this is a live demo I want to see if this generated code actually works let me copy paste it into a new cell and run it and yes it's actually generating prime numbers up to 1,000 so but how did Jimma do this we could try to understand this code line my line but actually I'm going I'm going to ask Jemma to explain so let's go thank you it works can you explain the code in plain English here are my explanations o that's a lot of explanations uh this is probably good but not very readable I was expecting my explanations to be in the code AS comment so that I know every line at every line what is going on uh that's not the formatting I wanted but let's ask Gemma to format this differently great now add those explanations as comments in the code line by line and here we are again live demo so I'm copy pasting this into a new cell and as a bonus you get syntax highlighting and you can actually read the explanations that Jimma has added uh it's it seems to be the actual implementation of the seeve of Austin algorithm I think it got that correct let's run it again it's still running so this is how you implement a chatbot us using the Kass API and the gimma model back to the slides please Gabriel just as a technical side Caris gives you the option to use multiple uh Precision modules so for example you can use float 32 float 16 or a mix of these two and also inate if you're looking for some quantization stuff why does this matter because if you're trying to get a speed up you can usually get a two three even 4X speed up just by enabling this one line of code to mix float 16 additionally when you're working with these larger llms when you're working with a 7 billion model that is just 17 GB uh on float 16 if you want to half that you're going to have to use inate uh for context Wikipedia's full text is about 22 gbt so float 16s or 32 can be pretty unwieldy back for you thanks Gabriel let's talk about fine-tuning Gemma Gemma is a general model so if you wanted to perform on a specific task or domain really well you need to find tuni for example if you want Gemma to perform really well on the translation t ask you can fine tune it on translation sentence purse or if you want Gemma to chat with you like a pirate you can find fine tune it on pirate speech Martin will show us how to do that later you can fine tune JMA 7B model in Kos Gemma 7B is a large model so let's talk about some techniques that helps us fine-tune the model the first technique is Laura or low rank adaptation instead of training all the model parameters Laura freezes the original model waste and only trains two lower rank matrices for example here instead of training this big red n byn Matrix Laura trains these smaller uh blue matrices it multiplies them and adds them to the original weight Matrix to create the updated Matrix because in Lura fine tuning number of uh trainable parameters is significantly smaller than the full model Bas the training runs faster and requires less resources we can easily Laura fine tune your model in kasas you can enable Laura for your model by calling the enable Laura API on the backbone of your causal language model after fine tuning you can save and load your Lura weight separately because Lura weights are smaller than the full model weight this makes them transferring and sharing and storing them much easier the second techniques that helps us fine-tune uh GMA 7B model is model parallelism K LP uh sorry caras has model parallel API and it abstracts the way all the complexities of parallel computation it allows you to specify how to split uh how to split all the layers and uh where to put the pieces Kos parallel API is Jack is backed by jacks uh and so it so you only need to specify how to split the largest layers and that's that are the bottleneck and the xcla the xcla compiler will handle the rest BBY so what the code in the previous slide was shown is was that if you have a large block of Base like this you can easily split it across multiple accelerators for example if you have a Transformer based model like Gemma you can split the heads across multiple accelerators now Martin will show us how to F tune the chat bot we just built so back to the demo please all right what can we fine-tune our model on uh this is a demo so we decided to do something a little bit fun and tune this model to talk like a pirate so we have a pirate speak data set and we'll be running our functioning on it and see if this model can uh pull off a good a good impersonation of a pirot so the first thing we are doing here sorry not that is uh as Sone said enabling Laura which means reducing the number of trainable parameters this cell is just my pirate speak data set uh it's already I loaded it uh I'm not actually going to run the fine tuning I did that beforehand it takes about 15 minutes I didn't want you to wait but this is the important line in the fine tuning I'm just calling the standard kasas model.fit on my data set um uh call this is what will launch the fine tuning process on my data set I'm also displaying the details of my model down here and there's one thing I want you to see it's here the number of trainable parameters sorry so you see there are 22 million trainable parameters in this model as I said we're using the 7B model which actually has 8.5 billion parameters but those are non-trainable so this is the Laura technique in action it has frozen most of the models parameters and is fine tuning just a little subset 22 million out of 8 billion all right so the fine-tuning uh went well I did it beforehand you see the loss going nicely down and now I'm going to load simply one of those fine-tuned checkpoints and see if we can continue chatting with this model all right this is loaded uh let's say hi AO there and while this is loading I actually wanted to show you something else at the very top very top here uh the code that is doing the partitioning of my model so as I said large model it will not fit on one accelerator I need to split it across multiple accelerators for that I'm using the model parallel API in kasas which is right here kasas distribution model parallel and what this part of the code is doing you see there are a bunch of regular Expressions targeting specific layers in my model so it's it's a bit of an art uh to partition your model the API allows you to uh specify layer by layer exactly how you want to split those layers on multiple accelerators now there are people who think about this and do research about how to split the models so I'm showing you the details in reality on the Gemma model there is a get default distribution API that gives you uh the settings that usually work and you don't have to figure out those details yourself if you want to it's it's fully available too so you have to do this uh this model parallel setup before you load your model so that when you load your model it ends up on multiple accelerators and you can run it thanks to the shared memory of all those accelerators all right back to chatting so I was asking I was just telling Jamal there and it answered what is it uh yeah it's right here hi it's nice to meet you what would you like to talk about that looks like a pirate voice so what can we talk about with a pirate why not prime numbers so yeah prime numbers for sure oh that be very interesting topic do you have a specific question about them well you see it's in character so what can we ask uh as a specific question well I want to see if this pirate can code so let's go yeah give me python Computing them primes up to 1,000 here be some python code and it's giving me the python code live demo again uh I want to make sure this is still working let's copy paste this code into a new cell run it and yeah those are my prime numbers from one uh to 1,000 so as you see the pirate can code and it stayed in character to the end you see there's some nice uh uh pirate speak here so this is fine-tuning uh in K back to you s thank you Mike can we go back to the slide thanks so far we showed you how easy it is to use Kos you can easily load Gemma model in Kos and you can build a chatbot with it you can find un need with Laura and you can distribute the model using the model parallel API in the second part of this session we want to talk about the flexibility of caros we want to show you that in addition to very routine tasks such as find tuning and building a chat bot you can use Kos to implement stateof thee art machine learning techniques here we focus on two recently published research papers control vectors and self extent we've chosen these approaches because they are very powerful techniques and they are also very useful the first one control vectors is similar to fine-tuning it's something to change the Model Behavior but the difference is that uh in fine tuning you need to retrain all the parameters so it comes at a high computer ational cost but in control vectors you only guides the model output into the desired behavior and it doesn't require any retraining self extend the second technique is a is a technique for extending the context window of an llm without any additional training so this self- extend method is helpful when you when your task requires understanding longer context of longer sequences of text let's dive deeper into these approaches the first one is control vectors so prompt engineering and uh fine-tuning are two commonly used approaches for changing Model Behavior control Vector is a new technique for doing that control vectors directly manipulates the internal activations of the model during inference the main idea is that if you if you feed a sentence through the model that has a specific tone and You observe the intermediate activations of the model and then you fit another sentence that has the opposite tone the intermediate activations of the model are in the opposite direction let's walk through an example together for example imagine you want to build an AI assistant chatbot for a car dealership and you want this AI assistant to have a very formal and sophisticated tone the first step is to create a set of opposite sentence pairs First We Take a set of neutral sentences a neutral sentence is something like the sun is shining and then we prefix these neutral sentences with positive and negative control a positive control is a sentence that drives the Model Behavior towards the desired direction for example in the car dealership uh example a positive control can be something like like you are a luxury assistant making a statement about the word and then we add our neutral sentence a negative control is something that drives the Model Behavior towards the opposite direction towards the undesired Direction a negative control here can be something like you are a teenager with Bad Manners and making a statements about the war when we create these opposite sentence pairs we feed them through the model and record the AC iations of the model when the model is completing these sentences for example with the Gemma model which is a Transformer based model we record the output of the the activations of the decoder block for example Vector A1 through A4 here when we record these activations we can see that these opposite sentence pairs create uh opposite directions in activations we can extract practice activations using statistical analysis techniques such as PCA these directions are control vectors and we have one vector for each layer now that we know what's the direction of the model when it's talking about when it's talking in a specific tone or voice for example when it's talking formally we can push the model towards that behavior by adding the control vectors related to the formal Voice or the sophisticated voice we can even control how Str the effect is by using a multiplier now Gabriel will show us how to implement control vectors in K thank youor so there are two things that we're going to be doing for this method we're going to first extract the control vectors so we're going to grab all those activations store them in a separate output in this case it's going to be an activations output and then later on we're going to re-inject them we're going to just legitimately just sum them uh how do we do this so our first part extracting the control vectors in our before picture this is just a plot of the Caris layers one of the most powerful parts of Caris is this functional model the functional API graph where you can grab each layer manipulate them however you want add new inputs add new outputs and just uh create a fully new Backbone in this case so for us we're going to go through each of the layers we're going in initially uh Gemma is just like all llms a series of layers first our input then our embedding then our decoder blocks eventually we get to our output and we're going to grab all the outputs so it's layer. output uh for Loop through and we're going to create this kind of activations uh stack which is just a stack of all the inner activations for our model uh later on we're going to reinjected these wrapped layers we'll show you how you to do this all this code is going to be available fully multi backin once again tensorflow jacks or pie torch uh we wrap our layers which is going to just eventually sum in whatever we learned through our PCA all that math that San was explaining and then we'll run it so let's go on over to the demo uh all this is available our bitly links will be uh on the slides of course we install klp before anything and then just to drive this point home everything we're doing is fully multi backend we are not running only Jax code or only tensorflow code we are doing Jax torch and tensorflow code and I guess numpy to change the backend to tensorflow for example we just contrl c contrl v you're done you are now running this intenser flow let's go back to Jax because that is one of our uh back ends we're going I already pre-and this because we're on a T4 this is a free tier GPU so you can run this at home for free with a free Google account uh if not uh feel free to sign up uh T4 gives you about 16 GB of GPU Ram which is more than enough to use a 2 billion parameter Gemma model so we pre-run this another thing to decide is that we're using these CLE usernames and keys just to be able to grab that model otherwise uh you'll have to add it in the secrets tab here are our utilities just to display everything nicely we're using the same chat state that Martin was using previously no differences there we're going load our instruction tune 2B 1.1 model 1.1 is it was just a model update a couple weeks ago instruction 2B meaning that it takes in all these conversational PRS it's able to do turn by turn by turn conversations so let's set the scene you are a used car salesman you're trying to get people interested in your cars come to your website and have them finally make a sale right so you're going to give your context to the model to Gemma you're going to say hey I have a Lexus and two Hondas and then if we chat with this let's pre-run a couple of these cells just so that we can see the output again this is completely free tier Google collab running on T4 no higham no anything if we ask it factual questions what is the first car in the listing it answers correctly it's an Alexus RX uh etc etc third question most affordable priciest car and then we get to something very interesting we are asking Gemma for its opinion again we're trying to sell these cars so what would Gemma recommend to your users or for whatever you're trying to find to uh for what they should buy so in this case which one of the three would you recommend for a family of four doing lots of school commutes and the occasional trip to the mountains and so Gemma answers the Honda CRV appears to be the most suitable option if I walk into a car dealership and the dealer says you the most suitable option for you is probably that car I am not buying a car right so how do we take Gemma and put it in a position where it's able to answer these questions with maybe a bit more passion right you can also of course follow up because we're adding in all the contexts of the turn by turn conversation so in this case why is this important it strikes a harmonious combination of features and price so let's go through the actual code for this this is everything that we explained in the theory this is how we generated the plots and caret is a oneliner k.s. plot Model A little bit of rewiring but that's just to clean up the the plot you're still going to be able to do it this is our llm I'm not going to go through all of this but this is exactly the same as what we had on the slides no differences we sack in each and every one of these activations we grab the activations from each Gemma decoder block and we output it to our stack and then that stack is where we're going to grab all the numbers we're going to run our math and we're going to be able to push our model where we want it so here is our prompts I pre-run this because it takes about four minutes on a T4 free tier if you are willing to go for an a100 is going to be maybe a minute and a half uh so I'm not going to run this too much but essentially these are all going to be the same prompts as what we saw in the slides it's going to be first uh a sentence saying pretend you're a luxury assistant which pushes the model into one of the directions we want and then your Falmouth pushing it in the other direction we're going to go down here uh this is how we extract it the important part is here all this is building up to running through our new backbone the thing that we've rewired creating this new stack and it here is where we can extract those activations so to to run something in Caris all you do is call the model that you've created in this case new backbone pass in your process prompts and you're done uh there we extract it all we do our PCA I'm not going to go into the math all this code is available it's extracting everything then here we get to our control Vector decoder layers this is what we did later on for step two to re-inject to add in what we wanted into our model um not going to run here and here we get to our actual demonstration so we've done all of this we have a vector that positions Us in either a formal place or an informal Place uh based on this we can push our model so let's start out by giving it informality so we set our formal voice to negative 1.9 let me pre-run all of these again this is all live they're taking about 5 seconds on a free tier T4 factual questions all correct just double check and then we're going to to get to our opinion which one of the three cars would you recommend so Gemma after having fine tuned for a family that does a lot of school commutes and the occasional trip to the mountains and the answer is uh informal it's the best car for a family is going to be the one that's reliable we have made our Gemma model informal more human something that we would potentially be able to talk to better if we ask it to elaborate we see it even more clearly uh for safety features you want a car that has good safety features because you don't want to get into it cuts off because we cut it for the demo but you can have it keep on generating text and all this is very very informal and with no changes just with what we've already generated let me pre-run over here here we see the other direction so we've just reversed the minus sign and converted to 1.9 we uh add in the context of all three cars we ask it as questions is correct second car third car and then while we wait for it to run we see which of the three would you recommend for family of four doing a lot of school commutes and the occasional trip to the mountains and now uh let me double check that we ran every cell yes this is through there we are while we wait for all this to run let us go back to the reinjection and we see this very important Line This is where we're doing that sum this is what it all built up to it's essentially just summing in our self do control Vector through our Caris layer model uh and then as this say finish to run okay back to our opinion based thing uh we once again get the appears to be the most suitable option but once we ask it to elaborate on this question is able to be a formal good salesman the Honda CRV strikes a harmonious balance between practicality and luxury offering a comp ing combination of features as you can imagine if you are able to send more GPU more examples more compute into this you can get it to do anything you want using the same exact prompts that we used before just with this control Vector system and we're going to go to the theory for the second one thank you Gabriel all right so uh this was one U pretty Advanced uh paper that we implemented in kasas we wanted to show you another one which is called self- extend this one solves a different problem the problem of context Gemma was initially trained with a context of 8,000 tokens which means that if you are running a technique for example like rag retrieval augmented generation where you have to put a lot of documents in the context for the model to answer uh in a knowledgeable way uh you might be limited and this is the problem with large language models uh their context seem to be fixed it's what they have been trained on not so this self- extend paper actually shows us how to extend it by hand by doing a bit of brain surgery in the model itself to understand what is how it's doing it we actually have to dive into the core uh algorithm that is running in a Transformer model and that is is called self attention so let's get into it and you will see if we understand what self attention does fixing this this uh this context problem is actually pretty easy so self attention is a mechanism that computes all the possible word combinations in your prompt and that is what helps the model reason about which words are in the prompt and how they interact so the way it is computed is uh by the time we get Computing self Attention our sentence is already transformed into vectors one vector per token here is one vector per word and since we we want to compute the interactions between word between words we actually have one set of vectors in red here that represents every token in the sentence and a second set of vectors that represents the same sentence with the same words but here in green and self attention is simply all the possible dot product between those words it's a matrix of all possible combinations but there is one additional ingredient it's what you see here in the equation as r o of n minus P it's called rotational positional encoding so I'm not going into the math not explaining why it's called rotational this is the part that encodes the relative positions of the words because of course it's not just because two words are in the sentence you you also need to know how far they are apart in so on if I have the words for example common and sense if they appear next to each other it's common sense and that has a specific meaning right so this positional embedding depends on N minus p n is the position of one word and P the position of another word so n minus p is the distance between words and this is the ingredient that is going to be problematic with respect to context so let's try to see what kinds of distances between words I can get in a in in a given prompt I have them here in this uh little green Matrix and let's see what happens if now I'm running my uh inference on a prompt size that is bigger than the context the model was was trained on so what is happening is those numbers in red suddenly word distances appear that the model was not trained on during training and this is going to be problematic it does it's not trained to handle such long word distances it is going to Output gibberish it with if it Encounters this how to fix it well the super clever solution that this paper found is just use smaller numbers no really that's it if you understand what is going on you can fix it easily you have to be just a little bit careful so let's get back to this Matrix we want those numbers in red to come back to the range that was seen during training um but you have to be careful at close range distant exact distances between words matter so we have to keep those in place we can't change those at close range if I say common sense that has a certain meaning if I say common comma in a sense different meaning still the same words just the distance uh is is different but at long range which is where my problematic long distances are I can just change those numbers and use use smaller ones a bit of interpolation so that the relative positions are good and that's it with this bit bit of surgery I'm suddenly fixing the context length of my model and I can extend it so now we want to see how to implement this in Keras on a pre-trained Model that you loaded from kasas NLP go ahead thank you Martin so we might recognize this code from a few minutes before this is the same patching system that we were using before for our uh context uh for our other demo what we're doing here is we're simply going through all of our layers we're overwriting our two methods in this case our rotational positioning embedding or rope for short and we're overwriting our computer tension with something called extend comput tension which we'll see throughout in the demo uh one just inside on this the reason that we have to get rid of the Rope is because when you actually do this whole Gemma first you do our positional embedding then we do attention to make all this easier let's get rid of our rope put it all in the attention and let's go to the demo see how this is done so we start by installing the latest Caris NLP I've already pre-run this we're going to be doing all this on an a100 the reason we're not doing this on a free T4 is that we're trying to really push the cap abilities of Gemma Gemma was not trained for any of this Gemma was trained for going up to about 8,000 a bit more than 8,000 tokens and we're trying to push that we're trying to get to 10,000 potentially you could get to 20 30 whatever you have the compute the gpus for once again we're going to be using Jax but fully multi backin want to use tensorflow tensorflow done let's go back to Jack this is all pre-run we're using our kagle username and key we're installing the same we're up grabbing the same Gemma so this is the Gemma one instruction 2B English it loads up collab is very helpful with this in that it has it all cached you're not downloading anything it would run in about 20 seconds so we're a car dealership we have about 150 models of cars 200 300 some very large number that's very difficult to work with and we're trying to use our open source models our Gemma powerful models in this case we want our users to be able to request whatever piece of information they want in this case the 2024 Google ioar uh discount code which is 1234123 this is not a real car just for disclaimer purposes this is our Hast stack we have a Hast stack of 45 cars and we're trying to hide our needle in this case that discount code so let's run through this let me so here so we're going to be using the same conversational system we're going to be using the start turn user start turn model and let's start out with just 45 cars as contact so that's about 2,700 tokens which is still a lot I mean 45 cars are a decent amount of cars but it's well within the range of gemma's capabilities so in this case we ask it for the discount code and it answers 1 2 3 4 1 two3 as a cool technical note here one of the things that we do is we do our user turn first so what is the discount code we end our turn and then we start the turn of the model with the discount code is so that we're already pre-loading it with what we potentially would want to see uh so let's try and do something that is not within gemma's capabilities let's quadruple the amount of cars so we're at about 200 we're running it with 10,000 tokens a bit north Jemma was not trained for this it is going to fail uh and that is fine because we have self extent to come to the rescue for this so we just get an enter and a du that is fine this is not expected to work now we're going to get into our self method uh we're not going to go too much into it we've gone through the theory that is just implementing this all this will be available later on this is the code from the Sid we're just overwriting our rot method ning it out and adding our new extend computer tension which is what we just defined uh we're going to recompile this all double check that it does work so this is all running live once again on a100 it's going to answer to a hi there prompt and we're going to see in this case the full uh output so as soon as this runs it's having to recompile it's having to do everything Jax is a compiled backend meaning that is going to have these static shapes and there we are here we see the actual full output so start of turn it answers to hide there start of turn model hello it's great to hear from you what would you like to talk about it's working perfectly fine now let's test it with a self- extend algorithm so let's begin with something that is uh within old Jam's capabilities so in this case we're going using about 8,000 tokens that's the tripled uh Hast stack meaning we're using about 150 cars still a lot and let's let it load and remember this is all running on that a100 so we're able to do these very long contacts in full control for us so it answers correctly the discount code is 1234123 awesome let's push it further let's see if gem is able to do something that it was not trained to do so we're going to quadruple this see how we're at 10 , 639 tokens that's about 2,000 3,000 more you can push this much further uh for this purposes we'll keep it at 10,000 it ran in about 5 seconds and it's able to answer correctly the discount code is 1234123 now Martin's going to send us off thank you Gabriel can we go back to the slides please all right so this is what we wanted to show you today we wanted to show you how to do simple things with kasas by loading a model from kasas NLP and implementing the basic workflows like a chatbot like fine-tuning is it as well but we also wanted to show you I mean all of those things in kasas are basically on liners but we also wanted to show you that kasas is a full-fledged modeling framework the entire architecture of the Gemma model is implemented in kasas using kasas layers and when you load a model from K NLP the entire uh graph of layers is available to you if you need it to change the behavior of the model to implement the latest and greatest research papers on your model kasas offers you that flexibility in addition to being super easy to use for the easy tasks that's it we hope you enjoyed this session if you need to go back to the three demos we showed you today you have the the the the links here and I just have to thank you for your attention [Applause]