ARC Challenge Live Coding

Key Takeaways

um hello hello i hope someone can hear me um i just need to quickly access the chat so i can even see you because i forgot but if you're here welcome it's this is nothing special we're just thank you we're just going to uh quickly jump into the ark game hi thanks for letting me know so the arc challenge is this game i don't know if you've seen that uh so you're given kind of these boards and there's always an input and an output and so the left one is always the input the right side is always the output by the way this is um by by alexa borski a nice website the arc game you can find it on github and a link to it so the left side is the input the right side is the output and you are supposed to do the pattern you know recognize the pattern from left to right from left to right from left to right so this one is an easy one it's basically the rule as you can see is there's always twice the same thing like next to each other or here on top of one another and you are required to fill out this test sample right here so i guess we can do that actually if you click on the left thing it will actually copy it over to the right side which just happens to be the uh the perfect solution here but you can you know fill in anything you want right here if we did it manually we would do it like this like this so i'll just copy like the top part here this and like this and you can check whether you get it correct and we got it correct okay very nice actually i should probably so let me obs uh let me put some sort of a banner right here of what we're doing no ah you cannot win a 30 90 at this time so we are solving arc cool uh yeah so arc challenge is this thing all of these are really simple for humans but really tough for machines so um here you see and you can't even formulate it right there's just kind of there's lines so you're kind of supposed to just take the color that is not the lines maybe i guess so we'll just all fill all of this with blue and we got it correct nice so they get from from simple uh to quite hard so um yeah you can check it out for yourself the data set has about 400 training example and every single example is um you have a link to arc yes sorry yeah i should probably so there's there's different things i'll just i'll put a link i'll put a link to i'll put a link to this thing um yeah i'll put a link here so yeah these are these go from easy to to to super hard but for humans they're never hard but for machines as you can imagine they're quite hard and every one of them uses kind of like a different uh notion so there's not twice there is rarely twice the same kind of logic behind how you solve these games um i've never tried google's fubar coding i don't even know what that is i'm sorry uh it's been a long time since i've done actual coding challenges however uh we at our discord have decided to give this a little bit of a shot so we have i should probably um i think this is public github.com arc community maybe yeah exactly so there you can see a bunch of of repos and we've just started so my goal in this repository right here is to provide a little bit of a framework um a little bit of a framework or of evaluating these things and i got pretty far so far so the idea is that i have a test right here the idea is that you would implement an agent so an agent is this thing and the agent implements a method called solve test sample and there you get a list of training samples these are these three samples or so and you get a test board and you're supposed to output top k list of solutions top k list is simply a list of boards so you're supposed to output boards you can make use of these hints right here although you will not get any points in the final result if you do but for start that's that you can use that the task data simply contains the top k that you're supposed to output so it's pretty pretty simple and you're simply you simply need to implement this method and then the goal here is that i then take your agent and evaluate it on the data set give it a nice report and so on um so that you can try yourself like how would you go about programming an algorithm that can solve these types of tasks which is really hard especially there's only 400 training examples as i said and 400 test examples and none no two examples use the same logic you just need to generalize from like the essentially the in-context learning right here and yes you could use a large language model uh for this but gpt 4chan made you notorious meaning i wasn't notorious before no yeah that was quite something but we're back to family friendly stuff no one absolutely no one's gonna complain about collary squares right so this is completely controversy free that's why it's a good good project so this is here is an implementation of a cheating agent so the cheating agent as i said you can use these hints right here and one of the hints is actually the output and the output will just give you the solution right so the cheating agent actually looks at the output and just returns that and you know good enough and um in fact it will be recorded in the hints that you actually access that so we'll have that in the evaluation so the goal now that i have today is to finish this setup of this evaluation code so people can actually use it productively let's say i already have code to download the data set and so on but i'm having a bit of trouble um how it when it comes to metrics so metrics in my mind is this thing and we can look at it so a metric could be like accuracy accuracy is like or correctness whether or not you solved a sample correctly that's kind of the most basic metric but there can be more metrics right there can be um how much of the pixels you got correct and so on so these aren't counted in the end you know you only get a point if you get it exactly correct however they can tell you something about the progress you're making so my idea of a metric is it has two methods one is compute and you would give that a single result right a single evaluation result you give to the metric it will produce something for example right did you solve the riddle correctly and the aggregate function then would aggregate this across all of essentially across all of the um evaluation results all of the riddles so one metric that i implemented here is the correctness metric as you can see right here so this would compute over the all the test examples of one riddle the minimum value so did you get essentially you get one if you get all the riddles correct and it would then aggregate across all of them in a mean fashion so the result of this metric would be how many or what fraction of riddles of the data set did you get correctly that's it um now so far what i've done is i have implemented these as property of the evaluation result you can see here is the evaluation result that's con that's constructed uh as your agent outputs a solution this is you know done and here i have metrics results in it and i don't really like it i think the metric you should be able to save the evaluation result and then compute maybe the metric even later on that evaluation result so we're going to rip out the metrics results from here and the aggregation results from this eval result list which i also construct and then hopefully hopefully um yeah how good is this the state of the art on arc i believe it's really weak like it's certainly below 20 or so or 30 solved or or even even weaker like it is it is quite weak uh there is not a good algorithm yet and it has some some properties right you actually need to you need to hard code human priors that that's actually it's in the paper so yeah it's tough um all right so i'm gonna rip essentially out these metrics results right here and i'm going to manage the metrics results externally so let's say so here we have yeah no no okay so here is a metric and the metric we can actually also rip out uh into its own file let's say we yeah into the metrics file why not so we have a metric right here like that and oopsie is the eval result in the interface yes okay so we'll import that we already have it okay uh and we need that the eval result here cannot be a string anymore okay i mean this is this is already good enough can you share you set up the vim rc um so yeah the the ide is neovim um and i am currently really strongly considering redoing my vmrc so first i want to use the the lua configuration of neovim and i want to like get different different plugins because these have become quite a mess you will also see if you watch me code that pirate like the loading bar is currently going all the time and stuff so it's it's really become not good and jumbled so i'll uh redo it and once i've redone it and i'm actually proud of the setup then i can i can maybe share it but i'm using relatively standard plugins i'm also using coc i'm using copilot and yum i don't actually know if copilot works in this because i'm using python 3 9 and neovim complained anyway so we get compute and aggregate and so the um yeah the problem here a little bit is well yeah uh i will make these actually underscore functions because we can then track so a pattern that i like to use often is anything wrong with vs code it's not vim that's that's kind of the the only thing [Music] so you know um okay and good i know this seems pointless but it is not [Laughter] because because yeah it is not pointless because you can give users kind of the opportunity to override this but you still have a hook into it right with the non-underscore functions uh as a framework developer it's an interesting pattern that i think makes sense a lot of times if you want to give people you know the opportunity to sub class things and you can still do logging and track access and so on in the non-underscore function especially here uh we now have the opportunity to instead of this give the eval results list no i missed by one okay see it's too it's kind of slow so yeah so now here we have to so out of the eval results list we have to fish out no actually yeah so now now we get into trouble right let's say we just cancel this for now and we make these into underscores so when we eval something we have a result and then we want to compute the metric on it and that gives us a metrics result right and in that case it's any now i'm going to play around with the typing or something but but in our case it's just kind of any so that's just a result we need to store that result somehow in a data structure so let's go over to the eval here so here we have apply metrics and yeah so what we'll do is we'll go see now there's the problem we'll go by metric by metric we'll go through all the results what are your thoughts on the minerva model i have not i will read the paper as soon as possible uh but i'm really trying to get this done so the community can actually use this code but it's it's super cool and i think that so i've read it a little bit but not in detail but my opinion so far is is data quality is really king right there there they just parse i've this is not i've also talked to some people about it and it seems like how well you can parse pdfs and stuff and and lottek is a big part of of your data quality and your data quality directly influences the results so better take good good care of your data right right it actually pays to pays off to to clean your data well and pre-process it well and so on seems to be a real boost all right so now we're going through every result so ah for eval result actually we can we can just use we can just ask copilot here see uh metric result okay so we get a result for each of these and we can probably make this into a list comprehension actually because um metric dot compute like this so that's metric results so for every riddle we now have a result and then what we're going to do is we're going to give that to the um to the aggregation function so metric dot aggregate now it depends on how we've implemented this right let's go over to the metric and so what do we have here we have the eval results and we also need a list of metrics results metric results these are the things that have been computed by compute above list any for now it's not pretty it's not pretty but you know that's that's what we get um yes i think that's that we can also have that exact same interface right here and we can just replicate this here i'm not even it's reckless to not check co-pilot output but sometimes you're just so confident that you can just give it to get to co-pilot all right so now we what we need is a data structure that keeps all of this stuff right um again we're going to probably go to metric and we are going to make we'll need pydantic if you don't know pedantic very good library like data classes except cooler metric result yeah why not let's call it metric results so uh it's yeah it's like data classes but nicely serializable nice print functions and to json from json and so on and and with validators really nice things so the metric result here now the only issue i have a little bit is the task has several riddles right and and they're just in a list now the metrics results also obviously you'll get a list of results the question is can we just simply like have two lists and sort of implicitly be like well the first element is the first element and the second corresponds to the second that's really not very nice but i don't know like we would need to make everything into like a dick um that's also not particularly sorry particularly um nice so hmm i think i'll just stick with the list so i'm going to say uh eval [Music] okay how are we calling these uh let's call them riddle results and this is a list and these and this is like aggregation result okay now let's call that compute results because the method is compute and aggregate results result and we'll still need to do something about that any but i will have to read the python documentation [Music] um on typing to to really see how we can do this with inheritance and so on because the idea is that you could define your output type here and then you the type checker could make sure that you'll get the same thing for now it's just any i'm i'm sorry okay i know you can do type var but i don't know how that behaves with inheritance although i think you could hear you should be able to do something and then have that show up can we okay maybe we'll we'll do this um all right and generics generic generic aha okay okay so a generic type is from an instantiation of this class yes generic okay must be a subtype of something [Music] we could do this right we could have inherit from generic and then we could yeah okay um from typing generic and then here we'd have generic t now t isn't known so we also need to type var t equals type for metric type what's wrong with that okay um let's just do it let's just say it's t so we can have all of these be t t t t right okay and then we need to just take care that here so now i'm wondering here this is a float and then we get a float right here right and this here is also float but i'm feeling like a a duper super programmer and obviously ah okay pedantic and generics do they play well do they do they go with one another let's check it out i'm not sure but okay okay i see the problem interesting yes we're doing we're doing api stuff generics from java yeah java is a mess especially generics however generic models there we go generic models are only supported that's fine that's fine too that's okay so here we have generic model generic data okay seems like we have to we have to inherit from generic model um can we do that you see the yeah pyrite is going and generic t and then here we get a t i'm still not super satisfied because because um well how about we have aggregate actually output the result right like that um like this and well here we can't have metric results we'll need to call it compute results too sorry i'm neglecting chat are we doing api stuff yes we're doing api stuff um hopefully hopefully compute results compute results now we're ready to everything's called result socket naming things so yes yes yes thank you co-pilot that is quite nice of you okay all right so the compute method is fine here is all fine all good and the aggregate method is fine and now we simply need to implement we would have to change the definition of this aggregate right here so here we have a class doing mean aggregation if you see this this needs to be a float obviously if it's some sort of a mean and yeah that's it um if not okay yep yep yep can you plug co-pilot in vim as yes tim pope himself actually wrote the copilot plugin and does not accept pull requests currently so but yeah exactly get value where do i have this get value gives a float i think this is fine luckily oopsy luckily i have written tests let's see how that goes that that is good very good very very excellent what do you mean what could not be does it but it says here oh generics okay all right nope generics there we go okay now our problem is that in eval we can't import metric that makes sense do we need it we actually need it okay good and metric compute eval result eval result tuple is not compatible oh no see now we get into typing typing issues so we're in the eval results list which is we actually need to access eval results eval result list dot eval results actually we could just extract that here like this and we're pretty good i think let's see wow that's slow all right that's better so this has no attribute oh oops this is no attribute aggregation results um okay aggregate i i called it aggregate result um yeah so now the tricky part is yeah we are not done by a long shot uh see here i say apply metrics okay this applies the metrics now what's it what what is it going to do it's going to call compute and that is fine or trying to make an api for metric eval yes exactly so i'm trying to write code for other people to use to evaluate their their agents on the ark challenge if you're new here the arc challenge is uh is uh uh this thing this yes so we are here and we then call aggregate aggregate takes the results list and the results from compute so we'll actually rename that to compute results we'll rename that to eval result list and we'll get a metric result out of that so now we get a result per metric now the metric result the metric itself has a name therefore the metric result should probably also have a name that makes a lot of sense um yeah yes it does it does make a lot of sense so good all right the metric result now has a name and metric results ta-da we'll add that to the list we'll return that list we'll say here that is a list of metric result a metric result will import from here all right so now we just need to um what's the console text editor this is neovim which is very slow because i'm streaming i have the camera as like an external camera feed over an hdmi i have a second screen that is streaming to the ipad uh and i'm running copilot and this is an api but i'm running a pyrite language server in the background so this thing is stressed uh but it it works it's fine um and my hard disk is almost full which i figured is is not good like if you're below 10 things get slow need to clean up all right um [Music] we'll have the eval test right here so this thing now if you apply metrics you should get a list of these metric results right which correspond to um or these these metrics here so the issue is it's just a list how would you explain the reasoning of the correct answer to a kid that failed the question yeah that is it's really it's like you can't um like a lot of times uh it is just human intuition e type of things so here for example look at that so there's just like the lower left or the top right thing goes to red and the lower left things go to blue now that contains so many human assumptions like you know what what is a thing what is an object right we recognize these same color thing as the object and maybe we recognize even the the back black pixels as the background but to an algorithm there's no objects there's just pixels and there's no things and there's no inherent separation and so on no come on what what that is mad what'd i do wrong i'm right [Laughter] see i don't even i don't even like i don't know what's wrong uh i'm pretty sure this is no oh come on well we'll have to live with with failing this one the bigger ones are red ah maybe it's the smallest the smallest one that's um or the one that no is or the one that taught no is codepal good or bad it's very good smallest is red okay let's try let's try that i'll make this blue and i'll make this red all right good job good job chat robbie encode the whole thing with clip then test the similarity will not work well prop no not gonna say it's not work will probably not work because clip is trained very much on natural images it like the the capability of like abstracting to like ob seeing objects in these types of of things is very hard um the challenge seems similar to the measure of intelligence paper uh yeah it's actually from that paper the last chapter in that paper describes this challenge so yeah a different object is different that could have been it too yes all right so the issue right now is we have a metric result right um but we will just get a list so i would like to have i would like to have i would like to have like a thing maybe i'll just return a dick but you it's it's good to have some sort of a type so we'll say we'll say a metrics result matrix dict is addict that goes from a string to a metric result right and we'll put that down here yup i guess that's cool so what we can then do and we actually don't need the any anymore that's good it's always a good sign so in eval what we can do instead of just having a list right here we can have a matrix result ticked we can instantiate that here um and we can not append but um at that here cool and then this is no longer a list but this is now a addict that seems fine so now the last thing we'll actually need is we'll need to create a kind of like a unified package of all of this so we now have um metrics that we call right and i think what we should do last is some sort of report right so let's call that reports um yeah why not report okay user bin and python um we'll make it pedantic report base model okay and all right so the report is going to be generated by eval so what do we need in a report on this thing let's make this a bit bigger so you all can see oopsie hey this crashed all right um let's split it so we'll certainly need the eval result list right um yeah so from interface import eval result list then we need these metrics results so import um metric result dict matrix metric result dick yes okay then what do we need we may need like what data set was used maybe maybe who knows um let's consider eval again we may need a date time like can we do that um so yeah we can do that right hmm sure why not this top of the screen is cut off oh i'm sorry that is how about this or this yeah better [Music] a bit more yeah so a date time to to check when the report was generated um do we do we want to call this like metadata or so maybe not uh import date time so um how about this field with a default factory oh okay now it's trying to get it right i think this is okay cool yep um [Music] we can give it i don't know tags uh [Music] how about oh yeah the data set maybe that is also a good thing so um i have a dataset utils class right here and um these sub dears let's say sub dears list of string and we'll also we'll also comment maybe comment string something like this right something that packages all of these things together and then uh and then saves it down so these are all either pedantic models or standard models so we are going to and this one doesn't even have to be here all we need is the eval results really and now we have we can have various methods to actually render this or yeah so we can have various representations of that however i'm not sure if i want to implement stir right here would augmenting tag the screen would stream with asmr okay uh would augmenting example with natural descriptions help maybe i don't know could you create a program to procedurally generate similar puzzles with a text label to train on like an l or an i system then learn to generate the text from the samples results from the text yeah probably i mean if you can come up with text descriptions but again there's 400 test examples which is not too many so a lot of deep learning type of things are out of the question at least without help but generating data is one of the things one of the routes we are considering yes but the question is how because you know you need to build in these human priors like gravity and objectness and so on so i don't want to um let's save to file uh file name no no no we'll go os so all of this um from path lib import path we'll make this a path like very good thanks copilot so this saves the report to a file now the eval here it currently evaluates agents on riddle and on riddles it applies metrics now we need a function that um well it's a good question what we now need is a function how can i improve my coding skills have you heard of bit tensor um i have heard of it my current opinion is that is buzzwords but i i've not looked into it very clearly i hear there is an interview on towards data science but i've not listened to it how can i improve my coding skills just code and look at other people's code and submit pull requests to stuff and take the feedback with a grain of salt but take it seriously um [Music] or or join a company that has some good standards or something or look at coding guidelines from big companies because you know they for for for some reason they have survived and that's a challenge if you code with a lot of people so they need good standards all right so we now need we now need a last function that does everything you know uh maybe at once nah i'm not sure that loads kind of loads so if we load the data set we say evaluate we say apply metrics and then we ship this all in we we shove it all into report um that should technically you know do all the things but we could do it in the cli yes probably we should do it in that cli so there's a cli uh by the way i'm i'm really proud of the cli so you can do something like so if you type arc that's the cli if you've installed the thing so you can do help you can do arc show random id it will show you one of the tasks that's a bit of a big task but maybe i zoom out for see it will show you like a random id uh puzzle which is pretty cool and then you can try to solve it this is all in console that's why it's especially cool so um yeah we need one thing that evaluates an agent on a data set and that's going to be the kind of last thing um now the question is can i load an agent from can i load an agent from this so eval can we import the data set thing from from eval because probably probably um clip happens to know physics he also fears its own mortality no not clip but yeah that was that was a fun certainly a fun fun thing um i'm just trying to think how we best do this uh how about we make an example so let's make not here let's make examples like that out here actually let's move that into a directory called examples so we'll take that cheating agent maybe i should maybe i should move this um to this like base like that seems pretty good and then we can do an init file right and just import the agent from import agent we'll all it out good all right and then we can have um the cheating agent inside of this one so let's go to the eval test rip out the cheating agent like this put it in here from agent um import task data or hints board pair board help k list i think we have it now port hints isn't exported oh that's from hints maybe i guess we we need to export all of these from arc anyway challenge seems to be solvable with only low level priors but high level prices sequence of translation with reasoning is made up of more primitive priors yes but also the com like there's not an arbitrary combination that's also one of the challenging things so i don't i don't really know how to yeah so we should be able to import all of these from arc i think like this so let's make that happen here yeah from import agent task data uh we'll also need from import board hints um hints certainly will also need the top k list right here i hope that's enough good all right so this cheating agent as i said is able to just look at from arc import star no no no no no no bad didn't someone ask how can i improve my coding not like not with star imports although sometimes it's it's just quick i think yeah especially if you're if you want to do something quick so the cheating agent uh will import that then in the eval test right here because we can we don't need any of these anymore except task data um from arc dot yep i need to restart whenever i make a new file i need to restart like that that's dumb something's wrong with my setup clearly so does this still run see how slow this is it's terribly slow yup has no aggregation results yeah so now is the part where we have to actually change the test so here is the metrics yep here we have the eval results i would call that result list um apply the metrics yes that was give you'll give us a metric result dict um [Music] yep and how about that garbage collector yeah maybe ah okay i need to [Music] aggregate result yes hopefully good test passes wow um let's look at it okay come commit uh metrics okay now we need like a last function that that just generates the report from the riddles and then maybe one that also loads the data set sorry and then we're good again okay any an interface ah yeah i see go go go go go go excellent okay ship it and yeah we'll go on so let's go to eval let's um evaluate agent on riddles no no how do we call this i want to take something that's takes riddles metrics and generates a report essentially it's just going to call the things above so it'd be evaluate agent on riddles and apply metrics and generate report uh no um is hard it's hard report from riddles that's not too bad all right let's do this um evaluate r3 uh and matrix so essentially we'll do the exact same thing as we did before no if not the evaluate on radios report from riddles that yes yes yes no not matrix results i want a report apply metrics um the report does it have uh how about instead because we we'll just call that eval results metric results that should be good all right return report i know results evaluate and report that is that that is that is a good good name very good name evaluate evaluate on riddles good here i want evaluate on data set and report so instead of riddles we'll take subdirectories no actually that that's that's fine that's just good enough very good biometrics has everything it needs good nice alright so format it good now we can [Music] i still don't know how we should get the agent now in there but we can make an example this we can rewrite the eval test right here um to this so let's have uh yup this that actually yeah let's just say evaluate and report and let's rename it good idea very good idea what plugins are using i don't know i've lost overview myself so you know [Laughter] um and that should probably optionally take metrics right because uh no let's do it right if matrix is none yep um yep cool we don't need apply metrics this will do it all so [Music] yep okay now the one thing i think we also need is we'll need some default metrics right um get default metrics and that should give us a list of metrics and that would return yeah why not these two so here we can actually instead of this we can say default metrics because most people are just going to use the default metrics test it again and good so i have about more five five to ten more minutes of time and i'm going to use that to probably make an example and the example is going to be pretty much just the the test so here is the example um ah [Music] copilot go go thank you arc agents cheating important cheating agent will from arc.metrics import default metrics and um [Music] evol so oh yeah the riddles yes now what we'll actually need is also the data set from utils maybe we should the evaluate and report maybe we could export that somehow in arc because it's it might be very common right so like this [Music] good um okay the task data how do we know if it's not a bot and co-pilot um yeah we'll need get riddles from the training subdirectory that's fine okay and the task data let's say top k is three um correct fraction now this is going to be one hopefully let's try it out let's go to examples let's try it and we'll see we should probably have some sort of um indicator of the progress and stuff i think that would be be something no well correct fraction one um yeah so do we have a logger already from three months ago he looked more jacked that is true that is actually true i have not had time to work out too much uh was busy getting cancelled so um um info evaluating agent on some riddles let's spin here evaluate agent on riddles ah this goes up here probably that's better um let's put in some tucodum here do we have it in the requirements do we have it in the setup we have it good so where was i just now there we go um all right let's see how that works nice does wearing shades make you a better code didn't know vim had auto complete functions it's plugins it's all it's language servers um downloading get where get riddles so here all right that should do then the last thing applying metrics how about this let's try we're loading the data set very nice evaluating very nice loading bar excellent now the apply metrics correct apply metric board size correct fraction 1. we should probably good and that is a really nice example it's pretty simple if you ask me and then you can store that to some file if you want and you can probably also implement your own agent but that's a task for another day so i yeah added example isn't it hard to see yes especially if they're polarized so if i do this i can't see anything but you know so in case anyone is interested in solving the arc challenge this is currently our community project on our discord which i hope you'll find if not it's like why culture dot com slash discord um in case you're interested i will be pushing all this code soon and i'll also make a bit of a i'll make more examples on how you can exactly interact with this but ultimately you can you'll also be able to use the the cli for it and yeah and i'll make the reports do prettier outputs and that will be that i think that will go right here you'll be able to load some sort of data set um hopefully probably yes you'll be able to load some sort of data set generate a report what is it yes this is a library that you can use to evaluate your agents what i don't know yet is how do apps like flask and so on um or uv corn you're given you give them like a file name right and they dynamically seem to load whatever is in the file name so uh uh crap how how can we look this up python dynamically load class from file by file name or something import from dynamic file name yeah see i have a python script config file yes exactly so that is my problem the import module to dynamic import modules this is what we need so this um i'll make it a bit bigger ah cool so import module and then get adder hey this isn't too hard we could do that we could totally do that um okay let's try let's try we got five minutes um so we'll make a command this is eval let's just call that yes um and that's going to take how should we how should we do this um that's going to take like agent i then agent path or something no i want something like flask has so let's just do this a string yeah it's not it's not particularly good it's not a particularly good interface yet but um hmm i'm not i'm not a fan of it yet sub dear i'm not a fan of it yet how about we just do this for now so what i want to do is i want to agent file path agent class name is as we split it by this so you would be able to do hopefully you would be able to do um arc eval and then you'd say something like arc or agents um cheating agent dot cheating agent like this right and use import lib yeah exactly that's pretty good we'll do it so import import module ah but it needs to be in path so we can't actually do this right here we'll have to do we'll have to do well this is super duper slow like this yes yes yes good uh agent module name okay agent class very good i co-pilot is doing all of this for me really there's nothing nothing i need to do um and now i need to to evaluate it right so um import evaluate and report so i'll get my i'll get my uh data set get riddles and we'll get like we'll make later parameters but report equals evaluate and report no that's just the function evaluate and report takes agent riddles task data and metrics riddles so um arc import task data and get default metric that's probably also something we should put into arc itself no i think so yes yes good um [Music] okay again what do we need here agent bidless task data and matrix okay and then we print from the report we probably want to print something like uh the metric results let's try if this works yes it actually appears to work wow okay now it just needs a bunch of parameters and then it's cool yay very nice very nice okay yeah it gets everything correct very cool so yeah we'll just be able to use the cli uh um oh added eval to cli this is going to change a little bit the eval but i hope it's going to be useful and that's that from me i'll just push this as i said the repository is open discord community is open um we're currently just in the beginning stages of solving this i hope we get farther than the people so far um and get actually something to work which would be nice all right thank you everyone for being here that was it uh just some chilling saturday afternoon coding and i have to do some actual work now all right thank you very much bye