Agents @ Work: Dust.tt — with Stanislas Polu

Key Takeaways

hey everyone welcome to the Len space podcast this is celesio partner and CTO at deible partners and I'm joined by m host swix founder of smalli hey and today we're in the studio with Stan po welcome thank you very much for having me visiting from par Paris and you have had a very uh distinguished career it's very hard to summarize but you went to a college in in both Eco poly technique and Stanford and then you worked in a number of places Oracle toems stripe and then open AI pre chbt uh we'll talk we'll spend a little bit of time about that about 2 years ago you left open AI to start dust I think you were one of the first open eye Alum Founders yeah I think it was about at the same time as the adep guys so was that first wave yeah and uh people really loved the David episode uh we love a few like sort of open the eye stories U you know for back in the day like like we're talking about pre-recording probably the statue of limitations on on some of those stories have has expired so it's you can talk a little bit more freely without com but maybe we'll just talk about like what was your journey into AI um you know you were at stri for almost 5 years there a lot of stripe alarms going into open the eye I think like stripe culture has come into open the eye quite a bit Yeah so I think the the bursters of stripe people uh really started flowing in I guess after chat GPD but yeah my journey into is a is aan Greg BR yeah from from GRE of course um and and danela actually back in the days danela mod yes she was uh coo I mean she is co yeah she had a pretty high high job at open at the time yeah for sure my journey started as anybody else you're fascinated about computer science and you want to make them think it's awesome but it doesn't work I mean it was a long time ago I was like maybe 16 so it was 25 years ago then the first big exposure to AI would be at Stanford and I'm going to like disclose a hold I am because at the time it was a class taught by Andrew Ang and it was there was no deing it was half features for vision and a star algorithm so it was fun but it was the early days of of deing at at the time I think few years after it was the first project at Google uh but you know that that cat face or the human face trained from many images went to uh hesitated doing a phg more in systems eventually decided to do uh to go into uh into getting a job went at horle started a company did a gazan mistakes got acquired by stripe walked with Greg Workman there and at the end of of tribe I started essing myself in AI again felt like it was the time you had the at games you had self-driving craziness at the time and I started exploring project it felt like the atams were incredible but there were still games and I was looking into exploring project that would have an impact on the world and so I decided to explore three things uh self-driving cars cyber security and Ai and math and AI it's like I think by a decreasing order of impact on the world I guess yeah discovering new map would be very foundational it is extremely foundational but it's not as direct as you know driving people around sorry you're do as a stripe you were like thinking after stripe kind of a bit of time where I I started exploring didn't of work with friends on trying to get LC cars to drive autonomously almost started a company in France or Europe about self drive driving trucks we decided to not go for it because it would like it was like probably very operational and I think the end of the of the team wasn't there and also I realized that if I wake up a day and because of bug I wrote I kill the family it would be a bad experience and so just decided like no that's that's just too crazy then I explored uh cyber security with a friend we're trying to apply Transformers to cut fuzzing so cut fuzzing you have kind of an sorry an algorithm that goes really fast and tries to mutate uh the inputs of a library to find bugs and we try to apply a Transformer to that and do reinforcement learning with the signal of how much you propagate within the the the binary uh didn't work at all because the Transformers are so so slow compared to evolutionary algorithms that it kind of didn't work and then started interesting myself in math and Ai and started working on set solving with AI and at the same time open a was kind of starting the reasoning team uh that were tackling that project as well I was in chat in touch with Greg and eventually get in touch with Ilia and finally found my way to open ey I don't know how much you want to dig into that the way to to find your way to open when you in Paris was kind of an interesting Adventure as well please uh and I want to note this was the two-month Journey you did all this in two months the search the search for what sorry your search for your next thing cuz you left in July 2019 and then you join opening at September I'm going to be ashamed to say that you were searching before yeah I was searching it before I mean it's it's normal it's normal no the truth is that I moved back to Paris through stripe and I just felt the hardship of being remoted from your team N9 away and so kind of freed a bit of time for me to start the expression before sorry Patrick sorry John hopefully they're listening um joining openi from Paris and from like obviously you had worked with Greg but not not anyone else no yes so I had work with I had worked with Greg but not IIA but I had started chatting with Ilia and IIA was kind of excited because he knew that I was a good engineer through Greg I presume but I was not a train researcher didn't do a PhD never did research and I start chatting and he was excited all the way to the point where he was like hey come past interviews it's going to be fun I think he didn't care where I was he just wanted wanted to try walking together so I go to SF go through the interview process get an offer and so I get Bob McGrew on the phone for the first time he's like hey Stan it's awesome you've got an offer when are you coming to SF I'm like hey it's awesome I'm not coming to the SF I'm rais in Paris and we just moved he was like hey it's awesome well you don't have an offer anymore oh my God no it wasn't as har as that but that's basically the idea uh and it took me like maybe a couple more time to keep chatting and they eventually decided to try a contractor setup and uh that's how I kind of started working at open officially as a contractor but um but in practice really felt like being an employee what did you work on so I was a solely focused on math and AI okay and in particular in the application so the study of the larage models mathematical reasoning capabilities and in particular in the context of formal mathematics okay the motivation was simple Transformers are very creative but yet they do mistakes and formal math systems are the ability to verify a proof and uh the tactics that you can use to solve problems are very mechanical so you miss the creativity and so the idea was to try to explore both together you would get the creativity of the LMS and the kind of a verification capabilities of the formal system a formal system just to give you a bit of context is a is a system in which a proof is a program and the formal system is a type system A typee system that is so evolved that you can verify the program if the type checks it means that the program is correct is the verification much faster than the than actually executing program it is ver verification is instantaneous basically yeah so the truth is is that what you code in involves tactics that may involve computation to search uh for Solutions so it's not instant you do have to do the computation to expand the tactics into the actual proof the verification of the the proof at the very low level is instantaneous yeah how quickly do you run into like you know halting problem P andp type things like impossibilities where you're just like that I mean you don't run into it at the time it was really trying to solve very easy problems so I think the can you example of easy yeah so that's the mass Benchmark that everybody knows today the Dan HRI Dan Hendrick one yeah and I think it was the low end part of the mass Benchmark at the time because that mass Benchmark in CL AMC problems amc8 AMC 10 12 so these are the easy ones then a am problems somewhat harder and someo problems like for listeners we covered this in our benchmarks 101 episode AMC is literally the the grade of like high school grade eight grade 10 grade exactly so you can solve this um just briefly to mention this because I don't think we'll touch on this again there's a bit of work with like lean and then with uh you know more recently with uh with de mine doing like scoring like silver on the Imo um any commentary on like how math has evolved from your early work to today I mean that result is mind blowing I mean from my perspective spent three years on that at the same time guom lump in Paris we were both in Paris actually he was at fair was working on same problems we were pushing the boundaries and the goal was the IMO and we cracked a few problems here and there but the idea of getting a medal at an IMO was like just remote yeah so this is an impressive result and we can I think the uh Deep Mind team just did a good job of scaling I think there's nothing too magical in their approach even if it hasn't been published there's a Dan silver talk from seven days ago where it goes a little bit into more details it feels like there's nothing magical there it's really applying reinforcement learning and scaling up the amount of data they can generate through autorization so we can dig into what autorization means if you want let's talk about the the tail line maybe of the so you join and you're like I'm going to work on math and do on all these things I saw in one of your block posts you mentioned you find two over 10,000 models at open using 10 million a100 hours how did the research evolve from you know the gpd2 and then getting closer to like the vinci3 and then you left just before CH gbd was released but like tell people a bit more about the research path that took you there I can give you my my perspective of it I think uh at open there's always been like a large R of the compute that was reserv to training the gpts which makes sense uh so it was pre- anthropic splits most of the computer was going to a product called Nest which was basically gpt3 and then you had a bunch of let's say remote not core research teams that were trying to explore maybe more specific problems or maybe the algorithm part of it the interesting part I know if if it was where you question was going is that in those labs you're managing researchers so by definition you shouldn't be man managing them but in that space there's a managing tool that is great which is computer location basically by managing the computer location you can message the team of where you think the priority should go and so it was really a question of you were free as a as a researcher to work on whatever you you you wanted but if it was not align with open a emission and it's fair you wouldn't get the the computer location as it happens solving math was very much align with with the computer with the the the direction of open and so I was lucky to generally get the computer I needed to make to make good progress what do you need to show as incremental results to get funded for further results it's an imperfect process if you're working on mass and AI obviously there's kind of a prior that it's going to be align with the company so it's much easier than to go into something much riskier I guess you have to show incremental progress I guess it's like you ask for a certain amount of a compute and you you deliver a few weeks after and you saw you demonstrate you have a progress progress might be a positive result progress might be a strong negative result and a strong negative result is actually often much harder to get or much much more interesting than a positive result and then it's a It generally goes into as any organization you would have kind of a people finding your project or any other project kind of a cool and fancy and so you would have that kind of phase of growing up computer location for it all the way to a point and and then maybe you reach a an a an apex and then maybe you go back to mostly to zero and restart the process because you're going in a different direction or something else that's how I felt explore exploit yeah yeah exactly exactly exactly it's PhD student the search process and you were reporting to Ilia like the results you were kind of bringing back to him or like what's the structure it's almost like when you're doing such cutting cedge research you need to report to somebody who's actually really smart to understand if the direction is right so we had a reasoning team which was working on on reason obviously and and some MTH in general so and that team had a manager but IIA was extremely involved in the team as an advisor I guess since he brought me in open a I was lucky to mostly for during the first years to have kind of a direct access to him he would really coach me as a training researcher I guess with good engineering skills and Ilia I think at openi he was the one showing the showing the North Star right he was his job and I think he really enjoyed that and he did it super for while was going through the teams and saying this is where we should be going and trying to you know flock the different teams together towards towards an objective I would say like the public perception of him is that he was the strongest believer in scaling oh yeah he was he always pursued like the compression thesis y you have worked with him personally what what do what does the public not know about how he works I think he really focused on building the vision and communicating the vision within the company which was extremely useful I was personally surprised that he spent so much time you know working on communicating that vision and getting the teams to work together versus and to be specific vision is Agi oh yeah the vision is like uh yeah it's it's the belief in compression and and spanning computes I remember when I started working on the reasoning team it it was the excitement was really about scaling the computer on reasoning and that was really the the belief he wanted to ingrain in the team and that's what has been uh useful to the team and and with the Deep Mind result shows that it was the right approach with the the success of gg4 and stuff shows that it was the right approach it was it according to the neural scaling laws the Kaplan paper uh that was published I think was before that because those ones came with gp3 basically at the time of gp3 being released or being ready internally but before that there really was a strong belief in in scale I think it was just the belief that the Transformer was a generic enough architecture that you could learn anything and that it was just a question of scaling any other fun stories you want to tell you know Greg you know any I didn't really I didn't work that much with Greg when I was at open he was uh he always been mostly focused on on training the gpts and rightfully so one thing about samman he really impressed me because when I joined he had joined not that long ago and it felt like he was uh kind of a a very high level CEO and I was mind blown by how deep he was able to go into the subjects within a year or something all the way to a situation where when I was having lunch by year two I was at open AI with him it would just quite know deeply what I was doing and and with no ml background like let's you know yeah with no but I didn't I didn't have any either so I guess that explains why but I think you can you it's it's a question about really you don't necess need to to understand the very technicalities of how things are done but you need to understand what's the goal and what's being done and what are the recent results and all of that in you and we could have kind of a very productive discussion and that really impressed me given this size at the time of open end which as not negligible yeah I mean you've been a you were a Founder before you're founder now and you've seen Sam as a Founder how's he affected you as a as a Founder I think having that capability of uh changing the the scale of of your attention in the company because most of the time you operate at a very high level but being able to go deep down and being in the known of what's happening at on the ground is something that I feel is really enlightening that's that's not a place in which I ever was as a Founder because first company we went all the way to 10 people current company there's 25 of us so the high level the the sky and the ground are pretty much at the same place no no you're being too humble I mean stripe was also like ack stri strip I wasn't a funer so it was like at open I was really happy being uh being on the ground pushing the machine making it work yeah uh last opening eye question y the the anthropic split you mentioned you were around for that very dramatic David also left in in around that time you you left this year we've also had a similar management shakeup let's just call it can you compare what it was like going through that split during that time and then like does that have any similarities now like are we going to see a new anthropic emerge from these folks that have just left that I really really don't know at the time the split was pretty surprising because they had been trying gpt3 it was a success and to be completely transparent I wasn't in the wids of the split what I understood of it is that there was a disagreement of the commercialization of that technology I think the the focal point of that disagreement was the fact that we started working on the API and wanted to make those models available through an API is that really the core disagreement that I don't know was it safety was it commercialization exactly or did they just want to start a company exactly exactly that I don't know but I think the what I was surprised of is how quickly open ey recovered at the time and I think it's just because the we were mostly a research org and the mission was so clear that some Divergence in some team some people leave uh the mission is still there we have the compute we have we have a sh so it just keep going yeah very deep bench like just a lot of talent yeah yeah so that was the opening eye part of the history exactly so then you leave open in September 2022 and I would say in Silicon Valley the two hottest comp compies at the time were you in Lang train what was that start like and what did you decide to start with a more developer focused kind of like a AI engineer tool rather than going back into do some more research and something else yeah first I'm not a trained researcher so going through op was really kind of the PHD I always wanted to do but research is hard you're digging into a field all day long for weeks and weeks and weeks and you find something you get super excited for 12 seconds and at the 30 seconds you're like oh yeah that's was a and you go back to to digging I'm not a trained like formally trained researcher and it wasn't kind of a necessarily an ambition of me of creating of having a a research career and I felt the hardness of it I enjoyed a lot of like that a ton but at the time I I decided that I I wanted to to go back to something more productive and the other fun motivation was like uh I mean if we believe in AGI and if we believe the timelines might not be too long it's actually the last train living a station to start a company after that it's going to be computers all the way down right and so that was kind of the true motivation for like trying to go uh to go there so that's kind of the core motivation at beginning personally and the the motivation for starting company was uh pretty simple I had seen GPT 4 internally at the time it was September 2022 so it was pretty GPT but GPT 4 was ready since a I mean i' been ready for a few months internally I was like okay that's that's obvious the capabilities are there to create an insane amount of value to the wall and yet the deployment is not there yet the revenue of open a at the time were ridiculously small compared to what it is today and so the tessis was there's probably a lot to be done at the product level to unlock the usage yep let's talk a bit more about the for factor maybe I think one of the first successes uh you have was kind of like the web GPT like thing like using the models to Traverse the web and like summarize thing and the browser was really the the interface why did you start with the browser like what was it important and then you built xp1 which was kind of like the browser extension so the the starting point of the time was so if you wanted to talk about LMS it was still a rather small community a community of mostly researchers and to some extent very early adopters very early Engineers it was almost inconceivable to just build a product and go sell it to the Enterprise though at the time there was a few companies doing that the one on on marketing I don't remember its name Jasper but so the uh the natural first intention first first first intention was to go to the developers and try to to create tooling for them to create product on top of those models and so that's what dust was originally it was quite different than L chain and L chain just uh bitted the [ __ ] out of us uh which is great it's it's a choice you were cloud in closed Source they were open source yeah so technically we were open source and we still are open source but I think that doesn't really matter I had the strong belief from my search time that you cannot create an LM based workflow on just one example basically if you just have one example you have a fit so as you develop your interaction your orchestration around the LM you need a dozen example obviously if you're running a dozen example on a multi-step workflow you start paralyzing stuff and if you do that in the console you just have like a messy stream of tokens going out and it's very hard to observe what's going there and so the idea was to go with an UI so that you could kind of introspect easily the output of each interaction with the model and dig into there through an UI which is was that open source I actually didn't oh yeah it was I mean dust is entirely open source even today we're not going for open it matters I didn't know that yeah know the reason why is because we're not open source because we're not doing an open source strategy yeah it's not an open source go market at all we open source because we can and it's fun open source is marketing you have all the downsides of Open Source which is like people can CL you um but I I think that D side is is a big FY okay yes anybody can clone dust today but the value of dust is not the current state the value of dust is number of eyeballs and and hands of developers that are creating to it in the future and so yes andbody can close today but that wouldn't change anything there is some value in being open source in a discussion with the security team you can be extremely transparent and just show the code when you have discussion with users and there's a bug or feature missing you can just point to the issue show the PO request show the show exactly oh welcome that doesn't happen that much but but you can show the progress if the int if the person that you're chatting with is a bit technical they really enjoy seeing the proquest advancing and seeing all the way to deploy and then the the down sides are mostly around security you never want to do security by ofation but the truth is that your vector of attack is facilitated by you being open source but at the same time it's a good thing because if you're doing anything like bug bonting or stuff like that you just give much more tools to the bug boners so that their output is much better so there's there's many many many trade-off I don't believe in the value of the code base per se I think it's really the people that are on the code base that have the value and the go to market and the product and all of the things that are around the code base obviously that's not true for every every Cod base if you're working on a very secret Kel to accelerate uh the inference of llms I would buy that you want to be open source but for product stuff I really think there's no there's very little risk y I signed up for xp1 I was looking January 2023 I think at the time you were on da vinci3 given that USC in gbd4 how did you feel having to push a product out that was like using this model that was like so inferior and you're like please just use it today I promise it's G to get better it's like just overall as a Founder like how do you build something that maybe doesn't quite work with the model today but you're just expecting the new model model to be better yeah so actually xp1 was even on a on a smaller one that was the Post jpt Release small version so it was it babage no no no no not that far away but it was the uh the small version of of CHT basically I don't remember its name yes you have a frustration there but at the same time I think expain one was design was an experiment but was design as a which we useful at the current capability of the model if you just want to extract data from a LinkedIn page that modable was just fine if you want to summarize an article on on a newspaper that model was just fine and so it was really a question of trying to find a product that works with the current capability knowing that you will always have Tailwinds as models get better and faster and cheaper so that was kind of a there's a little bit of a frustration because you know what's not there and you know that you don't have access to it yet but it's also interesting to try to find the product that works with the current capability and we highlighted xp1 in our anatomy of autonomy Post in April of last year which was you know where are all the agents right so now we spend 30 minutes getting to what you're building now yeah so you basically had a developer framework then you had a browser extension then you had all these things and then you kind of got to where dust is today so maybe just give people an overview of what dust is today yeah and the court Theses behind it yeah of course so dust really want we really want to build the infrastructure so that companies can deploy agents within their teams we are horizontal by Nature because we strongly believe in the emergence of use cases from from the people having access to creating an agents that don't need to be developers they have to be thinkers they have to be curious but they can like anybody can create an agent that will solve an operational thing that they're doing in their DayDay job and to make those agents useful there's two Focus which is interesting the first one is an infrastructure Focus you have to build the pipes so that the agent has access to the data you have to build the pipe such that the agents can take action can access the web Etc so that's really an infrastructure play Main in connections to Notions like GitHub all of them is a lot of work it is boring work boring infrastructure work but that's something that we know is extremely valuable in the same way that stripe is extremely valuable because it maintains the pipes and we have that dual Focus because we're also building the product for people to use it and there it's fascinating because everything started from the conversational interface obviously which is a great starting point but we're only scratching the surface face right I think we at the pong level of llm productization and we haven't invented the C3 we haven't invented Counter Strike we haven't invented cyberpunk 27 777 so this is uh really our our mission is to uh to really create the product that let people equip themselves to just get away all the work that can be automated or assisted by L&M and can you just comment on different takes that people had so maybe at the most open is like autog gbt it just kind of like just trying and do anything it's like it's all magic there's no way for you to do anything then you had the Adept uh you know we had divit on the podcast they're very like super hands- on with each individual customer to build super tailor how do you decide where to draw the line between this is Magic this is exposed to you especially in a market where most people don't know how to build with AI at all so if you expect them to do the thing they're probably not going to do it yeah exactly so the auto GPD approach obviously is extremely exciting but we know that the agent capability of models are not quite there yet it just gets lost so we starting with starting where it works same with X1 and where it works is pretty simple it's like uh simple workflows that involve a couple tools where you don't even need to have the model decide which tools it's used in the sense of you just want people to put it in the instructions it's like take that page do that search pick up that document do the work that I want in the format I want and give me the results there's no smartness there right in terms of orchestrating the tools it's it's mostly using English for people to program a workflow where you don't have the constraint of having compatible API between the two that kind of personal automation would you say it's kind of like um llm zappier type of thing like if this then that and then you know do this then this so very you're programming with English so you're programming with English so you're just saying oh do this and then that you can even create some some some form of apis you say when I give you the command X do this when I give you the command y do this and you describe the workflow but you don't have to create boxes and create the workflow explicitly it's just need to describe what are the tasks supposed to do it would be and make the tool available to the agent tool can be a semantic search the tool can be quaring into a structured database the tool can be uh searching on the web um and obviously the interesting tools that we only stting to scratch are actually creating external actions like reimbursing something on stripe uh sending an email clicking on a button in the ad me or something like that do you maintain all these Integrations today we maintain most most of the Integrations we do always have an escape atch for people to kind of custom but the reality is that the reality of the market today is that people just wanted to walk right and so it's mostly us maintaining the integration as an example a very good source of information that is key to productize is Salesforce because Salesforce is basically a database and a UI and they do the [ __ ] they want with it and so every company has different models and stuff like that so right now we haven't we don't support it natively and the type of support or real native support is will be slightly more complex than just oing into it like is the case with slack as an example because it's probably going to be oh you want to connect your Salesforce to us give us the soql that's the Salesforce ql language give us the queries you want us to run on it and inject in the context of dust so that's interest interesting not Integrations are call and some of them require a little bit of work on the user and for some of them that are really valuable to our users but we don't support yet they can just build them internally and push the data to us I think I understand the Salesforce thing but let me just clarify are you using brow browser automation because there's no API for something no no no no no in that case so we do have browser automation for all the use cases that imply the the public web but for most of integration with the internal system of the company it's really runs through API haven't you felt the pull to RPA browser automation that kind of stuff I mean what I've been saying for a long time maybe I'm wrong is that if the future is that you're going to stand in front of your computer and looking at an agent clicking on stuff then I'll eat my computer and my computer is a big Lenovo it's black doesn't sound good at all compared to a Mac I me if the API are there we should use them there always going to be a long tail of stuff that don't have apis but have the the the wall is moving forward that's that's disappearing so the core a value the uh in the past has really been oh this this old '90s product doesn't have an API so I need to use the UI to automate I think for most of the ICP companies the companies that ICP for us the scale ups that are between 500 and 5,000 people tech companies most of the S they use have apis not as an interesting question for the open web because there are stuff that you want to do that involve websites that don't have apis and the current state of web integration from which is us and open Ai and anthropic I don't even know if they have web navigation but I don't think so the current State of Affair is really really broken because you have what you have basically search and headless browsing but Ed bring I think everybody is doing basically body. inner text and and fill that into the model right there's parsers into markdown and stuff super excited by the companies that are exploring the the capability of of rendering a web page into a way that is compatible for a model being able to maintain the selector so that's the basically the the place where to click in the page through that process expose the actions to the to the model have the model SE an action in a way that is compatible with with model which is not a big page of a full dumb that is very noisy and then being able to decompress that back to the original page and take the action and that's something that is really exciting that what that will kind of change the uh the level of things that a model that that agents can do on the web that I feel exciting but I also feel that the bulk of the useful stuff that you can do within the company can be done through API the data can be retrieved by API the actions can be taken through API yeah for listeners I'll not that you're basically completely disagreeing with David one exactly exactly you know I mean adep is where it is and you know and dust is where it is so dust is still standing can we just quickly comment on function calling Yeah you mentioned you don't need the models to be that smart to actually pick the tools have you seen the models not be good enough or is it just like you just don't want to put the complexity in there like is there any room for improvement left in function calling or do you feel usually consistently get always the right response the right parameters and all that so that's the tricky product question because if you if the instructions are good and precise then you don't have any issue because it's scripted for you and the will just look at the scripts and just follow and say oh he's probably talking about that action and I'm going to use it and the parameters are kind of abused from the state of the conversation I'll just go with it if you provide a very high level kind of a auto gpts level at in the instructions and provides 16 different tools to your model yes we're seeing the models in that state making mistakes and there is um obviously some progress some some progress can be made on the capabilities but the in interesting part is that there is already so much work then and that can assist augment accelate by just going with pretty simply screw to four actions uh agents what I'm excited about by stalling in like pushing our users to create rather simple agents is that once you have those working really well you can create Mata agents that use the agents as actions and all of a sudden you can kind of have aarchi of of responsibility that will probably get you almost to the point of the auto GPT value it required the construction of intermediary artifacts but you you you're probably going to be able to achieve something great I'll give you some example we have or incidents are shared in slack in a specific Channel or Shi are shared in slack we have a meeting where we have a table about incidents and shipped stuff we're not writing that weekly meeting table anymore we have an assistant that just go find the right data on slack and create the table for us and that assistant works perfectly it's trly simple right take one week of data from that channel and just create the table and then we have in that weekly meeting uh some uh obviously some graphs and and and Reporting about our financials and our progress and our AR and we've created assistance to generate those directly and those assistants works great by creating those assistants that cover those small parts of that weekly meeting slowly we're getting to inall where we'll have a weekly meeting assistance we'll just call it you don't need to prompt it you don't need to say anything it's going to run those different assistant and get that notion page just ready and by doing that if you get there and that's an objective for us to us using dust get there you're saving I don't know an hour of company time every time you run it yeah that's my pet pet topic of npm for agents is like how do you build dependency graphs of agents and like how do you share them because why do I have to rebuild some of the smaller levels of what you built already I have a quick follow question on agents managing other agents it's a topic of a lot of research both from like Microsoft and even in startups what you've discovered best practice for let's say like a manager agent controlling a bunch of small agents that it's two-way communication I don't know is there should be a protocol format to be completely honest the the State we are at right now is creating the simple agents so we haven't even explored yet The Meta agents we know it's there we know it's going to be valuable we know it's going to be awesome but we're starting there because it's the simplest place to start and it's also what the market and distance if you go to a company random SAS B2B company not necessarily specialized in Ai and you take the an operational team and you tell them build some tooling for yourself they'll understand the small agents if you tell them build AO gbt they'll Auto what and I noticed that in your language you're very much focused on non-technical users you don't really mention API here you mention instruction instead of system prompt right that's very conscious yeah it's very conscious it's a mark ofall designer Ed who kind of pushed us to create a friendly product I was KNE deep into AI when I started obviously and my co-founder Gabrielle was uh was a strip as well uh we started a company gazer that got acquired by strip 15 years ago was at Allen a Healthcare company in in par after that it was a little bit less so kneep in AI but really focused on product and I didn't realize how important it is to make that technology not scary to end users it didn't feel scary to me but it was really seen by head our designer that it was feeling scary to the users and so we were very proactive and very deliberate about creating a brand that feels not too scary and creating a wording and language as you say that that really tried to to communicate the fact that it's going to be fine it's going to be easy you're going to make it and another big point that David had about adep is like we need to build on environment for like the agents to act and then if you have the environment you can simulate what they do how's that different when you're interacting with apis and you're kind of touching systems that you cannot really simulate like you know if you call the Salesforce API you're just calling it you know yep so I think that goes back to the DNA of the companies that are very different adep I think was a product company with a very strong research DNA and they were still doing research one of their goal was building a model and that's why they raised a large amount of money to Etc we are 100% deliberately Product Company we don't do research we don't train models we don't even run gpus we're using the models that exist and we try to push the product boundary as far as possible with the existing models so that creates an issue indeed so to answer your question when you're interacting in the real world well you cannot simulate so you cannot improve the models even interracting your improving your instructions is complicated for a builder the hope is that you can use models to evaluate the conversation so that you can get at least feedback and you could get conative information about the performance of your assistant but if you take actual trace of interaction of humans with those agents it is even for a human extremely hard to decide whether it was a productive interaction or really bad interaction you don't know why the person left you don't know if they left happy or not so being extremely extremely extremely pragmatic here it becomes a product issue we have to build a product that identifies user the end users to provide feedback so that as a first step person that is building the agent can iterate on it as a second step maybe later when we start training model and post training them Etc we can optimize around that for each of those companies yeah do you see in the future products offering kind of like a simulation environments the same way all SAS now kind of offers apis to build programmatically like in cyber security there are a lot of companies working on building simulative environments so that then you can use agents like red team but I haven't really seen that yeah no me neither uh that's a super interesting question I think it really going to depend on how much uh because you need to simulate to generate data you need to generate data to train models and the the question is at the end is are we going to be training models or are we just going to be using Frontier models as they are on that question I don't have a strong opinion it might be the case that will be training models because in all of those AI first products the model is so close to to the to the product surface that as you get big and you want to really own your product you're going to have to own the model as well owning the model doesn't mean doing the pre-training that would be crazy but at least having an internal post- trining realignment Loop makes a lot of sense and so if we see many companies going towards that over time then there might be incentives for this the sasses of the world to provide assistance in getting there but at the same time there's a tension because those s they don't want to be interacted by by assist they want by agents they want they want the human to click on the button so that's they got to sell seats yeah exactly exactly just a quick question on models I'm sure you've used many probably not just open the eye would you characterize some models is better than others do you use any open source models what have been the trends in models over the last two years we've seen over the past two years kind of a a bit of a race in between models and at uh at at times it's the open a model that is the best at times it's the anthropic models that is the best all take on that is that we are agnostic and we let our users pick the model oh they choose yeah so when you create an assistant your an agent you can uh you can just say oh I'm going to run it on GT4 or GT4 turbo or don't you think for the non-technical user that is actually an abstraction that you should you should take away from them we have a same default so we take we move the default to the latest model that is cool and we have the same default and it's actually not very visible in know flow to create an agent you you would have to go in advance and go pick your model so this is something that the technical person will will care about but that's something that obviously is a bit too complicated for the and do you care most about function calling or instruction following or something else I think we care most for function calling because you want to there's nothing worse than a function call including incorrect parameters or being a bit off because it just uh Drive the whole interaction off yeah so got the broccoli function calling board these days it's funny how the comparison between gp4 o and gp4 turbo is still up in the air on function calling I personally don't have proof but I know many people and I'm probably part of them to think that gp4 turbo is still better than gp4 on function calling wow we'll see what comes out of U the1 class if it ever gets function calling and clo 3 on fight son is great as well uh they kind of innovated in interesting way which was never quite publicized but it's that they have that kind of chain of thoughts step whenever you use a clo model or sunet model with function calling that chain of Step doesn't exist when you just interact with it just for for answering questions but when you use function coing you get that step and it it really helps getting better function coing Yeah we actually just recorded a podcast with the Berkeley team that runs that leader board this week so they just released V3 yeah uh it was V1 like two months ago then they V2 V3 turbo is on top turbo is on top turbo is over 40 and then the third place is xlam from Salesforce which is a large action model that been trying to popularize yeah mini is actually on here I think mini is number 11 but arguably mini has for that do you use leaderboards do you have your own evals I mean this is kind of intuitive right like using the older model is better I think most people just upgrade yeah what's the what's the eval process like it's funny because I I've been doing research for three years and we have bigger stuff to cook when you're deploying in a company one thing where we really spike is that when we manage to activate a company we have a crazy penetration the highest penetration we have is 88% daily active users within the entire employee of the company the kind of average uh penetration and activation we have in our current Enterprise customers is something like more like 60 to 70% weekly active so we basically have the entire company interacting with us and when you're there there is so many stuff that matters most than getting evals getting their best model because there is some of these places where you can create product or do stuff that will give you the 80% with the work you do whereas deciding if it's gp4 or gp4 Turbo or Etc and know it'll just give you the the 5% improvements but the reality is that you want to focus on the places where you can really change the direction or change the uh the interaction more drastically but that's something that we'll have to do eventually because we still want to be Ser because in some ways the the model labs are competing for you right you don't have to do any effort you just switch model and then it'll it'll grow what are you rate limited by is it additional sources it's not models right you're not really rate limited by quality of model right now we right now we are limited by yes the uh the infrastructure part which is avability of I mean ability to connect easily for users to all the data they need to do the the job they want to do because you maintain all your own stuff you know there there the companies out there there are starting to provide Integrations as a service right I used to work in an Integrations company yeah I know I know it's just that there is some in cases about how you Chun stuff and how you process information from one platform to the other if you look at the end of the spectrum you could think of you could say oh I'm going to support airb and airb hasir really sense the French Founders I know very well I'm seeing him today and the reality is that if you look at notion airb does the job of taking notion and putting it into in a structured way but that's a way that is not really usable to actually make it available to to model in a useful way yeah because you get all the blocks details Etc which is useful for many for data scientist not for AI the r of notion is get sometime you have a sometime you have so when you have a page there's a lot of structure in it and you want to you you want to capture the structure and shun the information in a way that respects that structure in notion you have databases sometime those databases are real tabulate data sometime those databases are full of text you want to get the distinction and understand that this database should be considered like text information whereas this other one is actually quantitative information and to really get a very high quality interaction with that piece of information I haven't found a solution that will work without us owning the connection n that's why I don't invest in these there's composeal there's um all hands from from Grim nuig there's all these other companies that are like we will do the Integrations for you you just we have the open source Community we we'll do off the shelf but then you are so specific in your needs that you want to own it yeah exactly you can talk to Michelle about you know he wants to put the AI in air bite you know I will I will cool what are we missing you know what are like the things that are like sneakily hard that you're tackling that maybe people don't even realize they're like really hard the real Parts as we we kind of touch base through the conversation is really building the infra that works for those agents because it's a tenous walk it's a an evergreen piece of work because you always have an extra integration that will be useful to non-negligible