Learn How Monday.com Built Their AI Agent Workforce with LangGraph | LangChain Interrupt

Key Takeaways

So great to meet you. I'm Assaf and today I'm going to talk to you about how we're building our digital workforce at monday.com. Um so very quickly about myself. So I'm the head of AI Monday. I'm a scar for SEOA. Um currently live in Israel, so I'm super jet-lagged right now. Uh and I've also been building AI products for the past decade including some you may know like GPT researcher and so monday.com um maybe some of you know a bit about them. Uh but basically we're a public company. We we basically build a work OS where you can manage and do all your work in one single platform rather your CRM dev service work management. Uh we've we've actually crossed $1 billion ARR just this year. And there's one important fact here that I think is worth noting and it's that um we're actually processing around 1 billion tasks per year. And I want I want you to just think about this for a second because when you think about 1 billion tasks, work tasks per year, just think about the opportunity for agents and AI that can actually take on those tasks. And this is a huge opportunity that we see at Monday when we think about AI. And we've actually launched our first AI feature around September last year. And we've seen insane hyperrowth. We've been growing 100% month over month with our AI usage. And just recently we launched our digital workforce. So when you think about what is our digital workforce, think about agents working around the clock, whether you're whether you're an SMB looking to scale up or an enterprise, imagine agents working within the Monday ecosystem on any given task you can think of. And what I'm going to show you today is very powerful lessons learned that we had in our experience building agents. And it was said earlier here today by the Harvey team and I think others that to build very successful agents, you have to focus on product and user experience. And we have a saying at Monday that the biggest barrier to adoption is trust. It's actually not technology. And I want to show you a few examples of things that we've learned. So when we think about autonomy, I think you know we're all uh engineers and we uh love to think about autonomous agents and you know agents doing everything around the clock. But actually the opposite is true. When we see how our users are using agents and what they think, imagine that every company, every user has a different risk appetite. And when you build AI agents, you should give users that control. And what we've learned by applying this is that we've actually increased adoption insane in insane way by giving the control in the user's hand to decide how they want to control their agents. Secondly is entry points. Now, if you're building a starter from scratch, that's something else. But as a huge company like Monday, one thing that we've learned is don't rebuild a new user experience. Try to think how you can create this experiences within your existing products. So when you think about how agents can work at Monday, we already have people working at Monday on Monday. We just assign people. So we can do the same with agents. Just think about how you can just assign digital workers or agents to actual tasks. And by doing that, our users have no new habits that they have to learn. It's seamless within their experience. Another super important thing that we've learned. So originally when we set when we released those agents, imagine that you can ask in the chat and say things you know like create this board, create this project, modify this item for our users. Monday boards are production data. I think a very good example I like to give is think about cursor AI which is an amazing product. We all vibe code as Andrew and G said earlier. Um but imagine that with cursor AI instead of you as developers seeing the code imagine it was pushed straight to production. I assume that none of you or maybe most of you would have not used it, right? And and that is just how important user experience is versus technology because techn technologically wise you we could do that. Cursor could have done that. And what we did is that we saw users on boarding testing them out and once came the time to actually push content to the board that's where they froze. So we introduced a preview and this preview increased adoption by insane because users now have the confidence that they know what's going they have this guard rail before they actually save and they know what's going to be the outputs before they see it saved. So when you think about building uh AI experiences, think about previews, think about human loop, think about how users can have that control and understanding before AI releases to production. And lastly is explanability. Now explanability we've heard a lot and it feels I think from when I talk with with people it sound it kind of feels like a nice to have but explanability is much more than that. Think about explanability as a way for your users to learn how to improve their experience with the AI over time because when they have an understanding of why the outputs happened, they have an ability to change the outcomes. So those these four are super important components that we've actually introduced in our product that have increase adoption very very nicely. Now let's talk about the tech. So we actually built our entire ecosystem of our agents on Langraph and Langmith and we've tested out various frameworks and we found Lingraph to be the number one by far and just a few examples. So what's great about Lingraph is that it's not really opinionated but it still does everything you don't want to deal with as an engineer. like interrupts and checkpoints, persistent memory, human in the loop. Those are critical components that we don't want to deal with, but we have that. On the other hand, we have super customiz super great options to customize it just for what we need. And we'll show you an example in one second. And additionally, native integration, we now process millions of requests per per month using Langraph. and it's proven to be super scalable. So let's take a look at how this is behind the hood. So we have langraph as the center of everything we're building and around around our langraph engine which uses also langraph lang for monitoring. We also have what we've built as what we call AI blocks which is basically internal AI actions that we've developed at Monday. We've actually built our own eval evaluation framework because we believe that evaluation is one of the most important aspects when you're building AI and I think today was a lot about evaluations as you've guys seen. So I'm not going to dive into that. And then we also have our AI gateway which is our way of preserving what kind of inputs and outputs are enabled in the system. Now let's take an example of our first digital worker that we released which is the Monday expert. So basically what you see here is a conversational agent using the supervisor methodology that are our the system holds four different agents. We have the supervisor, we have the data retrieval agent which is in charge of retrieving all data across Monday. For example, knowledge base, board data, we also use web search. Then we have our board actions agent that does actual actions on Monday. And lastly, we have the answer composer that based on the user, the past conversations, tone of voice, and all kind of other parameters that are defined by the Monday user actually composes the final answer. And we've even added a really awesome tool that we've learned which is called undo. So basically we give the supervisor the ability to to dynamically decide what to undo within the actions based on the user feedback which is by the way proven to be one of the coolest um use cases for building. And I want to share a bit of our lessons learned as we built this agent and what we're seeing. So when you're building conversational agents, assume that 99% of user interactions you're not going to know how to handle. And it's proven statistically, right? When you think about the infinite amount of things users can ask, probably you've only handled 1%. And for this, we learned to start with a fallback. What happens in the 99% of interactions that we don't know how to handle? So for example, what we did was if we detect and the user is asking some action that we don't know how to handle, we would search our knowledge base and give them an an answer for how they can do it themselves. This is an example of one way of resolving fallbacks. We've talked so much today so I'm not going to really dive into it but I think the bottom line with eval is that eval are your IP because models change technology is going to change so much over the next few years but if you have very strong evaluation that is your IP that will allow you to move much faster than your competitors human the loop critical we talked about this a lot at the beginning I'm think I'm for those who have really shipped AI to production, I think you've seen that it's one thing to bring AI to uh 80%. But then it takes another year to get to 99 99%. And this is a very important rule because we really felt confident when we were working locally and once we shipped to production, we realized how far we are from an actual product. Um I can see some of the audience resonate with me on that one. um all guardrails we highly recommend that you build outside the LLM right we've seen things like LLM as a judge even back to the cursor idea by the way I think cursor is such a great example for for a way to build a good product experience because I don't know if you've guys especially with VIP coding after 25 runs it stops right this is an external guard they put in no matter if it's actually running successful 25 runs and it stops. So just think about how you can create those guardrails outside the LM. And then lastly, and this is a very interesting one, is that it might be obvious that it's smart to break your agent into sub agents, right? Obviously, when you have specialized agents, they work better. But what we've learned is that there is a very important balance because when you have too many agents, what happens is what we'd like to call compound hallucination. So basically, it's a mathematic problem, right? 90% accuracy times 90% accuracy. Second agent times a third times a fourth. Even if they're all at 90%, you're now at 70%. So, and it's a mathematical, it's proven mathematically, right? So, I think there's a very strong balance between how much of agents you want in your multi- aent system versus having too much or too little. And it's something that I think there's no like rule of thumb. It's something you have to iterate on based on your use case. So, let's talk about the future of work. And we believe that the future of work as what we're working on at Monday is all about orchestration. And I want to give you an example. So this is a real use case that we try to work on internally. We just had our earnings report just a just a few days ago. And for those of you working in large public companies, you you probably or if you've been involved in this earning reports, it's a tedious process. There is so much data, narrative, information across the company that you have to gather and so many people involved. So we said, what if we automated this? What if we had a way to automate and create an entire workflow that would automatically create everything we need for earnings? That would be a dream, right? But there's one problem with this. And the problem is that it will only run once a quarter. We invest the entire month building an amazing workflow and then within a and then we run it once and the next time we run AI is going to change dramatically. New models are going to come out. Everything's going to change in the world and then we're going to have to rebuild everything. Right? So it got us thinking about how can we solve this? So I want you to imagine what if there was a finite set of agents that could do infinite amount of tasks. Now the irony is that this is not some big dream. This is exactly how we work as humans, right? When you think about us, we each have our specialized skills. Some are engineers, some are data analysts. And then every time there is a task at work, some of us do A and some of us do B. So there's no reason why they shouldn't work with agents and AI. So when we think about the future, we think about what you see here. Imagine that for the same given task that we had and I showed you earlier, we had a dynamic way to orchestrate and create dynamic workflow with dynamic edges and dynamic rules choosing dynamic very specific agents that are perfect for the task, run the task and then dynamically dissolve. So, this is super exciting and one of the things that we're working on with Linkchain and we really want to see this come to life in the future. So, lastly, we're actually opening our marketplace of agents to all of you and we'd love to see you join the weight list and join us in building and trying to tackle those 1 billion tasks that we are trying to uh complete. So, thank you very much everyone. This was a pleasure. Thank you.