Knowledge is Eventually Consistent // Devin Stein // MLOps Podcast #335

Key Takeaways

Um so code is like a really really good system of record. Um and maybe you can extend the analogy to other types of system records internally where you know they are truth and you can monitor changes on those and reflect those back into the knowledge base but unless you have like a digital system of record it's really hard to reconcile things. So there's still going to be meeting notes um and like other types of documents that might be floating around but uh certain types of knowledge like you know how does our product actually work like I think there should always be a very clear answer. [Music] Devon Stein uh CEO and founder of DOSU and drink my coffee black. Um, generally pourovers in the morning, but we have an espresso machine at the office, so also drink quite a bit of espresso. Let's talk about the fax agent. Can you break it down for me? What is it exactly before we go into like the details? >> Yeah. Uh, I mean, basically it's our so before we get into the details, you know, like I guess do a product we got our start um helping out with open source maintenance. Basically kind of the premise of DOSU was um as an engineer I actually don't spend that much time coding especially as I grew more senior in my career and as a open source maintainer like a lot of my time was spent answering questions and triaging issues. So I started DOSU to focus on like hey can we answer questions and triage issues like engineers can um by looking at code commits conversations and tickets like everything um kind of around the codebase and the product um kind of engineers are unique in organizations because code is truth like it really tells you how your product actually works and so whenever there's ambiguity you need to escalate something to an engineer um and so we kind of built DOSU kind of our initial agent to answer questions and triage issues whether they're like incoming in Slack um or on GitHub issues uh in the case of open source maintenance and Doge has been you know very popular within open source helping answer questions and triage incoming GitHub issues um and we just launched um kind of our second iteration of our agent we're calling our fact-based reasoning agent um and the premise is exactly this idea that like hey um users ask uh very related questions over and over again and do uh you know does an investigation every time um kind of starting from scratch like most agents do like hey okay we're going to now search the codebase we're going to look at recent PRs uh we're going to you know scroll through Slack um and but a lot of that work is kind of duplicated across different agent runs and so this new design um has it so as DOSU is doing research in a given uh run it's learning what we call facts which or like claims supported by evidence it found and it uses those facts when generating its response. Now if the response is correct which we get from you know either direct user feedback or from like a maintainer expert that jumps into the thread then those facts are committed to its knowledge base. Um and then next time someone asks a similar question about related topics to first take stock of like hey what do I know about these topics um in terms of the facts in my knowledge base do I have enough information to respond if not what am I missing and then it'll do additional research to find that information and respond and uh it's kind of this nice um sort of uh learning loop where uh the more you use the product the more facts it learns the better it gets and the faster it gets. >> Okay. So, the fax flywheel is fascinating to me and also the whole idea of how and when you get the agent to jump in is probably the most fascinating because how does it get solidified that a decision has been made? How does an agent know that a decision has been made and it's not just continuing a conversation or it needs to be picked up again when XYZ happens? like there's so many X factors and intangibles that feels like it's a very hard problem for an agent to solve or maybe not. >> Yes, in the general case I think it is very hard to have that human intuition of like when I should jump into this like when is my expertise needed. Um I think in the domain we work in it's usually who is asking the question like the audience actually matters a lot. If we have it's you know the kind of ticket or issue is coming from a maintainer they probably know generally about the problem and maybe like a good response would then be like pointers and like where in the codebase or recent works um or no response at all because they probably have it under control versus if a user is asking a question and they are maybe non-technical or less technical or new to the project then um usually help uh is like you know any information to help them go from where they are to kind of closer to where they want to be in terms of resolving or answering their question is welcome. Um, so I think audience plays like a pretty important role when thinking about like does an agent want to jump into this conversation. >> So do you have to have user profiles built up that the agent is aware of? >> Yes. uh we have kind of a pretty simple structure right now where we have basically experts who are users in the app, the curators of knowledge um and then uh normal users who are less familiar with the domain in the future. I think there's a lot of cool things we could do around um you know different types of audiences whether you're fully non-technical um and you can only understand things in product terms um versus you're an engineer that is just unfamiliar with this codebase but you can read and write code. >> Yes. And where does it triage items from? Do you plug into Jira? Do you plug into Confluent Notion like ClickUp? Is it all of that? because I know documentation and I was expressing this before we hit record. It can be such a pain and a lot of times it's a pain because it's just so dispersed. Yes. Um I think that's one of the uh really interesting things and sort of um about LLMs is that and humans right if you think about human memory um something you know engineers like our senior engineer at a company or an open source maintainer has been on a project for a while something that they do that is you know really really unique is they're able to see the activity across all these different apps see what's going on in Slack maybe in ClickUp in Jira um across PRs what has been merged what has been like you know what issues have come up and then actually connect the dots between these different um data sources um kind of see the connections in terms of topics how they relate together um and we try to take a similar approach um at DOSU around um basically trying to figure out you know what is the uh product or engineering ontology for your organization so you know what are the key concepts topics product features components on the engineering side and then how can relate the conversations that are happening across these disparate apps, the documentation that lives across these disparate apps back to those core concepts and start building the connections between them. It makes it easier for us to then, you know, when we're talking about a specific part of the product where that information might live across everything. This episode is brought to you by MLflow, the opensource platform trusted by teams worldwide to manage the entire ML and Gen AI life cycle. And with free managed MLflow on data bicks, you get all the benefits of MLflow plus automated infrastructure, unified experiment tracking, model versioning, observability, and enterprisegrade governance all in one place. Ship reliable models to production faster with less hassle. Get started at mlflow.org or check out the recent talk that we had from Eric Peter at RSF agentbuilder summit. mlflow.org links in the description. We had on here a few months ago Dona who created a data analyst agent and one of the hardest challenges that she talked about was how words don't mean the same thing always words are hard basically was the long and short of it and she used it in the context of if you're a data analyst you have all this jargon that you say and that you want you ask questions to an agent about. But those questions and those words, all that jargon, you have to explain to an agent what that means. Because and I always go back to the most simple answer that I can wrap my head around, which is an MQL at one company, a marketing qualified lead at one company can mean one thing at this company, but at the other company, you have to do five different things to become an MQL. So even though we're using the same word, it is not the same when the data analysts are asking questions of the LLM. And what they did to fix that is they essentially had to create a glossery of terms so that when a data analyst would use this jargon, the LLM could reference that and it could say, "Okay, cool. I understand what an MQL is at this company." How have you dealt with that? Because I can only imagine in your world it is multiplied times 100. Yeah, I mean we have a very similar approach where for topics they have aliases or synonyms um that you know also refer to as um it's kind of like the relationship. So you know we have a lot of examples of this. I think every company does you know where we call something uh one name in the front end but then the back end actually has a historic name that's completely different. Um and so we just go back and forth internally uh using either the front-end term or the backend term or another thing is like you know the word task. We have like uh three or four different terms for task in our backend codebase. U could be a celery task. It could be an LLM task. Uh it could be this background task. Um and you know how should we differentiate for given the same term? um you know what other what what is the meaning of this term in this context um is also like a related u and hard problem. Another thing that's kind of fun and relates back to like auto reply or like jumping into conversations is um like conversation implicature. Like what is the implied meaning of a a comment in Slack? Uh especially like you when you someone will post in Slack like you know how do we do this thing and you know what is we in this case? Is it like the team they're on? Is it uh you know the engineering team like of the channel? there's a lot of like implied meanings of like the way people converse in um shared channels when it's like you kind of know the do kind of domain of discourse like what people are generally talking about and having the LLM do the same sort of um you know figuring out what is the implied meaning what is this person uh actually looking for um is is fun and challenging >> it's so fuzzy isn't it it's just like yeah we who you calling we yeah right >> what do you mean by Yes. And how do you think about adding value but not being too noisy? Because an LLM is always going to say that it knows something and it's probably always going to want to jump into a conversation. But that actually adds cognitive load at the end of the day. If I have to read a one-page report for a simple question that I have and it really isn't getting to the essence of the question, I'm going to be pretty pissed at that tool. >> Yep. I think it's one of the hardest problems generally in the agent space. So, I think there's two pieces with that. One is uh conciseness is generally very hard with LLM. um partially the way I think they're both in pre-training and kind of fine-tuned um they tend to be more verbose than what a human counterpart would actually say. And so we've invested a lot in reducing uh verbosity like keeping responses concise because you're right like the it's one thing to be wrong and have a two sentence answer but to be wrong with like a two paragraph or page answer then it takes a lot of time and cognitive load to figure out hey the LM's not correct. Um, so I think the way you deliver information, um, we've thought a lot about it. I think we still have a long way to go in order to kind of be concise, be, um, you know, uh, point back to reference sources as well can help with that. So instead of like restating what exists in the kind of reference source or fact, kind of link back to it can help shorten things. The other side of it is, you know, when do LLMs know if they have uh, you know, do they know the answer? Um, are they confident enough to respond? Also like a open research problem. Um, I actually just saw a really interesting paper, I can't remember the name. It was something to do with abundance was in the title uh that came out of um fair at Meta yesterday where um they actually found that reasoning agents are worse at knowing that they don't know the answer. So they're because you know it's almost this overthinking problem where um they will even in their train of thought they might say they don't know but by the end of the time they finish thinking they're like I got this. Um >> they talk themselves into it. >> Yeah. Exactly. It's like yeah coaching them kind of like okay I can respond to this. Um and so um you know there's ways in which you know I think given a specific domain you can help figure out better like what is your confidence level. Um one is again like audience. So uh who is asking the question I think helps dictate quite a bit on like what is an appropriate response and like if they want help. Um and then the other side is uh kind of going back to the factbased agent is can you ensure that all these statements you make um are coming from facts and those facts have been confirmed before um either by a human expert or kind of uh you know from previous uh conversations. So the quality of your knowledge I think also dictates you know the quality of your responses. Let's put a pin in the quality of your knowledge because I want to dive into how you think that can improve over time. The last thing while we're on this topic of these these fact-based agents is do I need to give explicit consent or approval when a fact is immortalized so that it can go and then be rerouted and put into a knowledge base or is that just happening in the background and it's constantly updating? if there's been enough insinuating at it. Uh because that feels like something that could easily go off the wheels too, right? Like you just start going on this circular uh kind of false fake news type of vicious cycle and next thing you know you're like, "Wait, how did that become something that quote unquote we do here?" >> Yes. Uh this is something we think a lot about and actually have um I think it relates generally to like UI or I guess UX for agents in that when we started kind of rolling this out we actually said hey we're going to just generate facts based off all the data we ingest so no human loop fully automated and where we ended up is uh you know we would generate a lot of knowledge I think the knowledge is correct but you don't know um and so it is this sort of um scary middle ground of like maybe you're kind of propagating some uh like false claim and that's actually cascading failure. >> And so where we've ended up and I think like from a PL product philosophy just makes sense generally for agents is there's sort of like three stages to automation. Uh first is you know human the loop. So uh you know a like in the open source case a open source maintainer says you know like great response either by DOSU or they added a response and they're like I want to make sure that DOSU knows about this topic and it they can take an explicit action to save it to their knowledge base and so there's like you know the human is and they then see like a preview of what gets saved and they can edit it. So there's like you know very high quality knowledge that the human um you know took the initiative to say save say this to my knowledge base and reviewed it. >> It feels like that is you're asking a lot of someone to do that. >> Yes and no. I think what's unique somewhat about the domain we're in is that um uh usually do operates in like public forums. So whether that's like an internal slack or you know open source project where there's a lot of people asking questions or looking for help and then there are a few experts and there you know even if DOSU doesn't answer it like someone will generally you know it's like someone's going to respond to you in Slack hopefully and if you don't you'll keep pinging them and so we actually do usually get a resolution on all the threads that we're on whether it's dosu you know driven resolution or a human driven resolution and so it's like a lot less work to say, you know, a command to save this to the knowledge base than it is to like go and update your documentation, which is like kind of the the alternative. And there's also an incentive to do so because, you know, if you do this, then next time do is going to get that and you won't have to answer that same question. Um, so I think we're fortunate in like our domain that our users, our experts are very incentivized to make the product better. Um, and so that's the human in the loop modality. Then the next one is more um you know I would say maybe AI driven um it's or like kind of in the crawlwalk run. we're at like the walk stage where you know DOSU is reviewing threads and um you know whether it's a PR or like a conversation and it is saying like hey you said something that um I either one conflicts with what is in my knowledge base or um I don't have in my knowledge base but seems important which um and and then can actually reach out to that person either in the thread or in like a direct message or kind of in the app and say you know should I save this to my knowledge base based off what you suggested. So, it's kind of uh a little bit of a proactive um >> yeah, >> where you know, we're not saving it directly, but we're doing the work to kind of get everything ready in like a preview draft state and then you just have to approve it. >> So, it's it's like a sleeper agent that's there in the background recognizing. And how does it know when it seems important? Is that just like my use of uh like bad words and caps locks? Yeah, it's similar to you know a lot of it has been twoing um and I think we're still iterating again like we're kind of fortunate in that like the domain that is discussed is like very um it is well defined usually what you want to save to a knowledge base or to your right to your documentation it's like hey there was some investigation where this seemed like a lot of work a lot of back and forths or um a question that you know was very relevant to the domain of like how do I get started how do I trouble Shoot. So there's like a class of stuff that we're looking for that is likely what you want to put back into the knowledge base. >> Yeah. Nice. And so then what's >> Yeah, the third, you know, the fully automated is kind of where we originally thought we wanted to start, which is we need to get so confident and like we get enough approvals and all these drafts that we can uh within, you know, start generating knowledge where we're confident automatically. You don't have to be in the loop. And then maybe if we're not confident then that kind of get gets moved to a draft state for you to review. But generally it's like hands-off you know do is extracting knowledge for you generating previews when it needs your help. Um and uh and then you know important then is like actually maintaining that knowledge which is the second piece of the puzzle. So because of the rise of all the coding agents, how have you seen things change? Things are changing in a few different ways. So I think one is the like importance like ironically I think the importance of written knowledge is only increasing because uh unlike humans AI at least currently do not have good memories and so um you know it doesn't like you know there's the intern analogy is made often it's like oh you know agents like having a thousand interns or 100 interns but uh interns hopefully will get better over time. Um, agents are, you know, very very smart, but they are forgetful. You know, they typically like don't learn from doing in the same way. And so, uh, what we've seen from kind of our users and talking to customers is that agents do much much better when they have written knowledge to reference of like, you know, I'm working on billing. Like, where is billing? How does it work in this codebase? uh or like you know how should I think about billing conceptually for this product um and so they you know agents need more documentation and the also the format in which they consume them is different you know agents like unlike people uh can like read giant blobs of text and that's good docs for them you know it's like very information dense >> it doesn't have to be aesthetically pleasing um and also I think similarly like connecting like the best docs for agents connect product concepts back to the codebase. So it's not just talking about like billing in the abstract but it has references back to like where it's implemented in what directories. Um which is you know important for people but I think even more important for agents because they're often like operating on the codebase and they want to they don't want to like learn about a concept and then have to figure out where it lives in the codebase. They just want to know where to go in the codebase immediately. Um so that's one piece I would say. Uh the other side uh is that you know agents coding agents excel in uh smaller projects you know the vibe coding uh examples but at larger kind of at scale the it's kind of it can be very scary to use a coding agent because if you are an engineer and you're asked to make a change and you don't know how to make the change and then you ask an agent to do it it's you know really, you know, >> playing with fire. >> Yeah, you're playing with fire. And so kind of a prerequisite to making changes is generally understanding the system like what is the impact of your changes? And so that also is kind of where um documentation or just knowledge um becomes more important. But for humans just like understand like what is the impact of this change? Where should it change? So then you can actually like review um and be a sort of co-pilot for the coding agent. so incredible about the ways that they consume information because that lines up with everything. And it it's funny how making the link, so giving them almost like grounding them in code gives them a better chance of success. And then throwing as much information at them as possible really lines up with, yeah, like give all the context you can and they can sus out what is actually important for them. Now, this feels like a perfect segue into the quality of knowledge and how you're thinking about making knowledge higher quality in general because as I was mentioning before we hit record again was I love documentation. I'm a huge fan of trying to write. I think a writing helps you clarify your thoughts. It helps you get down the most important stuff that you want to then take forward. But what I've noticed is you have to constantly be updating docs as things change. And you also have to recognize that a lot of things are just going to not be relevant after a certain amount of time. and you kind of need to know which ones are relevant in those moments of time. So, how does that knowing that like if I look through my notion, which is where I keep all of my documentation for the community and all of that stuff, if I look through that, I'm not going to say like 80% is obsolete, but it's not it's definitely a majority is obsolete. And that's because like every podcast that we've had for the last 5 years, you know, I have a notion page on them. I'm not using those anymore, but they're there and maybe they can be referenced, maybe it can be something. Or there's like strategy documents that I've written up in 2022 and those are not relevant. Or like reflection documents. All of this stuff feels like it would muddy up the waters on documentation. And if you want the highest quality documentation for your business, how do you think about like keeping it high quality? So I think you're not alone in that. You know, 80% of your notion is stale. Uh I think that's probably the norm. You know, you know what's the the saying is like the instant you write docs, they're stale. And I think that's really true. I think the way we think about it and I think generally is like a good framing is you want to like lean on humans on for what they're good at and then AI for what they're good at and humans are really good about knowing what is actually important. Um so I think like you know writing is still a very useful and important exercise of like um you usually as like an expert on some do topic or having just done research like know what are the things that someone in the future need to know and sort of like maybe the story that you want to tell around it and you know AI can help you write but at the end of the day like you know your the nuggets of you know expertise or like knowledge that you as the expert like put into it like That's what's really really important to have the human in the loop. And so we focus on like what is like how do we make tools to make it really easy for you have something that you think is important. How do you get it down in a written format um you know save to a knowledge base or to your documentation as easily as possible? Because I think you know at least from my experience and I know from like a lot of other engineers is there's a lot of things I would like to document but I just don't have time. Um, and so I think part of the puzzle is really being making it easy like 10x 100x easier to get information out of experts. >> And then the second part of the story is like okay, how do we maintain it and how do we um, you know, what is what do you how do we know what is truth? the way we're approaching it and I think it's unique to our product and I think it's you know at least currently like it's easiest in the product engineering domain because you have code which is a source of truth that is very clear how it changes or when it changes really um and so the way we think I think there's like two types of I guess documentation or knowledge some of it is is more notes it's temporal in nature it's maybe a a meeting note uh you don't want to update a meeting note because it's a record of what happened, but there's pieces of that that maybe you do want reflected as like canonical, a source of truth. >> Um, and so we're very f focused on like uh the knowledge that you want to be source of truth. >> How do we keep that up to date? And you know, the nice thing is that code um is, you know, another source of truth that's like very clear what it is. And so as code changes, we can compare those to the state of your knowledge base um and try detect inconsistencies. So you know you have this in your you know source of truth but your code is now saying this or used to say this but now says that you should probably update these um sets of documents or these facts. So I think making the distinction between uh what what is information you want to last um versus something that is a kind of a record of the of a point in time I think is very important. >> That's so fascinating to try and think through that. Let's take a second to talk about our sponsors of today's episode. Hyperbolic GPU Cloud delivers Nvidia H100s at just $149 per hour. No sales calls, commitments, or surprise fees. Spin up one GPU or scale to thousands in minutes. With virtual machines and bare metal multi-node clusters featuring high-speed networking, attachable storage, and auto topups, you only pay for compute when you need it. Hyperbolic costs up to 75% less than legacy providers. You need longer runs? Well, get reserve clusters at predictable rates with instant quotes and fast onboarding. Try Hyperbolics H100 power ondemand. Try it now at app.hyperbolic.ai. And just so you know, the links in the description. Let's get back to the show. I've got a friend Willm who is working on like agents S sur agents right it feels like this is in some ways the first time that I've thought about wow both of these agents could be best friends like if the dou agent is talking with William's cleric agent and He's trying to root cause and doing more or less the same thing that you're doing, but with root cause analysis and trying to really figure out, wow, something went wrong or the whatever. Something is super saturated and servers are failing around the world or data dog is showing this blah blah blah and here's why. and then it can sync with those who and get maybe it it uses it for knowledge to help reference different things and and do its job better or when a root cause analysis is made and then something is committed it's thrown into DOSU. I wonder if you see that as a way that like the two agents will play together or do you feel like it is something that DOSU will eventually start doing? It's just not there yet. So, I know William as well. Uh, it's great. Shout out to Will. >> Shout out Will. Uh, so actually I think you're spot on. Like we want Dou to be other agents best friend. Um, so like you know DOSU can be a knowledge provider for cursor for cleric. Um because like you're saying um knowledge is really important for root cause investigations in the S sur world. Um and then importantly like you know you have run books. How do you make sure runbooks don't go out of out of sync and go stale? Because the worst thing you could do is give cleric like a runbook that isn't correct and then it's going off and doing something that it shouldn't be um or that used to be true and it's confused. Uh and so >> which is one of the things Yeah. I have I it was also a question that I had in my mind is how are you making sure that you're giving the most relevant information? Is it just that you're looking for the most recent? Because sometimes maybe the most recent isn't the thing that is the most relevant or I I guess it's maybe not ne necessary to say relevant because that's more of a search problem but if you want to update information how are you going about updating it and saying this is the source of truth now. >> Yeah. So I think the way we approach it. So there is a search element to it like kind of um but the I think the there's an interesting you know uh comparison between like search and documentation where you can almost think of documentation is as like a knowledge cache in in some ways where someone's done the searches they've compiled the information and then they've written it down so you don't have to do those searches again. Um and so the way we kind of think about it is when we are learning when do is like learning about a topic or you know that from book is associated with we are trying to keep you know that topic as like a source of truth document. So we're you know as you know conversations are happening we are like kind of reflecting those pieces back into the document and so as long as you can find the relevant document you can trust you know it's correct if that makes sense. So you don't have to go through the like work of searching through Slack or all these other documents to try to figure out like, okay, well this one happened like three months ago and they said this, but then this was one month ago, but then they said this again. >> Um, yeah, >> Ben, that feels like the headache. But sorry I cut you off from Okay, you're now other agents best friends and it makes complete sense like cursor does much better if it gets given I imagine these stuff that do can feed to it. Yeah, exactly. And I think there's interesting things where you know if you think about how agents are probably going to become like the majority consumer of documentation if they're not there already um they might be in the kind of coding domain where you know again like the format of docs like how you think about docs starts to change um the APIs you might want to expose also change um so I think like an interesting thing like I think a big challenge at least in like our notion or you know projects that I've worked on is information hierarchy is also like really hard to do in like in traditional documentation and knowledge bases but maybe don't matter as much in the world where you have like agents being the main consumers of information. Um and maybe there's a better experience that we can build around like how information is organized for um the more discovery kind of modality because that is when it's useful to see everything laid out is because you're trying to like figure things out. You're trying to learn maybe you don't have a question yet. Um >> it's fascinating. Yeah, information hierarchy is such a headache and I can't tell you how many docs I have lost because they were nested inside of like 10 other docs. Just this happened to me just the other day and I was so frustrated. I'm like where the hell is that database? I know it's around here somewhere and I couldn't find it and I used all the key search terms that I thought I would have called it, but since I haven't referenced it for like a year, I had no idea and and it was just like going off a feeling of like I think it was somewhere in here and then spent too much time on it and eventually gave up. >> Yeah. So I I think there's I mean I think there's some interesting experiences that I mean we haven't built but I think like you know I I would imagine like uh where if you think about like documentation there's two modalities for it really is like how we think there's the you want an answer and so you have a question you want an answer search chat is actually pretty good for that the other side is you are you know new to something and you're just trying to explore you know what is possible um how should I be thinking about things and that's where it's really nice to have that directory structurally kind of poke through. Um, and so I wonder if there's some more generative experiences there on the learning side to be built out of like, hey, maybe you can visualize your knowledge this way and you can be guided through it in a specific way. Um, that's even better than kind of like our traditional docs hierarchy we have today. Yeah, it just makes me think about those onboarding experiences being so much more custom and so much more tailored to the way that you like to learn and then you can just be getting exactly what you're talking about where I want to see it this way or I really want to know what are the main principles that we're working off of and you don't have to go and click through like the start here read me or watch this video or this loom that somebody put together a few months ago or years ago. >> Exactly. Well, what else do we want to hit on? Is there anything else that like um for you specifically is really top of mind? I think one piece that we I mean we touched a bit on like knowledge maintenance but I do think it's really like uh worth emphasizing that it's like something that humans just are it's like not a job for humans I think like maintaining knowledge is just the effort to do that to be able to like monitor all the different changes conversations happening in organization and reflect that in documentation is near impossible. I mean some organizations have technical writers that their job is to try to keep up and distill like what is important from all the activity and then make sure that's reflected. Um but even then it's very very hard to keep up. But for LLMs or AI like monitoring all these changes and then you know doing an analysis of like how this impacts uh your kind of current state of knowledge is a much more like straightforward routine operation. So it actually can um and I'm just like very excited about you know what are the implications of that of like when you can actually have knowledge that you trust um and that like things are always up to date. Um I don't know I think it's just like something that is like really built for machines that has only been possible as of recently. Well, it feels like a really lowhanging fruit that you probably already do is just giving executive summaries of here's the changes that happened last week or on whatever the cadence is that the person is looking for. Hey, here's everything that you should be in the loop on. And you can almost like subscribe to okay, I want to know all the stuff that's happening just so that I can have one eye on that and make sure if I can be valuable, I I can throw in my two cents too. >> Yeah. Yeah. Actually, that's an interesting one. Uh we haven't explored much is like the interplay like okay, I want to be aware of changes as an expert to know when I should also be involved as well. Um, yeah. >> Or as a sabotur, you can be like, "No, we're not doing that." Just total block on everything. But yeah, I think that that is a fascinating piece just like you know how you have the ability to watch when things happen, right? As there's repos or whatever, like I can just be lurking in the background and recognizing when things are going on. I also as I was thinking about that I was thinking through how specifically you're going about evouts because it feels like you can in a way get a lot of signal from people like if a PR is merged or if there's we talked through like all these different areas that you're touching on which can give you a ton of different signals and ways to evaluate if the agent is correct or it's it's doing what it needs to do. Have you found that certain signals should be weighted more than others? So, emails uh I mean always top of mind for us. I do think like you're saying we're pretty fortunate in that um unlike maybe you know chat GPT where it's a very one-on-one interaction because we're operating in public forums on repositories where PRs are merged there's usually truth um and that helps us get signal as to like how we're doing um when we make mistakes something that we've been doing recently that um I I think is kind of cool and uh is sort of dog fooding our product in how we are doing evals. And so we're sort of calling internally like living emails in that you know you know at the end of the day you know if we run the agent and it's someone has a question or an issue you've produced an answer or a document and um that question uh and the answer like is a piece of knowledge and so can we like save answers that you know from that users have asked and then try to detect like when our answers on the eval side has gone out of date. Um, and so that way we can kind of keep at instead of like having to before what we were doing is we were saving a version of every data set for that specific point in time where someone asked a question. Um, and that works, but it's a lot of like operational overhead. And wouldn't it be nice if that we could actually just save evals as they are today and only when relevant pieces of information change do we actually have to update that eval? Um, and so we're very early in this process, but I'm like I'm kind of excited about it because it reduces the friction of us collecting evals um like a ton and then we also get signal in the product side um as well. It's it's a great way to save it's like um it just reminds me of you know like the best engineers I know are lazy and this feels like that's like no shade to you. It's like the smart way to do it is just to think like how can I make sure that we don't have to do this all the time. How can we just take it's almost like are you thinking about it in a way of taking the delta of okay well this changed so we need to update our eval set just on that part. >> Yeah. Exactly. It's like if you think about an answer as like a a document or a knowledge artifact. Our job should be you know a user saves this question answer pair as like a thing to our knowledge base. um that is a good eval until you know that document goes stale and then can we update that eval or should we prune it from the knowledge base? >> How are you seeing folks want to deploy this inside of the companies? Uh, I imagine you get a lot of data on what kind of integrations they want and there's probably like us the 8020 principle where yeah, we got to have Jira, we got to have GitHub, and we got to have Slack. That's probably like the stack that I would imagine majority of folks run with. But do you see folks that want to have DOSU then deployed in their private cloud because documentation is it's like the special sauce in a way. Yes, security is top of mind for us. I mean we're so talk to compliant um but right now DOSU is um kind of a SAS product. uh we you know go to great lengths to make sure like customer data is partitioned multi-tenant um but and currently don't offer self-hosting um at least out of the box I think you know we're willing to work with customers on it because of the sensitivity of knowledge on the integration side I think that's you know it is a long tale um the way we think about it is like at the core of what we want is like we need code commits conversations and tickets and documentation is like the the key integrations And so Confluence, notion, um, Jira, linear, slack, teams, discord, github cover majority of people. But then there is a really long tale of information that lives in other places that people want access to. And something that we've been exploring is when does it make sense for like a uh data source to be something that we need a like formal integration with uh versus something where we can do um more like just in time authentication you know where um you know someone like gives DOSU a access token on their behalf to go and look at you know maybe the logs in this service um because uh or this private internal thing uh as as long as it's like O2 compliant. So, we're trying to figure out like maybe there's a a way for um you know firstparty kind of integrations but then longtail um you know people just authorizing dosu to just do just in time access. >> Yeah, it's funny how email is not in there at all. Like nobody's figuring out their documentation on email. >> It's true because email I mean maybe at some companies but most of the companies we work with like email is you know uh it's a lot more We really focus on like the product and engineering domain. And a lot of those conversations are happening kind of in shared channels or uh on like PR reviews less so kind of in more formal back and forth on emails. Yeah, it's it's like email is externally facing and all of the is the Slacks and the Discords or Teams or whatever it may be are internally facing and so you wouldn't you wouldn't expect email to be that. Um, I wanted to ask about it the and we don't have to like specifically call out glean, but we can talk about like different ways that knowledge sprawl happens. And we're talking about just knowledge kind of multiplying and how complex things get. Every time you add a new employee, every time you add a new service, every time you add a new feature, that knowledge sprawl just continues to happen. And it's not like anybody ever comes and says, "We have less documentation this year than last year." I think that's the same with data. Nobody ever comes and says, "Great news. Last year we only had or we had this data sprawl. Now we were managed to get it in order and we don't have as much data this year. You've got a vision for a flywheel of data that you just talked about. I think that there's other folks that are trying to go about it and say look data is messy. let's find the best ways to join the mess or give you access to the mess. You're taking a bit of a different approach. So, can you walk through that vision? Yeah, I think you know right now how do we deal with knowledge sprawl or like you know what happens as an organization grows and the answer ends up being something like search which is like we and that's what you see is people are building better and better tools for sifting through this knowledge and trying to you know let people reason about like what is truth or or you know agents to reason about what is truth. Um but I think you know if we kind of look forward into a world where you know you have companies that are started with um LLMs and AI first tools for knowledge management you can imagine where you have a store of knowledge that is um like source of truth that grows with you and uh you don't end up in the same situation where you know you have like six copies of a similar document and you're not really sure which one is truth and may you have to go ping the authors to say hey like did this is this still true. Um you can actually have a system that is doing that for you in real time like as you scale. Um and I think that this is true you know at least in our kind of view of the world like this can be true for like product and engineering knowledge or at least where you have a system of record. Um so code is like a really really good system of record. Um and maybe you can extend the analogy to other types of system records internally where you know they are truth and you can monitor changes on those and reflect those back into the knowledge base but unless you have like a digital system of record it's really hard to reconcile things. So there's still going to be meeting notes um and like other types of documents that might be floating around but uh certain types of knowledge like you know how does our product actually work like I think there should always be a very clear answer. Do you also think about the differences between internal and external documentation? >> Yes. Uh this comes up a lot. The I think internal there are some interesting differences today and I don't know how it's going to be in the future. So today you know I think the big difference between internal external docs is external docs is uh usually like a representation of your product. it should be very polished. Has your brand um you know style, tone, these little details really matter. Um versus internal knowledge, it's really about uh does it exist and is it correct? Um and can I find it? Uh and so and also like kind of the topics they cover are different. You know external docs uh are very focused on like kind of userf facing what users want to do with a product. internally you want to know all the messy stuff you know you want to know why uh you know why do we have this hacky code or why can't we handle this integration you know um or you know what what should I not do to break the product you know um those things are not going to be written publicly facing but are really important for internal collaboration and communication for people to do road mapping effectively um >> in the future yeah >> oh sorry to interrupt But this makes me think about one of those stories that you hear. It's one of the horror stories of like an engineer sit sifting through the documentation is trying to figure out how to optimize or refactor something and gets to this, you know, this function and it's like why do we need this function? We do not need this function. And there's only one comment above it that says do not ever change this function. you will deeply regret it. And they're like, "Whatever." They go and they say, "It actually doesn't do anything. I'm getting rid of it." And they delete the whole thing. And they come back and they realize they just crashed the whole product, whatever. And they recognize that it was them that put that comment five years ago or whatever. And they the it the real story is even funnier because I think the language they use is very colorful. And it's almost like wouldn't it be great if in those comments you can just have a link to the Slack conversation or the area that it was decided on in the actual documentation on how that was decided that that happened. Or maybe it's a link to the last outage of the person that tried to get rid of that function. Whatever. It's um just made me think of that. But sorry I cut you off. No, exactly. I think like that's the a great example of like, you know, uh machines can have better memory than humans. You know, I I could totally see myself doing something like that where, you know, made a change, everything broke, I was like, "Okay, never do this again." no links in the comments, the git blames buried so far deep you can't find it. Um, and then you, you know, just repeat the same thing and then all of a sudden all the memories come back to you and you're like, "Oh my god, I can't believe I did that again." Uh, and you know, uh, with LLMs like both making the connections for us and helping like link disparate, you know, conversations or reviews back to code and making that really really accessible. Um, hopefully we can avoid that. I also think there's a piece of um interesting like about sort of a knowledge continuity there too where you know it's that I think example is funny because it's same engineer doing the same thing like 5 years later but more often it's like that engineer leaves after four years and then there's a lot of engineers that are looking at that code being like can we delete it we don't know like you know Sarah left and no one knows what happens if this is deleted and someone's like I'm just going to try it uh and then you know the cycle continues. Uh, and I think that as we have better, we make it easier for people to get their knowledge kind of in a store and that store is kept up to date. I think you can have better sort of continuity as people come and go from companies and communities. Um, that can help them like be uh longer living uh kind of yeah over time. [Music]