MLflow Leading Open Source

Key Takeaways

All right, folks. I'm super excited for this conversation that I just had with Jules, Danny, and Corey. They are on the MLFlow team. They are veterans in this space. Jules literally wrote the book on Spark. Now he is leading the developer relations side of MLflow and Danny and Corey are both lead engineers crafting what we know and love about this classic tool. and they talked about how MLFlow is leading with opensource. They really want to go allin on the open- source ML flow and bring it to this new agent paradigm that we're in. Enough of me chatting. Let's get into the conversation with it. >> Not every day that I get to hang out with so many incredible people. I am graced by not one, not two, but three of the ML flow team. How y'all doing today? I'm super stoked to chat about agents in production. You guys have been doing a lot with it and I think each one of you individually has been on this podcast before, just in different phases of your lives. It's cool to see you all working on MLflow and doing things in this new agent era. >> Absolutely. No, it it I think all of us have come along way and at the same time some of us are continuing to uh focus on developer experience for AI in a radically different domain. Models to agents, feature stores to agents. It's kind of exciting. So, thanks for having us back. Yes, we all we all we we all we all will come a long way from from Spark to MLflow uh and back to Spark and back to MLflow. So, I'm delighted to be working again with Danny and with Corey and obviously talking to you, we have had many interactions. I just love you, me serenating on on on the stage with your with your ukulele. That was that was the highlight. Absolutely. That was the highlight. Um >> yeah. Well, I feel like a lot of this has reflected just with it's not just like with like what the three of us have been doing. It's also like the ML ops community has like been shifting as well. Like we've we kind of all were working in various versions of the the classic ML world before and then kind of all shifted over to this like AI agent kind of ops space, right? So yeah, it evolved real quick. And so speaking of evolving, like MLflow has evolved a ton. You guys have been seeing folks that are putting their agents into production. I would love to start out with just what have you been seeing out there? Each one of you comes at it from a different perspective. So maybe we can get your takes on the conversations you're having each day. >> Yeah, 100%. I I think as we talk about agents in production, the sort of first natural question to ask is what are people trying to put in production? What are they building and and what are they deploying? And after talking to a lot of our customers, open source users can pretty confidently declare that uh the age of the chatbot is is still here. It ain't going anywhere. that uh the the bulk of our customers, the bulk of our open source users, they're building chat bots. Yes. You know, some of them that started as textonly bots a couple years ago have become multimodal. We do get the occasional like voice application. Folks are doing speech to text and then they get the text back and they go text to speech. Some are processing images but at the end of the day like that chat interface remains pretty ubiquitous like well over uh you know 50 or 60% of use cases which is is massive. Uh the other thing that I'd say about chat bots and there there are a bunch of other use cases that I'll let Danny and others talk about uh is that they're pretty much all tool calling chat bots now. You know we we had the age of rag uh up until about a year ago. Everybody was really excited about document retrieval specifically and now yeah folks are continuing to retrieve documents. Uh they're they're gathering information from knowledge bases. They're indexing them but they're doing so much more around that as well. You know they're they're making real-time API calls to other service other services to fetch information. Um they're even having their chat bots take various actions. You I'm going to go ahead and save some information about this user uh to a knowledge base. So there's not just read there's a little bit of writes coming in as well. Um so this chatbot interface has remained somewhat consistent maybe has become multimodal but then under the hood the architecture has has evolved quite a bit thanks to the powerful models uh that seem to be being released every week or two now. Um I think that's what you can probably attribute to a lot of the upgrades behind the scenes. Uh so that that's my pitch on uh on chatbot use cases. Um, Danny Jules, what else are we seeing? >> Yeah, well, just to elaborate on that, I think chatbots have what I've seen is chatbots are getting more and more sophisticated. So, maybe initially was just like text based chat chat bots. Now, we're seeing like they're there's a lot more like rich kind of return types like maybe like rich HTML that's being returned or like more human in the loops or within this agent. And now you're adding like kind of multi- aent systems or um but it's still like a chat interface to the user or there's uh basically yeah the infrastructure underneath people are building like more complex like guard rails within the agents you know uh just to as they get closer and closer to production or working with higher stakes use cases. I think people are now just beginning to get some of those like oh you can't just willy-nilly launch something to production kind of use case into production now. So so that's been cool to see. Um, and uh, >> yeah, I've seen, >> let me jump in there, Danny, because >> when you talk about the advanced, the chatbot is a little bit more advanced. I don't know if you guys saw this website called AI UX design patterns. It is brilliant. It gives you all these different ways that folks are leveling up chat. And right now a lot of them are just kind of concepts but you have things like you can highlight certain parts of the response in a chat and then expand on only that part you know which is so nice as a user experience because a lot of times you want to say like oh yeah I like all of this but I don't like that part or I've also seen designs where they'll come back with a carousel of items for you that you can scroll through like we're natively used to using as opposed to only chat in chat out that type of thing. But yeah, you were going to say something else, Danny, so hit me with what else you were [clears throat] mentioning. >> Well, that kind of reminds me of a different thing that I'll touch on as well. But um I was going to say even though all the chatbots are getting more and more complex, it's surprising to me how uh nent the observability and eval space has been for for these chatbots. Uh like you'll notice that the current this course is a lot about like multi-turn evaluation and simulation how like basically every vendor or solution still is like really just kind of beginning to get their feet wet. But like multi-turns like conversations in chatbots are naturally multi-turns. like you know why hasn't there been kind of more robust tooling in that space and really a lot of customers have had to just work around that and build their own tooling that's like super bespoke. Um so yeah that that's been interesting. Yeah, I mean I mean talk talking to both what Corey and and Danny say that we actually begin to see according to my conversations having a lot of developers out there the chatboard is seems to be the favorite one and the and and as Danny pointed out more of these evaluation judges are being written for example deep eval and raas do provide very specific board builtin as well as custom judges you can actually do multi-turn evaluations so we begin to see more of that in that particular uh area where chat ports are being written but they begin to feel the confident that they can actually write these judges to evaluate the multi-turn sessions to evaluate the entire conversation and that somehow seems to be the the trend at least from what I've seen from developers uh trying to ask questions or or hit me with you know how are how are you actually evaluating your chatboard in production I'm using my I'm using either ML flow I'm using some framework that actually has a host of built-in uh judges that allow me to uh um gauge a particular metric or I can write my custom judges along with agent as a judge. So we we begin to see quite a bit of that trend. >> Yeah. But um yeah I I think it people are beginning to talk about multi-turn judges now but not bec not because like they didn't need it like I think as soon as we launched multi-turn judges or we started talking about it people were like yeah I need that. That's like the first thing I want to start with. Uh it's like you know I want to know is my chap is are humans like escalating to get like you know people trying to get escalate to get human support are they uh is the agents repeating it over and over again like all these like common like Asian problems you really start seeing in the context of a conversation rather than in a single turn and really as soon as we launched it people were like cool yeah we're going to use that immediately we needed that yesterday >> yeah an example of this I like to use a fair bit is uh yeah determining whether an agent is being repetitive. I can't figure that out just by looking at one input output. No clue. But if I can see a, you know, 20 turn conversation, my god, this agent keeps saying the same thing. It's giving me the same answer. I ask for more information and, uh, it continues to respond with basically the same stuff rephrased. It becomes a lot more obvious. Uh, and so that's where, you know, I I can't figure that out. If I have this little function, this thing called a judge that is is looking turn by turn. I have to have the bigger picture. Same thing comes with human feedback though. Uh that you know a lot of times uh folks rightfully so are using their end users or internal testers and whatnot to determine whether a chatbot meets their needs and they need that uh conversational feedback. Indicating that this particular message didn't make a ton of sense is is yeah kind of helpful. uh but it it's much more useful if I can say you know this entire interaction uh should have been half as long or you know I I got this is what I was trying to achieve uh with this interaction and this is where I got and what was missing. So I think there's an element of human feedback. There's there are these questions about how I automate the process of finding issues in these uh multi-turn interactions and bugs in my agent. And it's funny, it's taken us long this long to get here and that everything started with sort of single turn input output because this chatbot interface especially this multi-turn thing uh was the first thing launched you know November of uh of what three years ago now uh you know chat GPT was a multi-turn chat interface and so it's kind of funny that we've finally started to catch up in terms of uh the evaluation on the quality measurement side >> kind of you touched you touched on two very salient point one is the repetition Right? How do I know that if I had if I had a 20 turn conversation, something was not being repeated? And the third was the logical consistency. Is is there a logical flow between each and every steps or am I being consistent or I being contradictory? And I think those are the important elements where where the judges can actually help to detect in the multi-turn conversation. One, the coherence. Is there a logical flow? B is the context being retention. I referred to something you know 20 turns before. is my agent or is my conversation board has has the ability to remember that in the context. So I think th those are the important thing and that actually collectively those make the the chatbot more robust and and and and more cohesive. Yeah. So you you touched on that important point. >> Yeah. Because you kind of take it for granted us in this field that we understand oh potentially a lot of times to get the best result we should just start a new chat and clear the context window. But that's not like I would guess that your average user off the street doesn't know that. And so how can you make sure that what you're talking about Jules that stays true? Like if somebody wants to reference back and you're not doing it properly, you know that and you have a way to either be the janitor of that context or encourage them to start a new chat or do something because that just leads to poor user experience. >> Yeah, 100%. And I think it dovetales nicely with this whole uh topic of memory management for agents that that everybody seems very rightfully excited about. The idea that uh a very long message history gives an agent a ton of rich context. You know, if if I've been talking with a chatbot for 15 minutes and then I ask it to do something that follows naturally from that 15 minutes of conversation, I'm much more likely, I would guess, to to get the result that I'm looking for. At the same time, there is that trap door pitfall that I think you rightly point out where if I'm asking for something totally different, uh, you know, that 15 minutes of context can hurt more than it helps. Uh, and so to me, I view it as a problem of memory and state management in an agent across all the conversations that I've had with this particular chatbot. When I go and I ask for something, whether it's in the middle of an active conversation or a brand new one, what information about me or about my intense or previous uh interactions is this agent drawing from? uh and that's something that uh it we're taking a pretty close look at at data bricks and from a research perspective as well and and I think it's an important topic for the industry at large just agent memory. >> Yeah. Yeah. And how do you weight memories? >> Exactly. So, Cory, Cory, you're referring to the short-term memory and long-term memory where if the 15-minute conversation uh has has has um trespassed and nothing has been eluted with further conversation, that 15-minute turn can be actually put in long-term memory and can be retrieved on demand, right? And the short short-term memory is just like, okay, I'm going to flush this out and I'm going to start a new session, but I'll keep that in short-term memory. If there is a reference to certain thing that I talked about previously that doesn't exist in my short-term memory, I'm going to go and refer to the long-term memory to see if the context is there. And that way, as you said, it it maintains a good flow and you as a user has a better experience to say, okay, you know, this guy remembers everything for the last half an hour than I actually had. So, it's like a human really be paying attention to you and and referring to something that you had asked about 15 to 20 minutes half an hour ago. That is you and that is like a moment. Oh wow. This is this is brilliant. I can't I can't believe half an hour half an hour ago I asked it to do something and I referred it to it and now it actually comes back and gives me a full answer. That is definitely aha moment. >> Magical. I'll tell you what though, when I encounter chat bots in the wild and they don't work that well, I am very adamant caps locks and everything saying how bad this damn bot is and I just like am trying to flag it so whoever is doing eval like oh maybe we should try and make it better in this area. Yeah, it's funny because uh one customer that we worked with uh they were kind of like they launched a production without really doing too much robust evo and like at some point they were looking at all their like responses. It's just like like I think most of their chats were just people swearing at the chatbot and so really it was like trying to serve a different purpose but ended up just being a therapy chatbot for all of all their users. But but interesting Danny when you when you become abusive to the chatborn if you have a judge that actually that that that has the frustration like for example in MLflow we have the built-in evaluation called user frustration right and so it detects it detects your language and tonality and based on that it would actually respond sometimes it surprises you when you know you were frustrated about this particular topic and you you would come back with a nice polite answer to mitigate the frustration I think that's an interesting way how how chatboard codes can somehow appear like human interaction because of the fact that we actually have this all both built in as well as custom judges you can actually write and you can specify and tailor it to your to your chatbot right if the chatbot is talking about insects then you everything dealing with insects would be sort of more tailored to that. Uh the other piece of this uh beyond yelling at the chatbot is the degree to which folks are collecting like thumbs up, thumbs down, categorical uh and and free form feedback from their chat bots. I I noticed that a lot of folks the first time they deploy to production haven't really thought about collecting those signals as much as you might expect. it as a result, you know, like Demetrios, you resort to caps locking and yelling at uh an agent in the chat window. Um, but it's often high signal for developers if they can get a more structured set of feedback. If they can get thumbs up, thumbs down, or they can get a one to five rating and they can they can carve out a separate uh text window or field for an explanation about why things are going well or where that they or where they can improve. I think a lot of folks treat that as an afterthought. Uh, and it's a little bit surprising, at least to me, because it's pretty fundamental to a figuring out if you deployed something that's working reasonably well and then b automating that process of uh fixing and identifying quality issues in the future. Like it's hard to sort of automate the detection of a real quality problem uh without having some early signal from your users about what the quality problems are. Uh so the the upshot here would be feedback collection is massively important and to some extent underinvested in uh in the the chatbot ecosystem and the interesting thing is it's very easy to build you know all all the stuff that folks are doing around stitching together uh you know tool calling chat bots with knowledge databases and external APIs and putting that an awesome web front end storing data securely all that stuff is hard you collecting a little bit of end user feedback is pretty straight straightforward and I'll shamelessly, you know, plug the uh MLflow feedback collection APIs as being a particularly low friction and and easy product surface to use. Um, but the the point is like spend that extra hour. That's that's really all it should take to wire up some end user feedback uh to and understand the quality of your agents. >> Cory, you you you touch on a very important point. How do we actually take the human feedback and what and and operationalize it? From your, you know, from your uh experience talking to customers, even developers, how do you think they're actually doing it? So, when they have the emo flow deployed and and they're going through the UI to add additional feedback, how does it get injected into into the uh into the traces so they can improve that? >> Yeah, it's a great question. So what you have with most deployed agents where you've integrated some amount of tracing is effectively this massive log statements. Uh you whether you collected them with ML flow or a variety of other solutions. Uh you you have thousands, hundreds of thousands, even millions of these timestamped logs. You can open them up and you can see the statement bystatement stack trace. What feedback helps you do is filter those. So I want to find the needles in this hay stack. Most of my hundreds of thousands of interactions aren't necessarily going to be labeled. The quality might be reasonably good if folks didn't take the time to express their frustration or, you know, provide some feedback. Uh, and so what I would like to quickly do is ignore all the stuff that I know is working well and I don't have to look at closely. I want to you click in this UI, apply a filter for certain types of negative feedback. You know, for example, user frustration is uh is high. Okay, I'm going to apply that filter. Or the completeness of whatever task the agent was being asked to perform is is low. It's only providing partial information or not fully meeting the user's objective. That condenses this log of thousands and hundreds of thousands of uh of stack traces into a few things that I can then analyze more deeply and start to root cause. uh you know it's great to understand that the user was frustrated why well then now I have to actually take a look at those particular log statements uh then in the future once I've uncovered that root cause and I've started to implement a fix I don't want to have to go back to my end user and say is this better now uh I should be able to like take their their feedback and turn it into some sort of unit test and that's where these like eval you know automated uh techniques really help uh so that that's sort of the pipeline that I see it starts with the feedback collection, it proceeds to filtering. There is some human debugging and then you're fixing with these automated unit tests that you can build. And does those unit tests somehow help us to improve the the prompt as well, right? Not to not not to get repeat the same rut again, not not to get into the same frustration loop and all that. >> Yeah, I think there's a variety of of solutions out there. So, prompt optimization is certainly one. prompts uh and still remain I think a major determining factor of agent quality. Two or three years ago everybody was talking about hiring professional prompt engineers. A year after that it was well you know this prompt engineering thing is brittle. We should look at fine-tuning. Uh we you there are other techniques that uh that may serve us better and get us further. It seems like prompt engineering though has sort of remained front and center for a lot of agent developers. So you're absolutely right. In the process of fixing an issue, I may want to do a fair bit of prompt engineering to get the agent to make better decisions. Uh retrieve information from a knowledge source more proactively, issue queries uh against a web service somewhat differently and beyond or or take on a different persona. Uh these are all valid use cases for prompt engineering. And so something that we've been investing in quite a bit in ML flow is making that quick and and effective. I've personally spent uh days prompt engineering uh voice assistant that I build at home and it can be agonizing. uh wouldn't it be great if there was some automated technique that I could use uh to to make that process a little bit more efficient and that's where you know the research uh folks at data bricks but also just more broadly in academia have been pushing forward this concept of prompt optimization so given the collection of negative feedback given some information about what isn't working in an agent and given a set of prompts how can I come up with better uh versions of those prompts that uh that mitigate and fix those issues and and can I treat this basically like a uh optimization job and an automated uh automated solution rather than this human in the loop uh English instruction modification thing where you know I don't know if I'm making deterministic progress and I might be spending like hours or days coming up with better versions of my prompt. you're talking about that's where the gap optimization comes into play where you can actually optimize your prompt engineering using the the the gap optimization as part of the MLflow library. So one of the things as you mentioned that when you actually want to automate that the prompt engineering part would be okay here's a here here's a set of prompts uh run it through the gap run it through the gap optimization to get the best prompt to see how how how you can actually do that. One thing that I find interesting when it comes to trying to optimize your prompts from when you're out there and you're trying to add all these edge cases, you can quickly get into a place where you've got this massive prompt because you've added all these edge cases in and then you end up destroying your usability of the agent since you tried to cover for all this. And I I know I've seen folks that will focus on like dynamically adding in or injecting in the context. And that's why we have all heard, you know, the rise of the context engineer because you got to throw the right context in at the right time. And you really want to have not more than you need. It's if it's anything, it's like just enough but not too much certainly. And and if you're not careful, like you said, you end up rewriting a rule-based system, except this time you're not writing it in Python or TypeScript or or Java or whatever. You're writing it in English. Uh and you know that that effectively uh becomes lower quality and more brittle almost than just writing it in code. So you're 100% right and that's where some of these optimization techniques and optimizer like GPA uh frameworks like DSPI that we try to integrate closely with and ML flow can help you avoid that because they're you have to force them to generalize across a whole bunch of samples at once. So you can't necessarily overfit sample by sample by sample. ultimately, you know, you're that optimizer is not going to be able to produce good quality that way across thousands of of test cases. So that that's certainly one way to arrive at a prompt that a fixes your problem, but b avoids that overfitting and and sort of rule-based uh prompting that can quickly really get out of hands. I that really resonates. Absolutely. >> Yeah. I I don't know if we've seen our own kind of like customers hit the case where they're just like adding so much stuff into the prompt that it's like really polluting the the conversation. Uh I think they they're still more in the state of we've thrown something in there and we're still trying to figure out like how good is the the agent in general. But um yeah, also I feel like people have stopped short of that because they are cost conscious as well. Like they don't want to just blow their LLM kind of token budget and that is something they're tracking very closely. So, you don't want to construct this like massive prompts that just continuously sucks all your budget every single turn. >> And you were you were talking about something before, Gory, that I think is fascinating when it comes to the feedback collection and how like you normally need to bring in the subject matter experts for that type of a thing. When you get that feedback, it's great from users if they raise their hand and they're kind enough to take the time and tell you what's wrong with it. But a lot of times when you're looking at these multi-turn conversations, you're just going to bring in a subject matter expert and say like, is it doing what we want it to be doing? And I've heard a huge complaint from folks about how subject matter experts don't want to be looking at like JSON. They in fact they can't really be doing that. So you have to figure out how can I give these folks the data in a way that they can properly annotate it and add value to it that isn't just like logs dump 100%. Absolutely. And that's something that we've spent quite a bit of time at data bricks and in the MLflow open source community investing in attempting to address that challenge of as a domain expert how do I quickly like recreate that same view that that end user had overlay some extra information that helps the domain expert you get to the bottom of what went wrong uh and then very quickly uh provide their expertise and So that's a matter of if I have a chatbot out there and user said, "Hey, this really isn't going well. You know, I'm Demetrios. I'm yelling at the thing. I'm frustrated." Uh, whatever. I I as I then call in my domain expert, Danny or Jules, and they're reviewing this interaction. They're trying to figure out, you know, whether this agent uh that's, you know, designed to answer questions about like telecom customers or something uh is you what what is it that is making Demetrio so frustrated? uh they should be able to replay that exact chat. They they should have access to the turnbyturn chat first and foremost, but then oftentimes that isn't 100% uh sufficient. They should also be able to see, okay, this particular turn of the chat, the agent went and accessed information. Where did it access? Okay, it accessed uh information about the user's account rather than about like the the line of products. Demetrios was trying to get a new phone. uh he's not asking about his account or billing related questions. So if I can give Danny and Jules that information in a UI and they can look through it and very quickly go ah okay this thing is fetching the wrong info. Uh then that's very it's very easy for them to come back and say all right I found the source of Demetrios's frustration. It was you know the sort of knowledge retrieval problem. And so at Data Bricks, we've invested quite a bit of time building this uh review application where I can collect a bunch of these chat sessions and I can send them over to domain experts in my organization. They get back that full chat view with the overlay of what information was retrieved. That way they can dive deeper and they get this awesome feedback form where they can provide root cause analysis. Uh which then helps the developer go back and fix their agent. uh they can then send that review app to any other member of their organization and uh we've been really invested and actively working on bringing this into the open source community as well. The vision is that uh anybody running open source ML flow should be able to spin up exactly the same UI send it to any other member of their organization and start collecting these labels. Uh and so that's a problem that hits home is deeply I think personal to all of us on this call at this point. Well, maybe it's interesting to talk for a second about how MLFlow you traditionally were building for data scientists, right? And now do you feel like the persona has changed? And if so, who are you building for? There's more stakeholders involved. What does it look like now? 100%. We we are focused on continuing to serve that community of data scientists who want to build awesome high quality models. And I think to to make clear that's not going anywhere and is is super important to us. At the same time, what we found a couple of years ago is there's so much synergy between agent development and model development. You're going through this process of building an initial version of uh an AI application. You're measuring its quality. You're iterating on it. You're managing the deployment of that thing. you're collecting signals from your end users once you've deployed it. You know, that's true whether you're building an agent or a model. And so we felt like, wow, we can we can take on this problem. We can start uh addressing the needs of agent developers. But as you dive in, the next order of detail around okay, how do I measure quality? How do I deploy this thing and and understand whether it's operating well in in production as we've talked about looks a fair bit different? And so that's where you know you have to serve your agent developer who's building that initial version of an agent by giving them logs and observability and stack traces so they can debug the same way that you serve the model developer by giving them a way to track uh train test and val uh loss during the training process. It's uh you're meeting similar needs but you're doing it with different tooling. And then at the point of I'm trying to evaluate and improve quality. Okay. Well, the the for the model developer uh again they're going to take a look at that test loss and they're going to go and do feature engineering and things like that. Uh but for the agent developer to evaluate quality, they need to go talk to a domain expert within their organization or they need to have these awesome out of the box uh LLM judges that can look at a natural language response and determine whether uh it it meets the needs of the user who sent the query or not. So again, similar objectives, very different sort of sets of tools underneath. So we're still serving, you know, AI developers, model developers, and agent developers, but it's required us to tap these brand new spaces of agent evaluations and things like prompt optimization and build higher fidelity feedback collection into the platform. Uh so what you'll see now when you you go to mlflow.org, you install it all flow and open it up is you get this sort of structured experimentation and deployment platform for ML and Genai but you you pick the uh the objective am I building an AI agent or am I building an ML model we tailor the experience based on that selection I I think Cory is absolutely right in terms of this is sort of a natural progression that we're actually serving in two different audiences right as as you pointed out the data The scientists deal with models which are sort of deterministic and the evaluation criteria is is is more quantitative. You know you have you have a certain precision that you're actually referring to. You have certain recall whereas if you look at the experimentation they go through the same process but the evaluation process is quite different right you have more a non-deterministic model where you actually now have this set of built-in judges and a set of custom judges that evaluate based on their criteria. And I see I see a similarity between prompt optimization and fine-tuning a traditional model. They they are sort of similar sim similar tools and similar techniques you use but it's an actual progression and think we emlow currently serves both needs. So with your traditional ML engineer or ML data scientist, you can use eml flow for that particular need. And if you're the recent more engineers and geni developers with building agents, as Cory pointed out, we have a set of techniques and tools and uh judges that allow you to measure that in a way that that meets your meets your demands and meets your needs, >> right? But also very often it's the same person doing both, right? So uh I think a while back we were like you know who is actually building agents and I'm actually curious Demetrius like whether you've seen this as well. Um we're like oh maybe it's like all this like new class of engineers who are building agents and like we're not talking to them at all. Uh and then we went out and just talked to like a bunch bunch of people um including a lot of non MLflow users and it seems like actually >> a lot of people just shifted from being like okay I used to be a data scientist or data engineer collaborating on a plastic model and now I've just rebranded into like a geni engineer >> engineer >> and so that's that's still probably like you know like half the market is like these like former data scientists or data engineers or MLAs converted into these geni people and so that that's where they often then will come to us and be like hey we we've already used emlow in the past like we would love to continue using mlflow but but certainly there are these kind of like new like engineers who uh haven't really gone through this process before and then they're like what is an eval what is and so then there's kind of an education component there of like uh you know hey we've already built all these expertise over like decades of how to like how do you iterate quickly to get high quality you know outputs right uh let us teach you kind of what are the best practices there so >> well that answers my question that was going through my head as you guys were talking which is like why not make a different product? Why not separate the two? But it feels like if it is for almost this very similar persona then it makes sense because they're already familiar with MLflow. They're just going to extend the capabilities. >> Absolutely the case. I think that there are from what I've seen two temptations that uh product developers competitors out there uh have have had to grapple with. The first like you say is that temptation to I'm going to build a brand new product throw everything out. Uh you know it's a brand new world of agent development. We don't need all that ML stuff. uh people can install a totally separate package, hope host a separate service, manage uh the the work entirely separately. Our agent developers who, as Danny points out, are often the same folks, need a different platform. They're going to install something else that we build. Uh and we're going to have to maintain two different ecosystems. That that's sort of the the first temptation. Second temptation is there's nothing different here. It's all just kind of, you know, training jobs and uh and you know, quality measurement for agents and and for models. It all just comes down to metrics anyway. You know, what what's the difference? Uh and that that has its own problems. So, the agent developer needs traces. The model developer, yeah, I mean, they might want them like way in the future after they deploy something, but it's not even something they're thinking about uh as they start to build a new use case. Uh you know, the Asian developer needs LLM as a judge. I don't know why the heck I would use that if I was building like a traditional machine learning classifier. Uh so you definitely have to like understand the workflows are different and the tools are different. Uh but the the objectives and the the persona the person that's that's trying to meet those objectives remains the same. They don't want to you know have to learn and install a separate platform. Yeah. the the business uh stakeholders or folks that are making purchasing and uh decisions within an organization don't want to pay for multiple platforms and set them up and figure out how to uh make them available. Um so we believe we found a pretty good uh happy middle here. You know the one platform with the right tooling for those agent devs and uh model developers who were often just data scientists a fair bit of software engineers as well. Yeah. the personas really had to meet in the middle in a way like data scientists had to learn how to become better software engineers and then software engineers had to learn how to become better data scientists and so you have this nice like ven diagram of of both of those skill sets that you need and you want to take advantage of and I do see like the agent building very much as lots of software ware skills that we're talking about, but being able to fall back on fundamental data science type stuff is crucial. >> It really is, especially as the actual software development part of it is increasingly automated. The coding assistants a basically agents building agents uh are kind of becoming the deacto pattern for a new use case. So we talk to customers all the time where you know despite the fact that cloud code and codecs and other tools really only rose to prominence within the last year or so everybody has subscriptions throughout a lot of our customers. the entire dev team uh is using these things to build their agents and so it's easier than ever for them to integrate new uh tools into their agent or connect a new data source or or even ask uh you know something like cloud or codeex to prompt engineer and so then you because the software development life cycle is so accelerated they have to fall back even more quickly in their workflow to well okay like what does the quality look like how well is this going like those data science skills to your point become increasingly important important earlier. You I'm not thinking about it three or four weeks from now after I've handcoded an entire agent. I'm thinking about it three or four hours from now because the the code is already written because I you know I I asked a coding assistant to do it. Uh and so where this is all leading I believe is a ensuring that folks can do the data science properly themselves but the b why can't we enable coding assistants to do some of this too? Uh so there there is I think an appetite out there for you know if I'm asking claude or codeex to build a uh an awesome agent. Can I ask it to evaluate it as well? Can I give it some example inputs or tell it more about what I'm trying to build and let it come up with those inputs? Can I then ask it to assess quality and iterate? Uh and so as a result, we've tried to make all of MLflow's judges evaluation capabilities, feedback functionality available to coding assistants uh through MCP. So I I can basically drop MLflow uh into my coding assistant and have it act like the uh the agent developer, which includes the eval data sciency work. And that's something that folks seem to appreciate as well. >> Jules, you're off mute. I get the feeling you got something to say. No, I mean, you know, Corey just articulated very well that the the personas now are sort of merging along and it's hard to say I'm a data scientist or I'm a data engineer or I am an architect given the ability and the extensibility of code assistant to say go ahead and create a traditional model for me and I'm not even a data scientist but I'm just saying the developer might say I'm not a data scientist but I heard about you can actually create it a classical traditional model and it will do it for me. So I don't have to actually have the skills. The skills comes in. How do you actually measure the metrics and which metrics do you actually want to use for a classical model or for an agent? I think that's where the skills come in. That's where the background knowledge comes in. But the coding assistants can do one thing or the other is as Corey pointed out, you drop in an MCB server with code and you can say go ahead and go ahead and write write me an agent to to evaluate this particular chatbot that I've written and it'll spill one out. So I think we're entering we're entering this new frontier where where the personas are very gray area but you just need the background to be able to say how I'm going to assess and what are the things that I need to know to understand to to get the best out of this particular piece of software that the agent has actually created. >> Yeah. I I think one kind of cool thing that's at least I'm also seeing is that um yes these personas are emerging but also a lot of the innovation seems to be coming from the like more data science kind of persona or like people who are well versed in that space. Uh a couple examples of this would be um I think we've seen on our like very good success with if you want to switch between different so like you know you're using like a Gemini model you want to switch to GBT or or vice versa how do you actually like how do you make that switch your prompts are tied to your models uh I think there's kind of the data science world kind of was like well this kind of just looks like model distillation how do I like teach a model to act like a different model and I think we've seen that actually if you combine if you try to do that and use DS PSI in the mix to do that you get like really really good results. In fact, you know, we've uh we've even seen results where you can use like a a smaller cheaper model and that can really accurately reflect the outputs of like a much more expensive model. So, it's almost a cost optimization, right? So, um that's kind of like one innovation I think that's like coming out of like more traditional like researcher data science kind of spaces. I just want to reiterate on something what Corey said which was I think the MLflow team has done a fabulous job of even though having two distinct platform one for the agent developers and GNI developers and other one for the for the MLflow traditional machine learning scientists the concepts are not very dissimilar right both need experimentation both needs a way to to fine-tune as I said earlier which one one does the promptization the other does the finetuning both needs a set of data sets that you can actually provision over you know you need training data set you need a evaluation data set you need a test set we do the same thing in in in gen where you actually have the data sets so I think the concepts are very similar and the underlying tools and techniques used are are are pervasive across both both these particular uh platforms so somebody who's familiar with with with ML and is coming into jai can actually do that or somebody who's familiar with ji and is going into ML can easily transport those particular skies so I think conceptually and visually and and through workflow they're not very dissimilar. I think the emlow team has done a very conscientious job in making sure that the efforts are flawless from one to the other. It's not like you're you're in a complete different world. There is some concepts that actually come from there's a ven diagram as as as diminish there's kind of a vend diagram of set of tools and concepts that merge together. >> Absolutely the case. I I also wanted to dive deeper into that uh sort of model distillation piece that that Danny was bringing up because I think it's a great example of the capability that folks resonate with whether they're coming from a data science background doing model development or they're building agents. Everybody wants as high quality an AI powered application as possible and they want it for zero dollars. And like obviously uh you trade off cost and quality a fair bit. But the real upshot I think of a lot of this work in model distillation is uh you I can build a great agent using frontier models and then over time once I understand the quality of that agent and the factors that determine quality I can I can ramp that uh choice of model down to a less intelligent less expensive frontier without sacrificing quality. And what I need is a structured way to measure it. uh and and what I need is a structured way to adjust my agents prompts actually in particular uh if I'm going to change models. So the the task of moving from like a GPT 52 down to a GPT5 Mini uh for example or a Gemini Pro to to flash. Um that process requires me to understand what are the examples of great agent quality, what are the examples of negative quality that I'm trying to avoid. and um how do I tailor my agents instructions so that mini can understand those instructions just as well as uh as you know GPT52 etc. So if I have both of those pieces which MLflow provides I can get to a point of cutting my costs you know by a factor of 10 even 20x uh just by virtue of of switching the model and so that's pretty powerful as well. One other area I wanted to hit on, Danny, we talked about a little bit last time we chatted was the idea of governance and how folks are really trying to figure that out right now, especially with agents being able to read and write and you're like, um, so who we going to blame when they write the wrong thing? How are we going to keep tabs on this? Especially if we're at an organization with a 100,000 people and they all now are leveraging agents. >> Yeah. Um I I think governance is a super hot topic for us uh right now at data bricks and Cory's been actually in a lot of these conversations more recently than I have. Uh obviously I guess data bricks is we're known for governance. We think about governance a lot. Uh I think AI governance kind of breaks down into a couple of components here. probably the first thing that people talk to us about it is like well I have like PI and this and all my different like agents and traces like what do I do do with that? How do how do I kind of ensure only the right people can have access to that? Um and then uh there's also I guess it it funnels out into well what data does the agent have access to as well? Uh who can use what agents? uh and also kind of it really kind of gets into that like AI gateway kind of realm as well of like well can I like fall back to different models kind of control cost uh and whatnot so I think we're thinking about all that stuff right now on our side we do have an AI gateway uh kind of managed component uh I think Corey we've now is it controversial to say that we are kind of exploring open source for >> I would go beyond that and say you know that MLflow has and and for open source way, >> but like kind of to like we are I think the plan is to invest substantially more into kind of an open source governance kind of gateway solution here, >> right? >> 100%. Absolutely. Um and so going a level deeper on on that axis and I think Danny just did a fantastic job of discussing the the sort of aspects of model access to data and then developer access to the data that an agent produces. So going back to that that model access to data problem. Um we are invested pretty heavily in providing cost controls for model access as part of open source MLflow. Also providing data access controls. So I I've built a bunch of tools. I have like this awesome uh tool catalog with MCP. I'm going to hook it up to my agents. How do I make sure that uh you know the right tools are exposed to the right agents for different use cases and that you know they you don't have an agent accessing data that it shouldn't be allowed to for example uh about a completely different customer like these knowledge bases are are pretty sensitive and then so locking down tool access is really important but then as I mentioned just cost controls I have two teams building agents you know one is building a low QPS chatbot the other one is processing billions of docu documents uh every day, they require different budgets and blowing the budget out of one project and and sort of exceeding costs can mean that the other developers don't get to do anything. Uh and so like folks definitely need what we call this AI gateway that allows organizations to say, "Hey, all right, for your use case, here's your budget. Here are the models you have access to. Here are the tools that you have access to. If you need something else, you know, come and talk to us, talk to your admins, and we'll grant that access in a structured manner. Otherwise it's uh you know API keys flying around access to data unregulated it becomes chaos and so there's like real risks there from a model access to data and cost governance perspective and I think we could probably talk about that all day and as Danny mentioned in terms of developer access to agent data that that second consideration uh there's some big challenges around that as well PII um ensuring that only privileged guys can look at at certain uh slices of those traces. That's all super important. And that's where data bricks unity catalog has been you know crafted and and tuned for years to provide governed access to you know model data, agent data, uh BI data, all of all of basically that this is no different than any arbitrary uh set of sensitive data you're ingesting into our platform. Uh it's just that it's coming out of an agent. You need to govern and manage it and share it just as securely. Uh and so that's where like we've been building data bricks and all flow on top of unity catalog and this amazing secure enterprisegrade and governed data lake on data bricks. >> Yeah. Um yeah I I have a separate life of database where I also think a lot about PI detection and um and how it relates to unity catalog. And I think one thing is uh similar to what we're seeing here which is the easiest thing that people will start with is they will just redact all PI and say like nobody has access to this or totally lock it down but rarely is that the actual goal. So you need a catalog and there's been a lot of investments on like how do you do you know aback attribute based access control so that and what people are trying to do now is like okay well uh if there's a scalable way to manage governance then I would actually open up data as much as possible to it people who should have access to it but in absence of that technology then they'll say let me just fully lock it down nobody can see this information and that's kind of like now you're you're losing a lot of that business value that you that you could be kind of getting access to right um I think And that's also going to uh be playing out within the agent space too where if you want to really debug why is an agent going wrong or you want to do like downstream like BI insights out of this stuff sometimes you you are going to need to capture some sort of PI information just not everybody has access to that and so at least where we've seen some early kind of success here is uh having all this traces get ingested directly into adults table and then now you have access to all the governance controls you have with with the unity catalog and now um you know you can go back to kind of uh having open access and sharing and doing all your downstream insights right uh whereas a lot of I think the other competitors or or vendor solutions out there their default reaction is like let's give you a tool to just redact all your PI and have it never landed to begin with. Uh to me that's that's that's a great first step but it means that you're you're locking away a lot of that like super valuable data that you could be uh really analyzing and and parsing. Well, I've talked to people who can't use memory tools because the memory tools just by default, you know, the open source ones that you get off the shelf now expose all of this conversation that's happening with their users and and there's PI all over. It's just like littered throughout that. Even if you extract away sensitive data that's obviously sensitive, like a social security number, for example, inside of the enterprise, if you're asking a chatbot to help you craft an email to fire someone, that's also sensitive. And like that's much harder to get when you're thinking of it like in the traditional sense of PII. you know, it's not like, oh, that's not my credit card number, that's not my social security number, but that is very sensitive. >> So, you're talking about, De, you're talking about, you know, the the what Danny was talking about the PII that can be redacted and and that comes up quite a bit in in Meetup talks about how we actually going to redact the traces. But, I just I think you you bring up a point very important is I'm crafting an email which is going to fire someone or I'm crafting an email that's going to promote someone. That's considered kind of you know private information. >> Yeah. >> How do you actually read out that you know what's the context? So that the context is important. How do I actually take a piece of trace or piece of output that has come up and then evaluate and categorize that as sensitive or nonsensitive? Right. That's again an evaluation criteria. >> Yeah. Well, I was going to say and actually the process of redacting PI is actually a hard process. So I think a lot of people start thinking like let me just write some like random rejax. Uh for people who are a little bit more sophisticated, they might use like Microsoft Presidio. Uh but I think everybody I've talked to, including ourselves who who've tried it, is like you just get so many false positives, you you you miss everything. So you know what what we ended up having to build for our own like PI detection offering is we had to build an agent essentially like a whole like kind of complex like sequence of LLMs and heristics and what not to kind of actually figure out um you know with relatively low cost what is PIO what is not and that we've seen that is much much better than like kind of the off offtheshelf solutions out there. Yep. And and what makes this a lot easier is actually having access to the raw data. If I can treat this thing as a dataf frame or as a table and use all the brilliant, you know, SQL and Python and ecosystem tools for scrubbing that data, analyzing it, tweaking it, feeding only subsets that I know are safe into a uh a memory database so that when Demetrius's tool reads from it, I know that it's it's reading information that uh you know is is safe and and secure. That's really important. Uh unfortunately a lot of solutions out there don't actually give you access to the raw data like typically these traces get dumped into some uh shared like big multi-tenant uh like Postgress database and if you ask for like hey how can I get like a table or a data frame or the the raw records that I can do some sort of more complex analysis or find PII and redact it the answer is good luck it's you you can't actually get that data back and so that's where again this uh sort of datab bricks Unity catalog, delta table approach that stores all of your data as you know the beautiful tabular formats and and and data frames makes this kind of analysis a lot easier. You can plug in any tool from the ecosystem. You can use SQL uh you can express any sort of transformation or PI redaction task that you'd like. And so I think that's really powerful as well. Yeah, I I'm excited for kind of what future stuff we start building with linkbase as well because linkbase is kind of tailored to this kind of use case that Demetrius you're talking about, right? You you want like a database that [music] it has fast access retrieval and what not but is well governed, right? And and that is I think kind of the promise of lake based [music] on database as well. [music]