How Sierra AI Does Context Engineering

Key Takeaways

It is useful to understand energy levels and think about when light hits atoms, how electrons will change energy levels, but it is more accurate to think about the electron cloud. [music] And so that what you sort of modeled with the web is probably more accurate. But decision trees and flow diagrams are still potentially useful constructs to think about how an agent should behave. >> [music] >> The difference about AI is that instead of the software being super cheap, super fast, super reliable, it's actually quite expensive to run, kind of slow, it gets slower and slower the better the reasoning models get, >> and non-deterministic. So, it can be creative, it can be flexible, but it can also hallucinate. Mhm. >> And so the methodology that you need at each of those different junctures in this flywheel that you're trying to build is pretty different from with traditional software. >> So it's like traditional software does this. Let's just take the opposite of all of those. >> A good example is like unit testing, right? So with unit tests, uh you run them once, if they pass, you're good. >> Yeah. uh with simulations which is uh what we call our uh >> flagship testing product in Sierra. >> We might simulate a given conversation 5 10 15 times and then you'll look at what data changes happened as a result of the conversation which is a similar methodology to what we published with TaBbench which is the open source framework for EVAL but we'll also do uh using an LLM as a judge or a third agent. So you kind of have the user agent, the agent agent, and then the evaluator agent. And the evaluator agent will say, you know, did this conversation meet this checklist? >> And that's pretty different from unit tests or even integration tests in traditional software. >> Yeah. Do you even put that in the CI/CD pipeline? >> You do. Yes. So we have the concept of what we call critical simulations which are simulations that run you know when you actually will merge a commit to main or uh schedule a release or what have you. >> Feels like that could be really slow. >> It could be. Uh we have a high degree of parallelism. So if you have 300 simulations that you need to run, you're not waiting for them to run one after the other. They're going off and being executed in parallel. Mhm. >> And when you say simulation, are you only talking about it for voice or are you talking about it for everything? >> It's interesting you asked that question with voice first. Uh I feel like uh most of the time I get that question. It's are you only thinking about it for chat? >> But it is both. Um so it's actually pretty interesting to listen to voice simulations. The way that they work is we'll have a library of background noise, a library of voices, a library of all these sort of different entropic things that you can introduce to try to simulate what it would actually be like to get a phone call. >> And our customers are all over the world in all different environments and all different industries. >> Makes sense. >> Exactly. So, you have people calling in from the car. you know, SiriusXM is one of our uh larger customers, and people will call in because their radio's not working, and they're in the car or by the, you know, on a busy street or something. And so, you have the background noise, you have different accents, you have different microphone quality. Not everyone's talking into this, you know, magical podcast mic. And so, you have to deal with all that. And our simulations have kind of gone through that punch list, you know, rowby row and said, "Okay, we're adding this to the product." And then if you run it 5 10 15 times over 3 400 simulations, you have a pretty good idea that if those are all passing, your agent is doing what you would expect it to do. >> Do you also look at simulations a little bit like red teaming? >> Definitely. So there's I I don't know if I would say there's two classes of simulations, but there are definitely some where it's just the happy path. Oh, I I need to reset my radio because the encryption keys expired. And so an API call goes out and a satellite beams down the uh new encryption keys and you can listen to sports talk again or Howard Stern or whoever whoever you like to listen to. Uh but then there's also the more adversarial simulations where you're trying to um you know we call it abuse the agent >> and so we have a number of custom models that are just about detecting abuse. Uh and those are often customer specific as well where you know some customers they might have a younger audience and things like self harm and bullying are top of mind. Some customers might have different audience and things like hacking into the system are top of mind. So you have to be pretty customer specific in how you think of it. The nice thing is this is a very valuable problem for our customers and so they're deeply engaged with us and we've gone over the last six months or so making the entire product fully self-s serve so that it's self-explanatory. You can get up to speed. Uh and while it is this deep thought process, it's kind of up to you on the spectrum of how much you want to take that into your own hands, how much you want to work with Sierra's agent development team. >> How are you creating these nefarious simulations? There are three agents involved. Uh there's the user agent. Um there's the agent itself, the AI agent, although they're all AI, and then there's the LLM as a judge. And so the user agent basically can take a system prompt as part of the simulation, >> but you don't have to massage it to get it to be like really bad. >> Um so I think to get it to be really bad, uh you would. Yeah. And typically we won't have the really bad simulations as part of the the the run. >> Um you can also uh >> because you want to catch that before it's out or trying to get to production. >> I think the honest reason is that you typically the list of really bad stuff you just want to make it so not everyone has to look at that every couple minutes. And also I don't think that uh AI models are that good at imagining some of the terrible things that people say. >> Um probably because you know the uh model providers have gotten out in front of that. >> Yeah. >> That being said, we also have the concept of verbatim simulations where you can just hardcode a script of what the user agent should say when the AI agent says X. Uh and so >> for those ones where you really want to test against, you know, some of this behavior that you definitely want to prevent against or um you know, prompt hijacking or these kinds of things, you basically will do a verbatim simulation instead of an AI user AI simulation. Does that make sense? >> Yeah. The really bad stuff, I'm not necessarily thinking really bad in like what humans are capable of. I'm just thinking really bad of like as a prompt hijacking type of thing where I'm just trying to get some data out of the model. If I know that I'm now dealing with an AI agent, let's see what I can get from it in that way. Maybe it's naive. I feel like the models that are out there today are not that good at trying to like hijack themselves. There's two things that come into play here as well. One is that the actual access to sensitive information is always safeguarded in a deterministic fashion. >> So, so we'll have more traditional unit tests that will make sure that the access controls to a system are no different from what you would have if you were on the website, you know, logged in as a user, as an example. So we don't leave it up to the model in any case whether to expose sensitive information. Uh we leave that up to deterministic systems. >> How does that work? Like the connection between access and model. >> So often let's say you go to um I'll think like Sonos for example and you're logged in. The fact that you're logged in means that you have access to a set of data in Sonos's database. maybe the speakers that you've connected to the networks in your home and the AI model also has access to that same information. So you would never be able to use the model to get information that isn't already available to you, you know, as a loggedin user on the website. >> Yeah. So it's almost like as you log in, then the model just has this sandbox of your world. >> Correct. And then if it needs to go and do something because you have a complaint, then it like gets out of your world or it >> Well, if you have a complaint, that would be something you could do in your world as well. So the models, they're they're taking action all the time, but they're not taking actions that you wouldn't have permission to do as a user. >> Yeah. And so then you're just protecting on the permission side of things like >> and that doesn't seem like it's like that's a triedand-true thing that we've been doing in software. And that didn't necessarily flip like you were talking about back with that cycle thing. >> It's not one of those slow non-deterministic. If you can make sure that the model doesn't have access to it, it just gets access to what it gets. Then whether or not it gives the information is where things get wonky. >> That's exactly right. And you said earlier like, oh, so you're just kind of in doing the exact opposite of the software development life cycle. And I kind of just took it in a different direction because it's that's actually not true. A lot of the times you need to do the exact same thing that's tried and true from traditional software development. Sometimes you need to do the opposite and sometimes you need more of a hybrid approach. So >> it's more of a first principles analysis of each of the different steps of kind of plan, build, test, release, optimize, analyze, plan, build, test, release, you know, etc., etc. Um, but just kind of from the first principles of what's changed using AI systems along with deterministic systems as opposed to only deterministic systems. >> I wonder how different the simulations are for voice and for text or if they are I know that of course with voice you got to add in all these special things like you were saying like noise, background noise also. I I've talked at length with people about conversational design. You can't let somebody know that the model's going off and doing something, >> right? >> Or you don't want to have somebody thinking that somebody hung up, the agent hung up because the model is off and doing something. So, you want to give like cues and all of that stuff. So, I imagine that's one type of simulation and that you're only going to find in voice, >> right? And then in text you have different types of simulation and how do you see those two being different? >> It's a really good question and one that's very close to my heart. So because my personal story at Sierra was I spent the first nine months building agents with customers. So I mentioned Sirius XM because they're one of our uh larger customers that we're proud to work with but also because I helped build the very first version of their agent. And then SiriusXM, you know, as large as they are over chat, they're even larger over voice. And so just seeing that kind of having it beat us over the head of, hey, voice is actually the huge opportunity here, >> uh, you know, led uh, me and a couple others from the team to start prototyping what would it look like to uh, build out a voice version of their agent. And so we iterated on that and eventually kind of moved up the hierarchy of needs from just being able to go back and forth to actually responding to verbal cues and interruptions. One example is an interruption is not actually a super objective thing. If you start talking right now, you might be agreeing with me, encouraging me, interrupting me, uh sort of telling me to go in a little bit of a different direction, but most automated systems today, if you say something, they will just stop and start over no matter what you said. And so, >> so painful, too, as the user. >> Yes. Yes. And so it actually feels like they're talking over you because you're like, you go, no, I go. And I don't know if you've everyone's had this where you're walking down the hall and someone's walking towards you and you're like who's gonna move. >> Um and that's what it feels like all the time. >> End of dancing in the hallway. Yeah. >> And like obviously once you get so deep in the matrix, you notice everything. So as an example, the two of us were just talking at the same time, >> but it wasn't a problem. >> Yeah. >> Uh another example is if I pause for a very long time like I just did in the middle of a sentence, you know, to wait for me. But if I'm done speaking, you know that it's your turn to speak. And so context is so important. Both was this an interruption and is it my turn to speak? And it you can't just boil it down to a number of milliseconds. You need to actually think about what's being said >> and start planning your response, you know, before the other person's done talking. >> Everyone almost has their cadence. Their way that they talk is unique to them, right? And so I may give longer pauses and I may speak slower and I may not want to finish but then actually I do finish or things like I speak weird. Right. Right. >> And we all kind of have our own quirks when we speak and you can't have a universal setting of all right well the model's going to be like this. The model almost has to adapt to each individual way of speaking. But I haven't seen anywhere, maybe you've messed around with this, where you can have the model interpret the first whatever 10 seconds and go, "Oh, okay. This person takes long pauses in the middle of sentences or this person speaks a little bit slower and so I'm going to give them more time between words." >> Right? It's a really good point. At the very high level, you can do some of this uh for customer specific. as an example, you know, if a customer has mostly older callers, you might wait longer. Uh if a c if a customer has a lot of authentication flows, you know that it takes a while to, you know, talk through your email or your password or these kinds of things. So, some of it is shared. That being said, I think to do it in a personalized manner like humans do, you probably will have the most success with some of the speech-to-pech models where the audio tokens themselves are the things that are being inferred upon. And so that actually all kind of gets boiled down into the training data. >> That being said, right now most of those systems for most production environments are just a little bit too hallucinatory. >> Yeah. Our approach has been we built a modular architecture. So on the transcription side and on the synthesis side and by the way even if you're using the speechtospech model you need transcription because you can't make an API call with a bunch of voice tokens. Uh and so we transcription is part of the pipeline no matter what. But we do have support for uh models like um you know the the model that underlies the real-time API the uh GPT audio models uh as well as the Gemini audio models which are these kind of fully speechtospech models and I think that's going to be the future. Yeah. >> Uh it's just being honest with you right now. The uh amount of hallucination that we've seen in our production tests is such that for the larger customers, uh the text to speech is still uh the most reliable way. >> Yeah, I've heard that from just about everybody I've talked to where it's like we want it with that speech language model, but we can't have it because it's just I can't put that into production with any amount of confidence. >> Yeah. And I would I would go a step further than we want it, which is, you know, that I believe in it. >> I think if you and I were passing notes across the table right now, it would be a much less interesting and productive conversation. And that's basically what AI agents are doing. >> So, we're willing to make the long-term investment in our architecture where we support that out of the box today even though it's not quite ready. And we're following the um OpenAI in particular just released a new uh real-time audio model. And every time one of those things happens, we uh you know test it out, figure out are there places where we could use this. Um and obviously there's a risk tolerance and risk likelihood spectrum for each of our customers. So I think we will find that >> for certain languages, for certain customers, for certain use cases, this is going to be an absolutely magical experience and then over time it will kind of disrupt the texttospech approach. At least that's my personal belief. Do you have a pipeline that you'll run models through as soon as they come out just to see like because obviously when you look at benchmarks it's just a bunch of [ __ ] >> right we do some benchmarks are really cool as well like we love our tow bench benchmark it's really cool when the math olympiad is won by an AI but when it comes to putting systems in production for our customers I think what you said is roughly accurate uh we do immediately when a model comes out we have a whole suite of eval but it's usually an iterative process because one model might be better if you prompt it in a different way or if you fot it in a different way or even if you fine-tune it in a different way. So it's not you have a good instinct immediately but it's often a multiple days multiple weeks process to know if the ceiling of the new model is higher than the ceiling you know the local maximum that you've climbed to on the previous model. Um, so it's it's and then I would say also for voice, uh, language is a huge deal. Uh, as well as like dialect and local. So Brazilian Portuguese being different from Portugal Portuguese. Uh, and so you have to we basically have a rubric uh, and we have, you know, basically callers that we work with who can call in and test out each model. Oh, nice. um you know because you need people that actually speak the language that can be in realistic scenarios or else you're going to most likely overfit to some data or you know not be representative. >> Yeah. My buddy was telling me how he was doing a bunch of simulations on the OpenAI real time and one funny thing that he found was when you speak English with an Italian accent or no when you spoke in Spanish with an Italian accent >> Yeah. it would switch to this like madeup language that was like a mix between Spanish and Italian. Oh wow. >> And when you spoke ger when you spoke English with a German accent, it would just switch to German and speak to you in German. >> And so it was like what is going on? I speak in English, you know, like I want it to be in English, but it keeps switching to German. >> Yeah. >> And at the end of the day, you're like, but is it the end of the world? Because if the German accent is there, that probably implies they speak German and so why not just speak into in German? >> Exactly. It's like I don't know, do you speak any other languages besides English? Okay. So, if you go to um a restaurant, let's say, and someone greets you in English, but they have an accent that suggests they speak Spanish or Portuguese, you do you kind of feel that urge of like, oh, I want to practice my Spanish or like, you know, could we have a discussion in another language? Wouldn't that be fun? And I guess that's sort of what's going on in the model. >> The model's like, [clears throat and laughter] "Oh, yay. I get to touch." >> And on the uh simulation, I know that >> our mutual friend Will >> has told me about how he uses heat maps to see where the agents are good and where they're bad. And this is like on the task completion side of things. Have you tried to do any of that with like because that feels very orthogonal to simulation in a way? It's like you you simulate things and you see where it's good, where it's bad, but then if it's an agent, maybe it's like the next step is can it do this? Can I actually like complete the tasks that we ask it to do? >> It's a really important piece of it is not just what did I do wrong, but how do I do it right next time? >> And I think without a clear understanding of where you went wrong, >> which I think maybe what you're getting at with the heat map. Mhm. >> You can't really understand what the fix would be. >> Yeah. >> So to give a few examples, sometimes the agent will go in the wrong direction because it doesn't have the right answer. Sometimes the standard operating procedure is wrong. >> Sometimes it was asked to reason about too many things at the same time. And so you want to know those things at the point where there's an error so that you can fix it most quickly. And in simple cases, the agent can fix it itself. So, one example is a number of conversations from Sierra. It's uh remarkably low at this point with a lot of our agents, but a number of conversations uh end up getting transferred to a human. Um especially if there's something that you need to do that just requires oversight or regulated industries, etc. >> And so in those cases though, >> you can actually learn from what the human did post transfer and detect is there anything missing in my standard operating procedure? Is there anything missing in my knowledge base? And have AI author draft article of, hey, you should maybe you should add this knowledge. We had, you know, 150 different agents all mention this thing, but the AI agent doesn't seem to know about it yet. >> Uh, and sometimes it's like, no, no, I don't want the AI agent to know about that. And sometimes it's like, oh yeah, I was missing. And it's remarkable how often it's the latter because a lot of the companies we work with, >> one of the things we hear a lot on our discovery calls is like, "Oh, I don't know if I'm ready because my knowledge base isn't in a good place." >> Uhhuh. >> And so our response to that uh and one of the reasons that we built this product is well actually the best way to get your knowledge base in a good place is to launch an AI agent and just make sure that it's scoped appropriately, but you'll actually find the edges of what it knows. And our system, the Sierra platform, will create basically a prioritized list of the knowledge that your agent should have but doesn't have or where it's a little bit off or implying the wrong thing. >> It reminds me of like the continuous integration where you're just you're seeing where's the edge case. Okay, cool. Now, let's integrate that back into the knowledge base so that hopefully it's not an edge case next time. >> Yes. And I think there's two other things here. I know I'm doing a lot of twos. Um there's two other things here that are [laughter] that are very very important. One is it's a really important business problem for customers. So when you create the system that allows you to kind of pull in the right direction and just create that upward spiral of improvement, >> they're going to pull on it. So we have customers where they have, you know, 10 plus full-time people evaluating conversations and looking for opportunities for improvement >> because it's such an important business problem. And then the other piece, I know it's not a business podcast, but I think it's kind of geeky, too, is we have this outcome based pricing model where [clears throat] Sierra only gets paid when the agent does does the job well. >> Yeah. >> And so you have the incentives on the customer side and you have the incentives on the Sierra side. And so as sort of a systems thinker, it's this very very fun environment where if you create the system, the business incentives on both sides will just pull the agent in an upward spiral. And I think that's accountable for a surprising amount of our success. It's just that the incentives are so aligned >> and the table is set to improve the agent. >> I always [sighs and gasps] love the idea of outcomebased pricing, but feel like it is insanely dangerous. >> Why do you say that? >> Because you it's on you. And if your agent [ __ ] up and then you don't get the outcome, you pay money. Like you eat that price of having to do for us. Yeah. Exactly. Like >> there's a potential world where you don't get the outcome, but you end up spending money. Obviously, it's like we're talking cents here potent, but at gigantic scale and you don't recognize there's a problem soon enough, you can burn through some cash. I think though these are two correlated events in a way which is it kind of turns a lagging indicator into a leading indicator. We'd actually rather know that sooner rather than later that our agent isn't performing well. We'd rather feel the pain of lighting money on fire so that we have to fix it and then when it comes time to renew a contract a year from now, we actually might feel better about our relationship with our customers. So, I think our NPS with our customers is much higher because of this because we actually find out about issues and we have to fix them because we're feeling the pain of like a hole in our pocket uh in the moment instead of, you know, 12 months from now where they're like, "Hey, yeah, we, you know, it wasn't performing as well as we thought." >> Yeah, that's a great point. You're not going to get that conversation as often because you're going to be charging them when they have successful cases. I I could see that, >> right? One thing that I've heard folks talk about a lot is that getting the and especially in the uh customer success lane like getting it working for one or two um people when they're coming in and complaining or doing something is not I'm not going to say easy because it's it's not easy but it is doable. And then where it really gets hard is when you have to service a company that has millions of users and then the customer support is trying to service all of these tickets at scale. Have you How do you think about like that and is that right? Because I've heard it from folks. I I want to like fact check it real fast and then see what how you think about it. And you're talking about the AI agents or human agents because we we definitely see that on the human side where you know it's not a coincidence that so many of our customers are large consumer companies where they have millions of customers. I think I think it's true that most of our customers have millions of their own customers. Um most of our customers have over a billion dollars in revenue. So that kind of stands to reason if they're a consumer customer they probably have millions. >> Yeah. And they see this scale such that the power of bringing in AI just changes the nature of their relationship with their customers. Where previously maybe they had a strong incentive to hide their phone number on some backwater page of their website, now they can plaster it on the front and offer that company. >> Yeah. Where you're like trying [laughter] to click through. >> Do I contact these guys? >> Yeah. Yeah. It's like a meme at this point. And so I I think when you think about uh you know the the AI agent as kind of the analogy to the internet website or the mobile app. It's also about the actual addressable market of contacts with a business growing because now you can provide a great experience to anyone who calls in. Mhm. >> I think though uh what you said about you know seeing just a different distribution when you get scale uh the stat that's stuck with me a lot I spent the first seven years of my career at Google and to this day something like 20% of all Google searches have never been made before >> and oh wow >> that's because different things are happening every day new language changes uh you know the things people are curious about change etc >> and so you need a system that's actually like resilient because the same thing is true for our businesses. The reasons people are chatting in are going to change each day. You know, if uh someone is doing an interview in the Sirius XM uh studio, you're going to get different questions from a day where they had an outage or something like that. And so each day there's just different questions and you have to build a system that's truly resilient. And uh one of the things that I look back on Google really fondly about is just the amount of invention occurring in the company. You you'd go to the all hands which is called uh TGIF and you would hear oh you know we use machine learning to improve the way that cooling works in the data center and it's saving us a billion dollars. It's a lot of money for an algorithm or like uh you know they before the latest iterations of large language models you know we are understanding search queries better we created this BERT u classifier or analyzer which can tell us you know a lot more about the actual semantic meaning of a search query not just the keywords that are in it. um or you know our data centers they run 20 degrees hotter than any other data centers because actually machines failing isn't a big deal for us because we have failover uh different from everyone else and so there's this sense of invention and you know building things and creating differentiation at the technical level not just at the product and marketing level >> and Sierra is the first experience I've had since then where I feel like because AI is moving so fast and because we just have some of the most genius thinkers in the world and and builders in the world, you have that feeling of we're inventing new things. And I think the what we call our agent SDK, which is the software layer that our agents are built on, both the developer ergonomics and tooling and release cycle and kind of the what we call the agent development life cycle put into a product, but also the software constructs and the agent architecture itself are just novel and inventive and flexible. Um, and I think it's it's what I said. I'm I'm I definitely feel a lot of just admiration for the team, but it's also being out on the experience curve >> ahead of some other companies. I think uh just building with customers from the beginning starting early in 2023, less than 6 months after chatgpt came out and just getting reps with customers means that the abstractions that we have are a little bit less leaky uh a little bit more robust for all the different things that that could come in. Um at least that's my theory. Uh and then I think we have the right sorry to ramble a little bit here but I think we have the right layers of the stack where you have the platform uh but then you also have the flexibility of agent development. >> So most of our agents at this point are being built in no code. Many of them are being built by our customers directly. >> But you still have that uh ability to express anything you need by dropping down into code and kind of just interoperating between the two. And I think that's where you want to be working with the customers we work with where their business leaders might have the best understanding of the goal we're trying to achieve, the metrics we care about, the standard operating procedures, but then you still need to connect to APIs. You need your engineering team to get involved. Sometimes you need to drop down and do something with, you know, not your chef's knife, but your pairing knife. And so this platform was just thoughtfully architected by being so customer obsessed over the last two and a half years that you know what we have uh as a reward as a system that can handle scale in that way. >> Thinking about three years ago when you started versus now >> and how much you've had to reinvent or try and create new or try and like test out and it's perpetual R&D in a way. >> I think that's exactly right. And if there's one thing that I feel like has helped me, you know, have more good ideas than I otherwise would have, it's spending those first nine months at the company. Uh, you know, working as an agent product manager and agent engineer, and just being so close to customers, feeling their pain, >> you know, having to tell them directly face to face when the agent did something they weren't expecting, getting to celebrate and go out to dinner when it, you know, hit some business target. >> Yeah. Nice. those those experiences are kind of, you know, in my neurons in a way that uh I feel like is just I'm just so grateful for and I think anyone building a startup, the the more you can kind of feel what your customers are feeling, the the more integrated your decisions will be. >> Huge win. I remember a conversation I was having with a buddy of mine who was saying, "Look, man, we have found the biggest lift on our AI systems from just gathering the whole floor of the company together, no matter what their job role is, and having a labeling party and getting a bunch of pizzas, and we just get together and we look at the conversations that we've been having with the AI agent, the agent, and the user. like real conversations and everyone just labels across like five metrics. Did it do this? Did it do this? What what would you give it as a score? And by the end of it, the business side of the house understands the AI so much better and they can say like, "Oh yeah, okay. I see what's going on here." And then you catch all these anomalies that you probably if it was only engineers looking at it, they would have never caught because it would seem like, "Yeah, that seems fine." But then you have the subject matter expert looking at it and they're going why what? No, we can't say this. We shouldn't be saying this. This doesn't make sense. Or or sometimes it's just like this is stupid. No. >> Right. >> So I I think about that and enabling the low code no code is almost like another way of doing that same thing. >> Yeah, it's exactly right. And I think what we've seen working with hundreds of customers now is when we bring together the business side and the technical side, I wouldn't say it's the only way you can get results, but it's certainly a heck of a lot easier. Uh, and it it's the only way you can get really like the best results because these two stakeholders are so involved. And we've done that enough times now where we have some of those repeatable processes to bring everyone in the room and make sure that all of the different viewpoints are represented. The other thing, this is kind of like a pinch me moment from working at Sierra, but I don't know if you're familiar with the book Unreasonable Hospitality. >> Yeah. Why have I heard of that though? I think But why? Yeah. Why? Why do I know that? It's not a It's not a Greek guy that wrote it, is it? >> His name's Will Gera. I'm [clears throat] not sure at all [laughter] >> his actual origins. He's from New York. He ran a restaurant called 11 Madison Park for a while, which rose. >> Yeah. They got, you know, they became the number one restaurant in the world is the spoiler alert. >> Um, >> like the five Michelin star where it's like you could only get three. [laughter] >> How do you have five? >> Right. Right. Uh, and it's, you know, this beautiful ballroom environment in New York. Um, but was kind of a, you know, pretty good middle of the road beastro for a while and he just, you know, along with the team there took it to the next level in terms of service and hospitality. uh and he's one of our adviserss at Sierra and helps a lot with our how we think about tone and language. And so it it popped into my head because what you're saying of like get the whole floor together and have an eval party or labeling party. Uh we've also actually gotten to do that with him where he'll come in uh to the office and spend an hour just looking at conversations with us and we say, >> you know, what would a three Michelin star restaurant do in this scenario? >> Yeah. And of course, you know, sometimes they would do things that are not within the budget of our customers, but sometimes they would do things that are totally free. And so it allows us to think about aspirationally, how can we make an agent where it feels like it understands you the way the person, the host who greets you when you go to 11 Madison Park or one of the best restaurants in the world. And that kind of feeling of service. And I think that's one of the reasons why it's fun to work on is yes there's the part of you know companies are hiding their phone number and we want to just bring it up to a baseline but then there's also that part of how can we actually make the relationship that you have with your businesses that you know good and great and you know something that feels like you're going to a Michelin star restaurant. >> Yeah. Do you remember any of the things that he said like, "Oh, we could try this or maybe they would be that that were the free ones, not the right >> expensive [laughter] ones like going out to buy dinner." Yeah. >> Um I do I do I remember it very vividly because one of the things that stood out kind of to your point actually is that even though he at that point hadn't had as much experience with our technology, he had such an intuition of what a good answer would be. And so I, you know, had this picture of, you know, I understand each part of the stack and that helps me throughout the stack. Yeah. >> But then I would put him in this other category of just like soant at the one thing that matters. >> Uh, and so we were reading through a number of voice conversations and listening to them and hearing for different tone and awkwardness and um the ability to kind of just connect and and drop the barrier. Uh, and what actually stood out to me is there was one message we in a long I think it was like a 45minute conversation and there was one message where the agent sounded really empathetic and like connect clearly connected with the user and we were going through each and being hyperritical and I think on that one you know it was less of the uh I'm so sorry that this is frustrating and more like man this sounds really tough you know like we've been working through this and we can't figure it out yet. Uh, and it was a very difficult technical troubleshooting for like a uh basically um banking or like financial services application. So, not an easy problem to solve, >> but what really stood out was that ability to just connect and be like, I hear you. I see where I see you. Like, >> let's try and figure this out. >> Yeah. and the you know I would say I would give us you know maybe a constructively like a B minus on the particular conversation that that we were reviewing but in that particular message it was like that's what we want >> and I think someone like Will uh and and some of the people that have really good taste which is a one of the few like appreciating assets uh you know in the labor market right now maybe um the that really good taste often comes out when you're like not necessarily critic criticizing all the bad things but being like that that's what we want. >> Uh and so I was really impressed with his ability to kind of put his finger on those moments, those bright spots that we want to scale. >> So he tells you that and then what do you do? How do you incorporate a lot of that in in a way that it's not just like you're absolutely right, >> right? So yes, this is where the full stack understanding becomes relevant again. [laughter] Um >> that's classic. >> There's a number of different ways you could do this. we've had success um or I should say one of the ways we've had success actually in this particular case is with uh fine-tuning of models to >> phrase things in a certain way. >> The a lot of the base foundation models have been trained primarily for chat achieving AGI doesn't really care if it's written or spoken. And so I think where when you look at where the Frontier Labs are going, they're actually going sometimes in the opposite direction of a low latency conversational voice experience. >> Yeah. >> And so having a little bit of control at the model level for style can help a lot on that particular problem. Now when we do have fine-tuning, if it's with data, you know, that's in any way derived from the particular customer, then it's a customer specific model. Um, and if it's more just, you know, general tone and style, it can be useful across customers. But you also have to balance the fact that anything that is being put in the weights could be regurgitated. And so you have to make sure that there's nothing in those weights that any of the users who would interact with those weights wouldn't be able to see. Um, so the that process is, as you can imagine, much more labor intensive. And so it's really those specific circumstances like the style of voice where it's very specific, very high leverage, very important where it [clears throat] makes sense to kind of drop down to the weights. Most of the time I would say the solutions come at a higher level in the agent architecture or in some of the underlying task prompts um or you know changing the model itself, those kinds of things. >> Yeah. It's only when Will comes in for the day when he's like that piece right there, you're like [ __ ] I guess we're going to have to fine tune. Right. And you want to make sure it's durable, too, because uh you know, if like style is something that most brands don't want to change too often, >> too marketkedly, whereas if you're fine-tuning this new promotional offer into your weights, then two weeks later when the promotion ends, your model is out of date. So, the right tool for the job is definitely one of those things we think a lot about. >> This is probably a good segue into the ensemble of models and how you work with that. So, it's actually mind-blowing to look at one of the conversation traces with the Sierra agent. >> If I if the Sierra agent says, "Hello, how can I help you today?" and I say something like, "Oh, you know, um, my uh I'll try not to use SiriusXM. Uh, you know, my groceries weren't delivered and I'm not sure where they were. You know, what's going on here?" Between me finishing talking and the agent responding, there might be 10, 15, 20 different models that get invoked. >> And some of them, you know, might be as as simple as like an embedding model to do retrieval augmented generation. Some of them are frontier models, maybe even with some reasoning tokens to make sure that we're able to handle a complex reasoning space. Some of them are fine-tuned uh fast but cheap classification models to just you know understand the task at hand >> and it would be very slow to run these all in series. So sometimes uh there are certain models that need to return a token before the next model can start but most of the time this is happening in parallel. And so I think what's really impressive is the ability of the agent architecture to break problems down into tasks that can be handled by different models and then selecting the right models for each of those tasks and doing it super quickly. And we call that whole process kind of our our constellation of models that we use to respond to a message in a way. So it feels really simple, but I think you know my Google background showing underneath the iceberg there's just a lot going on between when you type something in or when you say something and when you get a response. >> Okay. So, this is wild, man. Tell me more about the >> way that you are. So, you're doing everything async. >> It's basically you're gathering it's that same input data, right? >> But then it just spurs out and there's a bunch of different variants of it >> and you do have dependencies on certain parts of the this output. like does it already know the taken path because I can't imagine that it would know all right well I >> how do I how does it know that it needs a dependency and it needs like I don't understand this break it down for me more >> so this is one of the reasons why I think voice is still um an area where we're doing extremely interesting things from an engineering perspective and where we're still you know significantly ahead on a product perspective and it's because latency is so important so to give you an example um retrieval augmented generation Like if we need to access a knowledge base to answer a question, >> the agent will often make a decision to look up the knowledge before it even knows or simultaneously while it's deciding whether it would be a good idea to use the knowledge. >> And so often times if you're just building something naively, you would think, oh, should I go look that up? Okay, I'll go look that up now. I'll give you the answer. But what an agent is doing because it's tokens are, you know, more parallelizable and um, you know, less expensive than ours. It's saying just in case I need to look something up, I'm going to go do that. Meanwhile, I'm going to use a smarter model to reason about whether it's even a good idea to have gone and looked that up. >> And then, uh, if I decide yes, I should have looked it up, then I already have the information. >> But you're using that as like a tool. It's a tool call that gets invoked or >> Yes. Conceptually, yes. Okay. >> Um, so the lookup tool would be running in parallel with a classifier to decide if the lookup tool was even necessary. >> Uh, and then you already have the information and that's just a very small microcosm, but >> there's tons of speculative execution throughout the uh each agent term >> and then you kind of like you know and this is one of the reasons why like that was not a term I was familiar with when I started working at Sierra. uh but we have a number of people that have you know worked on building compilers and uh you know very complex systems from scratch and kind of understand these concepts of software engineering. Um, you know, our Brett, our CEO, was one of the principal authors of Tornado, which is one of, I think, the first, maybe still the most popular like async driven Python web server. >> And so, we have this, you know, 20, 30 years of software engineering history and conceptual understanding that was very new to me when I started, but it's like a kid in a candy shop for someone who's just kind of a geek. >> Yeah, that specul I haven't heard that, but it makes a lot of sense. It's like it's better to have it and not need it than to need it and not have it. >> Right. Right. That's exactly right. And our agents are always thinking that exact same thought. >> And the So then this is only for voice or this for everything. >> Well, I don't think we would have been forced to build it this thoughtfully if it weren't for voice. >> But now everything is powered by this. >> Yeah. >> And you know sometimes like email, you probably don't need this. >> Yeah. >> Right. And you might just want to save the money on your little rag lookup and just wait because a 5-second email response versus a 3 second email response is not a very big deal. But in voice, 5 seconds and you know 1.5 seconds are very different. >> And so you want to make sure that you're doing everything in parallel. >> Yeah. Because real time is it's always been a hard problem in ML in general. I know that when it's whether it's like real time fraud detection or real time recommener systems, right? or now real time voice. It adds this level of complexity to an already complex thing. We were talking about how complex the voice is and that like rich medium but how we all have our own ways of talking and now >> to add on to that this whole piece of we need to make sure that we're getting the right information at the right time. Am I thinking about this correctly with decision tree type of thing like that style? I think conceptually if a decision tree would be useful to a human then it might be a useful concept to one of our agents. >> Similarly, if a standard operating procedure would be useful to a human, it might be a useful concept. But the ways that our agents are architected is probably more similar to how you and I would learn how to complete a task or how to help someone with a task >> than sort of a decision tree in the sense of traditional software and some of the legacy chatbot systems. >> So it's useful concept in that it models a flow that you might want to follow. >> Yeah. But a lot of times I think our agents are more correctly described as kind of goals and guard rails where you say, you know, the goal right now is to help the agent uh reset their radio or help them make a payment. And you mentioned fraud detection uh and recommener systems as other things where real time matters because there's often money involved and like click-through rates matter a lot and conversion matters a lot and that's true for a lot of voice conversations as well. So, for example, if you're, you know, uh, thinking about whether you want to maintain your subscription to a streaming service or whether you'd like to cancel and the agent can make you an offer for a 50% reduction for 3 months, but they take too long thinking it over, it starts to get frustrating and you're like, you know what, I just want to cancel. So, these things matter a lot. And so the goal might be, hey, you want to uh, you know, save the customer from cancelling, but some of the guardrails there might be around the types of offers that you're allowed to provide and uh, the experience that the customer should have and, you know, making sure you're telling the truth and only the truth and all that stuff. And so, >> you know, decision tree, flow diagram, etc. These are all useful concepts for just thinking about operations. The one that I think is the most aligned with our architecture is probably this goals and guard rails. Yeah. Architecture. >> And it does make sense now, the constellation metaphor in a way. It's like a web of different models in different ways as opposed to some straight path that you take a decision tree on. >> This is going to go, you know, to the side. I won't say it'll go over people's