How we ship models in VS Code | LIVE161

Welcome back. We're here at Microsoft Build, and I'm super excited to be here with Julia. How you doing, my friend? >> I'm doing fantastic. Thank you. >> The thing I love about Build is we get to talk about the cool stuff behind the scenes, and we get to ask all sorts of questions. We have a live audience here as well, so they might ask some questions. Why don't you tell us who you are and what you do? >> Cool. Um my name is Julia. I am a product manager on the VS Code team. I've joined the team last year in August, and yeah, it's been a fun ride. >> But the thing is is that you just don't work on VS Code. Like let me paint the stage. We're in a gen tech coding era. Like for me, it started last December, and we we kind of go and we pick the model, and we just happily just go along and pick a model, and it seems super easy. But that's what you work on. >> Yes. >> It's not as easy as it sounds, right? >> Yes. So, I thought the same thing. Funny that you say this. I'm like, you know, I'm in VS Code, I'm using the chat extension, and from time to time, like a new model shows up, and I'm like, oh great, I'm just going to use the new one. Um now, since I joined the VS Code team, I'm in charge of or I'm helping the team, not in charge of, but I'm helping the team. It's very collaborative. Um what all goes into releasing these new models in our VS Code extension. And it is a ride. I can tell you that much. It's not as easy as, oh, we get a new model endpoint, and boom, here it is, and it just works. There are so many things and pieces that go into this from us working very closely with the model providers and like getting their insights and what the new model is all about. So, yeah, there are a lot of things that go above and beyond here. >> So, I'm just picking the thing, and I'm like, these things are all compatible or the same. Can you give us a sense for how not the same? These are all like all very special flowers, whether they're for a certain company or from a certain type. Tell us about that. >> Yeah. So, I think the biggest aha moment for me was even within the same model family, let's pick the GPT ones. Um GPT to 5.5, even within this model family, it kind of had its own new personality. And we had to adjust our coding harness to it. And that's why we work very closely with them because for us the models are also a black box. Um, so we are getting their insights like what has their research team on work done in making the new model endpoint better. So truly is like every new time we are getting a new model, it has its own personality even more across different model providers like Anthropic and OpenAI. These even differ even more. But even within the same model family or like model provider um section, it's already very very different. >> So we talk about models even from 5.4 to 5.5. You talk about personality or characteristics, right? And you talked about the harness a little bit, too. Do you have to make changes to the harness for every model? And if people don't know what a harness is, can you help them understand what a harness is and how new models make it so you have to change some things? >> Yeah. So I brought a little bit of a >> Ooh, ooh, let's go to the screen here. >> flow chart diagram. >> Okay. >> Um, just to give a very high level of hey, these are all of the steps that we have to think about. Um, as I mentioned, we work very closely with the model providers. So it's very I thought it was very processy. Oh, yes, of course we learn about this new process. It's very casual. The model providers reach out to us. They are like, "Hey, our research team told us we have something that is working." And then our entire team from the Copilot API team that gets the new model onboarded to to also us and client-side like VS Code, we all go in and we start working. And in particular, we work on optimizing the harness. So what is this harness? Um, the harness itself is the system, the environment your AI or your LLM is running. Essentially, the LLM only provides or gives back tokens. But how do we make changes? How do we edit files? How do we run in terminal? So this is what we consider our harness and these are all of the changes we have to make sure it works once a new model comes up. And there are four very important things we always look at whenever we have a new harness. It's our system prompt. So for every model, for every different model across even different model providers, we have our own system prompt. We have different built-in tools that these models can call and even across different model providers, they tend to call different tools as well. So we have to make sure the right tool sets are being called and then also context management. Um different models have different or work with different model context. And then also the agent loop and how do we call it? How do we load them tools? So we just have to make sure it works end-to-end. >> on. So you effectively, every time you get a new model, every model gets its own special flower kind of situation in the harness so that for us as we're using this, we really can't tell. >> Yes, correct. Well, actually, so let me switch over to my stable. >> Oh, can you hit control plus a couple times? >> course. There you go. >> There you go. >> All right, so um this is a fun little project. It's called DJ Julia. >> Control plus one more time cuz I I'm I'm old. There you go. >> It's just like a fun little project that I have. Um so right before this session, I created this prompt which is "Hey, for my DJ app, I want you to create a new music pattern for me." And just to kind of show you first of all, everyone can go in and explore what the harness does whenever we make these calls to the LLM. So you see a bunch of styling. It does a bunch of different things, right? It searched. So whenever you switch to our uh chat debug logs, you can really see all of the different things that is happening behind the scene. So, >> Hold on, you went really fast. Let's pause for a second. So, there is a chat debug log where you can see effectively every step the harness has taken. >> Exactly. Every step, every tool call, what is the system prompt that is being applied. So, here for example, let me close this real quick, and you can see the request message is being separated between the system prompt, and you will notice, so this one is with the GPT 5.5 model. If you would do the exact same prompt with, let's say, Opus 4.6, it will have a different system message as well. That's how we optimize and we make sure every new model gets the latest and best together with the model providers that we work on because they clearly know the models so much better. And it also helps us to really optimize and make sure it works truly end to end as well. And then also, if you're interested in seeing all of the different tools that are being called as an example, the GPT model family always uses apply patch versus the Anthropic models they use insert insert files. just some nuances how different all of these models truly are. >> So, you talk about like how different they are. Is there a way you could show us like how they are different? >> Um sure. Let's I mean, let's try this here. Um So, use this. Let's go back. And for everyone to see, I used this using the GPT 5.5 one. So, let's just do this with the Opus 4.7 one. I'm doing the exact same prompt. So, first of all, also another challenge we have in the AI space, even using the same exact same model, it is non-deterministic. So, every time you run this, even within the same model family, you will always get a different input or output. Um but just to show you how much they differ, even across um within the same harness, and just having the different model here. >> Yeah, and because I mean this thing is sampling, you know, uh according to the distribution of the of the models output. I like to say stochastic process as much as I can. It's like a $5 word, makes you sound really smart. I don't know what it means all the way, but that's what they told me. But effectively, this is all probabilistic, and so it's going to be different every time, even within the same model. >> Exactly. Yeah. >> Okay. So what did you So what do you got? >> So all right, if you remember what we had before, our system prompt, let's go here. There you go. So this was the system prompt we had previously. Just going to close this. >> Uh-huh. >> Um it does like a bunch of um pre-classification checks and stuff like that. So now if you go to the latest one that we just did with the um uh Claude one, you can see the instruction files are different. With the GPT 5.5, we had something to make sure it inserts before, because the GPT 5.5 ones they are more prone to immediately go and run in terminal. Um but we wanted to hey, first explore a little bit, and then go and run it, because we have seen so much more success with these kind of system prompts, rather than using the ones that we previously had with 5.4. And all of this was worked together with the OpenAI folks. >> And this is This is really impressive, because like betwixt us, friends, I built my own harness in uh Rust, and I started changing models assuming that it wasn't working, and things just stopped working. So you effectively have to when it when a model comes out, it's like, all right, gang, let's get together, let's figure out what the right prompt is, let's figure out how they do tool calls, etc. You have to do all of that work. >> Exactly. Yeah. >> Okay. >> Um I I mean, I love that you're building your own harness, but one of the benefits for using our built-in one, and there's no harm in building your own, and of course making your own changes is we we do work very closely with them, and we are truly truly optimizing for all of the new checkpoints, and we are optimizing in a way that it will hopefully work for most folks out of the box, and you don't have to worry about it, because we know how much work that actually goes into always updating and making these changes, and that's one of the benefits for using GitHub Copilot in VS >> 100%. My my harness is nowhere near as skillful as theirs, and it's literally for helping me write stuff. And so, notice that even for a separate task, does your harness have to be different, or is this all optimized for writing code? >> That's optimized for writing code. That's why you see in the instruction file usually usually the very first one is hey, you're an expert AI pro programming assistant. We're so we're putting the model in context of where it is. It's in a coding environment. It has to make sure it is not just like giving you text. Um it is going in. It is using our built-in tools like run in terminal, all of these things to make sure it works in the environment we're kind of grounding it into. >> So, I'm assuming you're not like oh, we got a new model everybody. Let's just use the last one and change a couple of words, see what it does. I'm sure there's like a better process to do this kind of testing. Can you tell us that? >> Yeah. Um so, to be very honest, um a lot of the times whenever we get a new checkpoint, the very first thing and this is something we get from the model providers. They tell us just use the system prompts from the prior model, and let's just see how it goes. Um and we're like okay, I guess. Because they're also So, the reason why they want us to do that is they want us to see how good it already is without doing all of the optimization. Yeah. And how do we measure this? So, we measure this by running a bunch of offline eval benchmarks. We do this by internal dog fooding. So, these are two in parallel things that are going on. And once we have these benchmarks run, we share the results with the model provider, and then based on that they're like, "Oh, okay, I see. It's already maybe better, but let's start tweaking, and let's start going in. How about this?" So, then we sometimes have hardest like a model where we basically just removed everything, and we're like, "Okay, let me have to start from scratch here." And then we optimize it, and optimize it to make sure whenever the model launch day is, it truly is the the the perfect model to hit the ground crowd. >> So, but I but there's got to be like, for example, I'm I'm a traditional dev, I did some machine learning. We had things called evaluations, we do things as devs as unit tests. Do you have something like that that you systematically run through to improve the prompt or see like, "Hey, it's this good, but now it's this good." What what does that look like in your process? >> So, our process here, and this probably is going to touch a little bit of evals. So, yes, we have offline evals. Especially, we run these before the launch. After launch, we also still optimize the harness, which is very important. It's not never a ship and let's go. It's always like we go in, and we iterate even after. So, what are the benchmarks, or how do we evaluate it? There are a bunch of public benchmark out there. Sweet bench, one of the very known ones. And we still run these, even though there are a lot of conversations, controversy around these public benchmarks. One of them being, and the model providers have told us directly as well, the the models are so smart now that you're always kind of going to get this. So, use this as a regression test rather than really evaluating the quality of it. So, we still run these, we share it, we see if the model has regressed. We use it for optimization of our harnesses as well. But then, the more important one is we use or we have our own internal benchmark, which is called VS C bench. It currently has 100 and plus tasks or instances that we run every time. So, before a launch, we run across different reasoning efforts. We make some changes to the system prompt as an example. We see how it changes and we look at these what we call model report cards to see how it's performing against it. >> So, you're not just like yellowing a model out. You have a full bench benchmark of a bunch of stuff that you're testing. Can you show us a little bit that the card that that doubles out? Show us Show us that. >> Awesome. Um So, I did this this morning. >> Can you control plus a couple times? >> Yes. >> You're good. Thank you. >> So, I ran this this morning and basically how it works is we have a VS Code Insiders instance in an ACA. So, whenever we run our offline e-votes, we spin one up. It's always using the latest. So, I ran it this morning so that truly has the latest Insider version in it. And how I did this, I ran it against GPT 5.4 two times and GPT 5.5. Typically, we try to run it five times because of the non-deterministic problem that we have with AI. >> Um so, then our next step, this is where it gets really exciting, is the performance metrics. We look at the resolution rate. So, resolution rate is was this task a successful or not? And we evaluate this by certain assertions we have and we think of it like a checklist and it goes through and says, "For this test case, was it resolved? Did it hit all of the check marks?" Yes, and then it's resolved and it goes in and shows 153 out of the 248 test cases were resolved. Um and this is how we can compare across the different model families. >> This is pretty amazing. So, what should developers take away from this? You got about 45 seconds. >> Um First of all, always keep trying new models. Give us feedback if there's anything that doesn't work as expected. We are continuously operating and doing this. And also, I've done the same. I gave a model a one time kind of thing. But because we are putting so much effort in um these models, just make sure to maybe sometimes come back um and give it another try. And also, so then Evals offline evals are huge topic and anybody building their own coding harness, it has been one of the tools that we has we as Microsoft have been super successful and the model providers are loving. So definitely a space to um dig more into. >> Awesome. Well, thank you so much for being with us and thank you so much for watching. We'll see you after this. >> Thank you.