GHU: Building AI Agents with VS Code and GitHub

Key Takeaways

[music] Hello everyone. Thanks for joining. My name is April Gittens. I'm a principal cloud advocate here at Microsoft. And today I'm going to be sharing with you a workshop that we did this year at GitHub Universe. It's combining the best of both worlds. On one hand, we have the AI toolkit extension, which is a really great extension. If you haven't installed it yet, you should. It's really helpful for creating agents. You can also do tracing. You can also do evaluation as well. You can explore models. And then on the other hand, we're going to be combining that with GitHub Copilot. I'm sure if you're tuning in to the stream, many of you are already familiar with GitHub Copilot. So that's great. If not, it's okay because we're going to be diving into that a little bit. What's really awesome is that just recently the AI toolkit extension got a bit of an upgrade. Whereas GitHub Copilot has AI toolkit tools that are available that can be invoked to help you do a variety of tasks whether that's getting model recommendations for any AI system or app that you're building. There's also um the ability to create the agent as well and you can ask GitHub copilot to create an agent for you using like the Microsoft agent framework. You can also enable tracing in your own app. So, if you need some help with how to just put tracing in your app for your agent, the toolkit um combined with GitHub Copilot can actually do that for you. And we can also do some tracing um as well, evaluations. So, there's a lot that you can do with the extension. And there's even more you can do now that there are tools from the AI toolkit available with GitHub Copilot. I'm going to take you through some of those today. And also, by the way, in the event you want to try everything that I'm going to share with you, there is a repository where we have the instructions for you to go test all these things out and you can do it on your own. So, without further ado, I'm going to migrate over into Microsoft Foundry because that's where we are going to start today. All right. So, here we are in Microsoft Foundry. Why are we here? Well, we're going to be using Microsoft Foundry models as we're creating this agent. And so, the very first thing you'll want to do is just make sure that you have everything with your environment set up within Foundry. That means creating a Foundry project. I'm already in the new Foundry experience. We announced it recently at Ignite. And within here, what you'll want to do is create a project. That's all you're going to do. Create a project. Once you have that project created, we're going to head over into Visual Studio Code. I already have things set up here, but let me take you through some of the things I want to call out so that way you're aware that you have these as well. Starting off over in the extensions, there's going to be two extensions that you're going to need. First is going to be the AI toolkit for Visual Studio Code. That's the fabulous extension I mentioned in the beginning. When you uh search for it, it will say AI toolkit, and this is what it looks like. So, make sure you install this one. The second one that you're going to want is the Azure resources. We're installing this extension because this is how we're able to get access to the Foundry project that you would have created in that first step. Once you have both of these extensions installed, great. What you now need to do is actually sign in to this Azure resources extension and then from there you can set your default project. I'm already signed in, but I'll take you through what that looks like more or less. So over here on the left hand side once you're logged in you'll see a dropown for your subscription you're going to scroll all the way down to the Microsoft Foundry section. And in here you'll open that up. You're going to find your project. And so for mine I'm using my um GitHub Copilot demo one. You're going to rightclick and then from there you can open it in the Microsoft Foundry extension. The Microsoft Foundry extension is part of the AI toolkit extension. It is an extension pack. they come together, they're best friends. And so once you're in this extension, what you'll need to do is um at the very top, it'll show you that your project is now set as the default. So that's how you get it to be set as default. If you make a mistake, don't worry. You're going to rightclick on that and click switch default project in the Azure extension. Okay. Now that you have your project created, you have the extensions installed, you have your default project selected, the next thing that we want to do is dive into GitHub Copilot. So with that said, let me open up GitHub Copilot for you all. All right, so here I'll expand it for you. GitHub Copilot, if you haven't seen it, this is what it looks like. We're going to working in agent mode because those AI toolkit tools that I mentioned, they're available in agent mode. There's a couple of models that are available for use that we recommend that you use if you're going to be leveraging the AI toolkit tools with GitHub copilot. With that said, here in the model picker, um the models that you can use are going to be GPT 5 mini. There's also um any of the four Claude models. So either Claude Haiku 4.5, Opus 4.5, Sonnet 4, Sonnet 4.5. So if you have access to one of those models, I believe five mini might be free, but if you have access to any of those models, those are the ones we're going to recommend using when you're trying out GitHub copilot to invoke those AI toolkit tools. All right, all that's been set. Now, let me show you one of the cool things that you can do in here. I mentioned earlier you can get model recommendations for something that you're building. for GitHub Universe, we created the cutest pet planner agent, which essentially it uh it gives you recommendations of things to do with your pets. And so before we kick that off, we needed to get an idea of what model should we even use for this, right? So what you can do, and this is what I did p personally, is pass in a prompt into GitHub copilot. And what you can do is you can just give it some context as to what it is that you're creating. And in doing that, what it'll do is recommend a couple of models for you to choose from. So with that said, here is the prompt that I passed in. Again, this is in the repo, and I'll share that link towards the end. But this one says, I want to build a pet planner agent. It jobs to sniff out the perfect play date. It should check the weather. It should get activity ideas and it should point to the best spot in town. And I just want to know well which language models would it recommend for this scenario and why. I also want to know the tradeoffs uh things like reasoning ability, cost, latency, context length and I need this information because I want to make a really informed choice. So again in GitHub copilot I'm in agent mode. I personally love using cloud sonic 4.5. So that's why I have that one selected. And then from there you can go ahead and submit that GitHub copilot. What we should see is that it's going to invoke a I believe it's a model. Yep. Get AI model guidance. This tool is from the AI toolkit. And what it's doing is it's using this tool to help with forming a recommendation for you on which model or models would be best for your given scenario. So, we're going to give it just a little bit longer to keep working because it's working right now. But once we have a response, more likely we'll see three different options available for models. Um, I'm going to cheat a little bit and say GPT4.1 Mini tends to be the model that it recommends often when doing this. I'll see if it's going to give me that response. Let's see. Let's see. Let's see. And if not, I'll try another model just to see if we get a response a little sooner. But essentially, what's going to happen here is that we're going to get uh three different models recommended. And for each of those models that are recommended, it'll let us know things like uh cost, like something to consider like cost or maybe things like latency. And in addition to that, I'm trying to think what else would it recommend. It might also um it'll it'll it'll basically base its response around the provided scenario itself. So this seems to be going just a little slow. There we go. You must have heard me talking about it. All right. So we have our recommendations and now we don't have to use our imagination anymore because we have the response here. So starting off at the top for example it's top recommendation as I told you GPT4.1 mini and again this is for the scenario that I passed in here we get information about speed cost context as strong reasoning but it also gives us the tradeoffs I don't want to know all the good stuff about it I need to know the things to maybe consider a little more gpt 5 nano is another option that it gave there's also 54 mini um instruct and then below that we have a nice little comparison matrix as well. And so if you are overwhelmed by all the models that are available out there, I'd recommend using GitHub Copilot with the AI toolkit tools to help with figuring out which model you should go forward with. Now from here, you got some model recommendations, but like can you test these models? Absolutely. AI toolkit provides that ability. So if we go to the AI toolkit extension and within here we have the model catalog. So in the model catalog and I'll just give us some more space here is where you can find all the models available. We have GitHub hosted models, we have foundry models, we have models from Enthropic, we have models from Google, we have models from um uh I think hugging face. So if you're doing any like uh local models, you're going to find them here. All right. and we're adding new ones. And when new ones become available, most likely you'll find them here. And within here, what you'll need to do is find the model that you want to test. So in my case, it was GPT, let's see, 4.1 mini. And so once you found your model, you can then try that in the playground. Playground is going to be where you can actually go and test it. And so from here, I'm just going to migrate on over to that playground. It's called the model playground. And once you're in here, this is what that playground looks like. Now, if you recall, GitHub Copilot recommended a couple different models. If I wanted to compare more than one model at a time, I can do that here in the playground. And in order to do that, what I would need to do is click this compare button. I don't have any other models deployed right now. So, I would need to browse to find one. And let's say I think it had said like five nano at some point. I'll just use the GitHub hosted one that's here. So, we'll add that model. And so, what we're going to now do is compare the two together. So, here within the model playground, we are able to compare two models at the same time. We can pass the same prompt in. And I actually have a system prompt that I want to pass in. You can add your system prompts at this stage. If you go to I'll go back to it's this little icon that's a a little panel where you can add in your system prompt. Place that in there. Place it into each one. We'll close this up as well. It's the model preferences. And then from there you can type in your prompt. So the prompt that I'm going to pass in that I would use just to test is it's raining today. What should my dog and I do? Notice it populates into the other one. Once you submit that, you'll get a response from there. All right. So, 4.1 Mini, right off the bat, it was like, I'm here. I'm present. I'm awake. I'm here to help you on your pet. And we already have an answer. And in this answer, it's following up with me on um on uh some more information, whereas GBT 5 Nano just pretty much like dove right in to just recommending stuff. kind of want to get I want to make sure it it has an idea of what's happening with my dog. And so from here, you could follow up with more questions, see how each model does its output. At this stage though, I'm going to move forward with GPT4.1 mini. That's that's my homie. So from there, we then need to create the agent. I mentioned you can create the agents with GitHub copilot using the AI toolkit tools. In addition to that, you can also use what is called the agent builder. The agent builder is part of the AI toolkit and it enables you to create agents in this UI experience directly here in VS Code. I already created my pet planner agent that we see here on the screen. But this is what the agent builder looks like. In here, you can give your agent a name. You choose the model that you want to use. The instructions are going to be the system prompt. So, this one's quite lengthy. What I recommend the repo is where you can find all the system messages that are used if you're trying this out. And then just below that, we have everyone's favorite thing, MCP. We have a tool that's been added to this agent. It's actually a custom local model context protocol um server that I created. It's available for you to use as well. It is equipped with mock data. It's actually hard-coded data because this isn't an actual live server that's hosted somewhere in the cloud and it was for the sake of GitHub Universe. But what this server does is that it has an assortment of tools available. One gets the weather, one gets activity, one gets safety tips, and it might be a fourth tool. We'll take a look at it here in a second. But in any case, you can add that tool directly here in the agent builder. Now, how do I even create this? How does one how does a person even create an MCP server? Don't worry because the AI toolkit has guidance available for you. If you want to create your own MCP server, what you'll need to do in the AI toolkit is head down to the let me get to that section. MCP workflow. Within here, there is a create new MCP server selection. You're going to select that. When you do that, you're going to have two templates available. either a TypeScript template or a Python template. I'm a Python girly, so I went with the Python template. In that Python template, you'll give it also the server a name and then it'll open up a new project in VS Code for you. I'm actually in that project. So, we're going to take a look at what it actually gives you. First and foremost, and I think uh the most important thing is that we give you a readme. And so, this read me that's available in here, let me go into preview. It has all the steps you need to get started. All right. So hopefully you don't get tripped up and and stuck on creating the MCP server, but more important, we actually give you this template that's already filled out with the necessary information that you would need to create a server. And so from there, follow what's in there to help with creating the tools for your server. There's some environment setup that you'll need to do as well. And so that's available in the readme um too. And once you have that set up, from there, what I then recommend going to do is heading into the src folder because there's a server.py file in here, and that is where you're going to come in and actually um add your tools for your server. So, I mentioned I created this server with some mock data, the server file itself also in that repo that I mentioned. So, you can copy and paste that in. It's also helpful just in case you've never created a server on your own and you're not quite sure what to do with tools and such. So, in any case, I have uh this example here. I mentioned there's like a a a tool for getting the weather. There's a tool for getting pet activities. There's a tool for finding pet friendly locations as well. And that might be it. Oh, and some pet care tips. Those are super important and helpful. So there's an assortment of different logic in here that relies heavily on using random to select one of the items in the list at random from all the fake data. But in any case, once this server is uh good and ready to go, the server py file at least, we're going to use the um the debugger, the run and debug here in VS Code to actually start the server up. You can debug in the agent builder. That's that fabulous place we were just earlier where we created the agent. When you're in the agent builder, what it's going to do is uh once you start the debugger, and I already have it running right now, so I'm not going to rerun it. But once you're in here, it'll open up the agent builder. You'll make sure you are at your actual agent that you're testing things out. So in my case, the pet planner. And then from here, you'll come in and you'll add the server to your agent as well. to add tools um or even just the server, there's this add tool button. You have MCP server or a custom tool. That's how you go in and add that. One thing I want to call out here, once I added this tool, I actually went back and modified the original system prompt for my agent. There's a cool thing you can do within AI toolkit to help you out with system prompts. They're tricky. I get it. So, there's this button that says improve. If it doesn't say improve, there's one that says generate. If you have no system prompt here and you're just trying to figure out how to do a system prompt, click on that generate button that'll be there and then you can actually describe the task of your agent and the AI toolkit will generate a system prompt for you. 100% recommend reviewing that system prompt before you move forward just to make sure it's it reflects what you wanted to say. However, if you have a system prompt in here like I do, what you would do is click this improve button and then it'll improve that instruction and you'll describe what you want to change. I put in the repo. What I recommend passing in here at this point to make sure you get a good update. The major change in the system prompt is that it takes the tools into account for the agent to utilize. Now, once this is in here, you can start testing out the agent. Um, again, so I had put in my pool and I Los Angeles. What should we do today? And so it wants to know some information about him. All right. I like to say he's two. He likes outdoor walks in the park. What we should see here are some tools getting invoked. And I can see that we have let's see if it's going to let me show you because it's so fast. So we had the get weather tool invoked. We had the pet activities tool invoked vacation locations and pet care tips. So you can see in here that the tool is being utilized by this agent which is great. That's exactly what we want. And that's how you would go about setting this up. Now, what happens when you're ready to upgrade from doing things or I guess migrate from doing things here in the agent builder to doing some stuff with code because we love code. Down at the very bottom, we have a view code button. Click that button. You can then choose your SDK as well as your programming language and then it's going to give you the code for the agent. I already have one that's already set up. And so if I were to go to Not that one. Where are you? It's hiding. So pet planner agent. Here it is right here. This is what it looks like. Um I chose the Microsoft agent framework and then uh you can continue to chat with your agent now with code. I recommend doing that migration from the agent builder to generating the code because now you can do more things with the code. You can start adding all sorts of logic to it. But the agent builder did a lot of the heavy work in the beginning for you and now you can continue doing stuff with code. There is some environment set up in the beginning. Do recommend that you do that. Actually, it's required that you do that otherwise things won't run. And so make sure you have that set up and then you can start running. As a heads up at about um for me line 38 and onward to about 47 um actually not there sorry it's going to be line 31 to about 36. There's this user inputs um list here. That's going to be the inputs that's going to be submitted to the agent. So you can modify these to whatever it is that you want them to say. Once you know your agents working, you can do things like enable tracing that I as I mentioned earlier. And so you can use things like GitHub copilot again to invoke that. If I can I very quickly generate a new one of this so I can get this to work for you. Okay. Okay. So, like let's say this was my we'll just create this as a new one. Uh demo agent. All right. Just so we can have this. So, I say I want to add tracing to this. Tracing is going to enable you to see the steps that the uh the agent took to generate it output. Within GitHub Copilot, we're back here in agent mode. All you honestly need to say is enable local tracing for my my demo agent. py or just like my pet planner agent. I don't want it to get confused with the other file in here. So that's why I'm specifying the file. But because I already have this file selected, it's using that immediately as context. And then you can go ahead and submit that. What it's going to do is start either uh it'll install any dependencies that are missing. It'll also then actually modify the file itself to enable tracing. And what it's also going to do is start the tracing viewer that we have in the AI toolkit. And so right now, right off the bat, it gave you this command to start the tracing collector and viewer. I'll allow it. And this is what that looks like. So this tracing viewer is where you can come and view traces for um for your agents. Let me zoom out here. I have some examples of what those things look like in here. So, I have one from a prior run that I had did before. Let me just close these things out. And this is essentially what it's going to look like. Um, at the top for this invoke agent, we get a summary of everything that occurred. But you can break things down a little more. We can also see anytime tools were called um as well and what that output uh happened to be. But this is going to be super helpful for things like debugging. The last thing I'm going to show you as well, um, I'm going to have this one stop since I already have it already done, but just to show you that, let's see, did it add everything in? Here's an example. Agent mode, GitHub Code pilot came and updated my my file for me. All right, last thing. Evaluations. Evaluations. There's a lot that can go into that. If you're brand new to evaluations, not quite sure how to do those, GitHub copilot, you can ask it to add evaluations for your agent. Literally keep it as simple as that. And it'll actually go through a full workflow of helping you create your evaluations. It'll take a look at the files in your project. It will recommend evaluators. We're going to use the Azure AI evaluation SDK for the evaluation script that eventually gets created by GitHub Copilot. Um, it also creates your data set. So, if you don't have an existing data set, it will create synthetic data for you. Not only will it create the inputs, it'll also run your agent to get the outputs as well. So that is beyond helpful if you've never done evaluations. Last thing I'll just show you that I personally like having GitHub Copilot do once I have my evaluations and this is an example of what the output looks like. This is a lot going on, right? At least for me, it's a lot. So I like to then ask GitHub Copilot to create an evaluation report and recommend what's next or what I should do. and it actually gives me a really nice executive summary. I always like to throw this in as something extra to do. So definitely test that out. But all in all, the repo that we have available is within this AI toolkit samples repo. It's a pet planner workshop. All the instructions are there for you to go through uh the full workshop that we did at GitHub Universe. It takes about an hour to complete. So, just as a heads up, um, if you are giving this a try on your own, you're trying to figure out how much time uh, you should set aside to spend on this. And I mean, it's a it's it's a really great experience itself. With that said, thank you all for tuning in for this portion where we got to go through the GitHub Universe workshop that we did. AI toolkit extension, be sure to install it. Um, it's a really great extension. And yeah, I hope you enjoy the rest of the stream. Take care.