Building AI Agent Workflows with Semantic Kernel

Key Takeaways

Okay, now I have a single agent. That's great. I can interact with it. I can ask it questions, but I want it to start doing more complex tasks. Kind of one of the easiest is up in the top left corner, the sequential. I can now orchestrate multiple agents. Hello everyone, welcome back to the open at Microsoft show. I'm here with Evan and Vic. welcome to the show and we're going to talk about seam kernel on the AI agent era and Evan let's go straight to the point what's semantic kennel and how that can help people to build AI agents >> so semantic kernel is an open-source model agnostic SDK that enables developers to orchestrate robust AI agents and multi- aent workflows with enterprisegrade flexibility. So, we really want to provide a a an a G and open-source, you know, that enterprisegrade uh uh SDK for folks who really want to get started developing their agents and their applications uh really quickly. And so, we want to meet developers where they are, provide the necessary abstractions for them just to hit the ground running and developing without having to focus on everything underneath. >> That sounds great. And let me share here your slides. Uh V and Evan and I love that what's a Asian because that's what everybody's talking about now. That's right. Yeah. So it's interesting to see kind of how agents have evolved as as large language models have improved. But in the era of the AI agent, uh we really look at kind of an agent as a persona. And so you're giving this agent a task, some instructions. Uh you're providing input to the agent. Uh and and what really makes it an agent that can go off and do something useful is ability to use these tools, allow it to have a memory or context or uh you know, actions. And and what what is really cool is that the tools are are native code. And so you can bring your own tools, your IP and either Python, C or or Java. in in regards to semantic kernel and you're allowing that agent to go off and then work on your behalf. And so an agent scope to delivering weather or currency exchange or something like that can go off and call APIs and do all this stuff and give you the result and it really enriches your your AI application by having that agent do that work for you. I think yeah that's that's that's one way you know we we like to to to explain it in semantic kernel and building an agent in semantic kernel is really easy and we wanted to show you that uh construction in in one of the languages that we offer Python you can see in just these few lines of code we are giving an agent the a plugin as we referred to or a tool we give it a name and we give it instructions and we invoke that agent we get the response and it the with the large language model now and the tool calling to to the extent it works out now and so much better than it did before it's able to get this the response from the context what is the weather like today in Seattle calls into the function and gives us a result so getting started super quickly a couple imports a package install and this is similar both between Python uh and C and Java so really want to make the developer experience uh you easy. So there the barrier to entry isn't really high. This is all it takes really to start get, you know, interacting with an agent in a large language bottle. It is that's a good point, Evan. Some people think that's uh is a is a great effort to create a agent, but he just showed that like a few lines of code you can do it. And I guess what I really like of the open source project is that that brings that try to simplify the the complexity as well. And I think there's something to show us how you can get this out of the box templates or also one that you have on the project to go on more complex agents. >> Exactly. Yeah. So in in these uh in these orchestration patterns that we support and we have these samples in our repo and we'll point them to you in a little bit but really okay now I have a single agent that's great I can interact with it I can ask it questions but I want it to start doing more complex tasks kind of one of the easiest is up in the top left corner the sequential I can now orchestrate multiple agents I put together and they don't even have to be that track completion agent they can be Azure AI foundry agents. They can be uh Azure responses agent. They can be assistant agents. You could technically write an agent on your own if you wanted to. You could extend our our agent base class. And now you're bringing in this concurrent pattern. Oh, I want these agents to go off and do research for me and give me a result. or I want an agent to actually know about other agents available and maybe I have a triage agent and it's handing off to a different agent for me on my behalf because of the context of the conversation. A really popular one is group chat getting these agents together and and having them go off and work together and producing a result. So on top of that is magentic, you know, uh built off the the autogen magentic pattern and bringing in the ability to replan and refocus and having these agents and kind of intelligently work and not just say here's the result. Oh, that's not what I asked for. It knows it can help figure out based on the context. And a really strong one that I absolutely love is this workflow process. And now we're not just kind of stuck in prompting and saying, "Agent, go do this for me." we're able to give it these more guard rails. we're able to say work from step A to step B in this directed graph sense and it just really opens the doors to these more complex scenarios and you're able to orchestrate these agents you know in the stateful event driven manner which is is really powerful >> that sounds great and like as ko as a I see myself as a framework so means that the framework's providing functionality and features that simplify the life of the developers of the know the agent builders and I think Vic is going to show something for us very soon like how that works but how you see that how some can is helping the developers how we get the feedback from the community what they say about the project. Yeah, sure. Um, I can come in and show a real example of of that, but before we get into the practical aspects of things. Um, so what I heard a lot from customers and partners is that, you know, building Asians is hard and why is it not magically kind of work? Um, they they need to do kind of a lot more kind of tuning and all that all that all that things that they need to do to to make Asians behave. So, it really comes down to these challenges. um you know you you trying to ground your agent so that it's useful to your day-to-day operations your your business value. So for example what I do um dayto-day I'm running this you know program managing a a series a webinar series and what I do is converting uh a video so recorded videos into a blog post um so that it can be you know consumed in in in in writing form uh in reading form. So it really sounds promising. However, what you know what what sort of Evan kind of pointed out um is making Asian to to to behave and and have those guardrails is quite um important to make sure that it actually achieves your business goals. So first the main challenge is that agents um you know initially when when we talk about chatbt and and and the previous uh kind of incarnation of agents is that it's very much uh stateless context as in you you put in uh a message you know in a in a chat and then you get an output. Um but it it loses the context as you go along. Um and you know what we need to build with Asian is that it has that stateful uh ability. So it can carry context across steps um so that we can make sure uh it understands sort of the history and the context of of the previous interactions. And second is control. Um is that you're not just calling a function. You you're kind of giving some agency to your agent so that it it it kind of handle things um with certain certain thing but you also want to be able to um ensure that you guard rail them. Um so we we need to to come up with that you know solution. And then number number three is the p predictability of that. So agents don't always behave kind of the same way. So especially if in the context of business process typically things are quite rigid. So especially when powered with LLM because there's so much data and and training set um that that you go through um you need to you need some sort of strategy to constrain and and observe and debug agents. So here's uh an example of how agent can be built. Um, so we have a, you know, I call it semantic clip, which is an agentic system that takes a video and then convert it into a blog post. And then as a bonus as well, you can see there's ability for you to publish that blog post with with MCP, which is, you know, what's what's hot at the moment called model context protocol. And I'll show you that uh in demo here. As you can see, it's quite deterministic uh in a sense that you you give the agent some uh freedom, but at the same time, you guardrail that process and it carries the context from from preparing the video and then transcribing it uh into, you know, a text um and then you generate it and then you do evaluation and so on. So, let me show you quickly what it what it looks like. Um so we can here we can see here this is a you know my uh web page here. Um I can refresh it and I probably we can you publish and upload a new video here. Um, so I'm just going to uh choose one of these videos. Uh, the one that less than 10 meg going to process that video and it generates a a blog post and it's as simple as as that in terms of the usage. U but in the back end uh it's built with semantic kernel and has all those process from framework and workflow. So when you upload the video that the beginning of your workflow and like the first step of the workflow and they probably using something to generate the blog they calling a model. >> Yes 100%. So in the back end what what we do um is first you know get the the path of the video and then it transcribe it into um this text here. Right? So that's transferred as you can see here. Um and this uses um LLM um of an AI Azure Open AI model called Whisper. Um and next is we use another model uh to generate the blog posts. Um as you can see here um it's also uses LM but it's a separate uh kind of agent uh that you build on each of these um uh kind of process. So that each of these um has LLM associated with it. So for example, if you want to use a different model to you know transcribe the video and then you have a different model to uh to generate the blog post, you can. So you can mix and match and that's the benefit of using semantic bunnel is that it has that ability to you know plug in different different models as you see fit. So it may not be LLM, it could be SLM if you want a small language model. um earthwide. >> And I I just want to build off really quickly what you said is taking a business process in something like this that potentially a human was doing all these steps by themselves and then coming up with an output. Now we're automating it with AI. We're building the workflow. We know as we use semantic clip, we're going to go from the beginning to the end and I'm going to get a blog post. Um but this is a great example of like a really linear flow. you know, semantic kernel and other steps supports fan out, fan in. You can send off these concurrent patterns and let, for example, in a research scenario, let agents go and research. Then you have an aggregator or some other step that brings that research into something more tangible like a report. Um, and so you can start to build these and model these workflows human in the loop. What if I need a long running process and I need to stop it in a in a kind of a a scenario where someone needs to actually say yes or no? That's perfectly valid. You don't have to take AI and make it 100% AI, but all of a sudden we're bringing in the business process and and reducing the cognitive load of someone manually doing something before and it's still allowing someone to say yes or no because we're not at the point yet where we can just click a button and it all works auto, you know, automatically. >> 100%. Yeah. So, in this case, what I have is, you know, um exactly what your your point around um ability for you then be able to publish the blog post if you're if you're happy with this, right? Um you can then approve the blog post or you can regenerate it um and then if you're happy then you can just submit it um and it will um you know be published into a blog post using MCP in this case. Um, so I've got an example already. Uh, one of these things, um, that we we do is as soon as you click that button, um, it will publish that to a, you know, a tool, um, a service. It could be GitHub, it could be, you know, other other tools, um, or other services. And then you see your blog post, uh, being uploaded there. Um, so that's pretty much uh, the process uh, what we uh, wanted to demo quickly. >> That's amazing. We I like that that decision making that step-by-step process that's clear and how complex is that like how long it takes to create something like that. Yeah, it took it took probably, you know, less than a few days to actually come up with that. Um because with sematic kernel, it's such an easy way to kind of get all of these various building blocks if you will uh put together um because it's already got the you know the the facilities and the frameworks uh around it. Um so what I do is basically just come up with the steps and each of these steps then um com combine to be a process. Um and you can do a lot more advanced as what you know Evan mentioned uh around adding some human in the loop. You can sort of come up with a a fan in and fan out sort of uh method to basically hey go to a different department to to review the you know uh the blog post uh and then once you agree with that or maybe some make some modification you come back uh and then come up with the you know summary of all the changes um to to be approved by the content manager and then you publish that in in a blog post. So this is just a simple demo but there's a lot more that you can do with semantic model. >> Sounds good. Do I share the project web page and maybe Evan can talk a little bit about um how someone from the commit that want to really jump in on the project and collaborate contribute to the project. >> Yeah. So we can we can see the the semantic kernel GitHub repo here um on the screen. And when you visit this repo, um, we ideally would love for you to check out the code, start building with it if you haven't, uh, started already. We have many get start getting started, uh, with agent samples, getting started with processes. The biggest thing too is come and discuss stuff with us. Ask questions if things aren't clear. File an issue if something doesn't make sense. Our team regularly triages issues. Um, we can talk about issues in weekly office hours. we go through discussions, it's really a great place and and when we find that as more people interact with discussions or issues, people are coming back later when they say, "Oh, I get the same issue. Oh, it's already solved. It can get, you know, unblocked much more quickly than kind of scratching their head or or something just sitting there." Um, the team is is really open to this feedback. We we build this product for the community. Of course, it's used by uh you know, teams within Microsoft. It's used by lots of companies outside of Microsoft and some companies love to talk about their journey and we have a dev blog that also shows what they're building and things like that and it's really cool to see these these different applications. But as as we see here on the screen the the public community office hours every Wednesday at 8 a.m. Pacific time in the US. Um come in and ask questions. Tell us what you're building. show show us your projects whether it's through show and tell on GitHub or just showcasing it in office hours and and really we love to keep that openness um just discuss with us and tell us your ideas and we're we try to be as open as possible about our ideas and our road map and what we're building over the next six months um and so that's that's kind of the exciting thing about being open source is we can do that it's the goal >> amazing I just want to add a little bit that we also have APAC version of that uh that's happening every month uh I think second uh Thursday of every month um and uh you can see you can download the calendars um into your you know in calendar straight away from here >> I think that's it we did show very well the project is a very important project if you didn't know about the project I recommend you to follow also follow the open at Microsoft show at the Microsoft developer channel. Thanks a lot for you two, you know, for showing up and showing that project. I know there's a lot of new features and news coming very soon. So, it's a good timing. Everybody talk about NCP agents and um that's the time for anyone that looking to learn and and grow on AI to follow some ko. Thank you guys soon. >> Thank you so much, Ryan.