Deploying a GenAI Agent using Databricks Mosaic AI, Anthropic, LangGraph, and Amazon Bedrock

Key Takeaways

Okay. Uh, welcome everyone. Going to go ahead and get started. Um, so my name is, uh, Randy Defva. I work for AWS. And today in this, uh, quick 20-minut session, there's really a couple of, uh, converging or overlapping themes that I'd like to talk about uh, in the world of Gen AI and data. And so one is really about the direction that the Gen AI world is heading in terms of multi- aent patterns and how agents work together. And then the second related theme is you know how do we actually use generative AI and particularly agents to make working with our data state a little bit easier and more straightforward. Uh so this is the standard uh statement to not rely on anything I say in terms of road map. Um but the what I'd like to show you today is really again these two patterns of one where are where is the community moving in terms of agents what are the trends we're seeing there um and then I'm going to show you two quick demos mostly just uh Jupyter notebooks where I have some code I wrote and the first is going to show an example of starting to compose uh a simple multi- aent system using building blocks both from AWS investment from data bricks. Uh and so I'll talk there a little bit about how we start designing more mature agentic AI platform stacks. And then the second demo is really more related to that second theme about how can we start using Genai agents um to help us work more effectively with our data. So the bigger picture in terms of these trends. So, first in the world of agents, people are starting to want to have uh more standardized mature agentic AI stacks. And so, the way that that's manifesting is we're moving away from this paradigm of we build individual agents as bespoke one-off applications to we'd like to have cataloges of standard agents that we built that do certain things well and then be able to compose more complex applications by having multiple agents work together. uh and you know do that without having to write a lot of custom integration code. Um and so the first demo I'm going to show you again will show you an example of how to start composing multi- aent applications and I'll really be focusing on some of the underlying agentic platform building blocks that we use primarily from the data bricks world in order to make that a little bit easier. And then the second demo is going to be focused a little bit more in a higher level picture of how do we use a multi- aent application to help us with a a simple but realistic problem that data consumers face. When your data mesh has data that lives in multiple places and in this example it's going to be some data is in in data bricks on the unity catalog some data sits on the AWS side in the glue catalog. So just a quick recap of where agents are moving. You know, a couple of years ago, agents were more of a theoretical uh thought experiment. Last year, people started building simple agentic applications. And it was really these um first four levels of adapting agents to uh adapting large language models to work with tools to give them more capabilities and then adding some basic orchestration or workflow planning there. And that workflow aspect could be either more deterministic. So if you hear the term workflow agents, it's really saying that we kind of know what the workflow looks like and so we'll orchestrate that with a more typical workflow tool or just with our own Python code. Uh and then we move to more fully autonomous agents where you get into patterns like React or RWO. And the difference there is that we're relying on the LLM to do the planning for us. So rather than specifying here's the steps we think you need to take, we'll let the the large language model figure out the the steps it should take and basically figure out its own workflow. The thing that we're seeing more of this year is looking more at multi- aent applications. And the idea here, if you think about more complex scenarios, so the insurance industry is a great example. If you ever submit a claim to your homeowner's insurance, your car insurance, there's a lot of different steps that that claim goes through. And so you might have one agent that's really good at fraud detection, another agent that's really good at doing claims underwriting and making sure that the thing you're applying for is actually covered under your policy, and then you have a supervisor agent that kind of coordinates all of those other agents that are doing the more specific tasks. Um, but this today is still pretty early. And one of the reasons is that we don't want to have to keep rewriting all this boilerplate integration code. So the um and and again this just kind of illustrates that idea that in one pattern we have more deterministic workflows than autonomous agents can really think and reason and plan themselves. But then in reality a lot of times the uh the more comp more realistic problems we end up with a hybrid pattern where some parts of it we allow the agent to do the planning some parts of it we uh we kind of give the agent a little bit more guidance. But again going back to this world of multi- aent you know right now if you do a multi- aent if you use the crew AI pattern uh or if you build a multi- aent supervisor pattern in Lang graph you know you're basically rewriting the same orchestration code over and over again. What we really like to get to is a much more standardized paradigm. So just like anthropics MCP has become the standard way that LLM interact with tools. So rather than having to rewrite custom API wrappers to uh facilitate tool calling, now we just plug in MCP servers. We want to get to the same paradigm with agents. So we want to be able to have different parts of the business write agents, put them in a catalog somewhere, make them discoverable, and then you just start grabbing agents off the shelf and plugging them in uh and reusing them. Uh and so that multi- aent paradigm is is certainly a big trend. And there's a couple of open source protocols that are important part of that picture. Um, Anthropics MCP again has become the deacto standard for tool calling. Uh, and it's really nice. You know, I just last week I wrote an updated demo for this presentation. And because I was able to plug in MCP servers both to actually do the kind of the vibe coding with Klein as well as to actually provide the interface to my data catalogs, I was able to write that demo in probably 20% of the amount of time it would have taken me before. Um so MCP today um is really a standard that started about six months ago. It originally started out as a something you run locally. So it was a way to plug tools like your local file system into claw desktop. Now very quickly we've moved to MCP servers that we can host remotely. We've added authentication and authorization. The work that's going on going on to the end of this year is can we evolve MCP to be more of an agentto agent protocol as well as a tool calling protocol. And there's a couple of other competing open source standards like Google's 80A and Cisco's agency that are kind of working in the same area. But MCP has gotten so much traction that you know at AWS that's the one that we're putting the most effort behind right now. Um and right now there's there is a way that you can actually wrap agents effectively as tools uh and use expose them through MCP today and and that's part of the demo I'll show you later. But we want to move to a world where there's a more explicit way to make agents discoverable um and allow them to communicate with one another. And so again, the ultimate picture we want to get to in the in the agentic world is we have a much more standard way to create these teams of agents working together. So we write agents that are good at solving specific problems and then we compose bigger systems of agents working together. And this is very much the same pattern we saw in the software world with microservices. So um moving on to into a couple of demos. Um, again, the first one I'm going to show you is really looking at how do we compose these multi- aent applications. And, you know, one thing we've learned over the last year or so is that when you're building an agent, there's kind of a set of core building blocks that you need for the agent to run. You know, no matter if you orchestrate it with Langraph or Crew AI or something else, your agent needs memory, it needs infrastructure, it needs observability, it needs evaluation. So there's kind of those underlying building blocks that you need and then the next layer up um you have your agent orchestration framework. So again those are things like Langraph or Crew AI. Those are probably the two most popular open source options we see. Uh you know data bricks and mosaic they have an agent framework. AWS released a new one called the strands SDK uh about uh two and a half weeks ago and I'll be showing you that one in the second example. Uh and then at the top layer you need the higher level concepts like the catalog and the registry and the authorization and the identity and some of those pieces are still being worked out. But anyway, this first demo uh is a simple example of where it's helpful to have two agents working together. And so the scenario is we want to build a deep research agent that works in a very specific way. So we want to have an overall research planning step that breaks it down into a three wide by three deep layer. So we want to take our problem, we want to look for, you know, what are the three key um aspects of this problem and then for each one of those aspects or dimensions, we want to dig three layers deep and do our analysis. So that's kind of a very structured way of approaching research that some of our teams at Amazon use. Then for each one of these aspects. So again if you look at um this was actually an example of looking at the advancements in transformer architectures in the genai world. So if our three dimensions are the technical architecture performance uh benefits and implementation challenges for each one of those we're going to go three layers deep. So for performance benefits we say well let's examine the speed improvements uh with recent advances in transformers. And for each one of those areas where we where we actually want to do some research, then we have a dedicated worker agent that actually goes and researches a specific aspect of a specific dimension. And so this is an example where I built my worker agent in Amazon Bedrock. Um, and this agent is a fairly simple one. and it says, you know, in this problem space, you need to go do some deep research on a specific aspect of a specific problem, and we'll give you access to a set of tools like looking at academic research papers or doing web search to help you along the way. And so in in this example, you can see the instructions I gave it and I gave it a tool, a Tavilli web search tool as uh through MCP. Um, and you can see the simple agent definition there, just a few lines of code. So that's my basically worker agent. Now to supervise and coordinate the overall workflow. Um what I want to do is use langraph because that's a little bit more flexible in terms of how I actually coordinate uh a complicated workflow. And so the langraph workflow for this one I'm actually going to use an anthropic model directly through data bricks uh through their mosaic model serving. And this one will walk through a series of structured steps. It'll first take the problem statement. It decomposes it into three different dimensions. For each dimension, it breaks it each dimension down into three different aspects. And then for each aspect of each dimension, it calls my worker agent to go and do the actual research. And then when all of those workers are done, it takes all the results and puts them together into a final aggregated report. So this is actually a pretty good example of a hybrid workflow where there's some parts of it that are very structured. Like we know we always want to break things down into three dimensions. Uh but then there's some aspect of it where we let the a worker agent go and do its own thing and figure out how to do the web search. And so this is just a sample query I gave it. What are the most important advances in LLM uh transformer design? And I'll show you the more complete report in a minute here. But you can see it broke it down into a list of you know my first dimension is architectural innovations. And then the first aspect of that is attention mechanism improvements and it goes and just you know takes a few minutes and writes this entire report for me. Now I mentioned building blocks. So I' I've already used a couple of things here. I've used uh model inference from both bedrock and from data bricks mosaic. Um the other one of the other important building blocks in this demo I started to use was Mosaic's uh agent evaluation framework. You know, a lot of my colleagues who build a lot of prototypes for customers, they basically say that the three most important building blocks are security for obvious reasons and then observability and evaluation. And if you don't get evaluation right, your project is likely not going to go very far very fast. Uh, and in the agentic world, this is even more interesting because we often want to evaluate not only the final output, but how efficiently the agent got there. Did it plan the right way? Did it use the right tools in the right order? Uh, and that's a really interesting area of research both for fine-tuning the model as well as for evaluating the final output. And so here you can see that, you know, Mosaic just has a really nice agent evaluation framework and it's able to automatically look at some key metrics like was the answer correct? You know, was it grounded in the right context based on my web search, things like that. So I would say that, you know, there's a lot of really good model evaluation uh tools out there. you know, if you're working in the data bricks world, mosaic evaluation is fairly easy to use. Um, and but evaluation again is one of those three critical building blocks that you definitely want to use to get started. And so this is just a summary of the overall architecture. So I use a combination of Langraph as well as Bedrock to do my agent orchestration. Um, and then for the basic building blocks, I had MCP to facilitate tool calling. Uh and then I use inference both from bedrock and from Mosaic. Uh and I also used uh Mosaic's agent evaluation uh toolkit. So that example was a little bit more focused on how we put things together. What are the building blocks we use? How we have hybrid workflows. The second example is actually a little bit more realistic. You know, one of the problems that I I run into a lot is that companies, large companies, often do end up with data worlds that are kind of split in two or three different systems. It's very very common for the customers I work with to have some data living in data bricks, some is living in in the AWS world. And as a data consumer, it can be really challenging. You know, we on the one hand, I think we're all moving towards a world where we have cataloges that talk to one another and query tools that can federate across different systems, but it can still be challenging as a data consumer to know where is my data, which system is it is it in, where do I go look for it? And so, this is a really great example of one of the ways that generative AI can make working with data easier. And so, um, rather than worrying about trying to merge my two different catalogs together, I decided to build a simple multi- aent system. Um, and I think this is a another great example of of a pretty realistic pattern for multi-agent systems. So, starting from the bottom, you can see I have two different data catalogs set up. I have the Unity catalog from the data bricks world and I have the AWS glue catalog, which is think of it as like a hive meta store for big data. Uh and so I have different sets of data in each catalog. And so I the first step I did was I built two different MCP servers uh to effectively serve as tools that an agent could use to interact with these cataloges. So the um each MCP server knows how to interact with a specific data catalog. And so they'll do things like be able to list the tables in the catalog, try to find different columns by looking at the column names or other metadata. So some of the simple things that people would do at the next level up, I build two agents. Uh each agent knows how to work with one catalog by working with the appropriate tool. Um and so I have one agent that knows how to kind of talk to the Unity catalog, another that knows how to talk to my AWS Glue catalog. And then I wrapped, you know, at the moment this is a little bit of a workaround because we don't really have a production ready agent-to- agent protocol yet. So I just wrapped each of those agents as MCP servers themselves. Um, and so each of those agents is now available as an MCP server tool that I can use. And then I have my supervisor agent that the enduser talks to. And through MCP it can work with each of those uh more specific agents. And so it looks a little bit more complicated than it is. It actually only took about an hour to put this together. Um but the idea is that as a data consumer I could go and say look um I work in logistics. I am looking for any table that is related to trucking companies in my no matter which catalog it sits in. And I have agents that can go in and talk to each of my different data cataloges and figure out do I have you know data tables related to you know trucking and it'll give you a final answer. And I think the uh captioning is kind of um cutting off a little bit the top box. So the top box was just the user interfaces. So in the sample code I I have both a canal line interface and a little streamlin front end that I put on top of these. So, I think I've got just about three or four minutes left. So, if you're interested, I could do a quick walkthrough of the notebooks um before you fill out the survey. So, this uh notebook, and I'll make this a little bit bigger so it's easier to read. So, this notebook shows how to set up that first demo. Um, so if you recall, our first demo was basically I have a bedrock agent that works with a web search tool and then I have a langraph agent that sits on top of that and kind of instructs it to go and do different types of research. So configuring the MCP servers is fairly simple. um you know the MC Tavilli is one of the most popular kind of demo web search tools that you can use with Genai and they have an MCP server already built then the next thing I do is configure my bedrock inline agent uh and I'll do that by providing it a list of MCP tools to call giving it the instructions and then I define my inline agent so I tell it which model I want to use give it the instructions and then give it essentially the set of MCP servers it's going to use. And then the main workflow is built in langraph. So I'll tell it I'm going to use the clawed model uh through data bricks. Give it its instructions and skip down here to where I set up the final graph. Simply tell it, you know, here's all my edges in the graph. Essentially, it's just guiding it. Break the problem down into three dimensions. Break each dimension down into three different aspects. And then for each aspect of each dimension call my worker agent that actually goes and does all the searching and report writing. And this just shows you what the final output looks like a little bit formatted a little bit more nicely down here. So you can see uh it broke the um most important advances in LM transformers down into three different areas. So the first dimension was architecture improvements and then it broke that down into attention mechanism versus pre-training and fine-tuning and so on. So in the last minute here I'll just quickly jump to show you the uh the second demo which again has agents that interact with different data cataloges. So basically what I have is one agent that knows make this a little bigger. One agent that knows how to work with the glue catalog. And so this is very simply uh this I built using the new AWS strands SDK. So I define the agent. I tell it which model I want to use. I give it the list of tools it can use that know how to talk to my glue catalog. And you can see these tools do very specific things like list the databases, you know, list all the tables in a database, try to find tables by name or by whether they contain columns of a certain type, and then I give it the instructions. Uh then there's uh another agent that's very similar that knows how to work with the Unity catalog. And then the final agent, the supervisor agent, uses each of those worker agents as MCP servers. So I have one MCP server that works with my Unity agent, another that works with my glue catalog agent, and then I have the instructions for my supervisor agent. And essentially all this is doing is routing. It tells it take the user query go work with the the two agents that can go work with the two separate cataloges find out whatever information you can and and give the end result to the user. So I would say that now you know compared to where we were 6 months ago just my concluding remark would be it has gotten so much easier to build fairly complex multi- aent systems in a much cleaner way with much less boilerplate kind of throwaway code. Um so going forward I would say you know although things are going to be even better 6 months from now when we have full-blown agent cataloges and agent agent registries if you start using uh you know a nice clean agent framework like strands SDK or langraph or crew AI and you start uh using MCP as your abstraction layer for tool calling and even making agents available as tools to other agents you can start composing these much more complex multi- aent systems in a very short amount of time. Um, now there's still a lot of work to be done in terms of safety guard rails and evaluation and observability, but getting from zero to proof of concept again can be done in in not a huge amount of time anymore. Uh, and just to give a little quick plug, my favorite vibe coding assistant is uh Klein backed by Claude 3.7 with MCP. Um, and that's just for me like personally was a kind of a very transformative step forward in personal productivity. So, with that, I think I'm actually a couple minutes over. Thanks very much for your uh attention. And again, please do fill out the survey. Uh if you have any uh if you'd like to ask me any other questions, I have my LinkedIn information on the title slide. Always happy to uh chat offline. Thanks very much.