Building AI agents on Google Cloud

Key Takeaways

Hi, my name is Taran and I'm a product manager at Google Cloud. Today we are going to share how to build AI agents with Google Cloud. We'll first talk about what are AI agents, then how to run AI agents on Google Cloud Run, and finally how to run AI agents on Vert.x AI. Let's get started. AI agents are a new type of application architectures. They rely on a language model to reason. They have short and long-term memory and are connected to sources of context. They plan and execute complex tasks by orchestrating multiple smaller tasks or even other agents. Their user experience can vary. They can be synchronous via chat or run asynchronously. For example, when doing research or doing code reviews, they need to collaborate with the user asking for more guidance, confirmation, and decisions. That enables user moderation and control. Something we call human in the loop. task can be performed using tools. Let's talk a bit more about those tools. First, there are the basic tools. Large language models are not good at math or basic time or unit conversions. Then, data access. The agent can retrieve context before calling the LLM or even query a database for specific details, for example, a product catalog. APIs. The agent can read from APIs, either first party or third party APIs. It can potentially even have right access with human supervision. Image generation. The agent can call another image generation API or generate visualizations from generated data. Browser. The agent can be given control of a web browser and browse websites using a real chromiumbased web browser. This is often done with multiple back and forth between the LLM and the browser service. Code execution. Code is generated by the LM and the agent executes it in a sandboxed secured environment. So in term of architecture at a high level a user will interact with the agent via multiple requests responses. These responses are often streamed. The agent is composed of the serving and orchestration component. This component we leverage of course a model to reason memory and access to data as well as tools. So a runtime for AI agent needs to scale to handle many concurrent users using the agent. Of course be cost effective. Have good latency performance and higher reliability. Have a great developer experience for agent developers. Integrate with LLM and tools. be flexible of running the developer's favorite language in AI agent framework and have streaming built in so that responses can be streamed as fast as possible to the end user. Cloud run is a perfect runtime to host your AI agent on top of Google's scalable infrastructure. It scales via automatic ondemand and rapid scaling of instances. It is cost effective via a pay-per use pricing model. Notably, it doesn't have any flat fee and doesn't require pre-provisioning capacity. It is fully managed. It has built-in zonal redundancy and is enterprise ready in terms of security, networking, and compliance. Developers love Cloud Run. It has top satisfaction and usability scores. Agent developers go from source to URL in one command. Cloud Run services have built-in credentials enabling you to call the Gemini API securely without any API key. You can run any AI agent frameworks in any language, Python, JavaScript and Cloudron supports out ofthe-box HTTPS endpoint with streaming be HTTP chunk transfer encoding, HTTP2 or websockets. So let's look at how to implement the previously described architecture on top of cloudr run. A cloudr run service takes care of serving and orchestration. It runs an AI agent framework like langraph or agent development kit. It can call the Gemini API or models served via the Vert.Ex AI endpoint or even your own fine-tuned model running on cloud run with GPUs for memory fire store is a great nosql database and memory store can also be used retrieval augmented generation can be achieved using cloud SQL or DB for posgress SQL with the PG vector plugins and finally tools can be hosted themselves on Cloud Run or be calls to existing APIs. And now let's hear how to build an AI agent. Hi, my name is Vita and I'm an engineer in the Cloud Run team. I'd like to show you a concrete example of building an agent on Cloud Run. Cloud Run supports any agent framework, but I used Langra. Let's quickly dive into a few concepts before I show you the app. Let's start with LLMs. You give them an input, maybe a question, a prompt to write something, and they generate a relevant output. They are fantastic tools. But as powerful as they are, LMS used in isolation have some key limitations. For example, their knowledge is frozen at the time they were trained, and they cannot directly interact with the outside world. So to overcome these limits, the most common approach is to augment the LLM by combining it with other pieces of data and computation. This pattern is called a chain. Think of it like a fixed workflow. As a developer, you explicitly define a sequence of steps. A classic example of this is retrieval augmented generation or rag. The big advantage here is reliability. The steps are predetermined so you know exactly what will happen. The downside, it's inflexible. The chain follows a predefined path and can adapt dynamically. Now, a key development that enables a more dynamic approach is tool calling. You can give an LLM access to external tools. When the LLM thinks it needs one of these tools to answer a request, it doesn't run the tool itself, but it generates a structured output saying which tool it wants to use and what information to pass to it. This tool calling capability leads to this second main paradigm, the agent. Instead of a fixed chain here, we put the LLM in the driver's seat, running it in a loop. The LLM gets a task, calls a tool, feeds the result back to the LM, and lets it decide the next step. It continues this loop until it determines the task is complete. The big advantage here is flexibility. The agent is not locked into a predefined sequence. However, this flexibility often leads to lower reliability. These basic agent where the LLM is making all the decisions in a loop can often go off track. And here's why. Maybe the instructions given to the LLM were ambiguous. Maybe the tools available weren't quite right or their descriptions were misleading. Perhaps the LLM failed to reason correctly about the steps needed and tries to rely on its internal knowledge instead of using a tool. So while powerful, this simple looping approach can be unreliable in practice. So this brings us to the core challenge. On one hand, we have change reliable, predictable but rigid and on the other hand we have these agents flexible and dynamic but unreliable. So the question is can we have both? Can we build systems that have the reliability of structured workflows and also the flexibility of agents? One way to achieve this is to model your control flow with a graph like a state machine. In this graph, you have nodes representing specific decision points. Some might execute code, some might call an LLM, some might call a tool, and then you have edges that define how you transition between these nodes representing the logic and flow of the application. So this is the approach that langraph uses. Lang graph is a framework designed specifically to build these kinds of graph-based AI applications. This core ID is to let you explicitly define parts of the control flow using code giving you the reliability of chains when you need it while also giving you the flexibility of agents where that's needed. Now let's take a look at my demo to see an example of that in practice. This is my pretend online store from the future and I am acting as a support agent. Here is a list of my open cases. To resolve them, I can access a repository of standard operating procedures. For example, this is the one that tells me how to investigate failed payments. I also have access to the backend systems. So, here's orders, customers, payments, shipments, and finally, invoices. Let's go ahead and resolve one of the support cases. This one looks interesting. This customer really needs their Echo Weave device, but is having payment issues. I've built three AI agents that help me do my work more efficiently, so I can focus on providing the best support for my customers. First, my case context agent will help me find all the resources that are related to the case. Gemini use uses tools to get all the relevant information and it will link them to the case. That's great because that's very uh repetitive work to do manually. So, it linked an order for those two echoe devices. There is a payment which failed because of a gateway timeout and the standard operating procedure that's related to this case. Working through that manual is a lot of work. Let's see if my other agent can help me with that. The SOP agent will read the manual, make a plan for resolving it, and execute the plan. It looks like the payment was successful after a retry. To close this case, let's send a reply to the customer using my third agent. This agent uses all the information in the case to send a personalized message to my customer and saves it as a draft for me to add it before I send it. There it is. And the message looks good to me. Let's reply and close the case. Now that you've seen the demo, let's take a look at the application architecture of my app. I used Langraph to build my agents and fast API as the web framework to build my web application. And look, I'm more of a Go developer myself, but Python is clearly my destiny. Finally, I packaged my web app into a container image using a Docker file. And now that I have a container image, I can deploy it to Cloud Run and get an HTTPS endpoint back. Now I want to show you how I deploy a change to my app. So this is my local editor on my local machine. And I'm going to add some text to the homepage and then deploy it so you can see the change. I'll add a paragraph. Hello Google IO. I'll save that file and I go to terminal and do g-cloud run deploy and I will confirm that that this is the directory I want to deploy and that's the service name and then I will start building and deploying. So while that's running I'll take you through a short tour of the web console. So this is my service in cloud run and I you can see that I have access to all the metrics like the request count request latencies the contain number of container instances that I'm running uh how much CPU they're using memory utilization and also the startup time of my containers. Now let's check back and see if that deploy has completed. Looks like it's almost done. So, it's creating a revision of routing traffic. And when that's done, I can click on the link. And you can see hello Google IO. Let's break down that G-Cloud run deploy command. First, it uploads the sources to a GCS bucket. It kicks off a build in cloud build. And if you don't have a Docker file, then build packs is used to turn your source code into a container image for you. BuildPax is the open- source version of the technology that powered App Engine since 2008. Note that you don't have to run these commands. This is done for you by the G-Cloud run deploy command. Now, once the container image is built, it is pushed to artifact registry. That's Google's container registry service. Then, Cloud Run creates a new revision importing the full image. And finally, 100% of traffic is automatically migrated to the new revision. Cloud Run is designed to make developers more productive, and that's why it's great for running agents. Now, let's look at another way to build agents on Google Cloud with Vert.ex AI. Thank you, Vita, for walking through how to build AI agents with CloudRunner. And now, let's discuss the different capabilities in Vert.x AI to help us build these sophisticated agents. Google Cloud's Vert.Ex AI is a comprehensive platform which supports the full agent life cycle. And today we'll cover an overview of the entire workflow starting from models and tools in using Vert.x AI's agent builder to discover, build, and deploy agents and using the A2A protocol to extend the agents and make them interoperable. Let's break down each of these components. At the foundational level, agents need two things, intelligence and tool access. And for intelligence, Vert.x AI provides access to Google's cuttingedge Gemini models natively. It also provides access to hundreds of other open-source models via Model Garden and the flexibility to bring your own, ensuring you have the best model for your agent. And agents are only useful if they can interact with the external world. Vert.x AI enables this through native Google Cloud integration, allowing agents to securely ground responses and connect to various enterprise data sources and tools. Agents can leverage pre-built connectors, call custom APIs managed via Apoge, or even trigger complex workflows in application integration. Now with this foundation, let's talk about the central component which is a Vert.xai agent builder. It contains three main capabilities. The agent development kit to build agents, agent engine to deploy them, and the agent garden to discover and use other agents. Now, at the core of agent building is the ADK or the agent development kit. It's Google's open-source agent framework to build sophisticated and reliable agents. Now, it is drawn from internal Google's experience of building agents and is designed to provide a smooth developer experience by making agent development feel more like traditional software development. And a few key capabilities of ADK are we'll start with the built-in developer UI, a local web interface that spins up instantly when you run your agent. Let's see this with an example of a customer service agent. And here I have my simple home and garden agent, which can help customers with product selection, managing orders, or can even use video to help identify different kinds of plants. Running the ADK run cla command directly starts the agent in your terminal or even better running via ADK web opens up a new built-in UI. Let's go ahead and enter a prompt. Hey, can you help me with some indoor plan recommendations for winter? And we'll see the agent attempting to call the relevant tool to fetch the information. This not only speeds up our iteration and we can directly chat with our agent but also helps us inspect the reasoning steps, see which tools the agent uses and debug issues visually without writing any complex tests or any UI code. Now moving beyond just the text interaction, ADK is built for richer interactions like it natively handles audio and video streaming with minimal to no code changes if you're using ADK web. This allows us to build agents that can see, hear, and respond in real time directly within the framework. With all of these capabilities, while ADK is powerful on its own, it is designed for an open ecosystem, meaning it's model agnostic, deployment agnostic, and interoperable. You can bring in Gemini models, open- source models, or even custom models from model garden or light LLM. ADK is also crucially deployment agnostic, meaning you can package the ADK agents into containers and run them anywhere. Be it Vert.xi XI agent engine, Google Cloud Run, GKE or even any other clouds or onromises. But most importantly, ADK supports interoperability and extensibility through support for both model context protocol and the agentto agent protocol. You can integrate any MCP tool to your agents built with ADK. You can also extend your agent to talk to other agents which are A2A compatible which we'll briefly discuss in a few minutes. Agent development kit also provides other capabilities like LLM based agents, built-in agent evaluation or generating tools from an open API spec and many more. You can get started with ADK today by grabbing the Google ADK package. Check out the links to learn more. Now, once we've built this agent, they need a reliable home or a place to deploy. And that's where the Vert.x AI's agent engine offers a serverless runtime optimized for agents. You can deploy a containerized agent built using ADK or any other framework and agent engine will handle the underlying infrastructure like scaling, security configurations and integration with cloud monitoring. It's built with enterprise needs in mind offering robust security like VPCSA compliance, cloud trace integration, and even integration with Vert.ex AI's evaluation service, which is a way to quality check your agents before and after deployment. Now, with just a few lines of code, we can now deploy the customer service agent that we just built with ADK into agent engine. And that's it. Our agent is now running remotely and is ready to be integrated into other applications. Agent Engine effectively isolates agents execution logic from the underlying infrastructure concerns. And to help you get started quickly and avoid building these agents from scratch, Vert.xai's agent garden provides pre-built agent samples and modular tool components. In the Vert.xai ICS AI agent garden, you will find complete source code examples for common agent archetypes like data science agent, retail agent, or even the customer service agent that we just built. Or you can also grab reusable code snippets and implementations for specific functionalities such as connectors for particular databases, secure wrappers for external APIs, or optimized rack components. Using agent garden, you can find reference implementation, reduce your boilerplate code, and integrate tested components directly into your agent. Now, as we make huge investments in developing these valuable specialized agents with multiple tools and connectors, it's crucial to prevent them from becoming isolated silos. True power lies in enabling them to collaborate and share their unique capabilities. Think of stacking Legos over one another. And to facilitate this vital interoperability across your own agents, across various platform, or even across different companies, Google alongside over 50 other industry partners recently launched the agentto agent or A2A protocol. A2A is an open standard designed to allow agents, regardless of how or where they were built, to securely communicate, exchange information, and coordinate actions across longunning tasks. It is built on familiar protocols like HTTP and JSON RPC and facilitates interactions between a client agent initiating a request and a remote server agent fulfilling it. Now let's say you have an AI agent for financial trading which has a strategy agent to identify trading opportunities and a risk agent to do compliance assessment. A single agent in this scenario would be too complex and create a tangle of connections. But with A2A, these agents can now collaborate effectively. The strategy agent can now discover the risk agents capabilities via an agent card. It can then initiate a task sending the trade query as a message. 8 to8 defines how these agents can securely exchange information and coordinate actions even for complex longunning tasks and the risk agent can now provide the status updates on its assessment using either streaming or push notifications. 82A protocol also supports interaction via various modalities such as text, data or files and is designed with enterprisegrade security enabling reliable coordination. The goal is simple. It's universal interoperability. Away enables you to build powerful solutions by composing diverse specialized agents breaking down the technical barriers and ADK natively supports ensuring agents built with it are ready for this interconnected multi- aent future and here's the link to the 808 docs for more information to do a quick recap. AI provides a cohesive platform to build agents. It starts with the flexibility of models and secure access to enterprise tools. We can then use the Vert.xi's agent builder to build sophisticated testable agents using the open-source agent development kit. Deploy and manage these agents reliably via the serverless agent engine and accelerate discoverability and development using agent garden and critically enable powerful interoperable multi- aent solutions through the A2A protocol. It is this combination of comprehensive tooling, flexibility, and a clear vision for interoperability that allows you to build, deploy, and manage truly impactful AI agents more effectively on Vert.exai. Thank you. [Applause]