Introduction to CloudQuery | Amazon Web Services

Key Takeaways

Hi everyone, my name is Keegan Morazzi. Thanks for joining us. I'm a partner solutions architect here at AWS where my team's mission is to discover, develop, and deliver solutions with our partners. Today here I'm joined with Joe Carlson. He's a senior cloud developer advocate at Cloud Query. Uh say hi Joe. >> Hi Keegan. Thanks for having me. Okay. Uh Keegan, today we're going to be introducing Cloud Query as a cloud asset inventory for AWS and we're going to be digging into what it is, what problems it addresses, and then give a overview of its architecture. You can see here a little spoiler alert. Um so Keegan, I actually wanted to ask you a quick question, completely rhetorical. Um how many cloud resources are publicly available in environments you manage? How many are currently unencrypted? Which one of those costs the most? And Keegan, if you can't answer any of those questions, no problem. Um, you probably have a visibility problem with your cloud and not a cloud problem. >> So, we have these questions about our cloud environments. What can we really do to address them and get that visibility that you're talking about? >> Yeah. And and don't worry, like most teams can't and most teams don't have a ton of visibility into their cloud resources. Um, and oftentimes we find visibility can be pretty pretty fragmented. Getting data about your cloud can be kind of messy and siloed and most tools only solve a slice of the questions you have around that like security cost or just giving you a basic list of your inventory. Uh, and that's the problem that cloud query actually solves. So with cloud query teams actually get an AI powered visibility and actionable insights across all of their cloud resources and it can help you find things like what cloud resources exist. You can get complete cloud coverage or coverage across AWS including all your accounts and regions. So not just your EC2 and S3 buckets but Lambda functions, RDS snapshots and all the random cloud for stacks that you put together 6 months ago and forgot. multicloud account, multi-reion visibility, all in single queries. And you can get real-time updates every time someone spins up a new resource. You can also find things like how your cloud is currently configured. It makes it easy to get deep metadata from AWS and like configuration settings, relationships, who owns what resource and where and when they set it up. Uh and you can see things like EC2 instances plus the security groups, VPC, IM roles, who launched it and when. Uh complete network topology across your resources showing which resources talk to which and you can also govern different tags across your resources so you can know which team owns what and which environment they're you're currently looking at. The last thing too it can help you with is spotting risks and finding opportunities in your cloud. So things like security misconfigurations you can identify before they get exploited. So public S3 buckets, expired SSLerts, etc. Uh and it can also do things like find costs that are could be wasteful. So unused EBS volumes, idle RDS instances, wasted reserve capacity. It can help you find and build automated compliance monitoring for SOCK 2, PCI, CIS, whatever. Um, and it basically just helps you find operational trends before they become problems. >> Awesome. So, I feel like I have a good idea of what cloud query does. Can we talk a little bit about how it works? >> Yeah, let's dig into it. We have this uh beautiful drawing that I put together. Thank you very much. Um, but let's dig into it. So, um, first and foremost, cloud query is read only with your AWS environment, which means you don't have to make any envir or like modifications to your your infrastructure. There's no agents, no security access or nothing that's going to be edited or changed on there. And it connects to your AWS environments and accounts using IM roles um with describe and list permissions only. Um but the thing that makes it powerful, it's not just your AWS environments connects to can also connect multiple data sources like other SAS environments. Um, cloud query is able to extract data from multiple source different sources including you know AWS accounts but things like wise data dog github gitlab um can get like you can import things like your wise for security finding splunk for log analytics and data dog for monitoring metrics. So you can enrich your cloud configuration data with data from other sources to really get a holistic view of what's going on in your cloud and why things happened. And then here is where the magic happens in cloud query. So cloudquery takes all your data from these cloud sources. It transforms and normalizes them into a unified schema. So that means your EC2 instances, your GitLab projects, all your wise vulnerabilities and data dog metrics all get the same treatment. So it makes that you have consistent uh field names, standardized relationships, and unified timestamps to make it easy to query and analyze across everything. This isn't just about data collection. It's data normalization and everything gets transformed to format that makes it easy for you to analyze it. Okay, so we're going to move deeper down a ladder here. There are basically two options for managing and using cloud query. You can manage it on your own using the CLI. uh CLI is useful if you want to persist that data into your own data lake data warehouse postgres database whatever um or in the cloud where that gets managed so you can do automated syncs uh and our backend is built in click house so you have super speedy massively scalable uh analysis you can do on that that back end too but either way you end up with a unified view of your entire techn technology stack, AWS resources, application code, security findings, operational metrics, all queryable all through a single place. Um, and this architecture really solves a fundamental problem of visibility fragmentation in the cloud. So instead of having to jump between all these different dashboards and all these different places, it can all be in one single place for a single source of truth. That's a lot, but okay, let's wrap up what we covered here today. So, Cloud Query solves the cloud visibility problem that's been plaguing infrastructure teams. Cloud Query makes it easy for you to see everything across all your accounts, all your regions, and all your SAS platforms. That means no more blind spots, no more surprises when you're building in the cloud. It also helps you create and maintain a unified data model with consistent schemas across your entire cloud infrastructure. Whether it's an AWS EC2 instance or a GitLab project, the data is predictable and queryable. And it's more than just collecting that data. It makes it easy for you to make intelligent decisions and take action on that data.