Unlocking agentic data engineering with Lakeflow + Genie

Key Takeaways

Welcome to the stage, Databricks senior director of product management, Bilal Aslam. >> [screaming] >> Good morning. Good morning. I'm going to talk to you about Lakeflow, but just a little quick note first. I wore this I wore this purple jacket 2 years ago, talked about Lakeflow. I did not wear it last year, and the biggest piece of feedback was bring it back. So, here it is. Uh my name is Bilal Aslam. I'm a senior director of product management at Databricks. Super excited to talk to you about Lakeflow. Now, uh first of all, thank you to Mugesh and Arsalan and Ali and Kenan and everybody who's come here and talked about AI and agents. Now, uh super exciting future, uh but if you're anything like me, if you're a data engineer, uh you're probably sitting there thinking, "This is great, but all of this needs data." Uh all these apps, agents, AI, everything needs data, and you might be thinking, "My job got a little bit harder." Uh and that's because uh with data engineering, the destination is simple. The destination is get to the right. Get to the business outcomes we want. We want agents, applications, we want real-time operations, but we're actually starting from the left. We're starting from raw data sources. We have data in Kafka, Salesforce, NetSuite. We have on-premise databases like Oracle and SQL Server. And the messy middle, that's data engineering. That's our job to take that raw data, turn it into insights. And over the years, we have built architectures like these. I put this up here because it's representative. You don't have to look at every box here. It's a lot of logos. It's a lot of boxes. It's a lot of lines between boxes. And that's just to get data from one place to the other. And you know, we've made peace with complexity. Sure, there's a lot of tools, but you're probably sitting in here thinking, "Okay, at least it works." Uh even though you know instinctively that this complicated, not unified architecture is missing a bunch of critical things, we want to version control everything as data engineers, but in this stack, you can version control some things, not the others. Uh you can version version control your Spark code, but not your Kafka infrastructure. You want governance everywhere, but it's pretty much impossible. You're reading and writing to too too many different places, and there are parts of the stack, as we will see soon, that are just locked up. And finally, you want everything to scale, and you want to save money. Uh so you can build more applications and outcomes for your business, but not all parts of the stack scale equally well. I'm here to tell you that your job and my job as a data engineer is about to get quite a bit harder. What you're seeing here is Genie code. It's a purpose-built coding agent for data science, machine learning, and data engineering. Uh if you're using it, that's awesome because uh 60% of Lake Flow pipelines, this is a crazy statistic, are already written by Genie code in just 3 months. We just released this. It's pretty awesome. On the other hand, this statistic scares me a little bit as a data engineer. And that's because I have a crush on agents. Agents are great, but all agents mean new data, new pipelines, more to build, more to manage on that fragile infrastructure. We're going to get swamped. So what do we do? We start by removing the moving pieces. And actually, it's super important. We don't just want to remove complexity with more complexity. We want to replace that with open formats, open open frameworks, open tools. We ultimately need to simplify the data stack. I'm here to make a pretty bold claim, and my bold claim is that Lake Flow is that unified stack. That it gives you that open foundation for AI and agents so that you can build the future that your company deserves. And now, we work with hundreds of partners, lots of tools. They're all interoperable with Lake Flow and the lakehouse. You can use them as well. So let's get started. I'm going to do a little bit of a speed run through this. Uh how do you simplify data transformation? ETL. This is where you're spending most of your time and money. Uh you have Spark. You have lots of legacy Spark jobs. You may have a Spark distribution on EMR or some other vendor. I'm going to replace that with Apache Spark declarative pipelines. If you're not using them, please try them out on Lake Flow. They're open, you can run them anywhere including on your laptop, and they're declarative so you can just focus on the work to be done. And because they're open, agents are really really good at writing these pipelines. They unify batch and streaming. Many years ago, DBT was the only game in town when it came to SQL. I like writing ETL in SQL, but Spark declarative pipelines unifies that as well. It has SQL and Python. You don't have to choose. And finally, if you're using Flink for real-time streaming, that's probably because actually low latency streaming was really hard on Databricks and Spark. But now, last year we open sourced real-time mode, which is a brand new engine for low latency stream streaming, and I'm super excited to share that it's now available inside Spark declarative pipelines. So you can get millisecond streaming on an open framework that runs anywhere. Awesome. So we're taking care of this complexity, we're getting rid of this, we're building an open foundation for agents and AI. Well, what about all the shadow IT that's happening? So you see, you may have 20 data engineers in your company, but you have hundreds of analysts who are building pipelines in drag and drop tools. These pipelines are built on their laptops, proprietary formats, proprietary tools, and if somebody leaves the team, you're out of luck. Super excited to share that Lake Flow Designer, the no-code data prep tool powered by Genie, is now available. It's generally available. You can try today, you can try it right now. >> [applause] >> Thank you. Awesome. And by the way, one super cool thing is that Lake Flow Designer doesn't build any proprietary pipelines. It just builds Spark declarative pipelines under the hood. Again, you can just run them anywhere. I think you're starting to see a pattern. All right, we're going to go a little faster. I'm going to go ahead and simplify data ingestion. If you're using a SaaS that has hundreds of connectors, that's great, but it's probably landing data in proprietary formats. Your agents don't know how to operate this SAS. You cannot automate this. You can barely monitor it or observe it or govern it. As Ali mentioned, I'm super excited to share that LakeFlow Connect now has more than 100 connectors. If there's a connector you're thinking of, we've either built it, we're building it, or we're going to build it. And there's a new community that's building open-source connectors. You can even build your own. And guess what? I think you're going to guess what I'm going to say next. This isn't a special type of pipeline or ETL. It's just Spark declarative pipelines under the hood. That was pretty awesome. All right. Let's talk about Kafka. As data engineers, we use Kafka day in and day out for high-volume telemetry. We're bringing that in. Kafka's a buffer, but we also know Kafka's a real pain to manage, okay? Super excited to share that Zero Bus Ingest is a fully managed service that is now 100% wire compatible with Kafka. So, you can take your Kafka producing code, point it at Zero Bus Ingest, and you can land data into the lakehouse in open formats, ready for AI. So, no small files, millions of small files at 12 GB a second. So, it's pretty awesome. You can get rid of Kafka for your stack. Great. Uh okay, so we're almost there, and you're going to see we're still building everything on an open ETL framework that's declarative. But you see you see this little uh Airflow icon hanging out there. That's your orchestrator. You have to use that to trigger things, to schedule things. Let's simplify that as well. So, you're probably using Airflow because you like writing workflows in Python. Well, LakeFlow Jobs supports writing workflows in Python now. It's just uh pure Python. You can also build point-and-click DAGs. It's fully serverless, so you don't have to manage any infrastructure. And one interesting fact is that we did a survey of our customers, and 80% of our customers are using old Airflow distributions. So, they're managing them, maintaining them, they have security vulnerabilities, and they're not even getting all the new features, okay? Uh and I'm the last thing I'm really excited about because for the longest time, let's face it, with jobs, it was really good at orchestrating Databricks, not so great at orchestrating other systems. So, super excited. This probably one of my favorites in this conference is we're releasing 50 plus integrations. So, you can orchestrate everything. You can think of these as Airflow operators. They're open source and you can orchestrate everything including Snowflake. Okay. So, where where does this bring us? This brings us to a an open stack. Everything is pretty much building open source declarative pipelines. And this is not a brand new product. We've been at this for a while and it's ready for the world's toughest data workloads. I want to share some statistics with you. Spark declarative pipelines now process 200 trillion rows of data every single day. Lake Flow jobs is running the world's biggest data workloads. It's about 1.7 billion job runs per month. And finally, serverless compute is really popular. You don't want to manage clusters. You don't want to set up VPCs. 50% of our customers have now opted in to use serverless compute. That's pretty awesome. Great. So, we've I hopefully I have convinced you to some degree that we made it easier to build. And now you might be thinking, "Okay, we have an open stack. I can start building AI can write the code." Maybe you're going to go try out Genie and it's going to build something for you. But what about operations? I haven't really solved that for you. Actually, I've kind of made the problem worse, right? Because now you're going to say, "Great. Now more people can build more pipelines, more outages, more problems." And here's the reality. Simplifying pipeline creation is actually the easy part. Although this was pretty tough, right? It's data operations that are really, really hard. Here's a survey that is by one of our partners and it shows that data engineering teams spend more than 50% of their time on maintenance. So, that's where your time is going. It's only going to get worse. And even with this time spent on maintenance, you still see up to 60 hours of downtime every month. So, this sucks. Now, you might be thinking, uh but Bilal, you just told me that this is a unified stack. Surely, it has APIs and endpoints and tables. Maybe a coding agent can just fix it. And unfortunately, not really. Let me walk you very quickly through why. And the big idea here is that data engineering is not software engineering. It's something a bit more. In software engineering, your code is self-describing. You It's It just literally describes itself. And in data engineering, it's data and code. In software engineering, your tests either work or they don't. They're fully deterministic. In data engineering, you have code and data. You can't just test data. It's statistical. And finally, in software engineering, and this is the big one, failures are loud. When something goes wrong, you're going to see a trace. You're going to see an exception. And then you can roll back your deployment. In data engineering, failures are silent and permanent. You're probably going to get an angry phone call from a CEO or a stakeholder saying, "Hey, how come this data is bad?" Maybe a week later. Uh so, why do coding agents trip up here? Well, that's because when it comes to detection, they're just missing data. They're missing telemetry. And you can give them Spark logs, for example, but these are multi-megabyte traces. You have to be very careful to manage the infra- the the context window there. And similarly, when it comes to assessing the root cause, they're missing lineage. Now, you have to export your lineage out of the system. And to some degree, you can do this. Coding agents are getting better and better and context windows are getting better. And then you have to remediate. They have to write the code fix. But it's actually the final step, which I'm going to be a little pedantic about and call it verify. You can't with data, you can't just run your unit tests. You actually have to take your code fix, and you have to run it on production data. Okay. And if this is giving you shivers, that's exactly right. It really should. So, what does your agent need? Your operations agent needs to combine code and data. That's tools and skills to some degree it can do it. To figure out the root cause, it needs read access to production data. Uh, who's comfortable with that? It's an agent you didn't write and you don't trust. I I don't see a single hand raised here. But actually, your coding agent also needs write access to production data. That's the only way it can verify the fix. Like who's here wants to give a random agent write access to their production data? There's got to be one person. Okay, there's no one who wants to do that, right? Because it can cause chaos. It can drop everything. And this is the big insight. So, we've been looking at this problem, it's a tough problem, and the big realization we came to was that your operations agent actually needs to live in the data plane. It cannot live outside of the data plane. And if you think about it, that makes sense because the data plane has the data. It has the lineage. It's also the right governance boundary. So, I'm super, super excited to announce Genie Zero Ops, a new background Genie that puts your data and AI operations on autopilot. Great. I'm going to give you a demo, but let me quickly walk you through how it works. So, for detection, Genie Zero Ops autonomously builds per table machine learning models, and it continuously fine-tunes them. And it has native access to metrics, events, and logs. So, you don't have to build this plumbing yourself. We already have tens of thousands of these machine learning models in production with Genie Zero Ops. To assess, it does graph ranking on data lineage in Unity Catalog. So, I'll show you how it walks the lineage forward and backwards. And then to figure out the root cause, and this is where you spend a lot of your time. The root cause is basically what went wrong. It actually has a supervisor agent and a fleet of sub agents who do research and then come to a consensus on what is the likeliest cause. For remediation, Genie Zero Ops works with Genie code. It cooperates. If Genie code has access to your code, to your ticketing system, to your version control, it uses all that. So, it can also update tickets for you as it goes through the life cycle. And finally, verification is the toughest step. To verify, this is the 10 years of building an open lakehouse. It builds shallow clones of your production data. These are super cheap, super fast branches, similar to what we do in Lakebase. And then it utilizes native network and code isolation in Databricks. All right. So, I'm going to show you a quick demo of what Genie Zero Ops actually does. Great. So, let's see. All right. So, you're going to notice a couple of things here. First of all, this does not look anything like a dashboard. We're not flooding you with metrics, events, alerts, light uh you know, red lights and all that. What you see is something that looks and feels a lot like an email inbox. It's actually, we modeled it after a prioritized email inbox. So, for example here, one cool thing is that you get to see severity. So, every incident is ranked by severity, so you can only you have you can just focus on what really matters. And also, this isn't just a list of jobs and pipelines and failures and things like that. Over here, for example, here's an alert and Genie Zero Ops figured out that hey, 16 jobs are failing, they're actually the same incident. So, the natural grouping saves you time. All right. So, I'm going to go ahead and look at an actual incident. And what's happening here is that there's an upstream table called top fan voters, and something's going wrong with it about 10 minutes or so ago, uh the row count dropped quite a bit. And what Genie Zero Ops does is by the time you log in, your background agent has actually done all the thinking for you. It has done all the investigation for you. And as a data engineer, I can start looking at things like what's the impact. Now, when it comes to impact, not all tables are important, not all pipelines are important. So, it walks the lineage forward, but in this case it figures out that, "Hey, top fan voters is a pretty important table. Downstream tables depend on it, and this fan fan engagement dashboard depends on it." So, it marks it as critical. Then, it deploys the root cause agents to figure out what's going on. And what these agents do is this is where they're saving you hours of time. They start from this table, and they fan back using lineage, and they go investigate potential causes upstream. So, over here, this is a fairly simple DAG. For example, you know, it could be any one of these tables, but really it finds this one table, this fan interactions table, that's the one that's problematic. Okay. Then, it went ahead and autonomously wrote a fix for you, but this fix is not deployed to production. You're in control. What it did do, and I want to share this, this is super cool, is that it went and created a shallow clone. It's like the lake-based branch of your production data. By the way, you have to give it permission. It won't do it automatically. You have to tell it you can do it with this pipeline. And this could be petabytes of data. It's a shallow clone. It deployed the fix, and it actually verified that the right number of rows are returned. This is awesome. This is hours, if not days worth of work. And now at this point, I'm just going to go ahead and create a pull request, and I can follow my software development life cycle. Uh let me show you one other super cool thing that Genie Zero Ops can do. The one other thing Genie Zero Ops does is if you give it permission, it will autonomously scan tables for things like PII. So, in this case, our application is unwillingly exposing the PII of about a thousand users. Okay? That's pretty bad. And Genie Zero Ops creates a report for me. It finds out what the PII, the personally identifiable information is. It gives me a per-table breakdown. It looks at lineage to figure out this is important. Yeah, this is pretty bad. And then in this case, it comes up with a proposed fix that's not a code fix. It's a It's a Unity Catalog a policy fix, and I can deploy it. Okay, so that's Genie Zero Ops. We're super excited about it. And for data engineers like me, and hopefully for you, it means far less time building, far less time maintaining, and much more time building on a unified and simple data stack. Thank you very much.