Efficient data ingestion with Lakeflow Connect: Data Engineering with Databricks

Key Takeaways

[Music] My name is Elise and I'm on the product team here at Data Bricks and I'm going to discuss and demo efficient data ingestion with LakeFlow Connect. But first, why are we even here today? Well, it's pretty simple. Before you can take advantage of data bricks, you have to access your data. And that data is probably all over the place. You have it in SAS applications like Salesforce, Workday, Service Now. You have databases like SQL Server, Postgres, Oracle. You have file sources, message buses, APIs, and so on. But it can be really difficult to get that data into the lakehouse so that you can do meaningful things with it. To start, pipelines are hard to set up. For example, SAS APIs have tons of quirks and they can be really difficult to ramp up on. They also change a lot, so it's hard to keep up with them. Database pipelines are even harder to configure and manage, especially without impacting the data source. So customers do a lot of work to build these pipelines, to optimize them, and particularly to maintain them. And spending time on all that gets super expensive. Plus, it slows your time to value. It doesn't help that these pipelines often require a specialized data engineer. So the people who need to use this data downstream, say the analyst or the ML engineer, they get blocked. It's the opposite of democratization. And you might not even want to democratize your data when you have this patchwork of pipelines because it's so hard to govern it effectively. Think access control, lineage, monitoring. So at the end of the day, you're not fully utilizing your data. You probably run into pipeline outages and you might even have some security risks. Now, data bicks has already done a ton of work to address this, and we're going to cover some of those existing solutions so that we can contextualize where the new offering sits in our portfolio. At the most customizable layer of the stack, we have structured streaming, which is an API for incremental stream processing in near real time. Engineers love the performance, the scalability, the fault tolerance. But over time, they asked for more automation and less overhead. They wanted things like better autoscaling, data quality constraints, failure recovery and so on. So we added LakeFlow declarative pipelines which is a framework for building reliable and scalable data pipelines. You define the transformations that you want to perform on your data and declarative pipelines manage the orchestration monitoring data quality errors and customers love that it reduced the complexity I've been talking about while also improving the performance of their pipelines. But these days, we're hearing a lot of demand for something that is even easier to set up, even more hands-off, especially when dealing with those complex data sources. You don't want to spend your time learning about each of the SAS APIs and all of their quirks. You don't want to deal with the networking complexity of an on-prem database. You don't want to get paged in the middle of the night because, you know, you maxed out your API limits and now your data pipelines are down. That's why we're introducing managed connectors with LakeFlow Connect. Before we dig into Lakeflow Connect, it's important to note that data bricks lflow is a unified intelligence solution built on top of the data intelligence platform. Lakeflow connect is one component of this. In short, LakeFlow Connect enables efficient data ingestion into your lakehouse and there are three things we're most excited about. The first is simplicity and ease of use. You can set up a production ready pipeline in just a couple of steps. And once you've set a connector up, it's also fully managed with schema evolution, observability, alerts, automated retries, std type 2, and more. This lowers your maintenance costs, but more generally, it means you get to spend less time moving your data and more time getting value from that data. LakeFlow Connect is also unified with your existing tools, with your governance. And for this, for example, this product is governed by Unity catalog. It's orchestrated via Lakeflow jobs. It also integrates with data bricks asset bundles for CI/CD and Lakehouse monitoring for data quality. Finally, LakeFlow connect is efficient with incremental reads and writes wherever possible. Moreover, since ingestion is just step one, we actually write the data to support incremental processing downstream as it works its way through the medallion architecture. And along the way, it has a super simple costefficient pricing model that's simply based on the compute used by your pipelines. Let's take a look at these managed connectors. I'm going to walk you through pipeline creation in real time. This is not sped up. We're going to start in my data bricks workspace. And to begin ingesting data, I've clicked new and add data. Here I see a couple of the connectors that are already enabled on my workspace. There are actually a bunch more in various stages of preview and I can get early access by talking to my account team. For this demo, I'm going to click into Salesforce. To start, I'm going to give it a name. Let's call it a lease demo on the day I'm filming this. And I'm going to give it a location. So even though this is a fully managed service, we don't hide critical uh event logs and cluster logs from you. So you can be confident that you know what's going on under the hood, even as your pipeline continues to run. And finally, I'm going to select a UC connection that stores my Salesforce credentials. If I didn't already have one, it'd be super easy to create one right here. It just uses OOTH. Now I'm going to create the pipeline and fetch the source schema. Now on this step, the connector is pulling the schema of data that's available for me to ingest and that data is restricted to whatever data the authenticated user actually has access to. We always respect source permissions. Now the schema it can evolve over time and like flow connect can already handle a lot of different types of schema evolution. So for example, it automatically ingests new columns of my selected table. However, it is important to note that schema evolution is a constant work in progress. So, for example, we're currently designing for automated type widening uh to further expand that coverage. So, here my schema is loaded and I'm just going to select a couple tables to ingest for the the purposes of this demo. You'll notice here that I'm actually able to include and exclude specific columns. So, if my table has some junk columns that I don't necessarily need to ingest, I can exclude those right here. Uh, next I'm going to decide where to put that data into my lakehouse. So, I can use a destination schema that already exists or I can create a net new one in line. Uh, if you're using our API, you can also write to multiple destinations per pipeline. So, for example, sending table A to schema A or table B to schema B. So, now I'm going to click through and it'll validate my setup. With this, we're going to move to the next step. And while we wait for my pipeline configuration to validate, I'm going to talk about one more feature you don't see in the UI. Uh that is sedd type one and type two. For most of our connectors, we're actually able to track that history of data over time. We also compound it with change data feeds so that you get that real full history of data uh in your table uh over time. Now here in this last step, we have to set a schedule and uh we have to select our notifications. Lake Connect is orchestrated by the tried and trueue Lakeflow jobs platform. For each schedule that I choose to set, I'm simply creating a job with a pipeline task. I can then add any tasks that I'd like before or after that pipeline task. So, you know, I could create a net new one. I could add more schedules. I could delete all the schedules. Or I could just leave the default as it is right here. Um, I also have notifications. I'm going to leave this as is. So, this means I'm going to get notified if the pipeline fails. Now the connector is going to run and we've landed on the pipeline monitoring page. Here I can monitor the status of my data and my pipelines. Uh you can see the event log that I mentioned right here so I know what's going on at any given time as the pipeline continues to run. There's also some helpful information here on the right rail. Again trying to let you kind of see under the hood of the pipeline even though it is that managed service. Now, this pipeline is already production ready, but as I continue iterating on it, it can get more advanced. For example, connectors fully support data bricks asset bundles for CI/CD to help you unhold data engineering best practices as more and more people are able to create pipelines across your organization. And that's particularly important because as you just saw, it's really easy for anyone to create a pipeline. That said, you can lock down the necessary permissions with Unity catalog if you want to restrict who on the workspace is actually able to create these. Now, this data is starting to load. We're seeing it updated as that happens. And after this, I can go use the data in a transformation pipeline, in an AIBI dashboard, or whatever my use case needs. And that's about it. I hope this video helped you learn about LakeFlow Connect. uh hear about the latest tools we offer for data ingestion and see how easy it is to use Lake Connect to help you solve your toughest data engineering challenges. For more information, uh check out our website or dive into the technical documentation which includes demos and other resources. Or you can try it for yourself. You'll see the Salesforce connector that I just demoed in your ad data page. In the meantime, thanks for watching data engineering on data bricks. [Music]