Databricks Workflows

Key Takeaways

hi everyone I'm Frank from the tech marketing team at data breaks today I'm excited to Showcase data breaks workflows a managed orchestration service fully integrated with our data intelligence platform it simplifies creating managing monitoring and repairing workflows for all your ETL analytics and machine learning pipelines let's explore a practical workflow example using our data and AI Summit cookie data set our workflow has two key objectives it adopts a datadriven approach to identify Prime locations for new bous flagship stores leveraging fresh data it also generates AI power descriptions of these locations factoring in local ingredients let's Peak under the hood to see how it's constructed the workflow consists of several components a Delta life table DLT task for data ingestion and transformation switching to DLT you can see it seamlessly integrates continuously updated realtime sales data from SQL server with franchise information from sales force both data streams were ingested using fully managed Lake flow connectors showcasing data ingestion in a real world scenario back to workflows we have an if then task for conditional branches and a SQL task that leverages an AI function to interact with large language models like dbrx GPT or Lama and then two notebooks one for the false case of the if then branch and one for triggering a downstreams app this combination allows for deficient data processing flexible workflow orchestration and advanced Aid driven analytics the intuitive pointing click authoring experience allows you to add new tasks effortlessly for instance to iterate overall flagships to all locations and perform an API call out for each I could employ a for each Loop Additionally you can incorporate other task types such as python jar files dashboard refreshes triggering a DBT project or another workflow workflow itself is a service run by data breaks with more than 99.95% availability the tasks executed by workflows run on serverless compute serverless compute provides fast efficient and reliable processing without infrastructure management offering payper use pricing and elastic scalability workflows is fully integrated with unity catalog the unified governance solution that provides centralized data Discovery Access Control auditing and lineage for example here is the full endtoend lineage diagram of our cookies example including volumes tables streaming tables and materialized views let's talk about git integration for a moment git settings in data breaks workflows enable jobs to run version controlled code from git repositories streamlining production deployments now let's explore one of workflows most powerful features automated triggers in this example I've set the workflow to run every 10 minutes ensuring regular data processing another option would be using a file trigger if a file arrives in the Bak house volume it can kick off our workflow this feature supports various Cloud object stores and offers flexible configurations based on file patterns prefixes and directories another powerful option is the table trigger for example updates to the big house sales customer table could automatically trigger our analysis pipeline for a workflow I could also Define job parameters to dynamically customize task Behavior notifications for workflow start success failure or a workflow exceeding the duration threshold and also workflow permissions to control access and execution Rights workflows include centralized monitoring to simplify life for end users this view shows me the current active runs and completed runs for the last 60 days it also includes tasks that were triggered by external orchestrators when it comes to observability I appreciate the Matrix view this view represents jobs over time complete with execution details like run ID duration and task completion status the graphical display of total execution time facilitates quick identification of longer running jobs and tasks that failed datab breaks workflow now also offers a timeline view for visualization and debugging the timeline view displays job runs as horizontal bars on a timeline showing task dependencies durations and status users can easily understand the flow of execution identify bottlenecks and troubleshoot issues the timeline view includes a critical path highlighting and the integration of query profiles this intuitive tool helps to manage and optimize complex workflows efficiently so if you detected an issue in one of the workflows you just click on repair and rerun you then Define if you want to rerun an individual task or all Downstream tasks from the issue you can highlight the arrow easily and then you can click on diagnos error which will give you a precise error message and a recommendation how to fix that issue and then if you rerun that particular workflow it's just rerunning the affected tasks which will finish much quicker and successfully finish while we focused on the UI so far in this demo workflows also provides an API for integration with external orchestrators like a py airflow we offer terraform providers if your organization already has a robust devops framework or if your engineering team is well versed in terraform however for most users looking into cicd we recommend data bricks asset bundles sometimes also called DBS daps provide a more accessible and streamlined way to manage infrastructure daps simply the packaging and deployment through a yaml file that defines the structure and components of your project including workflows and notebooks and SQL and ml models you can use any IDE such as vs code to create this file but this yaml config file can also be generated from an existing workflow this allows data professionals to focus on their core tasks without being bed down by infrastructure complexities customers can use dabs to promote their project definition from a defa to a pro environment using any cicd tool of their choice here's the key takeaway from this demonstration data breaks workflows is the fully managed Lake House orchestration service for building reliable workflows on any Cloud we at data breaks believe it's the best orchestrator for the lak house adopting workflows now paves the way for a seamless trans ition into the forthcoming unified Lake flow platform eliminating the need for future migrations