Table Migration and Catalog Design with UCX | Part 5
Skills:
AI Workflow Automation90%
Key Takeaways
The video tutorial demonstrates table migration and catalog design using UCX, a tool by Databricks, covering initial catalog and table mapping, output updates for seamless migration, and optimization of the Unity Catalog upgrade process.
Full Transcript
hello everyone we are continuing with the ucx video series my name is Andrea and in this video we'll cover the first steps necessary to let ucx know what your future catalog and schema assign is going to look like for the migration we do have some prerequisites to make sure we cover so let's take a look at the repo those are ucx needs to be installed in the workspace of course we have some special configurations if we have an external meta store such as glue so make sure you review the right guides the assessment workflow job must be already executed we recommend having already completed the group migration process if applicable and finally Unity catalog needs to be enabled on the workspace so a metast stor should be already in place and linked the up process begins pretty simple we just need to execute the create table mapping command to take the output detected tables from the assessment from which ucx will create a new csb file we will need to modify this new CSV file to ensure that our predefined design is correctly configured before executing the command let's review my assessment results for demo purposes I only have two tables in Hive One external and one managed my external table is pointing to a storage location in the cloud okay let's execute the command the created CSV file will be saved into the applications ucx folder that was created during the installation of the tool here is the command that I will execute we have the first portion which is dat bricks just to make sure that we are calling the DAT brick CLI Labs which is the command group for the labs organization projects ucx to make sure that we are calling the right project and then the command which is create table mapin in this case I'm using this profile which is the one that has access to my workspace so it's executed ucx will take the authentication authenticate to the workspace and then execute the commands after the command runs a new tab will be opened on the browser this tab contains the newly created CSV which maps to the old tables to the new three- L naming configuration this CSV contains a set of columns those are the name of the workspace this is useful if you are working with multiple workspaces the target catalog to which we are going to migrate The Source schema and the destination schema this is Hive and unity catalog and the same thing for the table source and destination we must ensure that we choose the right catalog and schema and we can also modify the table name in this case I will modify my catalog name a bit let's call them ucx migrated catalog managed for the first one and ucx migrated C calog external for the second one just so we can keep track of the migration we'll leave the rest as it is this configuration is easy when we have a low number of Records such as in this case but when we are working with a large number of tables our recommendation is to export the CSV to your machine and modify it on any external tabula software this will help you avoid mistakes and make sure that you are inputting the right configuration for your environment in the next video we'll see what the commands necessary are to give ucx workflows access to the cloud storage to be able to make the necessary changes on the cloud thank you
Original Description
Master table migration and catalog design with UCX in this essential tutorial. Learn how to:
• Execute initial catalog and table mapping
• Update outputs for seamless migration
• Optimize your Unity Catalog upgrade process
GitHub repository for UCX:
https://github.com/databrickslabs/ucx/blob/main/README.md
Playlist
Uploads from Databricks · Databricks · 7 of 60
1
2
3
4
5
6
▶
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
Building AI Agent Systems with Databricks
Databricks
Databricks Workflows
Databricks
Automate Unity Catalog Upgrade with UCX Part 1: Overview
Databricks
Automate Unity Catalog Upgrade with UCX Part 2: Installation
Databricks
Automate Unity Catalog Upgrade with UCX Part 3 - Assessment
Databricks
Automate Unity Catalog Upgrade with UCX Part 4 - Group Migration
Databricks
Table Migration and Catalog Design with UCX | Part 5
Databricks
Setting Up Azure Access for UCX Table Migration | Part 6
Databricks
UCX Table Migration: Creating Catalogs and Schemas | Part 7
Databricks
Automate Unity Catalog Upgrade with UCX Part 8: Code Migration
Databricks
Streaming to Kafka Just Got Easier with DLT Pipelines
Databricks
Data Engineering From Data to Dashboards with DABs: Crunching the Cookies Dataset
Databricks
Epsilon helps businesses connect with their consumers using Databricks Data Intelligence Platform
Databricks
Unilever transforms operations with GenAI using the Databricks Data Intelligence Platform
Databricks
ActionIQ enables businesses to unlock customer data with the Databricks Data Intelligence Platform
Databricks
Mixed Attention & LLM Context | Data Brew | Episode 35
Databricks
Inside Databricks SQL: Engineering innovation with Hans
Databricks
Inside Databricks: Engineering innovation with Michael Armbrust
Databricks
The Money Team at Databricks: driving revenue and customer growth
Databricks
Unity Catalog unveiled: engineering data governance at scale
Databricks
Create a view in Databricks and share it with Power BI using Delta Sharing
Databricks
NDUS leverages Databricks Data Intelligence Platform to revolutionize higher education management
Databricks
Démo Databricks de AI/BI
Databricks
EMEA Data + AI World Tour 2024
Databricks
GenAI: The Shift to Data Intelligence - Customer Panel on Industry Use Cases
Databricks
GenAI: The Shift to Data Intelligence - Ft. Ash Jhaveri, VP of Reality Labs Partnerships at Meta
Databricks
Virtue Foundation leverages the Databricks Data Intelligence Platform to advance global health
Databricks
Announcing Synthetic Data Generation in Mosaic AI Agent Evaluation
Databricks
AI/BI Dashboards Embedding - A tutorial
Databricks
Bayer transforms global data management with the Databricks Data Intelligence Platform
Databricks
Databricks at AWS re:Invent 2024
Databricks
Hive Metastore and AWS Glue Federation in Unity Catalog
Databricks
Data + AI World Tour Paris 2024
Databricks
Retail reimagined: Currys data-first strategy to driving growth and improving operations
Databricks
Mixture of Memory Experts (MoME) | Data Brew | Episode 36
Databricks
Verana Health Data Curation and Innovation with Databricks and AWS
Databricks
Securing SaaS Applications: Obsidian Security on Their Journey with Databricks and AWS
Databricks
Twilio Eng VP on Data Intelligence & AI at AWS re:Invent 2024
Databricks
Chegg Eng SVP on Data-Driven Approach to Student Success with Databricks and AWS
Databricks
Ibotta Personalized Rewards Innovation with Databricks and AWS
Databricks
Simplify AI governance with #databricks AI Gateway
Databricks
Databricks SQL and Power BI Integration
Databricks
Databricks Serverless SQL Warehouses
Databricks
7 West powers audience growth with the Databricks Data Intelligence Platform
Databricks
Secret to Production AI: Tools & Infrastructure | Data Brew | Episode 37
Databricks
Skyflow CEO on Data Privacy with Databricks at AWS re:Invent
Databricks
Databricks Clean Rooms Product Demo
Databricks
Dun & Bradstreet Enrichment & Monitoring, powered by Delta Sharing & Databricks Marketplace
Databricks
Unpacking Libraries in Databricks
Databricks
Providence uses an AI agent system from Databricks to help doctors improve their communication
Databricks
How State Street Uses AI to Transform Millions of Trades Daily
Databricks
Vevo Therapeutics CEO on Curing Disease with Data at AWS re:Invent
Databricks
Over Architected with Nick & Holly: Databricks updates for Feb 2025
Databricks
The Power of Synthetic Data | Data Brew | Episode 38
Databricks
Use Databricks Lakehouse Federation to break down data silos
Databricks
AI's rugby score: National Rugby League rallies fans with analytics and unified data
Databricks
Open Variant Data Type in Delta Lake and Apache Spark
Databricks
How would you sort Ætheldred in the alphabet using Databricks?
Databricks
A guide on how to operationalize the Databricks AI Security Framework (DASF)
Databricks
Future-Proof Your Asset Performance Management with Generative AI - Field Assistant Live Demo
Databricks
More on: AI Workflow Automation
View skill →
🎓
Tutor Explanation
DeepCamp AI