BigQuery Migration Service: Validation and optimization

Key Takeaways

So, you've transferred data from a source data warehouse or data lakeink into Google Cloud and want to know if everything made it or what happens next. This video is [music] for you. One of the easiest ways to make sure your data has been transferred successfully is with automated validations. The BigQuery migration services validation checks for structural mismatches to discover missing data, content mismatches to discover mutated or incorrect data, and type fidelity to verify schema accuracy. The results of a validation are stored in storage buckets and will contain the sample data, generated queries, and validation summaries so you can easily pinpoint and fix missing records. Check the description for more information on the validation services. Once you know all the data has been transferred successfully, you can move on to checking that all governance and access controls have also been migrated correctly. Google Cloud has great tools to help you with data governance, granular access control, data quality, and metadata. If you're thinking of creating AI applications, this is a good time to start planning the foundations to clean and accurate data as you migrate your existing data governance from, for example, Snowflakes Polaris or datab bricks Unity catalog. You will see how the available encryption techniques and data loss prevention features become very useful in this process too. With a secure foundation in place, the next step is to validate adjacent workloads like ETL pipelines, business applications, and reporting layers. This is a good opportunity to start engaging business users and getting them excited with all the unlocked potential for innovation. As the workloads stabilize, you will find aspects to fine-tune and further optimize in your workloads. The best optimizations will be incremental and based on the capabilities and architecture of the new infrastructure. Trust me, this is not the time to rebuild your workloads from scratch. Start with those changes that may be easier to implement. For example, clustering and partitioning are an easy win and also big factors in reducing the amount of scanned records, thus improving performance and lowering costs. Similarly, reducing joins in BigQuery is usually a good idea. So, the normalized tables using nested and repeated fields are a good pattern to apply when possible. You will find many tips like this one in the links in the description. While you're integrating into your new data warehouse or data lake, avoid a common pitfall and consider if your current patterns are still adequate for your needs and the new infrastructure. For example, is EDL still better than ELT? Another common pitfall is overestimating the need for real time data. Will your users still need streaming data or is batch enough? If your integrations were set up some time ago, there may now be new premises and tools as well as more potential to unlock. You probably have some insight into this from the assessment service reports. Speaking of modernization, if this isn't the current practice, consider the path to CI/CD pipelines and AI assisted coding for your developers. Version control and automated testing for your SQL data pipelines and workflows can be implemented slowly and can go a long way. Last but not least, remember we spoke about cost of ownership and execution in our first video. Make sure you stay under budget and apply some guard rails and limits to how your data is processed and queried. Your Google Cloud expert team can help you choose the right big query edition and allocate reservations across projects for your initial estimates. Things that can influence your costs are ingestion mechanisms and using different types of storage wisely. I also recommend having login monitoring and alerts to see how usage patterns evolve and keeping an eye for further optimizations. BigQuery has a recommendation feature for that. I hope you feel ready for your migration and remember there are many experts at Google Cloud ready to help you out. There's also more information in the description of this video. You got this. >> [music] >> Hey, hey, hey. >> [music]