Accelerating Analytics: Integrating BI and Partner Tools to Databricks SQL

Key Takeaways

Hey everyone, morning. Nice to have you all here. Um, you know, we're happy to kick you kick off the morning for you all. Just just just a few things to talk about. Everything we're going to show you is for looking per standard plans might change. um make sure you do all your surveys and make sure you kind of give us feedback and you know both Tucson and I are excited to have you all here. So today we're going to talk about what we're doing on the partner side to accelerate what you're doing on your you know organizations with partner tooling whether that's BI whether that's things like DBT whether that things like um other partner integrations right and our mission as part of this team is you know you all have your tooling you all have your things that you built your applications with your analytics reporting layer your you know transformations layer and we want to make sure we can empower for this you can work seamlessly with any tooling with in any language with any um kind of user persona and to be able to achieve this right we've been kind of looking at it both from a breath perspective and depth perspective when I when we kind of zoom out for a sec right over the over the years what we've been focusing on is how do we build an ecosystem where you kind of have a set of partners from BI to governance to ML to integration where you have this big set of options you can pick from. But for the top used ones that we have for example for BI these are things like PowerBI, Tableau, you know, Looker, Sigma, right? How do we give you a deeper experience that makes your workflows easier, saves you time and makes your life overall easier? Um a part of those efforts for example is you know partner connect right where you know for any of for a lot of those tools you saw you can kind of come into our UI select the partner select the integration and then we will set up a trial experience for you and then kind of bring that back and make that work but this is just one example right and you know a lot of our customers start with partner connect but then kind of expand and use some of the other capabilities that we do and a lot of this is evident from the excitement we see in the market from our customers you guys right which is what this is showing and I know this is like a chart up to the right but uh is the number of queries that we see I think this is like rolling seven day even though uh from just partner tooling like such as your BI tools like such as DBT just on data brick SQL and this has been growing like crazy year after year and as we build you know more integrations and deeper integrations this seems to be kind of giving you guys more usage and a lot of this is obviously enabled by you know having the connectivity layer, having the connectors, having you know all of those tooling at the client plan but also you know at the platform level and data brick SQL is powering a lot of these things with unity catalog right when you kind of think about partners right you can kind of have that you know analytic solutions BI solutions or you know a lot of these solutions without having the data warehousing side kind of solved and you know with DBSQL you know they That is kind of how we get, you know, the all the, you know, all the way from governance to, you know, things like concurrency, lower latencies, being able to serve at scale and kind of give you all of that at great price performance. And with that, I kind of want to set the agenda of all the things Tusan and I are going to talk to you guys about today, specifically about the partner side and all the new and exciting things we're building there. So, we have three main kind of things to cover. We'll spend most of our time on the new integration side. Kind of give you an update on what we're doing in Microsoft ecosystem with Tableau, with Salesforce, a lot of these partners. Then we have a quick segue about what we're doing on the APIs and open drivers side to kind of make it a bit easier for all of you all to integrate with your applications, with your portals, with all of those things. And finally, probably more exciting than any of the stuff we talked about before is going to be showing you some demos of how all of this new stuff works. Uh, so with that, I'll hand it off to Tusan. Perfect. Uh, good morning everybody. Morning. Thank you. Thank you. Um, I appreciate you all showing up here at 8 a.m. Uh, as I look out into the crowd, I don't see many coffee cups, so I guess you all are morning people. Uh, or maybe faking it till you make it. Um today I'm here to talk to you about the new integrations that we have uh for connecting to partners. As John mentioned um our goal and our mission is to make connectivity as seamless as possible for you all to the tools that you know and you love. Um on a personal side this is my first presentation at data and AI summit. I recently joined data bricks. Uh so I'm very excited to talk to you all about these these new integrations where we built on existing experiences and also created net new ones. So to start we'll talk about PowerBI. Uh how many of you all use PowerBI today? Can I get a show of hands? Looks like the whole room. Um and so we we know this here at data bicks that PowerBI is a very very important partner to us and to you all. And so we want to make that experience great. Um so last year we started with uh publish to PowerBI from Unity catalog. This was an experience that allowed you to go directly from Unity Catalog, take a table or schema and publish it as a semantic model within PowerBI. Uh really trying to help you all with the those ad hoc analyses. And so we wanted to take that a step further this year. uh rather than just being for one-off analyses, being able to do this at scale and for production use cases. So we have PowerBI tasks which are now within LakeFlow jobs. So you can tack this on to the end of an existing pipeline uh to once you're going from like a bronze layer to gold layer and then just send that to to PowerBI uh so that your your analytics uh will be up to date and you can do all of the things that you need to do within PowerBI. Uh this is available in public preview today. So I recommend you all go and check it out uh and give us feedback and we'll hopefully uh meet your expectations. And coming soon for this we'll also be working on incremental refresh. So this means that we'll be able to publish um those semantic models in a more performant way uh for you all as well. So staying with task types, uh DBT is again one of our uh premier partners at data bricks and uh we want to make the experience with DBT cloud just as great as it is with DBT core uh their open- source product. So we have the DBT core task which allows you to handle your transformations directly within LakeFlow jobs. And we're now bringing that to uh DBT cloud as well. And so you'll be able to run an existing DBT cloud job or configure uh and schedule a new DBT cloud job. And this one actually just launched to private preview. So, if you're interested and you're a DBT cloud user, we would love for you all to reach out to your data bricks account team to get you all set up for the private preview. The next one is on Tableau. Uh it seems like this is a PowerBI room, but I'll ask how many people here use Tableau. I mean, it's I I see some some duplicated hands here. Uh so maybe you're using both. In this case, again, we wanted to make this a seamless experience. Uh previously when trying to use Tableau Cloud, you'd have to go to Tableau Cloud, uh find the data bricks uh schema or table that you're looking for, configure the connection, a lot of manual steps, and we really want to simplify that down to a few easy steps. Um and now directly from Unity catalog the same way with publish to PowerBI you can find a uh data set uh schema a table that you're interested in hit explore in Tableau and you will land in the Tableau UI uh Tableau cloud UI and be able to start playing with that data set. So zero setup on your behalf is very easy uh to get going with analytics. So now I'm going to talk to you about some new integrations um or new integrations in the sense that we didn't have them before. So we're not really building on in an existing experience. Uh and the first one is spreadsheet connectivity. Uh when I talk to customers um it's a lovehate relationship between spreadsheets. Uh people hate them but everyone uses them. So then you all continue to have spreadsheets uh in your data architecture. And as data bricks and as the partner ecosystem team, our goal again is to make it easy to connect to the tools that you love or hate, but the ones that you use. Uh and so we started with Google Sheets. Now within the Google Sheets uh UI, there's an add-on uh for data bricks that allows you to connect uh to your SQL warehouse, run SQL queries uh and now you no longer have to go into data bricks, write a query, download that as a CSV, and then open that in Google Sheets. Um we'll continue to improve on this experience as we develop. And then the uh big one would be Excel which is coming soon. Um, we want to offer a similar experience where instead of having to uh use the ODBC driver uh and download it or work with your IT team to get that downloaded and configured or have to use connection strings uh where you can just go in, see an add-on, install it, and then get uh get going with your analytics in in Excel. So, these are two uh pieces that we're really excited about to help you all use your spreadsheeting tools. um easier with data bricks. And the next one uh is one that I'm really excited about. So as I mentioned, this is my first data and AI summit. Uh I also recently joined data bricks and this is my first full end toend launch. Uh and it's the data bicks power platform connector. uh the every year when we set out to think about what it is that we want to achieve as the partner ecosystem team, we look at uh what are the asks from you all the customers coming through uh the reps of where do you want to see better connectivity, where do you want to see new connectivity and power platform was the number one ask uh from customers. Uh so with this connector you'll be able to build power apps uh create co-pilot studio agents using your data in data bricks and create automation workflows. Uh this feature just launched to to public preview today uh specifically for Azure data bricks and coming soon we'll be offering this uh experience for our customers on AWS and GCP. And so I just wanted to hammer home like how much of a better experience that this is. Uh before this you had to use custom connectors uh REST APIs uh copy data from data bricks into other systems to try to get it into power platform and we've completely simplified that now where you can still have the uh maintained governance that UC or Unity catalog provides. You no longer have to copy data because it's a live connection and it's all embedded within the uh Power Platform UI. You just go in, configure the connection, and you're off to the races with building uh your Power Platform applications. And so, don't just take it from me. Uh we've had a lot of customers that we really appreciate who've helped us test this product out and make sure that it's ready for you all uh in public preview. Uh one of them is Coti which is a global uh beauty brand and they use power uh platform to build applications that can help them manage safety incidents within their warehouses. Uh which is very crucial to keep warehouse employees uh safe and also make sure they're delivering highquality products. uh before power platform they were using manual reconciliation across Excel files or uh handwritten documents uh which obviously led to frustrations for the people who had to do that reconciliation uh but also led to errorprone uh issues and as a result they moved to power platform and they were able to develop these applications but the last step was being able to connect to data bricks uh they had to have a a custom connector that they were maintaining and had to keep updated and now we've completely simplified that experience for them uh so that they can focus on keeping their warehouse employees safe rather than how are they going to configure the connections. Um the next one is UK power networks. uh this is a leading energy and electricity bl uh company within the UK and uh for them they use power platform to uh help when there are when inclement weather has occurred and they're getting a lot of phone calls like where's my power my power is out uh and so they've built a power uh app that helps their call handlers be able to provide customers with real-time information which is crucial uh in situations where weather is bad, which I assume we all have have experienced. Uh, and so now again with the Azure data bricks connector for for power platform, they're going to be able to uh provide those live updates much much easier and there's going to be uh less of a learning curve for them to continue to set up new power uh platform applications with their data in Azure data bricks. So I'm going to hand it back to Jean to talk about uh drivers and APIs. Awesome. Thanks, Tusan. So, just a quick segue before we switch to the demos is a few things we've been kind of excited about on the drivers and API side. And just to zoom out a bit, right? You know, when we kind of think about the ecosystem, we kind of think about all the connectors for individual BI tools that we have, right? Things like Tableau, PowerBI, Power Platform stuff. But at the same time, we're also looking at it from like, hey, you know, our customers have applications. there's a lot of connectivity needs outside of these specific um you know built-in app packaged apps. So how can we serve you know your Python apps your Go apps your NodeJS apps in a way that's performant that's stable and that's you know scalable and that's why when we talk about the ecosystem a big you know 50% of the ecosystem is about you know giving you guys this programmatic capability across your stack and making sure uh you know that is solid. So this year, you know, there's two things that are kind of exciting and new that we've been working on. One of them is we actually releasing a new open source JDBC driver, soon to be open source. It's not too open source just yet, but it is actually up on our download page. You can go ahead and start using it. You'll hear more about it later on. This is a way for us to just a bit iterate a bit faster and kind of give back to the community. Um, on the second side, a bit more exciting. We're we partnered with Microsoft to also work when a on an AD ADBC driver which is arrow Apache arrow based database connectivity. Um and this is something that Microsoft is doing on their side with PowerBI so that they integrate this driver as opposed to the OBBC driver they use today to give a better overall experience in terms of you know DAX operator pushdowns faster refreshes faster data transfer and all of those things. So, we're excited about both of these things um kind of happening in the open source um ecosystem. Um so, with that, I kind of want to switch gears and start kind of showing you guys some stuff. So, we will go through like four demos, right? We're going to go through we're going to show you all, you know, using Google Sheets that integration. We're going to show you all about uh how we do the PowerBI tasks and how the publish that kind of works PowerBI pass type and then we're going to show you a few things about the you know power app side that we've been working on. All right. So with that let's switch over to show you all the Google Sheets experience. So one of the things I'll say is today right if I wanted to go from data in data bricks to Google sheets what I have to actually do is go to datab bricks UI download a CSV or excel come here say import and then you know do a lot of that but that involves a lot of steps you you need to remember which query lives were and kind of get all of that which is kind of horrible. What if instead of that I can come into the Google Sheets UI directly you know get the data bricks connector for Google sheets log in to my workspace using you know my single sign on experience that I have with you know my data bricks workspace from there right start by picking the warehouse I have again I don't remember to need my cluster URL I don't need to remember any configuration parameters I pick from a UI in this case I can pick a serverless warehouse for example or something and then I type my SQL query and then I can just run it. And behind the scenes, right, this actually uses the SQL REST API. So, one of those open APIs that we've been talking about. So, it does not have any other dependencies or it doesn't have any other services in between. Talks to your data bricks environment and then brings back uh the results just like that. You can refresh them. You can keep them up to date. You can manage multiple queries. We're going to enhance this experience so that you can see your historical runs and then you'll be able to manage more than one query refresh at the same time. But this is the direction we're kind of going on the spreadsheet side. That sidebar experience is going to be similar on Excel too. And as we improve and add capabilities to both of those things, for example, maybe like a lower code experience where maybe you don't have to write as much SQL, right? Those things are going to be available across both of those products. So we kind of want to have a consistent, you know, end user spreadsheet experience when you're kind of connecting to all of these things. And uh this is live already is a public preview. Excel is going to be coming soon. So excited to hear what you guys think about it once you play with it. So next up in our list is the PowerBI task type. Just to recap, right, publish to PowerBI, as you remember, as Tusan mentioned, is a way for you all to hand off your work of your gold tables and make that available in PowerBI for analysis. Right? How it works is it basically takes your metadata about the data set you have, you know, the exact path, the columns, the data types, you know, the cluster you're using to create it, the security details, sends that metadata over to PowerBI, right? so that you can directly start you know making use of it or your analysts start making use of it without any other configuration or having to remember all of these connectivity details. What we've done this year is we now added that to the workflows UI because we know you know not a lot of your admins are in the UI clicking stuff right so that you can now select PowerBI as a task type directly as you're building your workflows and as part of your workflow right let's say there is you're adding a column you're removing a column you're changing metadata what you can do is whenever you rerun this workflow now all of those metadata changes or whatever else you're doing will be published to PowerBI so you don't have to kind of think about the separate process to keep them in sync. You can just tag them along um to your workflow that you're doing already in the database UI here. Um and there's a bunch of configuration options that we propose here. You can pick for example whether something should be direct query, import mode, hybrid mode later on and a lot of things like that. We're excited about this one. Um but yeah. All right. So next up we have Tucson going to be talking about the publish to Tableau experience. Got it. Uh so this is the the explore and Tableau experience that I that I talked about previously. Um just a quick sidebar talking about the experience we also built for PowerBI. We have the publish to here. Um, so we're in Unity Catalog and say I'm an analyst and I'm like, I just need to visualize this data right now in Tableau for whatever the case may be. Maybe my boss is asking for this information and I need to do it as quick as possible. Uh, so directly from Unity catalog, I would take that information. Um, I would hit explore in Tableau cloud and then Tableau is going to launch. Um I'll have to configure my connection. One second here. And now the the uh connection is live. So as John mentioned for PowerBI, the metadata is just what was uh published to Tableau. And from here then I can start doing my my analytics workloads. Uh so say for example I took the pickup uh zip code information and now I have a visualization. So what I'm just showing you here is that uh the connection is live and as I interact with this uh Tableau dashboard the information is going to be queried from data bricks and then visualized into uh Tableau cloud. And so we're really just trying to simplify that experience. And so the last one is uh say for example you don't just want to visualize the data you want to be able to interact with it uh change the data uh that's where the power platform uh connector can come in and help you all. So as I mentioned it allows you to create uh power apps power automate uh automation workflows and and copilot studio agents. Uh I'll focus on the the power apps piece within this demo. Um within Power Apps we allow you to uh read from data bricks so you can show data uh but then we also allow you to uh write data back to your datab bricks instance as long as you have those permissions in unity catalog. So that's something I want to be very clear on. This is all governed by unity catalog and the permissions that you've set up. Uh and then just to talk touch on the other pieces with the uh Power Automate flows, we're hitting the SQL execution API directly. And so that is for those use cases where you need to query a large amount of data or maybe you want to do uh batch updates and you want to send 30 or 40 rows for a merge statement. Uh you can use the uh power automate capabilities of SQL. And then uh Copilot Studio, you can use data bricks as a knowledge source uh when building your agents. Uh so just give me one second here to set this up. Okay. So we're in um power uh apps and I've already created a connection here. So I I won't bore you with with that piece. Um but just to show you what the experience is like, you hit add data. You find the Azure Data Bricks connector because again this is only available for Azure Data Bricks right now. You click that. You select which connection that you want to use. Say for example, I want to use uh this connection here. Now, it's going to show me the cataloges that are available uh for me to use from Unity catalog. Uh let's say I click the one that says uh Power App, right? And it's going to show me then um the uh table name, but it's going to have the uh schema and catalog prepended to it so that if you have duplicated table names in a catalog, you won't get confused of of which table that you're actually trying to access. Uh so I'll close out of that. Um and then you would be able to see the data that is in your application directly within the Power Apps UI. Um, and then just to give you a quick overview of what's happening in this application, this is live pulling information from data bricks. So that same table that I just showed you um is what is populating the sidebar and uh then we're doing some filtering based on the search criteria. So if I hit play on the application um let's see if I'm able to type this. I type a name and basically what's happening is it's executing uh SQL query on the data brick side uh to filter for this particular bit of information. If I delete all right it's going to again refilter I can do uh sorting by name. Right? So all of the core power FX functions that you would be looking to support uh data bricks is able to translate those to SQL and execute against those. Um and I select one and it shows it here. So then that's the reading portion of it. But then if we talk about the ability to update records or create new records or delete records uh that capability also exists. Uh so if I create a record let's just do this really fast. Uh fair Okay, that's not a real fair. Uh, $45 and let's just call it primary key of,40. Um, so when I submit this, this is again translating the power FX command to um data bricks SQL insert statements and is inserting the data into your table that is already existing in Unity catalog. And we got a success metric uh uh notification here. So if I search for I think I typed AAA then you see the the data here as well. Uh this works for update statements as well as uh deletions. Um so that's the the power platform connector as it relates to Power Apps. And again this works for Power Automate. You can write SQL queries and copilot studio. You'll be able to use the agent workflows and ask questions of your data bricks data. Um, so that is the the end of the demos that we have. Um, so we want to open it up to you all to be able to ask questions. Um, and hopefully we can can help answer those questions live or maybe do a followup.