Databricks updates for December 2025 with Nick & Holly [Holidays Special]

Key Takeaways

Should we should Should Santa just give up? I got it. We're replacing Santa. AI is now Santa. We run AI query. We give it all this stuff. We don't ask [music] any questions about how it figures it out. And we say, "What is this person getting?" >> Okay. Okay. >> That's it. I I I like the idea of putting Santa out of a job. Don't listen to this with your kids. Hello. Hello. And welcome to Overarch Architected with Nick and Holly, where together we're experts in data and AI, but separately, not so much. This is the podcast where we talk about new features in data bricks and then try to shoehorn them into one architecture to see if it's actually realistic. And here are our favorite features from the last 30 days. And I think I'm going to go first and one of the first features I want to talk about is Unity Catalog convert foreign tables and where you can do it from hive meta store or glue. Now if that didn't make any sense to you, let's back up a little bit and start by talking about what is a foreign or even a federated table. So this is data and metadata that is part of another system and you can now access them via unity catalog. So in your datab bricks environment so that you've got all of that wonderful governance with it as well. So this is this is fantastic. So this is access to data in a different system without needing to do a migration. Within that you've got two different flavors of tables. So you've got external tables and you've got managed tables as well. So if you're converting one of those tables to an external table, you get things like uh the table history. Uh you can use things like delta, parket, ocro, json, csv or text. Or if you wanted to go for a little bit more, um you could make it a managed table. Uh if it's a delta table and then you get things like predictive optimization as well. If you wanted to, you could also optionally move the data as well with a single command. Um this does disable the source access as well and so then unity catalog becomes that system of record. Uh but you can still copy it if you want to. Now with this functionality comes a whole bunch of new features like new syntax. So you can do a dry run to see what is this doing to your tables before you actually go ahead and do it. Uh if you realize you made a mistake, you've got roll back. >> So we have a foreign table. >> It's already it's already registered, linked, federated. Yeah. >> Um, and this feature allows me to one click one click convert into a managed table. >> Managed or external table. Yes. >> Or external table. Okay. >> Got it. All right. So, so managed will actually copy the data over as well. >> So, you can optionally move the data if you want to um and then disable the access from the source. Um, but you could also do copy as well instead of uh doing the move. So what is a manage table that is not copied where the data remains? >> Unity catalog will do your predictive optimization. So it will manage what the file sizes look like. It will manage um the optimization and the kind of reads performance tuning as well. >> Okay. So here manage doesn't mean that the data comes over necessarily just that it's managed. Sorry [laughter] I'm catching up. Well I really want I really want to dial in on what managed means. It's like a historically very annoying thing. If you Google it and look up the difference between a managed and an external table, it's like the number one question on stack. >> And I think previously it was all down to kind of location and now we're really blurring the lines with >> well external used to mean it was like oh okay well the you know the file location the path was something that Unity Catalog wouldn't control. But now that's not necessarily true because you could have a manage table that lives in glue and it's not necessarily managing the kind of the file path the location that it's going to. But it is managing things like the snapshot versions, how often you're doing compaction and optimization and Z ordering. Oh, it's not called Z ordering anymore, is it? Um, liquid clustering. >> And then remind me, it's not just delta tables, right? It's >> so if it's a managed table, it is a delta table because only delta really needs things like liquid clustering or um version cleanup. Uh, but if it's you've got something that's not delta, then you'd use something that is an external table. So if it's pocket or OC or whatever whatever it can still be external and you still get all the kind of governance access management stuff and the table history too. >> Okay cool. In terms of getting our architecture started I'm just going to write a table. What does our table have? >> Uh so your table is probably either in the hive meta store or it's in a glue catalog. >> Let's make it let's make it somewhere somewhere far somewhere far far away in the land of glue. the land of glue. [laughter] >> We have a table. What's our use case here? What are we doing? >> Well, we did a Halloween special last week. Last week, last month. I think we can do >> So, we should do Christmas. >> Yeah. Okay, then. And it's far, far away. I think that is the North Pole. Frankly, that's where our table is going to be. >> It's going to be What should we put in it? >> I don't know. [laughter] >> Oh, we should put like a list of good little boys and girls. How about that? Uh well, all people are included. Let's just call them people. >> People in line for Christmas presents. Okay then. >> Uh yeah. So we have a people. We have our people table. People. >> Okay. >> And this is glue. So this is all like in a different universe, the north pole. >> And we have UC Unity catalog. We are going to I don't know which direction to make the arrow. Is that is that how I'm going to do it? >> Uh I feel like it's kind of going there and then back. Like the request comes from Unity catalog. That's where all of your access management happens about you know should Nick be able to access however many billion people are in this table and then the data is going to be returned via Unity Catalog. >> Yes. The answer is yes. I should have I should have access. So let's do is that the double arrow you're looking for? >> Yeah. Okay. I mean I would have done it. It's fine. It's fine. It's fine. It's fine. Okay. It's fine. >> [laughter] >> Okay, let's say we convert it. Convert it. >> Uh, something that's quite nice about this um is this you could do this as part of a migration. If you sort of want to start using some data, but you don't really want to commit to nine months of moving something over. You're like, "Hey, I just need to quickly set something up, get access to this other data, and then just do my project rather than trying to boil the ocean and do everything all at once." >> All right, AI features hit me. >> AI feature. We're going to start with one that we've talked about in the past, but it was always in betas and in previews and um that's AI parse document. Um I do think it's just worth calling it out again. Um I think one of the things that when we started talking about it, it only supported images. Now it supports docs like word docs, doc, doc x and it also supports uh well pdfs were supported before um and also powerpoints. So very useful. There's like a there's a new schema version. So there's now we've had iterations. So there's different versions of this, but the latest version two supports things like uh pulling out just descriptions of figures. So um when you actually run a parse document on some kind of document that has figures in it, you can say, hey, I don't want to parse everything. I just want the figures and the descriptions of the figures. And so then you'll get back the elements in a structured format. Um and that'll come with like the bounding box. So you get coordinates and some description and whatever the contents are of the figure. So um very very extremely useful particularly for like enterprise type things where your documents have things like graphs and charts uh tables things like that. >> Okay. So yeah so things like tables. >> Yeah. >> It actually puts the data the numbers in the right place rather than here's a random set of headings and then you get to guess where the numbers go. [laughter] >> Yes. And you also uh like before you get like the the headers, the footers um along with that metadata. So it's it's not just text. You'll actually get, hey, this is the element. It's a header element or a footer element. Here's the actual contents of that element. Um, so it's not just the thing, it's also the metadata. It's whatever it is about the thing, its placement. Um, so it's an incredible feature. So it's bestin-class. It's better than any any of the other existing uh foundation models out of the box. I think we are going to have a bunch of documents, uh, document store. Uh, and within that we have some PDFs of things. Uh, we've got like docx files. Um, what else was it that you said? >> PowerPoint, >> PowerPoint, EP. >> Right. I think in here we could have like a bunch of toy diagrams maybe of what Santa is building this year. Is that >> Yeah, sure. Manual. How about and and like manuals? Toy manuals. >> Yeah. Okay. Yeah. Yeah. Yeah. So, uh document >> safety considerations >> toy. Yeah. Toy documents >> for young children. >> Um sorry, said that again. [laughter] >> No, I'm laughing at the toy document store. Is it a toy document store or is it a toy document store? >> Uh well, that's what I always wanted as a child was a toy document. Like a toy PowerPoint. That's That's a good Yeah, that's why I'm interpreting it. [laughter] >> Here we go. All the good little girls and boys can get proper toys and the bad ones get a toy PowerPoint. [laughter] >> Start presenting. [laughter] >> Uh yeah, when I grow up, I want to be a salary man. Okay. Um outline designs. Uh okay then maybe should like some trend like a trend analysis um in our PDFs as well like what's going to be the and we hypothesize what is going to be the toy of the season is it convoluted yes do I like it >> all so these are all documents document all the documents basically is what I'm saying [laughter] >> okay >> then cool I [snorts] like it no connection to our people table yet right just making sure [laughter] we have people we have toys a good start it's a solid Okay then. All right. Well, our next feature is going to help us connect to more data as well, and that is that Unity catalog now has JDBC connections uh which you can now add into Unity Catalog. So, spoilers, this is part of the underneath workings of what makes federated queries work, but we're also exposing um some of the underlying technology so that you can be a little bit more flexible with it pre as well. So, previous we spoke about kind of Hive and Glue, but if you've got something that's neither of those things, which let's face it, you probably do, you can use uh this JDBC connector that is part of Unity Catalog, which means that it is governed. Uh so it doesn't have fine grain access control, but it does things like allows for writes. Um it allows for like Spark data source tuning. It allows for push down queries. is you can use a specific driver if you want to and just wider support of things that you can connect to and have that connection managed by Unity catalog as a thing in there. So you can decide who gets access uh to what because the authentication is with a username and password. So you might not want to have that in um a notebook. You probably want to have that as like an uh things something that is managed by Unity catalog. Um, and so all of that is also going to be captured and audited as part of system tables too. >> It's a JDPC connection, so I can point it at anything. >> Yes. >> Am I understanding that correctly? >> Yes. >> Um, okay. How about we just add some location data? Location, mapping, globe, reindeer path, reindeer paths, >> all kinds of stuff. >> Okay, so it's our custom reindeer. [laughter] custom reindeer system that we need to connect to a JDBC. Okay, got it. >> Yes, I'm totally into this. This is our brand new JDBC connection. >> And let's just move this slightly here so it's pretty. All right, we have I think this is good start. We have a lot of We have We have our data stuff figured out. >> So, I like how this is looking. Back to AI land. >> Yes, >> because we need to do something with this stuff. So in AI foundation model land uh some exciting stuff uh kind of three um series of models. So new model from Alibaba Quen 3 Next instruct is now available on data bricks. Open AAI's G GPT 5.1 also available on data bricks natively and Gemini 3 Pro preview which we might be a little bit out of sync cuz I Gemini 3 is out now so you know oops but that's okay. uh we are recording async. So um those are the three models. So that's very exciting. All part of the foundation model API all available natively within data bricks hosted and run by data bricks. Uh so these aren't just as always proxy calls to some other provider. This actually data bricks uh running. All these models are extremely capable. Um so it really comes down to testing them for your use case and choosing essentially you know for price to performance what makes sense. Yeah, I think that's an interesting point about just being able to quickly swap in and swap out and not necessarily having to set up like a big new thing every single time that you want to test out something new. Okay, then um so in terms of how to use these models, what what do we think we can do? Because we can't use them for AI pass document, can we? >> No. AI parse document we that's kind of our proprietary model but we can provide these different foundation model um endpoints to other AI queries uh in SQL. So that's one thing we could do. We have people we have presumably where they are and we have the toys. Should we should Santa just give up? He should just I got it. [laughter] We're replacing Santa. We're we're replacing Santa as it Santa is replacing himself. AI is now Santa. We run AI query. We give it all this stuff. We don't ask any questions about how it figures it out. And we say what is this person getting? >> Okay. Okay. That's it. >> Maybe we should do like a toy recommener for Santa. Be like match this up. Yeah. Based on >> I like the idea of putting Santa out of a job. Don't listen to this with your kids. Um, >> so we've got AI center query. Uh, what are we going to do? It's like, uh, select toy, uh, from AI query all the columns. [laughter] >> She's going to concatenate them all together, which got to like based on people's names, names, um, and location. maybe uh pick the toy for them. Uh oh, and then you could specify which model. Um, so yeah, we'll do a prototype and we'll start with Quen, which is spelled like this. Quen. >> Yep. Quen the third. >> Quen the third. >> Uh, I feel like that query makes absolutely no sense, but I like it. Anyways, [laughter] >> like this is this is excellent pseudo code. Um, [laughter] I like this idea that like every single Nick is going to get probably a similar um a similar toy for Christmas because it's based off your name. Your surname is carp. Are you going to get a chest set? Is that what you're going to end up with? >> Oh, that would be so lame. [laughter] >> Only [laughter] Only if it's life size. It was life size. You want a giant chest set? >> Yeah, like a Harry Potter style. >> That's the first the first Harry Potter >> uh wizard chest set. Okay, got it. >> Okay. >> Um maybe it's got your date of birth and it'll figure out that you are a millennial. You're like peak Harry Potter fan. Can I talk about um my next feature that I'm interested in? >> Sure. You millennial. If you are a Scala or a Java person, uh you up until now probably really haven't been able to use serverless unless you wanted to switch languages. That is not the case anymore because now with serless you can use JAR files, the Java archive files. >> So yeah, so you can now run your JAR files on serverless. Now unsurprisingly, these jar files do need to be uh data bricks compatible ones. Um, and it is serless version four and higher that uh you can do this with. So that's a nice one for our Java and Scava Java and Scava people. That's not right, is it? I'm going to leave that in there. That's fine. >> Or our Scava people. >> It's it's uh it's funny because I I started using serverless so much more and as a result, historically speaking, all that's done is I've become much better at Python. So now you're telling me I can go back >> Yeah. Yeah. >> And do stuff do stuff the old way. >> I'm saying no onwards onwards and forwards for me. Where are we going to put this? >> So sorry so you say about kind of new development, but I think it's also really useful for people who are who already have things that have been deployed in kind of traditional clusters, non-serless serverful cluster. That's not right, is it? Um, but I genuinely like some of the performance stuff that you get with serless is incredibly impressive. So if you've got something where you've kind of sort of half tuned it but not really had the time to think about it, test it out on serless, see what the performance is like and you might actually save yourself a bunch of money with maybe a week's worth of work rather than having to go through and like really think about the tuning that you need to do in Spark like you would with kind of classic Java or Scala. >> Nice. Good call. Good call on the older workloads. Didn't consider that. Here's what we're going to do. Speaking of old stuff and we're going to put like that another box on top of this and we are going to write jar. >> I can see some SQL written [laughter] in there if you just basically done spark.sql and now you're going to put [laughter] >> that's that's enough questions. [laughter] Uh, >> okay. So, >> straight up, we've got our first anti pattern. [laughter] >> The whole thing is an anti pattern. No, no, no. This fine. This makes this first of all, the amount of code that exists uh it could still be written in Scotland um or in Java or something like that. And it's doing a bunch of other stuff, but it has this query embedded uh in it because it was easier to write and maintain than uh instead of writing it in uh like using the actual Scola or Java APIs. >> Actually, like I Not I mean is it an ant pattern? Lots of people do this. >> Well to put exclusively SQL in a spark.sql statement and then put it in a Java file. >> Yeah. Yeah. Well, it's not the only thing that's going on. It's just like part of the code, right? So, >> I mean, if it's part of it, like that's fair game. But if you're just >> taking some That's what it is. That's what it is. Don't ask. >> If you've been holding off using SQL in serless until now and jar is finally the thing that has unlocked it, I think there's other things going [laughter] on. >> Uh, okay. Okay. So, we support it, but uh jars for serverless >> for serless. And it's like a job. So, I should basically write all this >> somewhere. Huh. >> It's your AI Santa. You You draw it the way you want. Nick, >> jobs aren't referenced in Unity catalog. Jobs are just else or elsewhere. There it is. Jobs. Jar job. >> Job. Lovely. Next AI feature. Hit me. >> Big month in agent bricks and MCP. First of all, MCPs have kind of been upgraded to a much more first class citizen uh within data bricks. So there is a MCP server tab now available. Um in that tab is uh kind of a list of the MCP servers that you may have deployed um or the existing managed MCP servers that come with data bricks. So I think historically we had Genie spaces which uh kind of automatically became an MCP server so you could include them in other things. So that's one. Um I believe the other one was vector search also came with an MCP server out of the box and the uh new one is Unity catalog functions. So if you have functions right that you've registered in Unity catalog you get MCP out of the box for those functions. So that's super useful. What's notable is the new managed um a new managed datab bricks MCP server is also um DBSQL. So SQL MCP server is available. Wait, sorry. SQL Server MCP. Wait, what? Sorry. [laughter] >> Uh, DBS SQL is ported through through the MCP uh interface. >> Yes. >> Right. >> So, if you attach So, if you attach the MCP uh server as a tool to your LM context, the LM will be able to actually execute SQL queries. And sorry, one more thing is that we also support them through uh marketplace. >> So that's uh that's that's now public. So you can if you have an MCP server that you want to publish, you can publish and uh I guess what is the right word? Not download. You can use other MCP servers that other people have uh published. >> Interesting. I was going to have this AI query thing as its own MCP server and have that as a tool, but we've put it in part part of a JAR file, so I don't think that's going to work anymore. Or we could make our naughty or nice MCP server for other people for targeted advertising. Nick Kov got a chess set. Chesset is sad. Now we know that we can advertise to him because he had a sad present and is more likely to buy more on Boxing Day sales. How about that? >> Sure. [laughter] Good storytelling. >> Very, very thorough story about me and my chess set. >> You're the one who said it was sad. You know, some months uh we build something that's like sort of sensible. I get a lot of comments from people being like, "Hi, Al. Be really useful if you could actually build this for us so that we could like see what it looks like in real life." I don't think anyone's going to do that this month. >> No, we haven't really It hasn't really come together. >> No. Um, unless Santa messages us and tells us to stop and >> we're on our way to our first one. One one out of 10. [laughter] >> MTV, uh, Naughty on Ice. Um, and what else is it going to do? Um, toy received. Uh, what else is this MCP server going to do? Toy Sentiment, should we say? [laughter] >> Oh, and um, because they're are they now part of Unity Catalog as well or the governable? No, they're not, are they? Oh, the tools are >> the tools are, but the MCP server itself is not under uh the Unity catalog. >> Okay. Well, I'm just gonna point this arrow to tool. Uh, and then MCP is not in Unity catalog. Okay, then. Are we happy? >> No, but we should continue. >> No, I'm not happy. So, therefore, I will not dwell on it. Okay. Well, our next feature is uh going to be very useful because we've got all of this personal data going around. Um, and that is that Aback is now in public preview. This is attributebased access control. Uh, and there are some changes because it's been in beta for a while. So, if you've been using it, um, now it has some changes to it. So, if you're not familiar with Aback, ultimately what you're doing is tagging assets. So, you can say that this contains personal identifiable information. Uh, and with that, you can either do things like row filtering or column filtering based on who the person is or what groups they belong to. You can enforce this with policies and then you can audit it all via system tables as well. In terms of the changes from beta, this used to be at a workspace level, which means it was very much a kind of girl guide promise of well, I can't access the data through one workspace, but I could do it through another one. Um, this has now changed. It is now account level um enforcement. Um I will say if you've been using dedicated compute uh for a while um if you are on a runtime that is earlier than 16.3 which is around March time um there are some changes um that could come to you which are going to be a bit of an issue. So you've got two options basically. Either you do nothing and it just rolls out to the public preview. If you want to disable it there are a few steps that you can go through to be able to disable it. And if you are using some dedicated compute you might want to do that. So, we have Santa who's using all this and then we have elves that are using all this. >> Santa is like an admin. Santa can see everything. Santa knows everything, >> right? I mean, it would be weird if Santa was like on data bricks and it was like then I it's like if I don't have access, who's got access? >> Yeah. >> Um, so Santa gets everything. The elves only get access to the toy document door because the elves are responsible for making the toys and they can't see who is getting them and where it's going. >> Okay. >> So, what do we think of that? >> Uh, yeah. Yeah. Yeah. So, we could tag the tables to be like to say, >> uh, this is this. I think you could also like redact the names for the elves. So, they could see like the people things like maybe they could see age or the year that you were born in. uh they can't see kind of like down to the down to the individual date level because that doesn't really make any sense. Um and then also things like location, you know, it might be useful to know the country um but um they might not want to give access to um like the individual kind of address level. >> How's this for a visual? Elves, Santa, and elves. [laughter] >> I like it's it's a two out of 10 now. [laughter] >> All right, then. And I think we've got one final feature and that is from you. >> Yes, AIB land. Two uh things. Uh first, the research agent is um is out. It's in beta. If you're familiar with kind of the research uh stuff in say I think I think like deep research was first the first one probably was OpenAI chatbt. Uh so it's where it goes off creates a whole plan takes its time to execute that whole plan. you come back in 20 minutes and after 20 minutes of work you get whatever whatever it is that the agent came up with. So that same kind of feature is now available within data bicks as the research agent. Um notably it like it doesn't have or it's not allowed to access the internet. So it's really just focused on only the assets that you exposed to it um and make available uh through Unity catalog. So research agent. So that's very useful. Um that's first one. And then the second one, this one's what I think is really cool. You can now attach a Genie space, or maybe that's not the right word. You can use a Genie space as a tool within Microsoft Copilot. So if you're in Copilot in Microsoft, you can now link a Genie space to that. So it has access to that Genie space as a tool. And if you kind of unbox that, the Genie space itself, right, is scoped to a specific set of tables, a specific set of instructions and metadata, um, things like that. So you can go and you configure your Genie space to expose whatever it is that you need to and then allow uh, other users to attach to it within Microsoft Studio, Microsoft Copilot Studio. So, if you have assets, assets within data bricks or something that's governed by Unity catalog that can help um an agent figure something out, then exposing it via this path would be the way that you could do it, >> right? >> Otherwise, you'd have to you'd have to stand that up somewhere else, worry about how that's governed, the permissions access, blah blah blah blah blah blah blah. This way, it's just make the genie space, expose it, and you're done. >> Right. Okay. Sorry. Sorry. In my head, it's like the chat thing in the corner is aware of like how many unique first names you have and like and I didn't understand how that was going to make you a better programmer, but that's not what it means. What it means is you could use this as part of like other applications that then don't use data bricks. >> Yes. Mhm. >> Right. Okay. Now I see how that's useful. Got it. Okay then. And I think I understand where our research agent is going to go. Research agent here, which I guess is part of a genie space. Uh and this is going to do deep research in terms of trend analysis. I think we're going to do like uh predicting uh next hot toy um toy trends. Uh what else are we going to do? I think we need to predict parts and like manufacturing processes and like maybe equipment that we might need for next year as well because maybe it includes a microchip that Santa is not available to make yet and so he needs a new um he needs a big printer for his microchips for next year for his GPU. So maybe that's what he needs. >> Even Santa can't get GPUs, man. >> Are we out of features? Is this a stun? >> Yes. Uh please let it be over. >> [laughter] >> No, this is fun, but this this I'm looking at the architecture and this is uh it's not even it's not an architecture. It's not over architecture. It's just >> box here. Yeah. Um >> it's funny cuz we did we did speak before and we thought we had something good and something happened in between and now oh this is awful. This is really bad. I mean, I think in terms of um creative uses of data, I kind of don't hate it. Um I do think the AI query in a Java file is kind of dumb. Um but >> you're you're never going to let it die. [laughter] >> Yeah, this is terrible. This is awful. I think I want to say like novel interesting use cases like I think for that it gets like a solid eight but in terms of execution this is like a four. Oh, we're doing we're doing like we're ranking specific aspects now. We don't have a >> score. I'm like no, for the architecture it's a four, but in terms of the ideas, >> it's a one. >> It's a [laughter] >> I think it's I think I mean we can go lower. There is a zero, but [laughter] doesn't matter. This is the worst so far. And I think we've given we've given we've given a lower score than four. That's why I'm that's why I'm calling you out on four. But um let's just bask in the richness of the features right we have Aback we have JDBC converting foreign and external tables using UC we have AI parse document we have NCP servers genie spaces co-pilot studio [laughter] >> it is a rich set of features I would I would grant you that >> basket features [laughter] >> oh they stuck together in a stupid way yeah I think if I paid someone to like go away and make a Santa architecture and they came back with this, I'd be very upset. >> Uh, no. Christmas spirit. Christmas spirit. [laughter] >> Uh, all right. Uh, I think that is it for this month. Do you agree with us? Do you not agree with us? Do you agree with Nick? Is it a one or a zero or is it even a four? Um, would you propose this to Santa? [laughter] I'm going to leave some links to documents uh all in the description. So, if you want to read more about this, then you absolutely can do. Next month, I say actually this month uh is a bit of a quiet month for releases because we uh take time off and releasing over Christmas is kind of a really dumb idea. So, actually the releases slowed down. So, we do have a very special episode planned for you for next month. So, come back and watch it then. Until then, have a wonderful holiday season and we'll see you next time. Bye bye bye. >> Happy [music] holidays. >> Wow. [music] Please give the happy holidays. That was really bad. >> Absolutely not. No.