Lec 18. Transfer Learning: Models

Key Takeaways

Let's get started. >> Um so today we're going to kick off the first of a little two lecture minieries on transfer learning. um specifically how we think about um taking information that might have been learned from a model and then transferring it to some new task, some new challenge, maybe some new distribution. We talked about distribution shifts last week. Um so at a high level, the idea of transfer learning is that you want to try to learn even if you don't have very much data. And so we've sort of tried to structure this framing of transfer learning into kind of these different sub areas. Um so first we're going to talk about how you can maybe transfer knowledge from a machine learning model um that you have about the mapping um from inputs to outputs. And so within that we're going to talk about fine-tuning. And then we'll also talk a little bit about domain adaptation. Um and then we'll talk about transferring knowledge about the outputs. And so in that state case we're going to be discussing knowledge distillation. Um and then we're going to be talking a little bit about foundation models and maybe what this new paradigm is in terms of uh transfer learning. Kind of introducing this idea of like in context learning or prompting as a mechanism for transfer learning by just asking. Um and then uh on next Tuesday we're going to kind of finish this up um with like one further discussion of how we might transfer knowledge about the inputs of the data. So why deep learning? This is a slide from Andrew Ang who of course I think gives a very has a very very famous lecture series on deep learning. Um and here he essentially is is pointing out that deep learning might be different from sort of older or standard learning algorithms. Um specifically in terms of how it scales with the amount of data that you have to train on. might be a little bit hard to see with the lights as they are. Um but essentially the idea is that um many learn learning algorithms older learning algorithms saturate um at some with some amount of data you just don't see performance continuing to improve. [snorts] But we have really um not seen that type of saturation as we scale the amount of data that we're giving these deep learning algorithms. Um and sort of next Thursday you guys will have a lecture from Phil around some of these scaling laws, these sort of relationships that people are starting to learn about the scale of the number of parameters, the amount of available data. Um but one kind of high level um recent um analysis that was done seem to suggest that with the current capacity of machine learning models we have um we we basically have far more capacity than data. So essentially the idea that um there's a lot more room to scale on the data dimensions still before we're really hitting the capacity of our largest deep learning models. Um and that's talking about billion data point data sets things that are like on the order of scale of the entire internet. Um we're still not hitting the the actual computational capacity of these models. Um so fshot learning is the idea that you can maybe learn even if you have very little data. Um so it turns out that humans are shockingly good at this. We're really good at kind of recognizing patterns and translating them even to things that we've never seen. So here um if I tell you that that symbol up at the top is a DAX, these are made up, right? This isn't real. But if I tell you that's a DAX, um, can you guys tell me which of the ones below is also a DAX? >> Yeah. >> Second row, second from the right. >> Second row, second from the right. Yeah. Right. It looks like a DAX. Um, so okay. What about this? Which on the bottom uh is an example of the same concept as the item in the box? Someone else. Yeah. >> Second row, first to the right. >> Yes. So those are both segways, right? But but then this kind of begs this like interesting question. Um because there's potentially ambiguity here. If you only have this single example and you're telling me which is the same, what you made was the the guess that was probably most similar, right? because the orientation of the wheels and the structure, but maybe the thing I was looking for was instead actually I wanted you to tell me all the things that had two wheels, right, instead of four or one. And so one of the things that's challenging when you're doing this sort of fshot learning type of um scenario is with less data it can be harder to understand without ambiguity what the boundaries of maybe the categories of interest are, right? Um, so if id showed you uh three examples, one was a Segway, one was a scooter, and one was a bicycle, and I said, "Okay, find me the ones that are that are also in this category." You would would have been much easier for you to sort of understand. Okay, she's looking for things with two wheels. Or, you know, at some different level of granularity, maybe I was trying to just learn wheeled things, in which case I might need to have again more diverse examples that cover sort of the space of the concept I want to learn. So, future shot learning really is just this idea that we can learn to maybe generalize um from our past lived experience um how to recognize similarity and patterns in a useful way. Um so up till this point a lot of what we've talked about um with deep learning is essentially you're you're going to have a data set and you start with an with an empty slate, right? And then you're essentially trying to learn sort of some physical rules, some concepts, some dimensions of similarity that are useful for a specific task. Um, but what humans do instead when they try to learn something new is we really have a lot of prior knowledge, right, coming from our experience in the world. Um, from the time that we're incredibly young, we're sort of physically embodied in the world, learning about concepts like physics, like structure. um we're learning through the entirety of our education and just in our experience in our lives like how to think about these measures of similarity and difference and different people can have different sort of mental models of the world. Um this actually comes up a lot um in currently when you're thinking about scaling largecale data labeling um that actually if you're getting data labeled from people from different cultures they might actually have different mental models than the one that you intend in your culture right so that there's this idea of like you know my mental model is based on some prior and we don't necessarily know or there really isn't a concept of which mental model of the world is correct um but these mental models are built and and They kind of capture sort of representations of things, kind of models we have of like how the world operates, maybe different skills. And that means that we're basing anything we learn on top of a bunch of prior knowledge, basically like systems that are really wellposed to learn new things. And actually, a lot of what you're doing in college is learning how to learn, right? Um you're you're learning how to learn new things efficiently. And so there's this question of like how can we give deep learning models or deep networks similar access to prior knowledge. Um and so a very very simple way to do that is just pre-train them on other tasks or on prior tasks. Right? So the idea that you can send your neural network to school and you can try to get it to learn how to learn. Um and the way that you might do that is maybe training on a huge amount of data. um with some other sort of set of categories that might not be the thing you're really explicitly interested in, but it's things that you have data for at scale. Um and then you can use some mechanism to transfer that knowledge from the pre-train nets so that you can solve new tasks. Um and so one view of this idea of like learning from little data or transfer learning is actually the point of deep learning is to enable us to be able to do this type of problem solving with little data. Um so you're trying to learn these representations. um these models that have a good sense of what's what's useful, what is a good thing to build on top of if we want to learn. So that's kind of the motivation for talking about um transfer learning, right? So um when we think about a model in general, usually you know we have some big system and it will map some set of inputs to some set of outputs um in some very you know fun complex nonlinear way. Um but we're thinking about um what actually are we transferring right? Um and so you can think about transferring some knowledge about the relationship for example um from the inputs to the outputs. You could also think about transferring knowledge in different ways, right? We can think about transferring knowledge about biases in a model. You can think about transferring knowledge about an optimizer perhaps. Um like how do we optimize models? Um, we can think about maybe transferring knowledge about potential error modes or ethical risks or societal implications of models. Um, so we we'll really be focusing on pretty simple mechanisms to think about knowledge transfer. But I do think that there's a lot of kind of more broad sets of transfer or sort of this idea of like sharing information um in a valuable way that are really not understood or not solved. Right? So, um, one thing that we still haven't really figured out what to do, how to do is building an agent that's able to actually learn some intrinsic motivation about how to explore in the world, which then could be transferred to a new agent, which would let them explore more efficiently. This idea of like transferring motivation within a robotic agent. So there's a lot of ways that you know we have these kind of brute force knowledge transfer mechanisms that we'll talk about but I think there's a lot of um space still to explore uh in the machine learning research arena just in terms of the best ways to try to share information and transfer knowledge. >> Yeah. How how much of that is like limited? Um like my my ba my my very basic understanding [clears throat] of some of the challenges in say like reinforcement learning is that it's more it's more of an issue of like sample efficiency and like thinking back to 2012 like one of the breakthroughs wasn't really that the algorithms changed but just that they GPU and that sped up the training process. [cough] How much do they like do people think that the bottleneck is the like algorithm or the ideas around how to do this versus just like we're not computationally efficient enough? >> Um so the question is basically like what's the bottleneck in terms of knowledge transfer? Is it fundamentally algorithmic or might it be related to instead like computational efficiency? And you brought up a really great example of essentially how figuring out how to use GPUs effectively was a huge breakthrough in terms of progress across many different areas of deep learning, reinforcement learning being one of them, but basically anything else as well. Um, I think it's a little bit of everything. And this question of the sample efficiency, like I I would not say that we have good algorithms for sampling data efficiently or for learning from that sampled data efficiently. like I mean we have algorithms that do it but I I feel like there's a lot of room to improve. Um cool. So we're first going to talk a little bit about this idea of transferring knowledge from the mapping. So essentially like transferring knowledge from the model you've learned itself how you go from inputs to useful outputs. Um, so you know, a simple example of this would be something like, okay, I have a lot of data on music with associated genres. And the idea being that being able to recognize a genre is potentially useful, right? It's teaching models something about the structure of music, something about what makes different songs similar. Two things being of the same genre kind of argues that they might be similar in some way. And then maybe what you want to then adapt or sort of transfer that knowledge to is something where you have a lot less data that's specifically predicting let's say like your individual preferences for songs right so maybe you know you can do genre recognition on the entire cadra of music on Spotify and then if you actually want to learn um how to predict what songs you're likely to enjoy that you're going to prefer you might only have you know a much smaller set of data which is like all the songs you've ever listened to, which is of course not even close to all the songs that exist. Um, and then the actual testing would then be taking this sort of fine-tuned model. You can see they have this like FP prime. Um, and using it to actually predict on new music whether you're going to like it or not. And then I don't know, see, do you like the music that it's suggesting for you? Um, and so this is kind of that broad simple thing for in terms of like what we call fine tuning. Um, and at a very high level, it's that you're pre-training your network, your sort of set of weights and biases on some task A. That's a pre-training task. And that will result in some parameters for your model, W and B, right? Weights and biases. Then you initialize a second network with some or all of W&B. And so there's like these kind of interesting question um of like how much knowledge about the mapping you want to transfer. Um, so usually like maybe the very final layer of a model that's going to be pretty specific to the pre-training task. So often it's it doesn't really make sense, especially if the the sort of output shape is not the same. Uh, you you cut that off. You know, you you add some different final layer on, but maybe you really only want those early layer features to be captured and so you actually let the model um sort of you initialize it with less of the initial initial model. Um, and then you think about training that second network on some task B. And so now you'll get some slightly adjusted parameters W prime and B prime. And the idea is like then then you have um this like learned representation that um is essentially an encoder and that's what we're transferring, right? So you're taking the representation learned from the initial model um that's encoding something useful about your data and then you're transferring that and then learning how to make use of that useful representation on something downstream. And we'll talk a little bit about um some of these different like empirical like choices you can make here around how much you let the model adapt, right? Like h how much you let it change from the initial representation. But empirically actually as long as like the types of input data are similar. So images and images for example what we often find is that a lot of the structure in the models like weights and biases actually stays the same particularly in those early layers after you've done the fine-tuning. So even if you haven't explicitly constrained the model not to change that very much um which you can do like keep a learning rate low or freeze some of the network. But if you don't do that, you actually find that a lot of the structure will stay the same, which does seem to argue that that some of that is just universal in some sense. It's universally useful um for let's say images or audio. So we previously saw a very specific case um when we were talking about similarity based representation learning um where we talked about self-supervised pre-training, right? Um, so here with this self-supervised pre-training, we're not necessarily explicitly or directly training the model to do something we might care about. Instead, we're just training the model to learn something about data similarity. Um, and then that ends up and has been shown to be quite useful uh for training representations that are good for downstream tasks we might care about. And one of the main benefits of this type of self-supervised pre-training is the fact that because it doesn't require labels for data um it's making that sort of pre-training on a lot of data like a lot a lot a lot of data right the amount of data that exists without any specific label is much larger than any sort of labeled or clean data set. Um so here you know you think about contrasted pre-training and then maybe you're going to test it on some sort of new recognition task. Um, but instead you could think about fine-tuning a model, which is how you take those initial representations and transfer them to then actually learn how to do that new recognition task. Okay, so now I'm going to get into a couple like just pretty simple but pretty practical considerations. So, what if you're pre-training on some images and you're predicting whether they're wheat or corn and then now we want to fine-tune our model. Um, but now we have an x prime and a yp prime that are not the same. So, essentially we have a model and our initial model was trained on images that are RGB and it was uh trained to predict two categories as the output. And now the model we actually want is going to actually take in images that are just grayscale. So we only have a single dimension and we want to predict on outputs that have an additional category. Um so like the structure of the model itself is actually a bit different. Um essentially if you're changing either your input dimension or your out output dimension or both. Um so one kind of very simple mechanism that you can do um is you can add kind of like a glue into the model um which is just like a way to kind of transfer data through to get it into the right format. So then when you're doing your pre-training, if you know that your downstream task is not going to have necessarily the same structure, um you just you define these new sort of like Z's, which are maybe models or even just like a few simple layers, fully connected layers, what have you. Um that enable us to then map each input and output to the same dimensionality. Um and so you would do this where you need it, right? Like if you if you still going to have RGB images, but then all you're doing is adding one more output dimension. Well, then like you know, one thing you can do is you can take the final output layer of your initial model and you can learn some reprojection to project it into more dimensions. Or you can do what I talked about sooner and you could kind of consider this Z to be like that final mapping from your representation, that encoder to the number of categories you want to predict. and then you can just uh swap that out and maybe learn some different um different one that's that's now in this new model. So you're just essentially defi defining points within the structure of your model that give you a bit of flexibility. So now like this f1 goes to f1 prime those will be different from each other. F3 goes to f3 prime those will be different from each other in terms of their structure like the actual number of layers um weights sort of size but then the f2 can stay the same. So like the vast majority of the structure you're not actually changing. Um and this actually comes up quite a lot uh particularly for like some types of data that have maybe more input dimensions. So most image models are pre-trained on things like imageet or you know these days lion where you have huge amounts of RGB data but then say you're someone who's working in robotics and you have RGBD right you have this additional depth dimension. Um, it's often true that you can still get a lot of value in doing transfer learning from models trained without depth, but now you have to kind of figure out how to add this additional depth dimension during your fine-tuning effectively. Um, and so often the way people will do that is they just learn some sort of additional encoding that takes that and sort of maps it into three dimensions. or sometimes depending on how much data you have whether you can really learn that type of mapping you do some sort of like strange just like very simple model of how to transform one input dimension into another. So if you're working with like very high scale hyperspectral satellite data and you have models that are pre-trained with three channel inputs, sometimes people will just run something like PCA and take the like principal components um of like the the three most informative principal components over all those hyperspectral bands and use that as their three channel input when they're fine-tuning. um because the ability to actually learn all of those 380 different input layers um effectively is one computationally very expensive and two if you don't have a lot of data that's a lot of parameters to learn and so that can be tricky. >> Yeah. >> So we're talking about sort of adding or changing the stacks and the dimensions. Do you ever add stuff in the width dimension for fine tuning or is that just >> um in the width? So there's like the input size would change. >> Uh so basically like say if you had F2 >> Uhhuh. >> sort of adding additional channels in parallel to F2 >> like concatenating >> kind of. >> Yeah. I mean so so definitely like uh a lot of these models will assume some sort of fixed input size and then say maybe you want the to train the model on a bigger input size. That's another way that you can think about building like this little F1 that will take a larger input size image and then like learn how to get it to that smaller input size that the pre-trained model was trained on. Um like but I think I think the only way to think about changing the width dimension is in terms of like the spatial extent right >> so just in terms of input not in terms of like the throughput of the models. >> Oh you mean like actually changing the structure of the model itself? I guess like stacking on top of like a new frame. >> Yeah. I mean, so one of the things we're trying to get across here is that you can kind of treat these things as like modular building blocks, right? And so you can take like a building block that someone else has built and then stick it in and then come you have to figure out what blocks you need to add to the system to maybe get that building block to be useful for you. But the point being that a really well-trained building block, if that analogy still is making sense, um a really well-trained building block is still useful and it's maybe worth coming up with this like modular structure that will make that work for your problem. Um but like as soon as you start actually trying to mess with that building block like changing the weights and the biases within that block, it gets more complicated. There are people who look at doing it and actually the sort of entire area of like model compression is kind of changing that structure by removing stuff to make it more efficient. And we'll talk a little bit about that later. In terms of adding more stuff, I don't know. I haven't seen a lot of work in that space, but I think as soon as you're adding more stuff, the value of the pre-training starts to go away, if that makes sense, because then like you change the structure of the mathematical formulas. Cool. So, essentially, we're just trying to figure out how to make things useful, right? And so in practing in practice, we'll essentially just learn mappings for any new stuff we add to make this work for our data. Um, and so those new things would need to be then initialized from scratch. But that's a lot less parameters that need to be sort of learned from scratch than if you're sort of going endtoend from scratch, starting from that blank slate. Um, and if you really don't have much data, you can also think about copying weights for that initialization. So for example in that RGBD case um sometimes what people will do is they will take the RGB channels that were learned from the original model and then they'll staple on an additional channel and they'll initialize it with for example like just like the blue channel parameters with the idea that like maybe there's some value in what's been learned for another channel that might still be transferable. again like this is all kind of like no one really know like no one no one has like a perfect um sort of set of heruristics for like if this is your data this is the best thing to do and this is really where the idea of like designing a benchmark for your data set where you understand how to do comparison across like some of these different choices so you can try things and see how they're affecting performance of your model. That's where that starts to become really important because most of the time like if it's some custom problem some little dimension of the system will probably be somewhat customizable and it's hard to say um what's like the best structure for a system um without actually doing some testing. So one of the issues is that if you only have a little bit of data and you let the entire model train and you train it until that loss is all the way saturated, right? Like we really have like we don't have anything else we can learn. We run into that catastrophic forgetting problem again. That thing we talked about um a bit when we're talking about sequences where it's like somehow the thing we learned from the original data we learned too long ago. the model's weights get changed too much and you you essentially lose capacity. Um so um this definitely happens more with smaller models than big models interestingly. Um but in order to avoid kind of overfitting on a really small amount of data for your specific task um there are a few different mechanisms that people use to try to kind of strike that right balance, right? learning enough about the new task that you can specialize while not forgetting too much about this general useful representation. Um, and so a couple things uh you can think about freezing most of the network. So basically saying I just want to [snorts] keep most of this model exactly the same and only training for example the final layers or so those initial things at the beginning that you need to have that are new. Um you can explicitly use really small learning rates. So basically just not letting the model move too much in any direction at any training step. You can use what's called early stopping, which is essentially you use a validation set and you make a decision about which model weights to keep based on the best performance on your validation set with the idea that um essentially you just haven't let the model train too much. So you sort of stopped it before it can forget too much about the original pre-tax task. Or another mechanism and one that tends to work pretty well though of course there's some complications in terms of data balance that you have to strike. sort of sampling effectively is to just continue to train on the original data as well. Um, so essentially if you have access to the original sort of pre-text training data, then as you're fine-tuning, sort of continuing to add some of that pre-text or some of that original training data is good. But of course, there's a complexity here which is like, all right, but if you have different input types and different goals, this becomes harder and harder to do, right? If you were doing like contrastive pre-training and now you're trying to do classification on top of those representations, still continuing to have the contrastive pre-training is a little bit more complicated. Um, but a lot of these choices like these types of things assume that you have some sort of good representative validation set that your benchmark is well posed for your task. This might not always be possible where you might see distribution shift sort of going forward. Um, and so it really does become quite important to like think about measuring relative performance as close as possible to like your representative problem, right? if you want to actually use the model on some data that defining measuring progress is just difficult and complicated and important. And this is now I'm going to be on a soap box a little bit, but one of the things that drives me absolutely crazy about modern machine learning is that we're obsessed with the idea of bolded numbers and tables. And so this kind of begs the question that like you can represent everything about what a model does with one single number that sort of captures every dimension of importance to that model. And I don't think that's true. I think we need nuance. And honestly, usually like people will there this is bad behavior. People will pick the metric that their model does best on, right? They're like, "Ah, this makes us look the best, so we're just going to like show that one in our in our paper and we're not going to talk about the fact that for some of these other metrics we don't do as well." Um, I really love it in a machine learning paper when people are honest about their flaws, [laughter] right? Where you're like, "Hey, all right, we see some really great progress in these sections." And then you have a really nice limitation section that's like like realistic and captures the dimensions of where there still is space to make progress. Um, maybe it's just personal preference. I'm a little bit allergic to salesmanship. So, I don't like it when I read a paper and I'm like, I don't know, man. I think you're sweeping some stuff under the rug. Essentially, you do not want me to be your reviewer. Number two. Um, yeah. So, okay. So, we'd like to avoid forgetting the general representation. And here's some like engineering hacks that you can take if you want to try and do that. Um, but then there's also this other kind of area of research. Um, that's specifically it's called domain adaptation. And the idea here is that maybe you have some source domain and you have a target domain and then you want to learn to do well on your target domain, but you also want to essentially make sure that the representation spaces of the source and target domain are well aligned. Um, and this shows up a lot when you have sort of these types of distribution shift problems that we talked about before. Um, but it also can be used for sort of different types of fine-tuning. Essentially, one of the motivations here is that um maybe if you sort of force these feature spaces to be well aligned generally um that you can learn usefully how to um map uh you know learn that mapping with much less data, right? And and one way that people do this is they actually they they learn how to get the model from the the source domain to the outputs and then they try to learn a projection or a mapping from the target domain to the source domain. So you basically are learning how to map into the domain space where you had a lot of data and then you can use the same model um to actually do your prediction. Um and there's lots of different ways to try to force these representation spaces to be aligned. Um but one kind of large um set of them takes an adversarial approach. So here you have both source and target data during training. And while you're training for every input, you want to predict the category you care about. And you also want to predict the domain that it came from. Something about the distribution or sort of sort. So a a classic example of a data set in this sort of domain adaptation space is um there's data set of different types of um different types of like technological hardware, but one data set is from Amazon and one is from eBay, right? So you can think that like okay the way that people take pictures for Amazon are very like nice and clean and then the way people are taking pictures for eBay is maybe like on their messy desk. Um and so there what you're trying to do is you're trying to predict like what kind of computer is it and is it from Amazon or eBay and then this is where it gets adversarial. What they do is they flip the gradient from the domain predictor. So you're essentially trying to train the model to be as good as possible at predicting the category while being as bad as possible at predicting the domain which tries to force essentially the representations across the two domains to align. Um and then the interesting thing is you can kind of extend this to where you might not actually have category data for the source and the target, right? Um, so, so maybe you had a bunch of category data for the source and now all you have for the target domain is just the knowledge of what domain it came from. You can think about trying to still keep really good at predicting the category, but be really bad at predicting which domain it's from. And the hope is that there um, even if you only have a very limited amount of labeled data in your target or even sometimes none, you might still be able to learn something useful that helps you find the right categories in that target domain. Um and then this has kind of expanded into this whole area of unsupervised domain adaptation where here the idea is you might have data but no labels from the target domain. Right? So now it's like you want to make a model that's as good as possible on some set of data but you don't necessarily have any training labels. Now why would we have that? So, so here's an example actually from our own research um where we have a bunch of labeled source data and in this case the labeled source data is sonar images or sonar video from rivers with fish and we're trying to count all the fish that are swimming upstream so we can make good sustainable fishing policies and we have some rivers where we have a bunch of labeled data but we want this to work on sort of any river or if something changes about the river they move the sensor we want it to still work even though there's that distribution shift. And so here what we're trying to do is learn from labeled source data and unlabeled target data how to get a model to work better on the target. So how have people done this in the past? Well, so the very very sort of simple version of this would be you take your target data set, you run it through a model that's been pre-trained on source and then you maybe just calibrate it. You pick some threshold where you say, "Okay, any prediction over this we're going to treat as ground truth." Okay, so that's one way to transfer knowledge, right? You just use a model from a different domain and hope it works. Turns out it doesn't usually work very well. So this kind of motivated this idea and this this is a mechanism to do sort of distillation which we'll talk about a little bit more um later. But what if instead we we make a new model that we're going to call a student model and we're going to use those predictions on the target data as weak supervision. So we'll use the predictions that our model trained on the source data gives and then we'll try to train a model to predict the same thing. Um and the idea here is that maybe your student model is able to kind of learn something useful that is better than just taking those raw predictions. And this does work. you get small gains sometimes over over just the original source predictions. But this threshold is really really finicky and hard to calibrate, right? So people started adding in other bells and whistles. One thing is you use EMA, an exponential moving average to try to kind of regularize the way that the weights change within the student and teacher model. So you let the teacher model itself also update sort of slower. Um, and then finally, you can also introduce additional training on the source data. So there maybe you're letting your student model learn from the source data and from the target data. And then you can add in a bunch of fancy stuff like um instead of explicitly taking the predictions with a threshold, you can use a softmax. You can use a distillation loss so that you're learning to instead of match the prediction, match the distribution over the outputs. And you can try to use some explicit feature alignment between source and target. So that's like basically trying to make sure that the features are overlapping as much as possible. Lots of fancy tricks here. Kind of complicated. Um we essentially wanted to point out that these components are essentially the main components that people use to do this type of distribution. Um unsupervised domain adaptation. Um, and the point here is that basically every modern domain adaptation architecture is some combination of these components. And so if you can kind of centralize and kind of unify the ways that people are actually doing these experiments, what we found is that you can often be more optimal than any um previous approach. And so these these methods um many of them can be quite effective in certain scenarios. Of course, certain scenarios is a big interesting thing there. They're almost never as effective as having labeled data for your source domain, right? So, you just have labels, that's better, but if you don't have labels, you can actually start to get pretty close to performance if you do have labeled data. um if you do these if you sort of build these things carefully and some intuition as to like why that might might be is that um essentially the model's able to learn something about the statistics of the target domain um that's useful right you're shifting the weights towards um being representative on those new statistics yeah >> um how can you measure how like the performance of this when deployed and you like if you deploy it in a field that you actually don't have the labels of anything. How do you know your model? Well, >> yeah, this is a really great point. I think it's actually the elephant in the room for most of the the domain adaptation literature is that they almost always are demonstrating performance in their B bold number in their table on labeled validation data for the target domain. But this kind of begs the question, if you have to label data to like do model selection and determine what mechanism works best, why don't you just use that labeled data for training, right? Um, and I think uh, yeah, I mean I I I think it's kind of like one of these interesting questions and also if you did have some training data, like how does that kind of fit into this system? Um, but yeah, I I do think that in the literature in this space, um, one of the big sort of missing pieces is that you still aren't very good at doing this if you don't have anything to kind of select the best model or calibrate it in any way at all. >> Cool. Yeah. But I guess following up on that, I mean, isn't the point to reduce costs and times like you can use bigger data sets? >> Um, so reducing costs and time are is is valuable here. So, so maybe like one dimension is you're trying to reduce the amount of like human input you need to get a model to work well. But I think probably a better motivated scenario for a lot of these is um places where we do not have the right bandwidths to be able to actually get the data to where a human could look at it easily. So we use these domain adaptation models on these rivers that are like in the remote Arctic where like the way for a human to get there is like fly into anchorage, take a sea plane, like you know hike like it's just like non-trivial to actually get to the sensor and we don't have the kind of satellite connectivity or power that we would need to like get the data to the cloud. So if you have like these strong resource constraints on your system where you can't necessarily just have a human easily label data I think that's where these types of models tend to make more sense. Similarly like uh if you're I don't know doing like navigation on the moon or deep sea um sometimes we just need models that hopefully can adapt to a distribution. But the problem is if you don't have any kind of quality control going in, anything like this that's kind of learning from data where we don't have any sort of true labels has the potential from like a reinforcement learning perspective to kind of like veer off in some catastrophic way. And so you can fall into these kind of cycles where the model kind of gets worse and then worse and then worse and then worse. Um and then the other question is kind of like how much source data do you still have to have access to? So I think I think um essentially there's a lot of engineering components to like why you would want to do this and where it's actually effective and what the value ad is. No. Um yeah, but I think u there's also kind of an interesting dimension here which is like these types of domain adaptation approaches also really benefit from the right starting point. So you can almost think of this as like a different mechanism for fine-tuning, but the performance of these is very very heavily influenced by the performance of the starting point model. So it's almost complimentary in terms of like trying to build the most generalizable model, the sort of most uh useful general representation. And then this is maybe a mechanism to fine-tune if you have a lot of data that doesn't have labels um instead of you know maybe a little bit of data that does have labels. Cool. So next I'm going to talk a little bit about how we might try to transfer information or knowledge about the outputs of a model. Um so if you recall before um we talked about you know how you train a classifier with something like cross entropy. So here you know we have some model um we have an input X we have some output Y which is the ground truth for that model or Y is the output prediction for that model. So maybe there's a bunch of different categories and the one that has sort of the largest prediction um score or sort of lowest score is cat. Um it maybe is taking a soft max actually because it's going from zero to one. So this is maybe our prediction score. And then the ground truth is going to be some sort of one hot encoded label right where we say like all right the true label for this input x is cat. And then we use cross entropy loss that basically tries to get our model to predict a distribution that's as close as possible to this ground truth distribution which is just one for cat and zero for everything else. Um so you could argue that this is also kind of a teacher and a student. Um, so here the teacher is the difference between the ground truth and our prediction and the student is kind of the way that we're capturing that distance. And so the idea of knowledge distillation is essentially now we have some big pre-trained model. We're going to assume that that model is fixed, right? So we have this little lock on here. So it's a locked model. Um and then we want to learn some student um where so now we go from X to Y teacher. We have a model that goes from X to Y student. And then the way that we try to get this model to kind of distill is we want the difference or the distance between the teacher model's predictions and the students model's predictions to be as close as possible. So now maybe your teacher model has this distribution and maybe it's predicting that cat is quite high but it's also predicting that tiger is quite high. Right? So now when we're actually training our student model we want it to similarly match that distribution. Now does anyone have any intuition um as to why this might be useful? Like why it might be useful to to train to a model to match sort of a distribution of predictions as opposed to just a ground truth. Yeah, >> maybe the new model is smaller and quicker. >> Yeah. So maybe it's a way to get a new model that's smaller, could be more computationally efficient, and can kind of learn the same thing. Um, anyone have any intuition about why actually like it might be valuable to match the distribution of the outputs. Um, maybe someone new. Yeah. >> Um, if there's a cat and a tiger, I guess in this example, are kind of similar, right? >> Yeah, exactly. animals maybe dogs more similar monkeys assign a little bit right like it'll give it some more information >> yeah so that's exactly the answer I was looking for it's giving the model some information about the similarity so here like cats and tigers are maybe similar and but it's not just the similarity of the category set it's the similarity of what's recognizable in a given input right so if you have an image of a cat, but it's an orange cat and it's in tall grass. You could see how like a model would say, "Hey, that's pretty similar to a lot of the data I've seen of tigers, right?" And so now what you're teaching the model is not just this is a cat. You're teaching it this is a cat that looks a little bit like a tiger. And that might actually be useful context for a model in terms of learning more than just sort of a set of separable categories. Um and so uh there's another kind of interesting point here that I think is important to raise that's not just about efficiency um but is about sort of the modern era of many closed source models. So if you have access to the outputs of a model, its predictive scores, but not the model itself, you could arguably learn a model that can match the predictive scores of a given output. And this is maybe one of the reasons that a lot of those closed source models don't actually give you output predictions um over the sort of whole category set. Um cool. So now uh you can essentially think about doing standard cross entropy but instead of the cross cross entropy being to that binary category it's just between the teacher and the student outputs. Um and so exactly like what you said before right you guys are great here. Uh the label you get for every input might be more informative than just the ground truth cast. And so this soft target is telling you something about the similarity and what things look like. Um, so this is a fun example. It's not just saying this is a dog, but it's saying this is a dog that looks a little bit like a cat and it looks very different from a bear. And so maybe that's telling you, oh, it's like a small dog, right? Like it's like, I don't know, one of those little yappy dogs. Um so this original sort of introductory paper which is at one of Jeff Hinton's um shows that you might actually get lower training accuracy when training this way but you end up getting higher test accuracy on this um speech recognition task. Um and so the other interesting things if these soft targets have really high entropy um they're giving a lot more information per training case than hard targets and they have a lot less variance in the gradient between training cases is another thing that they sort of showed in this paper. And so a small model can often be trained on a lot less data than the original model while using a much higher learning rate um when you're doing this type of distillation. But it's not just something you can use to sort of make a smaller model. You can also think about learning um from a distillation perspective across modalities. Um so here uh maybe you have a teacher that learns to predict categories from an RGB image and then you have a student that only has access to depth. So this might be useful if you have labeled data for RGB but you don't necessarily have labeled data for depth. Um and so sort of another example of this crossmodal distillation is this model um called Soundnet where they have um waveforms and unlabeled video and then they essentially are doing object distributions and scene distributions and then they're learning distillation across both. Um there's another interesting version of distillation from a computational complexity perspective where you can actually distill the results of an ensemble. Right? So often we find that even even in the modern era ensembles over lots of models are often more predictive um than a any single model. Um, one of the arguments for this is kind of a classical one, which is that um, every model is going to hopefully make the same correct answers more often than they make the same mistakes, right? Um, and so here what you can think about is like maybe you have an ensemble that's really great, an ensemble of, you know, three different models, but they're all kind of expensive to run and and you don't really want to be running three models all the time. So you can think about explicitly um taking that ensemble of model outputs and then trying to distill a model that matches the distribution of your ensemble. And again, each model is imperfect in different random ways. Errors cancel each other out and the truth is shared. Of course, this isn't always true, right? Because sometimes we have biases in models that are learned from our data that will be consistent for every model. So the thing you're not going to get out of this is something that's really good at, you know, the types of corner cases that every single model in your ensemble gets incorrect. Um, but it can be much more efficient. Um, cool. So what if you don't just want to distill the outputs, but you also want to capture and distill something about the knowledge of the representation itself, right? So you might want to um for example match the embeddings at some intermediate point within your model using contrastive training. Um so as opposed to just sort of like matching the logits or the output scores, you can actually think about distillation um specifically using contrastive learning but where you want to actually supervise embeddings internal to the model um to be kind of invariant to some viewing transformation for example. Um so that like viewing transformation we talked about like rotation invariance etc before but another version of viewing transformation could be the the different views of that representation from the teacher which is a big network versus the student which is a small network. Yeah. >> So in practice when you see this and what you were talking about >> how much smaller than the teacher is the student mode