AugSBERT: Domain Transfer for Sentence Transformers

Key Takeaways

today we're going to talk about how we can train language models across different domains now to do this we're going to be using what is called the augmented expert training strategy and we're going to be using the domain transfer flavor of that so the training strategy is used where you have some data but not enough or you in our case with domain transfer we have some data in one domain so say maybe we have quora question pairs but we need sac overflow question pairs and all we have on the stack overflow side are unlabeled question pairs so what we're essentially doing is using that source data set the for example quarry question pairs to train a model that can then transfer its knowledge and label the stuck overflow question pairs now we don't need to stick with question pairs and question pairs we can have and go from question pairs to semantic similarity pairs or anything else but we do have to make sure that our source domain and target domain are at least similar and we can think of this required similarity as a bridge across a gap between the source main and target main the greater that gap so the the less overlap those two domains have the greater this gap here is and if this gap is bigger we obviously need a bigger stronger bridge and we can think of that bridge as our model we need a bigger stronger model to bridge these longer or greater gaps between the do know two domains so in this video we're going to be going through one example but i have tested these five different data sets so what we can see is the engram similarity so this is how many engrams overlap between the two dates that's as a whole so this is not a perfect measure of similarity of course because it's only capturing the syntax overlap and it's not really capturing whether the two datasets have the same language structure or talking about the same topics that much although it does help and this is just a very simple measure to help us figure out whether [Music] the two domains that we're trying to transfer across will be easy to bridge or not so we would expect from this that the medical question pairs down here because of this these lower similarity scores it would probably not be the best domain transfer to these different domains whereas these different domains scsb rte mrpc and quara question pairs are all reasonably similar so we'd expect the performance or the transfer or ease of transfer between those domains to be much smoother but like i said this isn't perfect and in fact one of the best domain transfer performances that i saw was from the medical question pairs to the core question pairs which obviously makes sense semantically because question pairs will probably have more of a similar structure than entailment pairs which is what we have up here of rt or just semantic textual similarity pairs as we have up here so that does make sense logically but it's not really reflected in this chart here so this is the overall strategy that we're going to follow with the main transfer augmented experts so the first step is to get our data so we're going to have a labeled source data set which in the earlier example would have been the core equation pairs and we're also going to have the unlabeled target data set which would be our stack overflow question pairs we're not actually going to use those two examples we're going to use from here we're going to go with mr pc as our source data set and stsb as our target data set so we'll go ahead and we'll get our labeled source data so that's going to be mrpc and we'll train a cross-encoder model with it now the reason i'll just quickly mention this the reason we're training a cross-encoder model rather than a buying coder model which is a sentence transformer is because we need less data to train a good crossing coder and the cross encoder here is only used to label the target data so we need less data to train a good cross encoder but cross encoders are also slow which is why we don't use a cross encoder over here as our final model so we're basically using that greater performance with less data to build a data set over here that can train a fast sentence transformer over here to a similar level or similar performance to that cross encoder that's basically what we're doing is it's knowledge distillation from our cross encoder all the way over to our sentence transformer that's all we're doing so let's jump over to the code and we'll have a look at how we do those or perform those first two steps okay so we're starting with the source domain mrpc as i said and the target domain is scsb so first thing we do is get that source data so we use hooking face data sets for this so you may need to pick and solve that so just pip install data sets and from data sets we import load data so this is just going to load the mrpc data set from hogging face which is stored in the larger glue data set and mrpc is like a subset of that and then we're taking the train split from there and we can see the data set features here so we have sentence one and two which are sentence pairs that we're going to be learning from to in order to label the other scsb sentence pairs and we also have the label as well and we only have 3.6 000 samples here which is quite a small number particularly for a sentence transformer so that's why we're going with the cross encoder here so that we can build something that has good performance whereas a sentence transformer might need a little more data although not necessarily um so we need to first format that data for training and to do that we we just go through each row in the data set that we just downloaded and sentence transformers always uses this input example object so ordering here is creating a an empty list and then within that empty list we're populating it with these input examples which contain sentence pairs here and also a label and then behind the scenes we're using pi torch so we also need to initialize a pi torch data loader with those input examples and we're using batch size 16 and also shuffling that data as well so that's how our data's ready for training the cross encoder and then we move on to if we'd like to we can move on to adding this validation set as well which is is useful because then we get a performance from our source validation set if possible you really probably want to switch this out for the target data set validation set so this would be in our case it would be stsb i'm not going to do that here just to emulate in in like an actual project you may not have this validation data from your target domain but i would i would definitely recommend if you do have the time to just manually label something so that you can uh check that performance you want you will want to do that later anyway but for now we'll just use source data we're using this cross encoder because we're training a cross encoder and it's coming from this cross encoder section of sentence transformers library and it's a correlation evaluator okay so that's going to check the correlation between the labels that we have here so the true labels in our validation set and the predicted labels from the cross encoder because the cross encoder does not output sentence vectors like a sentence transformer it outputs a similarity score and then we just initialize that correlation value we're using from input examples so we're using input examples up here so we use that method there and yeah that's our training and validation data i'm ready to go so we can go ahead and start training the cross encoder so for training cross encoder we load this cross-encoder object from sentence transformers again just using a base case so that's the sort of initial model that we're using um to initialize our cross encoder from a search coming from the hooking face models hub um and the number of labels so this you can change this if you are training a cross encoder that you would like to predict nli labels for example so you'd have zero one two which would be your contradiction uh neutral and entailment classes you would put something like three because you you have three labels and you want to output three labels uh but for us we want to output a similarity score so we just have one label then we have here we have number of epochs now anything more than one you're probably going to overfit so i would i think always stick with one um unless you really have reason to do otherwise and here we have the percentage of warm-up steps that we're going to use so this is the number of sets where we're not going to be training at the full learning rate but we're going to be gradually increasing up to that full learning rate so it's quite high 35 percent but it's what tends to work best i've found find cross encoders a higher percentage of warm-up steps tends to work reasonably well and then over here we have our optimizer parameters which is another training parameter that i i've modified a little bit so here we're using slightly high then the default of two e to minus five we're using five e to the minus five now everything else is as you usually would with a cross encoder so we're use using the fit method that's we'll do that with a sentence transformer as well the train data loader this is slightly different to the sentence transformer um parameter but otherwise everything is exactly the same way so passing in our source data which is the training data loader that we created earlier we have our evaluator number of epochs warm-up steps learning rate and where we are going to save the model so but a source is going to be mrpc crossing code um so that trains our model it doesn't take long you can see here it took on this computer 14 seconds it's very quick and then we can go ahead and label the target data so that's on to the next step so let me actually go back to here so we've just we can kind of cross these off so we've labeled source data and we'll train our cross encoder okay so this is our performance of the cross encoder models so that is the the source models over here on different target data sets over here so the source model for mrpc on the stsb target data over here so this is a the validation set of the scsp target data is what we are aiming for so it's 0.63 now we should be able to get better performance in this i think but we'll have a look and we can also see i i said the sjsp is quite easy as a data set to get good results on and we can see that here because all of these uh do achieve better performance but what we or what i really want to focus on here is the the bert based on case performance as a benchmark because this is what you would get if you didn't have your any any training data or which we don't we don't have any target domain training data instead we're we're kind of using this org expert training strategy to train from the source data set so these are our benchmarks and we can see that the from from here so we have these red circles which represent that the um the model performance has decreased from that benchmark and then the blue ones over here they represent that the model performance has increased from the benchmark and then we also have gray which is to indicate that it was the same value or maybe a couple of points higher but not that much higher so we can see straight away that the worse and we can see that it does perform worse so all of these uh glue data sets don't really correlate so much with the medical question pairs and we can see a lot more red in this column so that is to be expected in the other glue data sets we see a lot more gray and blue so the results are either similar or better so obviously next bit is this target domain part we're not going to be doing this optional augment data because it doesn't tend to at least for these data sets i didn't find this to help much uh but if you if you do want to see how to do that uh the the last video that we did on augmented expert last article does cover that um augmentation of data using something called random sampling so we're just taking random samples from our data set and creating new pairs with that and then labeling them without cross encoder that's all all there is to it so but for this i didn't find any significant performance increase it did increase the performance for a couple of the models a little bit but not not a huge amount so i i think for the time it takes it's probably not worth covering in some cases but if you think it might help especially if you have a small data set i would check out the other video and an article so we need to get out and label debt target data and then we're going to skip that step so we'll just cross it out and we are going to label it with our cross encoder so from up here we're going to take that and label this this target data so let's have a look at how we do that so yeah the labeling that target data first we need target data it's same again we're just going to download that from the face datasets library so we're coming from the glue like larger data set again um but this time we're using stsb which is quite an easy data set to get good results on to be honest so i wouldn't expect this sort of performance for everything um but this is just the example what we're going to use here and so we're obviously using the training display as well and we can see here okay in that day it's like we have the features sentence one sentence two and we also have the label uh so obviously we don't actually need to use the cross encoder to create our labels uh but we are going to because that's the whole point of this training strategy and i just couldn't find a reasonable reasonably good data set that didn't have the labels already so we're just pretending they don't exist so you didn't this here it's not there um if you if you're struggling to train across encoder or you can just download it from here by the way um or if it's taking a long time i mean it shouldn't take that long but i know on on some machines it it can take a while so just if you're following along with this in real time just i'll do that if you're having any issues so we have our our target data and what we do is we zip those sentence one and sentence two together so i'll delete that let's say we have sentence one and we have the first i don't know what it could be it could be i think it's something like a plane a plane is taking off and we would have basically a tuple where we have the first sentence and we also have the second related sentence which is something like an airplane lands i i don't think it's lands it is something to taking off as well but i don't remember what exactly it is so you would get something like this and then the next one would be something else like um the dog runs and i don't know the the dog warps okay so something like that so we're getting these like um two pools of sentence pairs and we're feeding them into the predict method of our cross encoder and that will return a set of scores obviously these here would be a reasonably high similarity because they're they're talking about similar topic not necessarily the same got warps and runs and lands and taking off but they they would be so reasonably high and then what i'm going to do so this just makes things easier in terms of seeing what we actually actually working with what we're looking at so i'm just going to pass those sentence ones under sentence twos and then there's a new scores into a frame and then we can see everything here so we can see a plane is taking off and an airplane is taking off so that's that was the the first two i was talking about and then we have all the labels that our cross encoders predicted now all these are pretty high because these are all very similar semantically similar pairs so that's why that's what we would expect later on there are other pairs which are less similar and then we move on to training the buying coder so let's quickly switch back to our visual again okay so now we have identified or we've downloaded that one labeled target data crossed that off and we've also labeled it using our cross encoder model so the final step as we saw is training our buy encoder with that labeled target data so let's have a look at how we would do that so we have our labeled target data in this panel data frame here so we're just going to iterate through the rows in our data frame and i'm going to append all those as input examples as we did before to this data list here okay and we have our sentence pairs and we have the predicted labels in there so once we've created those that list of input examples again we're just pushing all of that to a pi touch data loader and from there we can go on and initialize how our sentence transforms so to do that we are using models and sentence transformers from the percentage transformers library and we so the sentence transformer takes the typical transformer like bert and what that outputs is 512 word vectors and what we want is one single sentence vector for from our sentence transformer so to do that we need a way of translating those 512 word vectors into a single vector and we do that by pulling all of those 512 tokens into one token by taking the average value across each dimension and that that's all we that's all we do there so that's why we have this bird layer followed by a pooling layer i wish it's taking the mean pooling and then we combine both of those into a single sentence transformer object and that's how that's our sentence transformer initialize but obviously we need to train it so we come down here to train on the data we have at the moment we have continuous values in our labels from zero to one so what we can use is the cosine similarity loss so we initialize that and then again we don't want to over fit so epox is set to one and at this time we're using a lower number of warm-up steps which is it's 10 this time and we train with that so once that is done uh we are completely ready to train and we just call model fit so train objectives is slightly different sort of cross encoder fit method but everything else is the same we're just using the default value here so two e to minus five so you don't really need you can remove that if you want but if you do want to change that i put that in so you can see where you would change that and yeah we're ready to train we go ahead and train and see how that performs um so we evaluate it this time we're not using cross encoder so the the evaluator is slightly different this time using a embedding similarity evaluator which is going to take two sentence vectors and it's going to calculate the similarity between them and then it's going to compare that to the that predicted similarity to the true similarity as per our validation set here so in that validation set we we do have these labels but in stsp those labels are from the or in the range zero to five so we use this lambda function to divide everything by five which brings us into a range of zero to one which is what we need for this embedding similarity evaluator and then as we did before we're creating that list of input examples and then we can initialize the evaluator so we're using the similarity evaluator from input examples again because we have input examples and we're passing in our input examples data and right csv is false that just means i will print the score to the um to our notebook and then to actually evaluate all we do is pass the model to our evaluator and it will do everything for us and we get a similarity here or a correlation score of 0.76 which is is pretty good um you would you would think of you can think of something like 0.5 is kind of like your moderate correlation and 0.8 is like high correlation so that's pretty good okay and if we for the final quick part i just want to have a look at the other performances that i found because not everything is going to be as good as that but we just got there so we can see so this is what we just did over here we got 76 just now so we got slightly higher than what i got before so we got 76 here um but obviously like i said stsb is an easy data set to get good results on and the others are more difficult so the rest of them are you're more in this moderate um similarity range or correlation range so the medical question pairs days that actually perform better with the buying holder training than it did cross-encoder training but nonetheless they're still within the same sort of range that i would expect it's not it's not a massive improvement it's this is a the benchmark down here and there are a few uh percentage points better which is is probably reasonable particularly because they were not these glue data sets were not that similar to our medical question pairs data set so that that makes sense and then the other ones yeah they're also within that moderate range um the one that that's which did surprise me although it does make sense i mean their question pairs um is this one here so the transfer from from medical question pairs to quarter question pairs was pretty good and i suppose if you have a look at here we from core question person medical question pairs it's not as good and maybe because the language in quora question pairs is simpler than the medical question pairs i'm not sure uh but from medical question versus core question pairs it worked quite well so that i suppose points out where that engram similarity doesn't always correlate exactly to what you would expect but anyway i i think the resultingness are probably pretty typical from what you can expect now i think this can be really useful if you if you really don't have any any label data within your target domain to at least um squeeze out a few percentage points of performance more than you would be able to without uh this training strategy so for that reason i i think this can be quite useful um whether or not it is the best approach to take will depend on your on your data um so i think it's useful not always the best option but definitely something useful to know about and be able to apply if needed so yeah that's it thank you very much for watching i hope it's been useful and i will see you again in the next one