Sentence Similarity With Sentence-Transformers in Python

Key Takeaways

and welcome to this video on using the sentence transformers library to compare similarity between different sentences so this generally pretty short video i'm not going to go really into depth i'm just going to show you how to actually use the library now if you do want to go into a little more depth i have another video that i'll be releasing just before this one and that will go into what is actually happening here how we are calculating similarity or or pulling the how the model that we'll be using is actually creating those embeddings and then how we're actually calculating the similarity there so if you're interested in that go check it out otherwise if you just want to get a quick similarity score between two sentences this is probably the way to go so we have these six sentences up here and this one three years later the coffin was still full of jello and this one the person box was packed with jelly many dozens of months later they're saying the same thing but the second one is saying in a way that most of us wouldn't normally say it instead of saying coughing we're saying person box instead of jello we're saying jelly i think that's kind of normal actually and instead of years we're saying dozens of months so it's not really sharing the same words but we're going to see that we can actually find that these two sentences are the most similar out of all of these so we're taking those and we're going to be importing the sentence transformers library and we want to import the sentence transformer and then from that we want to initialize a sentence transformer model so we write sentence transformer and then in here we're going to be using this model that i've already defined a model name for which is the bert base mli mean tokens model so initialize that i need to rerun that so we have our model and i'll just show you really quickly this model is coming from the hugging face transformers library behind sentence transformers so this is the actual model we are using now first thing we do here is create our sentence vectors or sentence embeddings so we'll call a sentence vex equals model and code and all we need to do here is pass our sentences so we can pass a single sentence or a list of sentences it's completely fine and then let's just have a quick look at what we have here so you see that we have this big array and if we look at the shape we see that we have a 6 by 768 array so the six refers to our six sentences here and the 768 refers to the hidden state size within the bert model that we're using so each one of these sentences is now being represented by a dense vector containing 768 values and that means that we already take those and compare similarity between them so to do that we're going to be using the sklearn implementation of cosine similarity which we can import like this so sklearn pairwise or metrics pairwise and we import cosine similarity and to calculate our cosine similarity all we do is take that function and inside here we pass our first sentence so this three years later the coffin is still full of jello i want to pass that sentence vector which is just in index zero of our sentence vector array and because we are extracting that single array value so if we just have a look at this you see that we have a almost like a list of lists here if we just extract this we only get a list so what we want to do is actually keep that inside a list otherwise we'll get dimension error and then we do sentence vex one onwards so this will be the remaining sentences okay so let's take these or let's just bring them down here calculate this and we can see that our highest similarity by quite a bit is just 0.72 now that means that between this sentence and this sentence we have a similarity score of 0.72 so clearly it's working it's scoring the highest similarity and you can play around this and and test multiple different words and sentences and just see how it works but that's the easy way putting all this together so i think it's really cool that we can do that so easily but i don't think there's really anything else to say about it so thank you for watching and i'll see you in the next one