Evaluating Embeddings with MTEB Massive text embeddings benchmark - Nils Reimers

this year we published the massive text embedding benchmarks um which is kind of like a follow-up work bit more you know more put it bit more broadly in perspective so we collected different tasks like where you can use embedding models you can use them for clustering you can use them for BX mining meaning finding sentence with the same meaning in different languages you can use them for retrieve for semantic textual similarity for summarization for text classification for pair classification and for reranking and so we so so this is like a project that started also a really long time ago so when I published the sentence bird paper sentence Informer papers it showed good results on STS and S eval which was like the defact standard in embedding evaluation but if you really use it didn't perform that that well and also the original sentence SP model didn't perform that well as universal sentence encoder even such that The Benchmark on STS showed the opposite and so over the years with different help of different people and so on uh I collected first internally like a lot more training data a lot more data sets to evaluate my models on then luckily this year we have been able to open Source this and run a lot of evaluation on this and here we tested different models on these 58 data sets uh what you see on the x-axis is the speed like how quick is the model on the y- AIS how good is the model and then the size of the circles like how big how how many output Dimensions do you have and yeah what you totally see is like some cluster around here here where people used like bird based like 110 billion parameter models then we see like different sizes like the one model on mpet and GTR they perform extremely well also here some minm models work really well and then at the more left side we have like these massive six billion parameter model that show um also kind of like a nice Improvement but if you really compare the difference between like this model so here the center point and the npet center point the difference is not so small um but it's like two two magnitudes slower than these smaller models so I always found like this interesting to find like a model that's not only good but also would it fast so another interesting application we showed this year is fot learning with embedding models so gpt3 popularize fre short learning with prompt and in Contex learning so so what opening I showed in their paper is that if you want to do sentiment classification you can create like a prompt like this describe the sentiment then you have some in context examples I love this place which you say it's positive I don't I don't go here which is negative then you have a new review great pizza which you want to classify and I was never a fan of prompting and in context learning because first it has an extremely high sensitivity to The Prompt so here it can make a difference if you put a colon or an exclamation mark or question mark which can absolutely change the the performance and so you never know do I have to put like a colon or a question mark at the end of this do I have to add like a new line or not um also writing the prompt can be challenging for sentiment classification it's kind of easy but if you have like more complex nuanced um tasks it can be challenging to to write the prompt for that further challenge is you have limited context lengths so the original gpt3 can do like up to like 2,000 word pieces so if you have like longer examples like movie reviews you might can only present like 10 training examples but what happens when you have like 20 training examples examples or 100 training examples there's like no way how you can use these in a in context learning example and also you have quite High compute overhead so the the encoding um time complexity is quadratic with the length uh of the input so if you have to provide like all these examples at every call you like a lot of compute overhead to it