Multilingual and cross lingual embeddings - Nils Reimers

Key Takeaways

so final topic or second final topic is like multilingual search and so that's the most recent project I've been working on at at kohir to work on a multilingual embedding approach so far the dominant method in searches like lexical search bm25 but if you go and have like a multilingual data set let's say you create like some platform I don't know you you create Reddit people post in all types of languages on Reddit or Twitter and you want to provide like search function for all these languages it's like really really ugly to build this first you need to do language identification on docs and queries because every language needs a different tokenizer so in English you can't just tokenize on white spaces and then indexes but this does not work for Chinese so for Chinese you need a different tokenizer than for English and so on also stock worthless must be different so in English you have certain stock words but these stock words are different for French German Chinese Russian Turkish Arabic also what you often do is like stemmer where you reduce words to the stem like docs plural docs goes to Dock and that's also like language specific so every document has like a different pipeline different tokenizer different stock words different stamma for every language you need a different index so you have an index for all the Arabic documents or the Chinese documents or the English document and then if I enter a search query I first need to know what search what language is this query in and this can also be challenging because there are words which it's like a beak if you take the English word die in Germany the word b written the same way as an article so just from the word you don't know what's the language and so you don't know which index do I need to hit do I have to search the English index or the German index so you might need to query like multiple indices and out of the box systems like an elasticsearch only support few languages because it's like really really painful to create these tokenizer stockwards demo for every language with multilingual embeddings it's quite easy so you take the model you take the text path through Transformer Network you get an embedding out of it you don't need to do any language identification you don't need to have like any stemming stop words and so on so everything can be the same pipeline but however there have been previous work on multilingual embeddings but a big challenge was the the lack of training data so what people did before is they used from the neural machine translation Community translate sentences and train models on this or light models English models on this but here we see that models only work well on the sentence level another line of researchers like people use machine translation like Google Translate to translate query answer pairs to other languages but here the models don't learn language and Country specific properties so that people have been using Ms macro which has like a lot of questions how to do taxes in the yes and how to file certain forms and then translated this to German so now the model knows in German how I could find my taxes in the yes but the model has absolutely no idea how to do taxes in Germany and how to file all the forms that's required in Germany to do my taxes and as mentioned before models are really bad and out of domain settings so if I hit the model and ask some German specific question how do I do how do I file taxes or how do I get a tax refund the model has like absolutely no idea how to do it and how to learn it so what we did at here we say Okay data as is the key so we collected large quantities of training data not from machine learning translation but actually written by people of that language so for example Germans asking questions how to file the taxes in Germany together with the answers written from Germans how to do the taxes or how to fast certain forms in Germany and overall we close collected up to like 500 million non-english pairs and which we carefully created cleaned extended augmented and so on which covers like a lot of different topics and so we hope that the models are really broadly applicable and can find relevant information across all these languages to minimize the Gap we see in these models have on unknown words and unknown domains and of course we we put it also to a test in different settings first was like clustering and search in English search on non-english languages and cross-lingual classification we have training data just in English and I want to do classification in other languages and here the biggest Improvement we saw for search in non-english where previous methods they all worked like on a sentence level and here we see like a really big boost of performance on the non-english languages and one approach which is absolutely amazing was these embedding approaches it's like cross-lingual search so you can type the query in any of the 100 languages so here it's Arabic I'm not speaking Arabic my colleague Amer was so kind to provide this but he said supposed to ask the question what's the capital of the United States yeah you search the English Wikipedia on this and obviously elasticsearch has no idea what this Arabic garbage is but if you do semantic search it has like no idea no problem to match this because this text is matched to a vector the vector is really close to the English what is the capital of the United States and then you get the perfect match to like Washington DC is the capital of the United States foreign aspect here's in terms of like language bias if you create these models so either some models they have like a strong language bias meaning they prefer certain language combinations so for example if you take the libse model you see all the Russian points are in the left corner or the English points and the right corner for other models it's more like mixed together and there's like no strong separation between languages foreign ERS like is language bias good or not or not good so side effects with language bias is that same language results are ranked higher just because of the language so if I search what's the capital of the United States in Arabic I preferably find Arabic search results in a multilingual Corpus even such that maybe the English hit is it's a better answer to my question um but so so you could think that a model was odd language bias is nicer that it finds you the perfect document the perfect information without language but here the challenge becomes in things that are specific to languages where languages are also really tightly coupled with countries so for example if I search in English for wedding in an image search system some happy picture like this so you know that's the traditional Western picture how a wedding should look like the the bright and the white dress and the man and the smoking will suit black suit uh white skin if I if the model does not know about the language I'm searching in and I search in the Hindi word for weddings exactly I'm not speaking Hindi but Google translate told me that's the only word for wedding and the the vector spaces does not have like any information about the language so it just sees the content in terms of wedding it will retrieve the same result and here it's like doubtful if people that search in Hindi for wedding would be really interested to get like a western picture of a wedding presumably such person would be more interested to get like some weddings how they are typically celebrated in India similar who's the president here we assume Joe Biden links to president but if you ask in in French who's the president you probably not so much interested in the US president but you're probably more interested in the French president or maybe I don't know you used to speak there are a lot more french-speaking countries maybe you're interested in in a in the president of the respective countries so there's the question like how can you still preserve these country language specific properties if you don't have a language bias in the model