Vocabulary shapes cross-lingual variation of word-order learnability in language models

1. 1Pretrain transformer language models on synthetic word-order variants of natural languages
2. 2Analyze model surprisal to measure learnability
3. 3Investigate the impact of word-order irregularity and sentence reversal on learnability

NLP researchers and language model developers can benefit from this study to improve model performance on diverse languages, while data scientists can apply these insights to enhance language understanding in AI systems

💡 Greater word-order irregularity reduces learnability in language models

💡 Vocabulary affects word-order learnability in language models!