Efficient training of language models to fill in the middle

1. 1Apply a straightforward transformation to the dataset by moving a span of text from the middle of a document to its end
2. 2Train autoregressive language models with a large fraction of data transformed in this way
3. 3Evaluate model performance using perplexity and sampling evaluations
4. 4Run ablations on key hyperparameters to prescribe strong default settings and best practices

NLP engineers and researchers can benefit from this technique to improve language model performance, while product managers can consider its applications in AI-powered products

💡 Training models with fill-in-the-middle technique does not harm original left-to-right generative capability

🚀 Efficient training of language models to fill in the middle! 🤖