Mastering Small Language Models Behind the Scenes Secrets

Key Takeaways

we did a lot of fancy trick to to make the small language model work uh the most important thing I think is because I was leading the training team of by three uh the most important thing is we need to have very high quality data which means our data has a minimum noise and uh we need to do some kind of data pre-processing data cleaning all all before we do the training and that'll take lot a lot of time and a lot of money this is fascina to me because there is such a large amount of interest that goes into the small language models and if small language models are going to be this solution for architectures to do things faster and cheaper now whether or not that's a reality today is debatable I think you have to do a lot of work to make the small language model be as good as it can be right