NEW Transformer for RAG: ModernBERT

Key Takeaways

hello Community we have a new secret Transformer that nobody is talking about and you might say wait wait a second I just have a look at the leaderboard so who are the best Transformer system on this planet Gemini Gemini 2.0 flash sinking from December 19 tach oan preview Gro okay Claud Sonet no these are all known and you say hey wait a minute he and the license those are only proprietary maybe he means here open source and you say yes I've read this the best fully open source llm available starting December 8 2024 is here absolutely marks in and this is great this is a development by Northeastern University Harvard University Coral University and and and beautiful and you think this is it no because look we have an Apache 2 license we have a beautiful full GitHub repo we have everen un hugging face available for you unfortunately because it is not hyped and nobody's talking about this it has no downloads it is not sexy enough but you say you know what it includes the code training data now and you say hey to construct this particular stack data set 220 million active GitHub repost were collected here between 2015 and 22 and after the old initial filtering we had have about 5.28 billion unique code files with an uncompressed side of close to 100 terabyte so you might say this is it and look we have everything fully open source and you might say everybody would be enjoying this now but I hardly see any reports on this everything is here absolutely specified how it is done the training they give you everything they absolutely transparent this is the best open- source model you can find out there give this open source but then you look and say hey wait a minute this is based here on a llama 27b mall and you think hm on this channel somebody would recommend an old-fashioned llama 2 model and say maybe this is not it and you say okay we have to go back so a Transformer a secret Transformer nobody is talking about what could that be and I say yeah you know it's a Transformer that is absolutely essential for the complete information retrieval in thei I said this is the Transformer that enables rack systems any rack system at all for the last 4 and a half years and I say is a Transformer that creates Vector embeddings there would be new Vector store without of this and it say transform it creates semantic Vector spaces where you can have a coign similarity calculation and I tell you it's a Transformer that creates new mathematical representation and new mathematical spaces to be able to operate here with certain mathematical function on this and you might say well what Transformer is able to do this well I give you another hint if we go back here to the Transformer this is a video 2 years old and I show you here my goodness I have here a complete YouTube playlist with 11 videos that I showed you here a T5 model you know a sequence to sequence large language mall and we went through everything coding doing this in Jacks with flags and everything but you remember the most important thing as I told you here the Transformer architecture if you look at the Mal you remember here this is our particular Transformer in working visualization and you remember that I showed you two years ago we can divide this up somehow now if you only look here at the decoder part of the Transformer without the encoder we call this here the GPT models and opening eye was two years ago there this was a unidirectional causal language model that was great for Next Step prediction for the next token prediction but on the other side we had a bir directional masked language model and we call this an encoder only Transformer model and they were called bird and they were great for building here new representation for example in Vector spaces and now you might say okay so if it's not any of those model by open eye by entropic by Google by I don't know what it must be here under encoder only Transformer architecture and yes you're right so for more than six years we had a classical Bird model but today today it finally happens but let me just make it absolutely clear what is the difference between all the EI models we use today we have here our GPT systems our decoder only transform architecture so we employ an auto regressive processing where the tokens only attend to the Past token and we have those generative task but on the other side the encod only Transformer they input you a bidirectional structure allowing every token to attend to every other token capturing here deep contextual relationship so much better than the gbt models the task specialization was much better and if you look at the efficiency bird in rag was unbeatable so there we have it starting today we have now modern bird and this is this is like a Christmas present finally we have a new bird system and here we have it December 19 2024 smarter better faster longer and much more beautiful a modern B directional encoder of a transformer for fast memory efficient and long context fine-tuning and inference of llms and you know what beautiful Consortium on. lighton John Hopkins University Nvidia this is so important we will see here the optimization Nvidia did Implement here also for Consumer gpus so not only the h100s and of course hugging face so I was simply amazed to see this that finally our encoder only Transformer models our bird models we bring them now we equip them with the latest technology and we bring them up to date a amazing we have here modern bird here our GitHub as you see examples were just updated for 14 hours ago the source code here released yesterday it is an Apache 2 license it is there for you it is an amazing achievement you have it already on hugging face Ona modern bird large and a base you have two new bird malls which is simply great so this transition from the classical old bird system to the modern bird we changed everything architecture training procedure we have new efficiency improvements the key updates are simply amazing for modern bird we have a new sequence length we have a new efficiency oriented design with an implementation of the latest of flash intention and some un padding techniques to reduce your the computational overhead and utilize here GPU specific utilization including a consumer 490 we have an improved inference speed up to three times compared here to the other models we have have a complete new architecture we finally now use here our rotary positional embeddings our rope embeddings we have a complete new attention mechanism that increases here the speed significant we have a local Global attention mechanism that I showed you also in my last video when we were talking about BLT models we have new much more powerful activation function nonlinearity with gated structures to optimize the performance and the hardware efficiency plus all of this has been trained on the latest data on the latest best available data we have now code integrated here everything from repap documents and we have no broad application coverage and a better understanding of everything that is connected here with programming task but notice it is done for the language of English so if you have some other languages please for the moment this is here English implementation and then with the integration of Nvidia in the team we have a hardware aware and optimized design team for everything you can optimize on gpus if you want to understand the technology that we implemented there and you want to have a short overview here of the Gated linear units the Glu units this technology here from this publication by Google from 2020 these Glu variants were used here in this new modern bird system and if you're looking for the specific tokenizer because in my last video I showed you that meta developed a new tokenizer free methodology which is hailed here as something special but if you look closer you just see that it is applying a different methodology but with Dynamic tokenization we can make much further developments and if you want to read here that the tooner that they implemented in the modern bird architecture this is the allo tokenizer and this is the research paper from alen Institute University of Washington Yale University New York University and con M University from June of 2024 great now if you're not sure about the position the new positional encoding optimization I have two videos for you here and this here explain here the rotary position and Bing up to 100K Contex length and in the second video I give it here the further technological details and the code implementation if you want to go to 1 million tokens great but what is also absolutely fascinating and this goes here with my last video from yesterday where I showed you here this bite latent new Transformer architecture by meter that they you do not use a tokenizer and I told told you that they have if you remember in their local encoder structure they have here the self attention and cross attention but focusing primarily here on the local attention and then in the main Transformer they have here the global attention so they separate Here Local and Global attention for a better computational performance we have here the same we have now alternating local and Global attention mechanism in modern bird but integrated also on a architectural level we will have specific layers in our bird architecture that are only for calculating Global attention so what we want to achieve we want to balance the computational efficiency with the ability to process long context length and as I told you the global attention remember every token within a sequence attends to every other token this is quite computational expensive but it provides You full contextual awareness across the entire input and local attention we have here the sliding Windows implemented here so those particular tokens attend only to the small sliding window of neighboring tokens let's say 128 or whatever you go with which reduces here the computational complexity while preserving here the local contextual information of our patch of our secret of tokens real intelligent implementation to improve the computational speed and complexity so going now on the architectural side in modern bird we see that every third layer employs now this Global attention calculation and the other remaining layers use here with this sliding window the local attention calculation and in modern bird was implemented now a window size of 128 tokens you might ask why we do this well remember that Global attention is in the normal system as it is scaling quadratically with the sequence length and now by reducing the frequency of the global attention layers and the calculation modern bird achieves a significant computational savings it's faster and cheaper talking about faster I told you here about the positional rope configuration that we implemented but we also have an implementation of flash attention too and even if you use an Nvidia h100 flash attention reimplementation which is amazing this is brand new and they are designed specifically to optimize it mamory and their computer efficiency across everything this is done I suppose the part by Nvidia in this team by modern bird to make bird even faster on consumer gpus like a 490 up to an h100 GPU so a amazing speed Improvement new architecture new positional encodings everything has been brought up here to let's say 2025 to the technology of 2025 and we have finally a new modern birth and my goodness it was time so thank you to all those beautiful orst so they enable uses here to improve our information retrieval architecture everything that is connected with r and everything that is here using here something like a vector embedding if you do not want to read the technical paper there's a beautiful article on hugging phas you go hugging face articles published December 19 20124 title finally a replacement for bird it was uploaded 216 time here you have the or what they did some great stuff just to remember the importance of this secret Transformers you will not hear the ORS conclude we expect to see modern Bird become the new standard in numerous application where encoder only Transformer models are now deployed such as in the r pipelines you remember retrieval augmented generation and more or less in high high level recommendation system and I think this sentence underlines the importance here of modern bird the new standard in everything that is connected here with information retrieval everything that is about Rec pipelines everything where you have a vector representation an embedding of specific information where we bring in external information where you build fast Vector stores and Vector spaces to have a similarity operation in this mathematical spaces or anything if you go with a higher recommendation system so modern bird finally a beautiful Christmas present for the complete AI community and thank you to all those authors who made this possible and published here modern Bird on a free open-source license for the community and I will immediately Implement your modern bird especially for my systems I I hope you enjoyed it I hope you found some news idea and maybe if you subscribe I will see you in my next video