Make LLMs Fly: Accelerating Bielik With NVIDIA Minitron and Data Curator

Key Takeaways

Good. Hey guys, we're live, I believe. How was everyone doing? So, it's my pleasure to to welcome all of you guys. I'm Igor, I'm from Nvidia, and today we'll talk a little bit about uh model compression, about uh data curator, and about the story of uh Bielik, which is Polish sovereign AI builder. Uh yeah, so uh today with me there is uh three engineers from Bielik, uh Remek, who's going to talk a little bit more about uh model compression, Paweł, who's going to tell us the whole story of Bielik, and Karol, who will talk a little bit about uh the data curator. Welcome, gentlemen, and uh I think let's get it started. >> Yep, hello everyone. Uh so, uh today we'll tell you a bit about how our journey uh turns out to be our journey to sovereign AI, but before we achieved sovereign AI, we started with a magic numbers. So, it's kind of small as you can see, and it's the exact amount of data which the Polish data which was in Lama 2 data set training training data set. So, given that Polish is kind of hard language at the end. It wasn't enough for us. So, then we decided to collect some data to make model speak Polish a bit more and a bit better. Then then this is how this Speak Polish Foundation was founded. Today, we have over 3.3 terabytes of data and over 1 billion of documents which we used to create sovereign AI models. We didn't know that we create sovereign AI until we discovered that it's a thing in the world. And for us, sovereign AI is based on three main principles. First and foremost, it's security and it's being proven by the Polish National Social Insurance System in which they use BILEC to cover all their all their internal processes and use it to be fully compliant with GP GDPR and data processing internally. Second key elements of sovereign AI is openness and since our model is open, it's on Apache 2.0, we need to remember that it has been used as to be the engine of official Polish EU Presidency app called Prosta Pisemna or Simple Writing in English, and its main purpose was to help citizens to understand the documents a bit a bit better and make them more comprehensible. And last but not least, sovereign AI for us is about data quality. And here I want to use an example of our friends from top Polish deep legal research startup called Gaius Lex uh proved that a smaller model, a uh focused model, can compete with with the world's best in a very small uh precise field. And this field is law and legal research. And as you can see, Bialek in both of its versions, so 2.6, which is the previous, and V3, which is current, uh both of those versions uh were on par or even outpaced the GPT models. But, we don't only create LLMs. We have a citizen Bialek project in which we uh get data from citizens to build a multimodal model. We have BialekGuard, which is a Bialek uh guardrail-type model to secure not only Bialek, but all of your models throughout the ecosystem. We also create benchmarks. We have apps like Chatterina and ChatBialekAI, where you can test our uh solutions. And uh last but not least, we do some research and R&D, which uh Minitron originated from. But, before we go to Minitron, now we will speak about data a bit. So, Karol, to you. >> Okay, so hi, my name is Karol. Uh I'm from BialekAI, and I want to guide you through what we did and why we did. Uh so, before we go to the engineering part, uh our main goal was to deduplicate data. And why we did it, it was because we uh we were suspecting a lot of duplicates. And in established papers, we saw that deduplication can influence the model quality. So, the duplicates will waste your compute with diminishing returns. Uh it will make the model memorize instead of generalize. Uh it will prevent uh the model from learning from the bigger space of linguistic and knowledge, because it will have a smaller space. Mhm. And there is one special case and that is data contamination where data can leak and you can deduplicate on test and train sets. >> [snorts] >> So, uh we had a little bit problem with our hybrid cluster. Uh the we were the first adopters of uh Nemo Curator. It was version 1.00 and we hit a little memory wall that prevented us from utilizing our GH200 superchips as we would want. Since it was a uh hybrid cluster, we did a filtering phase on CPUs. And we had a very constrained memory on each node which was 384 GB. The standard filtering was uh with sub base ID removal. Uh so, let's go to the next uh well, I will >> [laughter] >> show exactly how our pipeline is different from uh what was first at Nemo Curator. There were three stages stages at first. Uh it was uh detecting uh duplicates with exact matching, with fuzzy duplicates, and then removing them at the end. Uh what we did to have a little bit more space in our uh in our cluster with the RAM was we did uh pre-filtering uh stage which was taken from another version that was to release at the time with Nemo Curator. Uh and we normalized our schema. We did early quality gate filter and we split the data into very small chunks by with uh 80 to 128 MB. Uh and that helped us with the next stages of filtering because the only thing that was bothering us was the CPU stage. We removed the Right now the stage is with normalization quality filter and split. Then we detect exact duplicates which goes very fast on GH200 superchips and then we do removal. The removal was converted to vectorized range bounded anti-join instead of just set filtering. We did a little bit of schema as well optimization. So it will all fit in the GPUs. Our Our stage now has five five pieces instead of only three and it helped us move past the GPU constraint. Sorry, RAM constraint. Okay, so the results that we achieved the 808 million raw data was from our internal corpus. We reduced it to 360 million. It was not only by the duplications since we had since we had the early filter gate with which filtered the the data that was classified with our internal classifier as low quality. So what actually happened was the data that we later filtered by our classified data as high, medium and 80% of confidence. The share of the high data in the whole corpus increased by 4.5%. And the result for our our web web data that we filter with composite filter that incorporates not only high, medium but a little bit of low data as well increased by 12%. So the quality of the data increased. Uh and the filters uh that we used it was about 80 80% of similarity threshold for the data. And stage reduced by about 160 million documents. So, the takeaways from what we did it's not always more compute power since we had the GH200 super chips, but sometimes you have to adapt our software and tinker with it. Uh so, the Nemo Curator modular API kind of helped us with that and we did it in days not months. We didn't have to fork the code. We just adopted codes that already were there. And the Bialik is fed by cleaner and higher quality data in the pre-training stage. So, what's to come? Uh we got a work in progress which is we're trying to test which thresholds of data will produce the best corpora for our next pre-training. Thank [snorts] you. >> Hello. This is my part. I will talk today about the compression. My name is Remigius. I am a member of Bialik technical team. Okay, let me start with the problem formulation. LLMs, large language models are powerful, but unfortunately they are expensive to run. They need a lot of GPU memory, a lot of compute. We have a new great computer platforms. For example, introduced yesterday Nvidia RTX Spark platform or small personal computer based on Blackwell where we would like to run very very nice personal local agentic system. But often models doesn't fit into the GPU memory. This is the the biggest problem. What can we do? Firstly, we can quantize model. This is This is standard attitude. Second one is look for smaller models. Or maybe buy more hardware. Assuming that we, I mean the purely provider, have a strong 11 billion model because our model is billion parameters model. We ask a simple question. Instead of training a smaller one from scratch and paying for it all over again, can we just shrink the the model we already have and provide for users a smaller model but still have the same quality. And the answer was probably yes. But we ask another question, how to achieve this goal? And for this task we used that model compression. Today this is a standard step for many companies like when deep seek mystery and video. All do it because the big advantage of the compression is it uses less than three three to 10% of the original training data and compute to provide a small model. So this is a much cheaper attitude, much greener and we can get model almost as good as original one. So we set up together with NVIDIA. This was last year project and we set three clear goals. At least 25% smaller model. 40% faster and keep more than 90% of the quality. These numbers guided everything in this project. So let me show how we did it. Okay, our approach has two main stages and this is an inspiration from Nvidia MiT Run approach. You can find a lot of really good examples and really good repo about Nvidia MiT Run approach. Uh you can find really good examples how to prune and how to provide such small model and provide it to the the the the production. There are many ways to prune model. We have a lot of methods and a lot of frameworks. By but we as a Bialik chose MiT Run for two reasons. First, this is very important for us because this is a hybrid approach. It cuts in several direction at once. So, you have only one framework and you can just cut everything you want in your model and it supports even my architecture. The second one, Nvidia gives us production-ready tool for it. This is very nice because they implemented a lot of different pruning and optimization techniques. So, you have a one tool and we can just prune prune using different techniques. Stage one in our in our project, it's structured pruning. We remove the parts of the model that matters the least. We find them by looking at the the activation statistics, just forward passes, then calculate statistics, no training needed. So, this was very very easy process. Then we cut the Bialik into two directions. First is the depth. It means that we cut the whole transformer layer. So, when you have a transformer layer, we can just uh uh calculate the activations. We can just um choose the the transformer layers and we just cut this this layers. And second attitude with width pruning. And we cut the size inside each forward feed forward network. All right, this is the part of transformer block after the attention. So, the the one the one more important thing is that this is a structured pruning. That means that we remove whole blocks, not only random not not random weights. So, the tools here was the model Nvidia model optimizer. Second stage was the pruning. So, the quality drops uh Pavel previous slide because yeah, because we talk about the pruning. So, so after the pruning the quality drops. So, stage two was to repair the the model with the knowledge distillation. The pruned model becomes becomes the students. Original model was the teacher. So, here's the interesting part two because the student does not learn from labels. It learns from to copy the or mimic the the output of the teacher. So, we just learn logits distribution. We use here another another great tool from Nvidia. This is Nvidia Nemo framework. And this is great tool because it supports not only distillation but pre-training, self-supervised learning, and alignment. So, you have everything in one package. After distillation, we run our normal training pipeline pipeline on top. So, in our case it was a self-supervised fine-tuning, DPO, and finally reinforcement learning. We use in the pipeline GRPO. Okay, next slide. So, the question is how much did we cut into our and what lessons learned we we we get. This was not a guess. So, we during this project we do ablation study. We tested 15 different configuration. We could have done more, but unfortunately, we were also limited by time. So, as I said, we got DGX lepton from So, resources from Nvidia. So, we have only We had only two two weeks to do all ablation studies. And the three examples here tell the whole story. The first first pruning was to to jump. We uh got only 27% smaller model. This is what not enough enough. As you can see, this is over 8 billion parameters, but our goal was to have 7 billion parameters model because we want to compare it with a Mistral. The second was too aggressive, the second case. We cut too much fit forward part of the transformer block. And the benchmark scores dropped significantly. The one we selected is 35% smaller model, which the best balance between the speed of quality. And probably the most important lesson for our dense Gaelic model was that the most sensitive was to touch the hidden dimension and touch this part of model decrease performance performance significantly. Second sensitive part was head attention pruning. We made several experiments during the project, but unfortunately, without success. When we had to balance between the depth and width and and find sweet spot when we removed too many layers, the model's reasoning didn't just get a little worse, it collapsed. So, we need to find the golden ratio between the two or two parameters, pruning depth and width. Finally, we kept 40 of the 50 layers to protect the reasoning and make the fit forward part about 25% narrower. Okay, next slide. Okay, the process of the pruning is very fast. Why? Because pruning doesn't need special hardware. To be honest, I just made pruning process on local HP Z6 workstation. Just under the desk, I prune all all all models on on on local machine with one GPU. GPU is required because it requires pruning requires it for for inference because we use it for collecting activation statistics. Second reason is on the data side. We don't need a lot trillions billion tokens. We use only 2 billion tokens. The process were were very fast it was very fast. We prepared 30 billion tokens data set, but we came to the conclusion that 2 billion token so very fast process. The the loss of the the models was very good. So, so we decided to take only 2 billions of token. This is very tiny tiny amount of our our data set. And the last thing was the stuff we used the the the GPUs because we use the Nvidia H200 GPUs on GPU DGX Cloud laptop. Thank you again Nvidia for providing us two notes for this testing. >> [snorts] >> And the key feature here is memory because as you can probably know uh distillation needs two models uh both models at once in the in the GPU. So, we didn't use any offloading to slower memory for checkpointing. There were were no complicated parallelization tricks. So, only two to three days to to distill the the the the the the model and to to provide new new model. Okay, and here benchmarks because you can ask me, "Okay, did you hit your quality goal?" Next slide. Okay. And yes, uh we managed to recover the the the model 90% of the teacher's quality. So, the quality is very very good. And we got model about 30% smaller. So, this is very good and it is consistent across all benchmarks. So, Polish leaderboards almost 95% of the original model. Reading comprehension 94% function calling over 92% And it is very good model, but our 7 billion model beats other models on the same of the same size like Qwen and Mistral 13 on Polish Polish benchmark. Okay? Now, the speed goal. We passed it too. Our throughput went up to uh almost 50%. The time to produce each token dropped by a third. So, the model feels clearly faster. And it is smaller. You can just uh download this model or put this model onto the uh consumer GPU card like the RTX 40 80 or even on laptop GPL and use it locally. You can go even smaller. If you quantize quantize this model uh to 4-bit, it keeps about 90 90% of its quality. So, uh we tested it using the NVIDIA FP8 and FP4 formats, and it works great great, too. So, the result the result is a model that we have 7 billion parameters model. Almost anyone can can download it and run locally. Okay. And next slide, to recap, very short recap, uh as you can see, our goal were were more than 25% smaller model, more than 30% faster, more than 90% quality, and the final result is 4 33% smaller, almost 50% faster, and 90% of the quality kept. All hit all three We hit all three uh goals. So, um this is how we achieved this, and this is our um our journey with with NVIDIA during the model optimization and working with Nemo optimizer and with uh Nemo framework. So, thank you very much for your attention. >> Okay. Uh the results that we just saw were quite incredible with like the the the speed uh and the and the size that you guys managed to optimize. Um I'm going to go back to a couple of the topics that you mentioned and we also got questions from the audience. Specifically, we got question from Pranshu 041. So, um to what extent did the activation based pruning with a domain specific calibration data set affect the performance of the model and did the RL post pruning help in any way? >> Yeah, yeah, yeah. This is very good question because because we didn't have to do do such ablation during this this project, but this is really good question. We managed to We we tried to to provide the or prepare the data set which is the same uh the same which has the same character with we have during our uh pre-training. So, the uh the model was almost the same. So, it was very small, but the distribution of the tokens were probably very very close to our uh our pre-training data set. So, we had a very tiny representation of our pre-training model. This is This is This is why we attempt to to to to to this problem. >> Okay. That's fair. And on the other hand, um was the original So, that that's a question actually from Gatotkaca 1821. Um they're asking whether the the model that we're currently discussing was it trained from scratch and then maybe the follow-up from the original model that the guys trained. What was it based on? >> Yes. Yes, yes, yes. This is very tricky and this is very tricky because because how we provided the Bialik model. >> [snorts] >> We took the uh um Mistral original Mistral model 7B and we uh did something which is called the uh model scaling. So, we add on top uh additional 10 uh transformer layer and when we conti- uh we provide the continual continual pre-pretraining. So, this is the something like the the mix. So, we we just achieve 11B starting from 7B Mistral model. This is why I uh told during this presentation that we wanted to uh go back to the 7B to compare it to Mistral 7B. So, we uh only adapt the model to the Polish. So, we use the base base model, different one. Mistral is a main our uh our main model, but we use a Qwen as well for smaller one. But, this is why we prepared this model. So, we took the Mistral 7B, then we scaled this model to 11B B and then we pretrain it continual using the continual pretraining and then we provide the 11B and uh during the model compression we go went back to the 7B. I don't hear you. >> Yeah, sorry. Um That's That's great to hear. Quick question, uh Pavel, are we ready for the demo? >> Um yep, we're ready. So, uh yeah. So, we prepared a small demo uh to you guys, but the given our talk was a bit uh longer than we uh planned, I started and run it beforehand. So, uh the setup I have here is the LM Studio in which I have loaded simultaneously five models. So we have the main star of the show, we have the Eleuthera 7B. We have Qwen 3.5 9B, we have original Eleuthera 11B. We have Gemma 26B A4B, so the MOE model, and also we have the European Minstrel 14B with in the reasoning option. And given those are kind of same sizes, we wanted to show you how Eleuthera, a model which we assume you don't know, and we want you to know a bit better after today, compares to those well-known ones. So I did run a benchmark. As you can see, here are the all the logs. We can probably we will provide you the link to the GitHub repo, of course, after the after the show. And the results I want to share with you are as follows. Here we run models on two benchmarks, the throughput, so how fast they can generate, and F1 score on the NER benchmarks, so the entity recognition. And it's being done on five languages. So as you can see here, the languages we did this on are Polish, German, French, English, and Spanish. So leading European languages plus Polish. We did this because V3 Eleuthera models are currently also using and supporting over 30 European languages. And if we zoom in on those results, we can see very clearly that our star of the show, so Eleuthera Minstral 7B is kind of on par when it goes to performance I had to go out of here, sorry. Eleuthera goes out of uh performance is kind of compared to the other models which are a bit bigger, like Gemma 26B, or like ministrone but when it goes to raw speed it can all all the way uh it can almost be compared to Gemma which has only 4 billion parameters active when it responds. And given that our model is dense and it's only has 7B parameters is a kind of achievement I'd say. I don't think I don't know what you think about it. Uh so that's the one thing and the other thing I want to show you is how the heat map looks for all those languages. So as you can see uh across French, German, and Spanish all of the models are basically spotless. The The only model that is spotless throughout all the categories is when. So kudos for our Chinese friends. And when it goes to Belik Minutron we can see that it's on the last row and it's spotless in French, German, and Spanish. And it's small mistake in Polish as well as the 11B version and as we can see those are basically the same results. So we saved 33% of size of the model as Remek showed you before. We made it generate tokens over close to 50% faster. And it maintained the quality that it originally had and just to let you know those models we used FP16. I did this benchmark on the Q8 versions of those so we can save some time. All these results as you can see here summed up. The Belik Minutron is generating about 50 tokens per second on my local equipment. Everything is done locally and given we also encourage local inference and uh encourage you to do so, to try it out yourselves. And given our models are so small, especially the Minuteman version, which in QA takes only seven gates of either RAM or VRAM, depending on your hardware, uh you can have kind of uh powerful model on your computer, on your desk, to run all the tasks you need. And please try it out in the European languages. We know it's kind of good in Polish, but we want to know if it's any good uh in in your language. So, please try it out, provide us your feedback, and uh then we can surely make the best out of it as we did previously. And all the feedback we got on the previous models, uh we were able to make it out into a feature, not a bug, in the next version. >> Okay, then maybe about the the European languages, uh coming back. Uh Muhammad Salman Hussein, uh he also asked a good question on how do you exactly like decide the right balance between the model size, the quality when optimizing for like specifically local European languages? What was your take on it? >> Um well, I'd say that uh given that each and every of our models that we've created was better than the previous one, uh the secret sauce, I'd say, lies in the uh data quality uh and our dedicated data pipelines, so a bit uh which Carl talked about, uh but it's only the peak of the iceberg, to be honest. And uh it's it's much more complex than what Carl discussed. So, I say that the key here is the right um right quality of the data we feed the model through. And uh playing with this, we can achieve the goals we we want to achieve. >> And I would I would like I would like to know to add something. When we did the optimization during the pruning method each model was uh passed by our benchmarks. We have We had We have very sophisticated benchmark framework to test model not in not only in Polish but in many languages. Part of this benchmark are made here in Bialystok. So each pruned version was tested during this this process. So we saw which model is the best one. And this is why we could take best decision to to to to to to choose the the the right model for next stage. >> Okay. Fair. Um so okay guys, we should be probably wrapping up very soon. So I'll let myself ask the one last question to you. Um why did you even decide for model compression instead of training smaller model in the first place? >> Oh, this is because this is a very effective very efficient way to have a new model. You know? So we could do it and train model from scratch but this is very very expensive process. So we decided to try after after Nvidia first successes we read in your blog we decided to do the some kind of research and development project and try to to use this whole Nvidia machines, you know, frameworks, software and and to look how we can do together. And it appeared very fast just after two or maybe three days that it works. Yeah. So, we can provide smaller model to to our customer very very fast and don't use a lot of, you know, supercomputer we have. So, we did it for small model, but we we know that we can just do it for bigger one and many companies do it today and they distill distill model and prune prune model and provide smaller one. Yeah. And here is one more question I would like to ask if we plan more version of Bielik. Yes, probably yes. We'll We work on many different Bielik models now, so we'll we'll see, but probably next or maybe this this this month will provide you more models. Better one. >> [laughter and gasps] >> All right, guys. Then I guess the last thing that we should mention is stay tuned for new releases of Bielik. All of the all of the models that the guys are training are on Hugging Face, as you can see, and that's the link to the to the to the mini tron model, the one the compressed one that that we just discussed today. And with that, we'll be wrapping up. So, it was pleasure to to host all of you guys and yeah, thanks for tuning in and uh take care. Have a rest a great rest of your day, guys. Yeah. >> Thank you. >> You too. Bye-bye. It was a pleasure to be here. >> Bye. Bye-bye.