Nemotron 3 Explained: Hybrid Mamba + MoE for 1M Token Agents

Key Takeaways

Hey, welcome back to Bazer. Today we are going to break down Nvidia's Neatron 3, a hybrid mixture of expert model family that is literally built for multi- aent AI. Imagine you are running hundreds of agents in parallel planner tool callers evaluators all reasoning over a 1 million token context without melting your GPU or your cloud bill. That is exactly the problem Natron 3 is trying to solve. And in this video, a full technical walk through of how the hybrid mamba transformer mixer of expert architecture works. Why active parameters per token matter more than just saying 500 billion model and where nano super and ultra actually fit in real multi- aent systems you might build. Grab a coffee. This is deep dive not a hype real. What problem is Natron 3 solving? Let start with the plane. agentics systems are not a single chart bot anymore. You typically have a retrieval agent, a planner, one or more tool calling or coding agents, a verifier or judge model. All of them talk to each other. All of them write a shared memory and all of them extended the context window every time they speak. The result, a communication overhead explodes. context window fills up and your GPU build goes vertical. The traditional dance LLM are not designed for this. They active every parameter for every token whether the token is doing the complex reasoning or just passing the message along. Nevatron 3 says, "Let's keep the capacity of a big model, but only pay the competitions on a small slice of for each token. So, Neatron 3 family of open model, the Nano, Super, and Ultra, they all built around this hybrid Mamba transformer backbone. The Nano around 32 billion total parameters but only activated 3 billion per token and super roughly 100 billion total but about 10 billion active per token and Ultra is 500 billion total and it's activate 50 billions per token. Think of it like nano is high throughput worker model for spamming lot of agents and super and ultra or the heavyweight planners verifiers and long horizons. All of them target a context window up to 1 million token which completely changes how you design a long running agent workflow. Now let's understand the backbone of it and how natron 3 integrate all these three components. The mamba layer for ultraefficient sequency modeling over huge context and transformer attention layer for precise fine graining the reasoning. Mixer of expert layer for scale capacity without paying for every export on every token for numatron 3 nano. The technical reports is 23 mamba ch layer plus mixture of exports block plus handful of attention layers totaling 31 billion parameters with 3.2 to 3.6 6 billion active for every forward pass. What does active per token means? Let's imagine you have a big model and it's only activate certain experts within the big model. So you can you can have the specialized agents or experts to to agents to communicate only wakes up when roting says this token needs you. So instead of like one gigantic dense brain you get a cluster of small expert brains and each token only consult a few of them. And how this expert routing going to works? Let me zoom in the mixer of expert layer because this is where lot of magic happens in a pipeline how it's going to if you give a prompt the token get embedded and pass through the shared backbone layer and the router looks each token representation and scores for each expert and it picks the top expert for example six out of 128 it send the token representation only to that expert. So you can imagine if you go to the uh your your college or school if you wanted to talk to the max expert you have to go and reach to the max lecturer. The similarly the routing layer knows which particular expert it has to invoke depending upon the prompt comes in and where you put your equation out there and inside the expert it the agent can use a multiple agents with different reasoning and multiple step planning the driving everything going to be include the load balancing losses and it regularize the export. So it means you don't need to activate entire the huge dense brain to to consume the compute. It's it's only take the smaller brain the export and then activated them with a lesser cost and it has a huge uh contextual memory which is how the mamba layer is going to work it here and the computational cost is lower because I'm only activating certain expert now it comes to why we need the mamba here mamba 2 is a sequence model that gives you a linear time scaling So it means it keep track of long range dependencies with much lower memory. So Neatron 3 on the Mamba 2 handle the massive 1 million token contextual window without exploding the cache which means it understand entire code base sit in the context. The whole investigation timeline or incident report can live in a single window. Cross agent messages stay inside the model not scatter across the external store instead of playing a heavy tool to go to the uh rag models and get the agents operating it within the agent. It shared that huge contextual information with a shared memory that allow the neatron 3 to respond much faster and natron 3 this is not only the thing it has a multi-environment reinforcement learning and reasoning controlling so which enable neoim and multi-environment reinforcement learning align this model to use the tool usage, multi-step planning, code generation to actually runs and the oral data set include the trajectories for the agent workflows, tool calling calling sequences, verifiable outcomes just not one slice of it. You can on and off trigger your reasoning. You can enable the thinking budget to control your reasoning tokens, allow them to spend. It's you get a very cheap model with minimal reasoning that enable you to have uh a better uh open-source agentic multi- workflow agentic uh LLM model in behind and it's perfectly suitable for multi- aent workload and it has a massive throughput and massive context fixed scoring capabilities. That's why numatron is topnotch when compared to natron 2 nano. It's perform 4x higher tokens and thanks to the hybrid mamba and mixture of exports reduce the k caching and enable it to have 1 million token context. So you can keep long running histories and multi-documents in Windows and this agents works in parallel without blocking each other. You can generate a reasoning token aid the agentic looping amplification. This enable the perfect planner worker judge architecture. Natron 3 nano is very natural for this kind of a backbone for the modern agentic workflow and it's open family the weights data and training everything is going to be available you can download the natron 3 nanner today via vlm and hugging phase and you can start this understand the data set how it's been shrine and you can utilize Is this Neatron 3 on a platform like Nvidia's NM and microservices to the cloud providers which already integrating this uh model this enable you to work uh for tool heavy workflows and code and data ops agents and with the reasoning capabilities nano is perfectly suitable for background ground reasoning over long logs and traces. If you have the huge chunk of the data which isn't required to you to have a tool heavy workflows, nano is perfectly good for you. And if you wanted to have a global planner orchestration policy jet and high stake decision maker for top-notch, it's super ultra is perfectly fine as it's open source. You can finetune for your models. uh you can fine-tune for your agent trace evolution the logs and internal tooling capability title alignments for your specific workflows. So natron 3 is really a big shift towards the agentic world. So let me recap in one line. Neatron 3 is Nvidia's attempt to make big open model actually practical for the large noisy multi- aent systems by mixing mamba transformer and mixer of exports into single efficient backbone. And if you want to follow up where a full multi- aent stack is wired using Natron 3 nano with planner tools and evolution loop, drop me in a comment and hit me like and subscribe to the basai and share this with someone who really currently burning those GPUs on a dense models for agent workflows. Thanks for watching.