Will Unified Memory Kill Discrete GPUs for AI?

Key Takeaways

Are increasingly expensive GPUs really the final answer for local AI? Sure, they're very fast, but what about RAM and VRAM? LLMs are getting larger and larger, and AI video generation also demands ever more VRAM. At the moment, this only works with the most expensive GPUs, and even those hit their limits when faced with the massive memory requirements of modern models, often exceeding 100 GB. So, what are the alternatives and what are their pros and cons? One promising approach, though it's not entirely new, is unified memory. Apple was the first to make it a headline feature in mainstream computing. But the concept itself isn't exclusive to them. AMD and Nvidia have implemented similar architectures, though they often use different names like shared memory or coherent memory in their technical docs. You'll see it in AMD's Ryzen AIX and in Nvidia's DJX Spark. Unified memory allows CPU and GPU to share a large common memory pool, potentially solving the bottleneck of fixed VRAM sizes. In this video, we'll explore what these new approaches do better, especially for large language models and video generation. But of course, we'll also talk about the downsides. And yes, we'll take a close look at both performance and pricing. What do we actually mean by unified memory? Traditional computer architectures use segregated memory pools. system RAM that can be accessed by the CPU and dedicated VRAM used by the GPU. This means data must be copied whenever both processors needs the same information that introduces latency, increases complexity and creates performance bottlenecks, especially in memoryheavy workloads like large language models or video generation. Another architectural limitation is the classic tradeoff between speed and capacity. System RAM tends to offer a lot of space but is comparatively slow while GPU memory is fast but limited in size. Developers often need to juggle between them deciding which data must stay close to the compute units and which can be offloaded without impacting performance too much. Unified memory takes a different approach. Unlike traditional computing architectures, unified memory uses a single shared memory space that can be addressed by different components such as CPU, GPUs, and specialized AI units or accelerators. By the way, Apple calls it unified memory while AMD and Nvidia tend to use terms like shared memory or coherent memory. But the core idea is same, a large fast memory pool that everything can access directly because of the shared address space. There's no need to duplicate data in separate memory regions. This eliminates transfer overhead and allows workflows to flow more efficiently between compute units without the traditional memory bottlenecks. But wait, haven't we had shared memory before? Like on laptops with integrated graphics? Technically, yes. Systems with integrated GPUs, like many Intel and AMD laptops, have used system RAM for both CPU and GPU tasks for years. But these older solutions come with limitations. The memory bandwidth is much lower, often under 100 GB per second. CPU and GPU still use different memory mappings, so they can't access the same data directly without coordination. There's no automatic memory coherence and copying is still often required. In other words, those setups share them physically but not architecturally. And even in higherformance desktop systems with dedicated GPUs, we still see signs that memory separation is a bottleneck. Features like resizable bar found on many recent gaming and workstation PCs allow the CPU to access the full GPU memory range instead of working in small chunks. That helps in some workloads, especially in gaming. But it's still just an optimization to cross the memory gap, not a way to remove it. The CPU and GPU still live in separate memory spaces and require driver level coordination. Unified memory takes a different path. It removes the divide entirely. CPU, GPU, and AI units all access the same high bandwidth memory pool natively and coherently. That architectural shift is what makes modern unified memory so powerful. However, unified memory introduces its own technical considerations. Memory bandwidth must be sufficient to service multiple coherent access patterns from different compute units. And memory controllers have to prevent one unit from blocking others. As we'll explore in the next chapters, bandwidth is one of the key enablers for making unified memory truly effective. With these considerations in mind, let's examine how leading hardware manufacturers have implemented unified memory approaches. In this chapter, we're going to look at how Apple AMD and Nvidia are building unified memory into their systems. We'll also compare them with traditional GPU and CPU setups, so you get a clear picture of what's different. By the way, if you want to see more videos like this, hit the subscribe button and ring the bell. Apple was the first to bring unified memory into mainstream computing in a big way. With the launch of the M1 chip in 2020, they introduced the design where the CPU, GPU, and neural engine could all access the same memory pool. That idea has evolved through the M2 and now the M3 series, culminating in the M3 Ultra. As of now, it still offers the highest memory capacity and bandwidth in Apple's lineup, while the M4 has already been released. There's no Ultra version yet, and certainly nothing with the same bandwidth or memory capacity. That's why we're focusing on the M3 Ultra here. You can get up to 512 GB of super fast LP DDR5X with around 819 GB per second of bandwidth. For context, even high-end DDR5 6,400 memory in desktop PCs only reaches about 51 GB per second. So, Apple's implementation is on completely different level. That's slower than the best high-end GPUs with HPM. but way faster than regular desktop memory. The chip itself combines 24 performance and eight efficiency CPU cores, an 80 core GPU pushing about 28 flops, and Apple's own neural engine. It starts around 4,000 USD, though fully maxed out machines can go over 12,000 USD. AMD takes a slightly different route with the Ryzen AIX Plus, also known as Stricks Halo. You get up to 128 GB of LPDDR5X memory with about 256 GB of bandwidth. That's less than Apple, but the chip is more affordable. The GPU offers around 12 T flops and includes a dedicated NPU neural processing unit called XDNA2. It delivers 50 tops and is tuned for low power AI tasks like realtime image or speech processing. Depending on how much memory you choose, complete systems start around 2,800 USD. Note on availability. While the Ryzen AIX Plus 395 is mostly found in the pre-built systems, there are early signs of modular options. Framework, for example, offers a desktop setup where the stricks Halo based mainboard can be purchased separately. However, key components like GPU and RAM are solid, so upgradability is still limited compared to traditional do-it-yourself PCs. Notice something a bit different, the Nvidia DJX Spark. It's a compact machine built around the new Grace Blackville architecture. It has 128 GB of LPDDR5X memory shared between CPU and GPU with about 273 GB per second of bandwidth. The bandwidth is lower than Apple's M3 Ultra and the memory capacity is about a quarter of it. But in terms of raw compute, the DJX Spark holds its own. The GPU delivers 30 T flops FP32 or up to one P flop in low precision AI. The price is starting at 3,000 USD for consumer versions. The DJX Spark is sold as a complete integrated system. There's currently no option to buy the components like the Grace Blackville Super chip separately for do-it-yourself builds. GPUs like the RTX 5090 and 5080 don't use unified memory. Instead, they rely on fast local VRAM, 32GB GDDR7 for the 5090, 1.8 terabytes pers bandwidth, and 16GB for the 5080, 960 GB per second. They connect to the rest of the system via PCIe, which means data transfer have to be managed manually. That makes them less ideal for huge memory models or anything that requires fast back and forth between CPU and GPU. But for workloads that fit into VRAM, like rendering or self-contained model interference, they're still incredibly efficient and they're cheaper. 2,000 USD and 1,200 USD. For comparison, take the Intel Core i94900K. It supports up to 128 GB of DDR5 RAM, around 90 GB per second bandwidth, and offers around 6.1 T flops. Not amazing for AI, but very versatile. A full system costs around,500 USD. In the next chapter, we'll step back and look at what all this means, where unified memory makes the biggest difference, and what tradeoffs still matter when building or buying a system for local AI. The technical implications of unified memory architectures require deeper examination to understand their advantages and constraints in AI generative contexts. Regarding technical advantages, the biggest gain is model capacity. Up to 125GB is enough for huge models with hundreds of billions of parameters. That means you can run them locally without paging or offloading which is critical for performance. Another key advantage is data movement. When CPU, GPU, and NPU share a memory space, data doesn't have to be copied. just referenced. This reduces latency, cuts of bandwidth use, and simplifies coordination across compute units. For AI workloads that span pre-processing, interference, and post-processing, that's a big win. Energy efficiency improves, too. That's mainly because you avoid unnecessary data transfers, which normally eat more power than compute itself. Regarding technical limitations, unified memory isn't magic. One clear limit is bandwidth. Most use LP DDR5X, which tops around 800 GB per second. That's a lot, but not as fast as GDDR7 or HBM, which go far beyond 1 terabte per second. for tasks that need constant data flow like high resolution image or video generation that can become a bottleneck. Another consideration today unified memory is typically paired with integrated designs that prioritize efficiency over peak compute. As a result, the raw performance of CPUs and GPUs in these systems currently fall short of high-end discrete accelerators. Another issue, shared memory means shared contention. If CPU, GPU, and NPU all want excess at once, the memory controller has to juggle priorities. Even with good arbitration, that overhead can slow things down. And finally, no upgrades. Today, unified memory is usually soldered or on package. You can't add more later. What you get is what you buy. That means you need to plan ahead, especially if your models or workloads are likely to grow. Where's unified memory heading? Looking at current trends, we can get a solid idea of what to expect by 2026. We likely see 64GB unified memory setups become standard in mobile and mid-range systems. What used to be high-end will become normal. Apple's upcoming M4 Pro and AMD's next tricks point chips are expected to include bigger memory pools across the board. The reason is simple. AI workloads are becoming mainstream. Intel's Falcom shores is worth watching. It mixes on package HBM with socketed DDR memory that solves the key problem of most unified memory systems. limited expandability with both fast and scalable memory in your system. It gives you bandwidth where you need it and capacity where you want it. The tricky part is building software that puts the right data in the right place depending on how it's used. Raw compute power is still important, but unified memory changes what's possible with that power. It removes bottlenecks, allows bigger models, and simplifies data handling. That's not just convenience. It's a shift in what kinds of problems you can solve locally. And maybe it's worth stepping back for a moment. Many of the fixes we've seen like resizable bar or high-speed memory optimizations are really signs that the old separation between CPU and GPU is starting to show its age. Meanwhile, traditional GPU workloads like rasterization aren't scaling the way they used to. The big gains are now coming from AI and AI demands, massive memory, fast coordination and unified execution across compute units. So perhaps the division between CPU and GPU is becoming artificial just like the floating point crop processor once was. First it was separate, then it became standard and eventually disappeared into the main processor. Unified memory might be following the same path. It's not just a feature. It's part of a deeper architectural convergence. Will unified memory replace the old split between CPU and GPU or will we just keep patching around the limitations? Let me know in the comments. And if you don't have a keyboard nearby, just leave a