Why ChatGPT Needs GPUs | AI Hardware Explained

Why do AI systems run on GPUs? If you have ever wondered why training Chat GPT costs a hundred million dollars, in compute, or why Nvidia became the most valuable chip company on Earth, the answer is sitting inside this question. GPUs were designed in the 1990s to render video game graphics, but the mathematics of rendering pixels turned out to be almost identical to the mathematics of training neural networks. Both problems require the same thing, billions of small, independent calculations running at the exact same moment. Today on Neural Monk, we go deep on GPUs in AI. We will cover the architecture, the math, the memory, the software stack, and the real performance numbers. By the end of this video, you will know exactly why GPUs power the AI revolution, and what makes one chip faster than another for artificial intelligence. To understand GPUs, we first need to understand how they differ from CPUs. A CPU, the central processing unit in your laptop or server, is designed for sequential tasks with complex logic and branching. Modern server CPUs have 8 to 64 cores, each extremely powerful, running at 3 to 5 gigahertz with large caches to minimize latency. They are optimized for the kind of work your operating system does, handling one complicated task after another as fast as possible. A GPU was built for a completely different problem. Rendering a game requires computing the color of every pixel on screen. A 4K screen has over 8 million pixels, each needing its own calculation. These calculations are completely independent of each other. So, instead of a few powerful cores, GPU engineers packed in thousands of smaller, simpler cores that can all run simultaneously. The NVIDIA H100 has 16,896 CUDA cores. That is the fundamental architectural difference. CPU, depth. GPU, breadth. So, why is the GPU architecture such a perfect fit for AI? Because neural network training is almost entirely matrix multiplication. When your model does a forward pass, it multiplies the input vector by a weight matrix, adds a bias, applies an activation function, and passes the result to the next layer. That weight matrix multiplication is the core operation. And here is the crucial insight. Every single output cell in a matrix multiplication is independent of every other output cell. To compute row two, column three of the result, you do not need row one, column one. They have zero dependency on each other. This means you can hand each output cell to a different GPU core, and compute all of them simultaneously. A 3x3 matrix gives you nine independent results. A 16,000 by 16,000 attention matrix gives you 256 million independent results, all computable in parallel. A CPU would crunch through them sequentially. A GPU fires all of its cores at once and finishes the job in a fraction of the time. Let us go inside an NVIDIA H100 GPU and look at the three key hardware units. First, the streaming multiprocessor, or SM. The H100 has 132 streaming multiprocessors. Each SM contains 128 CUDA cores, along with its own registers, shared memory, and instruction scheduler. Think of each SM as a mini processor managing its own thread block. Second, and most important for AI, are the tensor cores. Nvidia introduced tensor cores in 2017 specifically for deep learning. A tensor core performs a 4x4 matrix multiply accumulate operation in a single GPU clock cycle. That is 16 multiply adds in one clock. The H100 has 528 tensor cores delivering 989 teraflops on half precision AI workloads. Third is the HBM memory, high bandwidth memory. The H100 has 80 GB of HBM3 memory sitting right beside the die. Connected by thousands of data pins delivering 3.35 terabytes per second of memory bandwidth. This combination of massive parallelism, tensor cores, and extreme memory bandwidth is why a single H100 can do what would take hundreds of CPU cores. Here is the bottleneck that most people overlook when thinking about GPU performance. Raw compute power means nothing if your cores are sitting idle waiting for data. Imagine a factory floor with 16,000 workers, but only one narrow door for materials to arrive. That door is your memory bandwidth, and it is the hidden constraint in AI workloads. A large language model inference run is almost entirely memory bound. The model weights must be loaded from memory into the compute units on every step. A 70 billion parameter model in FP16 precision weighs 140 GB. With CPU DDR5 memory at 150 GB per second, streaming those weights takes nearly 1 second per pass. The H100's HBM3 delivers 3,350 GB per second. Those same weights stream through in about 40 milliseconds. That is a 22 times bandwidth advantage translating directly into 22 times more throughput for inference. This is why the memory type matters as much as the core count. Bandwidth feeds the compute engine. No bandwidth, no speed. Hardware alone does not explain Nvidia's dominance. The other half of the story is software. In 2007, Nvidia released CUDA, the compute unified device architecture. CUDA gave developers a way to write general-purpose programs that run on GPU cores using a modified version of the C programming language. Before CUDA, using a GPU for anything other than graphics meant working around OpenGL or DirectX, a painful hack. CUDA made GPU programming accessible and created an ecosystem. Over the next 17 years, Nvidia and the research community built an enormous library of hand-optimized GPU kernels. cuDNN handles convolutions, batch normalization, attention mechanisms, and recurrent layers. cuBLAS handles matrix multiply for any precision. These libraries are tuned down to the microarchitecture level for each GPU generation. When you call a PyTorch function, it ultimately calls cuDNN or cuBLAS. When you use TensorFlow or JAX, same story. CUDA is the reason AMD, despite having competitive hardware, has struggled to match Nvidia in AI. The software mode took 17 years to build and cannot be cloned overnight. Let us put real numbers to the GPU advantage. Training ResNet-50, the classic image classification model, on a high-end server CPU takes approximately 14 days. The same training run on an Nvidia H100 takes about 4 hours. That is an 84 times speed up from one chip. For language model inference, GPT-2 with 1.5 billion parameters generates roughly 40 tokens per second on a CPU. On an Nvidia A100, the same model generates 3,000 tokens per second. 75 times faster. BERT fine-tuning drops from 80 hours on CPU to 30 minutes on H100. 160 times faster. Looking at raw specifications, the best server CPU delivers about 4 teraFLOPS of FP32 compute with 460 GB/s of memory bandwidth. The H100 delivers 67 teraFLOPS, 16 times more compute with 3,350 GB/s. Seven times more bandwidth. These are not incremental improvements. They are the difference between a research project taking a week versus an afternoon. And between serving one user versus thousands simultaneously. A single GPU, however powerful, has limits. GPT-4 was estimated to require approximately 25,000 A100 GPUs running for 90 to 100 days at a cost exceeding $100 million. That model That model cannot fit in the memory of a single GPU, let alone be trained on one. This is where distributed GPU training comes in. There are three main strategies. Data parallelism is the simplest. You replicate the model on every GPU and split the training batch. Each GPU processes its portion of the data, computes gradients, and averages them. Model parallelism splits the model itself across GPUs. GPU one holds the first half of the layers. GPU two holds the second half. The activations pass between them in sequence. Tensor parallelism goes deeper, splitting individual weight matrices across a group of GPUs that work on different rows and columns simultaneously. Connecting all these GPUs efficiently requires specialized hardware. NVLink is Nvidia's proprietary GPU-to-GPU interconnect, delivering 900 GB/s, seven times faster than PCIe. InfiniBand connects GPU nodes across servers at 400 GB/s. Together, they turn thousands of individual GPUs into one coordinated supercomputer. Nvidia dominates AI compute, but it is far from the only player. AMD's Instinct MI300X, launched in 2024, is the most credible H100 challenger. Its key differentiator is memory, 192 GB of HBM3 compared to the H100's 80 GB with 5.3 TB/s of bandwidth versus H100's 3.35. Microsoft Azure and Meta have adopted MI300X for AI workloads. Google builds its own AI chips, the Tensor Processing Unit, or TPU. The TPU v5p delivers 459 teraflops on BF16 and is used internally to train Gemini, PaLM, and other Google models. It is available to external users through Google Cloud. Amazon has Trainium for training and Inferentia for inference, available exclusively on AWS with competitive price-per-token economics. Intel's Gaudi 3 targets AI training and inference with an open software approach, Built on the PyTorch ecosystem. Despite all this competition, Nvidia retains 80 to 85% of the AI accelerator market driven by CUDA's library ecosystem. NVLink interconnects and the sheer familiarity the research community has with Nvidia tools. The gap is closing, but the moat is real. Let us bring it all together. GPUs are used for AI because the mathematics of neural networks is dominated by matrix multiplication. Every element of the output matrix can be computed independently, making the problem perfectly suited to parallel execution across thousands of cores. CPUs have 8 to 64 powerful cores optimized for sequential logic. GPUs pack in 16,000 or more cores that all fire simultaneously. Tensor cores at a second tier of acceleration, purpose-built units that perform a full 4 by 4 matrix multiply accumulate in a single clock cycle, delivering nearly 1,000 teraFLOPS on half precision AI workloads. HBM memory provides 3.35 terabytes per second of bandwidth, 22 times more than CPU DDR memory, keeping those cores fed with data. The CUDA software stack built over 17 years with cuDNN and cuBLAS is the software moat that makes Nvidia's hardware advantage compound. Real benchmarks show 80 to 160 times speed ups on GPU versus CPU. For models too large for one GPU, data, model, and tensor parallelism spread training across thousands of interconnected chips. AMD, Google, Amazon, and Intel are all building serious alternatives. But today, if you are training or running a large AI model, it almost certainly runs on an Nvidia GPU. Next in the AI infrastructure series, how AI training works. Backpropagation, gradients, and optimizers explained. Like and subscribe to Neural Monk for weekly deep dives on AI and machine learning.