CPU vs GPU vs TPU

Why can the same workload run one way on a CPU, very differently on a GPU, and sometimes faster still on a TPU? Because each chip is optimized for a different type of computation. CPU handles general-purpose tasks. GPU handles large amounts of math in parallel. TPUs are optimized for specific machine learning workloads. That's why the same problem can behave very differently on each one. A CPU is a general-purpose processor. It is built for flexibility. It handles web servers, databases, operating systems, and application logic. This is a kind of work where every step can be different. Read a request, check authentication, look up data, apply business rules, return a response. That is a lot of branching and decision-making. CPUs are good at that. They have a small number of powerful cores designed to handle many different tasks efficiently. Now compare that with workloads that repeat the same math over and over across large amounts of data. That could be graphics rendering. It could be scientific computing. It could be video processing. It could be machine learning. That's where GPUs come in. A GPU packs many more arithmetic units than a CPU, which makes it great for high-throughput parallel work. For example, in graphics rendering, many pixels can be computed independently. In scientific computing, the same numerical operation may be applied across a huge data set. In machine learning, the same math is repeated across large batches of inputs. That is a great fit for a GPU. To understand why, we need to talk about matrix multiplication. Today's video is sponsored by Snowflake. If your data lives in five different systems, your pipeline keep breaking, and your team spend more time fixing infrastructure than building product, this is for you. Snowflake's AI data cloud brings everything together in one unified platform. You can work across data, apps, and teams, spin up workspaces and notebooks, and build AI-powered solutions out of the box. And with native support for Apache Iceberg, there's no vendor lock-in. That's why thousands of enterprises trust Snowflake to move faster with their data. Start building a data lakehouse on Apache Iceberg in minutes. Get Snowflake's free 30-day trial using the link in the description. A matrix is just a grid of numbers. For example, a 2 by 3 matrix has two rows and three columns. Matrix multiplication takes two compatible grids of numbers and combines them to form a new grid. You do that by multiplying numbers and adding them together, row by row and column by column. That sounds simple, but when the matrices are huge, it becomes a massive amount of repetitive math. And that math shows up constantly in machine learning. When a neural network processes input, a lot of what it's doing under the hood is matrix multiplication. The input is one big set of numbers. The model weights are another big set of numbers. The model combines them through matrix multiplication to produce a next set of outputs. Then it repeats the process across many layers. That is one reason GPUs are so useful for AI. They are very good at doing the same operation many times in parallel. Now, let's talk about tensors. A tensor sounds intimidating, but it's basically a generalization of some familiar concepts. A single number is a scalar. A list of numbers is a vector. A grid of numbers is a matrix. In machine learning, you can think of a tensor as a higher dimensional arrays of numbers. For example, a color image can be represented as a tensor. It has a height, a width, and color channels. If you process many images together in one batch, that becomes an even larger tensor. That brings us to TPUs. TPU stands for Tensor Processing Unit. Unlike CPUs, which are general-purpose, and GPUs, which are highly parallel but still fairly general. TPUs are more specialized. They're designed specifically for machine learning workloads, especially tensor heavy workloads like training and inference for large neural networks. For example, if you're serving a large language model, part of the workload may involve huge tensor operations during inference. If you're training a transformer model, the workload is even more dominated by matrix multiplications on giant tensors. That is the kind of task where TPUs can shine. So, why not use TPU for everything? Because specialization is a trade-off. The more specialized the hardware is, the less flexible it becomes. A CPU can do almost anything reasonably well. A GPU is excellent for many parallel workloads. A TPU can be extremely efficient for machine learning workloads that fit its design. In practice, modern systems often use different chips for different parts of the workload. CPU often handle control flow and orchestration. TPUs often handle large-scale parallel compute like rendering and model training. TPU are most useful when the workload is heavily centered around tensor operations. Performance comes from matching the workload to the right architecture.