Anatomy of a GPU: Die, Package and Memory

What people call "a GPU" is really an assembly of three parts, and it pays to keep them separate in your head because each one imposes a different ceiling:

A kernel is bound by whichever of the three runs out first, and which one that is depends on the work as much as the card. A large matrix multiply runs out of arithmetic; generating text one token at a time runs out of bandwidth. Two rows of the spec sheet are enough to tell which, and a section below shows the arithmetic.

Two words are about to do a lot of work on this page, so they are worth pinning down. A consumer card is the kind that goes in a desktop or workstation — GeForce on Nvidia, Radeon on AMD — designed for graphics first and sold at retail. A datacenter card — H100, MI300 and their successors — is designed for compute in a server rack: several times the memory, a dedicated high-speed link to its neighbours, usually no display output at all, and an order of magnitude more expensive. All three parts above are built differently depending on which of the two you are holding, and the next three sections are largely about how.

The die is one piece of silicon tiled with dozens to hundreds of identical compute blocks — Streaming Multiprocessors on Nvidia, Compute Units on AMD — plus a shared L2 cache, the memory controllers, and the scheduling hardware that hands work to the blocks. The design is deliberately repetitive: the same block stamped out many times, which is what lets a vendor ship a bigger and a smaller version of the same architecture by simply including more or fewer of them, and what lets a partially defective die be sold as a lower-tier part with some blocks disabled.

Two consequences worth carrying forward. First, your work only scales if you give it enough independent pieces to occupy every block — a kernel that fills ten blocks on a hundred-block die wastes ninety percent of the chip no matter how well written it is. Second, "how many cores does it have" is a nearly meaningless question across vendors, because Nvidia and AMD count the insides of these blocks differently.

GPU memory achieves its bandwidth by being wide and close, not by being clocked absurdly high. Two packaging styles do this differently:

Either way the bus between die and memory is far wider and far shorter than the path from a CPU to its DIMM slots, which is the entire reason GPU memory bandwidth is measured in terabytes per second while a desktop CPU's is measured in tens of gigabytes. You pay for that with capacity: a GPU has one to two orders of magnitude less memory than the server it is plugged into, and running out of it is the single most common practical wall in deep learning.

From the operating system's point of view the whole card is just a PCIe device — the driver enumerates it much like an NVMe SSD or a network card, only with a much larger BAR, the memory-mapped window through which the host can reach device memory. That link is dramatically slower than the on-package memory bus: tens of gigabytes per second against thousands. Any design that copies data back and forth per iteration is bound by this number, which is why the standing advice is to move data onto the device once and keep it there.

Datacenter cards add a second, much faster network for GPU-to-GPU traffic — NVLink with NVSwitch on Nvidia, Infinity Fabric / xGMI on AMD — so that peer traffic in multi-GPU training does not have to be routed through the host at PCIe speed. Consumer cards generally do not have this, which is one of the real functional differences between a gaming card and a datacenter one.

Four rows do most of the work, and each has a catch:

Peak TFLOPS and memory bandwidth together answer the question every optimization starts from: will this kernel run out of arithmetic first, or out of bytes? It takes three steps.

1. The card's ridge point. Divide peak FLOPS by memory bandwidth, using the precision you actually run and the dense figure, not the sparse one. For an MI300X at BF16 that is 1307 TFLOPS ÷ 5.3 TB/s ≈ 247 FLOPs per byte. It is how much arithmetic a kernel has to do for every byte it moves before memory stops being the bottleneck.

2. The kernel's arithmetic intensity. Count the FLOPs the kernel performs and the bytes it reads from and writes to device memory, and divide. This is a property of the work, not of the card.

3. Compare. The best the kernel can do is min(peak FLOPS, bandwidth × intensity). Below the ridge point the second term is the smaller one and memory is the limit. Above it, the die is.

So neither half of the chip is "the" bottleneck in general. Training and the prefill phase of inference are dominated by large matrix multiplies and sit on the compute side. Token-by-token decoding, elementwise ops and normalization sit on the memory side. Precision moves the line as well: the same MI300X peaks at about 163 TFLOPS for plain FP32 vector math, a ridge point near 31, so FP32 code turns compute-bound far sooner.

Treat the results as ceilings, not predictions. Real kernels reach only part of either peak, so for planning, redo the division with the bandwidth and FLOPS you actually measure. The byte counts also assume every operand crosses the memory bus exactly once. A poorly tiled matmul re-reads its inputs and lands far below N/3. Plotting the ceiling across every intensity gives the roofline chart, which The Roofline Model covers along with how to read it.

Each of the three parts is visible from the command line.