SM vs CU: The Core Compute Block

A GPU die is one compute block repeated many times. Nvidia calls that block a Streaming Multiprocessor, AMD calls it a Compute Unit, and they are the same idea: a self-contained bundle of arithmetic lanes, schedulers, registers and on-chip scratchpad that can run many threads at once. Your kernel's thread blocks get handed out to these, one block per unit, and a GPU scales by having more of them.

They are not, however, directly comparable units, and the marketing numbers built on top of them are comparable even less.

Strip away the vendor names and both contain the same categories of thing:

The interesting differences are in the proportions, not the parts list.

Read the last row together with the row above it. AMD ships more units with fewer lanes each, Nvidia ships fewer units with more lanes each, and the totals land closer together than either number suggests on its own.

Both vendors publish a headline core count. Nvidia multiplies SMs by FP32 lanes and calls the result CUDA cores. AMD multiplies CUs by 64 and calls the result stream processors. Neither number is wrong, and comparing them across vendors is still meaningless, for three reasons.

First, a "core" here is a lane, not anything resembling a CPU core. It has no independent program counter and cannot run a different instruction from its neighbours. Second, the clock differs, so equal lane counts do not imply equal throughput. Third, and most importantly, the figure ignores the matrix units entirely, and on any modern AI workload those supply the overwhelming majority of the arithmetic.

The numbers that survive the comparison are the boring ones: achieved FLOPs at the precision you actually use, and memory bandwidth. Prefer them.

Take the two datacenter parts this site uses most, an Nvidia H100 SXM and an AMD MI300X, and follow the headline numbers down from the unit count.

The core counts differ by 15%. The FP32 figures differ by a factor of 2.4, and the matrix figures by 1.3. The core count predicts neither.

The H100 row is the plain formula from FLOPs and TFLOPS: 16,896 lanes × 2 FLOPs per fused multiply-add × 1.98 GHz ≈ 67 TFLOPS. The same formula gives the MI300X only 82 TFLOPS. The other factor of two comes from packed FP32 instructions, which make each lane do two FP32 operations at once. That doubling is real, but only for code the compiler can pack, which is why a spec sheet is a ceiling and not a forecast. The matrix row depends on neither lanes nor cores; it is set by the Tensor Cores and Matrix Cores, which the "core" count leaves out entirely.

Because a thread block is assigned to exactly one unit and stays there, everything that block needs has to be carved out of that unit's fixed supply. Registers, scratchpad and thread slots are all finite per unit, and whichever runs out first caps how many blocks can be resident at once.

That cap is what occupancy measures, and occupancy is how the machine hides memory latency: the more resident groups, the more likely one is ready to issue while others wait. This is the point where a design decision inside your kernel, say using a few more registers per thread, quietly reduces the number of blocks per unit and costs you performance somewhere else entirely.

It is also why a kernel that launches ten blocks on a two-hundred-unit GPU is slow no matter how good the kernel is. Most of the machine is simply not participating.

The companion sample prints the per-unit budget of your own GPU, then asks the runtime how many 256-thread blocks of one kernel fit on a single unit as the block's scratchpad request grows. The core of it is one call:

int blocksPerUnit = 0;
hipOccupancyMaxActiveBlocksPerMultiprocessor(&blocksPerUnit, scratchpadKernel,
                                             256, scratchpadBytes);
            

On a small Nvidia card, built as CUDA, it reported 96 KB of scratchpad per SM and this:

The first rows cost nothing, because a different resource was already the limit. Past 8 KB, every extra kilobyte of scratchpad costs resident blocks, and the drop comes in steps rather than smoothly. On an MI300X the budget is different, 64 KB of LDS per CU, but the shape of the table is the same.

Both vendors report the unit count and its resource limits directly.

• Never compare GPUs by core count. Compare achieved FLOPs at the precision you use, and memory bandwidth.
• Read the SM/CU count at runtime (multiProcessorCount) and size the grid to several blocks per unit.
• Know your per-unit budget: registers, scratchpad and thread slots. The sample prints all three.
• Before adding scratchpad or registers to a block, check what it does to blocks per unit with the occupancy API or a profiler.
• Do not hardcode 32 or 64 as the group size; read warpSize.