Kernel Fusion

Every kernel reads its inputs from device memory and writes its outputs back. Chain three elementwise operations and the intermediate values make two entirely pointless round trips to HBM and back, purely because you expressed them as three separate kernels.

Fusion collapses them into one kernel where the intermediates stay in registers. The arithmetic is identical; the memory traffic drops by a factor of the chain length. Since these kernels are bandwidth-bound, that factor is roughly the speedup.

Take a bias add followed by an activation, on a tensor of N elements. Unfused, that is two kernels:

Halving the traffic on a kernel whose entire cost is traffic roughly halves the time. Extend the chain to four operations and the unfused version moves 8N against the fused version's 2N, and the gap widens with every op you add.

The launch saving is real but secondary. Launches cost microseconds; the memory traffic costs far more on any tensor big enough to care about.

Elementwise chains are the ideal case. Each output element depends only on the corresponding input element, so one thread can carry a value through the whole chain in a register without ever talking to its neighbours. Bias, scale, activation, dropout mask, residual add: all of these fuse together freely.

Reductions with elementwise work fuse reasonably well: the elementwise part folds into the load phase of the reduction, so normalization layers can usually be done in one or two passes rather than five.

An elementwise epilogue onto a matmul is the highest-value case in practice, which is the next section.

A matrix multiply computes a tile of output into registers, then writes it to memory. If the next operation is a bias add and an activation, the naive pipeline writes the tile out, launches another kernel, reads it back, adds, activates, and writes again.

Fusing the epilogue means doing the bias and activation while the tile is still in registers, before it is ever written. The output is written exactly once, and the intermediate never exists in memory at all.

This is what the tuned BLAS libraries expose as epilogue options, and what CUTLASS and Composable Kernel let you write for cases the library does not cover. It is also the conceptual step toward fusing something much larger into a matmul's mainloop, which is precisely what a fused attention kernel is doing.

You mostly do not write these by hand any more. PyTorch's compiler traces the graph, finds fusable chains of elementwise and reduction operations, and generates a single kernel for each group, typically emitting Triton. That is why torch.compile gives real speedups on models full of small operations while doing very little for a model that is one enormous matmul.

The practical implication is that hand-fusing is worth your time only where the compiler cannot: unusual operations it does not recognise, fusions that cross a boundary it will not cross, or cases where you want control over the memory layout that a generated kernel will not give you.

Compute-bound kernels. Fusing an epilogue onto a large GEMM that is already at a high fraction of peak arithmetic saves a little traffic and changes almost nothing, because traffic was never the constraint.

When register pressure gets worse. Carrying more live values through a longer fused chain costs registers. Push far enough and the kernel spills, and a spill in the inner loop costs more than the memory traffic you saved.

When the intermediate is genuinely needed. An activation that the backward pass has to read again has to exist somewhere. Fusing it away means recomputing it later, which is a real trade rather than a free win.

Across a synchronization boundary. If the second operation needs values that the first operation computed in a different block, no fusion is possible without a kernel boundary, because there is no grid-wide barrier.