Arithmetic Intensity

One simple way to think about GPU kernels is in terms of two numbers: how many operations are performed, and how much data is moved. We call the ratio of these two numbers the "arithmetic intensity" of the kernel, $\alpha$$\alpha$, measured in $\frac{\text{ops}}{\text{byte}}$$\frac{\text{ops}}{\text{byte}}$.

The benefit of this idea is that the arithmetic intensity of the kernel can be compared to the arithmetic intensity of GPU hardware itself (which is defined by the ratio of its theoretical floating point throughput and memory throughput). For example, using the specs for a Titan V GPU, we have:

When ${\alpha }_{kernel}\ll {\alpha }_{gpu}$$\alpha_{kernel} \ll \alpha_{gpu}$, then the GPU is capable of processing data faster than it arrives. As a result, we expect this kernel to be bottlenecked by the memory subsystems on this GPU and say that the kernel is "memory-bound".

Similarly, when ${\alpha }_{kernel}\gg {\alpha }_{gpu}$$\alpha_{kernel} \gg \alpha_{gpu}$, then the data arrives faster than the GPU can process it. This leads to high compute utilization, but low memory throughput and we say that the kernel is "compute-bound".

 

Kernel Examples

Let's look at some common GPU calculations and estimate their arithmetic intensities. We'll assume that the all values in these kernels are 32-bit.

Kernel	Operations	Memory (bytes)	${\alpha }_{kernel}$$\alpha_{kernel}$ (op / byte)
axpy (n)	2$n$$n$	12$n$$n$	$\frac{1}{6}$$\frac{1}{6}$
dot product (n)	2$n$$n$	8$n$$n$	$\frac{1}{4}$$\frac{1}{4}$
dense mat-vec (n)	2$n^2$$n^2$	4$(n^2+2n)$$(n^2 + 2n)$	$\frac{1}{2}$$\frac{1}{2}$ + $\frac{1}{4n}$$\frac{1}{4n}$
sparse mat-vec ($\beta =\frac{nnz}{row}$$\beta = \frac{nnz}{row}$)	2$\beta n$$\beta n$	4$(2\beta +2)n$$(2 \beta + 2) n$	$\frac{1}{4}$$\frac{1}{4}$ + $\frac{\beta }{4n}$$\frac{\beta}{4 n}$
dense matix multiplication	2$n^3$$n^3$	12$n^2$$n^2$	$\frac{n}{6}$$\frac{n}{6}$

The first 4 of these kernels have arithmetic intensities that are smaller than 1, and practically independent of problem size. As a result, since the Titan V's arithmetic intensity is ${\alpha }_{gpu}\approx 15$$\alpha_{gpu} \approx 15$, those 4 kernels are expected to be strongly memory-bound on that hardware.

However, the matrix multiplication kernel has an arithmetic intensity that grows without bound as $n$$n$ gets larger. This means that when $n$$n$ is small the calculation will be memory bound, but as $n$$n$ gets bigger the arithmetic intensity eventually exceeds ${\alpha }_{gpu}$$\alpha_{gpu}$ and becomes compute-bound.

 

Interactive Simulation

One way to interpret the arithmetic intensity model is that the GPU is essentially a processor (with some compute throughput) attached to a memory bus (with some maximum data throughput). From there, kernel execution can be thought of as data moving through the memory bus, which is eventually operated on by the processor.

Below is an interactive simulation of this process. Try varying the hardware and recipe (i.e. kernel) specifications to see how the system behaves.

• What happens when ${\alpha }_{gpu}\ll {\alpha }_{recipe}$$\alpha_{gpu} \ll \alpha_{recipe}$? What about ${\alpha }_{gpu}\gg {\alpha }_{recipe}$$\alpha_{gpu} \gg \alpha_{recipe}$?

• What happens to the steady-state recipe throughput if you increase the device's memory throughput for a compute-bound kernel?