A roofline model names the limiting factor before I write any code. Then I build the kernel, measure it, and check the prediction against reality, honestly, including every time the model was wrong. I'd never written an attention kernel at the metal before this. With the help of Claude's Fable 5, I built seventeen kernels.
Three eras, each with a different limiting factor. The roofline predicted the limiter before every step; the honest misses are on the record.
I pre-registered “it won’t help,” then measured it on a B300: FP4 reaches only 5.8% of its 15 PF peak, FP8 21.7% of 5 PF, both HBM-bound. The packing gate is a red herring for decode; low precision is a KV-cache capacity + accuracy lever, not a decode-speed one.
A live benchmark run, correctness first, then the speedup and the counter-free %HBM that proves the limiter.
Every number on this site traces to a measured run in docs/results.md.
Limiter location right most steps; magnitude missed 5 straight, the reason the model became two-layered.
An independent review re-ran the roofline tool and reproduced every published arithmetic-intensity number exactly, and cross-checked the external claims against current literature, the B300 constants, the decode-is-work-starved characterization, and the “complement, not beat” positioning all held.
“The misses are productive failures. Each one ruled out a wrong explanation and redirected me toward the real limiting factor.”
A pure FLOPs/bytes roofline is blind to the schedule. It named the limiter location right most steps but missed the wall-clock magnitude five straight, because L2 caching, occupancy, and launch overhead live in the schedule, not the arithmetic. Recording that honestly is the whole contribution.
“Deleting 99% of DRAM traffic made the kernel slower. Nothing was bandwidth-bound.”
“The M=128 packing advantage is a red herring for decode, the shape stays memory-bound.”
“‘Per-CTA-bound’ was a statement about the engine, not the problem. The right engine converts work to throughput.”
From per-CTA-bound to work-starvation across MHA, GQA, and MLA
Twelve attention kernels, each paired with a pre-registered two-layer roofline prediction and evaluated against measured results on T4, A100, B200, and B300. Central finding: decode's apparent per-CTA ceiling is work-starvation: the right engine converts work into throughput (0.75 → 1785 TFLOP/s on B300), but only in the high-throughput serving regime. The per-CTA-corrected layer matched the measured limiter at every decode step; pure roofline predicted the wrong regime at every single-stream step.
I complement, not beat, production kernels: the contribution is the open, prediction-vs-measured characterization, the two-layer model, and the boundary results.
Open to ML-systems / GPU-kernel / research internships. The kernels are the foundation for a from-scratch mini-vLLM.
Applied bottleneck-hunting from mechanical systems to GPU kernels: 17 CUDA kernels across 4 architectures, 8–16× over PyTorch, and characterized NVIDIA's flagship B300.