Lab · live demo
The roofline.
Everyone quantizes models to "make them smaller." The real reason is subtler: LLM decode is memory-bound, not compute-bound. Slide the precision, model size and bandwidth below and watch the operating point move along the roofline — and feel why fewer bits per weight buys throughput almost for free.
// ROOFLINE · LLM DECODE · ARITHMETIC INTENSITY = 2 / bytes-per-weightmemory-bound
239
tokens / sec
14.0 GB
weights in memory
1.0
FLOP / byte
You're on the memory-bound slope. Decode reads every weight once per token, so throughput is set by bandwidth ÷ model bytes — not by how many FLOPs the chip can do. Drop the precision and the weights shrink, the operating point slides right and up, and tokens/sec rise almost 1:1. That is the whole case for quantization on the edge.