---
title: "KV-cache quantization"
description: "How KV-cache quantization affects context capacity, latency, and accuracy in agent workloads."
doc_version: "1.1"
last_updated: "2026-05-01"
canonical: "https://modiqo.ai/blog/kv-cache-quantization.txt"
---

# The Geometry of Forgetting: How Three Papers Cracked KV Cache Quantization

There's a problem at the heart of every LLM you use, and it's not the one most people talk about.

When a model generates text, every token it has already seen leaves behind two vectors — a key and a value — that get stashed in a buffer called the KV cache. This buffer is the model's working memory. To produce the next token, attention has to look back at every key and value already stored. So the cache grows linearly with context length, and at long contexts it dwarfs the model weights themselves. A 70B model with a 128K context window can have a KV cache larger than the parameters.

The fix sounds obvious: store those vectors in fewer bits. Instead of 16-bit floats, use 4 bits, or 2, or 1. This is quantization, and it's been the dominant compression strategy for years.

But there's a hidden tax that makes naive quantization much less effective than it looks on paper. To understand the three papers I want to walk through, you have to understand the tax first.

## The overhead nobody tells you about

Standard quantization works by chopping a floating-point range into discrete levels. If your numbers live somewhere in $[-3.7, 4.2]$ and you have 4 bits, you carve that range into 16 buckets and store the bucket index for each number. To reconstruct, you need to know where the buckets started and how wide they were. Two numbers per block: a *zero point* and a *scale*. Both stored in full precision.

Here's the catch. You can't use one global zero-point and scale for the whole tensor — outliers would crush the precision for everything else. So you group numbers into blocks (say, 64 numbers per block), and each block gets its own zero point and scale. With a block size of 64 and a 4-bit quantizer, you're paying:

$$\text{effective bits} = 4 + \frac{32 + 32}{64} = 5 \text{ bits per number}$$

That extra bit isn't a rounding error. At 2-bit quantization with a block size of 64, the overhead is *50% of your budget*. You think you're storing things in 2 bits; you're really storing them in 3.

This is the tax. Every paper I'm about to describe is, at its core, a different escape route from this tax. The escape is the same idea told three different ways: *if you can guarantee in advance that your numbers follow a known, fixed distribution, you don't need per-block parameters*. The codebook can be precomputed, baked in, and shared by everyone.

The trick to forcing that guarantee turns out to be a piece of geometry that's been sitting in linear algebra textbooks for a hundred years.

## Random rotation, and what high dimensions do to your intuitions

Take a vector $\mathbf{x}$ in $d$-dimensional space. Multiply it by a random rotation matrix $\mathbf{\Pi}$ — the kind you get by orthogonalizing a Gaussian random matrix. The vector $\mathbf{\Pi x}$ has the same length as $\mathbf{x}$, the same inner products with anything else you also rotate, but its coordinates have been completely scrambled.

Now here is the magical fact. If $\mathbf{x}$ has unit norm — which we can always arrange by scaling — then after rotation, **every coordinate of $\mathbf{\Pi x}$ follows the same fixed probability distribution, regardless of what $\mathbf{x}$ originally looked like**.

The distribution is

$$f_X(x) = \frac{\Gamma(d/2)}{\sqrt{\pi} \cdot \Gamma((d-1)/2)} (1 - x^2)^{(d-3)/2}$$

which is a scaled Beta distribution. In high dimensions, this collapses into a Gaussian $\mathcal{N}(0, 1/d)$ by the central limit theorem. The intuition is simple once you see it: a unit vector on a high-dimensional sphere, when randomly oriented, has its energy spread roughly evenly across all coordinates, with each coordinate around $\pm 1/\sqrt{d}$ in size and Gaussian fluctuations around that.

This is the load-bearing observation. Once the post-rotation distribution is fixed and known, you can design the optimal quantizer for that distribution *once*, store its codebook, and reuse it forever. No per-block parameters. No overhead.

The three papers I'm walking through all use this idea, but they exploit it in increasingly clever ways.

## QJL: one bit, zero parameters, unbiased inner products

The first paper to weaponize this in the LLM context was QJL — Quantized Johnson-Lindenstrauss. The construction is so short you can fit it on a napkin.

You sample a random Gaussian matrix $\mathbf{S} \in \mathbb{R}^{d \times d}$ with i.i.d. standard normal entries. To quantize a vector $\mathbf{x}$:

$$Q_{\mathtt{qjl}}(\mathbf{x}) = \mathtt{sign}(\mathbf{S} \cdot \mathbf{x})$$

That's it. You project $\mathbf{x}$ through the random matrix and keep only the sign of each output coordinate. One bit per dimension. To recover an estimate, you do:

$$Q_{\mathtt{qjl}}^{-1}(\mathbf{z}) = \frac{\sqrt{\pi/2}}{d} \cdot \mathbf{S}^\top \cdot \mathbf{z}$$

The reason this works is the geometry of half-spaces. Each row $\mathbf{s}_i$ of $\mathbf{S}$ defines a random hyperplane through the origin. The sign of $\mathbf{s}_i^\top \mathbf{x}$ tells you which side of the hyperplane $\mathbf{x}$ is on. That single bit, repeated $d$ times with $d$ different random hyperplanes, locates $\mathbf{x}$ on the unit sphere with surprising accuracy. The constant $\sqrt{\pi/2}$ is the rescaling factor that comes out of integrating $|\mathbf{s}^\top \mathbf{y}|$ against a Gaussian — basically, when you compute the expected dot product of a Gaussian vector with a fixed vector $\mathbf{y}$, weighted by the sign of its dot product with another vector $\mathbf{x}$, you get back $\sqrt{2/\pi} \cdot \langle \mathbf{y}, \mathbf{x}\rangle / \|\mathbf{x}\|$. The factor in the dequantizer just inverts that.

The two properties that come out of this are what make QJL useful:

1. **Unbiased.** $\mathbb{E}[\langle \mathbf{y}, Q_{\mathtt{qjl}}^{-1}(Q_{\mathtt{qjl}}(\mathbf{x}))\rangle] = \langle \mathbf{y}, \mathbf{x}\rangle$. The estimator hits the true inner product on average.
2. **Bounded variance.** The variance is at most $\frac{\pi}{2d} \|\mathbf{y}\|_2^2$, and that $1/d$ in the denominator is doing real work — in high dimensions, the noise vanishes.

For a transformer's KV cache, this is gold. Attention scores are inner products. If your inner-product estimator is unbiased and low-variance, attention with quantized keys behaves like attention with real keys, plus a small correction that averages out.

What QJL gave the field, that no prior method had, was a quantizer with **literally zero per-block overhead**. The matrix $\mathbf{S}$ is shared globally; you regenerate it from a seed. No zero points. No scales. One bit per dimension, end of story.

The catch is in that word "one." QJL is locked at exactly 1 bit per coordinate. There's no knob to turn for higher-fidelity reconstruction when you have more budget. And while inner products are preserved, the actual reconstruction error $\|\mathbf{x} - Q_{\mathtt{qjl}}^{-1}(Q_{\mathtt{qjl}}(\mathbf{x}))\|^2$ is large — you've thrown away the magnitudes entirely.

This is the gap TurboQuant set out to fill.

## TurboQuant: bridging MSE and inner product, in two stages

The TurboQuant paper starts from a sharp observation: there are *two* different distortion measures you might care about, and they pull in different directions.

The first is MSE — how close is the reconstructed vector to the original?

$$D_{\mathtt{mse}} = \mathbb{E}\left[\|\mathbf{x} - Q^{-1}(Q(\mathbf{x}))\|_2^2\right]$$

The second is inner-product error — how well does the reconstruction preserve dot products with other vectors?

$$D_{\mathtt{prod}} = \mathbb{E}\left[\left|\langle \mathbf{y}, \mathbf{x}\rangle - \langle \mathbf{y}, Q^{-1}(Q(\mathbf{x}))\rangle\right|^2\right]$$

These look related but they aren't the same. A quantizer that's optimal for MSE — one that picks the closest codebook vector to the input — turns out to be *biased* for inner products. The reason is subtle: the optimal MSE reconstruction lives at the centroid of its Voronoi cell, but centroids are pulled toward zero by the curvature of the high-dimensional sphere. The reconstructed vectors are systematically a little shorter than the originals, which means inner products are systematically underestimated. Bias is poison for downstream attention math, because errors compound across many tokens instead of cancelling.

TurboQuant's solution is to attack both problems at once with a two-stage construction. Here's the setup.

**Stage one: an MSE-optimal multi-bit quantizer.**

Apply a random rotation $\mathbf{\Pi}$ to your input vector. Now every coordinate is independently distributed according to that fixed Beta distribution we wrote down earlier. Because the coordinates are nearly independent in high dimensions, you can quantize each one separately using the optimal scalar quantizer for the known Beta distribution. This is a 1-D continuous k-means problem, and you solve it once, offline, using the Lloyd-Max algorithm:

$$\mathcal{C}(f_X, b) = \min_{c_1 \leq \ldots \leq c_{2^b}} \sum_{i=1}^{2^b} \int_{\frac{c_{i-1}+c_i}{2}}^{\frac{c_i+c_{i+1}}{2}} |x - c_i|^2 \cdot f_X(x) \, dx$$

You're partitioning the interval $[-1, 1]$ into $2^b$ buckets, with the boundaries at the midpoints between adjacent centroids — a Voronoi tessellation in 1-D. The centroids $c_i$ are chosen to minimize expected squared error against the Beta density $f_X(x)$. Since the density is fixed and known, you compute these centroids once, store them, and use them for every coordinate of every vector you ever quantize.

The MSE bound this achieves, for any unit vector $\mathbf{x}$ at $b$ bits per coordinate, is

$$D_{\mathtt{mse}}(Q_{\mathtt{mse}}) \leq \frac{\sqrt{3}\pi}{2} \cdot \frac{1}{4^b}$$

The information-theoretic lower bound — Shannon's bound, the absolute floor that no quantizer can beat — is $1/4^b$. So TurboQuant is within a factor of about $2.7$ of optimal at all bit-widths. That's about as close as anyone has gotten in a data-oblivious scheme.

**Stage two: QJL on the residual.**

After stage one, you've got a reconstruction $\hat{\mathbf{x}} = Q_{\mathtt{mse}}^{-1}(Q_{\mathtt{mse}}(\mathbf{x}))$ that's close to $\mathbf{x}$ in Euclidean distance but biased in inner product. The residual $\mathbf{x} - \hat{\mathbf{x}}$ is small in norm but it carries the bias.

So you spend one extra bit per coordinate applying QJL to the residual. The QJL estimator is unbiased by construction, so when you add it back to $\hat{\mathbf{x}}$, the combined estimator becomes unbiased too. You've borrowed QJL's debiasing property as a correction term on a multi-bit MSE quantizer.

For a target budget of $b$ bits per coordinate, you spend $b-1$ on the MSE stage and $1$ on the QJL stage. The result, for any unit $\mathbf{x}$ and any $\mathbf{y}$, has:

- $\mathbb{E}[\langle \mathbf{y}, Q_{\mathtt{prod}}^{-1}(Q_{\mathtt{prod}}(\mathbf{x}))\rangle] = \langle \mathbf{y}, \mathbf{x}\rangle$ (unbiased)
- $D_{\mathtt{prod}}(Q_{\mathtt{prod}}) \leq \frac{\sqrt{3}\pi^2 \cdot \|\mathbf{y}\|_2^2}{d} \cdot \frac{1}{4^b}$ (low variance)

The reason the second stage helps is that you're now estimating the inner product of $\mathbf{y}$ with the *residual*, not with $\mathbf{x}$ itself. The residual has much smaller norm than $\mathbf{x}$, so QJL's variance — which scales with the squared norm of what it's quantizing — drops accordingly. Each stage covers the other's weakness: the MSE stage shrinks the magnitude that QJL has to cover, and the QJL stage debiases what the MSE stage can't.

In practice, TurboQuant achieves *quality neutrality* on long-context KV cache compression at 3.5 bits per channel, and acceptable degradation at 2.5 bits. It doesn't need any calibration data. It doesn't need any per-block storage. The codebook is fixed, computed once at compile time, and shared across every layer of every model. The only per-tensor cost is the random rotation — a single matrix multiply.

This is what people mean when they say "zero overhead."

## PolarQuant: skipping Cartesian altogether

The third paper, PolarQuant, takes a completely different angle on the same problem — and arrives at something that, in retrospect, feels inevitable.

Here's the setup. After random rotation, your vector lives in a known distribution. TurboQuant quantizes its Cartesian coordinates. PolarQuant asks: what if we converted to polar coordinates first, and quantized those?

A vector in $d$ dimensions, in polar form, is one radius and $d-1$ angles. The radius is just the L2 norm, and you can store it cheaply in floating point — there's only one of them per vector. The angles are where most of the information lives, and the question is what their distribution looks like after preconditioning.

PolarQuant's first move is a recursive decomposition. Instead of using the standard hyperspherical coordinates (which don't have nice independence properties), it pairs up coordinates and converts each pair to 2-D polar form. Then it pairs up the resulting radii and converts those to 2-D polar form. After $\log_2 d$ levels of recursion, you end up with one final radius and a tree of angles organized into levels of size $1, 2, 4, \ldots, d/2$.

The level-1 angles are computed directly from coordinate pairs:

$$\psi_j^{(1)} = \tan^{-1}(\mathbf{x}_{2j} / \mathbf{x}_{2j-1})$$

The higher-level angles are computed from norms of slices of the vector:

$$\psi_j^{(\ell)} = \tan^{-1}\left( \frac{\|\mathbf{x}_{(j-1/2)2^\ell + 1 : j 2^\ell}\|_2}{\|\mathbf{x}_{(j-1)2^\ell + 1 : (j-1/2)2^\ell}\|_2} \right)$$

The reason for this strange recursion is what it does to the angle distributions when the input is preconditioned.

Apply the random Gaussian sketch matrix $\mathbf{S}$ to your input vector. The result is a multivariate Gaussian — that's the JL lemma, plus the elementary fact that linear combinations of Gaussians are Gaussian. PolarQuant then proves a beautiful joint distribution result. For an isotropic Gaussian vector in dimension $d$:

$$f_{R, \Psi_d}(r, \psi_d(\mathbf{x})) = f_R(r) \cdot \prod_{\ell=1}^{\log_2 d} f_{\Psi^{(\ell)}}(\psi^{(\ell)})$$

The radius and all angles at every level are jointly *independent*, and angles within the same level share the same distribution. The level-1 angles are uniform on $[0, 2\pi)$ — there's no special direction in a Gaussian, so the angle within a coordinate pair is uniform. But the higher-level angles, the ones derived from ratios of norms of larger slices, follow a tightly concentrated distribution:

$$f_{\Psi^{(\ell)}}(\psi) \propto \sin^{2^{\ell-1} - 1}(2\psi)$$

Look at what's happening here. As the level $\ell$ increases, that exponent $2^{\ell-1} - 1$ grows exponentially. So $\sin^{2^{\ell-1}-1}(2\psi)$ becomes an increasingly sharp spike around $2\psi = \pi/2$, which means $\psi = \pi/4$. The angles concentrate. At higher levels of the recursion, *all the angles look the same and they all sit in a vanishingly small neighborhood around $\pi/4$*.

This is why polar coordinates win. In Cartesian, every coordinate has the same Gaussian-like spread, and you're forced to use the same number of bits everywhere. In polar, the angles at higher levels of the recursion live in narrower and narrower windows, so you can quantize them with extreme precision using very few bits. You don't need a wide grid; you need a fine grid in a small neighborhood that's known in advance.

PolarQuant precomputes the optimal quantization codebook for each level's distribution and stores them all. At inference time, the procedure is: apply the rotation, do the recursive polar transform, look up each angle in its level-specific codebook, store the bucket index. The radius is stored separately in floating point — one number per vector, negligible cost.

Same payoff as TurboQuant: zero per-block parameters, no normalization step, no calibration. PolarQuant compresses the KV cache by over $4.2\times$ while staying competitive on long-context benchmarks. The geometry just happened to be more agreeable in polar form.

## What changes when you stop fighting geometry

The reason these papers are worth reading isn't that they shave a few bits off the cache. It's that they each demonstrate the same shift in framing.

The pre-2024 way to compress a KV cache: look at the data, find its outliers, choose per-block scales and zero points, fight to keep precision under a tight budget. Every block pays the parameter tax. Every model needs calibration. Every quantization scheme is one outlier away from breaking.

The QJL/TurboQuant/PolarQuant way: refuse to look at the data at all. Apply a random rotation to force the post-rotation distribution into a known form. Design the codebook offline against that known distribution. Bake it in.

You give up adapting to the data, and what you get back is enormous. No calibration. No per-block storage. The same codebook works for every layer, every head, every model trained on every dataset. The data-oblivious quantizer beats the data-dependent one because the structural overhead it eliminates is bigger than the structural advantage adaptation provides.

There's something philosophical here that's easy to miss. For decades, the assumption in compression was that adaptation = quality. Look at the data, fit to the data, win. These papers invert that. Concentration of measure in high dimensions means that random rotation regularizes everything. You don't need to know the data, because in high dimensions every reasonable distribution looks the same after a rotation. The geometry does the work for you.

The KV cache is a special case. The same idea applies anywhere you're storing a lot of high-dimensional vectors and want to access them via inner products: vector databases, retrieval systems, recommendation indices. Anywhere people have been paying the parameter tax, this rotation trick is a way out.

That's the real story across these three papers. Not the bit savings. The bit savings are downstream of something more fundamental: a reframing of what "good quantization" means when your data lives in a high-dimensional space. The new answer is that you don't earn quality by being clever about the data. You earn it by being clever about the geometry — and then the data has nowhere to hide.