How do you run a model that is far too large for any single chip? Cerebras, the wafer-scale chipmaker challenging the GPU orthodoxy, has published a detailed technical post answering that question with unusual specificity: spread the model across 59 of its WSE-3T wafers, keep every layer whole and on-chip, and let only the intermediate results travel between them.
The worked example is Kimi K3, an open-source model from Moonshot AI with roughly 3 trillion parameters. That scale is the whole problem. A single Cerebras wafer carries 44 GB of on-chip memory, while trillion-parameter models need terabytes just for their weights. Cerebras's answer is not to slice individual layers across chips, a common source of slowdowns, but to give most layers a wafer of their own.
The arithmetic starts with compression. Stored at 4 bits each, Kimi K3's 3 trillion parameters work out to about 16 GB per layer across its 93 layers. The model mixes two kinds of layers, and they pack differently. Its 69 linear-attention layers keep only a small, fixed-size state per request, so two of them fit on one wafer. Its 24 dense layers must also store keys and values for every token in every active request, so each dense layer gets a full wafer, with the leftover memory reserved for that cache. The tally is 35 wafers for the linear layers plus 24 for the dense ones: 59 in total.
Linked together, the wafers form a pipeline. Each wafer runs its assigned layers and passes activations, the request's intermediate results, to the next stage over RDMA over Ethernet, at roughly 2 microseconds per hop. The weights themselves never move. Cerebras stresses that point, because repeatedly fetching weights from external memory is one of the main costs in conventional inference, and keeping whole layers in fast on-chip SRAM is the core of its pitch.
A pipeline like that only pays off if it stays busy. Rather than sending one request through all 59 stages and waiting, Cerebras keeps the line full, with each stage working on a different request at the same time, and stages processing small groups of requests called micro-batches. Smaller micro-batches favour speed for each request; larger ones favour total throughput. The ceiling on how many requests can be in flight at once is set by cache space on the dense-layer wafers, under the post's assumptions of a 256k average request size and an 8-bit KV cache.
Some care is needed with the headline numbers. Cerebras says the approach runs multi-trillion-parameter models at 1,000 to 2,000-plus tokens per second, and the same approach, it argues, extends to still larger models simply by adding wafers. But the company presents Kimi K3 explicitly as an illustrative example, and the post offers no independently verified benchmark tying that exact 59-wafer layout to the advertised speed. Cerebras itself notes that the more meaningful measure is total token throughput per unit of power, measured at the same output speed, and it does not publish that figure for this layout.
Even with that caveat, the post is a useful marker in the AI infrastructure race. As frontier models grow past what one accelerator, or one rack, can comfortably hold, the contest is increasingly about memory, movement and power rather than raw compute alone. Cerebras has now laid out, layer by layer, how its very different hardware would carry one of the largest open models in existence, and invited the rest of the industry to argue with the maths.