By Thorsten Meyer
You size a machine for a model by doing the one calculation everyone knows. Qwen3 235B at 6-bit: 235 billion parameters times six bits, divided by eight, gives roughly 176 gigabytes of weights. You have a 512GB machine. 176 is comfortably under 512. You load it. It runs. Done.
Then, three thousand tokens into a long document, it slows to a crawl, or it falls over entirely, and you are left staring at a machine that had “plenty of headroom” wondering what happened.
What happened is that the weights are only one line item in the memory budget, and the line item that got you is the one almost nobody adds up in advance. This piece is that budget, itemized — because “will the weights fit” is the wrong question, and the right question is “will the weights plus everything else they drag along fit, at the context length I actually intend to use.” Those are very different numbers, and the gap between them is where local inference goes wrong.
You size a machine by one calculation: 235B at 6-bit = ~176GB of weights, under your 512GB, done. Then it crashes three thousand tokens into a long document. The weights are one line item. The one that got you is the one nobody adds up.
When a model runs, memory holds four distinct things, not one. Only the first is the number on the card.
235B × 6 / 8 ≈ 176GB. Same for a 10-token prompt or a 100k one. The only line item everyone budgets.It’s the only line item that’s both large and invisible at load time. The failure is deferred — which is exactly what makes it dangerous.
Itemize the budget before you trust the headroom. Four disciplines follow directly.
“Will the whole budget fit at my real context” is the one that decides if the session survives.
The four things competing for your memory
When a model runs, your memory is holding four distinct things, not one. Only the first is the number on the model card.
The weights. This is the well-understood part — the parameters themselves, sized by the count-times-bits arithmetic. For Qwen3 235B at 6-bit that is your 176GB, and it is a fixed cost: it does not change whether you send a ten-token prompt or a hundred-thousand-token one. If this were the only line item, sizing would be as easy as everyone assumes. It is not.
The KV cache. This is the one that gets people, and it deserves the most attention because it is both the largest variable cost and the one absent from every "will it fit" back-of-envelope. The KV cache is the model's working memory of the current conversation — the stored keys and values for every token in the context so far, so the model does not have to recompute the entire past on every new token. It is what makes generation fast. And it grows linearly with context length. A short prompt barely registers. A long document, a big codebase, a full agent transcript with tool calls — the cache for that can run to tens of gigabytes, and on a large model at long context it can rival or exceed a meaningful fraction of the weights themselves. The weights are fixed; the KV cache is the tide, and it comes in as your context fills.
The activations. The intermediate computations flowing through the network as it processes each token — the working scratch space of a forward pass. This is smaller and more transient than the other two, but it is real, it scales with how much you are processing at once, and it is part of the budget you cannot spend twice.
The overhead. The operating system, the inference runtime itself, the framework's own buffers, and — on the unified-memory Apple silicon a great deal of local inference actually runs on — whatever the rest of the system needs to stay alive. This is the floor you never get back, and it is larger than people expect. You do not get to spend all 512GB on the model. You get to spend what is left after the machine finishes being a machine.
high capacity RAM for AI inference
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Why the KV cache is the one that bites
I want to dwell on the KV cache, because every genuinely painful local-inference surprise I have seen traces back to it, and the reason is structural: it is the only line item that is both large and invisible at load time.
Here is the trap in its purest form. You load a model and it fits with room to spare, because at load time the context is empty and the KV cache is essentially nothing. The machine reports comfortable free memory. You start working. And as the conversation or document grows, the cache grows with it, silently eating the headroom you thought you had — until, at some context length you never explicitly chose, it crosses the line. At that point one of two bad things happens. Either the runtime starts evicting or offloading to keep going, and generation slows catastrophically as memory that should be resident starts streaming from somewhere slower. Or it simply runs out and the whole thing crashes, deep into a long task, having worked fine for the first several thousand tokens.
The cruelty of it is that the failure is deferred. A weights-only overflow fails immediately and honestly — the model will not load, you know at once, you pick a smaller quant. A KV-cache overflow fails late, after everything looked fine, on exactly the long-context work you most wanted the big model for. The load succeeding tells you almost nothing about whether the session will survive. That is why "it loaded, so it fits" is the most expensive false conclusion in local inference.
And it interacts with the mixture-of-experts shape of every current model in a way worth naming. An MoE already sets your memory floor by its total parameters — all the experts resident whether or not they fire. The KV cache stacks on top of that already-large fixed cost. So the modern frontier open model is memory-hungry twice over: hungry at load for the full expert set, and hungry again as context fills for the cache. Both are memory costs the active-parameter count never warned you about.
A-Tech 256GB Kit (8x32GB) DDR4 3200MHz PC4-25600 ECC RDIMM 2Rx4 Dual Rank 1.2V ECC Registered DIMM 288-Pin Server & Workstation RAM Memory Upgrade Modules (A-Tech Enterprise Series)
- Compatibility: For select DDR4 servers and workstations only
- Capacity: 256GB kit with 8 x 32GB modules
- Speed: Up to 3200MHz DDR4
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
The budget, done properly
So the correct sizing calculation is not weights against memory. It is all four line items against memory, at the context length you actually intend to run.
On my own fleet, this is a different discipline than the one-line arithmetic everyone starts with. For any model on the M3 Ultra, the real question is: weights (fixed, from count-times-bits), plus a KV-cache budget sized for the longest context I plan to use, plus activation headroom, plus the system floor — does that total sit under the ceiling, with margin? The 176GB of Qwen3 235B weights on a 512GB machine looks like it leaves 336GB free. It does not. It leaves 336GB minus the KV cache at my working context, minus activations, minus the OS and runtime floor. Provision for a long agent session with a big cache and that comfortable-looking margin can compress dramatically. The model still fits. The session is the thing that has to fit, and the session includes the tide.
This reframes several things I have written about. It is the memory-side complement to the quantization work: quantization shrinks the weights line item, but a lower-bit model with a huge KV cache at long context can still blow the budget — you cannot quantize your way out of a cache problem, because the cache is not the weights. It is why the total-versus-active MoE distinction matters so much: the total sets a high fixed floor that the cache then builds on. And it is why the pooled-memory direction of the hardware I described is the right one: when the cache is the swing factor, the size of your single coherent memory pool is what determines how long a context you can actually hold.
GPU-Accelerated Computing with Python 3 and CUDA: From low-level kernels to real-world applications in scientific computing and machine learning
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
The practical rules that fall out
A few things follow directly, and they are the discipline I actually use.
Size for context, not for load. The number that matters is total memory at your longest intended context, not the weights figure on the card. If you plan to run hundred-thousand-token agent sessions, budget the cache for hundred-thousand-token sessions, not for the empty prompt you load with.
Treat the KV cache as a first-class line item, always. Write it into the budget explicitly, next to the weights, before you decide a model fits. A model that fits at load and dies at depth did not fit; you just measured it at the wrong moment.
Leave real margin for the floor. The OS, runtime, and framework take more than you think, and unified memory shares that ceiling with everything else the machine does. The usable budget is meaningfully below the nameplate.
And when the budget is tight, remember there are two levers, not one. You can shrink the weights — a lower quant, per the cliff-and-loss tradeoff — or you can shrink the cache, by capping context or using a model or runtime with a more efficient cache. Reaching only for the quant lever when the cache is the problem is a category error that a properly itemized budget prevents.
Fanxiang 1TB NVMe SSD PCIe Gen4 M.2 Internal Gaming SSD for PS5, Up to 4800 MB/s, 3D NAND SLC Cache Solid State Drive Upgrade Storage for PC/Laptops S690Q
- High-Speed Data Transfer: Up to 4800 MB/s transfer speed
- Generous Storage Capacity: Available from 500GB to 4TB
- Efficient Heat Dissipation: Includes heat dissipation sticker
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
The one-line version
A model's memory cost is four things, not one: fixed weights, a KV cache that grows with context, transient activations, and a system floor that is bigger than you think. "Will the weights fit" is the question everyone asks; "will the whole budget fit at my real context length" is the question that actually decides whether the session survives. The weights are the number on the card. The KV cache is the number that gets you — invisible at load, largest when you least want a surprise. Itemize the budget before you trust the headroom, because the machine that had plenty of room to load is not the same machine that has to hold a hundred thousand tokens at hour three.
Analysis from a builder, founder, and post-labor economist running a local-first inference operation; a companion to the quantization pieces and the mixture-of-experts explainer. The four-component memory framing (weights, KV cache, activations, system overhead) and the linear growth of the KV cache with context length reflect the established behavior of transformer inference; exact figures vary by model, runtime, quantization, and configuration. Fleet-specific reasoning reflects the author's own DojoClaw setup. Point-in-time as of 10 August 2026.
