By Thorsten Meyer
Here is the number that reframes the whole topic. Kimi K3, Moonshot’s 2.8-trillion-parameter open-weight model, would need roughly 5.6 terabytes of memory to run at FP16 — the precision models were trained and shipped at until recently. The most aggressive usable quantization drops that to 594 gigabytes. And even that does not fit on a maxed-out 512GB Mac Studio.
Quantization is the lever between those numbers. It is the single most important technique for running frontier-scale models on hardware you own, and in 2026 it stopped being a simple “download the model, shrink it afterward” step. Understanding why requires understanding what quantization actually does — and what changed this year.
Quantization is the lever that turns a model needing a datacenter into one needing a workstation. In 2026 it stopped being a simple after-the-fact shrink — and Kimi K3 is the clearest example of why.
Quantization stores the same weights at coarser precision. Fewer bits per weight means less memory and bandwidth, and slightly less accuracy. The size scales almost linearly with bit-depth.
bytes ≈ parameters × bits ÷ 8. K3 figures are Unsloth-reported for the 2.8T model.“Quantized” isn’t one thing. The format decides which hardware, which loader, and which trade-offs you get.
For years, labs shipped at FP16 and the community shrank the model afterward. Kimi K3 inverts that — and it changes the advice.
- Precision reduced after the model is trained
- Exploits the slack between FP16 and 4-bit
- “Just download a smaller quant” — the old default
- K3 ships natively at MXFP4, MXFP8 activations
- The compression was spent before release
- Can’t be squeezed further uniformly — the slack is gone
If K3 can’t be squeezed uniformly, how does a 594GB 1-bit build exist? Mixed precision — most weights at 1–2 bits, the load-bearing layers upcast to 8-bit, the whole thing measured against a lossless reference.
Both distort the simple bytes-equals-params-times-bits math, and both bite hardest on the frontier models people most want to run.
The abstractions resolve into a hard boundary. Drawn on a 512GB M3 Ultra:
Choosing a quant is choosing a point on a curve — steep at the ends, flat in the middle.
Now the frontier labs are spending the compression before you download it.
What quantization is, in one paragraph
A model's weights are numbers. Train it and you get billions of them, stored by default as 16-bit floating-point values — FP16 or BF16 — each occupying two bytes. Quantization stores those same weights at lower precision: 8 bits, 4 bits, sometimes fewer. You are not removing weights; you are describing each one more coarsely, mapping a continuous range onto a small set of discrete levels. A 4-bit weight has 16 possible values instead of 65,536. The model gets smaller and faster because there is less data to store and move, and slightly less accurate because the coarser description loses information. The entire discipline is about spending that accuracy where it costs least.
The memory math is worth internalising, because it is the arithmetic every local-inference decision runs on. Bytes ≈ parameters × bits ÷ 8. A 32-billion-parameter model at 8-bit needs about 32GB for weights. The same model at 4-bit needs about 16GB. Your Qwen3 32B daily driver at 8-bit MLX sits right at that first figure — which is exactly why it runs comfortably as a daily driver and a 4-bit build would leave headroom you do not need on a 512GB machine.
Quantization loss isn't linear. From 16 bits down to 4, you give up almost nothing measurable. Below 4, uniform quantization falls off a cliff — and where you land depends entirely on whether the build was calibrated or converted blind.
Retained quality against bit-depth. The line is flat across the top, then knees hard at 4-bit. Dynamic mixed-precision bends the cliff into a slope; uniform quantization does not.
It isn't the model forgetting facts. Each weight gets mapped to the nearest available level, and the gap between the true value and the stored one is error that accumulates through every layer.
The same quantization hits different capabilities at different rates. A build that still chats fluently at 3-bit may have quietly lost its ability to reason or emit valid structured output.
The damage isn't spread across all weights. A small set carries most of it — which is precisely why calibrated, mixed-precision builds recover so much by protecting just those.
Below the safe band, loss stops being a percentage and starts being behaviour you can watch happen.
The trap isn't the loss on the benchmark. It's the loss the benchmark doesn't capture.
so the model still sounds fine long after it stops being fine.
Acer Veriton AI Mini Workstation Personal Computer GN100-UD11 Series
- Powerful AI Performance: 1 PFLOPS FP4 AI with Superchip
- Optimized for NVIDIA AI Stack: Pre-installed with NVIDIA DGX OS
- High Memory Bandwidth: Shared 128GB LPDDR5X-8533 memory
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
The format zoo, and what each is actually for
"Quantized model" is not one thing. The format determines which hardware, which loader, and which trade-offs you get.
GGUF is the format of llama.cpp — the workhorse for mixed CPU-and-GPU inference and the reason so much runs on Macs at all. Its quant tiers (Q8, Q6_K, Q4_K_M, and downward) are the lingua franca of local inference, and it offloads gracefully to system RAM when VRAM runs out. For most people running most models, a GGUF at Q4_K_M is the default sweet spot.
MLX is Apple's array framework, and MLX quantizations are compiled for Apple silicon's unified memory rather than retrofitted onto it. On an M3 Ultra this matters: an MLX build generally gives better tokens-per-second and cleaner memory behaviour than the equivalent GGUF, which is why your 32B daily driver runs MLX and not llama.cpp. The trade is a smaller ecosystem and slower support for brand-new architectures.
AWQ and GPTQ are calibration-based, GPU-first methods. They run a small dataset through the model during quantization to decide which weights tolerate coarse treatment and which do not, producing 4-bit builds that hold accuracy well on NVIDIA hardware under vLLM or TensorRT-LLM. They are the serving-cluster formats, not the Mac formats.
MXFP4 and MXFP8 — microscaling floating-point — are the 2026 story, and they are why Kimi K3 behaves differently from everything before it. These are hardware-native low-precision formats, accelerated directly on Blackwell-class GPUs, where a shared scale factor is applied across a small block of values so that 4-bit floating point retains far more dynamic range than 4-bit integer would. This is not a compression applied to a finished model. It is the precision the model was trained in.
Silicon, Power, and Intelligence (Volume-II): Model Compression and Efficient Inference (Silicon, Power, and Intelligence - A Hardware-Aware AI Engineering Series Book 2)
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
The shift that broke the old workflow: trained-in quantization
For years the pattern was fixed: labs released models at FP16, and the community quantized them afterward. The compression happened after release, as a lossy step you applied to a full-precision original.
Kimi K3 inverts that. It was trained quantization-aware and ships natively at MXFP4 — 4-bit weights with 8-bit (MXFP8) activations where numerical stability matters most. The full model is about 1.4TB of weights at its native 4-bit; the lossless reference build used for calibration is a 1.56TB 8-bit version; a hypothetical FP16 upcast would be the 5.6TB figure from the top. The practical consequence is blunt and counterintuitive: the compression that normally shrinks an open model after release was already spent before it. You cannot casually take K3 from 4-bit down to 3-bit with a uniform method and expect it to survive, because the usual slack — the gap between FP16 and 4-bit that community quants exploit — is not there. It was used during training.
This is the difference between quantization-aware training (QAT), where the model learns to be robust to low precision as it trains, and post-training quantization (PTQ), where precision is reduced after the fact. QAT models like K3 are better at their native bit-depth and far less forgiving below it. Most 2026 frontier open models are drifting toward QAT and native low-precision formats, which means "just download a smaller quant" is quietly becoming less universal advice.
vLLM and High-Performance Inference: Memory Optimization, Parallel Execution, Token Streaming, and Scalable Model Serving (Large Language Model Refinement and Inference Series)
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Dynamic quantization: why calibrated 1-bit works and blind 1-bit collapses
If K3 can't be squeezed uniformly, how does a 594GB 1-bit build exist at all?
The answer is dynamic, mixed-precision quantization — and it is the most important practical idea in the whole field right now. Instead of applying one bit-depth to every weight, dynamic quantization drops the bulk of weights to 1 or 2 bits while upcasting the layers that matter most back to 8-bit, then calibrates the whole thing against a lossless reference build so the damage is measured rather than assumed. Unsloth's K3 quants do exactly this: most of the network at 1–2 bits, the load-bearing layers preserved, validated against the 1.56TB 8-bit original.
The results tell the story of why calibration matters. Unsloth's dynamic 1-bit build lands at 594GB and holds roughly 78.9% top-1 accuracy; the 2-bit builds at 711GB and 861GB reach about 90%. Compare that to the older, cruder approach the same team warns against: community GGUFs converted blind, because nothing could run the 2.8T model to calibrate it, routinely arrived with broken expert routing and quality that fell off a cliff. A file existing is not evidence it is accurate, fast, or supported. The difference between a usable 1-bit model and a broken one is entirely in the calibration.
Edge AI Model Distillation: Optimizing Deep Learning for Mobile, IoT, and Embedded Devices Using Knowledge Distillation, TinyML, Quantization, and ... ... Intelligent IoT and TinyML Applications)
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Two wrinkles the parameter count hides: MoE and the KV cache
Two things distort the simple bytes-equals-parameters-times-bits arithmetic, and both bite hardest on exactly the frontier models people most want to run.
The first is mixture-of-experts. K3's 2.8T total parameters do not all fire per token — only about 104 billion are active on any given step, the rest sitting idle as specialised sub-networks the router selects between. This splits the hardware question in two: memory is set by the total parameter count, because every expert must be resident to be selectable; speed is set by the active count, because only those do work. It is why a 2.8T MoE can be faster per token than its size suggests and yet still demand hundreds of gigabytes to hold. Your Qwen3 235B is the same shape at smaller scale — you pay in memory for all 235B and in compute for the fraction that activates.
The second is the KV cache, the running memory of the conversation, which grows with context length and is separate from the weights entirely. At K3's million-token context the cache alone runs to tens of gigabytes. A quantization that fits the weights into your memory but forgets the cache will either swap to disk — catastrophic — or silently truncate context. Budget for it explicitly; it is the line item people forget until the model slows to a crawl at long context.
Where the line actually falls, on real hardware
The abstractions resolve into a hard boundary, and it is worth drawing it on machines that exist rather than in theory.
The governing rule, from Unsloth's own guidance, is simple: your total RAM plus VRAM should roughly equal the quantization size. Fall under it and the model streams weights from disk, and disk is thousands of times slower than memory. The concrete failure case is instructive — a 64GB M1 Max running K3 by streaming from a 2TB SSD produced around 16 seconds per token, with some configurations exceeding a minute. That is not slow; that is unusable, and it is what "it technically loads" looks like when the weights do not fit.
Now map the boundary. A 512GB M3 Ultra runs Qwen3 235B at 6-bit — roughly 176GB of weights — with comfortable room for a large KV cache and the OS. That is a frontier-class open model as a genuine local workhorse. But Kimi K3 even at dynamic 1-bit needs a ~650GB combined-memory floor, which sits above that same maxed-out 512GB machine. The honest path to running K3 locally is a two-machine cluster — a 512GB Mac Studio linked to a 128GB node — or an eight-GPU H100/B200 server. This is the real texture of "local" at the frontier in 2026: the 200-to-300-billion-parameter class is a single-workstation reality, and the multi-trillion class is still a cluster.
That boundary is exactly why a 70-to-90% local-inference target is a sorting problem, not a single decision. The 235B-class model handles the bulk of serious work on one machine; a 32B MLX build carries the high-volume daily driving; and the handful of tasks that genuinely need a 2.8T frontier model either wait for the cluster or go to a hosted API — where, for occasional frontier calls, the electricity and depreciation math usually favours renting anyway.
The practical decision, distilled
Choosing a quant is choosing a point on a curve, and the curve is steep at the ends and flat in the middle. Q8 or 8-bit is near-lossless and the right call when quality is non-negotiable and memory is not the constraint. Q6 — where your 235B runs — is the quality-first sweet spot for large models on ample memory, giving up almost nothing measurable. Q4_K_M is the universal default, the best balance of size and fidelity for most models on most hardware. Below 4-bit, you are in dynamic-quantization territory, where the question is no longer "how many bits" but "was this build calibrated against a lossless reference or converted blind" — and only the former is worth your disk space.
The one-line version: quantization is how a model that needs a datacenter becomes a model that needs a workstation. In 2026 the technique matured from a blunt after-the-fact shrink into a calibrated, mixed-precision, increasingly trained-in property of the model — and the frontier labs are now spending the compression before you ever download it.
Sources: Unsloth Kimi K3 documentation and dynamic-GGUF release (29 July 2026) for K3 quant sizes, accuracy figures, and the RAM-plus-VRAM rule of thumb; Moonshot AI technical descriptions for K3's MXFP4/MXFP8 native precision, MoE structure, and 1M context; contemporaneous hardware analyses from Kingy.ai, ExplainX, ModemGuides, and Wan27 for real-world memory floors and the streaming-from-SSD data point. Model sizes and accuracy percentages are as reported by the quantization authors and vendors; local performance varies by loader, kernel, and configuration. Fleet-specific figures reflect the author's own DojoClaw setup. Point-in-time as of 3 August 2026. Not hardware-purchasing advice.
