By Thorsten Meyer
Almost every chip serving AI in the world today was architected for a world that no longer exists.
The dominant silicon — the GPUs and the first wave of accelerators that run the models we all use — was conceived before the transformer became the only architecture that mattered, before inference overtook training as the dominant workload, and before “serve this to a billion people at once” was a sentence anyone said out loud. It is astonishingly good general-purpose hardware that has been retrofitted, generation after generation, onto a workload it was never designed for. That retrofit has held up remarkably well. It is also about to end.
I think we are at the start of a re-founding of AI hardware from the transistor up — and because I run a local-first inference fleet and think about this economy through a post-labor lens, I want to lay out where I believe it goes, what the real physics levers are, and who ends up holding the chokepoints when the dust settles.
Almost every chip serving AI today was architected for a world that no longer exists — training-dominant, general-purpose, conceived before the transformer became the only architecture that mattered. The next decade rebuilds silicon around inference at civilizational scale.
Strip away the hype and the gains in purpose-built inference silicon come from exactly three places. Each tells you where the roadmap goes.
Prefill and decode have opposite hardware appetites. Running both on one undifferentiated chip satisfies neither. The answer is disaggregation — a pipeline of specialized chips, each doing the part it was born for.
Today we make tokens the way the Renaissance made screws — one at a time, by hand, on general-purpose machines. The endpoint is fab-like: cost per token falls as the facility grows.
Capital believes the workload is specializing. But the physics bet and the adoption bet are not the same bet.
- Merchant inference ASICs arriving with working silicon, $1B+ in contracts, gigawatt-scale roadmaps
- Groq’s inference tech absorbed into NVIDIA (~$20B)
- Cerebras public at large valuations; custom-chip shipments projected to outgrow GPUs
- Architecture lock-in: a transformer ASIC is obsolete the day a post-transformer design wins. The GPU’s inefficiency is its insurance.
- No independent benchmarks yet — the numbers are vendor-claimed.
- NVIDIA’s moat is software. A proprietary toolchain asks customers to abandon what they know.
If token production becomes a majority of output, and national capacity is measured in agents per gigawatt, the token supply chain becomes the most strategic chokepoint on Earth.
This is the strongest argument I know for the local-first, open-weight posture: keep meaningful capability distributed — models you can run yourself, on hardware you own, close enough to the frontier to matter. Scale pulls one way; sovereignty and resilience pull the other. Both futures get built at once.
It’s who owns the factories when it does, and whether the answer is “many.”
Inference is the market now
Start with the demand, because it reframes everything downstream. Training was the story of 2023 and 2024 — the giant clusters, the hundred-thousand-GPU runs. But training as a share of total AI compute is on a path to asymptote toward something small. The workload that actually scales without limit is inference: serving models to users and, increasingly, to agents.
And the scale of that is genuinely hard to hold in your head. Only a small fraction of the planet uses these models seriously today. The demand curve does not bend when that goes from a few million people to a few hundred million, and then to a few hundred million agents running concurrently, around the clock, each consuming tokens at a rate no human ever could. The dominant industry framing has quietly flipped: the general-purpose GPU running a general-purpose software stack is now widely argued to be structurally inefficient for the workload that consumes the majority of AI compute spending. When the workload becomes the market, the hardware gets rebuilt around the workload. That is the whole thesis.
The metric shifts too. We are leaving the era where the only question was raw speed and entering one where the question is throughput at a fixed level of interactivity — how many users, or agents, you can serve at once while holding a usable per-token speed. Tokens per watt. Tokens per dollar. Agents per megawatt. Those are the numbers the next decade optimizes, and almost none of them were front-of-mind when the current silicon was designed.
Distributed AI Systems: A practical guide to building scalable training, inference, and serving systems for production AI
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
The three levers that actually move
If you strip the hype away, the gains in purpose-built inference hardware come from three places, and understanding them tells you where the roadmap goes.
The first is thermal, and it is the one people underrate. You cannot simply cram more floating-point units onto a chip and get more performance. Model FLOPS utilization — the fraction of a chip's advertised peak you actually achieve on real work — sits somewhere around twenty to fifty percent on today's GPUs, and it is capped by heat. Push utilization up and more transistors switch, more power draws, the chip runs hotter, and it throttles its own clock to avoid cooking itself. Dennard scaling is the governing physics: power rises with the square of voltage. Halve the voltage and you quarter the power. So the real unlock is not more flops; it is running at dramatically lower voltage so you can afford more flops without melting. Other chip industries — Bitcoin miners run at a fraction of GPU voltage — prove it is physically possible. The next generation of inference silicon will be low-voltage silicon, and everything else follows from solving thermals first.
The second is memory and the interconnect, and this is the lever I feel most directly. Decode — the token-by-token generation phase — is a memory game, not a math game. What matters is how fast you can move weights and the model's working memory around. And the dirty secret of today's clusters is that the bottleneck is not bandwidth on a single chip; it is the latency between chips. Going point-to-point on current top-end hardware can take on the order of four thousand nanoseconds. The speed-of-light floor is a few nanoseconds. That gap — three orders of magnitude — is where the future lives. The direction is treating an entire scale-up cluster as a single pooled memory: dozens, then thousands, then tens of thousands of chips that talk to each other nearly as fast as they talk to themselves. I watch a small-scale version of this on my own fleet every day, running large mixture-of-experts models across unified-memory Apple silicon, where the size of the coherent memory pool is the thing that decides what runs at all. The frontier is that same idea at a thousand times the scale.
The third is specialization, and it is the most philosophically interesting. The entire semiconductor stack is built on buffer — every layer, from the design tools to the power modules to the standard cells, is made general-purpose so it can serve datacenter, edge, and embedded alike. The moment you commit to one workload, you can break assumptions the general-purpose world cannot. A trivial example with enormous downstream consequences: the default timing tools assume a chip might run at zero degrees Celsius. No AI datacenter has ice in it. Drop that assumption and a cascade of optimizations opens up. Do that across the whole stack and the twenty-percents and two-exes compound into something an order of magnitude better — but only for the workload you specialized on.
AI Hand Warmers Rechargeable 2 Pack, 20Hrs Long Safe Heat, AI Smart Chips Electric Hand Warmers, Portable Pocket Heater, Gifts for Christmas, Outdoor, Golf, Hunting, Camping Accessories
- 2-in-1 Portable Hand Warmers: Dual-use design for sharing or solo use
- AI Temperature Control Chip: Precise 3-level heat regulation with smart feedback
- Long-Lasting Warmth: Provides up to 20 hours of heat
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Prefill, decode, and the coming disaggregation
The sharpest expression of specialization is that inference is not one workload but two, and eventually more.
Prefill reads in the prompt and gets the model's memory into the right state; decode uses that state to generate tokens. These have opposite hardware appetites — prefill is compute-bound and wants raw flops, decode is memory-bound and wants bandwidth and low interconnect latency. Running both on the same undifferentiated chip is a compromise that satisfies neither. The emerging answer is disaggregation: run prefill on one pool of hardware, hand the model's working memory to a separate decode pool, and let each be built for its job. Push it further and decode itself splits — attention on one kind of chip, the feedforward network on another, potentially an SRAM-based accelerator that has no business touching a general workload but is unbeatable at that specific one. The future datacenter is not a rack of identical chips. It is a pipeline of specialized ones, each doing the part it was born for.
dedicated AI inference accelerators
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
The destination is the token factory
Here is where I think all of this is heading, and it is the frame I find most clarifying.
Right now we make tokens the way the Renaissance made screws — one at a time, on general-purpose machines, essentially by hand. There are no economies of scale yet. That is a temporary condition. Consider that money does not buy you a better iPhone; a billionaire and an average person carry the same device, because manufacturing at scale collapsed the cost and flattened the quality curve. Token production is not there yet, but it is going there. The endpoint is the token factory: a single, monolithic, purpose-built facility — think the forty-billion-dollar fab, then the hundred-billion-dollar one, then, inevitably, the trillion-dollar one — serving one or a handful of models to an enormous population, with the cost per token falling as the thing gets bigger rather than rising. The unit economics of a fab, applied to thought.
That is what "production is the product" really means, and it is why the companies that win this will be the ones that can manufacture token capacity at gigawatt scale, not merely design a clever chip. Availability becomes the killer feature. A chip that is ten times better but exists in the thousands loses to one that is merely good and exists in the millions.
AI Systems Performance Engineering: Optimizing Model Training and Inference Workloads with GPUs, CUDA, and PyTorch
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
The re-founding is already visible — and so is the bear case
You can see the re-founding happening in the market structure right now, and I want to be even-handed about it, because there is real risk woven through the opportunity.
The specialists have arrived. Etched came out of a long stealth in mid-2026 with working first silicon on a four-nanometer process, over a billion dollars in signed contracts before shipping a production rack, and a stated march toward gigawatt-scale in 2027 — the first credibly-funded merchant challenger built only for transformer inference. Groq's inference technology was absorbed into NVIDIA in a deal reported around twenty billion dollars. Cerebras went public and briefly touched enormous valuations. Custom AI-chip shipments are projected to grow faster than GPUs. The signal is unmistakable: capital believes the workload is specializing.
But I evaluate these claims critically, and three cautions belong in any honest map. First, architecture lock-in is an existential risk that GPUs do not carry. A chip etched for the transformer attention pattern is obsolete the day a genuinely better architecture — a state-space model, something post-transformer — displaces it. The GPU's inefficiency is also its insurance policy: it can run whatever comes next. Betting the silicon on transformers persisting is a real bet, not a certainty. Second, none of the merchant inference claims are independently benchmarked in production yet — the performance numbers are, for now, the vendors' own, exactly the caution I apply to every model launch. Third, NVIDIA's moat was never only the chip. It is the software ecosystem — the tooling everyone already knows — and a specialized chip with a proprietary toolchain and no compatibility asks customers to abandon that, which is a steeper hill than the raw performance numbers suggest. The specialists are right about the physics. Whether they are right about the adoption is what the next year answers.
The layer I actually care about: who owns the token factory
Everything above is, for me, prologue to a question that sits at the intersection of my work — and it is a sovereignty question, not a spec-sheet one.
If token production is on a path to become a majority of economic output, and if national capacity comes to be measured in something like agents per gigawatt, then the supply chain that produces tokens becomes the most strategically important chokepoint on Earth. And that chokepoint is narrow: a small number of leading-edge fabs, a handful of high-bandwidth-memory suppliers, and the ability to energize gigawatts of power. Whoever controls that controls the means of producing intelligence itself. This is why I care so much about where the silicon is made and who can make it — and why Europe's position, largely as a consumer of this stack rather than a producer of it, is a genuine strategic vulnerability rather than an abstract policy concern.
It is also the strongest argument I know for the local-first, open-weight posture I build around. A world where thinking is produced only in a few trillion-dollar token factories owned by a few entities is efficient, and it is fragile, and it concentrates a civilizational capability in very few hands. The counterweight is not to out-build them — that is not on the table for most of us — but to keep meaningful capability distributed: models you can run yourself, on hardware you own, at a quality that stays close enough to the frontier to matter. The economies of scale will pull one way. Sovereignty, resilience, and plain independence pull the other, and both futures get built at once.
Where this lands
The hardware is being re-founded from the transistor up, and the shape of it is now legible: low-voltage silicon that solves thermals before adding flops, clusters pooled into single vast memories across near-speed-of-light interconnects, disaggregated pipelines of specialized chips, all bending toward the token factory and its fab-like economies of scale. The physics is not in serious doubt. The architecture bet and the adoption bet are.
The question that actually matters is not whether inference silicon specializes — it will — but who owns the factories when it does, and whether enough capability stays distributed that the answer is "many" rather than "three." That is the part of this future I am watching, and the part worth building against.
This is analysis and opinion from the perspective of a local-first builder and post-labor economist. Company-specific facts — Etched's mid-2026 stealth exit, working A0 silicon, ~$800M raised and $1B+ in contracts; the NVIDIA–Groq transaction; Cerebras's public listing — are drawn from contemporaneous coverage (June–August 2026) and remain, on the performance side, largely vendor-claimed and not independently benchmarked. Technical framing (MFU ranges, Dennard scaling, chip-to-chip latency, prefill/decode disaggregation) reflects well-established engineering physics. Point-in-time as of 5 August 2026. Not investment or hardware-purchasing advice.
