4-hi HBM stacks emerge as optimal AI inference memory configuration

Full analysis

SemiAnalysis, in a piece by Dylan Patel's team titled "Long Live the Short King," argues that the shortest High Bandwidth Memory stacks (4 layers of memory chips instead of 12) are the cheapest way to serve AI models. HBM is the fast, expensive memory that sits right next to the GPU. You pay for it by the gigabyte, but inference (the work of generating tokens once a model is trained) only cares about bandwidth (how fast bits move), not capacity (how many bits you can hold). A 4-hi stack gives you the same bandwidth at a third of the cost. The idle capacity in a taller stack is money you paid for nothing.

How hard is this to undo? For anyone speccing a rack in the next two quarters, fairly hard. You commit to a memory config with your hardware maker and live with it for years. For the memory vendors, the pricing response is easy to undo and they will undo it fast. So this is really a bet on whether a cost advantage that exists today survives the market noticing it.

What's actually being decided: not "is 4-hi better math" (it is, at current prices) but "does the cheap-token advantage hold once SK Hynix, Samsung, and Micron reprice, and once you leave the frontier-lab demo and run messy multi-tenant traffic."

What sets the deadline: NVIDIA's Rubin Ultra NVL576 design choices, and the memory vendors' 2026 HBM4 roadmaps. Those will tell you whether 4-hi goes mainstream or stays a niche cost-optimized SKU.

The Skeptic

The math is right and the conclusion is oversold. 4-hi wins only if three things all hold. Your workload has to be bandwidth-bound. You have to offload the KV cache (the running memory of each user's conversation) to slower DDR memory without wrecking your latency. And HBM pricing has to stay put once the vendors see everyone asking for short stacks. Frontier labs love 4-hi because they can afford to build custom offload plumbing. A mid-tier API shop running shared GPUs cannot. And the wafer shortage is doing quiet work here: 4-hi looks optimal partly because tall stacks are simply hard to get. Cheapest-token-with-one-trick is a satisfying story that glosses over how different everyone's traffic actually is.

The Compute Pragmatist

The reframing that matters is tokens per HBM wafer. Hynix, Samsung, and Micron are constrained by wafer supply. Every wafer that goes into a 12-hi stack could have made three 4-hi stacks, which means three GPU memory systems instead of one from the same scarce silicon. In a global HBM shortage, that is a straight tripling of how many inference boxes the industry can build. Micron gains the most, given where its yields sit. The catch: these vendors sank real money into tall-stack packaging tooling and charge a fat premium for capacity. They will resist handing that margin back even as demand shifts under them.

The Safety Lens

Pushing the KV cache off the chip onto cheaper DDR memory changes what can go wrong. That cache holds user conversation context. Move it to a lower-security memory subsystem with weaker isolation, and on shared infrastructure at high concurrency, context leaking between customers stops being theoretical. Nobody is auditing memory-tiering code for side-channel leaks today. This is the sort of thing found through an incident, not a review. The engineers shipping the cost win are the same ones who will underweight the surface they just opened.

The Enterprise Buyer

If you buy inference capacity rather than build it, this decides your bill and your risk. A vendor running 4-hi with DDR offload can quote you cheaper tokens. Good. But ask two questions before you sign. What is the P99 latency under load? The offload path adds tail latency that breaks an SLA before anyone notices the average slip, and the median number on a vendor's slide will not show that. And how is my context isolated from the next tenant on that box. If the answer to either is a shrug, the cheaper token is not cheaper, it is a liability you have not priced.

Where they part ways

The Compute Pragmatist sees an industry-wide tripling of supply from a fixed wafer base. The Skeptic sees a cost edge that evaporates the moment memory vendors reprice, because the whole advantage is that HBM costs scale with gigabytes and vendors set those gigabyte prices. Both are right, and the tension is the whole story: 4-hi is optimal at today's prices, and today's prices reflect a market that hadn't yet been told to want short stacks.

The second split is Skeptic versus Buyer on who this is for. Frontier labs building custom offload infrastructure are not the same as a mid-tier provider on shared GPUs. The DDR offload path that makes 4-hi work is exactly where the Safety Lens and the Buyer both get nervous, and it is the part the whitepaper treats as solved.

What it hinges on

Two things. First, does the KV-cache-to-DDR offload hold up under real concurrency without blowing the latency tail. That is testable, and anyone speccing a rack should test it under adversarial load before committing. Second, do the memory vendors reprice capacity to close the 3x gap. If they do, the math flips and 4-hi becomes a smaller edge. The council leans toward 4-hi being real for bandwidth-bound frontier inference and overstated as a universal prescription.

Prediction: SK Hynix, Samsung, or Micron will publicly commit to volume 4-hi HBM4 production for AI inference by the time NVIDIA's Rubin Ultra ships (2027), rather than 4-hi remaining a niche cost SKU.

Confidence: Medium — wafer scarcity forces the vendors' hand, but pricing resistance could delay it.

Why: The HBM wafer shortage is the binding constraint, and 4-hi turns one scarce wafer into roughly three sellable memory systems instead of one, so a vendor that leans in sells more units even at lower margin per gigabyte. Frontier lab hardware teams are already the loudest advocates, which means the demand signal is coming from NVIDIA's biggest customers, not from a blog. The opposite outcome (vendors holding the line on tall-stack capacity premiums) is less likely because in a shortage the customer with the wafer-efficiency argument also happens to be the customer writing the biggest checks, and NVIDIA will spec what its buyers demand into Rubin Ultra. The risk to the call is timing: vendors sank money into tall-stack tooling and will defend that margin as long as they can.

Revisit by 2027-06-30: We're right if a major HBM vendor announces or ships volume 4-hi HBM4 aimed at inference, or NVIDIA's Rubin Ultra offers a 4-hi inference config. We're wrong if 4-hi stays absent from vendor roadmaps and NVIDIA ships Rubin Ultra with only 8-hi and taller options.