Engram Architecture Reduces HBM Demand for Large AI Models

DeepSeek's V4.1-Flash model introduces a technique called Engram, covered by chip-analysis shop SemiAnalysis (run by Dylan Patel): instead of computing common word patterns on the fly, the model looks them up in a table. That table can live on ordinary server RAM or even a hard drive, bypassing the expensive high-bandwidth memory (HBM) soldered to the GPU. SemiAnalysis says it could dent HBM demand and reshape NVIDIA's memory roadmap.

The most concrete claim: Engram cuts the GPU count needed to host a model copy from 4 to 2 on NVIDIA's B300. If that holds, it's a real rack-level saving. The problem is the memory physics. Fast on-chip memory moves data at roughly 8 terabytes per second per GPU. Ordinary server RAM does about 2 across the whole box. A hard drive is a thousand times slower still. Offloading only wins if the model reads the table in large, predictable chunks. Random, scattered lookups stall it. The 1.6x throughput figure SemiAnalysis cites is a best case on one specific chip, not a general result.

The deeper disagreement is about which memory is actually scarce. Training, not serving, is what saturates HBM, and training still eats every fast byte available. An inference trick doesn't move that demand curve, which is the demand curve NVIDIA prices against. SemiAnalysis's "reshapes HBM demand" framing assumes inference serving is a large enough slice to matter to NVIDIA's roadmap. The skeptical read is that it isn't.

There's also a quality question nobody has answered. A lookup table trained on standard pretraining data will look great on benchmarks drawn from the same distribution. It may fall apart on unusual long-tail queries. The ablation separating real generalization from overfitting hasn't been published.

The call: by NVIDIA's next data-center GPU roadmap update (GTC, expected March 2027), NVIDIA will not cut HBM specs on any announced chip and will attribute Rubin Ultra's existing memory derating from 1,024 GB to roughly 200 GB per chip to packaging and yield, not to inference-side architecture gains like Engram. Confidence is medium. Training demand still saturates HBM regardless of what happens at the serving layer, and NVIDIA has every reason to frame its roadmap decisions that way rather than concede that model architecture is routing around its most profitable component.

If you run large models yourself, the thing to check is whether a 2-GPU Engram-style replica holds your latency at your actual batch mix when those builds land in vLLM or SGLang. That test hasn't been run yet.

DeepSeek shipped a new trick called Engram in its V4.1-Flash model, and SemiAnalysis (the chip-analysis shop run by Dylan Patel) says it could dent the demand for the most expensive memory on AI chips. The idea: instead of the model working out common word patterns on the fly using its heavy compute layers, it looks them up in a big table. That table doesn't need to sit on the pricey fast memory glued to the GPU. It can live on ordinary server RAM, or even on a hard drive. Same answer quality, less of the expensive stuff.

What's actually being decided: nothing you sign this quarter. This is a read on whether the cost of serving big models is about to drop, and whether NVIDIA's pricing power over the memory market is quietly eroding. Hard to undo? Nothing here commits you to anything, so it's a watching brief, not a decision. What sets the clock: NVIDIA's Rubin Ultra chips, whose memory spec already got cut from 1,024 GB to about 200 GB per chip. If architecture tricks like this spread, that derating story gets a new chapter at NVIDIA's next roadmap update.

---

The Skeptic

One architecture, one Chinese lab, one model variant, and SemiAnalysis is already floating "could reshape HBM demand." Slow down. Lookup tables for common token patterns are old. What's new is the framing that you can shove them onto cheap memory. Fine. But the memory that runs hot in this business is training memory, and training still eats every fast byte you can feed it. Inference serving is not what's tight. And the quality-parity claim rests on benchmarks, which is exactly where a lookup table trained on the pretraining distribution would look best and fall apart on the weird long-tail query. The tidy "China innovates around chip export controls" story is too satisfying to swallow whole.

The Compute Pragmatist

The one claim with real teeth: cutting from 4 GPUs to 2 per model copy on NVIDIA's B300. That's a genuine change to how you wire a rack, and a real 2x saving on overhead. But check the memory math before you get excited. Fast on-chip memory moves data at roughly 8 terabytes a second per GPU. Ordinary server RAM does about 2 across the whole box. A hard drive is a thousand times slower again. Offloading the table only wins if the model reads from it in big, predictable chunks. If the lookups are random and scattered, it stalls. The 1.6x throughput number is a best case on one specific chip. Do not treat it as a general result.

The Researcher

The primitive is clean and worth respect. Many word pairings are statistically stable, so caching them in a table and skipping the on-the-fly reconstruction is a sensible bet. It rhymes with retrieval ideas, but applied at the weight level, which is a nice move. The open question that decides everything: does the table generalize across domains, or does it just memorize the training data and crumble on anything unusual? Nobody has shown the ablation that separates real learning from overfitting. The 4-to-2 GPU claim is the most falsifiable thing here, so that's what an independent shop should reproduce first. Until then, it's a promising result from one lab, not a law.

The Safety Lens

Push the offload story to its end. If capable inference runs on commodity servers with consumer SSDs instead of datacenter-grade GPU clusters, the number of actors who can run a big model at scale goes up. That's good for access. It also means the group of people who could run a capable model for something ugly gets larger, and red-teaming capacity has not grown to match. This is not a reason to root against the technique. It's a reason to notice that "cheaper to serve" and "harder to contain" are the same sentence read two ways.

---

Where they part ways

The real disagreement is about which memory is actually scarce. The Compute Pragmatist and the Skeptic agree that training, not serving, is what saturates the expensive memory, so an inference trick doesn't touch the demand curve that matters. SemiAnalysis's whole framing assumes serving is a big enough slice to move NVIDIA's roadmap. Both can't be right.

Second split: the Researcher and the Compute Pragmatist both zero in on the same falsifiable claim, the 4-to-2 GPU cut, but for opposite reasons. The Researcher wants it reproduced because it would validate the architecture. The Pragmatist suspects it only holds when the lookup pattern is batched and predictable, and that real traffic breaks that assumption at the batch seams.

What it hinges on

Three things. One, does quality actually hold on messy long-tail queries, not just benchmarks. Two, does the 4-to-2 GPU saving survive real, mixed traffic where lookups scatter. Three, is inference serving a big enough share of HBM demand to move NVIDIA's pricing at all, given training still eats the fast memory. The council leans skeptical on the grand "reshapes HBM demand" claim and genuinely interested in the narrow engineering result. Those are different bets and SemiAnalysis blurs them.

If you run big models yourself, the thing to check is your own provisioning tier the next time an Engram-style build lands in vLLM or SGLang: does a 2-GPU replica hold your latency at your real batch mix. Nobody has done that memory-tier-aware scheduling work yet, so it's a next-quarter test, not a today savings.

---

Prediction: By NVIDIA's next data-center GPU roadmap update (GTC, expected March 2027), NVIDIA will not cut the HBM spec on any announced chip and will attribute Rubin Ultra's memory derating to packaging and yield, not to model-architecture efficiency gains like Engram.

Confidence: Medium. Training demand still saturates HBM regardless of inference tricks.

Why: The expensive fast memory on AI chips is consumed mostly by training runs, which still use every byte available, so an architecture trick that trims inference-serving memory doesn't shrink the demand curve NVIDIA prices against. Rubin Ultra's cut from 1,024 GB to ~200 GB per chip is far more plausibly a packaging and yield story than a response to one Chinese lab's model variant, and NVIDIA has every incentive to frame it that way rather than concede that architecture is routing around its most profitable component. The opposite outcome, NVIDIA publicly acknowledging Engram-class techniques as a reason to derate HBM, would mean admitting its pricing power over memory is eroding, which no chip vendor volunteers.

Revisit by 2027-03-24: We're right if NVIDIA's next roadmap keeps or raises HBM per chip and blames prior derating on packaging/yield. We're wrong if NVIDIA cuts announced HBM specs further and cites inference-side architecture efficiency as a driver.