HBC vs HBM vs SRAM: Part 1 - Comparing each approach's ability to scale the memory wall

What you should know:

• AI inference performance is increasingly constrained by data movement and memory bandwidth, not compute.
• High bandwidth memory (HBM) and on-chip static random access memory (SRAM) each require trade-offs between bandwidth, capacity, power and cost.
• High bandwidth compute (HBC) delivers not only on memory bandwidth and capacity, but also improves performance-per-watt and performance-per-dollar by moving compute closer to data and minimizing data movement.

The first blog post in this HBC series detailed how the bottleneck in modern AI has moved — from compute to the delivery of data. This blog post takes the next step and asks the practical question that follows: if performance is now set by how well a system feeds its compute, how do the leading approaches compare?

Three strategies dominate the conversation. One keeps the entire model in fast on-chip memory. Another surrounds a powerful processor with stacked HBM, which has dominated GPU-centric systems. A third — near-memory computing, the approach Qualcomm Technologies calls high bandwidth compute, or HBC — performs the most data-hungry work near the memory itself. Each makes a different bet, and each is defined less by its peak compute than by a multi-way tension between bandwidth, capacity, power and cost.

The central finding is simple. The first two approaches are forced to trade these axes against one another. The third loosens the trade-off — because the bandwidth that matters is the bandwidth inside the memory, not the narrow interface that crosses between chips. 

The right yardstick: Measuring the thing that actually limits performance 

Generative AI inference has two very different phases. When a model first reads a prompt (known as the prefill stage), it is compute bound: there is a large amount of arithmetic to do, and it maps well onto dense matrix engines. But when the model generates a response (known as the decode stage) — one token at a time — the character of the work changes completely. Producing each token requires reading the model's parameters and its accumulated context from memory and doing comparatively little arithmetic with them.

This generation phase is overwhelmingly memory bound. The compute engines are mostly idle, waiting for operands to arrive. And for most deployed applications — chat, agents, assistants — this is where the majority of the time, energy and cost is spent.

That single fact reframes how systems should be judged. A platform's headline compute throughput describes a peak it will rarely reach on this workload. The meaningful yardstick is delivered performance on memory-bound work — and its close cousins, performance per watt and per dollar. 

The dilemma: Working around bandwidth, capacity, power and cost constraints

Every memory strategy for AI is, at its heart, an answer to a multi-way problem. A system needs enough bandwidth to feed the compute, enough capacity to hold ever-larger models and their context, and it must do both at a cost and power that make large-scale deployment viable. The difficulty is that conventional ways of improving one axis tend to worsen others.

The SRAM only bet

This school of design keeps the model in very fast memory placed directly on the processor die. The bandwidth is very high and the latency is low. But on-chip memory is expensive per bit and physically limited in how much of it fits on a single device — so a large model must be spread across a great many devices, each holding only a fraction of it.

The result is superb speed at small scale, but a steep cost in silicon, power and system complexity as models grow — because capacity was traded away to win bandwidth.

It is worth being precise about where this bet is genuinely strong, rather than quickly dismissing it. On the feed-forward network (FFN) layers of a model at low batch size and tight latency — the regime it is built for — on-chip memory is formidable. It is typically the least costly approach per unit of throughput where it applies, matching or sometimes exceeding HBC on energy per token while clearing HBM-based accelerators comfortably. For this narrow set of workloads, it wins or ties on the metrics that count.

The constraints are the other side of the same coin. Achieving that large on-chip bandwidth imposes strong constraints on the compute design, and in practice the SRAM-only solution works well only for static, fixed-size data, which confines it to low-latency, low-batch FFN work. It does not extend to attention, whose footprint grows with context and refuses to sit still. Reaching useful capacity requires spreading the model across so many devices that the threshold investment is large and the system complexity is considerable. It is a specialist that is excellent inside its limited envelope and quickly out of its depth beyond it.

The HBM bet  

The mainstream AI accelerator approach surrounds a large processor with stacks of HBM placed beside it. This delivers both generous capacity and relatively high bandwidth — which is why it dominates today. But it purchases that bandwidth through advanced, tightly supply-constrained packaging that connects the memory stacks to the processor. This complex packaging is one of the most expensive and scarce resources in the entire supply chain.

And there is a deeper limit. However wide that connection becomes, every byte the computation needs must still travel out of the memory and across the boundary into the processor. Bandwidth was bought at a premium — and the data still has to make the journey. The data journey also adds up in terms of increased energy consumption per byte transferred.

The HBC bet  

The HBC bet is that AI performance will be determined by minimizing data movement rather than maximizing compute throughput, so the winning architecture moves compute to the data instead of moving data to the compute. This allows all the axes to improve together. HBC can offer high effective memory bandwidth, low energy per token and good memory capacity at an overall low TCO for AI inference. 

Qualcomm HBC is estimated to have 6x the bandwidth per watt versus HBM for large batch sizes (Qualcomm Technologies estimates compared to competing published product specifications normalized at card-level) and 200x the capacity per watt versus SRAM for small and large batch sizes (Qualcomm Technologies estimates compared to competing published product specifications normalized at rack-level).

Part 2 of this blog delves deeper into how these approaches compare across different AI inference scenarios from low-batch FFNs to long-context attention and provides a weighted verdict of the system-level winner.

Where the model lives: Holding the model and feeding it

A useful way to see the difference between these strategies is to ask two questions of each: where does the model physically live, and how far must each byte travel to be used?

The on-chip SRAM approach holds only a sliver of the model per device, so the model is scattered across many of them and stitched together over a network. The HBM approach holds the model in far fewer devices, but every operand still crosses from memory into the processor to be used.

HBC changes the second answer. The model lives in memory, as before — but the most data-intensive operations are performed near memory, using the vast bandwidth available inside the memory. Only compact results leave the stack. The data, in effect, barely moves at all.

The system-level payoff: Why the advantage compounds at scale

The consequences of moving computation to the data are not confined to a single chip. They accumulate — and they grow more decisive as deployments get larger.

At fleet scale, energy is both the dominant operating expense and a hard physical ceiling on how much compute a data center can hold. An architecture that structurally minimizes data movement therefore does not just win a benchmark and deliver the highest tokens per second but also wins on the two metrics that increasingly govern the economics of AI: tokens per watt and tokens per dollar.

This is the sense in which the comparison is not close on the axis that matters. The alternatives optimize the movement of data. HBC largely avoids it.

Addressing a common objection: “Isn’t HBC just custom HBM?”

As the industry continues to confront the memory wall, the mainstream memory roadmap has begun to evolve in an important way. The newest HBM widens the interface between the memory stack and the processor to 2,048 bits, pushing a single stack toward roughly two terabytes per second, and — most notably — it replaces the simple layer at the bottom of the stack with a far more capable one built on an advanced logic manufacturing process. Because that base layer can now be tailored to a particular customer, the approach has come to be called custom HBM. It is a genuine and welcome step. But it is not the same thing as near-memory computing, and the difference is worth making precise.

A smarter pipe is still a pipe

Customizing the base layer lets a memory maker tune how data enters and leaves the stack — the interface protocol, buffering, signal integrity, test and reliability features, sometimes a little control logic. What it does not change is the fundamental transaction: the data still lives in the memory, and to be used it must still be read out of the stack and driven across the interface into a separate processor. The base layer makes that journey wider and cleaner. It does not make the journey unnecessary.

This shows up in a single revealing ratio — how much bandwidth moves inside the memory versus across the interface to the outside. For HBM, custom base layer or not, those two are essentially the same: the interface memory bandwidth is the internal memory bandwidth, so the ratio is about one to one. Every byte the computation touches is a byte that left the memory.

HBC changes the ratio, not just the pipe

Near-memory computing attacks the transaction itself. The most data-intensive operations are carried out within the memory module, drawing on the enormous bandwidth that exists inside a DRAM stack but has never been exposed to the outside world. Only compact results are sent across the interface. The bandwidth working inside the memory is therefore many times larger than the interface leaving it — an internal-to-external ratio measured in multiples.

That distinction has real consequences. A custom base layer inherits the same costly, supply-constrained packaging and the same energy bill for moving every operand off-chip; it simply manages that traffic more gracefully. HBC removes most of the traffic to begin with — which is why it improves capacity, bandwidth and cost together rather than trading one for another.

What to watch: How to evaluate the claims

As memory-centric approaches move from research into products, buyers and builders will be presented with competing performance claims. A few principles help separate structural advantage from headline spin.

• Insist on delivered, memory-bound performance. Ask for throughput on realistic token-generation workloads at realistic batch sizes — not peak arithmetic figures.
• Normalize by watt and by dollar. At scale, these are the true currencies. A platform that wins on raw speed but loses on efficiency will lose in the data center.
• Count the whole system. A device that holds only part of a model must be judged together with the network and the other devices needed to complete it.
• Follow the packaging. Approaches that depend on the scarcest packaging inherit its cost and supply risk, however impressive the silicon.

Why moving compute to data changes the economics of AI

The contest between memory strategies is often framed as a race for peak performance. It is better understood as a disagreement about where computation should happen relative to data.

Keep the compute central and the model distant, and system's budget is spent moving data to it. Move the compute to the data, and that budget is freed for useful work. The first two approaches, HBM and on-chip SRAM, are refinements of the former philosophy. HBC is a commitment to the latter.

On the yardstick that governs AI inference — delivered performance on memory-bound work, per watt and per dollar — that commitment is what turns a clever idea into a structural advantage. 

Part two of this blog post takes an architect's view and offers a deeper comparison of how each architecture performs across different AI inference scenarios. 

Go Deeper

Why isn't peak AI compute performance the best way to compare AI systems?

During the token generation (decode) phase of AI inference, systems spend much of their time waiting for data to arrive from memory. As a result, delivered performance on memory-bound workloads — as well as performance per watt and performance per dollar — is often a more meaningful measure than peak arithmetic throughput.

If HBM provides very high bandwidth, why is data movement still a challenge?

HBM increases bandwidth by connecting memory and processors through advanced packaging. However, every byte still has to travel from memory to the processor before it can be used, which consumes energy and creates an inherent bottleneck.

What are the trade-offs of keeping a model entirely in on-chip SRAM?

On-chip SRAM can deliver exceptional bandwidth and low latency, making it attractive for certain low-latency workloads. However, SRAM is expensive and capacity-constrained, requiring large models to be distributed across many devices, which increases system complexity and cost.