HBC vs HBM vs SRAM: Part 2 – A balanced comparison from an architect’s view

What you should know:

• AI token generation is bandwidth-bound, so reducing data movement improves performance more than adding compute.
• While on-chip static random access memory (SRAM) and high bandwidth memory (HBM) excel at specific workloads, high bandwidth compute (HBC) is architecturally advantaged because it performs computation where the data resides.
• For AI inference’s workload mix, HBC’s ability to minimize data movement increasingly improves system-level performance, power efficiency and economics.

Part one of this blog post compared three AI-inference memory strategies — SRAM, HBM and HBC. Since decode is memory-bound, HBC wins by computing near data, relaxing interface bandwidth, capacity, power and cost trade-offs instead of pitting them against each other. Part 2 presents an architect’s view and an honest comparison for a full decode solution.

The architect’s view: Reading performance off the roofline

The language computer architects use to reason about performance is the roofline. The model plots attainable performance against a workload’s arithmetic intensity — the number of operations performed for every byte moved. A workload can only reach peak compute once it does enough arithmetic per byte to hide the cost of fetching that byte. Below that point it sits on a sloped roof whose height is set entirely by bandwidth.

This is exactly where autoregressive generation lives. Producing each token reads a large volume of parameters and context but does only a small multiply-accumulate with them — an arithmetic intensity of roughly one to two operations per byte at low batch sizes. Batching lifts it somewhat, but latency targets and the memory cost of holding more context cap how far. The workload is pinned to the bandwidth roof — and no amount of extra arithmetic silicon moves it. 

Three ways to change the intensity that matters 

Here is the architect’s insight. Arithmetic intensity is not a fixed property of the model; it depends on which boundary you measure it across. The operation count is the same for everyone, so the way to raise intensity is to shrink the denominator — the bytes that must cross the expensive boundary between memory and compute. That is precisely where the three approaches differ. 

• HBM-based accelerators set the baseline. Every parameter byte is read out of the memory and across the interface each time it is used. The boundary intensity is the workload’s own — low, and firmly on the bandwidth roof. 
• On-chip SRAM approaches raise it modestly by keeping resident data off the external bus — but a large model no longer fits on one device, so the bytes reappear as traffic on the network fabric between the many devices sharing the model. 
• HBC changes the denominator directly. The bandwidth-heavy work is done where the data already sits, so the bytes crossing the boundary are only compact results — several times more useful work per byte that leaves the memory. 

Seen this way, the debate stops being about peak throughput and becomes a single architectural question: which design raises the arithmetic intensity at the boundary that limits you? The roofline says that is the only quantity that moves memory-bound performance — and HBC is the one approach that raises it structurally, rather than renting more bandwidth to feed the same wasteful crossing.

The case for an honest comparison

Previous blog posts in this HBC series argued that AI inference is limited by how well a system feeds its compute, and that HBC relaxes the trade-off between bandwidth, capacity, power and cost. That argument is sound — but told at the level of a whole system, it can flatten an important truth: no single memory approach wins everywhere.

Modern inference is not one workload. It is a mix of very different kinds of work, and the best architecture for one is not the best for another. A fair comparison has to say where each approach genuinely leads, ties or trails — and only then draw the system-level conclusion. That is the point of this piece.

One server, two very different jobs

Generating a token exercises two dominant kinds of computation, and they stress memory in opposite ways.

• Feed-forward network (FFN) layers apply the model's fixed weights. The data is static and regular — the same parameters, the same shapes, every step — which makes it a natural fit for hardware that thrives on predictable, fixed-size operands.
• Attention reads the growing history of the conversation — the accumulated context. Its footprint grows with the sequence and refuses to sit still, so it punishes any design that assumes a small, fixed working set.

Add batch size as a second axis. At low batch and tight latency — a single user waiting on a fast response — there is very little arithmetic to amortize each byte fetched, so the work is at its most memory-bound. As batch size grows, more useful work rides on each fetch, and the picture shifts again. Any claim about who “wins” is meaningless until it names the workload and the batch regime.

The scorecard: Where each approach leads, ties or trails

With those regimes named, the comparison becomes concrete. The table below is deliberately even-handed — it hands each approach the wins it has earned.

On-chip memory: A low-batch FFN specialist

Keeping the model in very fast on-die memory delivers extraordinary bandwidth at very low latency. In the regime it is built for — FFN layers at low batch size and tight latency — this is genuinely the approach to beat. It is typically the least costly per unit of throughput where it applies, it matches or exceeds HBC on energy per token, and it clears HBM-based accelerators comfortably on both. 

The limits are structural, not incidental. Sustaining that on-chip bandwidth imposes strong constraints on the compute design, and the memory works well only for static, fixed-size data — which is exactly why it excels at FFN work and does not extend to attention's growing, irregular footprint. Capacity per device is small, so a large model must be spread across many devices, carrying a large threshold investment and real system complexity. It can be thought of as a narrow specialist.

HBM-based accelerators: The versatile generalist

Surrounding a large processor with stacked HBM buys both generous capacity and good bandwidth, which is why this approach runs a broad range of workloads today. It handles FFN and attention networks, short context and long, without falling off a cliff. Its ceiling is the boundary itself: every byte the computation touches must still be read out of the memory and driven across the interface, and that bandwidth is bought through expensive, supply-constrained packaging. It is rarely the outright winner on any one regime, but it is also seldom the big loser — the safe generalist.

HBC-based accelerators: Attention and the whole system champion

Computing the most data-intensive work inside the memory changes what has to move. On attention — whose ever-growing context is precisely the traffic that dominates long-running chats and agents — doing the work in place, and sending out only compact results, is decisive. Avoiding data movement rather than merely managing it compounds the advantage at system scale, where data movement is the dominant energy cost and the binding physical limit.

Where it does not claim an outright win is the specialist's narrow home turf: on low-batch FFN work, a well-matched on-chip design can equal or better it. HBC is competitive, but not the peak. 

The weighted verdict: Why the mix decides the economics

If each approach excels in a different regime, how can there be a system-level winner at all? Because a real deployment does not run one regime — it runs a weighted mix, and the weights are moving.

As models take on longer context and more agentic, multi-turn use, attention's share of time and energy rises. The regime where on-chip memory is strongest — small, fixed, low-batch FFN — is a shrinking slice of the total; the regime where HBC is strongest is a growing one. The weighted average tilts accordingly.

So the fair statement is a careful one. On tokens per watt and tokens per dollar, the specialist can win an individual layer, but the architecture that minimizes data movement wins the realistic mix — and wins it by more as context grows. The claim is a weighted one, not a clean sweep, and it is stronger for being stated that way.

Why the future of AI depends on moving less data, not adding more compute

As AI inference workloads evolve, no single memory architecture wins every scenario. When HBC, HBM and on-chip SRAM are viewed through the lens of the roofline model, each excels in different workload regimes. While on-chip SRAM leads in low-batch FFN processing and HBM remains a versatile generalist, HBC’s ability to perform computation where data resides fundamentally reduces data movement, making it the strongest approach for attention-heavy workloads and increasingly favorable as context lengths continue to grow. 

Inference is a mix, and the mix is shifting toward exactly the work that rewards minimizing data movement. That is why the system-level conclusion survives honest accounting: not because one approach wins every regime, but because HBC wins the regimes that increasingly matter most — and wins them by more over time.

The future economics of AI infrastructure will therefore be determined not by peak compute alone, but by which architecture does the most useful work per byte moved. The next blog post in this HBC series turns from why the advantage exists to what it takes to build it at scale.

Go Deeper

If on-chip SRAM can match or beat HBC at the FFN layer level, does that mean the two approaches are roughly equivalent in practice?

No, not for a deployment that covers a broad set of AI Inference applications. On-chip SRAM wins its regime — low-batch FFN layers at tight latency — but that regime is a shrinking slice of the total workload mix as context lengthens and agentic, multi-turn use grows. The architecture that minimizes data movement wins the weighted average, and it wins by more as that average shifts toward attention's growing, irregular footprint.

What actually makes HBC structurally different from simply adding more bandwidth to an HBM-based accelerator design?

In an HBM-based accelerator design, more bandwidth still requires every parameter byte to cross the interface each time it is used — it rents more of the same expensive crossing rather than eliminating it. HBC does the bandwidth-heavy work where the data already sits, so what crosses the boundary is only compact results; the denominator shrinks, the arithmetic intensity at the boundary rises, and that is the only quantity the roofline rewards for memory-bound workloads. It is a structural change to what moves, not a larger pipe for the same movement.

What are the system-level implications of evolving AI inference workloads?

The system-level implication is already in the weighted verdict: data movement is the dominant energy cost and the binding physical limit at scale, so the architecture that reduces it structurally — rather than managing it more efficiently — holds the compounding advantage as inference workloads grow longer and more complex.