How to write fast AArch64 SVE/SME Kernels with Ripple and LLVM
Sign up for Developer monthly newsletter
Join thousands of developers around the globe who receive latest news and updates from our monthly curated newsletter.
Sign up
Come for support, stay for the community
Get support from experts, connect with like-minded developers, and access exclusive virtual events.
Join Developer Discord
Writing fast AArch64 kernels often means choosing between auto-vectorization that may not deliver, and intrinsics or assembly that are tied to one ISA extension, difficult to write, verbose, and hard to maintain and tweak.
Ripple, an open-source, LLVM-based API for SPMD and loop-annotation parallelism, offers a middle path: you write plain C/C++ and tell the compiler how you want it vectorized. In this post, you’ll learn how Ripple targets AArch64—including SVE and the Scalable Matrix Extension (SME)—and walk through new GEMV and GEMM examples that get close to hand-tuned performance with a fraction of the code.
What is Ripple?
Ripple is a compiler-interpreted API to express SPMD (Single Program, Multiple Data, as with CUDA® and OpenCL®) and loop-annotation parallelism (as with OpenMP®). It has a C and C++ implementation through Clang/LLVM, and it does not modify the underlying language syntax—it complements it with parallel semantics.
The philosophy is to let you directly indicate, through the API, the key decisions on how a kernel should be vectorized, while Ripple automates the secondary decisions and bookkeeping.
The core of the API is target-independent, clear rules let you reason about how your expression of the algorithm turns into vector code.
A few design points make Ripple a good fit for SIMD hardware:
- Blocks of processing elements: You declare a block shape with ripple_set_block_shape() and index its lanes with ripple_id(). Scalar, vector and tensor computations can coexist in the same function.
- Two models, one core: Loop annotations (ripple_parallel) are syntactic sugar over SPMD and automatically split a loop into full-vector iterations and a masked epilogue.
- No new IR: Ripple does not require new kinds of LLVM instructions or types, so porting it to a new target does not require a new backend.
- Plays well with others: SIMD intrinsics can still be used in a function that includes Ripple code.
Ripple was proposed to the LLVM community in an RFC, with the long-term goal of making it part of the main LLVM toolchain. In the meantime, everything you need is open source:
- Ripple RFC: https://discourse.llvm.org/t/rfc-ripple-a-compiler-interpreted-api-to-support-spmd-and-loop-annotation-programming-for-simd-targets/88241
- Ripple compiler (LLVM fork): https://github.com/qualcomm/ripple
- Ripple User Manual: https://qualcomm.github.io/learn-ripple/abstract.html
- Ripple test suite and examples: https://github.com/qualcomm/ripple-test-suite
Ripple on AArch64: from plain C to NEON, SVE and SME
The open-source Ripple release supports SIMD code generation for AArch64. Because Ripple produces standard LLVM IR, AArch64 support builds on the existing AArch64 backend: you compile with clang, enable Ripple with -fenable-ripple, and select the architecture features you want with the usual -march flags (for example, -march=armv9.2-a+sve2+sme).
If both SVE and NEON are available, then the compiler will automatically try to select the best instructions for the architecture features.
Streaming mode and ZA state are controlled with the standard Arm C Language Extensions (ACLE) attributes, such as __arm_locally_streaming and __arm_new("za"), so Ripple kernels fit naturally into existing AArch64 code. The SME examples in this post use fixed-size Ripple blocks and assume a 512-bit streaming vector length, set with -msve-streaming-vector-bits=512.
For matrix workloads, SME adds an outer-product engine with two-dimensional ZA tile storage. Rather than relying on pattern matching to discover matrix multiplications, Ripple exposes these features through the Ripple Vector Library for AArch64 (compiler-rt/lib/ripple/aarch64 in the Ripple repository): a small set of header-only functions—for example, to zero an accumulator tile, accumulate an outer product, and read back tile slices—that you call from ordinary Ripple code.
Ripple maps those calls to the corresponding SME instructions, while the rest of the kernel (address arithmetic, loop structure, masking and epilogues) stays in plain C. Users also have the ability to add their own implementations of vector library functions.
What this means for AArch64 developers:
- Less code to write and maintain: our Ripple FP32 and INT8 GEMM kernels are 57 and 71 lines of C (including comments, formatting, and blank lines), compared with 318 and 464 lines for reference assembly-based implementations. The FP32 GEMV kernel is about 17 lines of code, compared to the ~1200 lines of assembly, which is a 98.5% reduction in code size.
- Faster turnaround for validating optimization variants: modifying ripple-based C-code to iterate over different code variants is easier than rewriting and iterating over assembly code to see which version performs the best. For example, tweaking the unroll-factor of a loop is as simple as providing a compiler hint, as with regular C/C++ code. Interleaving can be done by just increasing the ripple block size. To switch between compiling for NEON or SVE, you can just change the compiler -march flag.
- Automatic masked operation handling: scalar C/C++ code inside if conditional blocks are automatically converted to masked vector operations using ripple.
- You stay in control: you choose block shapes, data layout and which hardware features to use; the compiler handles middle-end optimizations, register allocation and instruction scheduling.
Example 1: GEMV (matrix-vector multiplication)
GEMV, or matrix-vector multiplication, is central to the decode phase of LLM inference. It is primarily used when a transformer layer multiplies an activation vector by learned weight matrices. We can represent this as Y=Wx, where x is the 1xK activation vector and W is the KxN learned weight matrix. A scalar implementation for an FP32 version of this might look like:
// Simplified illustration – see ripple-test-suite for the tuned kernel.
void gemv_f32(size_t K, size_t N, const float *x, const float *W, float *y) {
for (size_t n = 0; n < N; ++n) { // columns distributed across lanes
float acc = 0.0f;
for (size_t k = 0; k < K; ++k)
acc += x[k] * W[k * N + n]; // x[k] is broadcast to all lanes
y[n] = acc;
}
}
We know that the outer loop is data-parallel, so we can use Ripple to vectorize it by simply annotating the loop. We only need to make a few changes – include the ripple header, define the block size, and indicate which dimension of the loop needs to be parallelized. Ripple takes care of the rest, including producing efficient masking and tail code for non-full iterations.
#include <ripple.h>
#define PE_TYPE 0
// Simplified illustration – see ripple-test-suite for the tuned kernel.
void gemv_f32(size_t K, size_t N, const float *x, const float *W, float *y) {
ripple_block_t BS = ripple_set_block_shape(PE_TYPE, 16);
ripple_parallel(BS, /*dimension*/ 0);
for (size_t n = 0; n < N; ++n) { // columns distributed across lanes
float acc = 0.0f; // scalar-looking; Ripple expands it to a vector
for (size_t k = 0; k < K; ++k)
acc += x[k] * W[k * N + n]; // x[k] is broadcast to all lanes
y[n] = acc;
}
}
At a high level, this matches the tuned kernel in the ripple-test-suite. The real kernel differs in two ways: it works on one tile at a time, and it adds a bias and clamps the result using std::min and std::max.
The weight matrix is split into tiles of 32 × K F32 values. That way each accumulator vector computes 32 output elements completely before they're stored. We build the ripple-test-suite for a 512-bit streaming SVE vector length, which holds 16 F32 values. Because of that, we set the Ripple block size to 32, or two vectors, and Ripple handles the vectorization.
The resulting kernel runs two fmla instructions per iteration of its hot loop and keeps the accumulators in registers until the result is complete. Ripple's scalar expansion turns the scalar-looking float accumulator into a vector the size of the block.. Thus, we can write a developer-friendly scalar-looking implementation and easily vectorize it with Ripple.
Performance: Across square sizes from 1K to 40K, the Ripple GEMV for FP32 performed within 5% of the hand-optimized SME reference kernel. Because the kernel is written in terms of a block of lanes rather than a specific register width, the same source can be retargeted by adjusting the block size to the native vector length.
It can also be tailored to different hardware with a different vector length that may pack larger tiles, simply by changing the Ripple block size. Additionally, in comparison to assembly kernels, it can easily take scheduling models into account.
You can also generate a NEON kernel by removing the SVE2 arch flag.
Example 2: GEMM with SME outer products
GEMM dominates the prefill phase of LLM inference and many other ML and signal-processing workloads. The GEMM examples in the ripple-test-suite (linked above) cover FP32 (FP32 inputs and outputs) and INT8 (8-bit quantized inputs with 32-bit accumulation). Each kernel is ordinary C code that walks the packed left- and right-hand-side matrices, plus a few calls into the Ripple Vector Library for AArch64 that interface with SME storage. The kernels look like this:
- Zero the ZA accumulator tiles at the start of each output block.
- In the inner K loop, load a column slice of A and a row slice of B as Ripple blocks and accumulate their outer product into ZA (FMOPA for FP32, integer SMOPA for INT8).
- Read the result back slice by slice, perform any fused operations like clamping, requantization etc. and store it with plain C; Ripple generates the predicated stores needed for partial tiles at the matrix edges.
The excerpt below is the core of the FP32 kernel. It is a Ripple re-implementation of a KleidiAI SME MOPA matmul micro-kernel and uses the same packed LHS/RHS layout.
With a 512-bit streaming vector length, each 32-bit ZA tile is 16×16. The kernel uses all four tiles (za0–za3) as one 32×32 accumulator, which matches its 32×32 Ripple block (Figure 3).
The library functions are ordinary scalar declarations with vector implementations marked as Ripple vector candidates using attributes (See learn ripple docs). The vector implementations are written using intrinsics. When Ripple vectorizes a call such as outerProductAccumulate_f32 on 32-wide float blocks, it swaps in the vector implementation.
That implementation splits each operand into two 16-lane halves and issues four FMOPA instructions, one per tile. The bias is pre-loaded into the accumulator as the outer product of 1.0 and the bias row. The M edge is handled by the loop bound, and the N edge by a plain if statement that Ripple turns into a predicated store. The code below shows what the implementation looks like:
#include <ripple.h>
#include <ripple_sme_veclib.h>
#define PE_TYPE 0
#define BLOCK_LEN (2 * (512 / 8) / sizeof(float)) // 32 floats = 2 vectors
__arm_new("za") __arm_locally_streaming
void ripple_matmul_f32_f32p_f32p_sme_mopa(/* M, N, K, packed LHS/RHS, DST, ... */) {
// ... pointer casts and stride setup elided ...
ripple_block_t TileShape = ripple_set_block_shape(PE_TYPE, BLOCK_LEN, BLOCK_LEN);
ripple_block_t VecShape = ripple_set_block_shape(PE_TYPE, BLOCK_LEN);
size_t VecIdx = ripple_id(VecShape, 0); // Expands to <0, 1, 2, 3…>
for (size_t m = 0; m < M; m += BLOCK_LEN) {
for (size_t n = 0; n < N; n += BLOCK_LEN) {
size_t LHS_BeginIdx = m * K;
size_t RHS_BeginIdx = n * (K + 1); // +1 for the packed bias
zeroAccumulator_f32(TileShape);
outerProductAccumulate_f32(1.0f, RHS[RHS_BeginIdx + VecIdx]); // bias
RHS_BeginIdx += BLOCK_LEN;
// matmul+accumulate k-loop.
for (size_t k = 0; k < K; ++k) {
// Normal C loads followed by a library call.
float LHS_Vec = LHS[LHS_BeginIdx + (k * BLOCK_LEN) + VecIdx];
float RHS_Vec = RHS[RHS_BeginIdx + (k * BLOCK_LEN) + VecIdx];
outerProductAccumulate_f32(LHS_Vec, RHS_Vec); // 4 x FMOPA
}
for (size_t m_slice = 0; m_slice < std::min(M - m, BLOCK_LEN); ++m_slice) {
size_t DST_RowIdx = (m + m_slice) * dst_stride_row;
if (n + VecIdx < N) { // masking along N
float T = getAccumHorizSlice_f32(TileShape, m_slice); // block-length slice
T = max(CLAMP_MIN, min(CLAMP_MAX, T));
DST[DST_RowIdx + n + VecIdx] = T;
}
}
}
}
}
Performance: compared with hand-written SME assembly, the Ripple FP32 GEMM is within about 1–2% for large matrices and within 5–10% for very small matrices. The INT8 GEMM is within about 1–2% for large matrices; smaller sizes still have a larger gap, which we are closing with further code-generation improvements in LLVM.
Ripple for AArch64 is a work in progress. It’s important to note that the GEMM examples do not do any manual optimizations like software pipelining, interleaving, unrolling etc. like you might see in hand optimized kernels, and that doing so in C code would be a lot more trivial than doing so in assembly.
In other words, for large problems you get near assembly-level performance from a fraction of as many lines of code as an assembly implementation. The code is more flexible, easier to write, and easier to adapt to target different architecture features.
Try the examples
The GEMV and GEMM examples live in the Ripple test suite (https://github.com/qualcomm/ripple-test-suite) under the mini_apps directory. Each example checks its results against a reference implementation and reports cycle counts. To build the suite for AArch64 using a cross-compiler and testing using QEMU follow the instructions in the repo readme.
The SME kernels assume a 512-bit streaming vector length, and the library provides verify_ripple_sme_api_fixed_vscale() to check this at runtime. If you run under QEMU, the flags above configure its SME vector length to 512 bits. Please see https://qualcomm.github.io/learn-ripple/ to learn more about Ripple.
Your Turn
Ripple brings SPMD and loop-annotation programming to AArch64 through the LLVM toolchain you already use. With it, you can write compact, readable C/C++ kernels that reach close to hand-tuned SME performance.
Clone the Ripple compiler, build the test suite, and run the GEMV and GEMM examples on your AArch64 hardware or under QEMU.
Then tell us what works, what doesn’t, and which kernels you’d like to see next: open an issue or a pull request on GitHub, share your thoughts on the Ripple RFC thread on LLVM Discourse, or join our Developer Discord to share your feedback.
