Building External Android Kernel Modules with Bazel, Kleaf and the DDK – Are You Ready?
Have you started moving your Android kernel build processes to Google's Bazel build system and Kleaf framework yet? Bazel and Kleaf are strongly recommended for building the Android kernel and its artifacts starting with Android 14, so this migration should be on your to-do list.
In the Qualcomm Innovation Center, Inc. (QUIC), we’ve taken the migration step, moving all kernel build processes to Kleaf. A big part of Kleaf is the open-source Driver Development Kit (DDK) for building external modules. In this post, I’ll describe how we used to build those modules, and how we build them now with the DDK. I’ll explain some of the pros and cons in the hope that you’ll have a smooth transition.
(This is a summary of my presentation “Driver Development Kit (DDK) and Vendor Workflow” at the Linux Plumbers Conference. Details at bottom.)
How we built external modules before the DDK
Kernel
First of all, we have a kernel team that manages our internal kernel_platform tree, with things like common build definitions and API compliance. build.sh builds the kernel and in-tree (that is, in the downstream msm-kernel tree) vendor modules:
Tech packs
For particular tech areas like display, audio, WLAN and Bluetooth, we have tech teams, or “tech packs.” Each one has its own Git repo and location in the vendor tree. Some tech teams include kernel modules in their builds, so we do out-of-tree builds for about 50 of those.
The source code for tech packs lives outside of the kernel platform tree. Besides module source code in the tech pack repo they also have a Kbuild file and a Makefile:
Also in that repo is the Android.mk file. It contains an include statement to the shared makefile Build_external_kernelmodule.mk, which is used by every tech pack. That’s how we define a target for the top-level Android Open Source Project (AOSP) build. That shared makefile in turn calls build_module.sh, which kicks off the classic make -C
Note that our builds call for a lot of custom behavior, so they were fragile. It was common that a build issue would crop up that we could fix simply by cleaning the workspace. That indicated that the build was incorrect, to the point that it often did not rebuild things properly.
How we build external modules with the DDK now
Along came Bazel, Google’s language-agnostic build system for ensuring build correctness without sacrificing speed. Google uses Bazel internally for many things, and the logical next step was to extend it to Android.
Kernel
Since we’ve migrated to Bazel and the DDK, the tree for building our kernel module and in-vendor tree modules has changed:
build.sh is gone. Instead, we have Kleaf Bazel build definitions (BUILD.bazel), and we define a kernel_build() target using Kleaf. Instead of build.sh, we run bazel build //msm-kernel:${target}_${variant}. All of that is defined in our Bazel files in repo.
Tech packs
Kbuild and Makefile are gone, and the tech packs now have a BUILD.bazel file. They also have a ddk_module() definition that loads the ddk_module rule from Kleaf where we define inputs, outputs, any dependencies and the kernel to build against.
Until we’ve converted our top-level AOSP build to use Bazel, we’ll continue using the same Android.mk and call out to Build_external_kernelmodule.mk. But instead of a raw make command, when we go down the chain to our build script, we run bazel build //vendor/qcom/opensource/techpack:my_module to build the DDK target.
One other thing to know is that Kleaf assumes all of your sources are underneath the kernel platform tree, which is not the case for us. So we’ve inserted the vendor -> ../vendor symlink to trick Kleaf into thinking that the modules’ sources are under kernel_platform.
Pros and cons of the DDK
We’re finding that the pros of switching to the DDK outnumber (and outweigh) the cons.
Pro: Limited visibility of kernel headers
This is the biggest benefit our kernel team has seen.
When a DDK module is built, Kleaf copies all of the public kernel headers and sources into a sandbox. If a module tries to use a private kernel header, Kleaf generates an error like this:
In this example, mm/slab.h is private, so Kleaf didn’t copy into the sandbox. That’s why the build can’t find it.
This feature showed us numerous places in several modules where we were using private headers and weren’t even aware of it. We’ve since changed those.
Pro: Upstream support and collaboration
Google has been very cooperative during our migration. Their engineers have helped us get Kleaf to work in our environment and get features and fixes into the DDK that our tech packs need. Any developer can submit a change to AOSP and, if the feature is valuable, Google will likely work with you to get it merged.
Pro: Less boilerplate
In some cases, we've been able to drop about half the lines of code that go into expressing our builds, largely because we can vectorize and loop with Starlark macros. The DDK enables you to reduce your build definition logic elegantly.
For example, the DDK auto-generates the Kbuild and Makefile. Although it can be tricky to port behavior from existing driver builds, we’ve found it to be a big benefit for new drivers.
Pro: Dependency graph plotting
With bazel query, you can query your directed acyclic graphs (DAG). You can ask questions of your build system like “Which modules depend on this module?” “Which headers does this module use?” and “Which kernel modules does this module depend on?” The answers are useful when you’re trying to migrate a large environment.
Pro: Build performance profiling
You can pass an extra flag to your Bazel build command to generate an output file with performance data. You load it into Chrome to get a performance profiling graph similar to this:
You then study it for any specific logjams, unnecessary dependencies and opportunities for parallelization.
Pro: Easier interdependencies and API separation
In our previous build system, it was common for tech packs to depend on symbols or headers from one another. There was no clean way to facilitate that. It was tedious to keep passing the Makefile includes and concatenate the Module.symvers.
The DDK simplifies all of that. If you want a module to provide something to other modules, you can list the public headers and the ones in your build directory that you want exposed to other modules. You can also use native Bazel visibility to define which modules are allowed to depend on a given module.
If you want module “foo” to depend on module “bar”, you’d simply add deps = ":bar" to your definition of module “foo”. When you run the build command, it will copy the public headers from the dependent modules. It will also handle all the Module.symvers symbol resolutions.
Con: A new build system
Let’s be honest: Learning a new build system is non-trivial. Few kernel developers are familiar with Bazel or Starlark (the Python-like language that Bazel uses), so that was a heavy lift for us.
Then, the different build phases of Bazel can be confusing. You might attempt building a module and get an error about a completely unrelated part of the build.
Bazel prohibits you from doing things like polluting your build with environment variables. Our kernel developers had been using environment variables for years, so there was pushback and frustration when they couldn’t do that anymore. Mind you, the guardrails are there for a good reason, but they do break some existing workflows. You have to think of them as the cost of ensuring the hermeticity of the build.
Con: Debugging
The limited visibility of kernel headers is a big advantage, as I mentioned above. But the sandbox can complicate debugging. By default, after the build is complete, Bazel cleans up the sandbox, wiping out any breadcrumbs you might rely on when debugging problems in the build. If you want to preserve them, you have to pass a special flag.
Also, the sandbox paths in debug messages are long and unwieldy. They tend to crowd your output needlessly.
Next steps
Anyway, we’re Android developers just like you, and we’re all busy just like you. We’ve switched from Make to Bazel and the DDK, and you can too. On balance, we’re glad we’ve made the transition.
Sure, there’s a learning curve with Bazel. It’s important to know the architecture before you start a large-scale migration like ours, and the Half-Day Bazel Bootcamp is a good place to start. It’s a two-part video that helped us; I strongly recommend it.
For more details and code snippets, see my presentation “Driver Development Kit (DDK) and Vendor Workflow” at Linux Plumbers Conference, with links to my slide deck and video.
Snapdragon- and Qualcomm-branded products are products of Qualcomm Technologies, Inc. and/or its subsidiaries.
