Skip to content

feat(os): publish the guest kernel build tree for out-of-tree modules - #1226

Open
kvinwang wants to merge 8 commits into
nextfrom
feat/os-kernel-headers
Open

kvinwang wants to merge 8 commits into
nextfrom
feat/os-kernel-headers

Conversation

@kvinwang

@kvinwang kvinwang commented Sep 17, 2026

Copy link
Copy Markdown
Collaborator

Problem

The guest image ships kernel modules but no kernel build tree, so an application that needs its own .ko has no supported way to build one against the exact kernel that boots in the CVM (Module.symvers, generated headers, config, matching compiler).

Fix

  • Export. The mkosi kernel component runs upstream's scripts/package/install-extmod-build via make run-command (the same path make bindeb-pkg uses for linux-headers), adds .config, and drops scripts/mod/devicetable-offsets.s, a -fverbose-asm intermediate that embeds the absolute build path and breaks reproducibility.
  • Release artifact. mkosi.finalize moves the tree out of the rootfs before measurement; make-release-artifacts.sh archives it as the new optional artifacts.kernel_devel manifest entry, and assemble.sh publishes dstack-<version>-kernel-devel.tar.gz. It is not in sha256sum.txt, so os_image_hash is unaffected. parity.json forbids it in the prod rootfs, and repro-check compares the archive byte for byte.
  • Builder image. os/image/kernel-builder/ builds ghcr.io/dstack-tee/dstack-kernel-builder:<version>: the tree plus a compiler from the pinned Debian snapshot, with KDIR set. The Dockerfile fails if the tree's kernel.release does not match. CI builds it on dispatch and pushes it on release tags.
FROM ghcr.io/dstack-tee/dstack-kernel-builder:<version> AS kmod
COPY mymod/ /src
RUN make -C $KDIR M=/src modules

Verification (local)

  • ./os/mkosi/build.sh --archive image succeeds; produces dstack-0.6.0-kernel-devel.tar.gz (13 MB, 7756 entries, contains .config and Module.symvers, no devicetable-offsets.s). sha256sum.txt and the rootfs contain no kernel-devel entries.
  • os/image/kernel-builder/build.sh builds the image; a hello-world module builds in it as a non-root user with vermagic: 6.18.40-dstack SMP preempt mod_unload.
  • ./os/mkosi/build.sh lint and prek pass.

Not run locally: repro-check and a module load test inside a CVM.

Replace source-text assertions with artifact behavior tests, including the tar SIGPIPE regression. Share the builder command between CI and local usage, narrow intermediate cleanup, and clarify smoke-test limits.
Drop the fixture tests, the kernel-devel.json metadata and the standalone
export script; inline the export into the kernel component. Trim the
workflow reporting, the builder image and the duplicated docs, and stop
tagging the builder image with the kernel release, which is not unique
across dstack versions.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant