build: add CUDA_PYTHON_TOOLCHAIN override for compiler/linker selection by juenglin · Pull Request #2903 · NVIDIA/cuda-python

juenglin · 2026-09-17T18:42:21Z

Description

closes

Introduces a CUDA_PYTHON_TOOLCHAIN build-time override (mirroring CUDA_PYTHON_PARALLEL_LEVEL) that switches the C/C++ compiler and linker as a unit, without naming them explicitly. Valid values: gnu (default on Linux), llvm (clang + lld) on Linux, msvc (default on Windows). The defaults reproduce the previous build behavior; a non-default toolchain (or an externally-supplied compiler) is honored.

Motivation: experiments (see the build-wheel-riscv64-clang branch in the private repo) showed >3x compile+link speedup with clang/lld vs gcc on Linux. This PR lands the build-backend support; a follow-up makes Linux CI toolchain-agnostic so llvm can be selected there.

Naming

gnu / llvm / msvc name toolchain families (compiler + linker + binutils), not just a compiler, which matches switching compiler and linker together. llvm (clang + lld) is preferred over clang because the linker also switches to lld; clang would under-describe that. Values are case-insensitive; an invalid value or platform mismatch raises a clear error listing the platform's valid values.

Resolution rules

_resolve_toolchain_name (shared by both packages) picks the toolchain as follows:

CUDA_PYTHON_TOOLCHAIN set — it takes precedence. The compiler/linker are set to the toolchain's (clang/clang++ + -fuse-ld=lld for llvm). If an externally-set CC conflicts with it, a warning is emitted (CC only; CXX commonly defaults to c++ and is not a reliable user-intent signal) and the external CC is overridden.
CUDA_PYTHON_TOOLCHAIN unset, CC/CXX set (Linux) — the toolchain is inferred from the external compiler (CXX preferred, fall back to CC): a value containing clang selects llvm, else gnu. The inferred path uses that toolchain's flags but does not override CC/CXX, so a wrapper like CC=sccache clang survives and gets the llvm flags. Inferred llvm runs the preflight so a missing lld fails fast.
Neither set — the platform default (gnu/msvc), which does not touch CC/CXX (so the existing CC=sccache cc in CI keeps working).

Preflight

_check_toolchain_available probes the toolchain's tools on PATH and fails fast with a helpful message (tool name, install hint, and how to fall back) instead of a cryptic compile error. No-op for the platform default; for llvm it checks clang, clang++, and ld.lld.

Changes

cuda_bindings/build_hooks.py, cuda_core/build_hooks.py: add the toolchain helpers. The genuinely-shared logic (constants, name resolution/validation, env application, preflight) is in a single block delimited by # --- begin/end shared toolchain helpers --- markers, duplicated verbatim with a "keep in sync" comment — mirroring the existing _import_get_cuda_path_or_home precedent. PEP 517 build isolation forbids a shared module (the sibling package isn't installed in the isolated build env), so the block is duplicated rather than imported; a test (test_shared_toolchain_block_is_in_sync) enforces the byte-identical invariant. The per-package _resolve_toolchain() is a thin wrapper that calls the shared helpers and assembles only its own package-specific flags (cuda.bindings: c++14, -fpermissive, -O3; cuda.core: c++17, -O2).
cuda_bindings/setup.py: drops the now-redundant _is_clang strip (the inference replaces it). cuda_core/setup.py is untouched.
Docs: CUDA_PYTHON_TOOLCHAIN documented in both packages' environment_variables.rst.
Tests: toolchain resolution, preflight, and inference tests in both test_build_hooks.py; a module-level autouse fixture cleans CC/CXX/LDSHARED per test (needed because _apply_toolchain_env writes them directly to os.environ, which monkeypatch does not revert).

Build paths verified

All current build entry points keep working: cibuildwheel (build-wheel.yml, Linux CC=sccache cc, Windows MSVC), python -m build/pip wheel (test-sdist-linux.yml, test-sdist-windows.yml), coverage.yml, pixi pixi-build-python, the Cython-test build_tests.py drivers, and local rebuild-cuda-python. The default path (neither CUDA_PYTHON_TOOLCHAIN nor CC/CXX set) is unchanged. An externally-supplied CC=clang is now handled by inference (gets the llvm flag set) instead of the old build_ext-time strip.

Checklist

New or existing tests cover these changes.
The documentation is up to date with these changes.

Introduce a CUDA_PYTHON_TOOLCHAIN build-time env var (mirroring CUDA_PYTHON_PARALLEL_LEVEL) that switches the C/C++ compiler and linker as a unit. Valid values: gnu (default on Linux), llvm (clang + lld) on Linux, msvc (default on Windows). The defaults reproduce the previous build behavior exactly and do not touch CC/CXX, so an externally-set compiler (e.g. the sccache wrapper used in CI) keeps working. Each build_hooks.py gains two helpers: - _resolve_toolchain(): reads CUDA_PYTHON_TOOLCHAIN, validates it against the platform's allowed set, and returns the toolchain name plus its cc/cxx and extra_compile_args/extra_link_args. A non-default toolchain sets CC/CXX/LDSHARED so distutils' customize_compiler picks up clang/lld. - _check_toolchain_available(): a preflight that probes the toolchain's tools on PATH and fails fast with a helpful message (tool name, install hint, and how to fall back) instead of a cryptic compile error. cuda_bindings/setup.py drops the now-redundant _is_clang strip: the llvm flag set from _resolve_toolchain is clang-correct from the start (no -fpermissive, no -fno-var-tracking-assignments). No workflow changes; the default path composes with the existing hardcoded CC='sccache cc' in CI. CI toolchain selection and the CUDA_PYTHON_COMPILER_LAUNCHER companion var land in a follow-up.

copy-pr-bot · 2026-09-17T18:42:25Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

mdboom

No real objection to this as-is, but it would be nice to reduce the duplication between the two build_hooks.py scripts somehow if possible.

I think in the long run, we will want to migrate to a more robust build backend, the top contender for which is probably scikit-build-core (based on CMake). In that environment, you wouldn't hardcode the flags to run explicitly, but have the configure step figure out which ones are available etc. My worry is that if we implement this now, we would have to duplicate this somehow inside of a better build system for backward compatibility and it may not make sense there. Would it be better to migrate now?

sylvesterkaczmarek

Removing the _is_clang handling breaks the externally supplied clang path that this PR says should keep working. When CUDA_PYTHON_TOOLCHAIN is unset, _resolve_toolchain() selects gnu and keeps the GCC-only -fno-var-tracking-assignments; because the default path intentionally does not override CC/CXX, a caller with CC=clang / CXX=clang++ now gets that unsupported flag. Previously build_ext detected clang and removed it. Please preserve compiler-based flag filtering on the default path (or resolve flags from the actual external compiler) and add a regression with externally supplied clang and no toolchain override.

Extract the toolchain logic that is identical across the two build_hooks.py files (constants, name resolution/validation, env application, preflight) into a single shared block delimited by 'begin/end shared toolchain helpers' markers, duplicated verbatim with a 'keep in sync' comment. This mirrors the existing precedent set by _import_get_cuda_path_or_home. The per-package _resolve_toolchain() is now a thin wrapper that calls the shared _resolve_toolchain_name()/_apply_toolchain_env() and assembles only its own package-specific flags (cuda.bindings: c++14, -fpermissive, -O3; cuda.core: c++17, -O2). The cc/cxx compiler mapping moves into a shared _TOOLCHAIN_COMPILERS table, so it is no longer re-assigned per branch. PEP 517 build isolation forbids a shared module (the sibling package is not installed in the isolated build env), so the block is duplicated rather than imported. A new test_shared_toolchain_block_is_in_sync enforces the byte-identical invariant so drift is caught locally. No behavior change: the defaults (gnu/msvc) reproduce the previous build flags exactly, and all existing build-hooks tests still pass.

juenglin · 2026-09-18T00:23:28Z

No real objection to this as-is, but it would be nice to reduce the duplication between the two build_hooks.py scripts somehow if possible.

...

I tried extracting the shared toolchain helpers into a small distributable package (cuda-python-build-helpers, parallel to cuda-python-test-helpers), and I had it working locally. On reflection I kept the duplication, for two reasons:

PEP 517 build isolation. A build backend can only import what's in build-system.requires. cuda-python-build-helpers would have to be installable in the isolated build env for every wheel build — meaning publishing it to PyPI (or wiring a local wheelhouse into each cibuildwheel container), the same release burden cuda-pathfinder already carries. That's heavy for ~60 lines of code.
Test-environment coupling. Once build_hooks.py imports the package at module top, the build-hooks tests (which load build_hooks from source via importlib) need it installed in the test env too — so it has to be added to the test dependency groups, and pixi.lock regenerated. The self-contained build_hooks.py has none of that.

FWIW there is precedent: _import_get_cuda_path_or_home is already duplicated verbatim between the two build_hooks.py with a "keep in sync" comment.

... scikit-build-core ...
My worry is that if we implement this now, we would have to duplicate this somehow inside of a better build system for backward compatibility and it may not make sense there.

Hm, I don't think we promise backward compatibility for how the packages are built. Am I missing your point?

…clang regression When CUDA_PYTHON_TOOLCHAIN is unset on Linux, infer the toolchain from the externally-set CC/CXX (CXX preferred, fall back to CC): a value containing 'clang' selects llvm, else gnu. This fixes the regression reported by sylvesterkaczmarek: previously the default 'gnu' flag set (incl. -fno-var-tracking-assignments) reached an externally-supplied clang because the default path did not override CC/CXX and the old _is_clang strip was removed. Now clang is inferred and the llvm flag set (no gcc-only flags, -fuse-ld=lld) is used, and the external compiler is left in place, so a wrapper like CC='sccache clang' survives and gets the llvm flags. When CUDA_PYTHON_TOOLCHAIN is set it takes precedence over an externally- set CC: a mismatch warns (CC only; CXX commonly defaults to 'c++' and is not a reliable user-intent signal) and the external CC is overridden. The shared toolchain helpers remain byte-identical across the two build_hooks.py via the 'keep in sync' markers. _resolve_toolchain_name now returns an 'explicit' flag so _apply_toolchain_env only overrides CC/CXX when the toolchain was chosen explicitly (not inferred). Tests: regression tests for externally-supplied CC=clang (infer llvm, no gcc-only flags, CC survives), CC=gcc (infer gnu), and the explicit mismatch/no-mismatch cases. A module-level autouse fixture cleans CC/CXX/LDSHARED per test because _apply_toolchain_env sets them directly in os.environ, which monkeypatch does not revert.

juenglin · 2026-09-18T00:31:56Z

/ok to test f8245a5

…om dedup) The dedup refactor accidentally changed gcc's '-std=c++17' (equals) to '-std:c++17' (colon) in cuda_core/build_hooks.py's gnu and llvm branches. gcc rejects the colon form ('unrecognized command-line option'), breaking all Linux gcc builds of cuda.core (the pixi smoke build and the linux-aarch64 wheel builds). cuda.bindings was unaffected (it uses -std=c++14, correctly). Restore the equals form.

juenglin · 2026-09-18T02:22:28Z

/ok to test 8ce0a8b

juenglin added this to the cuda.core next milestone Sep 17, 2026

juenglin added the packaging Anything related to wheels or Conda packages label Sep 17, 2026

juenglin self-assigned this Sep 17, 2026

github-actions Bot added cuda.bindings Everything related to the cuda.bindings module cuda.core Everything related to the cuda.core module labels Sep 17, 2026

juenglin mentioned this pull request Sep 17, 2026

ci: make Linux wheel builds toolchain-agnostic via CUDA_PYTHON_TOOLCHAIN juenglin/cuda-python#1

Open

2 tasks

mdboom reviewed Sep 17, 2026

View reviewed changes

sylvesterkaczmarek reviewed Sep 17, 2026

View reviewed changes

juenglin added 2 commits September 17, 2026 16:19

shorten some docstrings

653ab15

juenglin marked this pull request as draft September 17, 2026 23:41

juenglin force-pushed the toolchain-override-backend branch from b0289f6 to 67ac9fc Compare September 17, 2026 23:56

github-actions Bot added the CI/CD CI/CD infrastructure label Sep 17, 2026

juenglin force-pushed the toolchain-override-backend branch from 67ac9fc to f8245a5 Compare September 18, 2026 00:28

Navigation Menu

build: add CUDA_PYTHON_TOOLCHAIN override for compiler/linker selection - #2903

build: add CUDA_PYTHON_TOOLCHAIN override for compiler/linker selection#2903
juenglin wants to merge 5 commits into
NVIDIA:mainfrom
juenglin:toolchain-override-backend

juenglin commented Sep 17, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented Sep 17, 2026

Uh oh!

mdboom left a comment

Uh oh!

sylvesterkaczmarek left a comment

Uh oh!

juenglin commented Sep 18, 2026

Uh oh!

juenglin commented Sep 18, 2026

Uh oh!

juenglin commented Sep 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

juenglin commented Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Naming

Resolution rules

Preflight

Changes

Build paths verified

Checklist

Uh oh!

copy-pr-bot Bot commented Sep 17, 2026

Uh oh!

mdboom left a comment

Choose a reason for hiding this comment

Uh oh!

sylvesterkaczmarek left a comment

Choose a reason for hiding this comment

Uh oh!

juenglin commented Sep 18, 2026

Uh oh!

juenglin commented Sep 18, 2026

Uh oh!

juenglin commented Sep 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

juenglin commented Sep 17, 2026 •

edited

Loading