Summary

Spun out of the CI investigation on #786. Gets the GitLab pipeline off end-of-life base images, and moves the shipped binaries onto a toolchain that still receives security rebuilds.

Root cause of the current failures

bullseye-security's pool files have been pruned upstream while the index still advertises them — Debian 11 left LTS on 2026-08-31. So apt-get update succeeds and apt-get install then 404s on the individual .debs:

E: Failed to fetch http://deb.debian.org/debian-security/pool/updates/main/g/glibc/libc6-dev_2.31-13+deb11u14_amd64.deb  404  Not Found
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

Reproduced outside CI, straight from Fastly with no proxy in the path, so it is neither the fabric egress gateway nor a stale mirror. The +deb11uNN versions and glibc 2.31 also identify the running image as bullseye.

Bullseye cannot be rescued in place: bullseye-security is not published on archive.debian.org under either its own name or the older bullseye/updates name, and the main archive alone leaves held broken packages. A bullseye image can no longer even be rebuilt, since its own apt-get install layer fails. Verified: the job's exact package list installs cleanly on trixie.

Why the shipped-binary image matters more than it looks

Release builds link -static-libstdc++ -static-libgcc (ConfigurationPresets.kt:121-122), so the C++ runtime and unwinder in the shipped libjavaProfiler.so come out of the build image's toolchain. The build image's support status is therefore a property of the artifact, not just of CI.

On CentOS 7 those objects can never get another security rebuild. Worse for consistency: no gcc newer than devtoolset-10 was ever built for aarch64 anywhere in the EL7 ecosystem — the dts-11 collection ships binutils 2.36.1, elfutils, dwz and its own toolchain metapackage for aarch64, but no compiler. Confirmed against CentOS vault altarch, Oracle's SCL repo and the CentOS SIG buildlogs. Oracle Linux 7 is the best-maintained EL7 derivative (live CDN, both arches, working collections) but its public repos sit at the same 7.9 package levels, with later content behind an ELS subscription.

Changes

1. Shipped binaries: AlmaLinux 8 + gcc-toolset-14, both arches — new .gitlab/base/el8/Dockerfile, replacing .gitlab/base/centos7/Dockerfile and its vault.centos.org / Rocky-mirror repository plumbing. Supported until 2029, and the toolset is published for x86_64 and aarch64 alike, so one image definition covers both architectures with one compiler version. almalinux:8 is a multi-arch OCI index (amd64, arm64, ppc64le, s390x), so a single BASE_IMAGE_GLIBC digest serves both.

2. Test/tooling images: Debian 13 trixie, both arches. With the shipped arm64 build moved off the Debian image, that image no longer sets a customer-facing floor, so it moves to current stable alongside amd64 — which is what restores apt on the arm64 sanitizer jobs.

3. The archived buster default is gone with centos7/Dockerfile. It defaulted to openjdk:11-slim-buster; buster's repos are archived (does not have a Release file) and rebuild-images.sh reaches that default whenever the base-image variable is unset, since it only log_warns.

4. .gitlab/base/Dockerfile: no more swallowed install failures. Both the apt-get install and apk add lines ended in || true, so a total package-install failure produced a "successful" image whose tools were simply missing. The apk branch was dead — Alpine/musl has its own Dockerfile.musl.

5. Names describe purpose, not a version that has now changed twice: BUILD_IMAGE_{X64,ARM64}_GLIBC, BASE_IMAGE_GLIBC, tag suffixes {x64,arm64}-glibc-base.

6. The shipped artifact's ABI floor is now asserted in CI rather than being a property of whichever image the pipeline pulled — see below.

7. ddprof-lib/src/main/cpp/glibcCompat.h holds the one .symver pin needed to keep the floor at 2.17, guarded to glibc (musl has no symbol versioning, and a GLIBC_* .symver fails to link there) and carrying its own deletion instructions for when the oldest supported host moves.

The glibc floor is held at 2.17

The floor does not move. Building on a newer glibc does not by itself raise what
the artifact requires — symbol versions are per-function — but two references
did creep above 2.17, and both are now closed:

The version to pin is per-architecture because a symbol's oldest version is the
baseline of the glibc release that introduced that port. expf@GLIBC_2.17 does
not exist on x86_64, where the symbol predates 2.17 by a decade.

Measured on the real artifact, built in this image on both architectures:

And loaded on a real EL7 host rather than inferred from symbol tables — Oracle
Linux 7.9, glibc 2.17, both architectures:

unresolved version requirements: NONE — all symbol versions resolve on glibc 2.17

So RHEL 7.9 and its derivatives keep working, and this PR has no
customer-facing consequence. Everything in it is CI-internal.

An ABI floor check, so this cannot drift again

The 2.27 regression above reached a green pipeline and was found by accident
while chasing an unrelated failure. Nothing checked what the shipped library
requires, so the floor was whatever the current base image happened to produce.

.gitlab/scripts/check-abi-floor.sh now asserts two properties of every glibc
artifact, at the end of the build job that produces it:

1. the highest versioned glibc symbol it references is at or below
   SHIPPED_GLIBC_FLOOR (.gitlab/config.env, declared next to the image pin
   that determines it), failing with the offending symbols named;
2. its DT_NEEDED set is within an allowlist — which catches a dropped
   -static-libstdc++, the more dangerous regression of the two, since the
   library would then need a C++ runtime from the host JVM's environment.

Three conditions fail loudly rather than reporting a pass: an artifact with no
versioned glibc symbols, one with no NEEDED entries, and a sort that cannot
order versions. Each would otherwise leave the check silently inoperative,
which is worse than not having it — and one of them earned its place during
development, when a parser that read only bare (non-parenthesised) symbol
versions saw a .symver-pinned artifact as having no versioned references at
all.

The checker reads through $OBJDUMP, so its tests drive it with canned output
and need no compiler or shared object: 25 assertions in
.gitlab/scripts/tests/check_abi_floor_test.sh, run by a new ci-script-tests
job alongside shellcheck. Every guard was mutation-checked — disabling it
individually turns a named assertion red.

Images rebuilt and repinned

The images were rebuilt from this branch's definitions in pipeline 137281788 and the digests are pinned here. Each was verified by pulling it:

The musl, datadog-ci and benchmark images come from the same rebuild; their definitions are unchanged on this branch.

For the record on why the pins are here rather than in the image-update PR that rebuild opened against main (#791, closed): main has no BUILD_IMAGE_*_GLIBC variables, so the two shipped-binary pins silently did nothing there, while BUILD_IMAGE_ARM64 — which main still uses for build:arm64 and stresstest:arm64 — would have moved the shipped arm64 binary onto a trixie image and its glibc 2.41 floor.

Worth knowing for the future: the pins live in this repo, and every rebuild creates new pipeline-ID-prefixed tags rather than overwriting old ones, so rebuilding affects no branch until a repin is merged.

Verified locally

• .gitlab/base/el8/Dockerfile builds for both platforms; gcc-toolset-14 provides libstdc++.a/libgcc.a, the profiler's exact link flags succeed, and jq needs no EPEL.
• ./gradlew :ddprof-lib:buildRelease succeeds inside the image on both architectures, and the measurements above come from those artifacts.
• buster's apt repos fail as described; bullseye's install fails as described; trixie's succeeds.
• The floor check passes at 2.17 on both architectures' release artifacts, and both load on Oracle Linux 7.9 (glibc 2.17) with no unresolved symbol versions.
• The expf pin links under clang on trixie — the sanitizer jobs' toolchain — on both architectures, and under musl, where the guard correctly skips it.

One coverage regression this introduces, and how to undo it

The stress tests run in the same image as the build they consume, so moving the
build host moved them too:

arm64 moves toward the floor. x64 moves away from it, and that was the only
place anything executed against glibc 2.17 — the functional matrix runs on
ubuntu-latest and alpine, and the sanitizer jobs on trixie, so nothing else
comes near it.

Net: ABI coverage at the floor is much better than before (the check above, plus
loading the artifact on a real EL7 host), while runtime coverage at the floor
went from thin to zero. For a profiler that hooks PLT entries, walks
dl_iterate_phdr, and unwinds through libc's own frames, that is the half worth
having.

It is separable from this PR, which is why it is not fixed here: build:x64
genuinely cannot stay on CentOS 7 — that is the point of the change — but
stresstest:x64 has no such constraint. It compiles nothing, consuming the
prebuilt library through -Pskip-native -Pwith-libs (stresstests.sh:36-37),
so it can run in an EL7 container while the build stays on EL8. That is
better than the old arrangement, because it would exercise the el8-built
artifact on 2.17 rather than a 2.17-built one.

Proposed follow-up, happy to open it alongside this:

1. give .stresstest_job its own image variable so the runtime environment stops
   being tied to the build environment, and point the glibc targets at an EL7
   image;
2. add a small functional job on the same image, consuming the same artifact.
   Temurin 8, 11, 17, 21 and 25 all start on glibc 2.17 (verified), so JDK
   support is not a constraint — a two- or three-JDK subset on a nightly cadence
   would cover the libc-sensitive suites without multiplying the matrix.

Worth being precise about what that would and would not test: a container gives
EL7 userspace on the host kernel, so it covers the glibc surface (linking,
PLT layout, dl_iterate_phdr, TLS, unwinding through libc) but not
perf_events, timer_create or signal delivery, which are the host kernel's.
Real EL7-kernel coverage needs a VM and is a separate question.

Also noticed, not changed

• DOCKER_IMAGE: …/images/docker:24.0.4-gbi-focal (images.yml) is Ubuntu 20.04, past standard support since April 2025. Left alone because I cannot enumerate replacement tags in that ECR registry.
• .gitlab/fuzzing/.gitlab-ci.yml:20 and .gitlab/jdk-integration/.gitlab-ci.yml:32 also apt-get install at job time, so they depend on a live archive on every run, exactly as the sanitizer jobs do. Moving those packages into the images would make the pipeline independent of archive availability.
• GitHub Actions is unaffected: ubuntu-latest/ubuntu-22.04 runners and alpine:3.23 containers are all still supported.

🤖 Generated with Claude Code