feat: eBPF codegen toolchain for the uSID TC-BPF program by privateip · Pull Request #282 · datum-cloud/galactic

privateip · 2026-08-05T01:03:21Z

Summary

Second PR in the eBPF uSID datapath stack (on top of #281, which adds the usid.c program this generates from).

Teaches task build to regenerate the TC-BPF program's Go bindings via go generate (a new build:ebpf step ahead of build:binaries) and the CNI container image to do the same at image-build time — clang/llvm stays a build-time-only dependency and never ships in the runtime image.

Also adds a root-privileged rerun of the unit test suite in CI (test-unit-root): the existing test-unit job runs as an unprivileged user, so every test gated on requireRoot(t) — including the BPF_PROG_TEST_RUN cases in usid_test.go — skips instead of running. And a step that diffs the regenerated eBPF artifacts against the committed ones, so usid.c and its compiled output can't silently drift apart.

Note: test-e2e's dependency list isn't updated to include test-unit-root yet — that lands in the deploy/e2e PR later in this stack, once there's an eBPF datapath for e2e to actually depend on.

Test plan

go build ./...
task --list (Taskfile parses, build:ebpf task registered)
CI itself exercises the new jobs once opened (no clang available in this local sandbox to run build:ebpf directly)

Part of the eBPF uSID datapath cutover stack (base: #281).

ecv · 2026-08-05T02:44:38Z

Same disclosure as on #281 — ecv reviewed this through an AI shell, apologies. Questions, not verdicts.

The drift check pins correctness to whatever clang ubuntu-latest happens to ship. apt-get install clang is unversioned, and BPF object bytes move with the compiler (BTF, debug info, section layout), so the next runner-image clang bump makes git diff --exit-code fail on every PR — including ones nowhere near usid.c — with a diff nobody introduced. Is it worth pinning an explicit clang-NN here, or comparing something less brittle than raw bytes?
The Dockerfile regenerates the object during the image build, so the .o that ships is the builder image's clang output, not the committed blob that CI verified and reviewers read. That leaves three artifacts that can differ: committed, CI-regenerated, shipped. What guarantees the shipped datapath is the one that was reviewed? If the answer is "nothing, and that's fine because it's regenerated from the same source," is committing the blob still earning its keep — and if it is, would embedding the committed object in the image build (rather than regenerating) be the stronger guarantee?
build:binaries going internal: true makes clang a hard requirement for task build, so a contributor touching unrelated Go now needs the eBPF toolchain or gets a hard failure. Is that intended? A build:ebpf that warns and skips when the committed artifacts are already current would keep the toolchain optional for everyone except the people actually editing usid.c.
The Dockerfile installs linux-libc-dev alongside clang/llvm; the CI step installs only clang llvm. doc.go's -idirafter workaround exists specifically to find that package's headers, so CI seems to be relying on it being preinstalled on the runner image. Should the two dependency lists match, so a slimmer runner image doesn't break the build in a way the Dockerfile already anticipated?
test-unit-root isn't in anyone's needs (test-e2e is [lint, test-unit, build]) — is it in the required-checks list for branch protection? If not, the job can go red without blocking a merge, which would make the coverage it adds advisory only. Related to the note you already left about wiring it into test-e2e later.
It re-runs the entire suite as root rather than just the root-gated packages. Beyond the duplicated runtime, any test that asserts a permission-denied path behaves differently as root. Worth scoping to the eBPF packages, or is whole-suite-as-root deliberate?

One note rather than a question: because this PR targets pr0-ebpf-prog and the workflow triggers on pull_request: branches: [main], CI hasn't run here at all — only the CLA and image-publish jobs. So the clang install, the drift check, and test-unit-root itself are all still unexercised, which matches your unchecked third box. Same applies to every PR in the stack until the base lands on main.

ecv reviewed PR #282 and raised six questions about the eBPF codegen toolchain (commit 1e4a2e1). This addresses each: - Pin the drift check's compiler (clang-18/llvm-18, explicit versioned packages) instead of the unversioned clang/llvm meta-packages, whose underlying version moves with runner-image bumps and would otherwise flip `git diff --exit-code` red on PRs that never touched usid.c. doc.go's go:generate directive drops the hardcoded `-cc clang` so bpf2go falls back to its own $BPF2GO_CC env override, letting CI pin the compiler without the directive itself needing to change. - Stop regenerating the eBPF object during the container build. The Dockerfile now ships the committed usid_bpfel.o/usid_bpfeb.o as-is (go:embed'd into the binary), so the object shipped in the image is bit-for-bit the one committed to git, reviewed in the PR, and verified against usid.c by CI's drift check -- not a separate build produced by the builder image's own clang. This also means the Dockerfile no longer needs a toolchain at all, which resolves the apt dependency-list mismatch between it and CI in the same move. - `build:ebpf` now warns and skips (building against whatever's already committed) instead of hard-failing when clang isn't installed, so contributors touching unrelated Go don't need the eBPF toolchain. CI's drift check remains the actual correctness gate regardless of what any individual contributor has installed locally. - CI's `build` job now installs linux-libc-dev explicitly alongside the pinned clang/llvm, rather than relying on it happening to be preinstalled on the runner image -- doc.go's -idirafter workaround exists specifically to find that package's headers. - `test-unit-root` is now in test-e2e's `needs`, so a failure there blocks the merge chain instead of being advisory-only. (Branch protection's required-status-checks list should also list it directly -- that's a repo setting, not something this commit can change.) - `test-unit-root` now runs scripts/ci.sh's new `unittest-root` case, which discovers requireRoot(t)-gated packages by grep instead of re-running the entire suite as root. Scopes the real-kernel run to just the packages that need it (today: internal/cni, internal/plumbing/ebpf/prog, internal/plumbing/loaddr, internal/vmtap) and picks up future root-gated packages automatically as later milestones add them. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

ef73ddf changed the BPF ELF license section from "Dual BSD/GPL" to "AGPL-3.0-or-later" on the premise that "none of the helpers this program calls today are gpl_only." That premise is false: the VRF FIB lookup already calls bpf_fib_lookup(), which the kernel marks gpl_only (net/core/filter.c's bpf_fib_lookup_proto). The kernel's license_is_gpl_compatible() check only recognizes a fixed whitelist of exact strings, and "AGPL-3.0-or-later" isn't on it, so the program now fails to load at all: "cannot call GPL-restricted function from non-GPL compatible program". This surfaced as PR #282's new CI jobs failing -- the Build job's drift check flagged the stale committed .o/.go (usid.c had changed but they hadn't been regenerated since), and once regenerated, Unit Tests (root) then hit the verifier rejection above on every FIB-lookup-path test. Revert the license string to "Dual BSD/GPL" and correct the surrounding comment accordingly -- the ELF license section governs which helpers the verifier allows and is independent of (doesn't relicense) the file's own AGPL-3.0-or-later SPDX header, same as Cilium, Katran, and every other AGPL/Apache/BSD-licensed project embedding a BPF datapath. Regenerate usid_bpfel.o/usid_bpfeb.o and their Go bindings from the corrected usid.c (also picking up the vrf_value.dropped_packets field ef73ddf added, which needed the same regeneration its own commit message flagged as still pending). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

ef73ddf changed the BPF ELF license section from "Dual BSD/GPL" to "AGPL-3.0-or-later" on the premise that "none of the helpers this program calls today are gpl_only." That premise is false: the VRF FIB lookup already calls bpf_fib_lookup(), which the kernel marks gpl_only (net/core/filter.c's bpf_fib_lookup_proto). The kernel's license_is_gpl_compatible() check only recognizes a fixed whitelist of exact strings, and "AGPL-3.0-or-later" isn't on it, so the program now fails to load at all: "cannot call GPL-restricted function from non-GPL compatible program". This surfaced as PR #282's new CI jobs failing -- the Build job's drift check flagged the stale committed .o/.go (usid.c had changed but they hadn't been regenerated since), and once regenerated, Unit Tests (root) then hit the verifier rejection above on every FIB-lookup-path test. Set the license string to plain "GPL" -- not "Dual BSD/GPL"; this program isn't itself dual-licensed, so it declares the license it actually means rather than a disjunction it doesn't. Correct the surrounding comment accordingly: the ELF license section governs which helpers the verifier allows and is independent of (doesn't relicense) the file's own AGPL-3.0-or-later SPDX header, same as Cilium, Katran, and every other AGPL/Apache/BSD-licensed project embedding a BPF datapath. Regenerate usid_bpfel.o/usid_bpfeb.o and their Go bindings from the corrected usid.c (also picking up the vrf_value.dropped_packets field ef73ddf added, which needed the same regeneration its own commit message flagged as still pending). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

ecv · 2026-08-05T16:31:09Z

this one's got failing tests @privateip

ecv reviewed PR #282 and raised six questions about the eBPF codegen toolchain (commit 1e4a2e1). This addresses each: - Pin the drift check's compiler (clang-18/llvm-18, explicit versioned packages) instead of the unversioned clang/llvm meta-packages, whose underlying version moves with runner-image bumps and would otherwise flip `git diff --exit-code` red on PRs that never touched usid.c. doc.go's go:generate directive drops the hardcoded `-cc clang` so bpf2go falls back to its own $BPF2GO_CC env override, letting CI pin the compiler without the directive itself needing to change. - Stop regenerating the eBPF object during the container build. The Dockerfile now ships the committed usid_bpfel.o/usid_bpfeb.o as-is (go:embed'd into the binary), so the object shipped in the image is bit-for-bit the one committed to git, reviewed in the PR, and verified against usid.c by CI's drift check -- not a separate build produced by the builder image's own clang. This also means the Dockerfile no longer needs a toolchain at all, which resolves the apt dependency-list mismatch between it and CI in the same move. - `build:ebpf` now warns and skips (building against whatever's already committed) instead of hard-failing when clang isn't installed, so contributors touching unrelated Go don't need the eBPF toolchain. CI's drift check remains the actual correctness gate regardless of what any individual contributor has installed locally. - CI's `build` job now installs linux-libc-dev explicitly alongside the pinned clang/llvm, rather than relying on it happening to be preinstalled on the runner image -- doc.go's -idirafter workaround exists specifically to find that package's headers. - `test-unit-root` is now in test-e2e's `needs`, so a failure there blocks the merge chain instead of being advisory-only. (Branch protection's required-status-checks list should also list it directly -- that's a repo setting, not something this commit can change.) - `test-unit-root` now runs scripts/ci.sh's new `unittest-root` case, which discovers requireRoot(t)-gated packages by grep instead of re-running the entire suite as root. Scopes the real-kernel run to just the packages that need it (today: internal/cni, internal/plumbing/ebpf/prog, internal/plumbing/loaddr, internal/vmtap) and picks up future root-gated packages automatically as later milestones add them. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

ef73ddf changed the BPF ELF license section from "Dual BSD/GPL" to "AGPL-3.0-or-later" on the premise that "none of the helpers this program calls today are gpl_only." That premise is false: the VRF FIB lookup already calls bpf_fib_lookup(), which the kernel marks gpl_only (net/core/filter.c's bpf_fib_lookup_proto). The kernel's license_is_gpl_compatible() check only recognizes a fixed whitelist of exact strings, and "AGPL-3.0-or-later" isn't on it, so the program now fails to load at all: "cannot call GPL-restricted function from non-GPL compatible program". This surfaced as PR #282's new CI jobs failing -- the Build job's drift check flagged the stale committed .o/.go (usid.c had changed but they hadn't been regenerated since), and once regenerated, Unit Tests (root) then hit the verifier rejection above on every FIB-lookup-path test. Set the license string to plain "GPL" -- not "Dual BSD/GPL"; this program isn't itself dual-licensed, so it declares the license it actually means rather than a disjunction it doesn't. Correct the surrounding comment accordingly: the ELF license section governs which helpers the verifier allows and is independent of (doesn't relicense) the file's own AGPL-3.0-or-later SPDX header, same as Cilium, Katran, and every other AGPL/Apache/BSD-licensed project embedding a BPF datapath. Regenerate usid_bpfel.o/usid_bpfeb.o and their Go bindings from the corrected usid.c (also picking up the vrf_value.dropped_packets field ef73ddf added, which needed the same regeneration its own commit message flagged as still pending). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

ecv reviewed PR #282 and raised six questions about the eBPF codegen toolchain (commit 1e4a2e1). This addresses each: - Pin the drift check's compiler (clang-18/llvm-18, explicit versioned packages) instead of the unversioned clang/llvm meta-packages, whose underlying version moves with runner-image bumps and would otherwise flip `git diff --exit-code` red on PRs that never touched usid.c. doc.go's go:generate directive drops the hardcoded `-cc clang` so bpf2go falls back to its own $BPF2GO_CC env override, letting CI pin the compiler without the directive itself needing to change. - Stop regenerating the eBPF object during the container build. The Dockerfile now ships the committed usid_bpfel.o/usid_bpfeb.o as-is (go:embed'd into the binary), so the object shipped in the image is bit-for-bit the one committed to git, reviewed in the PR, and verified against usid.c by CI's drift check -- not a separate build produced by the builder image's own clang. This also means the Dockerfile no longer needs a toolchain at all, which resolves the apt dependency-list mismatch between it and CI in the same move. - `build:ebpf` now warns and skips (building against whatever's already committed) instead of hard-failing when clang isn't installed, so contributors touching unrelated Go don't need the eBPF toolchain. CI's drift check remains the actual correctness gate regardless of what any individual contributor has installed locally. - CI's `build` job now installs linux-libc-dev explicitly alongside the pinned clang/llvm, rather than relying on it happening to be preinstalled on the runner image -- doc.go's -idirafter workaround exists specifically to find that package's headers. - `test-unit-root` is now in test-e2e's `needs`, so a failure there blocks the merge chain instead of being advisory-only. (Branch protection's required-status-checks list should also list it directly -- that's a repo setting, not something this commit can change.) - `test-unit-root` now runs scripts/ci.sh's new `unittest-root` case, which discovers requireRoot(t)-gated packages by grep instead of re-running the entire suite as root. Scopes the real-kernel run to just the packages that need it (today: internal/cni, internal/plumbing/ebpf/prog, internal/plumbing/loaddr, internal/vmtap) and picks up future root-gated packages automatically as later milestones add them. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Teaches task build to regenerate the TC-BPF usid program's Go bindings (bpf2go via go generate, split into a new build:ebpf step ahead of build:binaries) and the CNI container image to do the same at image-build time, so clang/llvm is a build-time-only dependency and never ships in the runtime image. Adds a root-privileged rerun of the unit test suite in CI (test-unit-root) so tests gated on requireRoot(t) -- the BPF_PROG_TEST_RUN cases in usid_test.go, plus later root-gated tests -- actually execute instead of skipping, and a step that diffs the regenerated eBPF artifacts against the committed ones so usid.c and its compiled output can't silently drift apart. Stacked on #281 (adds the usid.c program this generates from). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

ecv reviewed PR #282 and raised six questions about the eBPF codegen toolchain (commit 1e4a2e1). This addresses each: - Pin the drift check's compiler (clang-18/llvm-18, explicit versioned packages) instead of the unversioned clang/llvm meta-packages, whose underlying version moves with runner-image bumps and would otherwise flip `git diff --exit-code` red on PRs that never touched usid.c. doc.go's go:generate directive drops the hardcoded `-cc clang` so bpf2go falls back to its own $BPF2GO_CC env override, letting CI pin the compiler without the directive itself needing to change. - Stop regenerating the eBPF object during the container build. The Dockerfile now ships the committed usid_bpfel.o/usid_bpfeb.o as-is (go:embed'd into the binary), so the object shipped in the image is bit-for-bit the one committed to git, reviewed in the PR, and verified against usid.c by CI's drift check -- not a separate build produced by the builder image's own clang. This also means the Dockerfile no longer needs a toolchain at all, which resolves the apt dependency-list mismatch between it and CI in the same move. - `build:ebpf` now warns and skips (building against whatever's already committed) instead of hard-failing when clang isn't installed, so contributors touching unrelated Go don't need the eBPF toolchain. CI's drift check remains the actual correctness gate regardless of what any individual contributor has installed locally. - CI's `build` job now installs linux-libc-dev explicitly alongside the pinned clang/llvm, rather than relying on it happening to be preinstalled on the runner image -- doc.go's -idirafter workaround exists specifically to find that package's headers. - `test-unit-root` is now in test-e2e's `needs`, so a failure there blocks the merge chain instead of being advisory-only. (Branch protection's required-status-checks list should also list it directly -- that's a repo setting, not something this commit can change.) - `test-unit-root` now runs scripts/ci.sh's new `unittest-root` case, which discovers requireRoot(t)-gated packages by grep instead of re-running the entire suite as root. Scopes the real-kernel run to just the packages that need it (today: internal/cni, internal/plumbing/ebpf/prog, internal/plumbing/loaddr, internal/vmtap) and picks up future root-gated packages automatically as later milestones add them. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

The new test-unit-root CI job (this PR) was the first thing to ever actually run these requireRoot(t)-gated tests -- and several failed: TestCleanupContainerNetnsNonVeth, TestCmdDelFlushesGuestNetnsConfig, TestConfigureInterfaceInNetns*, TestFlushGuestNetnsConfig*. All of them go through createTestNetnsWithDummy, which used ns.TempNetNS(). Its Path() is /proc/<pid>/task/<tid>/ns/net -- valid only while that specific OS thread stays alive and stays in that netns. Every caller of the helper immediately hands netnsPath to production code (configureInterfaceInNetns, cleanupContainerNetns, flushGuestNetnsConfig) that reopens it independently via its own ns.GetNS(netnsPath) call, and by then nsObj.Do() has already returned and unlocked its OS thread, which the Go runtime is free to reuse for an unrelated goroutine in a different netns. Confirmed with a minimal repro: a second ns.GetNS(nsObj.Path()).Do() after the first Do() returns can no longer see interfaces created in the first call. Switch to testutils.NewNS() (github.com/containernetworking/plugins/ pkg/testutils), which bind-mounts the netns to a stable path under /var/run/netns instead of pinning it to one thread's lifetime -- the same repro passes cleanly with it. Also fixes TestCleanupContainerNetnsVeth's own separate ns.TempNetNS() usage, and removes its "peer survives in host namespace" assertions: netlink.Veth's LinkAdd creates both ends of a pair in whatever netns the call runs in (no automatic host/guest split without an explicit peer-namespace move, which this test never did), and a veth pair is one kernel object -- LinkDel on either end removes both, wherever they live -- so those assertions could never have passed and weren't testing anything cleanupContainerNetns actually does. This is a pre-existing bug on main, unrelated to the eBPF work in this PR stack; it surfaced here because test-unit-root is the first CI job to ever run these tests as root instead of skipping them. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Adds the Go control-plane library for the TC-BPF uSID datapath: - preflight: kernel capability gate (BTF, HASH maps, SCHED_CLS, bpf_fib_lookup with VRF-tbid support) run before attempting to load the program at all. - usidmap: typed read/write/reconcile API for the three kernel maps (locator_table, function_table, vrf_table) the program consults. - attach: load/pin/attach/detach/watch lifecycle for the TC-BPF ingress hook, including netlink-driven re-attachment on interface or route change, gated by the new GALACTIC_CNI_EBPF_INTERFACES env var (internal/config) for multi-homed nodes where auto-detection is ambiguous. - metrics: Prometheus collector plus load/attach event counters. Every package here is independently unit-testable against fakes (faketable_test.go, a mock kernel prober, fake closers) -- nothing outside this tree calls any of it yet. Stacked on #281 (usid.c/uformat) and #282 (codegen toolchain). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

ecv

can you tech-debt ticket yourself to move the comments into documentation, AFTER merge?

Adds the Go control-plane library for the TC-BPF uSID datapath: - preflight: kernel capability gate (BTF, HASH maps, SCHED_CLS, bpf_fib_lookup with VRF-tbid support) run before attempting to load the program at all. - usidmap: typed read/write/reconcile API for the three kernel maps (locator_table, function_table, vrf_table) the program consults. - attach: load/pin/attach/detach/watch lifecycle for the TC-BPF ingress hook, including netlink-driven re-attachment on interface or route change, gated by the new GALACTIC_CNI_EBPF_INTERFACES env var (internal/config) for multi-homed nodes where auto-detection is ambiguous. - metrics: Prometheus collector plus load/attach event counters. Every package here is independently unit-testable against fakes (faketable_test.go, a mock kernel prober, fake closers) -- nothing outside this tree calls any of it yet. Stacked on #281 (usid.c/uformat) and #282 (codegen toolchain). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

privateip mentioned this pull request Aug 5, 2026

feat: eBPF uSID datapath control-plane packages #283

Merged

4 tasks

privateip assigned ecv, 0xmc, scotwells and slindseysr and unassigned ecv, 0xmc, scotwells and slindseysr Aug 5, 2026

privateip marked this pull request as ready for review August 5, 2026 01:58

privateip requested a review from a team as a code owner August 5, 2026 01:58

privateip requested review from 0xmc, ecv, ronggur, scotwells and slindseysr August 5, 2026 01:58

ecv mentioned this pull request Aug 5, 2026

feat: Add TC-BPF uSID program and bit-layout library #281

Merged

2 tasks

ecv mentioned this pull request Aug 5, 2026

feat: eBPF vrf_table GC sweep and control-daemon startup #287

Merged

4 tasks

privateip force-pushed the pr1-ebpf-ci-toolchain branch from 7a672d4 to 10d1fe8 Compare August 5, 2026 16:10

privateip force-pushed the pr1-ebpf-ci-toolchain branch from 10d1fe8 to c54f5ed Compare August 5, 2026 16:12

privateip force-pushed the pr1-ebpf-ci-toolchain branch from c54f5ed to 7ac4585 Compare August 5, 2026 16:33

privateip force-pushed the pr1-ebpf-ci-toolchain branch from 7ac4585 to 1d36cd8 Compare August 5, 2026 16:46

Base automatically changed from pr0-ebpf-prog to main August 6, 2026 11:08

privateip force-pushed the pr1-ebpf-ci-toolchain branch 2 times, most recently from a8f7afc to 0e45a3c Compare August 6, 2026 12:41

privateip and others added 3 commits August 6, 2026 14:40

privateip force-pushed the pr1-ebpf-ci-toolchain branch from 68ba958 to 5ae36ae Compare August 6, 2026 18:40

ecv approved these changes Aug 6, 2026

View reviewed changes

privateip mentioned this pull request Aug 6, 2026

Move eBPF CI/build rationale out of code comments and into documentation #296

Open

privateip merged commit ec0c283 into main Aug 6, 2026
1 check passed

privateip deleted the pr1-ebpf-ci-toolchain branch August 6, 2026 21:37

Navigation Menu

Uh oh!

feat: eBPF codegen toolchain for the uSID TC-BPF program - #282

feat: eBPF codegen toolchain for the uSID TC-BPF program#282
privateip merged 3 commits into
mainfrom
pr1-ebpf-ci-toolchain

privateip commented Aug 5, 2026

Uh oh!

ecv commented Aug 5, 2026

Uh oh!

ecv commented Aug 5, 2026

Uh oh!

ecv left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

privateip commented Aug 5, 2026

Summary

Test plan

Uh oh!

ecv commented Aug 5, 2026

Uh oh!

ecv commented Aug 5, 2026

Uh oh!

ecv left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants