Skip to content

Enforce copyright, CLAUDE.md and ticket-reference rules in CI (PROF-16469) - #938

Open
rkennke wants to merge 6 commits into
mainfrom
rkennke/PROF-16469-ci-checks
Open

rkennke wants to merge 6 commits into
mainfrom
rkennke/PROF-16469-ci-checks

Conversation

@rkennke

@rkennke rkennke commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do?:

Puts a CI check behind three rules that AGENTS.md states but nothing enforced, and runs the utils/ test suites in CI.

  • Copyright headers: utils/check-copyright.py, part of qualityChecks, applies the AGENTS.md copyright rule to the files a branch adds, modifies or renames. In a PR it compares with origin/$GITHUB_BASE_REF, elsewhere with origin/main. It reads regular files with a code extension (C/C++ incl. .inc, Java, Kotlin, Groovy, Python, shell, TS/JS) and extensionless #! scripts. It skips symlinks, directories and submodule entries.
  • CLAUDE.md: the file is committed, no longer git-ignored. utils/check-claude-md.py, also in qualityChecks, keeps it the canonical redirect stub, which replaces the bootstrap comment in AGENTS.md. The cairn plugin's SessionStart hook injects one managed block into the working copy. The check ignores that block only while it is uncommitted and HEAD:CLAUDE.md is canonical. A clean CI checkout therefore always compares the exact file, so text cannot be smuggled in inside cairn markers.
  • Ticket reference: the new PR ticket reference workflow fails a PR whose title and body contain neither PROF-<n> nor #<n>. Bot accounts are exempt.
  • Script tests in CI: the new script-tests job in ci.yml runs utils/tests/*.sh and the ticket-check test, with Node.js 22 for the Pi adapter cases.
  • AGENTS.md:
    • the duplicated CI-script rules are merged into one section;
    • Work Tracking is shortened to the binding rules;
    • each rule that a check now enforces names that check.

Motivation:

Step 1 of PROF-16469: advisory text cannot gate, so rules that a machine can check get a check. AGENTS.md also claimed that CI validated ticket references, but no workflow, script or ruleset did.

Additional Notes:

  • Flagged for reviewer acknowledgment: the copyright check will fail many PRs. Run against the last 20 merged PRs, it would have failed 12. Most failures are modified files with no copyright line at all (build-logic Kotlin files, tests), the rest are Datadog end years that were not bumped. The check follows the AGENTS.md rule as written. If that is too strict, relax the rule ("add a line to files that have none") rather than the check.
  • The ticket check also bites: 12 of the last ~66 human-authored PRs had no reference. Bot PRs are exempt by the [bot] login suffix, not by title.
  • Why a separate workflow for the ticket check: ci.yml does not run on edited, so fixing the description would not clear the failure.
  • Existing tests were broken: the utils/tests/ suites never ran in CI, and two had broken unnoticed. Both are fixed here.
    • test_quality_check_scripts.sh T1g pinned a sigsetjmp call count that the tree no longer has. It now pins only the exception count.
    • test_review_marker.sh failed, instead of skipping, its Pi cases on Node.js older than 22.6.
  • Copyright years: the required end year is the year of the file's last commit, not the clock, so a December PR still passes in January.
  • Upstream files: a file with only the async-profiler copyright needs nothing, because the rule covers only Datadog lines and files without any copyright.
  • Rollout: anyone with a local, untracked CLAUDE.md must delete it before pulling. Otherwise git refuses: "untracked working tree file would be overwritten".
  • Not in AGENTS.md: there is no link to the internal SDLC playbook, because the repository is public.
  • Known limits, left as is:
    • check-copyright.py needs refs/remotes/origin/main, or a --base. A local qualityChecks run in a clone with a differently named remote fails with exit 2. CI fetches full history and is not affected.
    • The check reads tracked files only, so a new file is checked once it is added to git.
    • A direct push outside a PR (no GITHUB_BASE_REF) compares with main, also for release/* branches.
  • Pre-PR review: Sphinx on the full range (standard, then deep), and standard on each fix range. All Critical/High/Medium findings are fixed. The deep run found the cairn-marker bypass (Medium), fixed as described above. The two Low test gaps from the last run are covered (T6j, T6k). Remaining Lows are test gaps the fixtures do not exercise: a copyright line below line 1, author versus committer date, mixed-case "Datadog", and a #<n> right after a word character. They are left as is.

How to test the change?:

  • bash utils/tests/test_quality_check_scripts.sh, now 84 groups. New:
    • T5: check-copyright.py over a fixture branch with new, stale, ranged, renamed, deleted, excluded, upstream-only, Present, December-committed and uncommitted files, plus .ts/.inc, shebang scripts, non-ASCII and non-UTF-8 names, symlinks, a binary, a submodule entry, and GITHUB_BASE_REF versus --base;
    • T6: check-claude-md.py, including cairn blocks (uncommitted, two blocks, committed, plus drift), --root below the repository top, no git, and a non-UTF-8 committed file;
    • T4: wiring.
  • .github/scripts/tests/test_check_pr_ticket.sh: 14 cases, including the template's JIRA-XXXX placeholder, HTML entities, anchors and bot authors.
  • Removing each rule in check-copyright.py makes the suite fail.
  • ./gradlew :ddprof-lib:qualityChecks (run through build-and-summarize) passes and runs both new checks.
  • CI emulation: a shallow clone of the branch in node:22-bookworm, with the job's steps, passes all 8 suites.

For Datadog employees:

  • If this PR touches code that signs or publishes builds or packages, or handles
    credentials of any kind, I've requested a security review (run the dd:platform-security-review
    skill, or file a request via the PSEC review form).
    bewaire also runs automatically on every PR.
  • This PR doesn't touch any of that.
  • JIRA: PROF-16469

🤖 Generated with Claude Code

rkennke and others added 2 commits November 4, 2026 18:42
…6469)

AGENTS.md stated these rules, but no check enforced them, and it claimed a
CI check for ticket references that did not exist.

- utils/check-copyright.py (in qualityChecks) applies the AGENTS.md
  copyright rule to the files a branch adds or modifies. The end year is
  compared with the file's last commit year, so a December change still
  passes when checked in January.
- CLAUDE.md is now committed instead of git-ignored. utils/check-claude-md.py
  (in qualityChecks) keeps it the canonical redirect stub, which replaces
  the bootstrap comment in AGENTS.md.
- The PR ticket reference workflow fails a PR whose title and body have no
  PROF-<n> or #<n>. It is a separate workflow so that editing the
  description re-runs it.
- The new ci.yml script-tests job runs utils/tests/*.sh and the ticket-check
  test with Node.js 22. CI never ran these suites, and two had broken
  unnoticed: T1g pinned a sigsetjmp call count the tree no longer has, and
  the Pi adapter cases failed instead of skipping on Node.js older than 22.6.
- AGENTS.md: the duplicate CI-script rules are merged into one section,
  Work Tracking is shortened to the binding rules, and each enforced rule
  names its check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
utils/tests/test_prepare_patch.sh runs utils/prepare-patch.sh, which checks
for jq. ubuntu-22.04 runners ship it today; installing it keeps the job
independent of the runner image.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rkennke
rkennke requested a review from a team as a code owner November 4, 2026 19:27
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-04T19:30:48.035976Z f499864 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f499864398

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread utils/check-copyright.py Outdated
ap = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])
ap.add_argument("--root", default=os.path.dirname(os.path.dirname(os.path.abspath(__file__))),
help="repository root (default: the checkout containing this script)")
ap.add_argument("--base", default="origin/main", help="base branch (default: origin/main)")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Compare copyright changes with the PR's actual base

When a PR targets one of the repository's release/* branches, as utils/backport-pr.sh and utils/prepare-patch.sh do, qualityChecks invokes this script without --base, so the default compares the branch with origin/main rather than the PR base. The check then examines all changes accumulated on the release branch and can block a backport because of stale headers in unrelated earlier cherry-picks; pass the pull request's github.base_ref (or its corresponding remote ref) to the check.

AGENTS.md reference: AGENTS.md:L403-L410

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 79ff87112: in a pull request the base defaults to origin/$GITHUB_BASE_REF, so a backport into release/* is compared with its own branch. An explicit --base still wins (tested). A direct push outside a PR still compares with main.

Comment thread utils/check-copyright.py Outdated
import subprocess
import sys

SOURCE = (".c", ".cc", ".cpp", ".h", ".hpp", ".java", ".kt", ".kts", ".py", ".sh")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Cover every source extension in copyright enforcement

When a PR adds a TypeScript source or a C++ .inc test source—both types already exist under .pi/extensions/ and ddprof-lib/src/test/cpp/—path.endswith(SOURCE) is false, so the checker returns before validating its header. This lets new source files bypass the rule the check is intended to enforce; include all source extensions used by the repository or classify source files by their tracked source directories.

AGENTS.md reference: AGENTS.md:L403-L410

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 79ff87112: .inc, .ts, .js and .groovy are checked now; .inc counts as C++ for the missing-line rule. Tested in T5.

@dd-octo-sts

dd-octo-sts Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Scan-Build Report

User:runner@runnervm48cm3
Working Directory:/home/runner/work/java-profiler/java-profiler/ddprof-lib/src/test/make
Command Line:make -j4 all
Clang Version:Ubuntu clang version 18.1.3 (1ubuntu1)
Date:Wed Nov 4 20:53:37 2026

Bug Summary

Bug TypeQuantityDisplay?
All Bugs2
Logic error
Dereference of null pointer1
Unused code
Dead initialization1

Reports

Bug Group Bug Type ▾ File Function/Method Line Path Length
Unused codeDead initializationreferenceChains.cppapplyOomRamp4971
Logic errorDereference of null pointerperfEvents_linux.cppunregisterThread72026

@datadog-datadog-prod-us1-2 datadog-datadog-prod-us1-2 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: FAIL

The copyright gate silently skips non-ASCII filenames under Git’s default quoting, along with TypeScript and extensionless scripts. These gaps allow missing or stale headers through CI.

Open Bits AI session

🤖 Bits Code Review · Commit f499864

Comment thread utils/check-copyright.py Outdated
Comment on lines +46 to +49
out = git(root, "diff", "--name-status", "-M", "--diff-filter=AMR", "--no-relative", mb)
for line in out.splitlines():
parts = line.split("\t")
yield parts[0] == "A", parts[-1]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Parse Git filenames without retaining quoting

With Git’s default core.quotePath setting, a filename such as café.java is returned quoted with escaped bytes. The parser retains that quoting, so path.endswith(SOURCE) rejects the filename and CI silently accepts missing or stale copyright headers. NUL-delimited parsing preserves the actual filenames, including rename destinations.

Suggested change
out = git(root, "diff", "--name-status", "-M", "--diff-filter=AMR", "--no-relative", mb)
for line in out.splitlines():
parts = line.split("\t")
yield parts[0] == "A", parts[-1]
out = git(root, "diff", "--name-status", "-z", "-M", "--diff-filter=AMR", "--no-relative", mb)
fields = iter(out.split("\0")[:-1])
for status in fields:
path = next(fields)
if status.startswith("R"):
path = next(fields)
yield status == "A", path

React: 👍 useful · 👎 wrong · 😕 unclear · 👀 right, but not acting on it

What each reaction means
  • 👍 Useful: correct, and you're fixing it or will.
  • 👎 Wrong: the finding is incorrect, for example the bug can't happen or the code was misread.
  • 😕 Unclear: you can't tell what it means or why it matters.
  • 👀 Right, not acting: correct, but intended, out of scope for this PR, a known risk, or too minor.

🤖 Bits Code Review · Open Bits AI session

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 79ff87112: the script parses git diff --name-status -z, so non-ASCII names are no longer quoted. 855f127b6 also stops non-UTF-8 names from aborting the output. Both cases are in T5.

Comment thread utils/check-copyright.py Outdated
Comment on lines +72 to +74
if not path.endswith(SOURCE) or path.startswith(EXCLUDED):
return None
full = os.path.join(root, path)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Include TypeScript and extensionless source scripts

The extension filter skips source types already present in the repository, including .pi/extensions/require-fresh-review.ts and the extensionless Python .githooks/pre-push. Adding these sources without headers, removing their existing Datadog headers, or leaving stale years passes the gate despite the documented policy. Include TypeScript and recognize extensionless interpreter scripts.

Suggested change
if not path.endswith(SOURCE) or path.startswith(EXCLUDED):
return None
full = os.path.join(root, path)
if path.startswith(EXCLUDED):
return None
full = os.path.join(root, path)
if not path.endswith(SOURCE + (".ts",)):
if os.path.splitext(path)[1]:
return None
with open(full, "rb") as f:
if f.read(2) != b"#!":
return None

React: 👍 useful · 👎 wrong · 😕 unclear · 👀 right, but not acting on it

What each reaction means
  • 👍 Useful: correct, and you're fixing it or will.
  • 👎 Wrong: the finding is incorrect, for example the bug can't happen or the code was misread.
  • 😕 Unclear: you can't tell what it means or why it matters.
  • 👀 Right, not acting: correct, but intended, out of scope for this PR, a known risk, or too minor.

🤖 Bits Code Review · Open Bits AI session

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 79ff87112 and 855f127b6: .ts is checked, and extensionless files count only when they start with #!. Symlinks, directories and submodule entries are skipped. Tested in T5.

@dd-octo-sts

dd-octo-sts Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

CI Test Results

Run: #41910612046 | Commit: 0161780 | Duration: 20m 4s (longest job)

✅ All 32 test jobs passed

Status Overview

JDK glibc-aarch64/debug glibc-amd64/debug musl-aarch64/debug musl-amd64/debug
8 - ✅ - -
8-ibm - ✅ - -
8-j9 ✅ ✅ - -
8-librca - - ✅ ✅
8-orcl - ✅ - -
11 - ✅ - -
11-j9 ✅ ✅ - -
11-librca - - ✅ ✅
17 ✅ ✅ - -
17-graal ✅ ✅ - -
17-j9 ✅ ✅ - -
17-librca - - ✅ ✅
21 ✅ ✅ - -
21-graal ✅ ✅ - -
21-librca - - ✅ ✅
25 ✅ ✅ - -
25-graal ✅ ✅ - -
25-librca - - ✅ ✅

Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled

Summary: Total: 32 | Passed: 32 | Failed: 0


Updated: 2026-11-04 21:14:46 UTC

@dd-octo-sts

dd-octo-sts Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

✅ All 40 integration tests passed

📊 Dashboard · 👷 Pipeline · 📦 74916289

rkennke and others added 4 commits November 4, 2026 20:16
…6469)

- check-copyright.py defaults to origin/$GITHUB_BASE_REF in a pull request,
  so a backport into release/* is compared with that branch, not with main.
- It also checks .inc, .ts, .js and .groovy files and extensionless scripts
  that start with #! (git hooks, .claude/commands); .inc counts as C++ for
  the missing-line rule.
- It parses `git diff -z`, so file names with non-ASCII bytes are no longer
  quoted, which made them skip the extension test.
- check-claude-md.py ignores the block that the cairn plugin's SessionStart
  hook injects into CLAUDE.md; without that, every session with cairn left
  a CLAUDE.md that failed the check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- check-claude-md.py ignored any text between cairn markers, so a commit
  could put arbitrary content in CLAUDE.md and pass CI. It now ignores one
  cairn block only while it is uncommitted and HEAD's CLAUDE.md is
  canonical; a clean CI checkout always compares the exact file.
- check-copyright.py reads only regular files with a code extension or no
  extension: no symlinks, directories or submodule entries, and no binary
  reads before the extension filter. File names that are not valid UTF-8
  no longer abort it.
- Tests pin --base over GITHUB_BASE_REF, `#` versus `#!`, two cairn blocks
  and a committed block.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…OF-16469)

- check-claude-md.py reads HEAD:./CLAUDE.md, so a --root below the
  repository top compares against its own committed file, and a missing
  git binary or non-UTF-8 output no longer aborts the check.
- README lists the new skips (symlinks, directories, submodule entries).
- Tests: the symlink fixture points inside the fixture repo, the
  non-UTF-8 case names its file, a cairn block plus other drift fails, and
  --root below the top reads the right committed file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- T6j: without git on PATH, an uncommitted cairn block is not ignored.
- T6k: a committed non-UTF-8 CLAUDE.md gives a clean mismatch, not a
  decode traceback.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant