Skip to content

[TEST] AI Release Notes Reviewer with v24.1.29 (Clean)#23295

Open
mikeCRL wants to merge 4 commits into
mainfrom
test-ai-reviewer-v24.1.29-clean
Open

[TEST] AI Release Notes Reviewer with v24.1.29 (Clean)#23295
mikeCRL wants to merge 4 commits into
mainfrom
test-ai-reviewer-v24.1.29-clean

Conversation

@mikeCRL
Copy link
Copy Markdown
Contributor

@mikeCRL mikeCRL commented May 11, 2026

Purpose

This is a clean test PR to validate the AI-powered release notes reviewer (from PR #23174) against actual WIP release notes (from PR #23291).

What's in this PR

This PR contains:

Unlike PR #23294, this PR contains only the release notes changes in the diff, not all the Python code from the AI reviewer implementation.

Changes in this PR

  1. src/current/_data/releases.yml - Added v24.1.29, v24.1.28, v24.1.27 entries
  2. src/current/_includes/releases/v24.1/v24.1.29.md - Release notes content

Expected Behavior

When this PR is created, it should trigger the release-notes-review.yml workflow, which will:

  1. ✅ Parse the YAML and Markdown changes correctly (no Python code conflicts)
  2. ✅ Run schema/format checks on the YAML and Markdown
  3. ✅ Validate PR references (#168782, #169973)
  4. ✅ Run AI-powered content quality checks
  5. ✅ Post results as a check run and PR comment

Comparison

Note

This PR is for testing only and should NOT be merged. The original PRs (#23174 and #23291) should be handled separately.

Supersedes #23294.

ebembi-crdb and others added 2 commits April 8, 2026 17:34
Add AI-powered release notes reviewer GitHub Action
684bb95 
Introduces a GitHub Action that automatically reviews PRs touching
release notes files using schema validation, technical accuracy checks,
and OpenAI GPT-4o content quality analysis.

- Copies core reviewer logic into .github/scripts/release_review/
- Adds standalone runner script (.github/scripts/run_release_review.py)
- Adds workflow that triggers on PRs to main touching releases.yml or
  _includes/releases/**
- Posts results as a neutral (advisory) check run and updatable PR comment
- Never blocks merge; requires OPENAI_API_KEY repository secret
@mikeCRL @roachdev-claude
Add v24.1.29 release notes for testing AI reviewer
eb9c4e2 
This commit adds only the v24.1.29 release notes (YAML + Markdown)
to test the AI-powered release notes reviewer from PR #23174.

Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>
@mikeCRL mikeCRL requested a review from a team as a code owner May 11, 2026 17:34
@netlify
Copy link
Copy Markdown

netlify Bot commented May 11, 2026 •
edited
Loading

✅ Deploy Preview for cockroachdb-api-docs canceled.

Name Link
🔨 Latest commit 99499b8
🔍 Latest deploy log https://app.netlify.com/projects/cockroachdb-api-docs/deploys/6a021cde37f95f0008188265

@netlify
Copy link
Copy Markdown

netlify Bot commented May 11, 2026 •
edited
Loading

✅ Deploy Preview for cockroachdb-interactivetutorials-docs canceled.

Name Link
🔨 Latest commit 99499b8
🔍 Latest deploy log https://app.netlify.com/projects/cockroachdb-interactivetutorials-docs/deploys/6a021cdd3b4fd40008dcc602

@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 11, 2026

Files changed:

  • .github/scripts/release_review/__init__.py
  • .github/scripts/release_review/config.py
  • .github/scripts/release_review/github_client.py
  • .github/scripts/release_review/reporter.py
  • .github/scripts/release_review/reviewer.py
  • .github/scripts/release_review/schemas.py
  • .github/scripts/run_release_review.py
  • .github/workflows/release-notes-review.yml
  • src/current/_data/releases.yml
  • src/current/_includes/releases/v24.1/v24.1.29.md:

@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 11, 2026 •
edited
Loading

Release Notes Advisory (AI) — No issues found. Release notes look good!

High: 0 · Medium: 0 · Low: 0

Links

Posted by docs-fast-agent — advisory only, does not block merge.

@netlify
Copy link
Copy Markdown

netlify Bot commented May 11, 2026 •
edited
Loading

❌ Deploy Preview for cockroachdb-docs failed. Why did it fail? →

Name Link
🔨 Latest commit 99499b8
🔍 Latest deploy log https://app.netlify.com/projects/cockroachdb-docs/deploys/6a021cdd2d009e0008938275

@mikeCRL @roachdev-claude
Fix YAML/Markdown diff parsing to avoid false matches
21f3d62 
The parse_yaml_diff() and parse_markdown_diff() methods were matching
any line containing "releases.yml" or ".md", which caused them to
incorrectly parse Python code that contained these strings in comments,
docstrings, or string literals.

This fix ensures we only start parsing when we encounter the actual
diff header (lines starting with "diff --git"), making the parsing
more robust and preventing false matches.

Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>
@mikeCRL mikeCRL force-pushed the test-ai-reviewer-v24.1.29-clean branch from e75cb23 to 21f3d62 Compare May 11, 2026 18:13
@mikeCRL @roachdev-claude
Update path filter to only trigger on release note files
99499b8 
Changes the workflow path filter from 'releases/**' to 'releases/**/v*.md'
to prevent triggering on support files that are not release notes.

This ensures the workflow only runs on actual release note files (which
follow the naming convention vX.Y.Z.md) and excludes support files like:
- backward-incompatible.md
- cluster-setting-changes.md
- deprecations.md
- upgrade-finalization.md
- whats-new-intro.md

Prevents wasted workflow minutes and reduces noise on PRs that aren't
adding release notes.

Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>
@mikeCRL
Copy link
Copy Markdown
Contributor Author

mikeCRL commented May 12, 2026 •
edited
Loading

Example output

This shows what an automated PR comment would look like if the 'AI Release Notes Reviewer' (name TBD) found issues across all three check types.

We are not yet able to test the actual flow against ChatGPT, so until that test is set up, this is Claude's approximation of what the output would look like.

Release Notes Advisory (AI) — Found 12 issue(s): 5 HIGH, 6 MEDIUM, 1 LOW

High: 5 · Medium: 6 · Low: 1

HIGH (5)

  • Missing required YAML field: release_date: The 'release_date' field is required in releases.yml but is missing or empty.

    src/current/_data/releases.yml.
    Suggestion: Add the 'release_date' field with an appropriate value.

  • Invalid date format: release_date '2026-5-12' is not in YYYY-MM-DD format.

    src/current/_data/releases.yml
    Suggestion: Use format: YYYY-MM-DD (e.g., 2026-05-12)

  • Invalid version format: release_name 'V24.1.30' doesn't match expected pattern vXX.X.X[-suffix].

    src/current/_data/releases.yml
    Suggestion: Use format: vXX.X.X or vXX.X.X-beta.1 (lowercase 'v')

  • Missing PR link definition: PR #123456 is referenced in the text but has no link definition.

    src/current/_includes/releases/
    Suggestion: Add: [#123456]: sql/opt: missing error handling for metadata staleness check cockroach#123456

  • Referenced PR #999999 does not exist: PR #999999 was referenced but could not be found in cockroachdb/cockroach.

    src/current/_includes/releases/
    Suggestion: Verify the PR number is correct.

MEDIUM (6)

  • Orphaned link definition: Link definition for #88888 exists but is never referenced in the text.
    src/current/_includes/releases/

    Suggestion: Either remove the link definition or add a reference to #88888 in the release notes.

  • Passive voice detected: This release note uses passive voice ("was fixed by the team") instead of active voice, which makes it less clear and direct.

    Suggestion: Rewrite using active voice. For example, change "The bug was fixed by the team" to "Fixed the bug" or start with an action verb describing what changed.

  • Release note too short: This note contains only 6 words ("Fixed bug in SQL parser"), which may not provide sufficient detail about the change or its impact.

    Suggestion: Add more context about what the bug was, how it affected users, and what specifically changed. Aim for at least 10 words to provide adequate detail.

  • Placeholder text detected: The text contains "TODO" which indicates incomplete content.

    Suggestion: Replace the placeholder with the actual content before publishing the release notes.

  • Hyperbolic language: The word "simply" is used, which is considered hyperbolic and should be avoided in technical documentation.

    Suggestion: Remove "simply" and describe the actual improvement or change objectively without subjective qualifiers.

  • Incorrect capitalization: The text uses "cockroachdb" but should use the proper capitalization "CockroachDB".

    Suggestion: Change "cockroachdb" to "CockroachDB" (capital C and DB).

LOW (1)

  • Minor style inconsistency: Consider using a more descriptive action verb to start the release note.

    Suggestion: Review the CockroachDB style guide for examples of strong action verbs that clearly describe what changed.

Links

Posted by docs-fast-agent — advisory only, does not block merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@mikeCRL