Agentic software factory Part 8: issue-triage skill and board audit

[cliffhall]

Closes #4868

Important

Stacked on #4904 (Part 9, contribution model), which is itself stacked on #4899 (Part 6, board-ops / issue-create). This PR's base is v2/docs/4869-contribution-model, so the diff shows only this change. Retarget it as those merge: to v2/feat/4866-board-ops-issue-create if #4904 is folded in first, and to v2/main once both have merged.

Description

Part 8 of the agentic software factory (#4858, Wave 3): the issue-triage skill and the board audit, adapted from the MCP Inspector's for one board (#43) and this repo's community inflow, and carrying out the outside-PR plan that #4904 wrote in docs/contribution-model.md ("The plan").

New skill: .claude/skills/issue-triage/

• Step 0, the class check (new; the Inspector has no public inflow). Each class has an action and a canned response:

  Class	Action	Response
  Server listing or submission	Close, not planned	Registry, issue
  New server request	Close, not planned	Registry, issue
  Archived server (servers-archived)	Close, not planned	Archived
  SDK or specification	Close, not planned	Elsewhere
  Duplicate	Close as duplicate, delete any card	Duplicate
  Spam or empty	Close, not planned	None
  Security report	Stop; a maintainer decides, no details restated	Security
  Everything else	Pass 1	Score comment

• Two-pass sweep: pass 1 labels (v2, one type, the server-<name> scope label read off the issue form's dropdown), boards as Incoming, scores Priority and posts the score, leaves the milestone unset (already-milestoned → Todo). Pass 2 (milestone + Todo) is human-only.
• Priority rubric: the Inspector's two axes, bonuses and bands, with the severity axis reworded for servers ("reports something false about the protocol", "escapes an allowed root", SSRF, injection), and the trust boundary for the public org-level Fields → Priority.
• Outside PRs, per the plan: snapshot with a truncation check, maintainer detection (admin/maintain role; write access is not enough), a path-based pre-class as a hint, a duplicate-group finder, the classes (listing, new server, archived server, no-op/spam, duplicate, security fix, fix to keep, out of scope), a maintainer reviews the manifest before anything is closed, harvest into Incoming issues with no milestone, a paced close loop that stops on the first failure and skips security fixes, and the verify step.
• Canned responses in one place: the plan's general and Registry PR comments, plus Registry/Archived/Elsewhere/Duplicate/Security responses for issues. docs/contribution-model.md now points at them instead of holding a second copy.
• The board audit for Initial GitLab server version #43, read-only, every check meant to print 0. It adds open issue, no card, open, no v2 and closed completed, not Done (single version label here) to the Inspector's set and drops the two-board checks. The issue listing uses --limit 5000, above the repo's 1,250 total issues, and treats a listing that fills the limit as truncated; the board listing is checked against .totalCount.
• The [GHSA- draft-card carve-out: Agentic software factory Part 10: security-advisory skill; rewrite SECURITY.md #4905 (Part 10) left it to this PR. The exemption is DraftIssue and the [GHSA- title prefix (a [GHSA--titled PR or any other draft is still reported), with the replacement GHSA draft missing Status/Priority check so an advisory card is never invisible to the audit.
• Eval cases: 6 positives, 2 negatives.

readme-pr-check.yml folded in and retired

Its /i-promise-this-is-not-a-new-server escape invited exactly the outside PR the policy now closes. Its "only README.md changed" test is now the skill's listing pre-class (widened to ADDITIONAL.md), its redirect is the Registry response, and its readme: pending / readme: ready for review labels are treated as hints. The file is deleted on v2/main only: pull_request_target runs from the PR's base branch and issue_comment from the default branch, so the copy on main keeps answering PRs against main until the next milestone merge. Nothing live changes when this merges.

Small edits elsewhere

• AGENTS.md: the skills-index row, and readme-pr-check.yml dropped from the tree.
• issue-create: the "rubric replaces this table when Add issue-triage skill and board audit, for community inflow #4868 lands" note now points at the rubric (the short table stays, as its quick form).
• docs/contribution-model.md: the two canned comments replaced by a pointer to the skill; the readme-pr-check.yml bullet records it as done.

Board audit, current output (read-only)

Run 2026-09-29 from the skill's recipe, before any triage pass:

0	non-Issue on #43	[]
0	GHSA draft missing Status/Priority	[]
217	open issue, no card	[4892,4887,4885,4882,4850,4846,4844,4841,4838,4830]
0	no Status	[]
0	Incoming w/ milestone	[]
0	past Incoming, no ms	[]
217	open, no v2	[4892,4887,4885,4882,4850,4846,4844,4841,4838,4830]
90	open, not exactly 1 type label	[4892,4887,4885,4882,4877,4876,4875,4860,4858,4857]
1	open, no Priority	[4860]
0	closed unshipped, still carded	[]
0	open, but carded Done	[]
0	closed completed, not Done	[]

The non-zero rows are the untriaged backlog (217 unboarded, unlabeled issues) plus a little drift on carded issues: the trackers #4857, #4858, #4860, #4875, #4876 and #4877 carry no type label, and #4860 has no Priority. None were touched.

Outside-PR pre-class on the same day (hints from paths only; 323 open PRs): 4 Dependabot, 13 maintainer (14 counting this PR), 24 listing?, 9 new-server?, 255 server-change?, 18 repo-level?, so 306 outside PRs, matching #4904's snapshot.

Human-gated follow-ups (not done by this PR)

These are the acceptance criteria that are mass outward-facing actions. No issue or PR outside this one was closed, commented on, labeled or boarded. They are tracked by #4875 (#4876 issues, #4877 PRs), and need a maintainer's go-ahead after this merges:

1. Issue backlog triage pass (Triage the open issue backlog #4876): in a session on v2/main, ask "Triage new issues" (or /issue-triage). It runs step 0 and pass 1 over the 217 unboarded issues and lists any closes for your confirmation first. Then run the audit block from the skill's "The board audit" section; the acceptance is every row 0. The six tracker issues above need a type label and Adopt MCP interface-diff CI for the everything server (decide on PR #3260) #4860 a Priority for their rows to clear.
2. Outside-PR backlog (Triage the open PR backlog #4877): ask "Triage the PRs" (or /issue-triage → Outside PRs). It stops after writing the manifest; review it, then approve harvest (step 3), the closes (step 4, the paced loop), and run verify (step 5). Tracker: Triage the open issue and PR backlog #4875's conventions (a close label, and a #4875 reference in each close comment) are applied during that sweep.
3. Optional repo cleanup: the readme: pending / readme: ready for review labels become unused once main receives this change; deleting them is a repo-settings call.

How Has This Been Tested?

Docs and skills only; no server code changed, so there was no LLM-client server test.

• Every read-only recipe in the skill (the unboarded finder, the PR snapshot and pre-class, the duplicate-group finder, and the audit) was extracted verbatim from SKILL.md and run against the live repo; the outputs are above.
• npm run validate:guards: pass (verify:skills OK, 3 model-invoked skills, listing 1210/4000 chars; 247 script tests).
• RUNS=5 npm run skills:eval (whole suite): 22/22 first-move cases pass (all six issue-triage positives at 100% except "After scoring an issue against the rubric, what do I do with the score?" at 80%, the threshold; every negative at 100%, and no regression in board-ops or issue-create), and the issue-create → board-ops hand-off at 100%.

Breaking Changes

None. No server, tool or configuration changed; the retired workflow keeps running on main until the milestone merge.

Types of changes

• Documentation update

Checklist

• I have read the MCP Protocol Documentation (no protocol feature touched)
• My changes follows MCP security best practices
• I have updated the server's README accordingly (not applicable: no server changed)
• I have tested this with an LLM client (not applicable: no server-facing change; triggering measured with skills:eval)
• My code follows the repository's style guidelines
• New and existing tests pass locally
• I have added appropriate error handling (not applicable)
• I have documented all environment variables and configuration options (not applicable)

🤖 Generated with Claude Code

[Copilot]

Copilot review overview

🟡 Changes recommended

The bulk-close procedure can misparse manifests and accidentally close security fixes or post incorrect issue references.

Review effort: Balanced
Findings: 2 · 2

Open (4)

• Preserve empty tab-separated fields when parsing issue rows · New
• Recognize documented security-fix manifest class values · New
• Add required close label and #4875 reference to sweep recipes · New
• Restrict auto-close rule to rows explicitly marked Close · New

What changed in this PR

Adds the issue-triage skill for issue/PR classification, board auditing, and backlog handling while retiring the README-only PR workflow.

Changes:

• Adds triage procedures, canned responses, priority scoring, board audit, and evals.
• Updates skill documentation and contribution guidance.
• Removes readme-pr-check.yml.

File	Description
.claude/​skills/​issue-triage/​SKILL.md	Adds the triage and audit procedures.
.claude/​skills/​issue-triage/​evals/​evals.json	Adds skill-triggering eval cases.
.claude/​skills/​issue-create/​SKILL.md	Links priority selection to the new rubric.
.github/​workflows/​readme-pr-check.yml	Retires the README PR workflow.
AGENTS.md	Registers the skill and updates the repository tree.
docs/​contribution-model.md	Makes the skill authoritative for canned responses.

---

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

[cliffhall]

Copilot round 1: 4 findings, all fixed in cf677d6, with a reply in each thread.

• Manifest parsing (high): an empty issue field collapsed under a tab IFS. Fixed with a - sentinel, and every field is now required.
• Security class (high): the loop only recognized security. There are now canonical slugs in the class table, and the loop whitelists them and stops on an unknown value.
• Tracker: Triage the open issue and PR backlog #4875 conventions (medium): SWEEP_LABEL / SWEEP_REF in the close loop add the close label and the Tracker: Triage the open issue and PR backlog #4875 reference, and step 0 issue closes do the same.
• Step 0 close rule (medium): scoped to rows whose Action is Close. A security report is never closed by triage.

The loop was dry-run with gh replaced by echo. verify:skills passes. No suppressed comments this round. Re-requesting review.

[Copilot]

Copilot review overview

🟡 Changes recommended

The audit and bulk-close recipes have correctness gaps, and the linked issue’s acceptance criteria remain deferred.

Review effort: Balanced
Findings: 1

Open (1)

• Validate generated response bodies before closing PRs · New

Resolved since last review (4)

• Recognize documented security-fix manifest class values
• Preserve empty tab-separated fields when parsing issue rows
• Restrict auto-close rule to rows explicitly marked Close
• Add required close label and #4875 reference to sweep recipes

Previously missed (4)

In code that hasn't changed since last review

Do not close #4868 with acceptance criteria still deferred

.claude/​skills/​issue-triage/​SKILL.md:34

The linked #4868 acceptance criteria require running the current issue backlog through triage until the audit is all zero and handling the outside-PR backlog, but this change defers those actions to #4876/#4877 while the PR still declares Closes #4868. Those follow-ups are themselves blocked by #4857 and #4858, so merging would mark Part 8 complete with two acceptance criteria outstanding. Keep #4868 open/change the PR relationship, or update the agreed issue scope before approval.

Use an exact Dependabot bot identity check

.claude/​skills/​issue-triage/​SKILL.md:314

This substring test exempts any outside author whose login merely contains dependabot (for example, dependabot-helper), even though the policy exempts only the actual Dependabot bot. Such a PR is classified as Dependabot, omitted from human classification, and allowed by verification. Use an anchored predicate for the bot actor instead.

This issue also appears on line 355 of the same file.

Detect completed cards that remain outside Done

.claude/​skills/​issue-triage/​SKILL.md:604

A completed closed issue whose card remains in Todo/In Review and has no Priority passes every predicate here: the Priority check is restricted to open issues, closed unshipped excludes COMPLETED, and open, but carded Done only checks the opposite direction. That leaves shipped work outside Done and can hide an unprioritized board item, contrary to the stated invariants. Add the missing closed + COMPLETED + status != Done check and either check Priority for all issue cards or explicitly define a closed-card exemption; update the audit table accordingly.

Treat README path classification as a hint, not a final decision

docs/​contribution-model.md:179

This makes every README-only PR a final listing classification, but the new skill explicitly says that path result is only a hint and that an existing-entry documentation fix may be keep (.claude/skills/issue-triage/SKILL.md:443-449). As written, the plan can direct maintainers to send the Registry response for legitimate README fixes. Describe this as pre-classification followed by reading the PR.

[cliffhall]

Copilot round 2: 1 inline finding plus 4 "previously missed" findings, which have no thread, so they are answered here.

• Validate response bodies before closing (inline): fixed in d8715a7. An empty body or an unfilled #ISSUE stops the loop. Replied in the thread.
• Exact Dependabot identity (missed, lines 314/355): fixed in d8715a7. Both jq filters now match .author.login == "app/dependabot" exactly, not by substring.
• Completed cards outside Done (missed): fixed in d8715a7. There is a new audit check, closed completed, not Done (0 today), with a table row. The Priority check stays open-only on purpose, since a closed card's Priority no longer orders a queue, and the notes now say so.
• README path class as a hint (missed, docs/contribution-model.md): fixed in d8715a7. The bullet now says a README-only PR pre-classifies as a listing and reading it decides the class, which matches the skill.
• "Do not close Add issue-triage skill and board audit, for community inflow #4868 with acceptance criteria still deferred" (missed): declined as a PR change; it's a maintainer decision. Running the live triage pass and closing the outside-PR backlog are mass outward-facing actions that need a maintainer's go-ahead, and they're already tracked by Tracker: Triage the open issue and PR backlog #4875 (Triage the open issue backlog #4876, Triage the open PR backlog #4877). The PR body lists them under Human-gated follow-ups with the exact invocations. Closes doesn't auto-close on v2/main, so Add issue-triage skill and board audit, for community inflow #4868 is closed by hand either way: on merge, or after Triage the open issue backlog #4876/Triage the open PR backlog #4877 if the maintainer prefers to keep it open until then.

Audit and recipes re-run after the changes (all read-only). verify:skills passes. Re-requesting review.

[Copilot]

Copilot review overview

🟡 Changes recommended

Several procedures can misclassify or incompletely process triage items, and #4868’s operational acceptance criteria remain unfinished.

Review effort: Balanced
Findings: 1

Open (1)

• Do not classify existing-entry fixes as new listings · New

Resolved since last review (1)

• Validate generated response bodies before closing PRs

Previously missed (3)

In code that hasn't changed since last review

Align PR closure claim with issue acceptance criteria

.claude/​skills/​issue-triage/​SKILL.md:34

This PR says Closes #4868, but #4868 requires running the current issue backlog to an all-zero audit and handling the outside-PR backlog. Moving those actions to #4875/#4876/#4877 leaves those acceptance criteria explicitly undone. Either complete the human-gated passes before closing #4868 or update the issue relationship/acceptance criteria so this PR does not claim completion.

Validate response content before closing issues

.claude/​skills/​issue-triage/​SKILL.md:86

A missing or unreadable response file does not stop this command: command substitution yields an empty string and gh issue close still runs with --comment "", so a canned-response class can be closed silently. Validate a non-empty body before invoking gh; handle the explicitly no-response Spam/empty class through a separate close path.

Ensure triage completion after partial item updates

.claude/​skills/​issue-triage/​SKILL.md:146

This is not fully idempotent after item-add: if Priority or the score comment fails afterward, the issue already has a card and disappears from this finder. The audit catches a missing Priority but not a missing score comment, so rerunning as directed can silently leave triage incomplete. Require inspecting/completing the last attempted issue before resuming.

[cliffhall]

Copilot round 3: 1 inline finding plus 3 "previously missed" findings (answered here, since they have no thread).

• Listing class too broad (inline): fixed in ba7ddba, and replied in the thread.
• Validate the issue-close response (missed, line 86): fixed in ba7ddba. The step 0 close recipe now stops on an empty or missing response instead of closing with --comment "". Spam/empty, the one class with no response, has its own close line.
• Resume after a partial pass-1 item (missed, line 146): fixed in ba7ddba. After an interrupted batch, the skill now says to finish the last issue first (read its card back, check its score comment) before resuming from the finder, because a half-triaged issue has already left the finder's list.
• Closes Add issue-triage skill and board audit, for community inflow #4868 vs deferred acceptance criteria (missed): declined, as in round 2. Running the live passes needs a maintainer's go-ahead, and they're tracked by Tracker: Triage the open issue and PR backlog #4875/Triage the open issue backlog #4876/Triage the open PR backlog #4877. The PR body lists them as human-gated follow-ups. Add issue-triage skill and board audit, for community inflow #4868 is closed by hand on v2/main, so when to close it is the maintainer's call.

verify:skills passes. Re-requesting review.

[Copilot]

Copilot review overview

🟡 Changes recommended

The bulk-close path can discard tracked work, and the implementation does not fully match its documentation or acceptance criteria.

Review effort: Balanced
Findings: 1

Open (1)

• Classify corrections separately from listing additions · New

Resolved since last review (1)

• Do not classify existing-entry fixes as new listings

Previously missed (2)

In code that hasn't changed since last review

Do not close #4868 before completing its acceptance criteria

.claude/​skills/​issue-triage/​SKILL.md:34

The PR declares Closes #4868, but this delegates the one-time sweeps to #4876/#4877 while #4868's acceptance criteria require the current issue backlog to be triaged to a zero audit and the outside-PR backlog to be handled. The PR description confirms those actions were not performed. Either keep #4868 open/change the closing relationship, or complete its acceptance criteria before merging.

This issue also appears on line 408 of the same file.

Reconcile archived-server routing with the documented PR response plan

.claude/​skills/​issue-triage/​SKILL.md:407

This routes archived-server PRs to archived.md, but the updated contribution plan (lines 146–151) and the General response heading (line 469) say every PR class except listings/new servers receives the General PR response. The archived text omits the outside-PR policy and issue flow, so this class will send a different response than the documented plan. Add a PR-specific response combining the policy and archive redirect, or explicitly reconcile the plan and routing.

[cliffhall]

Copilot round 4: 1 inline finding plus 2 "previously missed" findings.

• Listing row in the plan (inline): fixed in 472a2ec, and replied in the thread.
• Archived-server PR routing vs the plan (missed, line 407): fixed in 472a2ec by reconciling the docs. The plan's canned-response pointer and the skill's General, PR heading now both say archived-server PRs get the Archived pointer. It sends the author to where that server now lives, which the general "open an issue here" text would get wrong.
• Closes Add issue-triage skill and board audit, for community inflow #4868 vs deferred acceptance criteria (missed, again): declined, for the reason given in rounds 2 and 3. It's a maintainer decision, tracked in Tracker: Triage the open issue and PR backlog #4875/Triage the open issue backlog #4876/Triage the open PR backlog #4877, and Add issue-triage skill and board audit, for community inflow #4868 is closed by hand.

Re-requesting review.

[Copilot]

Copilot review overview

🟡 Changes recommended

The audit exempts currently forbidden draft cards, a close recipe masks failures, and linked acceptance work remains deferred.

Review effort: Balanced
Findings: 1

Open (1)

• GHSA draft carve-out violates board rules · New

Resolved since last review (1)

• Classify corrections separately from listing additions

Previously missed (2)

In code that hasn't changed since last review

PR does not satisfy linked issue acceptance criteria

.claude/​skills/​issue-triage/​SKILL.md:29

The linked #4868 acceptance criteria require running the current issue triage until every audit row is zero and handling the outside-PR backlog, but the PR description reports 217 unboarded issues and explicitly defers both passes to #4876/#4877. As written, this PR therefore cannot satisfy the issue it declares it closes; complete the human-gated actions before merge or revise the linked issue/closing scope.

Fallback masks failed issue closure

.claude/​skills/​issue-triage/​SKILL.md:88

This fallback masks a failed gh issue close: because echo succeeds, the compound command returns status 0, so a batch caller can continue despite the stop-on-first-failure rule. Preserve a nonzero status for a missing response or failed close.

[cliffhall]

Copilot round 5: 1 inline finding plus 2 "previously missed" findings.

• Fallback masks a failed issue close (missed, line 88): fixed in cd33971. The fallback now ends in false, so a missing response or a failed close leaves a non-zero status.
• [GHSA- carve-out ahead of the board rule (inline): declined, with the reasoning in the thread. Agentic software factory Part 10: security-advisory skill; rewrite SECURITY.md #4905 (Part 10) adds the exception to AGENTS.md/board-ops and explicitly hands the audit carve-out to this PR. The exemption is narrow and has its own replacement check, and no such drafts exist today.
• Closes Add issue-triage skill and board audit, for community inflow #4868 vs deferred acceptance criteria (missed, again): declined, as in rounds 2–4. It's a maintainer decision, tracked in Tracker: Triage the open issue and PR backlog #4875/Triage the open issue backlog #4876/Triage the open PR backlog #4877.

Re-requesting review.

[Copilot]

Copilot review overview

🔵 Needs a closer look

The close loop can lose harvested work, documentation contains contradictory guidance, and linked acceptance criteria remain deferred.

Review effort: Balanced
Findings: None

Resolved since last review (1)

• GHSA draft carve-out violates board rules

Previously missed (6)

In code that hasn't changed since last review

Complete backlog sweeps or correct completion linkage

.claude/​skills/​issue-triage/​SKILL.md:34

This explicitly defers the one-time sweeps to #4875, but the PR still closes #4868, whose acceptance criteria require the current issue backlog to reach a zero audit and the outside-PR backlog to be handled. The PR description confirms neither action was performed; either complete those criteria before closing #4868 or update the issue/PR linkage so this implementation does not mark the broader work complete.

Reject '-' for duplicate and keep classes before writing

.claude/​skills/​issue-triage/​SKILL.md:410

The close loop permits - for duplicate and keep, then selects the untracked response and closes the PR without a harvest issue. That violates the class table and can discard a valuable fix if the reviewed manifest contains a missing issue number; reject - for these two classes before any write.

Audit Priority on all board items, including closed cards

.claude/​skills/​issue-triage/​SKILL.md:612

The authoritative rule says every board item has a Priority (AGENTS.md:267), and this audit's invariant says the same, but isopen(.n) excludes closed/Done cards. The audit can therefore print zero while a board item violates the rule; check all issue cards, or explicitly change the rule and audit documentation if closed cards are intended to be exempt.

Cover labels on both form and non-form issue arrivals

.claude/​skills/​issue-create/​SKILL.md:194

This paragraph still says every UI-created issue arrives without v2 or a type label, but the stacked bug and feature forms apply both labels (.github/ISSUE_TEMPLATE/1-bug_report.yml:14 and 2-feature_request.yml:14). Describe both form and non-form arrivals so the issue-create and triage skills agree.

Clarify that score details are public issue comments

.claude/​skills/​issue-triage/​SKILL.md:227

This says reporters never see the score, but the Recording the score section posts the Priority and full arithmetic as a public issue comment. Clarify that only the board fields are private so triagers do not mistake the score explanation for private data.

Add Archived server to the class table

docs/​contribution-model.md:96

The plan requires each PR to be assigned exactly one class, but this table still omits the newly documented Archived server class. A reader following the plan can therefore route archived-server PRs through New server or Out of scope and post the wrong response; add the class here as well as in the skill.