feat(security): vendor Semgrep r/all ruleset with tracked exclusions

[scottschreckengaust]

Closes #211: vendor the Semgrep r/all ruleset locally so every scan (pre-commit, CI, mise) runs against a single pinned, offline-capable ruleset with exclusions baked in, instead of fetching r/all live on every run.

Related

• Closes RFC: Vendor Semgrep r/all rules locally with tracked exclusions #211 (RFC) — see that issue for the bug-bounty-probe finding that expanded this effort.
• Pre-existing local pre-commit failures surfaced during this work are tracked separately in Pre-existing pre-commit run --all-files failures (tracking) #213 (sub-issues pre-commit: 12 code_templates/*.py files fail Python-parse hooks (check-ast, ruff-format) #214–pre-commit: bandit hook fails (exit 2) — missing .bandit-baseline.json #219); they are not introduced by this PR.

Changes

New tools/semgrep/ (goals A/B/C):

• rules-vendored.yaml — pre-filtered rules; the only file semgrep loads. Script-generated byte-for-byte from r/all minus excluded rules. Byte-preserving (removes whole rule blocks, never re-serializes YAML — re-serializing was verified to silently break rules) and deterministic (sorts rules by id, so a regeneration with no upstream change yields an empty diff).
• rules-status.toml — human-owned source of truth: explicit [rules."id"] decisions (status = excluded/active, pr, reason). Every exclusion is listed individually. The script only reads it, so hand-written # comments are preserved.
• rules-update.py — stdlib-only update mechanism. Downloads r/all, drops excluded rules, rewrites rules-vendored.yaml, and reports derived new/orphaned rules.

Explicit exclusions (no implicit dropping):

• A rule is excluded only when a human lists it in rules-status.toml with status = "excluded". Nothing is dropped implicitly, so a rule appearing in a future r/all can never be silently excluded — it surfaces as new for triage.
• The list includes 79 bug-bounty-probe rules that leaked into r/all (SSRF-via-metadata, stored-XSS in message/fix, command-injection/secret-exfil tests, PWNED-PRODUCTION-RULE, and a rule impersonating a real id). These all carry source = https://semgrep.dev/r/None (no canonical registry source) in one contiguous r_id burst and never appear in semgrep/semgrep-rules. See RFC: Vendor Semgrep r/all rules locally with tracked exclusions #211 for the full provenance analysis. They are listed explicitly (rather than auto-excluded by source) so a legitimate rule that ever carried /r/None could not be silently dropped.
• When an excluded rule is later removed upstream, rules-update.py reports it as an orphaned state on the console — a human then prunes the rules-status.toml entry or keeps it. Nothing auto-deletes. (new and orphaned are derived statelessly from the prior rules-vendored.yaml + rules-status.toml; no state file is stored.)

PR provenance (from git history): the exclusions carry their real originating PR — 21 from #11 (2026-02-07), 1 from #89 (OpenAI), 1 from #200 (bbp-pattern-inject, a broken rule that made semgrep exit 2), and 78 from this PR (the remaining probe rules).

Integration points:

• mise.toml: add SEMGREP_RULES env (bare semgrep uses vendored rules with zero flags) + semgrep:update task; drop the 22 --exclude-rule flags.
• CI (security-scanners.yml) + pre-commit hook: --config=tools/semgrep/rules-vendored.yaml; drop exclude flags. Baseline/SARIF logic unchanged. All three invocations aligned on --verbose --max-log-list-entries=0 (log verbosity only; see the flag-alignment comment below).
• .semgrepignore: skip tools/semgrep/*.yaml (rules would self-match).
• .gitleaks.toml: allowlist the active file (contains secret-detection regexes).
• .pre-commit-config.yaml: exclude the generated files from content/whitespace/large-file hooks so autofix hooks never corrupt the byte-for-byte ruleset. rules-update.py/rules-status.toml stay in scope.
• Remove the stub .semgrep.yaml (never loaded by Semgrep; misleading).

Hardening: GHAS code-scanning findings in rules-update.py (dynamic-urllib/CWE-939, string-concat-in-list) were remediated with real code changes — an HTTPS-only OpenerDirector that makes file:///ftp:// unreachable, and explicit string concatenation — not suppressions.

Result

3005 active / 126 excluded blocks (101 distinct excluded ids: 21 PR#11, 1 PR#89, 1 PR#200, 78 PR#220; the remaining excluded blocks are duplicate ids the r/all feed reuses). The real subprocess-shell-true rule survives while its /r/None impersonator is dropped, showing the exclusion list discriminates valid from invalid.

Verification

• mise run build passes: lint (0 errors), fmt:check, validate, bandit ("No issues"), gitleaks (no leaks), checkov, grype. All CI checks green (24/24).
• Vendored ruleset validated against Semgrep 1.167.0: parses, loads, and scans with no fatal errors (exit 0, 0 parse errors); excluded rule IDs confirmed absent; retained rule blocks confirmed byte-identical to the upstream download.
• rules-update.py is deterministic: two consecutive runs produce byte-identical output.
• Packed size ≈ 869 KB (matches the RFC's ~875 KB estimate).

Note: GitHub push protection flagged a Slack-webhook placeholder (T00000000/B00000000/XXX…) that is part of the detected-slack-webhook rule's own pattern-not matching logic — a false positive accepted via the repo's push-protection unblock. The ruleset was intentionally not modified, to keep it byte-for-byte faithful to upstream.

Merge

This branch is intended to be merged with "Squash and merge". The commit history is intentionally granular (it records the iterative design, including an [auto_exclude] approach that was later replaced with explicit listing); squashing collapses it into a single clean commit on main.

Acknowledgment

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of the project license.

Generated with Claude Code

[scottschreckengaust]

Aligned semgrep flags across the three invocations (b50735b)

Coordinated the cosmetic log/verbosity flags so all three semgrep entry points behave consistently. These flags affect stderr verbosity only — never findings or exit code (verified empirically: --max-log-list-entries=2 prints <SKIPPED DATA (too many entries…)>, =0 prints the full rule list, i.e. 0 = disable truncation, show all).

Before → after

Flag	mise security:semgrep	CI security-scanners.yml	pre-commit hook
verbosity	--quiet → --verbose	--verbose (unchanged)	--verbose (unchanged)
--max-log-list-entries=0	✅ (was a no-op under --quiet)	✅ (unchanged)	❌ → ✅ added

Net: all three now use --verbose --max-log-list-entries=0.

Notes / gotchas

• --quiet and --verbose are mutually exclusive in semgrep (it exits with an error), so adding --verbose to the mise task required dropping --quiet. Confirmed the swap changes neither exit code nor findings — only output volume. mise run security:semgrep is now verbose locally.
• Under the old --quiet, mise's --max-log-list-entries=0 did nothing (quiet already suppresses those lists); it's now meaningful.
• pre-commit was --verbose without the flag, so with the ~3000-rule vendored config it printed <SKIPPED DATA> for the rule list instead of the full list — now consistent with CI.

Intentional differences that remain (by design, not inconsistencies)

• --baseline-commit — mise runs a full local scan; CI ($BASELINE_SHA on PRs) and pre-commit (main) diff against a baseline so only new findings block.
• --sarif-output / --error — only CI and pre-commit emit SARIF / gate on findings; the mise task is interactive.

[isadeks]

Excellent, carefully-reasoned PR. Vendoring r/all with explicit-over-implicit exclusions, fail-loud regeneration, and byte-preserving determinism is the right design, and the security rationale (probe-rule removal + supply-chain pinning) is well-substantiated. Reviewed the script + all integration files against the final HEAD tree.

Correctness — verified

• Integration paths are internally consistent: all final files reference the renamed rules-vendored.yaml / rules-status.toml / rules-update.py. The intermediate-commit names (r-all.active.yaml, update.py, exclusions.toml, EXCLUSIONS.md, rule-state.json) visible in gh pr diff --patch are all cleaned up before HEAD — the final tree has exactly 4 files with no dangling references. ✓
• .gitleaks.toml allowlists both the new and old filenames (deliberate, to cover branch history / #224). ✓
• Byte-preserving split is sound: 3005 ^- blocks = 3005 distinct ids; the 14 rules leading with - patterns:/- fix: correctly resolve id: via the later-line regex. ✓
• .semgrepignore filters scan targets not the --config file, so the ruleset still loads while being excluded from self-match. ✓
• HTTPS-only OpenerDirector genuinely closes CWE-939 (no FileHandler/FTPHandler → non-HTTPS has no dispatch handler), backed by an explicit scheme guard. Real fix, not a suppression. ✓
• Fail-loud aborts before the tracked file is written; output written only after all parsing succeeds; new/orphaned derivation is stateless and bootstrap-safe. ✓

Suggestions (minor, non-blocking)

1. Determinism has a latent gap on duplicate active ids. Sort key is id-only (rb[0]) with a stable sort, so same-id blocks fall back to the registry's nondeterministic order. Harmless today (no active dups — the dups are all excluded). Tie-break on content to make the guarantee total: active_blocks.sort(key=lambda rb: (rb[0], rb[1])).
2. Split assumes rule content never begins a column-0 - line (e.g. inside a message/fix block scalar). Fail-loud rule_id_of prevents silent corruption, so this is a resilience note — worth a one-line comment acknowledging the assumption.
3. security:semgrep switched --quiet → --verbose (intentional flag alignment; noting the local-output change).

Risks / tradeoffs

• +6 MB / 171k committed lines is the core RFC tradeoff (offline + pinning + supply-chain control vs. size). Deliberate; byte-preserve + sort-by-id keeps no-op regenerations at zero diff, which is the right mitigation.
• Maintenance is manual by design (semgrep:update + human triage of new/orphaned) — correct for a security ruleset, clearly documented.
• No automated test for rules-update.py; a small fixture test of split_rules/rule_id_of would guard the parser against future regressions. Acceptable for a manually-run tool.

Verification is thorough (24/24 CI, validated against Semgrep 1.167.0, two-run determinism, real-vs-impersonator discrimination). Approving — suggestions are optional polish; only the duplicate-id tie-break is worth considering before the next r/all refresh.

🤖 Generated with Claude Code

[theagenticguy]

Continued review — verified against the new head (8b2010b0), fresh checkout.

The review-nit commit addresses all four items, and I confirmed each against the actual tree rather than the commit message:

Item	Status
rules-status.toml header named renamed/deleted files (exclusions.toml, r-all.active.yaml, EXCLUSIONS.md, update.py)	✅ Fixed — header now names rules-status.toml / rules-vendored.yaml / rules-update.py; EXCLUSIONS.md references dropped (new/orphaned are console-reported).
Determinism gap on duplicate active ids	✅ Fixed — sort is now key=lambda rb: (rb[0], rb[1]), a total tie-break on block text.
split_rules column-0 - assumption	✅ Documented — docstring now states the assumption and that a mis-split aborts loudly via rule_id_of rather than corrupting silently.
security:semgrep --quiet → --verbose	✅ Correctly identified as intentional flag alignment; no change needed.

Ground-truth re-checks at 8b2010b0:

• rules-vendored.yaml is byte-identical to the prior head — the tie-break is a no-op today (0 duplicate active ids; the dups are all excluded), exactly as the commit claims.
• Exclusions still fully honored: 101 excluded ids, 0 leak into the 3,005-rule vendored file; blocks remain id-sorted.
• No stale filename references survive in any active config or doc (the r-all.active.yaml entry in .gitleaks.toml is the intentional history allowlist).
• 24/24 CI checks green.

Design and implementation are sound: explicit-over-implicit exclusions, byte-preserving + deterministic regeneration, fail-loud download, and a real CWE-939 remediation (HTTPS-only OpenerDirector) rather than a suppression. From a technical standpoint this is ready to merge.

Posting this as a review comment rather than a formal approval — the approving review here is a required-reviewer gate I'm deferring to @lalsaado to cast.

_{Reviewed by Bonk (Laith's collective agent teammate).}

[scottschreckengaust]

@isadeks thanks for the thorough review — all three suggestions are taken care of in 8b2010b:

1. Duplicate-active-id determinism gap — sort now tie-breaks on block content: active_blocks.sort(key=lambda rb: (rb[0], rb[1])). Verified the vendored file is byte-identical afterward (latent today — 0 active dups, the dups are all excluded).
2. split_rules column-0 - assumption — added a docstring note stating the assumption and that a mis-split aborts loudly via rule_id_of rather than corrupting silently.
3. security:semgrep --quiet → --verbose — confirmed intentional (flag alignment across the three invocations); no change needed.

Also folded in the related nit: rules-status.toml's header comments named the old files (exclusions.toml / r-all.active.yaml / EXCLUSIONS.md / update.py) and now name the files that actually exist.

On your optional note about a split_rules/rule_id_of fixture test — agreed it'd guard the parser; I've left it out to keep this PR scoped, and it's a good follow-up.