fix(lychee): drop excludes that pass on a GitHub runner and pace two hosts

[kyle-sexton]

No related issue: follow-up to #772 from the lychee online sweep, approved by the repository owner.

Summary

A GitHub-runner retest of the shared URL excludes (temporary probe step in claude-code-plugins run 38163276512, removed before merge) passed six of them, so they go. Two hosts that answer a runner's burst with 429 get per-host pacing instead of an accepted 429.

Fix

• Dropped (passed lychee 0.24.2 on the runner): the NTIA SBOM report (the old TLS-trust-store exclude), help.miro.com 31624028247058, isdown.app/status/anthropic, w3.org WorkingWithTimeZones, host-wide linkedin.com (its one cited post passed), host-wide x.com (status posts passed; a made-up status answered 404).
• x.com narrowed: profiles answer 403 to the runner (x.com/KentBeck), so profiles stay excluded with the untested article and terms pages; status posts are checked again.
• Kept with reasons restated: bsky.app (200 for a made-up post on the runner too, so a pass proves nothing), the two Medium articles, npmjs, dev.mysql.com, ACM, ISO, docs.genius.com, cursor.com, lib.rs (all 403 on the runner).
• [hosts."testing.googleblog.com"] and [hosts."jackrighteous.com"]: concurrency = 1, request_interval = "2s", the per-host keys in lychee v0.24.2's lychee.example.toml ([hosts."blog.example.com"], concurrency, request_interval). The no-accept rule stands.
• lychee.test.sh and fixtures/good/Exclusions.md: the dropped URLs are now asserted to be checked; x.com profile and article URLs asserted excluded, a status URL asserted checked.

Verification

• bash components/lychee/lychee.test.sh: 102 PASS, 0 FAIL (the config with [hosts] tables loads in every lychee call there).
• Runner probe results: 11 passes (the 7 URLs behind the six dropped excludes, 3 bsky.app URLs including a made-up post, and a made-up claude.ai path), 14 x 403, 1 x 404 (a made-up x.com status).

Related

• fix(lychee): drop the redundant example-domain exclude and add check-links #772, docs: repair links reported by an online lychee run claude-code-plugins#7199 and #7222.

🤖 Generated with Claude Code

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-11T19:34:41.702300Z	a7fef83	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[claude]

Claude finished @kyle-sexton's task in 2m 10s —— View job

---

Code review complete

• Gather context (PR diff, changed files)
• Review lychee.toml changes
• Review lychee.test.sh changes
• Review fixtures/good/Exclusions.md changes
• Post findings

Summary

Reviewed the three changed files (lychee.toml, components/lychee/lychee.test.sh, components/lychee/fixtures/good/Exclusions.md) against the PR's stated goals: dropping excludes that passed a GitHub-runner retest, narrowing the x.com exclude so status posts are checked again, and adding per-host pacing for testing.googleblog.com and jackrighteous.com.

One finding posted inline:

• lychee.toml:91 — the new '^https?://(www\.)?x\.com/i/' exclude is an unanchored prefix match, so it also excludes x.com/i/status/<id> (X's canonical non-username tweet-permalink form), contradicting this PR's own goal of having status posts checked again. The sibling pattern on line 90 deliberately carves status out (and has a test for it via assert_row_count), but this /i/ line has no equivalent scoping and no test exercises the /i/status/... shape.

Everything else checked out:

• The dropped excludes (Miro, IsDown, w3.org, NTIA, LinkedIn, host-wide x.com) are each backed by a stated runner-retest result and mirrored by new assert_contains checks in lychee.test.sh and corresponding fixture moves in Exclusions.md.
• The [hosts."testing.googleblog.com"] / [hosts."jackrighteous.com"] pacing blocks use valid lychee v0.24.2 [hosts.*] keys and sit correctly outside the exclude array; no accept override was added, consistent with the file's existing no-429-acceptance policy (lines 18-23).
• Test coverage for the new/changed behavior looks thorough aside from the /i/status gap above; author-claimed 102 PASS, 0 FAIL and runner-probe results in the PR description are unverified by me (not independently re-run).

[claude]

Claude finished @kyle-sexton's task in 55s —— View job

---

Security review — PR #775

• Gather context (PR diff, files changed)
• Invoke /review:security-review for PR fix(lychee): drop excludes that pass on a GitHub runner and pace two hosts #775
• Post findings

No security issues found.

This PR changes only lychee.toml (link-checker exclude regexes and new per-host
[hosts.*] rate-limiting entries), components/lychee/lychee.test.sh, and
components/lychee/fixtures/good/Exclusions.md. All three files are link-checker
configuration/test fixtures, not application code, auth/authorization logic, or
input-handling paths — there's no injection surface, credential handling, or
trust-boundary change introduced.

Checked specifically for the instruction-surface-deletion case this lane flags
(a diff that narrows/removes a standing rule): the removed lychee.toml exclude
entries (e.g. the NTIA SBOM, miro, isdown.app, w3.org, and host-wide
linkedin.com/x.com excludes at
lychee.toml#L80-L116)
are link-checker skip rules, not guardrails over agent/CI behavior — removing
them only means lychee checks more URLs online, which is the intended effect of
this PR and not a control bypass. AGENTS.md / rules files are untouched by this
diff, so the protected-class register doesn't apply here.

The new [hosts."testing.googleblog.com"] / [hosts."jackrighteous.com"]
pacing blocks
(lychee.toml#L119-L128)
only set concurrency/request_interval; no accept status codes are added, so
429s still fail the check as the PR body states.

[github-actions]

claude-security-review has reviewed this pull request through 19d3339; a later push is reviewed from there.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a7fef83971

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[github-actions]

claude-review has reviewed this pull request through 19d3339; a later push is reviewed from there.

[claude]

Claude finished @kyle-sexton's task in 1m 18s —— View job

---

Code review complete

• Gather context (incremental diff since a7fef83, prior review comments)
• Review changes in lychee.toml
• Review changes in components/lychee/lychee.test.sh
• Review changes in components/lychee/fixtures/good/Exclusions.md
• Post findings

Summary

Reviewed only what changed since a7fef83 (the incremental diff in .claude-lane/incremental.diff): lychee.toml#L91, and the corresponding additions in lychee.test.sh and Exclusions.md.

This commit narrows the x.com /i/ exclude from '^https?://(www\.)?x\.com/i/' to '^https?://(www\.)?x\.com/i/article/', which is exactly the fix both prior review findings (mine and Codex's on a7fef83) asked for: the unanchored /i/ prefix was silently excluding x.com/i/status/<id> and x.com/i/web/status/<id>, contradicting the PR's goal of having status posts checked again.

Verified by reading (tool execution wasn't available to me in this environment, so I traced the regex and test wiring by hand rather than running the suite):

• The new pattern only matches x.com/i/article/..., no longer touching x.com/i/status/... or x.com/i/web/status/....
• lychee.test.sh adds 'https://x.com/i/status/1' and 'https://x.com/i/web/status/1' to the assert_contains "runner-passing URL is checked..." loop (lychee.test.sh#L197-L198), i.e. asserting they are checked, not excluded — the right direction for this fix. The pre-existing assert_not_contains 'x.com i/ article is excluded' check (line 183, unchanged) still covers x.com/i/article/1 staying excluded.
• Exclusions.md lists both new URLs under "Pages that passed a GitHub-runner retest and are checked again," matching the test assertions. The fixture's section headers are documentation only — lychee.test.sh asserts directly against dump_out/the raw fixture text, not by parsing headers — so the placement is cosmetic but consistent.

No new issues found in this incremental change. No inline comments needed since there's no outstanding finding on the changed lines.

Note: I was not able to execute bash components/lychee/lychee.test.sh or invoke lychee directly in this environment (no tool-execution permission for those commands), so the "102 PASS, 0 FAIL" claim from earlier rounds remains author-claimed and unverified by me — my check here is static (regex tracing + reading the test file), not a run of the suite.

[claude]

Claude finished @kyle-sexton's task in 1m 4s —— View job

---

Security review — PR #775 (incremental, since a7fef83971acd506e3ab2ad96eec785d4d5be3ba)

• Gather context (incremental diff since a7fef83)
• Invoke /review:security-review for PR fix(lychee): drop excludes that pass on a GitHub runner and pace two hosts #775
• Post findings

No security issues found.

This increment (a7fef83...19d3339) touches only lychee.toml, components/lychee/lychee.test.sh, and components/lychee/fixtures/good/Exclusions.md — link-checker configuration and its test fixtures, not application code, auth logic, or any input-handling path. No injection surface, credential handling, or trust-boundary change.

It also directly resolves the gap the two earlier reviews flagged on this PR (the inline Claude review comment and the Codex comment on the previous commit): the unanchored '^https?://(www\.)?x\.com/i/' exclude silently skipped checking X's canonical status-permalink form (x.com/i/status/<id>, x.com/i/web/status/<id>). The fix narrows it to
'^https?://(www\.)?x\.com/i/article/', and I confirmed against the sibling pattern on line 90 that neither regex now matches x.com/i/status/1 or x.com/i/web/status/1 — both fall through to being checked, as intended.

Applying the instruction-surface-deletion lens from the review brief: this isn't a guardrail over agent/CI behavior (not CLAUDE.md/AGENTS.md/a rules file/a hook's allow/deny list), and the net effect of the change is to narrow a skip rule so more URLs get checked — tightening, not loosening, so the protected-class register doesn't apply here.

Test coverage for the fix is present and consistent: lychee.test.sh adds assert_contains checks that both new status-permalink URLs are checked, and Exclusions.md lists them under "Pages that passed a GitHub-runner retest and are checked again."