feat(scanner): enable SkillSpector LLM semantic pass (Anthropic Claude)#10
Merged
Conversation
Document the new llm.provider config knob (default nv_build) and the workflow contract: empty flags + workflow appends --no-llm dynamically when the matching credential secret is unset. Removes --no-llm from the static flags list now that the workflow drives it. This commit was prepared with help from Coder Agents.
Document what flipping LLM mode on does (and does not do) to the verdict math, the precision delta we expect, and what to expect for the five in-tree skills. Adds "LLM provider changes" to the "When to revisit" list. This commit was prepared with help from Coder Agents.
Update step 3 of the architecture summary to reflect that the scheduled scan now runs SkillSpector with the LLM semantic pass on by default. Add a new "One-time setup on the repo" section that lists the three repo-level configurations needed for a useful scan, including the new LLM credential secret. Mirror the LLM secret note into "Forking for your own catalogue". This commit was prepared with help from Coder Agents.
There was a problem hiding this comment.
Pull request overview
Updates the scanner configuration and documentation to support running NVIDIA SkillSpector with an optional LLM semantic pass (with a fallback to static-only mode), as part of the scheduled scan pipeline described in this repo.
Changes:
- Extend
config.yamlwith ascanners.skillspector.llmconfiguration block and makescanners.skillspector.flagsempty by default. - Document the LLM semantic pass behavior, expected precision delta, and repo one-time setup steps in
README.mdanddocs/CALIBRATION.md. - Shift responsibility for driving
--no-llmto the scheduled workflow (though the workflow change itself is not included in this PR’s diff).
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 7 comments.
| File | Description |
|---|---|
| README.md | Updates architecture and adds one-time setup guidance, including LLM credential setup. |
| docs/CALIBRATION.md | Adds an “LLM semantic pass” section explaining expected effects and limitations. |
| config.yaml | Adds scanners.skillspector.llm block and removes the default --no-llm flag from config-driven flags. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Swap the default LLM provider from nv_build (free NVIDIA Build) to anthropic with model pinned to claude-sonnet-4-6. Rationale: - Removes the second-vendor signup. The Coder org already has an Anthropic billing relationship, so the credential is one secret away from working. - Sonnet 4.6 is roughly 5x cheaper than the anthropic default (Opus 4.6) and is well matched to SkillSpector's LLM pass, which is finding-by-finding intent classification rather than long-form reasoning. Cost ballpark for 5 skills x 4 scans/day is small. - The other provider options (anthropic_proxy via Vertex, openai via any OpenAI-compatible gateway, nv_build) stay documented in the config comments and are still a one-line swap. This commit was prepared with help from Coder Agents.
Follow-up to the provider swap. The one-time-setup section now points at console.anthropic.com and ANTHROPIC_API_KEY instead of build.nvidia.com / NVIDIA_INFERENCE_KEY. This commit was prepared with help from Coder Agents.
DevelopmentCats
changed the title
Addresses copilot-pull-request-reviewer review on config.yaml line 46
and line 57: the previous comments described workflow behavior as
implemented ("the workflow appends --no-llm dynamically") when in
fact .github/workflows/scan.yaml still hardcodes --no-llm in this PR.
The new wording describes the contract between this file and scan.yaml
and explicitly notes that the matching workflow edit is committed
separately because the Coder Agents GitHub App lacks `workflows: write`.
Also swaps the provider from anthropic/claude-sonnet-4-6 to
openai/gpt-4.1-mini. The anthropic provider hardcodes api.anthropic.com
and ignores ANTHROPIC_BASE_URL, so it cannot route through Coder's
aibridge. The openai provider does, and gpt-4.1-mini was empirically
validated against the five in-tree skills (results in CALIBRATION.md).
This commit was prepared with help from Coder Agents.
Addresses copilot-pull-request-reviewer review on docs/CALIBRATION.md line 121: the LLM section described auto-enablement based on the credential secret, but scan.yaml in this branch still hardcodes --no-llm and that contract is not in effect yet. This pass: - Adds a measured table showing the five-skill before/after under LLM mode (was upstream's hand-wavy 70%-to-87% precision estimate). Catalogue-wide findings: 25 to 2; coder/setup: malicious to clean. - Adds an explicit "workflow gap" callout explaining that scan.yaml still hardcodes --no-llm in this branch and pointing readers at the PR description for the matching diff. - Documents the provider choice in plain language: anthropic provider cannot be steered at aibridge because it hardcodes api.anthropic.com and ignores ANTHROPIC_BASE_URL; openai provider works and gpt-4.1-mini is the cost/quality sweet spot. - Drops the previous "bringing coder/setup below suspicious requires the permissions-manifest layer" framing. With LLM mode on, coder/setup is already clean. This commit was prepared with help from Coder Agents.
…aibridge Addresses copilot-pull-request-reviewer reviews on README.md (architecture step 3 at line 14, the one-time setup block, the graceful-fallback paragraph, and the forking step 5 at line 125): the text described workflow behavior that is not implemented in this branch because .github/workflows/scan.yaml still hardcodes --no-llm. Fixes: - Add a note block right after the architecture summary spelling out that step 3's LLM behavior requires the matching scan.yaml edit, and pointing readers at the PR description for the diff. - Update step 3 of the one-time-setup section to set the OpenAI provider's secret and add OPENAI_BASE_URL as a variable. Adds a second note block immediately after the setup list flagging the workflow-file dependency again at point-of-use. - Update forking instructions step 5 to point at the new setup section and call out that confirming scan.yaml exports the secret is required for LLM mode. - Swap the API-key reference from console.anthropic.com / ANTHROPIC_API_KEY to a Coder AI Gateway token / OPENAI_API_KEY. The anthropic provider in SkillSpector hardcodes api.anthropic.com and ignores ANTHROPIC_BASE_URL, so it cannot be steered at aibridge; the openai provider works. This commit was prepared with help from Coder Agents.
DevelopmentCats
changed the title
SkillSpector's anthropic provider hardcodes api.anthropic.com and ignores ANTHROPIC_BASE_URL, and aibridge does not mount /v1/chat/completions on its /anthropic path (only the native /v1/messages shape). Routing SkillSpector at Claude through aibridge is therefore not possible today without either patching SkillSpector or adding a new route to aibridge. Use the Anthropic API directly instead. This trades the aibridge billing-line consolidation for a Claude-class model; the choice and the trade-off are documented in docs/CALIBRATION.md.
…RL var Three-step one-time setup now: Pages, workflow permissions, and a single secret (ANTHROPIC_API_KEY from console.anthropic.com). The OPENAI_BASE_URL variable is no longer used since the provider is anthropic against api.anthropic.com directly. The workflow-file note spells out the matching workflow env block (SKILLSPECTOR_PROVIDER, SKILLSPECTOR_MODEL, ANTHROPIC_API_KEY) that has to land in scan.yaml.
… caveat The measured 25 -> 2 false-positive reduction was captured against gpt-4.1-mini through aibridge during development; production hits claude-sonnet-4-5 via the Anthropic API directly. Verdict-band outcomes are robust to the model swap because every non-coder/setup in-tree skill scores well below the 51 suspicious cutoff even without LLM filtering, but the per-finding counts may shift one or two either way. Recalibration follow-up planned once production data lands. Also captures the four reasons aibridge cannot route SkillSpector at Claude today.
DevelopmentCats
changed the title
…-4-6 sonnet-4-6 is the bundled meta_analyzer-slot default in SkillSpector's anthropic provider — the slot whose docstring explicitly describes it as 'cheaper for the high-volume filter pass,' which is exactly the per-finding intent classification this scanner does. It also widens the context window from 200K (Sonnet 4.5) to 1M, reducing chunking for larger skills, and SkillSpector's bundled model_registry.yaml already knows its token limits so there is no runtime fallback warning. Bare alias used rather than a dated suffix: probing Anthropic's /v1/models endpoint on 2026-06-23 confirmed every dated suffix for the 4-6 series (e.g. claude-sonnet-4-6-20251015) returns 404, while the bare alias returns 200. The bare alias is the canonical ID Anthropic accepts.
config.yaml: collapse the llm: block to provider + model + a one-line flags: comment. The aibridge backstory, model justification, provider options table, and "contract with scan.yaml" prose all live in docs/CALIBRATION.md and the PR description; reproducing them next to the two values that actually matter is noise. README.md and docs/CALIBRATION.md: drop the two "this PR is in a weird in-between state" callout blocks. The PR description already covers the workflow split; the docs do not need to carry it.
The measured numbers are a point-in-time snapshot — they shift as SkillSpector evolves, as the LLM model swaps, and as upstream skills change. The README should not advertise them as a comparison.
The file was not consumed by SkillSpector, the scanner code, or the workflow; it was a 230-line design-decision retrospective justifying two YAML values. The load-bearing rationale (verdict thresholds map to SkillSpector's HIGH/CRITICAL bands and SkillSpector itself escalates at the same boundary) already lives in the config.yaml comment block right next to the thresholds. Drops the four inline references in scanner/verdict.py, README.md, RiskBar.tsx, and VerdictExplanation.tsx. The docs/ directory is removed because the file was its sole occupant.
The section was instructions for setting up the canonical coder/coder-skill-scanner GitHub repo (Pages source, workflow permissions, ANTHROPIC_API_KEY secret). Those are one-shot operator steps, not user-facing docs about the product, and are tracked outside the repo. The Forking section keeps its own minimal, self-contained equivalent for downstream forks.
…Spector steps Three edits inside the scan job: 1. New 'Determine LLM mode' step (id=llm) that emits extra_flags=--no-llm when ANTHROPIC_API_KEY is unset, and extra_flags= when it is set. Also emits a workflow-level ::warning:: in the unset case so the degraded mode is visible on the run page. 2. SkillSpector (JSON) step: passes SKILLSPECTOR_PROVIDER=anthropic, SKILLSPECTOR_MODEL=claude-sonnet-4-6, ANTHROPIC_API_KEY into env, and uses steps.llm.outputs.extra_flags instead of hardcoded --no-llm. 3. SkillSpector (SARIF) step: same env block, same flag swap. With the secret set on the repo this turns on SkillSpector's LLM semantic pass; without it the workflow falls back to --no-llm.
DevelopmentCats
changed the base branch from
cat/step-1-scanner-pipeline
to
main
coder/registry declares the same upstream skills in two formats:
in-tree as .agents/skills/coder-modules and .agents/skills/coder-templates,
and via registry/coder/skills/README.md frontmatter as sources[].skills
keyed 'setup', 'modules', 'templates' pointing at the coder/skills repo.
scanner/enumerate.py dedupes on (namespace, slug); the different slug
names ('coder-modules' vs 'modules') let both rows survive, so the
catalogue scan sees 5 skills when it should see 3.
The external-sources frontmatter is the canonical declaration in
coder/registry going forward. Drop the in_tree block from this
config so the enumerator only walks the README frontmatter.
in_tree remains supported in scanner/enumerate.py for forks
that maintain an .agents/skills/ layout; they re-enable by adding
the block back to their config.yaml fork.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
Flips the scheduled scan from
--no-llmto SkillSpector's two-stage analyser (static rules + LLM semantic pass).Provider is anthropic against
api.anthropic.comdirectly, modelclaude-sonnet-4-6(SkillSpector's bundledmeta_analyzerslot default; 1M context window; SkillSpector'smodel_registry.yamlalready knows its token limits so no runtime fallback warning). SkillSpector cannot be routed through Coder's AI Gateway today, so the Anthropic API key is on a separate billing line from Coder usage. Short version: SkillSpector pipes every provider throughlangchain_openai.ChatOpenAI(OpenAI/v1/chat/completionsshape), aibridge only exposes that path under/openai, and SkillSpector's anthropic provider hardcodeshttps://api.anthropic.com/v1/and ignoresANTHROPIC_BASE_URL. Empirically verified end-to-end against the workspace's aibridge.Why the LLM pass matters
SkillSpector's static-only pass is loud on real catalogues (~49% of skills get at least one finding per the ClawHub paper). Its LLM pass reads each finding's surrounding context, classifies intent, and filters context-aware false positives. Two
SQP-2findings oncoder/setup(OAuth token and session config written to disk without a user-visible notification) are the only real signal that survived the filter on the in-tree five during development; cleanest fix for those is a one-lineechoupstream incoder/skills, not a change here.Graceful fallback
If the
ANTHROPIC_API_KEYsecret isn't set, the scan workflow emits a warning and runs--no-llm. A fresh fork keeps producing valid (lower-precision) scans without any setup.Decision log
anthropicprovider hardcodesapi.anthropic.comand ignoresANTHROPIC_BASE_URL; SkillSpector pipes every provider throughlangchain_openai.ChatOpenAI(OpenAI/v1/chat/completionsshape); aibridge does proxy Claude, but only in Anthropic's native/v1/messagesshape under its/anthropicpath, and does not mount/v1/chat/completionson that path. Verified empirically:POST $ANTHROPIC_BASE_URL/v1/messagesreturns Claude output;POST $ANTHROPIC_BASE_URL/v1/chat/completionsreturnsroute not supported. Usingopenaiagainst aibridge withgpt-4.1-miniis a viable alternative; the trade-off is real (separate billing line vs Claude-class model), and the provider line flips back toopenaiwithout any workflow change if aibridge later adds the missing route or SkillSpector gains a native-Anthropic integration.claude-sonnet-4-6. SkillSpector's bundledmeta_analyzerslot default; the provider docstring describes that slot as "cheaper for the high-volume filter pass," which is precisely the per-finding intent classification this scanner does. Bumped from an earlierclaude-sonnet-4-5-20250929pick because 4-6 is one generation newer, widens context from 200K to 1M (reduces chunking on larger skills), and is in SkillSpector'smodel_registry.yaml(no runtime fallback warning). Bare alias is used because the dated suffixes for the 4-6 series return 404 from Anthropic's API as of 2026-06-23; the bare alias is the canonical ID. Override inconfig.yamltoclaude-opus-4-6(SkillSpector'sDEFAULT_MODEL, ~5x cost) or toclaude-opus-4-8/claude-fable-5if a higher tier is warranted later.--no-llmfallback. A fresh fork should produce something useful immediately, not 404 the publish pipeline because the operator hasn't added the secret yet. The workflow warning makes the degraded state visible.ci.yamluses inline pytest fixtures and never invokesskillspectorlive, so no inference cost on PR review.coder/skillsrather than justified via a manifest. The manifest design is captured for future use when we onboard third-party skills.This PR was prepared with help from Coder Agents.