Skillscope

A test harness for AI agent skills. Install the harness, or point a workflow at your repo. Skillscope enables three main types of tests:

• Structural: Checks whether the skill is well structured (contains the expected files, a proper hierarchy, etc.).
• Routing: Checks whether the skill triggers when it should and stays quiet when it shouldn't. You may also point to more than one skill here to explore how skills interfere with each other's routing.
• Behavioral: Runs the agent end to end and uses an LLM judge to confirm the skill behaves correctly.

All three read one dataset per skill, <skill>/evals/evals.json. How to write one is in docs/authoring-evals.md.

Important

Early Days: Flags, workflow inputs, and report formats may evolve quickly as this repo takes shape.

Quick start

Run from the root of the repo you want tested.

uv tool install --system-certs git+https://github.com/amd/skillscope

skillscope structural                        # no agent, no tokens
skillscope structural --external             # the same, plus checking external URLs
skillscope behavioral --skill my-skill       # needs an authenticated `claude` CLI
skillscope routing --routing-room my-skill,its-neighbour

Run it from your CI

Simply add an evals.json file to your skill and a workflow that points to our reusable workflow. This will triger all graders to run directly from your repo.

jobs:
  evals:
    uses: amd/skillscope/.github/workflows/reusable.yml@v0.1.0
    secrets:
      api_key: ${{ secrets.ANTHROPIC_API_KEY }}
    with:
      skills: skills/*

You may also choose to customize or even disable some tests if you prefer it. See docs/usage.md for details.

Commands

Command	Does
skillscope structural	Skill folders, datasets, and markdown references.
skillscope routing	Which skill fires, with several installed together.
skillscope behavioral	What a skill does once it has fired.
skillscope select	The CI plan for a change, as JSON.
skillscope list-skills	The skills that have a dataset, as JSON.
skillscope template	The dataset template a new skill starts from.

--help on any of them is the reference. Reports go to stdout as markdown, to $GITHUB_STEP_SUMMARY under Actions, and to JSON under .skillscope/runs/ in the repo under test — worth adding to .gitignore.

Docs

• docs/authoring-evals.md: writing a skill's tests.
• docs/usage.md: every flag, every workflow input, and what each check actually asserts.

Development

python -m pip install -e .
python -m unittest discover -s tests -t .

Standard library only, apart from PyYAML for frontmatter, so a graded run installs nothing beyond this package. Tests build a throwaway repo in a temp directory rather than reading the tree they run in.

License

MIT. See LICENSE.