Summary
test_repair_2_consumer_parser_outputs_match_frozen_golden in
scripts/test_validate_zsh_standard_policy.py hashes the parsed output of seven
consumer surfaces into a single frozen digest:
.github/instructions/zsh-scripting.instructions.md
.github/agents/zsh-plugin-standard-reviewer.agent.md
.github/skills/new-zsh-plugin/SKILL.md
.github/skills/new-zsh-plugin/templates/plugin.plugin.zsh
.github/skills/zunit-test/SKILL.md
PATTERNS.md
.github/README.md
Any edit to any of those files changes the digest and fails
Validate Agent Instructions. That is the test working as intended. The problem
is that nothing says so anywhere a person would look first, and the failure
itself is two bare SHA-256 strings:
AssertionError: '607f548e...' != '905120ed...'
There is no guidance in AGENTS.md, runbooks/, decisions/, or
.github/instructions/; grep -ri golden across all four returns nothing.
Evidence
The test file has been amended nine times. #582, a documentation change to a
skill file, refreshed the digest as a follow-up commit. #586 hit it again while
appending one section to PATTERNS.md. In both cases the regeneration step was
discovered by failing CI rather than known in advance.
Suggested fix
Attach the explanation to the assertion itself with a msg= argument, so the
seven inputs and the regeneration step appear in the failure output where
someone is already looking. That covers all seven surfaces at once, rather than
a per-file comment in each.
A note in runbooks/instruction-update.md is the alternative, but the failure
message reaches the person at the moment they need it and the runbook does not.
Summary
test_repair_2_consumer_parser_outputs_match_frozen_goldeninscripts/test_validate_zsh_standard_policy.pyhashes the parsed output of sevenconsumer surfaces into a single frozen digest:
Any edit to any of those files changes the digest and fails
Validate Agent Instructions. That is the test working as intended. The problemis that nothing says so anywhere a person would look first, and the failure
itself is two bare SHA-256 strings:
There is no guidance in
AGENTS.md,runbooks/,decisions/, or.github/instructions/;grep -ri goldenacross all four returns nothing.Evidence
The test file has been amended nine times. #582, a documentation change to a
skill file, refreshed the digest as a follow-up commit. #586 hit it again while
appending one section to
PATTERNS.md. In both cases the regeneration step wasdiscovered by failing CI rather than known in advance.
Suggested fix
Attach the explanation to the assertion itself with a
msg=argument, so theseven inputs and the regeneration step appear in the failure output where
someone is already looking. That covers all seven surfaces at once, rather than
a per-file comment in each.
A note in
runbooks/instruction-update.mdis the alternative, but the failuremessage reaches the person at the moment they need it and the runbook does not.