Skip to content

chore(ci): explain how to regenerate the frozen consumer-parser golden #588

Description

@ss-o

Summary

test_repair_2_consumer_parser_outputs_match_frozen_golden in
scripts/test_validate_zsh_standard_policy.py hashes the parsed output of seven
consumer surfaces into a single frozen digest:

.github/instructions/zsh-scripting.instructions.md
.github/agents/zsh-plugin-standard-reviewer.agent.md
.github/skills/new-zsh-plugin/SKILL.md
.github/skills/new-zsh-plugin/templates/plugin.plugin.zsh
.github/skills/zunit-test/SKILL.md
PATTERNS.md
.github/README.md

Any edit to any of those files changes the digest and fails
Validate Agent Instructions. That is the test working as intended. The problem
is that nothing says so anywhere a person would look first, and the failure
itself is two bare SHA-256 strings:

AssertionError: '607f548e...' != '905120ed...'

There is no guidance in AGENTS.md, runbooks/, decisions/, or
.github/instructions/; grep -ri golden across all four returns nothing.

Evidence

The test file has been amended nine times. #582, a documentation change to a
skill file, refreshed the digest as a follow-up commit. #586 hit it again while
appending one section to PATTERNS.md. In both cases the regeneration step was
discovered by failing CI rather than known in advance.

Suggested fix

Attach the explanation to the assertion itself with a msg= argument, so the
seven inputs and the regeneration step appear in the failure output where
someone is already looking. That covers all seven surfaces at once, rather than
a per-file comment in each.

A note in runbooks/instruction-update.md is the alternative, but the failure
message reaches the person at the moment they need it and the runbook does not.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:ciContinuous integration or GitHub Actions work.type:maintenanceNon-feature maintenance, cleanup, or org work.

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions