fix(ci): move eval report to workflow_run for fork PR support#662
Closed
morluto wants to merge 4 commits intogarrytan:mainfrom
Closed
fix(ci): move eval report to workflow_run for fork PR support#662morluto wants to merge 4 commits intogarrytan:mainfrom
morluto wants to merge 4 commits intogarrytan:mainfrom
Conversation
…roid
## Problem
Droid scans ~/.factory/skills/ for skill directories. It follows *relative*
symlinks pointing to ../../.agents/skills/{skill}, but ignores absolute paths.
When setup ran with --host factory, it created absolute symlinks:
ln -snf /Users/will/gstack/.factory/skills/gstack-qa ~/.factory/skills/gstack-qa
Droid ignored these entirely — no skills appeared in the '/' menu.
## Root Cause
Droid's filesystem scanner only resolves relative symlinks. Absolute symlinks
pointing to paths outside ~/.agents/skills/ are silently skipped.
## Reproduction
1. Clone gstack: git clone https://github.com/garrytan/gstack.git ~/gstack
2. Run setup: cd ~/gstack && ./setup --host factory
3. Check symlinks: ls -la ~/.factory/skills/ | grep gstack
4. Observe: gstack symlinks are absolute paths — WRONG
5. In Droid, type '/' — no gstack skills visible
## Fix
Change symlink creation from absolute to relative paths:
ln -snf "../../.agents/skills/$skill_name" "$target"
Also removes the 'gstack' skip that prevented linking the root skill.
## Validation
After fix:
- ls ~/.factory/skills/gstack-qa points to '../../.agents/skills/gstack-qa'
- Droid '/' menu shows gstack skills after restart
- Verified all working skills (debug, fix, test) use same relative pattern
## Problem
Skills were installed to ~/gstack/.factory/skills/ (a project subdirectory).
Droid never scans this location — it only scans ~/.factory/skills/ and reads
skill files from ~/.agents/skills/.
Even with correct relative symlinks in ~/.factory/skills/, Droid could not load
skills because the actual skill files didn't exist in ~/.agents/skills/.
## Root Cause
Droid's skill loading requires skill files to exist in ~/.agents/skills/.
Symlinks in ~/.factory/skills/ point to skill locations, but the files must
actually be present at the target.
## Reproduction
1. Run: cd ~/gstack && ./setup --host factory
2. Check: ls ~/.agents/skills/ | grep gstack
3. Observe: no gstack directories — SKILL.md files are in ~/gstack/.factory/skills/
4. Droid reports skills as unavailable
## Fix
Copy skills as real directories into ~/.agents/skills/:
mkdir -p "$HOME/.agents/skills"
for skill_dir in "$factory_dir"/gstack*/; do
cp -r "$skill_dir" "$HOME/.agents/skills/"
done
This ensures Droid can read the actual SKILL.md files.
## Validation
After fix:
- ls ~/.agents/skills/ shows all 31 gstack skill directories
- Each directory contains SKILL.md and supporting files
- Droid can read skill metadata from ~/.agents/skills/
…=github
## Problem
Droid maintains a skill registry at ~/.agents/.skill-lock.json. It ONLY loads
entries where sourceType=github. Entries with sourceType=local are silently ignored.
The old setup registered skills with sourceType=local — producing dead lockfile
entries that Droid never read.
## Root Cause
Droid's lockfile reader checks 'sourceType' to determine if a skill should be
loaded. No working Droid skill uses sourceType=local — it's a no-op.
## Reproduction
1. Run: cd ~/gstack && ./setup --host factory
2. Check lockfile: python3 -c "import json; l=json.load(open('/Users/will/.agents/.skill-lock.json')); print({k:v['sourceType'] for k,v in l['skills'].items() if 'gstack' in k})"
3. Observe: all gstack entries have sourceType=local — NOT LOADED
4. Compare: working skills (debug, fix, test) have sourceType=github
## Fix
Register skills with sourceType=github in ~/.agents/.skill-lock.json:
lock['skills'][entry] = {
'source': 'gstack/' + entry,
'sourceType': 'github',
'sourceUrl': 'file://' + os.path.join(agents_skills, entry),
...
}
This matches the format of all working Droid skills.
## Validation
After fix:
- python3 -c "import json; l=json.load(open('/Users/will/.agents/.skill-lock.json')); print(sum(1 for v in l['skills'].values() if v.get('sourceType')=='github'))"
- Shows count of github-sourced skills includes gstack
- Droid '/' menu shows gstack skills (e.g. /qa)
- Skill name comes from 'name:' field in SKILL.md, not directory name
The report job uses gh pr comment which requires write access to the base repo. Fork PRs triggered by pull_request get a read-only GITHUB_TOKEN, causing 'Resource not accessible by integration (addComment)'. Split the report into a separate workflow_run-triggered job that always runs in the base repo context with write access.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Fork PRs trigger the
reportjob viapull_requestevent, which givesGITHUB_TOKENread-only access to the base repo. Thegh pr commentstep fails with:Fix
Split the report job into a separate
evals-report.ymlworkflow triggered byworkflow_run:evals.yml— keepspull_requesttrigger, runs evals in Docker (safe for untrusted fork code), drops thereportjobevals-report.yml— triggered byworkflow_runafter evals complete, runs in base repo context with full write access, posts the PR commentThis fixes the comment failure for all fork PRs, not just this one. No security tradeoffs — untrusted code never runs with elevated permissions.