Add anti-probing guidance to submit_pull_request_review tool description by Copilot · Pull Request #51469 · github/gh-aw

Copilot · 2026-08-08T21:50:15Z

Run analysis showed agents calling submit_pull_request_review with empty args ({}) as a schema-discovery probe, hitting ERR_VALIDATION, then retrying with correct fields — 4 occurrences across 2 unrelated workflows. The tool's description documented its validation rule but lacked the imperative WRITE-ONCE anti-probing preamble that sibling tools (add_comment, create_issue) already carry.

Changes

Anti-probing preamble prepended to submit_pull_request_review's description, matching the wording pattern of add_comment/create_issue.
Inline minimal-call example so the first call has a concrete non-empty template to copy (mirrors what push_to_pull_request_branch does).
Closing sentence reinforcing that the call records a real submission intent, with noop/report_incomplete as the escape hatch.
Applied to both copies of safe_outputs_tools.json (pkg/workflow/js/ compiler copy and actions/setup/js/ runtime copy), which TestSafeOutputsToolsJSONInSync requires to be identical.

WRITE-ONCE: do NOT call this tool with empty or placeholder arguments to probe or
discover its schema — this call must include a non-empty `body`, or be preceded by
at least one create_pull_request_review_comment call; if you are not ready to submit
the real review, call `noop` instead. A minimal valid call looks like
{"body": "Looks good overall.", "event": "COMMENT"}. Submit a pull request review ...

Notes

Descriptions are served by the safe-outputs MCP server at runtime rather than baked into .lock.yml, so make recompile produces no lock-file diffs; the change takes effect on the next workflow run without recompiling.

Effectiveness should be evaluated against subsequent daily optimizer scans — push_to_pull_request_branch shows this preamble reduces but does not eliminate probe calls.

Fixes [safeoutputs] Safe-output tool descriptions: submit_pull_request_review lacks anti-probing guidance (empty-args probe pattern, 4 occurrences/2 [Content truncated due to length] #51467

run: https://github.com/github/gh-aw/actions/runs/31293411688> Generated by 👨‍🍳 PR Sous Chef · gpt54 · 3.76 AIC · ⌖ 6.55 AIC · ⊞ 8.5K · ◷

_{Comment /souschef to run again}

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

github-actions · 2026-08-09T00:40:04Z

PR Triage

Category: docs/chore
Risk: low (2-line change across 2 identical JSON tool-description files)
Score: 30/100 (impact 8, urgency 8, quality 14)
CI: unknown — no check-run data returned yet; PR still draft
Recommendation: defer (low value, but easy/safe) — good batch_review companion once marked ready; trivial anti-probing wording addition, low risk to merge quickly once out of draft.

Generated by 🔧 PR Triage Agent · auto · 51 AIC · ⌖ 2.57 AIC · ⊞ 8K · ◷

Copilot

Pull request overview

Adds anti-probing guidance for pull request review submissions to reduce invalid empty-argument calls.

Changes:

Adds WRITE-ONCE guidance and a minimal-call example.
Keeps compiler and runtime tool definitions synchronized.

Show a summary per file

File	Description
`pkg/workflow/js/safe_outputs_tools.json`	Updates the compiler tool description.
`actions/setup/js/safe_outputs_tools.json`	Mirrors the runtime tool description.

Review details

Tip

Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

Files reviewed: 2/2 changed files
Comments generated: 2
Review effort level: Balanced

  {
    "name": "submit_pull_request_review",
-    "description": "Submit a pull request review with a status decision. By default this tool targets the pull request that triggered the workflow. When the workflow is configured with `target: \"*\"`, you must specify `pull_request_number` to indicate which PR to target. REQUIRED: every call must include either a non-empty body or be preceded by at least one create_pull_request_review_comment call; calling with no body and no prior comments is rejected with ERR_VALIDATION. All preceding create_pull_request_review_comment outputs are automatically attached as inline comments. If this tool is not called, buffered review comments are submitted as a COMMENT review at workflow end. Use COMMENT for non-blocking feedback; use REQUEST_CHANGES only for merge-blocking. Example (inline-only review): call create_pull_request_review_comment one or more times, then call this tool with event: COMMENT and no body.",
+    "description": "WRITE-ONCE: do NOT call this tool with empty or placeholder arguments to probe or discover its schema \u2014 this call must include a non-empty `body`, or be preceded by at least one create_pull_request_review_comment call; if you are not ready to submit the real review, call `noop` instead. A minimal valid call looks like {\"body\": \"Looks good overall.\", \"event\": \"COMMENT\"}. Submit a pull request review with a status decision. By default this tool targets the pull request that triggered the workflow. When the workflow is configured with `target: \"*\"`, you must specify `pull_request_number` to indicate which PR to target. REQUIRED: every call must include either a non-empty body or be preceded by at least one create_pull_request_review_comment call; calling with no body and no prior comments is rejected with ERR_VALIDATION. All preceding create_pull_request_review_comment outputs are automatically attached as inline comments. If this tool is not called, buffered review comments are submitted as a COMMENT review at workflow end. Use COMMENT for non-blocking feedback; use REQUEST_CHANGES only for merge-blocking. Example (inline-only review): call create_pull_request_review_comment one or more times, then call this tool with event: COMMENT and no body. This tool records a real review-submission intent. Do not use it for placeholder reviews, auth checks, or probing. Call it only when the final review is ready; otherwise use noop or report_incomplete.",


  {
    "name": "submit_pull_request_review",
-    "description": "Submit a pull request review with a status decision. By default this tool targets the pull request that triggered the workflow. When the workflow is configured with `target: \"*\"`, you must specify `pull_request_number` to indicate which PR to target. REQUIRED: every call must include either a non-empty body or be preceded by at least one create_pull_request_review_comment call; calling with no body and no prior comments is rejected with ERR_VALIDATION. All preceding create_pull_request_review_comment outputs are automatically attached as inline comments. If this tool is not called, buffered review comments are submitted as a COMMENT review at workflow end. Use COMMENT for non-blocking feedback; use REQUEST_CHANGES only for merge-blocking. Example (inline-only review): call create_pull_request_review_comment one or more times, then call this tool with event: COMMENT and no body.",
+    "description": "WRITE-ONCE: do NOT call this tool with empty or placeholder arguments to probe or discover its schema \u2014 this call must include a non-empty `body`, or be preceded by at least one create_pull_request_review_comment call; if you are not ready to submit the real review, call `noop` instead. A minimal valid call looks like {\"body\": \"Looks good overall.\", \"event\": \"COMMENT\"}. Submit a pull request review with a status decision. By default this tool targets the pull request that triggered the workflow. When the workflow is configured with `target: \"*\"`, you must specify `pull_request_number` to indicate which PR to target. REQUIRED: every call must include either a non-empty body or be preceded by at least one create_pull_request_review_comment call; calling with no body and no prior comments is rejected with ERR_VALIDATION. All preceding create_pull_request_review_comment outputs are automatically attached as inline comments. If this tool is not called, buffered review comments are submitted as a COMMENT review at workflow end. Use COMMENT for non-blocking feedback; use REQUEST_CHANGES only for merge-blocking. Example (inline-only review): call create_pull_request_review_comment one or more times, then call this tool with event: COMMENT and no body. This tool records a real review-submission intent. Do not use it for placeholder reviews, auth checks, or probing. Call it only when the final review is ready; otherwise use noop or report_incomplete.",


gh-aw-bot · 2026-08-09T04:01:06Z

@copilot Please refresh this branch if needed, address any remaining review feedback, and run the pr-finisher skill.

Generated by 👨‍🍳 PR Sous Chef · gpt54 · 3.76 AIC · ⌖ 6.55 AIC · ⊞ 8.5K · ◷
_{Comment /souschef to run again}

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

Copilot · 2026-08-09T04:09:57Z

@copilot Please refresh this branch if needed, address any remaining review feedback, and run the pr-finisher skill.

Generated by [👨‍🍳 PR Sous Chef](https://github.com/github/gh-aw/actions/...

Addressed the remaining review feedback and pushed commit 211062c148.

Qualified the body-omission guidance to APPROVE/COMMENT only (REQUEST_CHANGES requires body).
Scoped the minimal valid example to the triggering PR and noted pull_request_number is required when target: "*".
Kept both safe_outputs_tools.json copies in sync.

github-actions · 2026-08-09T06:37:18Z

PR Triage

Category: chore
Risk: low
Priority: low
Score: 30/100 (impact 10 + urgency 8 + quality 12)
Recommended action: defer

Tiny doc/tool-description tweak, low impact, no urgency.

Generated by 🔧 PR Triage Agent · auto · 58.3 AIC · ⌖ 2.48 AIC · ⊞ 8K · ◷

github-actions · 2026-08-11T16:28:36Z

🎉 This pull request is included in a new release.

Release: v0.86.2

Initial plan

1cf6e2c

Copilot AI assigned Copilot and pelikhan Aug 8, 2026

Copilot started work on behalf of pelikhan August 8, 2026 21:50 View session

Copilot AI linked an issue Aug 8, 2026 that may be closed by this pull request

[safeoutputs] Safe-output tool descriptions: submit_pull_request_review lacks anti-probing guidance (empty-args probe pattern, 4 occurrences/2 [Content truncated due to length] #51467

Closed

5 tasks

Add anti-probing guidance to submit_pull_request_review description

96ff5cf

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Update safe-output tool descriptions with anti-probing guidance~~ Add anti-probing guidance to submit_pull_request_review tool description Aug 8, 2026

Copilot finished work on behalf of pelikhan August 8, 2026 21:59

Copilot AI requested a review from pelikhan August 8, 2026 21:59

github-actions Bot added pr-action:defer pr-agent:copilot pr-priority:low pr-risk:low pr-type:chore labels Aug 9, 2026

github-actions Bot mentioned this pull request Aug 9, 2026

[PR Triage Report] PR Triage Report — 2026-08-09 (6 PRs) #51485

Closed

pelikhan marked this pull request as ready for review August 9, 2026 03:11

Copilot AI balanced review requested due to automatic review settings August 9, 2026 03:11

Copilot started reviewing on behalf of pelikhan August 9, 2026 03:11 View session

Copilot AI reviewed Aug 9, 2026

View reviewed changes

Merge branch 'main' into copilot/safeoutputs-update-tool-descriptions

fda4500

github-actions Bot mentioned this pull request Aug 9, 2026

[pr-sous-chef] PR Sous Chef run report - 2026-08-09 #51481

Closed

Copilot started work on behalf of gh-aw-bot August 9, 2026 04:01 View session

Clarify submit_pull_request_review validation guidance

211062c

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

Copilot finished work on behalf of gh-aw-bot August 9, 2026 04:10

Copilot AI requested a review from gh-aw-bot August 9, 2026 04:10

github-actions Bot mentioned this pull request Aug 9, 2026

[PR Triage Report] PR Triage Report — 2026-08-09 (13 PRs) #51544

Closed

pelikhan merged commit bff252c into main Aug 9, 2026

pelikhan deleted the copilot/safeoutputs-update-tool-descriptions branch August 9, 2026 10:57

Navigation Menu

Uh oh!

Add anti-probing guidance to submit_pull_request_review tool description - #51469

Add anti-probing guidance to submit_pull_request_review tool description#51469
pelikhan merged 4 commits into
mainfrom
copilot/safeoutputs-update-tool-descriptions

Copilot AI commented Aug 8, 2026 •

edited by github-actions Bot

Loading

Uh oh!

github-actions Bot commented Aug 9, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

gh-aw-bot commented Aug 9, 2026

Uh oh!

Copilot AI commented Aug 9, 2026

Uh oh!

github-actions Bot commented Aug 9, 2026

Uh oh!

github-actions Bot commented Aug 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

Copilot AI commented Aug 8, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Notes

Uh oh!

github-actions Bot commented Aug 9, 2026

PR Triage

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Review details

Uh oh!

gh-aw-bot commented Aug 9, 2026

Uh oh!

Copilot AI commented Aug 9, 2026

Uh oh!

github-actions Bot commented Aug 9, 2026

PR Triage

Uh oh!

github-actions Bot commented Aug 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Copilot AI commented Aug 8, 2026 •

edited by github-actions Bot

Loading