Add comprehensive PR review safe outputs testing to smoke-claude workflow with per-safe-output staged mode by Copilot · Pull Request #15684 · github/gh-aw

Copilot · 2026-02-14T13:48:49Z

Extends smoke-claude workflow to validate all 7 PR review-related safe outputs. Previously only tested basic operations (issues, comments, labels); now covers complete PR review lifecycle.

Changes

Frontmatter updates:

Increased max-turns: 50 → 100 to accommodate additional test steps
Added safe output configurations for:
- update-pull-request - Modifies PR title/body
- close-pull-request - Closes PRs without merge (per-safe-output staged mode)
- create-pull-request-review-comment - Adds inline review comments
- submit-pull-request-review - Submits consolidated review with status
- resolve-pull-request-review-thread - Resolves review conversations
- push-to-pull-request-branch - Pushes changes to PR branches (per-safe-output staged mode)
- add-reviewer - Assigns PR reviewers

Test cases (tests #11-17):

Comprehensive coverage of PR review operations with conditional handling for scenarios where operations may not be applicable (e.g., no threads to resolve, no safe PR to close)
Tests use proper tool parameters and validate both success and graceful degradation paths
Output section updated to report three statuses: ✅ pass, ❌ fail, ⚠️ skipped

Safety mechanism:

Per-safe-output staged mode allows granular control over which operations emit step summaries vs make actual API calls
close-pull-request and push-to-pull-request-branch operate in preview mode (emit step summaries only)
Other safe outputs make actual API calls for comprehensive testing
Allows testing potentially disruptive operations without affecting production resources

Infrastructure changes:

Implemented per-safe-output staged mode support
Added Staged field to BaseSafeOutputConfig struct
Updated JSON schema to allow staged property on individual safe outputs
Rebuilt binary with updated embedded schemas

Example configuration:

on:
  pull_request:
    types: [labeled]
    names: ["smoke"]

safe-outputs:
  close-pull-request:
    staged: true
    max: 1
  push-to-pull-request-branch:
    staged: true
    target: "*"
    if-no-changes: "warn"
  create-pull-request-review-comment:
    max: 5
    side: "RIGHT"
    target: "*"
  submit-pull-request-review:
    max: 1
    footer: true

[!WARNING]

Custom agent used: agentic-workflows
GitHub Agentic Workflows (gh-aw) - Create, debug, and upgrade AI-powered workflows with intelligent prompt routing

Original prompt

Update smoke Claude to tests all the safe outputs related to pull request reviews

find all the safe output related to pull request review

add to workflow and add prompt to use

increase max turns to 100

Custom agent used: agentic-workflows
GitHub Agentic Workflows (gh-aw) - Create, debug, and upgrade AI-powered workflows with intelligent prompt routing

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

- Updated max-turns from 50 to 100 - Added all PR review safe output configurations: - update-pull-request - close-pull-request - create-pull-request-review-comment - submit-pull-request-review - resolve-pull-request-review-thread - push-to-pull-request-branch - add-reviewer - Added comprehensive test requirements for each PR review safe output (#11-17) - Updated output section to handle pass/fail/skip status - Recompiled workflow successfully Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

pelikhan · 2026-02-14T13:58:27Z

.github/workflows/smoke-claude.md

+      body: true
+      max: 1
+      target: "*"
+    close-pull-request:


@copilot this one as draft mode only.

Done. Added draft: true to pull_request trigger so this safe output only operates on draft PRs. (b1de163)

.github/workflows/smoke-claude.md

pelikhan · 2026-02-14T13:59:02Z

.github/workflows/smoke-claude.md

+    push-to-pull-request-branch:
+      target: "*"
+      if-no-changes: "warn"
+    add-reviewer:


@copilot this one as draft mode

Done. Added draft: true to pull_request trigger so this safe output only operates on draft PRs. (b1de163)

Set pull_request trigger to draft: true so safe outputs (close-pull-request, push-to-pull-request-branch, add-reviewer) only operate on draft PRs during smoke testing Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot · 2026-02-14T14:06:49Z

Just as a heads up, I was blocked by some firewall rules while working on your feedback. Expand below for details.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

https://api.github.com/graphql
- Triggering command: /usr/bin/gh /usr/bin/gh api graphql -f query=query($owner: String!, $name: String!) { repository(owner: $owner, name: $name) { hasDiscussionsEnabled } } -f owner=github -f name=gh-aw (http block)
https://api.github.com/user
- Triggering command: /usr/bin/gh gh api user --jq .login (http block)

If you need me to access, download, or install something from one of these locations, you can either:

Configure Actions setup steps to set up my environment, which run before the firewall is enabled
Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

pelikhan · 2026-02-14T14:10:29Z

@copilot

remove draft: true from pull request trigger
add staged: true to close-pull-request

- Remove draft: true from pull_request trigger - Add staged: true to safe-outputs for preview mode - All safe outputs now emit step summaries instead of GitHub API calls Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot · 2026-02-14T14:19:07Z

@copilot

remove draft: true from pull request trigger...

Done. Removed draft: true from pull_request trigger and added staged: true to safe-outputs. All safe outputs now operate in preview mode (emit step summaries instead of making GitHub API calls). (b723052)

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

https://api.github.com/graphql
- Triggering command: /usr/bin/gh /usr/bin/gh api graphql -f query=query($owner: String!, $name: String!) { repository(owner: $owner, name: $name) { hasDiscussionsEnabled } } -f owner=github -f name=gh-aw rror-quality.md t@users.noreply.staged: grep -l staged: ion.md /usr/local/sbin/grep (http block)
https://api.github.com/user
- Triggering command: /usr/bin/gh gh api user --jq .login staged: s/code-scanning-fixer.md /home/REDACTED/.local/bin/grep (http block)
- Triggering command: /usr/bin/gh gh api user --jq .login staged: pt.md k/_temp/ghcca-node/node/bin/grep (http block)

If you need me to access, download, or install something from one of these locations, you can either:

Configure Actions setup steps to set up my environment, which run before the firewall is enabled
Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

pelikhan · 2026-02-14T14:20:40Z

.github/workflows/smoke-claude.md

  go:
    version: "1.25"
 safe-outputs:
+    staged: true


@copilot remove top level staged

Done. Removed top-level staged from safe-outputs. (1195145)

pelikhan · 2026-02-14T14:21:18Z

.github/workflows/smoke-claude.md

+      title: true
+      body: true
+      max: 1
+      target: "*"


@copilot remove target * and add staged : true

Done. Removed target: "*" and added staged: true to close-pull-request. (1195145)

pelikhan · 2026-02-14T14:21:50Z

.github/workflows/smoke-claude.md

+      target: "*"
+      if-no-changes: "warn"
+    add-reviewer:
+      max: 2


@copilot add staged: true to push to pull request branch

Done. Added staged: true to push-to-pull-request-branch. (1195145)

- Remove top-level staged from safe-outputs - Add staged: true to close-pull-request (removed target: "*") - Add staged: true to push-to-pull-request-branch - Update BaseSafeOutputConfig to include Staged field - Update JSON schema to allow staged property for individual safe outputs - Rebuild binary with updated embedded schema Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot · 2026-02-14T14:31:06Z

Just as a heads up, I was blocked by some firewall rules while working on your feedback. Expand below for details.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

https://api.github.com/graphql
- Triggering command: /usr/bin/gh /usr/bin/gh api graphql -f query=query($owner: String!, $name: String!) { repository(owner: $owner, name: $name) { hasDiscussionsEnabled } } -f owner=github -f name=gh-aw (http block)
https://api.github.com/user
- Triggering command: /usr/bin/gh gh api user --jq .login (http block)

If you need me to access, download, or install something from one of these locations, you can either:

Configure Actions setup steps to set up my environment, which run before the firewall is enabled
Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

Copilot

Pull request overview

This PR extends the smoke-claude workflow to test all 7 PR review-related safe outputs by adding comprehensive test cases and implementing per-safe-output staged mode for potentially disruptive operations.

Changes:

Increased max-turns from 50 to 100 to accommodate additional testing
Added 7 new safe output configurations for PR review operations
Added 7 new test cases (#11-17) in the workflow prompt
Implemented per-safe-output staged mode for close-pull-request and push-to-pull-request-branch

Reviewed changes

Copilot reviewed 3 out of 4 changed files in this pull request and generated 1 comment.

File	Description
`pkg/workflow/compiler_types.go`	Added `Staged` field to `BaseSafeOutputConfig` to enable per-safe-output staged mode
`pkg/parser/schemas/main_workflow_schema.json`	Added `staged` property to `close-pull-request` and `push-to-pull-request-branch` schemas; extensive formatting changes (array/enum multi-line formatting)
`.github/workflows/smoke-claude.md`	Updated max-turns to 100; added 7 PR review safe output configs; added tests #11-17 for comprehensive PR review testing
`.github/workflows/smoke-claude.lock.yml`	Compiled workflow with new safe outputs, tool schemas, permissions (contents:write), git setup steps, and handler configurations

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-14T15:17:32Z

.github/workflows/smoke-claude.lock.yml

          mkdir -p /tmp/gh-aw/mcp-logs/safeoutputs
          cat > /opt/gh-aw/safeoutputs/config.json << 'GH_AW_SAFE_OUTPUTS_CONFIG_EOF'
-          {"add_comment":{"max":2},"add_labels":{"allowed":["smoke-claude"],"max":3},"create_issue":{"expires":2,"group":true,"max":1},"missing_data":{},"missing_tool":{},"noop":{"max":1}}
+          {"add_comment":{"max":2},"add_labels":{"allowed":["smoke-claude"],"max":3},"add_reviewer":{"max":2},"create_issue":{"expires":2,"group":true,"max":1},"create_pull_request_review_comment":{"max":5},"missing_data":{},"missing_tool":{},"noop":{"max":1},"push_to_pull_request_branch":{"max":0,"target":"*"},"resolve_pull_request_review_thread":{"max":5},"submit_pull_request_review":{"max":1},"update_pull_request":{"max":1}}


The close_pull_request safe output is missing from the MCP config.json, but it appears in tools.json (line 368-391). This means the AI can use the tool even though staged: true is set in the frontmatter.

Per-safe-output staged mode should set max: 0 in the config to hide the tool from the AI while keeping the handler active for preview. This pattern works correctly for push_to_pull_request_branch which has "max":0 in the config, but close_pull_request is completely missing from the config.

The config should include: "close_pull_request":{"max":0} to properly implement staged mode for this safe output.

Suggested change

{"add_comment":{"max":2},"add_labels":{"allowed":["smoke-claude"],"max":3},"add_reviewer":{"max":2},"create_issue":{"expires":2,"group":true,"max":1},"create_pull_request_review_comment":{"max":5},"missing_data":{},"missing_tool":{},"noop":{"max":1},"push_to_pull_request_branch":{"max":0,"target":"*"},"resolve_pull_request_review_thread":{"max":5},"submit_pull_request_review":{"max":1},"update_pull_request":{"max":1}}

{"add_comment":{"max":2},"add_labels":{"allowed":["smoke-claude"],"max":3},"add_reviewer":{"max":2},"close_pull_request":{"max":0},"create_issue":{"expires":2,"group":true,"max":1},"create_pull_request_review_comment":{"max":5},"missing_data":{},"missing_tool":{},"noop":{"max":1},"push_to_pull_request_branch":{"max":0,"target":"*"},"resolve_pull_request_review_thread":{"max":5},"submit_pull_request_review":{"max":1},"update_pull_request":{"max":1}}

Initial plan

fc81fb2

Copilot AI assigned Copilot and pelikhan Feb 14, 2026

Copilot started work on behalf of pelikhan February 14, 2026 13:49 View session

Copilot AI changed the title ~~[WIP] Update smoke tests for safe outputs in pull request reviews~~ Add comprehensive PR review safe outputs testing to smoke-claude workflow Feb 14, 2026

Copilot AI requested a review from pelikhan February 14, 2026 13:57

Copilot finished work on behalf of pelikhan February 14, 2026 13:57

pelikhan reviewed Feb 14, 2026

View reviewed changes

Copilot started work on behalf of pelikhan February 14, 2026 13:59 View session

Add draft filter to smoke-claude workflow

b1de163

Set pull_request trigger to draft: true so safe outputs (close-pull-request, push-to-pull-request-branch, add-reviewer) only operate on draft PRs during smoke testing Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI requested a review from pelikhan February 14, 2026 14:07

Copilot finished work on behalf of pelikhan February 14, 2026 14:07

Copilot started work on behalf of pelikhan February 14, 2026 14:13 View session

Enable staged mode and remove draft filter

b723052

- Remove draft: true from pull_request trigger - Add staged: true to safe-outputs for preview mode - All safe outputs now emit step summaries instead of GitHub API calls Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot finished work on behalf of pelikhan February 14, 2026 14:20

pelikhan reviewed Feb 14, 2026

View reviewed changes

Copilot started work on behalf of pelikhan February 14, 2026 14:22 View session

Copilot AI changed the title ~~Add comprehensive PR review safe outputs testing to smoke-claude workflow~~ Add comprehensive PR review safe outputs testing to smoke-claude workflow with per-safe-output staged mode Feb 14, 2026

Copilot AI requested a review from pelikhan February 14, 2026 14:31

Copilot finished work on behalf of pelikhan February 14, 2026 14:31

pelikhan marked this pull request as ready for review February 14, 2026 15:12

Copilot AI review requested due to automatic review settings February 14, 2026 15:12

pelikhan merged commit 8e0b098 into main Feb 14, 2026

pelikhan deleted the copilot/update-smoke-claude-tests branch February 14, 2026 15:12

Copilot started reviewing on behalf of pelikhan February 14, 2026 15:12 View session

Copilot AI reviewed Feb 14, 2026

View reviewed changes

Conversation

Copilot AI commented Feb 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Uh oh!

pelikhan Feb 14, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 14, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pelikhan Feb 14, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 14, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI commented Feb 14, 2026

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

pelikhan commented Feb 14, 2026

Uh oh!

Copilot AI commented Feb 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

pelikhan Feb 14, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 14, 2026

Choose a reason for hiding this comment

Uh oh!

pelikhan Feb 14, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 14, 2026

Choose a reason for hiding this comment

Uh oh!

pelikhan Feb 14, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 14, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI commented Feb 14, 2026

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Feb 14, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Copilot AI commented Feb 14, 2026 •

edited

Loading

Copilot AI commented Feb 14, 2026 •

edited

Loading