From 75da75ce504701630e9b283d077082d8048ac1b2 Mon Sep 17 00:00:00 2001 From: openhands Date: Thu, 28 May 2026 14:19:09 +0000 Subject: [PATCH 01/19] docs: add event-driven automation prerequisites and setup guides event-automations.mdx: - Add 'Prerequisites for GitHub Event Automations' section with step-by-step setup: GitHub App install, team org creation, org claim, service accounts - Add troubleshooting accordion for silent event drops code-review.mdx: - Replace generic 'Automate This' with full 'OpenHands Automations' section - Add Warning callout: set up event-driven automations + enable org feature - Complete copy-paste prompt for PR review automation - Trigger events, filter, session URL, progress comment documentation - Single-repo vs org-wide configuration and testing instructions qa-changes.mdx: - Add 'Automate This' section with Warning callout matching code-review - Copy-paste prompt for automated QA on PRs using qa-changes plugin Co-authored-by: openhands --- .../usage/automations/event-automations.mdx | 60 +++++++++ openhands/usage/automations/overview.mdx | 2 +- openhands/usage/use-cases/code-review.mdx | 122 ++++++++++++++++-- openhands/usage/use-cases/qa-changes.mdx | 27 ++++ 4 files changed, 199 insertions(+), 12 deletions(-) diff --git a/openhands/usage/automations/event-automations.mdx b/openhands/usage/automations/event-automations.mdx index cc4a1ace..efe2012a 100644 --- a/openhands/usage/automations/event-automations.mdx +++ b/openhands/usage/automations/event-automations.mdx @@ -5,6 +5,66 @@ description: Trigger automations from GitHub events or custom webhooks instead o Event-based automations run when something happens—a PR is opened, an issue is commented on, or a webhook fires—instead of on a schedule. This is ideal for responsive workflows like auto-reviewing PRs, triaging issues, or reacting to external service events. +## Prerequisites for GitHub Event Automations + +GitHub event automations require some one-time setup before events will flow. If any step is missing, automations will appear to work (manual triggers succeed) but GitHub events will silently never arrive. + +### 1. Install the OpenHands GitHub App + +The OpenHands GitHub App must be installed on the GitHub organization that owns the repositories you want to monitor. Install it from your [GitHub integration settings](/openhands/usage/cloud/github-installation). The app needs access to the repositories that will generate events. + +### 2. Create an OpenHands Team Organization + +If you're working with repositories owned by a GitHub organization (e.g., `myorg/my-repo`), you need an OpenHands **team organization** — not just a personal account. GitHub events for org repos are routed to team orgs, not personal orgs. + +If you don't already have one, create a team organization from the [OpenHands Cloud settings](https://app.all-hands.dev/settings). + +### 3. Claim Your GitHub Organization + + +**This is the most commonly missed step.** Without it, GitHub events have nowhere to be routed and will be silently dropped. + + +Your OpenHands team org must **claim** the GitHub organization to establish the link between GitHub webhooks and your OpenHands org. Claiming tells the event router: _"Events for repos in this GitHub org should go to this OpenHands team org."_ + +To claim a GitHub org: + +1. Switch to your team org using the org switcher in the sidebar +2. Go to **Organization Settings** +3. In the **Git Conversation Routing** section, find your GitHub org +4. Click **Claim** + +You must have admin access to the GitHub org to complete the claim. See [Claiming Git Organizations](/openhands/usage/cloud/organizations/settings#claiming-git-organizations) for full details. + +### 4. Create the Automation Under the Team Org + +Make sure you are switched to the **team org** (not your personal org) when creating the automation. The automation must live in the same org that claimed the GitHub organization — otherwise events won't match. + +### 5. (Optional) Add Service Accounts to the Team Org + +If you're using a service account (like a bot account) to create or own automations, that account must be a **member of the team org**. Invite them from the [Organization Members](/openhands/usage/cloud/organizations/managing-members) page. + +### Troubleshooting + +If your automation doesn't trigger on GitHub events: + + + + The most common cause. Go to **Organization Settings → Git Conversation Routing** and check if your GitHub org shows as claimed. If not, click **Claim**. See [Claiming Git Organizations](/openhands/usage/cloud/organizations/settings#claiming-git-organizations). + + + GitHub events for org repos are routed to the **team org** that claimed the GitHub org. If you created the automation under your personal org, events will never reach it. Switch to the team org and recreate the automation. + + + Double-check that the event type (e.g., `pull_request.labeled`) and filter expression match the action you're testing. Use wildcards like `pull_request.*` to match all actions during debugging. + + + Verify the automation is enabled. You can check via the automations list or by asking OpenHands to list your automations. + + + +--- + ## Built-In vs. Custom Integrations | Type | Setup | Best For | diff --git a/openhands/usage/automations/overview.mdx b/openhands/usage/automations/overview.mdx index 2ceb734e..cb268bcd 100644 --- a/openhands/usage/automations/overview.mdx +++ b/openhands/usage/automations/overview.mdx @@ -118,7 +118,7 @@ Each use case has a ready-to-use automation prompt. Click a card to see the full Functionally test PR changes by exercising the software as a real user would. diff --git a/openhands/usage/use-cases/code-review.mdx b/openhands/usage/use-cases/code-review.mdx index 5251a977..20ee7719 100644 --- a/openhands/usage/use-cases/code-review.mdx +++ b/openhands/usage/use-cases/code-review.mdx @@ -367,23 +367,123 @@ See real automated reviews in action on the OpenHands Software Agent SDK reposit ## Automate This -You can schedule daily code reviews using [OpenHands Automations](/openhands/usage/automations/overview). -Copy this prompt into a new conversation to set one up: + +Before setting up an event-driven automation you must complete two one-time steps: +1. **Set up [event-driven automations](/openhands/usage/automations/event-automations#prerequisites-for-github-event-automations)** — install the GitHub App, create a team org, and claim your GitHub organization. +2. **Enable the [organization feature](/openhands/usage/cloud/organizations/overview)** in OpenHands Cloud so GitHub events can be routed to your team. + +See the [full prerequisites guide](/openhands/usage/automations/event-automations#prerequisites-for-github-event-automations) for details. + + +[OpenHands Automations](/openhands/usage/automations/overview) is an event-triggered automation system that can replace per-repo GitHub Actions workflows. You define the trigger once and it covers all repositories the bot account has access to — no per-repo workflow files needed. + +**Trade-offs vs GitHub Actions:** Simpler to set up and maintain across many repos, especially at the org level. Also leverages the full OpenHands runtime (browser, tools, sandbox), which GitHub Actions cannot. GitHub Actions gives more control over the execution environment and integrates directly with your CI pipeline. + +### Setup: Create the Automation via Prompt + +Log in to [OpenHands Cloud](https://app.all-hands.dev) as your bot account (or under your team org) and send the following prompt in a new conversation. Replace the placeholders with your values: + +- `YOUR_ORG` — your GitHub organization name (e.g., `mycompany`) +- `YOUR_BOT_LOGIN` — the GitHub username of your bot account (e.g., `mycompany-bot`) + +```` +Create an OpenHands Cloud automation using the Plugin Preset with the following configuration: + +**Name:** PR Review: YOUR_ORG/* (ready for review, review-this, or reviewer requested) + +**Plugin:** github:OpenHands/extensions (repo_path: plugins/pr-review) + +**Trigger events:** +- pull_request.opened +- pull_request.ready_for_review +- pull_request.review_requested +- pull_request.labeled + +**Filter:** +``` +glob(repository.full_name, 'YOUR_ORG/*') && ( + label.name == 'review-this' + || requested_reviewer.login == 'YOUR_BOT_LOGIN' + || (!label && !requested_reviewer + && pull_request.author_association != 'FIRST_TIME_CONTRIBUTOR' + && pull_request.author_association != 'NONE' + && !pull_request.draft) +) +``` + +**Timeout:** 3600 seconds +**Prompt (use this exactly):** ``` -Create an automation called "Daily Code Review" that runs every weekday at 9 AM. +Before starting the code review, complete these steps in order: + +Step 1 — Build the session URL. +Run this in terminal: + SESSION_URL="${AUTOMATION_SESSION_URL:-${AUTOMATION_API_URL:-https://app.all-hands.dev}}" + echo "SESSION_URL=${SESSION_URL}" + +Step 2 — Extract PR info from the event payload: + PR_NUMBER=$(echo "$AUTOMATION_EVENT_PAYLOAD" | python3 -c "import sys,json; p=json.load(sys.stdin); print(p['pull_request']['number'])") + REPO=$(echo "$AUTOMATION_EVENT_PAYLOAD" | python3 -c "import sys,json; p=json.load(sys.stdin); print(p['repository']['full_name'])") + +Step 3 — Post a progress comment and save the comment ID: + COMMENT_ID=$(curl -s -X POST \ + -H "Authorization: Bearer $GITHUB_TOKEN" \ + -H "Accept: application/vnd.github+json" \ + "https://api.github.com/repos/$REPO/issues/$PR_NUMBER/comments" \ + -d "{\"body\": \"🔍 **Review in progress…**\\n\\nWe are performing the review through OpenHands Cloud Automation. You can log in and [view the conversation here](${SESSION_URL}).\"}" \ + | python3 -c "import sys,json; print(json.load(sys.stdin)['id'])") + +Step 4 — Review the pull request using the pr-review plugin. Post a comprehensive code review on GitHub with inline comments on specific changed lines where appropriate, and a concise overall summary. Avoid duplicating existing unresolved review comments. At the end of the top-level review body include exactly: + _This review was generated by an AI agent (OpenHands) on behalf of the user through OpenHands Automation. [View conversation](${SESSION_URL})_ + +Step 5 — After the review is posted, update the progress comment: + curl -s -X PATCH \ + -H "Authorization: Bearer $GITHUB_TOKEN" \ + -H "Accept: application/vnd.github+json" \ + "https://api.github.com/repos/$REPO/issues/comments/$COMMENT_ID" \ + -d "{\"body\": \"✅ **Review complete.**\\n\\nThis review was performed through OpenHands Cloud Automation. You can log in and [view the conversation here](${SESSION_URL}).\"}" +``` +```` + +### What This Produces + +When the automation is created and a qualifying PR event occurs, the bot will: + +1. **Post a progress comment** on the PR: "🔍 Review in progress…" with a link to the live conversation +2. **Run the pr-review plugin** which analyzes the diff and posts a structured code review with inline comments +3. **Update the progress comment** to "✅ Review complete." with the conversation link + +The automation triggers on four conditions: +- **`opened`** — when a new non-draft PR is created (for established contributors only) +- **`ready_for_review`** — when a draft PR is marked ready (for established contributors only) +- **`review_requested`** — when your bot account is requested as a reviewer +- **`labeled`** — when the `review-this` label is added to any PR -It should: -1. Find all open PRs that have no reviews yet -2. For each PR, review the diff for bugs, style issues, and security concerns -3. Post a summary of findings as a comment on each PR + +The `$AUTOMATION_SESSION_URL` variable is automatically injected by the automation runtime and resolves to a direct link to the conversation (e.g., `https://app.all-hands.dev/conversations/{uuid}`). No additional configuration is needed. + + +### Single-Repo vs Org-Wide + +The prompt above uses `glob(repository.full_name, 'YOUR_ORG/*')` to cover **all repos** in your org. To target a single repo instead, replace the filter's first condition: -Learn more at https://docs.openhands.dev/openhands/usage/use-cases/code-review ``` +repository.full_name == 'YOUR_ORG/YOUR_REPO' && ( + label.name == 'review-this' + || requested_reviewer.login == 'YOUR_BOT_LOGIN' + || ... +) +``` + +### Testing + +After creating the automation: -For inline review comments on every push, use the -[pr-review plugin](https://github.com/OpenHands/extensions/tree/main/plugins/pr-review) -as a GitHub Action instead. +1. Open a non-draft PR in a covered repo (the bot should trigger automatically for established contributors), or: +2. Add the `review-this` label, or request your bot as a reviewer on any PR +3. Watch for the "🔍 Review in progress…" comment — it should appear within a few seconds +4. The full review will follow within 2-5 minutes ## Related Resources diff --git a/openhands/usage/use-cases/qa-changes.mdx b/openhands/usage/use-cases/qa-changes.mdx index 5291334c..6e991259 100644 --- a/openhands/usage/use-cases/qa-changes.mdx +++ b/openhands/usage/use-cases/qa-changes.mdx @@ -193,6 +193,33 @@ The QA agent is most powerful when used alongside the [code review agent](/openh +## Automate This + + +Before setting up an event-driven automation you must complete two one-time steps: +1. **Set up [event-driven automations](/openhands/usage/automations/event-automations#prerequisites-for-github-event-automations)** — install the GitHub App, create a team org, and claim your GitHub organization. +2. **Enable the [organization feature](/openhands/usage/cloud/organizations/overview)** in OpenHands Cloud so GitHub events can be routed to your team. + +See the [full prerequisites guide](/openhands/usage/automations/event-automations#prerequisites-for-github-event-automations) for details. + + +You can run QA automatically on every PR using [OpenHands Automations](/openhands/usage/automations/overview). +Copy this prompt into a new conversation to set one up: + +``` +Create an automation called "Automated QA" that triggers on pull_request.opened +and pull_request.labeled (with label "qa-this") for my repositories. + +It should use the qa-changes plugin from github:OpenHands/extensions to: +1. Check out the PR branch +2. Run the QA agent to exercise the changed behavior as a real user would +3. Post a structured QA report as a PR comment with evidence (commands run, outputs, screenshots) + +Learn more at https://docs.openhands.dev/openhands/usage/use-cases/qa-changes +``` + +For automated QA on every push, use the [qa-changes plugin](https://github.com/OpenHands/extensions/tree/main/plugins/qa-changes) as a GitHub Action instead. + ## Related Resources - [QA Changes Plugin](https://github.com/OpenHands/extensions/tree/main/plugins/qa-changes) — GitHub Actions plugin From cb4abbe0c398b8fca9c70f98324c977a57b62449 Mon Sep 17 00:00:00 2001 From: openhands Date: Thu, 28 May 2026 20:35:45 +0000 Subject: [PATCH 02/19] docs: improve event-driven automation docs accuracy and consistency Co-authored-by: openhands --- .../usage/automations/event-automations.mdx | 37 ++++++++++++------- openhands/usage/use-cases/code-review.mdx | 12 +++--- openhands/usage/use-cases/qa-changes.mdx | 15 ++++---- 3 files changed, 35 insertions(+), 29 deletions(-) diff --git a/openhands/usage/automations/event-automations.mdx b/openhands/usage/automations/event-automations.mdx index efe2012a..2a409ad6 100644 --- a/openhands/usage/automations/event-automations.mdx +++ b/openhands/usage/automations/event-automations.mdx @@ -34,7 +34,11 @@ To claim a GitHub org: 3. In the **Git Conversation Routing** section, find your GitHub org 4. Click **Claim** -You must have admin access to the GitHub org to complete the claim. See [Claiming Git Organizations](/openhands/usage/cloud/organizations/settings#claiming-git-organizations) for full details. +You must be an **Owner** of the OpenHands team org and have **admin access** to the GitHub org to complete the claim. See [Claiming Git Organizations](/openhands/usage/cloud/organizations/settings#claiming-git-organizations) for full details. + + +Each GitHub organization can only be claimed by one OpenHands team org. If another team has already claimed it, coordinate with them or contact support. + ### 4. Create the Automation Under the Team Org @@ -69,7 +73,7 @@ If your automation doesn't trigger on GitHub events: | Type | Setup | Best For | |------|-------|----------| -| **Built-in (GitHub)** | None—just create the automation | PR reviews, issue triage, push-triggered tasks | +| **Built-in (GitHub)** | One-time org setup ([see above](#prerequisites-for-github-event-automations)), then create the automation | PR reviews, issue triage, push-triggered tasks | | **Custom Webhooks** | Register webhook first, then create automation | Linear, Stripe, Slack, and other services | ## GitHub Events (Built-In) @@ -118,25 +122,30 @@ Use wildcards like `pull_request.*` to match all actions for an event type. Filters let you narrow which events trigger your automation. They use [JMESPath expressions](https://jmespath.org/) to match fields in the event payload—so you can trigger only on specific labels, users, branches, or other conditions. + +OpenHands extends standard JMESPath with custom functions including `icontains` (case-insensitive string match) and `glob` (wildcard path matching). These are not part of the JMESPath specification. + + **Common filter patterns:** -```javascript -// Match a specific label +``` contains(pull_request.labels[].name, 'openhands') -// Case-insensitive mention in comment icontains(comment.body, '@openhands') -// Match repos in your org glob(repository.full_name, 'myorg/*') -// Push to main branch only ref == 'refs/heads/main' -// Combine conditions glob(repository.full_name, 'myorg/*') && contains(pull_request.labels[].name, 'bug') ``` +- `contains(...)` — match a specific label +- `icontains(...)` — case-insensitive mention in a comment body +- `glob(...)` — match repos in your org with wildcards +- `==` — exact match (e.g., push to main branch only) +- `&&` — combine multiple conditions + --- ## Custom Webhooks @@ -250,12 +259,12 @@ When registering any custom webhook, these parameters define how OpenHands proce These are example configurations for popular services. **Always verify with each service's webhook documentation**, as signature headers and payload formats may change. -| Service | Signature Header | Event Key | -|---------|-----------------|-----------| -| Linear | `Linear-Signature` | `type` | -| Stripe | `Stripe-Signature` | `type` | -| Slack | `X-Slack-Signature` | `type` | -| Twilio | `X-Twilio-Signature` | `type` | +| Service | Signature Header | Event Key | Notes | +|---------|-----------------|-----------|-------| +| Linear | `Linear-Signature` | `type` | | +| Stripe | `Stripe-Signature` | `type` | Uses a custom `t=timestamp,v1=signature` format — verify compatibility | +| Slack | `X-Slack-Signature` | `type` | | +| Twilio | `X-Twilio-Signature` | `type` | Uses HMAC-SHA1 of request URL + params — verify compatibility | --- diff --git a/openhands/usage/use-cases/code-review.mdx b/openhands/usage/use-cases/code-review.mdx index 20ee7719..448d472f 100644 --- a/openhands/usage/use-cases/code-review.mdx +++ b/openhands/usage/use-cases/code-review.mdx @@ -368,14 +368,10 @@ See real automated reviews in action on the OpenHands Software Agent SDK reposit ## Automate This -Before setting up an event-driven automation you must complete two one-time steps: -1. **Set up [event-driven automations](/openhands/usage/automations/event-automations#prerequisites-for-github-event-automations)** — install the GitHub App, create a team org, and claim your GitHub organization. -2. **Enable the [organization feature](/openhands/usage/cloud/organizations/overview)** in OpenHands Cloud so GitHub events can be routed to your team. - -See the [full prerequisites guide](/openhands/usage/automations/event-automations#prerequisites-for-github-event-automations) for details. +Before setting up an event-driven automation, complete the one-time [prerequisites for GitHub event automations](/openhands/usage/automations/event-automations#prerequisites-for-github-event-automations) — install the GitHub App, create a team org, and claim your GitHub organization. Without these steps, GitHub events will silently never arrive. -[OpenHands Automations](/openhands/usage/automations/overview) is an event-triggered automation system that can replace per-repo GitHub Actions workflows. You define the trigger once and it covers all repositories the bot account has access to — no per-repo workflow files needed. +[OpenHands Automations](/openhands/usage/automations/overview) is an event-triggered automation system that can replace per-repo GitHub Actions workflows. You define the trigger once and it covers all repositories matching your filter — no per-repo workflow files needed. **Trade-offs vs GitHub Actions:** Simpler to set up and maintain across many repos, especially at the org level. Also leverages the full OpenHands runtime (browser, tools, sandbox), which GitHub Actions cannot. GitHub Actions gives more control over the execution environment and integrates directly with your CI pipeline. @@ -461,7 +457,9 @@ The automation triggers on four conditions: - **`labeled`** — when the `review-this` label is added to any PR -The `$AUTOMATION_SESSION_URL` variable is automatically injected by the automation runtime and resolves to a direct link to the conversation (e.g., `https://app.all-hands.dev/conversations/{uuid}`). No additional configuration is needed. +The `$AUTOMATION_SESSION_URL` variable is injected by the automation runtime and resolves to a direct link to the conversation (e.g., `https://app.all-hands.dev/conversations/{uuid}`). The prompt includes fallbacks (`$AUTOMATION_API_URL`, then the default app URL) for environments where the variable is not yet available. + +The `$AUTOMATION_EVENT_PAYLOAD` variable contains the full GitHub webhook event as JSON. Both variables are provided automatically — no additional configuration is needed. ### Single-Repo vs Org-Wide diff --git a/openhands/usage/use-cases/qa-changes.mdx b/openhands/usage/use-cases/qa-changes.mdx index 6e991259..50f7220e 100644 --- a/openhands/usage/use-cases/qa-changes.mdx +++ b/openhands/usage/use-cases/qa-changes.mdx @@ -196,19 +196,16 @@ The QA agent is most powerful when used alongside the [code review agent](/openh ## Automate This -Before setting up an event-driven automation you must complete two one-time steps: -1. **Set up [event-driven automations](/openhands/usage/automations/event-automations#prerequisites-for-github-event-automations)** — install the GitHub App, create a team org, and claim your GitHub organization. -2. **Enable the [organization feature](/openhands/usage/cloud/organizations/overview)** in OpenHands Cloud so GitHub events can be routed to your team. - -See the [full prerequisites guide](/openhands/usage/automations/event-automations#prerequisites-for-github-event-automations) for details. +Before setting up an event-driven automation, complete the one-time [prerequisites for GitHub event automations](/openhands/usage/automations/event-automations#prerequisites-for-github-event-automations) — install the GitHub App, create a team org, and claim your GitHub organization. Without these steps, GitHub events will silently never arrive. You can run QA automatically on every PR using [OpenHands Automations](/openhands/usage/automations/overview). -Copy this prompt into a new conversation to set one up: +Log in to [OpenHands Cloud](https://app.all-hands.dev) under your team org and send the following prompt in a new conversation. Replace `YOUR_ORG` with your GitHub organization name: ``` -Create an automation called "Automated QA" that triggers on pull_request.opened -and pull_request.labeled (with label "qa-this") for my repositories. +Create an automation called "Automated QA" that triggers on pull_request.opened, +pull_request.ready_for_review, and pull_request.labeled (with label "qa-this") +for repos matching glob(repository.full_name, 'YOUR_ORG/*'). It should use the qa-changes plugin from github:OpenHands/extensions to: 1. Check out the PR branch @@ -220,6 +217,8 @@ Learn more at https://docs.openhands.dev/openhands/usage/use-cases/qa-changes For automated QA on every push, use the [qa-changes plugin](https://github.com/OpenHands/extensions/tree/main/plugins/qa-changes) as a GitHub Action instead. +For a more detailed automation setup with progress comments and session links, see the [Automated Code Review automation guide](/openhands/usage/use-cases/code-review#automate-this) — the same pattern applies to QA. + ## Related Resources - [QA Changes Plugin](https://github.com/OpenHands/extensions/tree/main/plugins/qa-changes) — GitHub Actions plugin From 7dd422188096405e3e030e72cb7c7243ec88035e Mon Sep 17 00:00:00 2001 From: openhands Date: Thu, 28 May 2026 20:45:47 +0000 Subject: [PATCH 03/19] docs: address review feedback - team requests, JMESPath null semantics, draft exclusion This is an AI-generated contribution on behalf of the user. --- openhands/usage/use-cases/code-review.mdx | 6 ++++++ openhands/usage/use-cases/qa-changes.mdx | 1 + 2 files changed, 7 insertions(+) diff --git a/openhands/usage/use-cases/code-review.mdx b/openhands/usage/use-cases/code-review.mdx index 448d472f..2acf47a6 100644 --- a/openhands/usage/use-cases/code-review.mdx +++ b/openhands/usage/use-cases/code-review.mdx @@ -407,6 +407,12 @@ glob(repository.full_name, 'YOUR_ORG/*') && ( ) ``` + +**Team review requests:** The `requested_reviewer` field is only populated for individual reviewer requests. When a *team* is requested as reviewer, GitHub uses `requested_team` instead. To also match team requests, add `|| requested_team.slug == 'YOUR_TEAM_SLUG'` to the filter. + +**How `!label` works:** JMESPath treats absent fields as `null`, and `!null` evaluates to `true`. This means the third branch fires for `opened` and `ready_for_review` events (which have no `label` or `requested_reviewer` in the payload), while correctly staying silent for `labeled` and `review_requested` events where those fields are set. + + **Timeout:** 3600 seconds **Prompt (use this exactly):** diff --git a/openhands/usage/use-cases/qa-changes.mdx b/openhands/usage/use-cases/qa-changes.mdx index 50f7220e..d3bb3930 100644 --- a/openhands/usage/use-cases/qa-changes.mdx +++ b/openhands/usage/use-cases/qa-changes.mdx @@ -206,6 +206,7 @@ Log in to [OpenHands Cloud](https://app.all-hands.dev) under your team org and s Create an automation called "Automated QA" that triggers on pull_request.opened, pull_request.ready_for_review, and pull_request.labeled (with label "qa-this") for repos matching glob(repository.full_name, 'YOUR_ORG/*'). +Exclude draft PRs (!pull_request.draft) from the opened/ready_for_review triggers. It should use the qa-changes plugin from github:OpenHands/extensions to: 1. Check out the PR branch From 406681c19685d93a66a754bbfaf7b064c56911d9 Mon Sep 17 00:00:00 2001 From: openhands Date: Thu, 28 May 2026 20:59:01 +0000 Subject: [PATCH 04/19] docs: address second review - move Note outside code block, document GITHUB_TOKEN, align QA preset Co-authored-by: openhands --- openhands/usage/use-cases/code-review.mdx | 14 +++++++------- openhands/usage/use-cases/qa-changes.mdx | 16 ++++++++-------- 2 files changed, 15 insertions(+), 15 deletions(-) diff --git a/openhands/usage/use-cases/code-review.mdx b/openhands/usage/use-cases/code-review.mdx index 2acf47a6..3d427fc4 100644 --- a/openhands/usage/use-cases/code-review.mdx +++ b/openhands/usage/use-cases/code-review.mdx @@ -407,12 +407,6 @@ glob(repository.full_name, 'YOUR_ORG/*') && ( ) ``` - -**Team review requests:** The `requested_reviewer` field is only populated for individual reviewer requests. When a *team* is requested as reviewer, GitHub uses `requested_team` instead. To also match team requests, add `|| requested_team.slug == 'YOUR_TEAM_SLUG'` to the filter. - -**How `!label` works:** JMESPath treats absent fields as `null`, and `!null` evaluates to `true`. This means the third branch fires for `opened` and `ready_for_review` events (which have no `label` or `requested_reviewer` in the payload), while correctly staying silent for `labeled` and `review_requested` events where those fields are set. - - **Timeout:** 3600 seconds **Prompt (use this exactly):** @@ -448,6 +442,12 @@ Step 5 — After the review is posted, update the progress comment: ``` ```` + +**Team review requests:** The `requested_reviewer` field is only populated for individual reviewer requests. When a *team* is requested as reviewer, GitHub uses `requested_team` instead. To also match team requests, add `|| requested_team.slug == 'YOUR_TEAM_SLUG'` to the filter. + +**How `!label` works:** JMESPath treats absent fields as `null`, and `!null` evaluates to `true`. This means the third branch fires for `opened` and `ready_for_review` events (which have no `label` or `requested_reviewer` in the payload), while correctly staying silent for `labeled` and `review_requested` events where those fields are set. + + ### What This Produces When the automation is created and a qualifying PR event occurs, the bot will: @@ -465,7 +465,7 @@ The automation triggers on four conditions: The `$AUTOMATION_SESSION_URL` variable is injected by the automation runtime and resolves to a direct link to the conversation (e.g., `https://app.all-hands.dev/conversations/{uuid}`). The prompt includes fallbacks (`$AUTOMATION_API_URL`, then the default app URL) for environments where the variable is not yet available. -The `$AUTOMATION_EVENT_PAYLOAD` variable contains the full GitHub webhook event as JSON. Both variables are provided automatically — no additional configuration is needed. +The `$AUTOMATION_EVENT_PAYLOAD` variable contains the full GitHub webhook event as JSON. The `$GITHUB_TOKEN` (from the configured GitHub integration) is also automatically available. No additional configuration is needed for any of these variables. ### Single-Repo vs Org-Wide diff --git a/openhands/usage/use-cases/qa-changes.mdx b/openhands/usage/use-cases/qa-changes.mdx index d3bb3930..eca5bdbb 100644 --- a/openhands/usage/use-cases/qa-changes.mdx +++ b/openhands/usage/use-cases/qa-changes.mdx @@ -203,17 +203,17 @@ You can run QA automatically on every PR using [OpenHands Automations](/openhand Log in to [OpenHands Cloud](https://app.all-hands.dev) under your team org and send the following prompt in a new conversation. Replace `YOUR_ORG` with your GitHub organization name: ``` -Create an automation called "Automated QA" that triggers on pull_request.opened, -pull_request.ready_for_review, and pull_request.labeled (with label "qa-this") -for repos matching glob(repository.full_name, 'YOUR_ORG/*'). -Exclude draft PRs (!pull_request.draft) from the opened/ready_for_review triggers. +Create an OpenHands Cloud automation using the Plugin Preset with the following configuration: -It should use the qa-changes plugin from github:OpenHands/extensions to: +Name: Automated QA: YOUR_ORG/* +Plugin: github:OpenHands/extensions (repo_path: plugins/qa-changes) +Trigger events: pull_request.opened, pull_request.ready_for_review, pull_request.labeled +Filter: glob(repository.full_name, 'YOUR_ORG/*') && (label.name == 'qa-this' || (!label && !pull_request.draft)) + +The QA agent should: 1. Check out the PR branch -2. Run the QA agent to exercise the changed behavior as a real user would +2. Exercise the changed behavior as a real user would 3. Post a structured QA report as a PR comment with evidence (commands run, outputs, screenshots) - -Learn more at https://docs.openhands.dev/openhands/usage/use-cases/qa-changes ``` For automated QA on every push, use the [qa-changes plugin](https://github.com/OpenHands/extensions/tree/main/plugins/qa-changes) as a GitHub Action instead. From c7997bd1c69a7a21e478bd28b011b13f7ce28b83 Mon Sep 17 00:00:00 2001 From: openhands Date: Thu, 28 May 2026 21:11:50 +0000 Subject: [PATCH 05/19] docs: address third review - FIRST_TIMER guard, valid single-repo filter, QA author guards Co-authored-by: openhands --- openhands/usage/use-cases/code-review.mdx | 13 +++++++++++-- openhands/usage/use-cases/qa-changes.mdx | 2 +- 2 files changed, 12 insertions(+), 3 deletions(-) diff --git a/openhands/usage/use-cases/code-review.mdx b/openhands/usage/use-cases/code-review.mdx index 3d427fc4..ceae2f10 100644 --- a/openhands/usage/use-cases/code-review.mdx +++ b/openhands/usage/use-cases/code-review.mdx @@ -402,6 +402,7 @@ glob(repository.full_name, 'YOUR_ORG/*') && ( || requested_reviewer.login == 'YOUR_BOT_LOGIN' || (!label && !requested_reviewer && pull_request.author_association != 'FIRST_TIME_CONTRIBUTOR' + && pull_request.author_association != 'FIRST_TIMER' && pull_request.author_association != 'NONE' && !pull_request.draft) ) @@ -476,10 +477,18 @@ The prompt above uses `glob(repository.full_name, 'YOUR_ORG/*')` to cover **all repository.full_name == 'YOUR_ORG/YOUR_REPO' && ( label.name == 'review-this' || requested_reviewer.login == 'YOUR_BOT_LOGIN' - || ... + || (!label && !requested_reviewer + && pull_request.author_association != 'FIRST_TIME_CONTRIBUTOR' + && pull_request.author_association != 'FIRST_TIMER' + && pull_request.author_association != 'NONE' + && !pull_request.draft) ) ``` + +The `review-this` label and `requested_reviewer` branches do not exclude draft PRs — labeling a draft or requesting the bot on a draft will still fire the automation. This is intentional: explicit review requests should be honored regardless of draft status. + + ### Testing After creating the automation: @@ -487,7 +496,7 @@ After creating the automation: 1. Open a non-draft PR in a covered repo (the bot should trigger automatically for established contributors), or: 2. Add the `review-this` label, or request your bot as a reviewer on any PR 3. Watch for the "🔍 Review in progress…" comment — it should appear within a few seconds -4. The full review will follow within 2-5 minutes +4. The full review will typically follow within a few minutes, depending on PR size ## Related Resources diff --git a/openhands/usage/use-cases/qa-changes.mdx b/openhands/usage/use-cases/qa-changes.mdx index eca5bdbb..d22ff4a2 100644 --- a/openhands/usage/use-cases/qa-changes.mdx +++ b/openhands/usage/use-cases/qa-changes.mdx @@ -208,7 +208,7 @@ Create an OpenHands Cloud automation using the Plugin Preset with the following Name: Automated QA: YOUR_ORG/* Plugin: github:OpenHands/extensions (repo_path: plugins/qa-changes) Trigger events: pull_request.opened, pull_request.ready_for_review, pull_request.labeled -Filter: glob(repository.full_name, 'YOUR_ORG/*') && (label.name == 'qa-this' || (!label && !pull_request.draft)) +Filter: glob(repository.full_name, 'YOUR_ORG/*') && (label.name == 'qa-this' || (!label && !pull_request.draft && pull_request.author_association != 'FIRST_TIME_CONTRIBUTOR' && pull_request.author_association != 'FIRST_TIMER' && pull_request.author_association != 'NONE')) The QA agent should: 1. Check out the PR branch From 9f1b621c1e3129d26dbb9fdff29059c45fbd2b8c Mon Sep 17 00:00:00 2001 From: openhands Date: Thu, 28 May 2026 21:21:58 +0000 Subject: [PATCH 06/19] docs: address fourth review - document no-resync behavior, reorder testing steps Co-authored-by: openhands --- openhands/usage/use-cases/code-review.mdx | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/openhands/usage/use-cases/code-review.mdx b/openhands/usage/use-cases/code-review.mdx index ceae2f10..c0e87446 100644 --- a/openhands/usage/use-cases/code-review.mdx +++ b/openhands/usage/use-cases/code-review.mdx @@ -463,6 +463,8 @@ The automation triggers on four conditions: - **`review_requested`** — when your bot account is requested as a reviewer - **`labeled`** — when the `review-this` label is added to any PR +The automation does not re-run when new commits are pushed to an existing PR (`pull_request.synchronize` is intentionally excluded to avoid noisy re-reviews). To request a follow-up review after addressing feedback, re-add the `review-this` label or re-request the reviewer. + The `$AUTOMATION_SESSION_URL` variable is injected by the automation runtime and resolves to a direct link to the conversation (e.g., `https://app.all-hands.dev/conversations/{uuid}`). The prompt includes fallbacks (`$AUTOMATION_API_URL`, then the default app URL) for environments where the variable is not yet available. @@ -493,8 +495,8 @@ The `review-this` label and `requested_reviewer` branches do not exclude draft P After creating the automation: -1. Open a non-draft PR in a covered repo (the bot should trigger automatically for established contributors), or: -2. Add the `review-this` label, or request your bot as a reviewer on any PR +1. Add the `review-this` label to any open PR in a covered repo — this is the most reliable test since it works regardless of author history +2. Alternatively, request your bot as a reviewer on any PR, or open a new non-draft PR (note: the auto-trigger on `opened` requires the PR author to already have contributor history in that specific repo — `FIRST_TIME_CONTRIBUTOR`, `FIRST_TIMER`, and `NONE` associations are excluded) 3. Watch for the "🔍 Review in progress…" comment — it should appear within a few seconds 4. The full review will typically follow within a few minutes, depending on PR size From 113c2b64daf3057fa7a7a427049440619a27ce46 Mon Sep 17 00:00:00 2001 From: openhands Date: Thu, 28 May 2026 21:36:47 +0000 Subject: [PATCH 07/19] docs: address fifth review - multi-line QA filter, GitHub App troubleshooting --- openhands/usage/automations/event-automations.mdx | 3 +++ openhands/usage/use-cases/qa-changes.mdx | 10 +++++++++- 2 files changed, 12 insertions(+), 1 deletion(-) diff --git a/openhands/usage/automations/event-automations.mdx b/openhands/usage/automations/event-automations.mdx index 2a409ad6..0c085785 100644 --- a/openhands/usage/automations/event-automations.mdx +++ b/openhands/usage/automations/event-automations.mdx @@ -53,6 +53,9 @@ If you're using a service account (like a bot account) to create or own automati If your automation doesn't trigger on GitHub events: + + The OpenHands GitHub App must be installed on the GitHub organization that owns your repositories. Go to [GitHub integration settings](/openhands/usage/cloud/github-installation) and verify it is installed with access to the relevant repos. Without this, no webhook events are sent to OpenHands. + The most common cause. Go to **Organization Settings → Git Conversation Routing** and check if your GitHub org shows as claimed. If not, click **Claim**. See [Claiming Git Organizations](/openhands/usage/cloud/organizations/settings#claiming-git-organizations). diff --git a/openhands/usage/use-cases/qa-changes.mdx b/openhands/usage/use-cases/qa-changes.mdx index d22ff4a2..d28ca7c5 100644 --- a/openhands/usage/use-cases/qa-changes.mdx +++ b/openhands/usage/use-cases/qa-changes.mdx @@ -208,7 +208,15 @@ Create an OpenHands Cloud automation using the Plugin Preset with the following Name: Automated QA: YOUR_ORG/* Plugin: github:OpenHands/extensions (repo_path: plugins/qa-changes) Trigger events: pull_request.opened, pull_request.ready_for_review, pull_request.labeled -Filter: glob(repository.full_name, 'YOUR_ORG/*') && (label.name == 'qa-this' || (!label && !pull_request.draft && pull_request.author_association != 'FIRST_TIME_CONTRIBUTOR' && pull_request.author_association != 'FIRST_TIMER' && pull_request.author_association != 'NONE')) +Filter: +glob(repository.full_name, 'YOUR_ORG/*') && ( + label.name == 'qa-this' + || (!label + && !pull_request.draft + && pull_request.author_association != 'FIRST_TIME_CONTRIBUTOR' + && pull_request.author_association != 'FIRST_TIMER' + && pull_request.author_association != 'NONE') +) The QA agent should: 1. Check out the PR branch From c0fd82734c3242511e5d177070e4198174befeff Mon Sep 17 00:00:00 2001 From: openhands Date: Thu, 28 May 2026 21:48:52 +0000 Subject: [PATCH 08/19] docs: address sixth review - QA prompt formatting, JMESPath operators, label creation hint Co-authored-by: openhands --- openhands/usage/automations/event-automations.mdx | 2 +- openhands/usage/use-cases/code-review.mdx | 2 +- openhands/usage/use-cases/qa-changes.mdx | 9 +++++---- 3 files changed, 7 insertions(+), 6 deletions(-) diff --git a/openhands/usage/automations/event-automations.mdx b/openhands/usage/automations/event-automations.mdx index 0c085785..96b68444 100644 --- a/openhands/usage/automations/event-automations.mdx +++ b/openhands/usage/automations/event-automations.mdx @@ -126,7 +126,7 @@ Use wildcards like `pull_request.*` to match all actions for an event type. Filters let you narrow which events trigger your automation. They use [JMESPath expressions](https://jmespath.org/) to match fields in the event payload—so you can trigger only on specific labels, users, branches, or other conditions. -OpenHands extends standard JMESPath with custom functions including `icontains` (case-insensitive string match) and `glob` (wildcard path matching). These are not part of the JMESPath specification. +OpenHands extends standard JMESPath with custom functions including `icontains` (case-insensitive string match) and `glob` (wildcard path matching). It also supports `!` (negation), `&&` (AND), and `||` (OR) as boolean operators. These extensions are not part of the [JMESPath specification](https://jmespath.org/specification.html). **Common filter patterns:** diff --git a/openhands/usage/use-cases/code-review.mdx b/openhands/usage/use-cases/code-review.mdx index c0e87446..d8dc95a7 100644 --- a/openhands/usage/use-cases/code-review.mdx +++ b/openhands/usage/use-cases/code-review.mdx @@ -495,7 +495,7 @@ The `review-this` label and `requested_reviewer` branches do not exclude draft P After creating the automation: -1. Add the `review-this` label to any open PR in a covered repo — this is the most reliable test since it works regardless of author history +1. Add the `review-this` label to any open PR in a covered repo — this is the most reliable test since it works regardless of author history (you may need to [create the label](https://docs.github.com/en/issues/using-labels-and-milestones-to-track-work/managing-labels#creating-a-label) in your repo first if it doesn't exist) 2. Alternatively, request your bot as a reviewer on any PR, or open a new non-draft PR (note: the auto-trigger on `opened` requires the PR author to already have contributor history in that specific repo — `FIRST_TIME_CONTRIBUTOR`, `FIRST_TIMER`, and `NONE` associations are excluded) 3. Watch for the "🔍 Review in progress…" comment — it should appear within a few seconds 4. The full review will typically follow within a few minutes, depending on PR size diff --git a/openhands/usage/use-cases/qa-changes.mdx b/openhands/usage/use-cases/qa-changes.mdx index d28ca7c5..489c02fd 100644 --- a/openhands/usage/use-cases/qa-changes.mdx +++ b/openhands/usage/use-cases/qa-changes.mdx @@ -205,10 +205,10 @@ Log in to [OpenHands Cloud](https://app.all-hands.dev) under your team org and s ``` Create an OpenHands Cloud automation using the Plugin Preset with the following configuration: -Name: Automated QA: YOUR_ORG/* -Plugin: github:OpenHands/extensions (repo_path: plugins/qa-changes) -Trigger events: pull_request.opened, pull_request.ready_for_review, pull_request.labeled -Filter: +**Name:** Automated QA: YOUR_ORG/* +**Plugin:** github:OpenHands/extensions (repo_path: plugins/qa-changes) +**Trigger events:** pull_request.opened, pull_request.ready_for_review, pull_request.labeled +**Filter:** glob(repository.full_name, 'YOUR_ORG/*') && ( label.name == 'qa-this' || (!label @@ -217,6 +217,7 @@ glob(repository.full_name, 'YOUR_ORG/*') && ( && pull_request.author_association != 'FIRST_TIMER' && pull_request.author_association != 'NONE') ) +**Timeout:** 3600 seconds The QA agent should: 1. Check out the PR branch From e3bbe16161b68bb78b354af39f0d566639954435 Mon Sep 17 00:00:00 2001 From: openhands Date: Thu, 28 May 2026 21:58:34 +0000 Subject: [PATCH 09/19] docs: add qa-this label creation hint to qa-changes.mdx Co-authored-by: openhands --- openhands/usage/use-cases/qa-changes.mdx | 2 ++ 1 file changed, 2 insertions(+) diff --git a/openhands/usage/use-cases/qa-changes.mdx b/openhands/usage/use-cases/qa-changes.mdx index 489c02fd..20005ef2 100644 --- a/openhands/usage/use-cases/qa-changes.mdx +++ b/openhands/usage/use-cases/qa-changes.mdx @@ -225,6 +225,8 @@ The QA agent should: 3. Post a structured QA report as a PR comment with evidence (commands run, outputs, screenshots) ``` +When testing, you may need to [create the `qa-this` label](https://docs.github.com/en/issues/using-labels-and-milestones-to-track-work/managing-labels#creating-a-label) in your repo before you can apply it. + For automated QA on every push, use the [qa-changes plugin](https://github.com/OpenHands/extensions/tree/main/plugins/qa-changes) as a GitHub Action instead. For a more detailed automation setup with progress comments and session links, see the [Automated Code Review automation guide](/openhands/usage/use-cases/code-review#automate-this) — the same pattern applies to QA. From c7dd78d21655190f506c33c758172bf73f6d8795 Mon Sep 17 00:00:00 2001 From: openhands Date: Thu, 28 May 2026 23:12:05 +0000 Subject: [PATCH 10/19] docs: add APPROVE/REQUEST_CHANGES/COMMENT guidance to automation prompt --- openhands/usage/use-cases/code-review.mdx | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/openhands/usage/use-cases/code-review.mdx b/openhands/usage/use-cases/code-review.mdx index d8dc95a7..84e838b9 100644 --- a/openhands/usage/use-cases/code-review.mdx +++ b/openhands/usage/use-cases/code-review.mdx @@ -431,7 +431,16 @@ Step 3 — Post a progress comment and save the comment ID: -d "{\"body\": \"🔍 **Review in progress…**\\n\\nWe are performing the review through OpenHands Cloud Automation. You can log in and [view the conversation here](${SESSION_URL}).\"}" \ | python3 -c "import sys,json; print(json.load(sys.stdin)['id'])") -Step 4 — Review the pull request using the pr-review plugin. Post a comprehensive code review on GitHub with inline comments on specific changed lines where appropriate, and a concise overall summary. Avoid duplicating existing unresolved review comments. At the end of the top-level review body include exactly: +Step 4 — Review the pull request using the pr-review plugin. Post a comprehensive code review on GitHub with inline comments on specific changed lines where appropriate, and a concise overall summary. Avoid duplicating existing unresolved review comments. + +When submitting the review, choose the appropriate event type: +- Use "event": "APPROVE" when the PR is ready to merge with no blocking issues (minor suggestions are fine) +- Use "event": "REQUEST_CHANGES" when there are blocking issues that must be fixed before merging +- Use "event": "COMMENT" only when you need more information or are providing an informational review without a clear verdict + +Default to APPROVE for clean PRs — don't use COMMENT when you mean to approve. + +At the end of the top-level review body include exactly: _This review was generated by an AI agent (OpenHands) on behalf of the user through OpenHands Automation. [View conversation](${SESSION_URL})_ Step 5 — After the review is posted, update the progress comment: @@ -454,7 +463,7 @@ Step 5 — After the review is posted, update the progress comment: When the automation is created and a qualifying PR event occurs, the bot will: 1. **Post a progress comment** on the PR: "🔍 Review in progress…" with a link to the live conversation -2. **Run the pr-review plugin** which analyzes the diff and posts a structured code review with inline comments +2. **Run the pr-review plugin** which analyzes the diff and posts a structured code review with inline comments — approving clean PRs, requesting changes when there are blocking issues, or leaving an informational comment when the verdict is unclear 3. **Update the progress comment** to "✅ Review complete." with the conversation link The automation triggers on four conditions: From 9257bba4653f90f6c49ad8dd77a82f6b3cd236a0 Mon Sep 17 00:00:00 2001 From: openhands Date: Fri, 29 May 2026 15:37:36 +0000 Subject: [PATCH 11/19] docs: fix team org link to point to organizations overview page Co-authored-by: openhands --- openhands/usage/automations/event-automations.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/openhands/usage/automations/event-automations.mdx b/openhands/usage/automations/event-automations.mdx index 96b68444..8d001369 100644 --- a/openhands/usage/automations/event-automations.mdx +++ b/openhands/usage/automations/event-automations.mdx @@ -17,7 +17,7 @@ The OpenHands GitHub App must be installed on the GitHub organization that owns If you're working with repositories owned by a GitHub organization (e.g., `myorg/my-repo`), you need an OpenHands **team organization** — not just a personal account. GitHub events for org repos are routed to team orgs, not personal orgs. -If you don't already have one, create a team organization from the [OpenHands Cloud settings](https://app.all-hands.dev/settings). +If you don't already have one, create a team organization — see [What Are Organizations](/openhands/usage/cloud/organizations/overview#what-are-organizations) for details and how to get started. ### 3. Claim Your GitHub Organization From 7a3fbe3266f7c67d5146f339b2d2ff980bd90953 Mon Sep 17 00:00:00 2001 From: openhands Date: Fri, 29 May 2026 16:16:29 +0000 Subject: [PATCH 12/19] docs: add GitHub Action option alongside automation approach for code review Restructure the 'Automate This' section into two options: - Option A: GitHub Action (per-repo) with a ready-to-use workflow example - Option B: OpenHands Automation (org-wide) with the full prompt config This preserves the original GH Action approach while presenting the automation as the newer org-wide alternative. Co-authored-by: openhands --- openhands/usage/use-cases/code-review.mdx | 68 ++++++++++++++++++++--- 1 file changed, 61 insertions(+), 7 deletions(-) diff --git a/openhands/usage/use-cases/code-review.mdx b/openhands/usage/use-cases/code-review.mdx index 84e838b9..a97e5c88 100644 --- a/openhands/usage/use-cases/code-review.mdx +++ b/openhands/usage/use-cases/code-review.mdx @@ -367,15 +367,69 @@ See real automated reviews in action on the OpenHands Software Agent SDK reposit ## Automate This +There are two ways to automate PR reviews with OpenHands: as a **GitHub Action** (per-repo) or as an **OpenHands Automation** (org-wide, event-driven). Choose the approach that fits your needs, or use both. + +### Option A: GitHub Action (Per-Repo) + +Add the [pr-review composite action](https://github.com/OpenHands/extensions/tree/main/plugins/pr-review) to any repository as a GitHub Actions workflow. This gives you fine-grained control over the execution environment and integrates directly with your CI pipeline. + +Create `.github/workflows/pr-review.yml` in your repository: + +```yaml +name: PR Review by OpenHands +on: + pull_request: + types: [opened, ready_for_review, labeled, review_requested] + +# Cancel in-flight reviews when a new one is triggered for the same PR +concurrency: + group: pr-review-${{ github.event.pull_request.number }} + cancel-in-progress: true + +permissions: + contents: read + pull-requests: write + issues: write + +jobs: + review: + name: Code Review + if: > + (github.event.action == 'labeled' && github.event.label.name == 'review-this') + || (github.event.action == 'review_requested' + && github.event.requested_reviewer.login == 'YOUR_BOT_LOGIN') + || (github.event.action == 'opened' && !github.event.pull_request.draft + && github.event.pull_request.author_association != 'FIRST_TIME_CONTRIBUTOR' + && github.event.pull_request.author_association != 'NONE') + || (github.event.action == 'ready_for_review' + && github.event.pull_request.author_association != 'FIRST_TIME_CONTRIBUTOR' + && github.event.pull_request.author_association != 'NONE') + runs-on: ubuntu-latest + steps: + - uses: OpenHands/extensions/plugins/pr-review@main + with: + llm-model: anthropic/claude-sonnet-4-5-20250929 + llm-api-key: ${{ secrets.LLM_API_KEY }} + github-token: ${{ secrets.GITHUB_TOKEN }} +``` + + +The action supports many additional inputs — `llm-base-url` for custom LLM endpoints, `use-sub-agents` for file-level review delegation, `require-evidence` to enforce proof-of-work in PR descriptions, and more. See the [action.yml](https://github.com/OpenHands/extensions/blob/main/plugins/pr-review/action.yml) for all available options. + + +**When to use this:** You want per-repo control, need to integrate with existing CI checks, or want to pin specific action versions per repository. + +### Option B: OpenHands Automation (Org-Wide) + Before setting up an event-driven automation, complete the one-time [prerequisites for GitHub event automations](/openhands/usage/automations/event-automations#prerequisites-for-github-event-automations) — install the GitHub App, create a team org, and claim your GitHub organization. Without these steps, GitHub events will silently never arrive. -[OpenHands Automations](/openhands/usage/automations/overview) is an event-triggered automation system that can replace per-repo GitHub Actions workflows. You define the trigger once and it covers all repositories matching your filter — no per-repo workflow files needed. +[OpenHands Automations](/openhands/usage/automations/overview) is an event-triggered automation system that replaces per-repo GitHub Actions workflows. You define the trigger once and it covers all repositories matching your filter — no per-repo workflow files needed. It also leverages the full OpenHands runtime (browser, tools, sandbox), which GitHub Actions cannot. -**Trade-offs vs GitHub Actions:** Simpler to set up and maintain across many repos, especially at the org level. Also leverages the full OpenHands runtime (browser, tools, sandbox), which GitHub Actions cannot. GitHub Actions gives more control over the execution environment and integrates directly with your CI pipeline. +**When to use this:** You want a single configuration that covers all repos in your org, or you need the full OpenHands runtime for more advanced review workflows. -### Setup: Create the Automation via Prompt +#### Setup: Create the Automation via Prompt Log in to [OpenHands Cloud](https://app.all-hands.dev) as your bot account (or under your team org) and send the following prompt in a new conversation. Replace the placeholders with your values: @@ -408,7 +462,7 @@ glob(repository.full_name, 'YOUR_ORG/*') && ( ) ``` -**Timeout:** 3600 seconds +**Timeout:** 600 seconds **Prompt (use this exactly):** ``` @@ -458,7 +512,7 @@ Step 5 — After the review is posted, update the progress comment: **How `!label` works:** JMESPath treats absent fields as `null`, and `!null` evaluates to `true`. This means the third branch fires for `opened` and `ready_for_review` events (which have no `label` or `requested_reviewer` in the payload), while correctly staying silent for `labeled` and `review_requested` events where those fields are set. -### What This Produces +#### What This Produces When the automation is created and a qualifying PR event occurs, the bot will: @@ -480,7 +534,7 @@ The `$AUTOMATION_SESSION_URL` variable is injected by the automation runtime and The `$AUTOMATION_EVENT_PAYLOAD` variable contains the full GitHub webhook event as JSON. The `$GITHUB_TOKEN` (from the configured GitHub integration) is also automatically available. No additional configuration is needed for any of these variables. -### Single-Repo vs Org-Wide +#### Single-Repo vs Org-Wide The prompt above uses `glob(repository.full_name, 'YOUR_ORG/*')` to cover **all repos** in your org. To target a single repo instead, replace the filter's first condition: @@ -500,7 +554,7 @@ repository.full_name == 'YOUR_ORG/YOUR_REPO' && ( The `review-this` label and `requested_reviewer` branches do not exclude draft PRs — labeling a draft or requesting the bot on a draft will still fire the automation. This is intentional: explicit review requests should be honored regardless of draft status. -### Testing +#### Testing After creating the automation: From a7e1dd2ec983d9889c3438804932e32a56fcc3b1 Mon Sep 17 00:00:00 2001 From: openhands Date: Fri, 29 May 2026 16:24:50 +0000 Subject: [PATCH 13/19] docs: add FIRST_TIMER exclusion to GH Action example for consistency Address review feedback: - Add missing FIRST_TIMER exclusion to GitHub Action if-block to match the automation filter (both now exclude FIRST_TIME_CONTRIBUTOR, FIRST_TIMER, and NONE) - Add comment on llm-model noting users should replace with their model Co-authored-by: openhands --- openhands/usage/use-cases/code-review.mdx | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/openhands/usage/use-cases/code-review.mdx b/openhands/usage/use-cases/code-review.mdx index a97e5c88..743a6916 100644 --- a/openhands/usage/use-cases/code-review.mdx +++ b/openhands/usage/use-cases/code-review.mdx @@ -400,15 +400,17 @@ jobs: && github.event.requested_reviewer.login == 'YOUR_BOT_LOGIN') || (github.event.action == 'opened' && !github.event.pull_request.draft && github.event.pull_request.author_association != 'FIRST_TIME_CONTRIBUTOR' + && github.event.pull_request.author_association != 'FIRST_TIMER' && github.event.pull_request.author_association != 'NONE') || (github.event.action == 'ready_for_review' && github.event.pull_request.author_association != 'FIRST_TIME_CONTRIBUTOR' + && github.event.pull_request.author_association != 'FIRST_TIMER' && github.event.pull_request.author_association != 'NONE') runs-on: ubuntu-latest steps: - uses: OpenHands/extensions/plugins/pr-review@main with: - llm-model: anthropic/claude-sonnet-4-5-20250929 + llm-model: anthropic/claude-sonnet-4-5-20250929 # Replace with your preferred model llm-api-key: ${{ secrets.LLM_API_KEY }} github-token: ${{ secrets.GITHUB_TOKEN }} ``` From 001427f669448a0eb01bd18138bbe05e67fe90c5 Mon Sep 17 00:00:00 2001 From: openhands Date: Fri, 29 May 2026 17:01:23 +0000 Subject: [PATCH 14/19] docs: replace inline GH Action YAMLs with links to example workflows Both code-review.mdx and qa-changes.mdx now link to the canonical example workflows in OpenHands/extensions rather than duplicating the YAML inline. This avoids drift between docs and the actual workflows, and keeps the docs focused on setup instructions. Both Automate This sections follow the same two-option pattern: - Option A: GitHub Action (link to example + action.yml) - Option B: OpenHands Automation (full prompt config) Co-authored-by: openhands --- openhands/usage/use-cases/code-review.mdx | 48 +------------------- openhands/usage/use-cases/qa-changes.mdx | 55 ++++++++--------------- 2 files changed, 20 insertions(+), 83 deletions(-) diff --git a/openhands/usage/use-cases/code-review.mdx b/openhands/usage/use-cases/code-review.mdx index 743a6916..db727e37 100644 --- a/openhands/usage/use-cases/code-review.mdx +++ b/openhands/usage/use-cases/code-review.mdx @@ -371,53 +371,9 @@ There are two ways to automate PR reviews with OpenHands: as a **GitHub Action** ### Option A: GitHub Action (Per-Repo) -Add the [pr-review composite action](https://github.com/OpenHands/extensions/tree/main/plugins/pr-review) to any repository as a GitHub Actions workflow. This gives you fine-grained control over the execution environment and integrates directly with your CI pipeline. +Use the [pr-review plugin](https://github.com/OpenHands/extensions/tree/main/plugins/pr-review) as a GitHub Actions workflow. Copy the [example workflow](https://github.com/OpenHands/extensions/blob/main/plugins/pr-review/workflows/pr-review-by-openhands.yml) into `.github/workflows/pr-review.yml` in your repository, add your `LLM_API_KEY` to **Settings → Secrets and variables → Actions**, and customize the trigger conditions and model as needed. -Create `.github/workflows/pr-review.yml` in your repository: - -```yaml -name: PR Review by OpenHands -on: - pull_request: - types: [opened, ready_for_review, labeled, review_requested] - -# Cancel in-flight reviews when a new one is triggered for the same PR -concurrency: - group: pr-review-${{ github.event.pull_request.number }} - cancel-in-progress: true - -permissions: - contents: read - pull-requests: write - issues: write - -jobs: - review: - name: Code Review - if: > - (github.event.action == 'labeled' && github.event.label.name == 'review-this') - || (github.event.action == 'review_requested' - && github.event.requested_reviewer.login == 'YOUR_BOT_LOGIN') - || (github.event.action == 'opened' && !github.event.pull_request.draft - && github.event.pull_request.author_association != 'FIRST_TIME_CONTRIBUTOR' - && github.event.pull_request.author_association != 'FIRST_TIMER' - && github.event.pull_request.author_association != 'NONE') - || (github.event.action == 'ready_for_review' - && github.event.pull_request.author_association != 'FIRST_TIME_CONTRIBUTOR' - && github.event.pull_request.author_association != 'FIRST_TIMER' - && github.event.pull_request.author_association != 'NONE') - runs-on: ubuntu-latest - steps: - - uses: OpenHands/extensions/plugins/pr-review@main - with: - llm-model: anthropic/claude-sonnet-4-5-20250929 # Replace with your preferred model - llm-api-key: ${{ secrets.LLM_API_KEY }} - github-token: ${{ secrets.GITHUB_TOKEN }} -``` - - -The action supports many additional inputs — `llm-base-url` for custom LLM endpoints, `use-sub-agents` for file-level review delegation, `require-evidence` to enforce proof-of-work in PR descriptions, and more. See the [action.yml](https://github.com/OpenHands/extensions/blob/main/plugins/pr-review/action.yml) for all available options. - +See the [action.yml](https://github.com/OpenHands/extensions/blob/main/plugins/pr-review/action.yml) for all available inputs (`llm-model`, `llm-base-url`, `use-sub-agents`, `require-evidence`, and more). **When to use this:** You want per-repo control, need to integrate with existing CI checks, or want to pin specific action versions per repository. diff --git a/openhands/usage/use-cases/qa-changes.mdx b/openhands/usage/use-cases/qa-changes.mdx index 20005ef2..f9062476 100644 --- a/openhands/usage/use-cases/qa-changes.mdx +++ b/openhands/usage/use-cases/qa-changes.mdx @@ -52,37 +52,7 @@ The QA agent knows when to give up: after exhausting multiple approaches without ### GitHub Actions -Create `.github/workflows/qa-changes.yml` in your repository: - -```yaml -name: QA Changes - -on: - pull_request: - types: [opened, ready_for_review, labeled] - -permissions: - contents: read - pull-requests: write - issues: write - -jobs: - qa: - if: | - (github.event.action == 'opened' && github.event.pull_request.draft == false) || - github.event.action == 'ready_for_review' || - github.event.label.name == 'qa-this' - runs-on: ubuntu-latest - steps: - - name: Run QA Changes - uses: OpenHands/extensions/plugins/qa-changes@main - with: - llm-model: anthropic/claude-sonnet-4-20250514 - llm-api-key: ${{ secrets.LLM_API_KEY }} - github-token: ${{ secrets.GITHUB_TOKEN }} -``` - -Add your `LLM_API_KEY` to your repository's **Settings → Secrets and variables → Actions**. +Copy the [example workflow](https://github.com/OpenHands/extensions/blob/main/plugins/qa-changes/workflows/qa-changes-by-openhands.yml) into `.github/workflows/qa-changes.yml` in your repository and add your `LLM_API_KEY` to **Settings → Secrets and variables → Actions**. See the [action.yml](https://github.com/OpenHands/extensions/blob/main/plugins/qa-changes/action.yml) for all available inputs. ### In a Conversation @@ -195,12 +165,23 @@ The QA agent is most powerful when used alongside the [code review agent](/openh ## Automate This +There are two ways to automate QA testing with OpenHands: as a **GitHub Action** (per-repo) or as an **OpenHands Automation** (org-wide, event-driven). The pattern mirrors the [Automated Code Review](/openhands/usage/use-cases/code-review#automate-this) setup. + +### Option A: GitHub Action (Per-Repo) + +Use the [qa-changes plugin](https://github.com/OpenHands/extensions/tree/main/plugins/qa-changes) as a GitHub Actions workflow. Copy the [example workflow](https://github.com/OpenHands/extensions/blob/main/plugins/qa-changes/workflows/qa-changes-by-openhands.yml) into `.github/workflows/qa-changes.yml` in your repository, add your `LLM_API_KEY` to **Settings → Secrets and variables → Actions**, and customize the trigger conditions and model as needed. + +See the [action.yml](https://github.com/OpenHands/extensions/blob/main/plugins/qa-changes/action.yml) for all available inputs. + +**When to use this:** You want per-repo control, need to integrate with existing CI checks, or want to pin specific action versions per repository. + +### Option B: OpenHands Automation (Org-Wide) + Before setting up an event-driven automation, complete the one-time [prerequisites for GitHub event automations](/openhands/usage/automations/event-automations#prerequisites-for-github-event-automations) — install the GitHub App, create a team org, and claim your GitHub organization. Without these steps, GitHub events will silently never arrive. -You can run QA automatically on every PR using [OpenHands Automations](/openhands/usage/automations/overview). -Log in to [OpenHands Cloud](https://app.all-hands.dev) under your team org and send the following prompt in a new conversation. Replace `YOUR_ORG` with your GitHub organization name: +[OpenHands Automations](/openhands/usage/automations/overview) lets you define the trigger once to cover all repositories matching your filter. Log in to [OpenHands Cloud](https://app.all-hands.dev) under your team org and send the following prompt in a new conversation. Replace `YOUR_ORG` with your GitHub organization name: ``` Create an OpenHands Cloud automation using the Plugin Preset with the following configuration: @@ -217,7 +198,7 @@ glob(repository.full_name, 'YOUR_ORG/*') && ( && pull_request.author_association != 'FIRST_TIMER' && pull_request.author_association != 'NONE') ) -**Timeout:** 3600 seconds +**Timeout:** 600 seconds The QA agent should: 1. Check out the PR branch @@ -225,11 +206,11 @@ The QA agent should: 3. Post a structured QA report as a PR comment with evidence (commands run, outputs, screenshots) ``` -When testing, you may need to [create the `qa-this` label](https://docs.github.com/en/issues/using-labels-and-milestones-to-track-work/managing-labels#creating-a-label) in your repo before you can apply it. +**When to use this:** You want a single configuration that covers all repos in your org, or you need the full OpenHands runtime for more advanced QA workflows. -For automated QA on every push, use the [qa-changes plugin](https://github.com/OpenHands/extensions/tree/main/plugins/qa-changes) as a GitHub Action instead. +When testing, you may need to [create the `qa-this` label](https://docs.github.com/en/issues/using-labels-and-milestones-to-track-work/managing-labels#creating-a-label) in your repo before you can apply it. -For a more detailed automation setup with progress comments and session links, see the [Automated Code Review automation guide](/openhands/usage/use-cases/code-review#automate-this) — the same pattern applies to QA. +For a more detailed automation setup with progress comments and session links, see the [Automated Code Review automation guide](/openhands/usage/use-cases/code-review#option-b-openhands-automation-org-wide) — the same pattern applies to QA. ## Related Resources From 45fa00d228082bfeeb7691457fe3fddceca28a74 Mon Sep 17 00:00:00 2001 From: openhands Date: Fri, 29 May 2026 17:11:03 +0000 Subject: [PATCH 15/19] docs: add /codereview conversation trigger to code-review Quick Start Add 'In a Conversation' section matching the pattern in qa-changes.mdx, showing how to install the code-review skill and invoke /codereview manually in any OpenHands conversation. Co-authored-by: openhands --- openhands/usage/use-cases/code-review.mdx | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/openhands/usage/use-cases/code-review.mdx b/openhands/usage/use-cases/code-review.mdx index db727e37..7a114f1e 100644 --- a/openhands/usage/use-cases/code-review.mdx +++ b/openhands/usage/use-cases/code-review.mdx @@ -101,6 +101,27 @@ The PR review workflow uses the OpenHands Software Agent SDK to analyze your cod +### In a Conversation + +You can also trigger a code review manually in any OpenHands conversation. First, install the skill: + +``` +/add-skill https://github.com/OpenHands/extensions/tree/main/skills/code-review +``` + +Then invoke it: + +``` +/codereview +``` + +The agent will ask for the PR to review, or you can provide context directly: + +``` +/codereview — Please review PR #123 on my-org/my-repo. +Focus on the new authentication middleware. +``` + ## Composite Action From ff32092d7dda72a05d2be2fe455bb52209706077 Mon Sep 17 00:00:00 2001 From: openhands Date: Fri, 29 May 2026 17:34:03 +0000 Subject: [PATCH 16/19] docs: add /codereview skill trigger to automation prompt Step 4 Include /codereview at the start of Step 4 so the code-review skill is explicitly activated when the automation creates a new conversation. Co-authored-by: openhands --- openhands/usage/use-cases/code-review.mdx | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/openhands/usage/use-cases/code-review.mdx b/openhands/usage/use-cases/code-review.mdx index 7a114f1e..571da6f2 100644 --- a/openhands/usage/use-cases/code-review.mdx +++ b/openhands/usage/use-cases/code-review.mdx @@ -464,7 +464,8 @@ Step 3 — Post a progress comment and save the comment ID: -d "{\"body\": \"🔍 **Review in progress…**\\n\\nWe are performing the review through OpenHands Cloud Automation. You can log in and [view the conversation here](${SESSION_URL}).\"}" \ | python3 -c "import sys,json; print(json.load(sys.stdin)['id'])") -Step 4 — Review the pull request using the pr-review plugin. Post a comprehensive code review on GitHub with inline comments on specific changed lines where appropriate, and a concise overall summary. Avoid duplicating existing unresolved review comments. +Step 4 — /codereview +Review the pull request using the pr-review plugin. Post a comprehensive code review on GitHub with inline comments on specific changed lines where appropriate, and a concise overall summary. Avoid duplicating existing unresolved review comments. When submitting the review, choose the appropriate event type: - Use "event": "APPROVE" when the PR is ready to merge with no blocking issues (minor suggestions are fine) From 84cf83bf651de8c4df4ccb54359a8d9852d2d839 Mon Sep 17 00:00:00 2001 From: openhands Date: Fri, 29 May 2026 17:46:10 +0000 Subject: [PATCH 17/19] docs: add bot account prerequisite and expand review_requested explanation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add a 'Prerequisites: Bot Account' section explaining why org-level automations need a dedicated bot account — it's the identity that approves PRs, requests changes, and posts review comments. Expand the review_requested trigger bullet to explain the workflow: team members request the bot from the Reviewers sidebar, and the bot posts reviews under its own GitHub identity. Co-authored-by: openhands --- openhands/usage/use-cases/code-review.mdx | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/openhands/usage/use-cases/code-review.mdx b/openhands/usage/use-cases/code-review.mdx index 571da6f2..7dc0b8ec 100644 --- a/openhands/usage/use-cases/code-review.mdx +++ b/openhands/usage/use-cases/code-review.mdx @@ -408,6 +408,10 @@ Before setting up an event-driven automation, complete the one-time [prerequisit **When to use this:** You want a single configuration that covers all repos in your org, or you need the full OpenHands runtime for more advanced review workflows. +#### Prerequisites: Bot Account + +For org-level automations, you should create a dedicated **bot account** (a separate GitHub user) and add it to your [OpenHands organization](/openhands/usage/cloud/organizations-overview). The bot account is the identity that will approve pull requests, request changes, and post review comments — keeping automated actions separate from human activity. Team members can then request this bot as a reviewer to trigger on-demand reviews. + #### Setup: Create the Automation via Prompt Log in to [OpenHands Cloud](https://app.all-hands.dev) as your bot account (or under your team org) and send the following prompt in a new conversation. Replace the placeholders with your values: @@ -503,7 +507,7 @@ When the automation is created and a qualifying PR event occurs, the bot will: The automation triggers on four conditions: - **`opened`** — when a new non-draft PR is created (for established contributors only) - **`ready_for_review`** — when a draft PR is marked ready (for established contributors only) -- **`review_requested`** — when your bot account is requested as a reviewer +- **`review_requested`** — when your bot account is requested as a reviewer. This is the primary way team members trigger an on-demand review — they simply request the bot from the PR's "Reviewers" sidebar. The bot then posts its review under its own GitHub identity, so approvals and change requests come from a clear, dedicated account. - **`labeled`** — when the `review-this` label is added to any PR The automation does not re-run when new commits are pushed to an existing PR (`pull_request.synchronize` is intentionally excluded to avoid noisy re-reviews). To request a follow-up review after addressing feedback, re-add the `review-this` label or re-request the reviewer. From bc64dc98d7da3c3ab955667e6340d762d257cc34 Mon Sep 17 00:00:00 2001 From: Xingyao Wang Date: Fri, 29 May 2026 13:55:48 -0400 Subject: [PATCH 18/19] Apply suggestion from @xingyaoww --- openhands/usage/use-cases/code-review.mdx | 2 -- 1 file changed, 2 deletions(-) diff --git a/openhands/usage/use-cases/code-review.mdx b/openhands/usage/use-cases/code-review.mdx index 7dc0b8ec..92ede4b8 100644 --- a/openhands/usage/use-cases/code-review.mdx +++ b/openhands/usage/use-cases/code-review.mdx @@ -476,8 +476,6 @@ When submitting the review, choose the appropriate event type: - Use "event": "REQUEST_CHANGES" when there are blocking issues that must be fixed before merging - Use "event": "COMMENT" only when you need more information or are providing an informational review without a clear verdict -Default to APPROVE for clean PRs — don't use COMMENT when you mean to approve. - At the end of the top-level review body include exactly: _This review was generated by an AI agent (OpenHands) on behalf of the user through OpenHands Automation. [View conversation](${SESSION_URL})_ From 6e1660e1696668d4f74c5bad7b05d41c58314e6d Mon Sep 17 00:00:00 2001 From: openhands Date: Fri, 29 May 2026 18:03:56 +0000 Subject: [PATCH 19/19] fix: correct broken internal link to organizations overview page Change /openhands/usage/cloud/organizations-overview to /openhands/usage/cloud/organizations/overview to fix the Mintlify broken link check. Co-authored-by: openhands --- openhands/usage/use-cases/code-review.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/openhands/usage/use-cases/code-review.mdx b/openhands/usage/use-cases/code-review.mdx index 92ede4b8..0260fab4 100644 --- a/openhands/usage/use-cases/code-review.mdx +++ b/openhands/usage/use-cases/code-review.mdx @@ -410,7 +410,7 @@ Before setting up an event-driven automation, complete the one-time [prerequisit #### Prerequisites: Bot Account -For org-level automations, you should create a dedicated **bot account** (a separate GitHub user) and add it to your [OpenHands organization](/openhands/usage/cloud/organizations-overview). The bot account is the identity that will approve pull requests, request changes, and post review comments — keeping automated actions separate from human activity. Team members can then request this bot as a reviewer to trigger on-demand reviews. +For org-level automations, you should create a dedicated **bot account** (a separate GitHub user) and add it to your [OpenHands organization](/openhands/usage/cloud/organizations/overview). The bot account is the identity that will approve pull requests, request changes, and post review comments — keeping automated actions separate from human activity. Team members can then request this bot as a reviewer to trigger on-demand reviews. #### Setup: Create the Automation via Prompt