diff --git a/openhands/usage/automations/event-automations.mdx b/openhands/usage/automations/event-automations.mdx index cc4a1ace..8d001369 100644 --- a/openhands/usage/automations/event-automations.mdx +++ b/openhands/usage/automations/event-automations.mdx @@ -5,11 +5,78 @@ description: Trigger automations from GitHub events or custom webhooks instead o Event-based automations run when something happens—a PR is opened, an issue is commented on, or a webhook fires—instead of on a schedule. This is ideal for responsive workflows like auto-reviewing PRs, triaging issues, or reacting to external service events. +## Prerequisites for GitHub Event Automations + +GitHub event automations require some one-time setup before events will flow. If any step is missing, automations will appear to work (manual triggers succeed) but GitHub events will silently never arrive. + +### 1. Install the OpenHands GitHub App + +The OpenHands GitHub App must be installed on the GitHub organization that owns the repositories you want to monitor. Install it from your [GitHub integration settings](/openhands/usage/cloud/github-installation). The app needs access to the repositories that will generate events. + +### 2. Create an OpenHands Team Organization + +If you're working with repositories owned by a GitHub organization (e.g., `myorg/my-repo`), you need an OpenHands **team organization** — not just a personal account. GitHub events for org repos are routed to team orgs, not personal orgs. + +If you don't already have one, create a team organization — see [What Are Organizations](/openhands/usage/cloud/organizations/overview#what-are-organizations) for details and how to get started. + +### 3. Claim Your GitHub Organization + + +**This is the most commonly missed step.** Without it, GitHub events have nowhere to be routed and will be silently dropped. + + +Your OpenHands team org must **claim** the GitHub organization to establish the link between GitHub webhooks and your OpenHands org. Claiming tells the event router: _"Events for repos in this GitHub org should go to this OpenHands team org."_ + +To claim a GitHub org: + +1. Switch to your team org using the org switcher in the sidebar +2. Go to **Organization Settings** +3. In the **Git Conversation Routing** section, find your GitHub org +4. Click **Claim** + +You must be an **Owner** of the OpenHands team org and have **admin access** to the GitHub org to complete the claim. See [Claiming Git Organizations](/openhands/usage/cloud/organizations/settings#claiming-git-organizations) for full details. + + +Each GitHub organization can only be claimed by one OpenHands team org. If another team has already claimed it, coordinate with them or contact support. + + +### 4. Create the Automation Under the Team Org + +Make sure you are switched to the **team org** (not your personal org) when creating the automation. The automation must live in the same org that claimed the GitHub organization — otherwise events won't match. + +### 5. (Optional) Add Service Accounts to the Team Org + +If you're using a service account (like a bot account) to create or own automations, that account must be a **member of the team org**. Invite them from the [Organization Members](/openhands/usage/cloud/organizations/managing-members) page. + +### Troubleshooting + +If your automation doesn't trigger on GitHub events: + + + + The OpenHands GitHub App must be installed on the GitHub organization that owns your repositories. Go to [GitHub integration settings](/openhands/usage/cloud/github-installation) and verify it is installed with access to the relevant repos. Without this, no webhook events are sent to OpenHands. + + + The most common cause. Go to **Organization Settings → Git Conversation Routing** and check if your GitHub org shows as claimed. If not, click **Claim**. See [Claiming Git Organizations](/openhands/usage/cloud/organizations/settings#claiming-git-organizations). + + + GitHub events for org repos are routed to the **team org** that claimed the GitHub org. If you created the automation under your personal org, events will never reach it. Switch to the team org and recreate the automation. + + + Double-check that the event type (e.g., `pull_request.labeled`) and filter expression match the action you're testing. Use wildcards like `pull_request.*` to match all actions during debugging. + + + Verify the automation is enabled. You can check via the automations list or by asking OpenHands to list your automations. + + + +--- + ## Built-In vs. Custom Integrations | Type | Setup | Best For | |------|-------|----------| -| **Built-in (GitHub)** | None—just create the automation | PR reviews, issue triage, push-triggered tasks | +| **Built-in (GitHub)** | One-time org setup ([see above](#prerequisites-for-github-event-automations)), then create the automation | PR reviews, issue triage, push-triggered tasks | | **Custom Webhooks** | Register webhook first, then create automation | Linear, Stripe, Slack, and other services | ## GitHub Events (Built-In) @@ -58,25 +125,30 @@ Use wildcards like `pull_request.*` to match all actions for an event type. Filters let you narrow which events trigger your automation. They use [JMESPath expressions](https://jmespath.org/) to match fields in the event payload—so you can trigger only on specific labels, users, branches, or other conditions. + +OpenHands extends standard JMESPath with custom functions including `icontains` (case-insensitive string match) and `glob` (wildcard path matching). It also supports `!` (negation), `&&` (AND), and `||` (OR) as boolean operators. These extensions are not part of the [JMESPath specification](https://jmespath.org/specification.html). + + **Common filter patterns:** -```javascript -// Match a specific label +``` contains(pull_request.labels[].name, 'openhands') -// Case-insensitive mention in comment icontains(comment.body, '@openhands') -// Match repos in your org glob(repository.full_name, 'myorg/*') -// Push to main branch only ref == 'refs/heads/main' -// Combine conditions glob(repository.full_name, 'myorg/*') && contains(pull_request.labels[].name, 'bug') ``` +- `contains(...)` — match a specific label +- `icontains(...)` — case-insensitive mention in a comment body +- `glob(...)` — match repos in your org with wildcards +- `==` — exact match (e.g., push to main branch only) +- `&&` — combine multiple conditions + --- ## Custom Webhooks @@ -190,12 +262,12 @@ When registering any custom webhook, these parameters define how OpenHands proce These are example configurations for popular services. **Always verify with each service's webhook documentation**, as signature headers and payload formats may change. -| Service | Signature Header | Event Key | -|---------|-----------------|-----------| -| Linear | `Linear-Signature` | `type` | -| Stripe | `Stripe-Signature` | `type` | -| Slack | `X-Slack-Signature` | `type` | -| Twilio | `X-Twilio-Signature` | `type` | +| Service | Signature Header | Event Key | Notes | +|---------|-----------------|-----------|-------| +| Linear | `Linear-Signature` | `type` | | +| Stripe | `Stripe-Signature` | `type` | Uses a custom `t=timestamp,v1=signature` format — verify compatibility | +| Slack | `X-Slack-Signature` | `type` | | +| Twilio | `X-Twilio-Signature` | `type` | Uses HMAC-SHA1 of request URL + params — verify compatibility | --- diff --git a/openhands/usage/automations/overview.mdx b/openhands/usage/automations/overview.mdx index 2ceb734e..cb268bcd 100644 --- a/openhands/usage/automations/overview.mdx +++ b/openhands/usage/automations/overview.mdx @@ -118,7 +118,7 @@ Each use case has a ready-to-use automation prompt. Click a card to see the full Functionally test PR changes by exercising the software as a real user would. diff --git a/openhands/usage/use-cases/code-review.mdx b/openhands/usage/use-cases/code-review.mdx index 5251a977..0260fab4 100644 --- a/openhands/usage/use-cases/code-review.mdx +++ b/openhands/usage/use-cases/code-review.mdx @@ -101,6 +101,27 @@ The PR review workflow uses the OpenHands Software Agent SDK to analyze your cod +### In a Conversation + +You can also trigger a code review manually in any OpenHands conversation. First, install the skill: + +``` +/add-skill https://github.com/OpenHands/extensions/tree/main/skills/code-review +``` + +Then invoke it: + +``` +/codereview +``` + +The agent will ask for the PR to review, or you can provide context directly: + +``` +/codereview — Please review PR #123 on my-org/my-repo. +Focus on the new authentication middleware. +``` + ## Composite Action @@ -367,23 +388,162 @@ See real automated reviews in action on the OpenHands Software Agent SDK reposit ## Automate This -You can schedule daily code reviews using [OpenHands Automations](/openhands/usage/automations/overview). -Copy this prompt into a new conversation to set one up: +There are two ways to automate PR reviews with OpenHands: as a **GitHub Action** (per-repo) or as an **OpenHands Automation** (org-wide, event-driven). Choose the approach that fits your needs, or use both. + +### Option A: GitHub Action (Per-Repo) + +Use the [pr-review plugin](https://github.com/OpenHands/extensions/tree/main/plugins/pr-review) as a GitHub Actions workflow. Copy the [example workflow](https://github.com/OpenHands/extensions/blob/main/plugins/pr-review/workflows/pr-review-by-openhands.yml) into `.github/workflows/pr-review.yml` in your repository, add your `LLM_API_KEY` to **Settings → Secrets and variables → Actions**, and customize the trigger conditions and model as needed. + +See the [action.yml](https://github.com/OpenHands/extensions/blob/main/plugins/pr-review/action.yml) for all available inputs (`llm-model`, `llm-base-url`, `use-sub-agents`, `require-evidence`, and more). + +**When to use this:** You want per-repo control, need to integrate with existing CI checks, or want to pin specific action versions per repository. + +### Option B: OpenHands Automation (Org-Wide) + + +Before setting up an event-driven automation, complete the one-time [prerequisites for GitHub event automations](/openhands/usage/automations/event-automations#prerequisites-for-github-event-automations) — install the GitHub App, create a team org, and claim your GitHub organization. Without these steps, GitHub events will silently never arrive. + + +[OpenHands Automations](/openhands/usage/automations/overview) is an event-triggered automation system that replaces per-repo GitHub Actions workflows. You define the trigger once and it covers all repositories matching your filter — no per-repo workflow files needed. It also leverages the full OpenHands runtime (browser, tools, sandbox), which GitHub Actions cannot. + +**When to use this:** You want a single configuration that covers all repos in your org, or you need the full OpenHands runtime for more advanced review workflows. + +#### Prerequisites: Bot Account + +For org-level automations, you should create a dedicated **bot account** (a separate GitHub user) and add it to your [OpenHands organization](/openhands/usage/cloud/organizations/overview). The bot account is the identity that will approve pull requests, request changes, and post review comments — keeping automated actions separate from human activity. Team members can then request this bot as a reviewer to trigger on-demand reviews. + +#### Setup: Create the Automation via Prompt + +Log in to [OpenHands Cloud](https://app.all-hands.dev) as your bot account (or under your team org) and send the following prompt in a new conversation. Replace the placeholders with your values: + +- `YOUR_ORG` — your GitHub organization name (e.g., `mycompany`) +- `YOUR_BOT_LOGIN` — the GitHub username of your bot account (e.g., `mycompany-bot`) + +```` +Create an OpenHands Cloud automation using the Plugin Preset with the following configuration: +**Name:** PR Review: YOUR_ORG/* (ready for review, review-this, or reviewer requested) + +**Plugin:** github:OpenHands/extensions (repo_path: plugins/pr-review) + +**Trigger events:** +- pull_request.opened +- pull_request.ready_for_review +- pull_request.review_requested +- pull_request.labeled + +**Filter:** +``` +glob(repository.full_name, 'YOUR_ORG/*') && ( + label.name == 'review-this' + || requested_reviewer.login == 'YOUR_BOT_LOGIN' + || (!label && !requested_reviewer + && pull_request.author_association != 'FIRST_TIME_CONTRIBUTOR' + && pull_request.author_association != 'FIRST_TIMER' + && pull_request.author_association != 'NONE' + && !pull_request.draft) +) +``` + +**Timeout:** 600 seconds + +**Prompt (use this exactly):** ``` -Create an automation called "Daily Code Review" that runs every weekday at 9 AM. +Before starting the code review, complete these steps in order: + +Step 1 — Build the session URL. +Run this in terminal: + SESSION_URL="${AUTOMATION_SESSION_URL:-${AUTOMATION_API_URL:-https://app.all-hands.dev}}" + echo "SESSION_URL=${SESSION_URL}" + +Step 2 — Extract PR info from the event payload: + PR_NUMBER=$(echo "$AUTOMATION_EVENT_PAYLOAD" | python3 -c "import sys,json; p=json.load(sys.stdin); print(p['pull_request']['number'])") + REPO=$(echo "$AUTOMATION_EVENT_PAYLOAD" | python3 -c "import sys,json; p=json.load(sys.stdin); print(p['repository']['full_name'])") + +Step 3 — Post a progress comment and save the comment ID: + COMMENT_ID=$(curl -s -X POST \ + -H "Authorization: Bearer $GITHUB_TOKEN" \ + -H "Accept: application/vnd.github+json" \ + "https://api.github.com/repos/$REPO/issues/$PR_NUMBER/comments" \ + -d "{\"body\": \"🔍 **Review in progress…**\\n\\nWe are performing the review through OpenHands Cloud Automation. You can log in and [view the conversation here](${SESSION_URL}).\"}" \ + | python3 -c "import sys,json; print(json.load(sys.stdin)['id'])") + +Step 4 — /codereview +Review the pull request using the pr-review plugin. Post a comprehensive code review on GitHub with inline comments on specific changed lines where appropriate, and a concise overall summary. Avoid duplicating existing unresolved review comments. + +When submitting the review, choose the appropriate event type: +- Use "event": "APPROVE" when the PR is ready to merge with no blocking issues (minor suggestions are fine) +- Use "event": "REQUEST_CHANGES" when there are blocking issues that must be fixed before merging +- Use "event": "COMMENT" only when you need more information or are providing an informational review without a clear verdict + +At the end of the top-level review body include exactly: + _This review was generated by an AI agent (OpenHands) on behalf of the user through OpenHands Automation. [View conversation](${SESSION_URL})_ + +Step 5 — After the review is posted, update the progress comment: + curl -s -X PATCH \ + -H "Authorization: Bearer $GITHUB_TOKEN" \ + -H "Accept: application/vnd.github+json" \ + "https://api.github.com/repos/$REPO/issues/comments/$COMMENT_ID" \ + -d "{\"body\": \"✅ **Review complete.**\\n\\nThis review was performed through OpenHands Cloud Automation. You can log in and [view the conversation here](${SESSION_URL}).\"}" +``` +```` + + +**Team review requests:** The `requested_reviewer` field is only populated for individual reviewer requests. When a *team* is requested as reviewer, GitHub uses `requested_team` instead. To also match team requests, add `|| requested_team.slug == 'YOUR_TEAM_SLUG'` to the filter. + +**How `!label` works:** JMESPath treats absent fields as `null`, and `!null` evaluates to `true`. This means the third branch fires for `opened` and `ready_for_review` events (which have no `label` or `requested_reviewer` in the payload), while correctly staying silent for `labeled` and `review_requested` events where those fields are set. + + +#### What This Produces + +When the automation is created and a qualifying PR event occurs, the bot will: -It should: -1. Find all open PRs that have no reviews yet -2. For each PR, review the diff for bugs, style issues, and security concerns -3. Post a summary of findings as a comment on each PR +1. **Post a progress comment** on the PR: "🔍 Review in progress…" with a link to the live conversation +2. **Run the pr-review plugin** which analyzes the diff and posts a structured code review with inline comments — approving clean PRs, requesting changes when there are blocking issues, or leaving an informational comment when the verdict is unclear +3. **Update the progress comment** to "✅ Review complete." with the conversation link -Learn more at https://docs.openhands.dev/openhands/usage/use-cases/code-review +The automation triggers on four conditions: +- **`opened`** — when a new non-draft PR is created (for established contributors only) +- **`ready_for_review`** — when a draft PR is marked ready (for established contributors only) +- **`review_requested`** — when your bot account is requested as a reviewer. This is the primary way team members trigger an on-demand review — they simply request the bot from the PR's "Reviewers" sidebar. The bot then posts its review under its own GitHub identity, so approvals and change requests come from a clear, dedicated account. +- **`labeled`** — when the `review-this` label is added to any PR + +The automation does not re-run when new commits are pushed to an existing PR (`pull_request.synchronize` is intentionally excluded to avoid noisy re-reviews). To request a follow-up review after addressing feedback, re-add the `review-this` label or re-request the reviewer. + + +The `$AUTOMATION_SESSION_URL` variable is injected by the automation runtime and resolves to a direct link to the conversation (e.g., `https://app.all-hands.dev/conversations/{uuid}`). The prompt includes fallbacks (`$AUTOMATION_API_URL`, then the default app URL) for environments where the variable is not yet available. + +The `$AUTOMATION_EVENT_PAYLOAD` variable contains the full GitHub webhook event as JSON. The `$GITHUB_TOKEN` (from the configured GitHub integration) is also automatically available. No additional configuration is needed for any of these variables. + + +#### Single-Repo vs Org-Wide + +The prompt above uses `glob(repository.full_name, 'YOUR_ORG/*')` to cover **all repos** in your org. To target a single repo instead, replace the filter's first condition: + +``` +repository.full_name == 'YOUR_ORG/YOUR_REPO' && ( + label.name == 'review-this' + || requested_reviewer.login == 'YOUR_BOT_LOGIN' + || (!label && !requested_reviewer + && pull_request.author_association != 'FIRST_TIME_CONTRIBUTOR' + && pull_request.author_association != 'FIRST_TIMER' + && pull_request.author_association != 'NONE' + && !pull_request.draft) +) ``` -For inline review comments on every push, use the -[pr-review plugin](https://github.com/OpenHands/extensions/tree/main/plugins/pr-review) -as a GitHub Action instead. + +The `review-this` label and `requested_reviewer` branches do not exclude draft PRs — labeling a draft or requesting the bot on a draft will still fire the automation. This is intentional: explicit review requests should be honored regardless of draft status. + + +#### Testing + +After creating the automation: + +1. Add the `review-this` label to any open PR in a covered repo — this is the most reliable test since it works regardless of author history (you may need to [create the label](https://docs.github.com/en/issues/using-labels-and-milestones-to-track-work/managing-labels#creating-a-label) in your repo first if it doesn't exist) +2. Alternatively, request your bot as a reviewer on any PR, or open a new non-draft PR (note: the auto-trigger on `opened` requires the PR author to already have contributor history in that specific repo — `FIRST_TIME_CONTRIBUTOR`, `FIRST_TIMER`, and `NONE` associations are excluded) +3. Watch for the "🔍 Review in progress…" comment — it should appear within a few seconds +4. The full review will typically follow within a few minutes, depending on PR size ## Related Resources diff --git a/openhands/usage/use-cases/qa-changes.mdx b/openhands/usage/use-cases/qa-changes.mdx index 5291334c..f9062476 100644 --- a/openhands/usage/use-cases/qa-changes.mdx +++ b/openhands/usage/use-cases/qa-changes.mdx @@ -52,37 +52,7 @@ The QA agent knows when to give up: after exhausting multiple approaches without ### GitHub Actions -Create `.github/workflows/qa-changes.yml` in your repository: - -```yaml -name: QA Changes - -on: - pull_request: - types: [opened, ready_for_review, labeled] - -permissions: - contents: read - pull-requests: write - issues: write - -jobs: - qa: - if: | - (github.event.action == 'opened' && github.event.pull_request.draft == false) || - github.event.action == 'ready_for_review' || - github.event.label.name == 'qa-this' - runs-on: ubuntu-latest - steps: - - name: Run QA Changes - uses: OpenHands/extensions/plugins/qa-changes@main - with: - llm-model: anthropic/claude-sonnet-4-20250514 - llm-api-key: ${{ secrets.LLM_API_KEY }} - github-token: ${{ secrets.GITHUB_TOKEN }} -``` - -Add your `LLM_API_KEY` to your repository's **Settings → Secrets and variables → Actions**. +Copy the [example workflow](https://github.com/OpenHands/extensions/blob/main/plugins/qa-changes/workflows/qa-changes-by-openhands.yml) into `.github/workflows/qa-changes.yml` in your repository and add your `LLM_API_KEY` to **Settings → Secrets and variables → Actions**. See the [action.yml](https://github.com/OpenHands/extensions/blob/main/plugins/qa-changes/action.yml) for all available inputs. ### In a Conversation @@ -193,6 +163,55 @@ The QA agent is most powerful when used alongside the [code review agent](/openh +## Automate This + +There are two ways to automate QA testing with OpenHands: as a **GitHub Action** (per-repo) or as an **OpenHands Automation** (org-wide, event-driven). The pattern mirrors the [Automated Code Review](/openhands/usage/use-cases/code-review#automate-this) setup. + +### Option A: GitHub Action (Per-Repo) + +Use the [qa-changes plugin](https://github.com/OpenHands/extensions/tree/main/plugins/qa-changes) as a GitHub Actions workflow. Copy the [example workflow](https://github.com/OpenHands/extensions/blob/main/plugins/qa-changes/workflows/qa-changes-by-openhands.yml) into `.github/workflows/qa-changes.yml` in your repository, add your `LLM_API_KEY` to **Settings → Secrets and variables → Actions**, and customize the trigger conditions and model as needed. + +See the [action.yml](https://github.com/OpenHands/extensions/blob/main/plugins/qa-changes/action.yml) for all available inputs. + +**When to use this:** You want per-repo control, need to integrate with existing CI checks, or want to pin specific action versions per repository. + +### Option B: OpenHands Automation (Org-Wide) + + +Before setting up an event-driven automation, complete the one-time [prerequisites for GitHub event automations](/openhands/usage/automations/event-automations#prerequisites-for-github-event-automations) — install the GitHub App, create a team org, and claim your GitHub organization. Without these steps, GitHub events will silently never arrive. + + +[OpenHands Automations](/openhands/usage/automations/overview) lets you define the trigger once to cover all repositories matching your filter. Log in to [OpenHands Cloud](https://app.all-hands.dev) under your team org and send the following prompt in a new conversation. Replace `YOUR_ORG` with your GitHub organization name: + +``` +Create an OpenHands Cloud automation using the Plugin Preset with the following configuration: + +**Name:** Automated QA: YOUR_ORG/* +**Plugin:** github:OpenHands/extensions (repo_path: plugins/qa-changes) +**Trigger events:** pull_request.opened, pull_request.ready_for_review, pull_request.labeled +**Filter:** +glob(repository.full_name, 'YOUR_ORG/*') && ( + label.name == 'qa-this' + || (!label + && !pull_request.draft + && pull_request.author_association != 'FIRST_TIME_CONTRIBUTOR' + && pull_request.author_association != 'FIRST_TIMER' + && pull_request.author_association != 'NONE') +) +**Timeout:** 600 seconds + +The QA agent should: +1. Check out the PR branch +2. Exercise the changed behavior as a real user would +3. Post a structured QA report as a PR comment with evidence (commands run, outputs, screenshots) +``` + +**When to use this:** You want a single configuration that covers all repos in your org, or you need the full OpenHands runtime for more advanced QA workflows. + +When testing, you may need to [create the `qa-this` label](https://docs.github.com/en/issues/using-labels-and-milestones-to-track-work/managing-labels#creating-a-label) in your repo before you can apply it. + +For a more detailed automation setup with progress comments and session links, see the [Automated Code Review automation guide](/openhands/usage/use-cases/code-review#option-b-openhands-automation-org-wide) — the same pattern applies to QA. + ## Related Resources - [QA Changes Plugin](https://github.com/OpenHands/extensions/tree/main/plugins/qa-changes) — GitHub Actions plugin