Skip to content

Select affected Cloud integration suites #413

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
sdairs wants to merge 1 commit into from
Open

Select affected Cloud integration suites #413

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
156 changes: 138 additions & 18 deletions .github/workflows/cloud-integration.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,8 +30,8 @@ concurrency:
cancel-in-progress: false

jobs:
cloud-integration:
name: Cloud integration tests
plan:
name: Plan Cloud integration tests
if: >-
github.event_name == 'schedule' ||
github.event_name == 'workflow_dispatch' ||
Expand All @@ -40,6 +40,127 @@ jobs:
github.event.pull_request.head.repo.full_name == github.repository &&
github.event.pull_request.user.login != 'dependabot[bot]')
runs-on: ubuntu-latest
outputs:
service: ${{ steps.selection.outputs.service }}
postgres: ${{ steps.selection.outputs.postgres }}
organization: ${{ steps.selection.outputs.organization }}
clickpipes: ${{ steps.selection.outputs.clickpipes }}
test_sha: ${{ steps.selection.outputs.test_sha }}
requested_scope: ${{ steps.selection.outputs.requested_scope }}
selected_suites: ${{ steps.selection.outputs.selected_suites }}
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
fetch-depth: 0
persist-credentials: false

- name: Select suites and log plan
id: selection
env:
EVENT_NAME: ${{ github.event_name }}
EVENT_REF: ${{ github.ref }}
EVENT_SHA: ${{ github.sha }}
PR_BASE_SHA: ${{ github.event.pull_request.base.sha || 'n/a' }}
PR_HEAD_SHA: ${{ github.event.pull_request.head.sha || 'n/a' }}
DISPATCH_SCOPE: ${{ inputs.scope || '' }}
run: |
set -u
all_suites="service,postgres,organization,clickpipes"
test_sha="$EVENT_SHA"

case "$EVENT_NAME" in
pull_request)
test_sha="$PR_HEAD_SHA"
requested_scope="affected (PR immediate diff)"
if selected="$(python3 scripts/classify-cloud-integration.py "$PR_BASE_SHA" "$PR_HEAD_SHA")"; then
:
else
echo "::warning::Classifier command failed; selecting all suites"
selected="$all_suites"
fi

@cursor cursor Bot Aug 14, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suite selection trusts PR classifier

Medium Severity

The plan job runs classify-cloud-integration.py from the PR head to decide which live suites run, including whether to skip the environment job entirely. ALWAYS_ALL_PATHS is meant to force all suites when that script changes, but that rule lives inside the same PR-controlled script, so a head revision that still emits a valid canonical string such as none can skip cloud integration while the workflow stays green.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 1a4fd7d. Configure here.

;;
Comment on lines +72 to +82
workflow_dispatch)
requested_scope="$DISPATCH_SCOPE"
if [[ "$DISPATCH_SCOPE" == "all" ]]; then
selected="$all_suites"
else
selected="$DISPATCH_SCOPE"
fi
;;
schedule)
requested_scope="all (schedule)"
selected="$all_suites"
;;
*)
echo "::warning::Unexpected event; selecting all suites"
requested_scope="all (fallback)"
selected="$all_suites"
;;
esac

service=false
postgres=false
organization=false
clickpipes=false
[[ ",$selected," == *",service,"* ]] && service=true
[[ ",$selected," == *",postgres,"* ]] && postgres=true
[[ ",$selected," == *",organization,"* ]] && organization=true
[[ ",$selected," == *",clickpipes,"* ]] && clickpipes=true

canonical=""
[[ "$service" == true ]] && canonical="service"
[[ "$postgres" == true ]] && canonical="${canonical:+$canonical,}postgres"
[[ "$organization" == true ]] && canonical="${canonical:+$canonical,}organization"
[[ "$clickpipes" == true ]] && canonical="${canonical:+$canonical,}clickpipes"
canonical="${canonical:-none}"
if [[ "$selected" != "$canonical" ]]; then
echo "::warning::Classifier emitted invalid suite output; selecting all suites"
selected="$all_suites"
service=true
postgres=true
organization=true
clickpipes=true
fi

checked_out_sha="$(git rev-parse HEAD)"
echo "Event: $EVENT_NAME"
echo "Event SHA: $EVENT_SHA"
echo "PR base SHA: $PR_BASE_SHA"
echo "PR head SHA: $PR_HEAD_SHA"
echo "Selected SHA: $test_sha"
echo "Checked-out SHA: $checked_out_sha"
echo "Event ref: $EVENT_REF"
echo "Requested scope: $requested_scope"
echo "Selected suites: $selected"

{
echo "service=$service"
echo "postgres=$postgres"
echo "organization=$organization"
echo "clickpipes=$clickpipes"
echo "test_sha=$test_sha"
echo "requested_scope=$requested_scope"
echo "selected_suites=$selected"
} >> "$GITHUB_OUTPUT"

if [[ "$checked_out_sha" != "$test_sha" ]]; then
echo "::error::Checked-out SHA does not match selected test SHA"
exit 1
fi

cloud-integration:
name: Cloud integration tests
needs: plan
if: >-
${{
needs.plan.result == 'success' &&
(needs.plan.outputs.service == 'true' ||
needs.plan.outputs.postgres == 'true' ||
needs.plan.outputs.organization == 'true' ||
needs.plan.outputs.clickpipes == 'true')
}}
runs-on: ubuntu-latest
environment: cloud-integration
env:
CLICKHOUSE_CLOUD_API_KEY: ${{ secrets.CLICKHOUSE_CLOUD_API_KEY }}
Expand All @@ -61,25 +182,29 @@ jobs:
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
ref: ${{ needs.plan.outputs.test_sha }}
persist-credentials: false

- name: Log revisions
env:
EVENT_NAME: ${{ github.event_name }}
EVENT_SHA: ${{ github.sha }}
PR_BASE_SHA: ${{ github.event.pull_request.base.sha || 'n/a' }}
PR_HEAD_SHA: ${{ github.event.pull_request.head.sha || 'n/a' }}
SELECTED_REF: ${{ github.ref }}
SELECTED_SHA: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
REQUESTED_SCOPE: ${{ github.event_name == 'workflow_dispatch' && inputs.scope || 'all (non-dispatch)' }}
EVENT_REF: ${{ github.ref }}
SELECTED_SHA: ${{ needs.plan.outputs.test_sha }}
REQUESTED_SCOPE: ${{ needs.plan.outputs.requested_scope }}
SELECTED_SUITES: ${{ needs.plan.outputs.selected_suites }}
run: |
echo "Event: $EVENT_NAME"
echo "Event SHA: $EVENT_SHA"
echo "PR base SHA: $PR_BASE_SHA"
echo "PR head SHA: $PR_HEAD_SHA"
echo "Selected ref: $SELECTED_REF"
echo "Selected SHA: $SELECTED_SHA"
echo "Requested scope: $REQUESTED_SCOPE"
echo "Checked-out SHA: $(git rev-parse HEAD)"
echo "Event ref: $EVENT_REF"
echo "Requested scope: $REQUESTED_SCOPE"
echo "Selected suites: $SELECTED_SUITES"

- name: Resolve test run label
run: |
Expand All @@ -101,29 +226,25 @@ jobs:
- name: Run cloud integration suite
if: >-
${{ !cancelled() &&
(github.event_name != 'workflow_dispatch' ||
inputs.scope == 'all' || inputs.scope == 'service') }}
needs.plan.outputs.service == 'true' }}
run: cargo test -p clickhouse-cloud-api --test integration_test -- --ignored --nocapture

- name: Run cloud Postgres integration suite
if: >-
${{ !cancelled() &&
(github.event_name != 'workflow_dispatch' ||
inputs.scope == 'all' || inputs.scope == 'postgres') }}
needs.plan.outputs.postgres == 'true' }}
run: cargo test -p clickhouse-cloud-api --test integration_postgres_test -- --ignored --nocapture

- name: Run cloud Org integration suite
if: >-
${{ !cancelled() &&
(github.event_name != 'workflow_dispatch' ||
inputs.scope == 'all' || inputs.scope == 'organization') }}
needs.plan.outputs.organization == 'true' }}
run: cargo test -p clickhouse-cloud-api --test integration_org_test -- --ignored --nocapture

- name: Run ClickPipe Postgres CDC integration test
if: >-
${{ !cancelled() &&
(github.event_name != 'workflow_dispatch' ||
inputs.scope == 'all' || inputs.scope == 'clickpipes') }}
needs.plan.outputs.clickpipes == 'true' }}
run: cargo test -p clickhouse-cloud-api --test clickpipe_postgres_cdc_test -- --ignored --nocapture

# ClickPipe create smoke tests run against a long-lived ClickHouse Cloud
Expand All @@ -134,7 +255,6 @@ jobs:
- name: Run ClickPipe create smoke suite
if: >-
${{ !cancelled() &&
(github.event_name != 'workflow_dispatch' ||
inputs.scope == 'all' || inputs.scope == 'clickpipes') &&
needs.plan.outputs.clickpipes == 'true' &&
vars.CLICKHOUSE_CLOUD_TEST_CLICKPIPE_SERVICE_ID != '' }}
run: cargo test -p clickhouse-cloud-api --test clickpipe_smoke_test -- --ignored --nocapture
1 change: 1 addition & 0 deletions .github/workflows/test-cloud-api.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ on:
- "crates/clickhouse-cloud-api/**"
- "crates/clickhouse-openapi-analyzer/**"
- "scripts/check-openapi-drift.py"
- "scripts/classify-cloud-integration.py"
- "scripts/tests/**"
- "Cargo.toml"
- "Cargo.lock"
Expand Down
2 changes: 1 addition & 1 deletion crates/clickhouse-cloud-api/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,7 @@ cargo test --test clickpipe_smoke_test -- --ignored --nocapture # creat

All require `CLICKHOUSE_CLOUD_API_KEY`, `CLICKHOUSE_CLOUD_API_SECRET`, `CLICKHOUSE_CLOUD_TEST_ORG_ID`, `CLICKHOUSE_CLOUD_TEST_PROVIDER`, and `CLICKHOUSE_CLOUD_TEST_REGION` in the environment, and are wired into the scheduled `Cloud Integration` GitHub Actions workflow. The ClickPipes E2E suites additionally need AWS credentials and an `eu-west-1` region quota; `clickpipe_smoke_test` reads a pre-provisioned service ID from `CLICKHOUSE_CLOUD_TEST_CLICKPIPE_SERVICE_ID`.

Manual `Cloud Integration` dispatches accept `scope=all`, `service`, `postgres`, `organization`, or `clickpipes`. The focused scopes run only their corresponding suite; `clickpipes` runs Postgres CDC plus the fixture-gated smoke test, while `all` runs all four mandatory suites plus that optional smoke test. For full-stack validation, manually run `scope=all` against the top stack branch.
Applying `run-cloud-integration` to an eligible PR selects suites from that PR's immediate base-to-head diff; known changes without a live suite finish without entering the environment-bearing job, while unknown API source or test paths select all suites. Manual `Cloud Integration` dispatches accept `scope=all`, `service`, `postgres`, `organization`, or `clickpipes`. The focused scopes run only their corresponding suite; `clickpipes` runs Postgres CDC plus the fixture-gated smoke test, while `all` runs all four mandatory suites plus that optional smoke test. Because immediate stacked-PR diffs exclude inherited changes, manually run `scope=all` against the top stack branch for full-stack validation.

`spec_coverage_test` sends the checked-in sources and snapshot through the
private `clickhouse-openapi-analyzer` crate. The analyzer recursively traverses
Expand Down
Loading