ci: fold the Delta build gate and PyArrow UDF suite into the merge queue tiers

.github/workflows/README.md

@@ -17,9 +17,10 @@ ruleset in `.asf.yaml`. That splits CI into two tiers:
 - **PR tier** (`pr`): fast feedback while a change is being iterated on.
   The Linux build, Spark 4.1 (catalyst and `sql_core` only) and Iceberg 1.11.
 - **Queue tier** (`queue`): the authoritative gate. Everything the PR tier
-  runs, plus the macOS build, the benchmark compile check, the Spark 4.1
-  `sql_hive` shards, Spark 3.5/4.0 and Iceberg 1.8/1.9/1.10, evaluated
-  against the merge result rather than against the PR head.
+  runs, plus the macOS build, the benchmark compile check, the Delta contrib
+  build gate, the PyArrow UDF suite, the Spark 4.1 `sql_hive` shards, Spark
+  3.5/4.0 and Iceberg 1.8/1.9/1.10, evaluated against the merge result
+  rather than against the PR head.
 
 Every queue-only job has a `run-*` label that opts a pull request into it
 early, listed in the diagram below.
@@ -82,7 +83,9 @@ to `pr_build_linux.yml` without either the guard or an entry in
   ---------------                     -----------------         ---------------------------
   pr_build_linux (+ push, cache only) docs                      pr_build_macos      run-macos-tests
   spark_4_1 (catalyst + sql_core)                               pr_benchmark_check  run-benchmark-check
-  iceberg_1_11                                                  spark_4_1 sql_hive  run-spark-4.1-hive-tests
+  iceberg_1_11                                                  delta_build_gate    run-delta-build-gate
+                                                                pyarrow_udf_test    run-pyarrow-udf-tests
+                                                                spark_4_1 sql_hive  run-spark-4.1-hive-tests
                                                                 spark_3_5           run-spark-3.5-tests
                                                                 spark_4_0           run-spark-4.0-tests
   label or dispatch only                                        iceberg_1_8         run-iceberg-tests
@@ -101,8 +104,8 @@ to `pr_build_linux.yml` without either the guard or an entry in
   reusable workflows invoked via `uses:`:
     pr_build_linux.yml         spark_sql_test_reusable.yml
     pr_build_macos.yml         iceberg_spark_test_reusable.yml
-    pr_benchmark_check.yml
-    docs.yaml
+    pr_benchmark_check.yml     delta_build_gate.yml
+    docs.yaml                  pyarrow_udf_test.yml
 ```
 
 ## What runs when
@@ -114,6 +117,8 @@ to `pr_build_linux.yml` without either the guard or an entry in
 | `pr_build_linux`     | PR, merge group or push to main, paths matched; on push only the cache-writing jobs, via `build_linux_full`            | `dev/ci/compute-changes.py`         |
 | `pr_build_macos`     | merge group, **or** PR with `run-macos-tests`                                                                          | `dev/ci/compute-changes.py`         |
 | `pr_benchmark_check` | merge group, **or** PR with `run-benchmark-check`                                                                      | benchmark sources only              |
+| `delta_build_gate`   | merge group, **or** PR with `run-delta-build-gate`                                                                     | main sources, poms, `contrib/delta` |
+| `pyarrow_udf_test`   | merge group, **or** PR with `run-pyarrow-udf-tests`                                                                    | map-in-batch and Python runner code |
 | `docs`               | push to main, paths matched                                                                                            | `.asf.yaml`, `docs/**`, `docs.yaml` |
 | `spark_3_5`          | merge group, **or** PR with `run-spark-3.5-tests`                                                                      | Spark 3.5 sources                   |
 | `spark_4_1`          | PR or merge group, paths matched; the `sql_hive` shards only in the merge group **or** with `run-spark-4.1-hive-tests` | Spark 4.1 sources                   |
@@ -187,6 +192,8 @@ umbrella doesn't watch, or operate independently of the rest of CI:
 | `pr_build_linux.yml`              | `pr_build_linux`                                             |
 | `pr_build_macos.yml`              | `pr_build_macos`                                             |
 | `pr_benchmark_check.yml`          | `pr_benchmark_check`                                         |
+| `delta_build_gate.yml`            | `delta_build_gate`                                           |
+| `pyarrow_udf_test.yml`            | `pyarrow_udf_test`                                           |
 | `docs.yaml`                       | `docs`                                                       |
 | `spark_sql_test_reusable.yml`     | `spark_3_4`, `spark_3_5`, `spark_4_0`, `spark_4_1`           |
 | `iceberg_spark_test_reusable.yml` | `iceberg_1_8`, `iceberg_1_9`, `iceberg_1_10`, `iceberg_1_11` |

.github/workflows/ci.yml

@@ -25,7 +25,8 @@
 #   pull_request  fast feedback. Linux build + Spark 4.1 (catalyst and
 #                 sql_core only) + Iceberg 1.11.
 #   merge_group   the authoritative gate. Everything the PR tier runs, plus
-#                 the macOS build, the benchmark compile check, the Spark 4.1
+#                 the macOS build, the benchmark compile check, the Delta
+#                 contrib build gate, the PyArrow UDF suite, the Spark 4.1
 #                 sql_hive shards, Spark 3.5/4.0 and Iceberg 1.8/1.9/1.10,
 #                 evaluated against the merge result rather than the PR head.
 #
@@ -162,6 +163,8 @@ jobs:
       build_linux_full: ${{ steps.compute.outputs.build_linux_full }}
       build_macos:  ${{ steps.compute.outputs.build_macos }}
       benchmark:    ${{ steps.compute.outputs.benchmark }}
+      delta_gate:   ${{ steps.compute.outputs.delta_gate }}
+      pyarrow_udf:  ${{ steps.compute.outputs.pyarrow_udf }}
       docs:         ${{ steps.compute.outputs.docs }}
       spark_3_4:    ${{ steps.compute.outputs.spark_3_4 }}
       spark_3_5:    ${{ steps.compute.outputs.spark_3_5 }}
@@ -259,6 +262,26 @@ jobs:
     if: needs.changes.outputs.benchmark == 'true'
     uses: ./.github/workflows/pr_benchmark_check.yml
 
+  delta_build_gate:
+    name: Delta Contrib Build Gate
+    needs: changes
+    # Queue-only by default; PRs need the `run-delta-build-gate` label.
+    if: needs.changes.outputs.delta_gate == 'true'
+    # The called workflow needs only a checkout.
+    permissions:
+      contents: read
+    uses: ./.github/workflows/delta_build_gate.yml
+
+  pyarrow_udf_test:
+    name: PyArrow UDF Tests
+    needs: changes
+    # Queue-only by default; PRs need the `run-pyarrow-udf-tests` label.
+    if: needs.changes.outputs.pyarrow_udf == 'true'
+    # The called workflow needs only a checkout.
+    permissions:
+      contents: read
+    uses: ./.github/workflows/pyarrow_udf_test.yml
+
   docs:
     name: Deploy Comet site
     needs: changes
@@ -427,6 +450,8 @@ jobs:
       - pr_build_linux
       - pr_build_macos
       - pr_benchmark_check
+      - delta_build_gate
+      - pyarrow_udf_test
       - spark_3_4
       - spark_3_5
       - spark_4_0

.github/workflows/delta_build_gate.yml

@@ -29,36 +29,10 @@
 
 name: Delta Contrib Build Gate
 
-concurrency:
-  group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }}
-  cancel-in-progress: true
-
+# Reusable: invoked by ci.yml. Triggering, path filters, and concurrency
+# live in the umbrella workflow.
 on:
-  push:
-    branches:
-      - main
-    paths-ignore:
-      - "benchmarks/**"
-      - "doc/**"
-      - "docs/**"
-      - "**.md"
-      - "dev/changelog/*.md"
-      - "native/core/benches/**"
-      - "native/spark-expr/benches/**"
-      - "spark/src/main/scala/org/apache/comet/GenerateDocs.scala"
-      - "spark-integration/**"
-  pull_request:
-    paths-ignore:
-      - "benchmarks/**"
-      - "doc/**"
-      - "docs/**"
-      - "**.md"
-      - "dev/changelog/*.md"
-      - "native/core/benches/**"
-      - "native/spark-expr/benches/**"
-      - "spark/src/main/scala/org/apache/comet/GenerateDocs.scala"
-      - "spark-integration/**"
-  workflow_dispatch:
+  workflow_call:
 
 permissions:
   contents: read

.github/workflows/pyarrow_udf_test.yml

@@ -17,46 +17,11 @@
 
 name: PyArrow UDF Tests
 
-concurrency:
-  group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }}
-  cancel-in-progress: true
-
+# Reusable: invoked by ci.yml. Triggering, path filters, and concurrency
+# live in the umbrella workflow. The list of files this suite covers is the
+# `pyarrow_udf` entry in FILTERS in dev/ci/compute-changes.py.
 on:
-  push:
-    branches:
-      - main
-    paths: &feature-paths
-      - "pom.xml"
-      - "common/pom.xml"
-      - "native/shuffle/src/spark_unsafe/row.rs"
-      - "spark/pom.xml"
-      - "spark/src/main/java/org/apache/comet/vector/**"
-      - "spark/src/main/java/org/apache/spark/sql/comet/execution/shuffle/SpillWriter.java"
-      - "spark/src/main/scala/org/apache/comet/CometConf.scala"
-      - "spark/src/main/scala/org/apache/comet/rules/EliminateRedundantTransitions.scala"
-      - "spark/src/main/scala/org/apache/comet/vector/**"
-      - "spark/src/main/scala/org/apache/spark/sql/comet/CometMapInBatchExec.scala"
-      - "spark/src/main/scala/org/apache/spark/sql/comet/shims/MapInBatchInfo.scala"
-      - "spark/src/main/spark-3.4/org/apache/spark/sql/comet/shims/ShimCometMapInBatch.scala"
-      - "spark/src/main/spark-3.5/org/apache/spark/sql/comet/shims/ShimCometMapInBatch.scala"
-      - "spark/src/main/spark-4.0/org/apache/spark/sql/comet/shims/ShimCometMapInBatch.scala"
-      - "spark/src/main/spark-4.0/org/apache/spark/sql/execution/python/CometArrowPythonRunner.scala"
-      - "spark/src/main/spark-4.1/org/apache/spark/sql/comet/shims/ShimCometMapInBatch.scala"
-      - "spark/src/main/spark-4.1/org/apache/spark/sql/execution/python/CometArrowPythonRunner.scala"
-      - "spark/src/main/spark-4.2/org/apache/spark/sql/comet/shims/ShimCometMapInBatch.scala"
-      - "spark/src/main/spark-4.2/org/apache/spark/sql/execution/python/CometArrowPythonRunner.scala"
-      - "spark/src/main/spark-4.x/org/apache/spark/sql/comet/shims/Spark4xMapInBatchSupport.scala"
-      - "spark/src/main/spark-4.x/org/apache/spark/sql/execution/python/CometArrowPythonRunnerBase.scala"
-      - "spark/src/test/resources/pyspark/conftest.py"
-      - "spark/src/test/resources/pyspark/test_pyarrow_udf.py"
-      - "spark/src/test/resources/pyspark/test_pyarrow_udf_dictionary_shuffle.py"
-      - "spark/src/test/spark-3.5/org/apache/spark/sql/comet/CometMapInBatchSuite.scala"
-      - "spark/src/test/spark-4.x/org/apache/spark/sql/comet/CometMapInBatchSuite.scala"
-      - "spark/src/test/spark-4.x/org/apache/spark/sql/execution/python/CometArrowPythonRunnerSuite.scala"
-      - ".github/workflows/pyarrow_udf_test.yml"
-  pull_request:
-    paths: *feature-paths
-  workflow_dispatch:
+  workflow_call:
 
 permissions:
   contents: read

dev/ci/check-ci-config.py

@@ -100,12 +100,16 @@
     "iceberg_1_11",
 }
 
+# The two contrib/UDF gates also run ./mvnw, but consume no shared artifact.
+MVN_JOBS = BUILD_JOBS | {"delta_gate", "pyarrow_udf"}
+
 ROUTING_CASES = [
     # The Maven wrapper and its config feed every job that runs ./mvnw: the
-    # Linux/macOS builds, setup-spark-builder, and the Iceberg `mvnw install`.
-    ([".mvn/maven.config"], BUILD_JOBS),
-    ([".mvn/wrapper/maven-wrapper.properties"], BUILD_JOBS),
-    (["mvnw"], BUILD_JOBS),
+    # Linux/macOS builds, setup-spark-builder, the Iceberg `mvnw install`, the
+    # Delta gate's effective-pom check and the PyArrow suite's `mvnw install`.
+    ([".mvn/maven.config"], MVN_JOBS),
+    ([".mvn/wrapper/maven-wrapper.properties"], MVN_JOBS),
+    (["mvnw"], MVN_JOBS),
     # The artifact wrappers are used by every producer and consumer of a
     # shared artifact. Without these, an edit confined to one of them routes
     # to nothing at all and merges having been exercised by no consumer.
@@ -116,6 +120,16 @@
     # Spot checks that the additions above did not widen unrelated routes.
     (["docs/source/user-guide/overview.md"], {"docs"}),
     (["native/core/benches/parquet_read.rs"], {"benchmark"}),
+    # The Delta gate script is read by nothing else; the contrib crate feeds
+    # only the gate. The PyArrow pytest lives under spark/, so the Linux and
+    # macOS builds see it too, but no Spark SQL or Iceberg suite does, and
+    # neither does the Delta gate, which only inspects build output.
+    (["dev/verify-contrib-delta-gate.sh"], {"delta_gate"}),
+    (["contrib/delta/native/src/lib.rs"], {"delta_gate"}),
+    (
+        ["spark/src/test/resources/pyspark/test_pyarrow_udf.py"],
+        {"build_linux", "build_linux_full", "build_macos", "pyarrow_udf"},
+    ),
 ]
 
 # Event policy. Each case is (event, expected set of jobs allowed to run),
@@ -130,7 +144,7 @@
 # than quietly accept it coming back.
 SPARK_DEPRECATED = {"spark_3_4"}
 ICEBERG_OPT_IN = {"iceberg_1_8", "iceberg_1_9", "iceberg_1_10"}
-BUILD_OPT_IN = {"build_macos", "benchmark"}
+BUILD_OPT_IN = {"build_macos", "benchmark", "delta_gate", "pyarrow_udf"}
 QUEUE_TIER = PR_TIER | SPARK_OPT_IN | ICEBERG_OPT_IN | BUILD_OPT_IN
 ALL_JOBS = QUEUE_TIER | SPARK_DEPRECATED | {"docs"}
 
@@ -166,6 +180,26 @@
         {"name": "pull_request", "action": "synchronize", "labels": ["run-benchmark-check"]},
         PR_TIER | {"benchmark"},
     ),
+    # The Delta build gate and the PyArrow UDF suite were standalone workflows
+    # that ran on every pull request and again on push to main. Folded in as
+    # queue-only jobs, each with its own label, they follow the same rules.
+    (
+        {"name": "pull_request", "action": "synchronize", "labels": ["run-delta-build-gate"]},
+        PR_TIER | {"delta_gate"},
+    ),
+    (
+        {"name": "pull_request", "action": "synchronize", "labels": ["run-pyarrow-udf-tests"]},
+        PR_TIER | {"pyarrow_udf"},
+    ),
+    (
+        {
+            "name": "pull_request",
+            "action": "labeled",
+            "label": "run-pyarrow-udf-tests",
+            "labels": ["run-pyarrow-udf-tests"],
+        },
+        {"pyarrow_udf"},
+    ),
     (
         {
             "name": "pull_request",

dev/ci/compute-changes.py

@@ -100,6 +100,67 @@
         "native/spark-expr/benches/**",
         "spark/src/test/scala/org/apache/spark/sql/benchmark/**",
     ],
+    # dev/verify-contrib-delta-gate.sh proves the default cargo, Maven and
+    # libcomet builds carry no Delta surface and that the gated build does.
+    # It reads the cargo tree, the effective pom, the compiled classes and the
+    # dylib symbol table: main sources and build inputs, never tests.
+    "delta_gate": [
+        "native/**",
+        "common/src/main/**",
+        "spark/src/main/**",
+        "contrib/delta/**",
+        "pom.xml",
+        "**/pom.xml",
+        ".mvn/**",
+        "mvnw",
+        "Makefile",
+        "rust-toolchain.toml",
+        "dev/verify-contrib-delta-gate.sh",
+        ".github/workflows/ci.yml",
+        ".github/workflows/delta_build_gate.yml",
+        ".github/actions/setup-builder/**",
+        "!**.md",
+        "!native/core/benches/**",
+        "!native/spark-expr/benches/**",
+        "!spark/src/main/scala/org/apache/comet/GenerateDocs.scala",
+    ],
+    # A real Python worker against each Spark 4.x Arrow runner. The list is
+    # deliberately narrow: the suite builds Comet three times, once per Spark
+    # version, and only the map-in-batch wiring can change its verdict.
+    "pyarrow_udf": [
+        "pom.xml",
+        "common/pom.xml",
+        "native/shuffle/src/spark_unsafe/row.rs",
+        "spark/pom.xml",
+        "spark/src/main/java/org/apache/comet/vector/**",
+        "spark/src/main/java/org/apache/spark/sql/comet/execution/shuffle/SpillWriter.java",
+        "spark/src/main/scala/org/apache/comet/CometConf.scala",
+        "spark/src/main/scala/org/apache/comet/rules/EliminateRedundantTransitions.scala",
+        "spark/src/main/scala/org/apache/comet/vector/**",
+        "spark/src/main/scala/org/apache/spark/sql/comet/CometMapInBatchExec.scala",
+        "spark/src/main/scala/org/apache/spark/sql/comet/shims/MapInBatchInfo.scala",
+        "spark/src/main/spark-3.4/org/apache/spark/sql/comet/shims/ShimCometMapInBatch.scala",
+        "spark/src/main/spark-3.5/org/apache/spark/sql/comet/shims/ShimCometMapInBatch.scala",
+        "spark/src/main/spark-4.0/org/apache/spark/sql/comet/shims/ShimCometMapInBatch.scala",
+        "spark/src/main/spark-4.0/org/apache/spark/sql/execution/python/CometArrowPythonRunner.scala",
+        "spark/src/main/spark-4.1/org/apache/spark/sql/comet/shims/ShimCometMapInBatch.scala",
+        "spark/src/main/spark-4.1/org/apache/spark/sql/execution/python/CometArrowPythonRunner.scala",
+        "spark/src/main/spark-4.2/org/apache/spark/sql/comet/shims/ShimCometMapInBatch.scala",
+        "spark/src/main/spark-4.2/org/apache/spark/sql/execution/python/CometArrowPythonRunner.scala",
+        "spark/src/main/spark-4.x/org/apache/spark/sql/comet/shims/Spark4xMapInBatchSupport.scala",
+        "spark/src/main/spark-4.x/org/apache/spark/sql/execution/python/CometArrowPythonRunnerBase.scala",
+        "spark/src/test/resources/pyspark/conftest.py",
+        "spark/src/test/resources/pyspark/test_pyarrow_udf.py",
+        "spark/src/test/resources/pyspark/test_pyarrow_udf_dictionary_shuffle.py",
+        "spark/src/test/spark-3.5/org/apache/spark/sql/comet/CometMapInBatchSuite.scala",
+        "spark/src/test/spark-4.x/org/apache/spark/sql/comet/CometMapInBatchSuite.scala",
+        "spark/src/test/spark-4.x/org/apache/spark/sql/execution/python/CometArrowPythonRunnerSuite.scala",
+        ".mvn/**",
+        "mvnw",
+        ".github/workflows/ci.yml",
+        ".github/workflows/pyarrow_udf_test.yml",
+        ".github/actions/setup-builder/**",
+    ],
     "docs": [
         ".asf.yaml",
         ".github/workflows/docs.yaml",
@@ -371,6 +432,13 @@
     # Benchmark sources are compiled and linted, never run, so a break there
     # cannot affect a PR's correctness verdict; the queue catches it.
     "benchmark": ["queue", "label:run-benchmark-check"],
+    # The Delta build gate only proves a build-system property, and the
+    # PyArrow suite builds Comet once per Spark 4.x version to drive a real
+    # Python worker. Neither changes often enough to earn a PR-tier slot; the
+    # queue catches a regression before it lands, and the label is the escape
+    # hatch for a change to the surface they cover.
+    "delta_gate": ["queue", "label:run-delta-build-gate"],
+    "pyarrow_udf": ["queue", "label:run-pyarrow-udf-tests"],
     # docs deploys to asf-site, so it must not run from a pull request or from
     # the queue's throwaway branch.
     "docs": ["push"],

docs/source/contributor-guide/ci.md

@@ -40,6 +40,8 @@ Which jobs run also depends on the event:
 | Iceberg Spark SQL tests, Iceberg 1.11             | yes          | yes         |
 | macOS build and Comet test suites                 | with label   | yes         |
 | Benchmark compile and lint check                  | with label   | yes         |
+| Delta contrib build gate                          | with label   | yes         |
+| PyArrow UDF tests, Spark 4.0 / 4.1 / 4.2          | with label   | yes         |
 | Spark SQL tests, Spark 3.5 / 4.0                  | with label   | yes         |
 | Iceberg Spark SQL tests, Iceberg 1.8 / 1.9 / 1.10 | with label   | yes         |
 | Spark SQL tests, Spark 3.4                        | with label   | no          |
@@ -86,6 +88,8 @@ Each suite outside the PR tier has a label that runs it on a pull request:
 | -------------------------- | ---------------------------------------------------- |
 | `run-macos-tests`          | macOS build and Comet test suites                    |
 | `run-benchmark-check`      | Benchmark compile and lint check                     |
+| `run-delta-build-gate`     | Delta contrib build gate                             |
+| `run-pyarrow-udf-tests`    | PyArrow UDF tests against Spark 4.0/4.1/4.2          |
 | `run-spark-4.1-hive-tests` | Spark SQL hive tests against Spark 4.1               |
 | `run-spark-3.4-tests`      | Spark SQL tests against Spark 3.4                    |
 | `run-spark-3.5-tests`      | Spark SQL tests against Spark 3.5                    |