[SPARK-56811][UI] Restore sub-execution grouping on the SQL tab listing

[yaooqinn]

What changes were proposed in this pull request?

This PR restores sub-execution grouping on the SQL tab listing that was silently dropped when the listing was switched to server-side pagination in SPARK-56140.

• Backend (SqlResource.sqlTable): accept a new groupSubExecution query parameter (default = cluster config spark.ui.groupSQLSubExecutionEnabled). When enabled, paginate over root executions only and embed each root's children as subExecutions: [...] on the row. Sub-executions whose root is missing from the filtered set surface as roots so they are never hidden. Fix recordsTotal / recordsFiltered to count root rows in grouped mode so the DataTables "Showing X to Y of Z entries" matches the visible rows.
• Frontend (allexecutionspage.js): read the existing group-sub-exec-config data attribute, forward groupSubExecution=true|false to the server, and append a trailing Sub Executions column showing a +N sub toggle on roots that have children. Toggling expands the row using DataTables row().child() with a nested table — matches the SPARK-41752 / Spark 4.1 layout.
• CSS (webui-dataTables.css): minimal styling for the toggle link and the nested child table (indent + tertiary background).

Why are the changes needed?

SPARK-41752 introduced sub-execution grouping in Spark 4.1, and SPARK-55875 carried it over when the listing moved to client-side DataTables. SPARK-56140 then switched the listing to server-side pagination but didn't carry over the grouping logic — every sub-execution now shows up as its own flat row, regressing the UX for queries such as CACHE TABLE and nested CTAS.

Does this PR introduce any user-facing change?

Yes — the SQL tab listing again folds sub-executions under their root, with a +N sub toggle to expand. Default behaviour is controlled by the existing spark.ui.groupSQLSubExecutionEnabled config (default true).

The screenshots below were taken from the same workload (SELECT, CACHE TABLE, broadcast join, DROP TABLE, CREATE TABLE … AS) — CACHE TABLE and the inner CREATE TABLE each spawn a sub-execution.

Before — flat listing (today, post-SPARK-56140): sub-executions appear as flat sibling rows; e.g. id 2/3/4 are all CACHE TABLE people_eng_cached and id 7/8/9 are all nested CTAS.

After — grouped, collapsed: only roots are listed, with a trailing +1 sub toggle on rows that own a sub-execution.

After — grouped, expanded: clicking +1 sub expands a nested table inline.

How was this patch tested?

• New unit test in SqlResourceWithActualMetricsSuite covering both grouped and flat modes against a session that runs CACHE TABLE (which produces a root + sub-execution pair).
• Existing SqlResourceWithActualMetricsSuite and AllExecutionsPageWithInMemoryStoreSuite continue to pass.
• Manual verification in a local Spark UI (screenshots above): ran the same workload twice — once with spark.ui.groupSQLSubExecutionEnabled=false (flat) and once with the default true (grouped) — and confirmed both modes render correctly.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: GitHub Copilot CLI 1.0.44-2 with Claude Opus 4.7 (Extra high reasoning)

[LuciferYang]

The actual config (from core/src/main/scala/org/apache/spark/internal/config/UI.scala) is spark.ui.groupSQLSubExecutionEnabled. Fix the docstring so users don't waste time searching for a non-existent key.

[LuciferYang]

This asserts that every grouped row satisfies id == rootExecutionId, but the backend (line 133-134 of SqlResource.scala) deliberately surfaces orphan sub-executions (whose parent is missing from the filtered set) as roots. Those orphans have id != rootExecutionId. The test passes today only because CACHE TABLE happens not to produce orphans and the test runs with no filter (so every parent is present); the assertion encodes a property that holds only in the unfiltered case but is documented as a universal "grouped row is a root" invariant.

[LuciferYang]

The orphan branch on line 134 of SqlResource.scala is one of the most subtle pieces of new logic — sub appears under a root unless the root is filtered out, in which case the sub surfaces. The new test doesn't exercise this path. Worth a second test case using a search[value] that matches the sub but not the root (or a status filter that produces the same shape).

[LuciferYang]

// Always load all execs once. We need the full set to (a) identify orphan
// sub-executions whose root is filtered out and (b) count root rows for
// `recordsTotal`. `sqlStore.executionsList()` is already a full
// materialization, so there is no separate "KVStore-pagination" path being
// disabled here.

[LuciferYang]

CACHE TABLE spark_56811_cached leaves a cached table behind that persists into subsequent tests in this suite. Wrap with withTempView and an explicit UNCACHE TABLE, or add a try { ... } finally { spark.sql("UNCACHE TABLE IF EXISTS spark_56811_cached") } block.

[LuciferYang]

When groupSubExec && needsFilter, this does a fresh O(N) Set construction + O(N) count on allExecs. The work is the same flavor as the filtered-set partition above (line 132-135) but on a different domain (all execs vs. filtered execs), so it can't be folded directly. Consider precomputing both partitions (filtered + unfiltered) up front behind a single helper, or memoizing allRootIds so the unfiltered count and the filtered set both consult the same lazily-built data.

[LuciferYang]

data (queryId) is interpolated into both a title attribute and visible text without escapeHtml. The other rendered columns (description, errorMessage) use escapeHtml. Today queryId is a UUID, so this is safe in practice, but the inconsistency is the kind of thing that bites later when a schema changes. Consider wrapping in escapeHtml(data) to match the rest.

[LuciferYang]

The disclosure is a <a href="#"> with a click handler — functionally a button. Screen readers see it as a generic link with no expanded/collapsed state. Worth adding:

• role="button" (so AT announces it as a control, not navigation),
• aria-expanded="false" on render, toggled to "true" / "false" in the click handler alongside the shown class,
• preferably aria-controls= referencing the generated child row id.

Existing components in the Spark UI are already inconsistent with such accessibility standards, and we can unify and standardize this in subsequent iterations.

[LuciferYang]

A single CACHE TABLE on a large nested DAG, or a deeply-nested CTAS, can produce dozens of sub-executions under one root. The current design embeds every sub of every visible root into the same JSON response, so a page of 20 roots could carry hundreds of sub rows even if the user never expands any disclosure. For most workloads this is fine, but worth noting in the PR description — and worth considering a follow-up that lazy-loads sub rows on first expand (a separate ?rootId=N endpoint) if a user reports payload size issues.

[sarutak]

Parent rows use descriptionHtml() which wraps in a link to the execution detail page, but child rows use plain escapeHtml(). Is this intentional, or should child rows also use descriptionHtml({ id: child.id, description: child.description })?