Skip to content

fix(spark): correct the CTAS claim on VortexSessionCatalog.createTable - #9332

Merged
robert3005 merged 1 commit into
vortex-data:developfrom
jackylee-ch:spark-session-catalog-ctas-doc
Aug 11, 2026
Merged

fix(spark): correct the CTAS claim on VortexSessionCatalog.createTable#9332
robert3005 merged 1 commit into
vortex-data:developfrom
jackylee-ch:spark-session-catalog-ctas-doc

Conversation

@jackylee-ch

Copy link
Copy Markdown
Contributor

Rationale for this change

The javadoc on createTable said Spark writes a CREATE TABLE ... AS SELECT result into the table this overload returns. It does not. DelegatingCatalogExtension sends the Column[] overload straight to the delegate, and that is the overload both supported Spark versions call — 3.5's CreateTableAsSelectExec directly, 4.1's via the TableInfo default. This override never runs; CTAS gets its Vortex table from the loadTable override instead.

Had it run it would have thrown: V2SessionCatalog.createTable returns null on its normal path, and asVortexTableIfVortex dereferenced the argument immediately.

What changes are included in this PR?

Tolerate null in asVortexTableIfVortex, and replace the javadoc claim with what actually happens.

  • ./gradlew :vortex-spark_2.13:test --tests '…VortexSessionCatalogTest' --tests '…VortexSqlTest' — 10 pass
  • same on :vortex-spark_2.12:test
  • spotlessCheck (both variants) and javadoc — clean

VortexSessionCatalogTest already covers the managed-table lifecycle on both versions, which is the path CTAS actually takes.

What APIs are changed? Are there any user-facing changes?

None.

AI assistance

Prepared with agentic AI assistance. I verified the dispatch against the compiled DelegatingCatalogExtension, CreateTableAsSelectExec and V2SessionCatalog in both Spark versions rather than reasoning from the source I had.

The javadoc said Spark writes a CTAS query result into the table this
overload returns. It does not: DelegatingCatalogExtension sends the
Column[] overload straight to the delegate, and that is the one both
supported Spark versions call, so this override never runs. CTAS gets its
Vortex table from the loadTable override instead.

Had it run, it would have thrown: V2SessionCatalog.createTable returns
null on its normal path and asVortexTableIfVortex dereferenced it
immediately. Tolerate null there and describe what actually happens.

Signed-off-by: jackylee <qcsd2011@gmail.com>
@robert3005

Copy link
Copy Markdown
Contributor

does this mean we can remove the override? I might have mixed up the version/feature matrix

@robert3005 robert3005 added changelog/fix A bug fix changelog/chore A trivial change and removed changelog/fix A bug fix labels Aug 11, 2026
@robert3005
robert3005 enabled auto-merge (squash) August 11, 2026 12:52
@codspeed-hq

codspeed-hq Bot commented Aug 11, 2026

Copy link
Copy Markdown

Merging this PR will improve performance by 13.04%

⚡ 1 improved benchmark
✅ 1943 untouched benchmarks
⏩ 85 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation decompress[u64, (1000, 16)] 73.3 µs 64.9 µs +13.04%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.


Comparing jackylee-ch:spark-session-catalog-ctas-doc (125801a) with develop (4b37237)

Open in CodSpeed

Footnotes

  1. 85 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@robert3005
robert3005 merged commit acc85b7 into vortex-data:develop Aug 11, 2026
112 of 113 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/chore A trivial change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants