fix(spark): correct the CTAS claim on VortexSessionCatalog.createTable - #9332
Merged
robert3005 merged 1 commit intoAug 11, 2026
Merged
Conversation
The javadoc said Spark writes a CTAS query result into the table this overload returns. It does not: DelegatingCatalogExtension sends the Column[] overload straight to the delegate, and that is the one both supported Spark versions call, so this override never runs. CTAS gets its Vortex table from the loadTable override instead. Had it run, it would have thrown: V2SessionCatalog.createTable returns null on its normal path and asVortexTableIfVortex dereferenced it immediately. Tolerate null there and describe what actually happens. Signed-off-by: jackylee <qcsd2011@gmail.com>
robert3005
approved these changes
Aug 10, 2026
Contributor
|
does this mean we can remove the override? I might have mixed up the version/feature matrix |
robert3005
enabled auto-merge (squash)
August 11, 2026 12:52
Merging this PR will improve performance by 13.04%
Performance Changes
Tip Curious why this is faster? Comment Comparing Footnotes
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
The javadoc on
createTablesaid Spark writes aCREATE TABLE ... AS SELECTresult into the table this overload returns. It does not.DelegatingCatalogExtensionsends theColumn[]overload straight to the delegate, and that is the overload both supported Spark versions call — 3.5'sCreateTableAsSelectExecdirectly, 4.1's via theTableInfodefault. This override never runs; CTAS gets its Vortex table from theloadTableoverride instead.Had it run it would have thrown:
V2SessionCatalog.createTablereturnsnullon its normal path, andasVortexTableIfVortexdereferenced the argument immediately.What changes are included in this PR?
Tolerate
nullinasVortexTableIfVortex, and replace the javadoc claim with what actually happens../gradlew :vortex-spark_2.13:test --tests '…VortexSessionCatalogTest' --tests '…VortexSqlTest'— 10 pass:vortex-spark_2.12:testspotlessCheck(both variants) andjavadoc— cleanVortexSessionCatalogTestalready covers the managed-table lifecycle on both versions, which is the path CTAS actually takes.What APIs are changed? Are there any user-facing changes?
None.
AI assistance
Prepared with agentic AI assistance. I verified the dispatch against the compiled
DelegatingCatalogExtension,CreateTableAsSelectExecandV2SessionCatalogin both Spark versions rather than reasoning from the source I had.