Spark: Support CREATE TABLE LIKE with Spark by MaxNevermind · Pull Request #14269 · apache/iceberg

MaxNevermind · 2025-10-07T04:07:38Z

To be done:

Add similar changes to Spark v3.x modules, I saw a discussion about deprecation of Spark 3.4 so not sure if it is needed for all 3.x versions

…12936-spark-create-table-like

MaxNevermind · 2025-10-07T04:10:01Z

Hey @anuragmantri
Can you look up this draft of implementation for some red flags?
I don't contribute to Iceberg too often so not entirely certain in my approach.

singhpk234

Thank you for the change @MaxNevermind !
Its useful to have this and there had been attempts to introducing CREATE TABLE LIKE in spark for DSV2 for a while now

At this point it seems reasonable to have custom analyzer rules and nodes to support this feature like done for views.

Let us know when ready for review, happy to help !

anuragmantri

Thanks for adding this @MaxNevermind, I agree with @singhpk234, we probably do not need to wait for this in Spark.

Overall, it looks good, feel free to move it out of draft after implementing the table resolution logic. Also add more test coverage. I will take another look when it's ready.

Welcome to Iceberg community!

anuragmantri · 2025-10-07T22:04:37Z

...0/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTables.scala

+case class ResolveTables(spark: SparkSession) extends Rule[LogicalPlan] {
+
+  override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperatorsUp {
+    case x @ CreateIcebergTableLike(_, _, _, _) => x


This should resolve the catalogs and tables. See https://github.com/apache/iceberg/blob/main/spark/v4.0/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveViews.scala

Actually I just realized that entire ResolveTables class might not be needed it as tables are resolved in ExtendedDataSourceV2Strategy.

anuragmantri · 2025-10-07T22:06:16Z

spark/v4.0/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestTables.java

+  }
+
+  @TestTemplate
+  public void testCreateTableLike() {


Thanks for adding the test, but we may beed to cover a more

IF NOT EXISTS behavior (tested in test but not validated)

TBLPROPERTIES override functionality (not tested at all) - Partition spec preservation (not validated) - Sort order preservation (not validated) - Cross-catalog table copying - Error cases (source table doesn't exist, non-Iceberg source, etc.) - Property merging behavior

Anything else I may have missed.

@anuragmantri @singhpk234

The current state of tests coverage and some questions:

✅ Partition spec preservation - 2 tests for partitioned an not partitioned tables

✅ IF NOT EXISTS behavior

✅ TBLPROPERTIES - a single test for merge and override

❓ Sort order preservation - Do we want to support Sort order functionality in CREATE TABLE LIKE? I see that Spark's DDL doesn't have it. In Iceberg DDL we also add ordering separately by using ALTER TABLE.

❓ Cross-catalog table copying - Do we want this functionality? I might be wrong but I don't see test for that in other DDL statements, does Iceberg even support cross-catalog DDL?

❓ Error cases (source table doesn't exist, non-Iceberg source, etc.) - I'm not sure what I'm supposed to test for that. Do I have to test that exception will be thrown?

…reate-table-like

MaxNevermind · 2025-10-28T04:29:36Z

@anuragmantri @singhpk234
Can you please look at it again?

The current state of tests coverage and some questions:

✅ Partition spec preservation - 2 tests for partitioned an not partitioned tables
✅ IF NOT EXISTS behavior
✅ TBLPROPERTIES - a single test for merge and override
❓ Sort order preservation - Do we want to support Sort order functionality in CREATE TABLE LIKE? I see that Spark's DDL doesn't have it. In Iceberg DDL we also add ordering separately by using ALTER TABLE.
❓ Cross-catalog table copying - Do we want this functionality? I might be wrong but I don't see test for that in other DDL statements, does Iceberg even support cross-catalog DDL?
❓ Error cases (source table doesn't exist, non-Iceberg source, etc.) - I'm not sure what I'm supposed to test for that. Do I have to test that exception will be thrown?

github-actions · 2025-11-28T00:17:58Z

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

MaxNevermind · 2025-11-30T19:54:00Z

@anuragmantri @singhpk234
Can you please look at it again?

github-actions · 2026-01-01T00:22:10Z

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

github-actions · 2026-01-09T00:19:24Z

This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.

…12936-spark-create-table-like # Conflicts: # spark/v4.0/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/parser/extensions/IcebergSparkSqlExtensionsParser.scala

Copilot

Pull request overview

Adds Spark 4.0 Iceberg SQL extensions support for CREATE TABLE ... LIKE ..., including parsing, logical planning, execution, tests, and documentation updates.

Changes:

Extend the Iceberg SQL extensions grammar/parser/AST to recognize CREATE TABLE (IF NOT EXISTS) ... LIKE ... (TBLPROPERTIES ...).
Add a logical plan node and Spark planner strategy + exec node to create the new table by copying schema/partitioning and merging properties.
Add end-to-end extension tests and update Spark DDL documentation.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
spark/v4.0/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestTables.java	Adds coverage for schema/spec copying, IF NOT EXISTS behavior, and property merging.
spark/v4.0/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ExtendedDataSourceV2Strategy.scala	Plans the new logical command into a physical exec node.
spark/v4.0/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CreateV2TableLikeExec.scala	Implements table creation based on a source Iceberg table.
spark/v4.0/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/CreateIcebergTableLike.scala	Introduces a new logical command representing `CREATE TABLE LIKE`.
spark/v4.0/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/parser/extensions/IcebergSqlExtensionsAstBuilder.scala	Builds the new logical command from the parsed AST.
spark/v4.0/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/parser/extensions/IcebergSparkSqlExtensionsParser.scala	Routes `CREATE TABLE ... LIKE ...` statements through the extensions parser.
spark/v4.0/spark-extensions/src/main/antlr/org.apache.spark.sql.catalyst.parser.extensions/IcebergSqlExtensions.g4	Adds grammar for `CREATE TABLE LIKE` + `TBLPROPERTIES`.
docs/docs/spark-ddl.md	Documents `CREATE TABLE ... LIKE ...` support under Iceberg SQL extensions.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

spark/v4.0/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestTables.java

+    assertThat(table.schema().asStruct())
+        .as("Should have expected nullable schema")
+        .isEqualTo(expectedSchema.asStruct());
+    assertThat(table.spec().fields()).as("Should not be an partitioned").isEmpty();
+  }


spark/v4.0/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestTables.java

+    assertThat(table.schema().asStruct())
+        .as("Should have expected nullable schema")
+        .isEqualTo(expectedSchema.asStruct());
+    assertThat(table.spec().fields()).as("Should not be an partitioned").isEmpty();


...ons/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CreateV2TableLikeExec.scala

+import org.apache.iceberg.SortDirection
+import org.apache.iceberg.SortOrder


.../scala/org/apache/spark/sql/catalyst/parser/extensions/IcebergSparkSqlExtensionsParser.scala

+      isCreateTableLike(normalized) || (
+      normalized.startsWith("alter table") && (
+          normalized.contains("add partition field") ||


...sions/src/main/antlr/org.apache.spark.sql.catalyst.parser.extensions/IcebergSqlExtensions.g4

+    : CREATE TABLE (IF NOT EXISTS)? multipartIdentifier LIKE multipartIdentifier (TBLPROPERTIES '(' tableProperty (',' tableProperty)* ')')? #createTableLike
+    | ALTER TABLE multipartIdentifier ADD PARTITION FIELD transform (AS name=identifier)?   #addPartitionField


docs/docs/spark-ddl.md

+```sql
+CREATE TABLE prod.db.new_table


anuragmantri · 2026-03-16T16:21:33Z

Hi @MaxNevermind, we discussed this PR in the Iceberg - Spark Community Sync but I forgot to update it here in the PR. In general, the community is trying to move away from extensions as they can be fragile and hard to maintain. We should aim to add the syntax in Spark if feasible. As an interim, you could add a procedure to do this same.

MaxNevermind · 2026-03-16T17:15:34Z

Hi @anuragmantri
Is it worth working on that then? I'm completely fine with switching to something else if this code will become obsolete very soon. Let me know.

anuragmantri · 2026-03-16T20:29:27Z

Is it worth working on that then?

Looks like there is a recent PR in Spark that adds supports for CREATE TABLE.. LIKE... We should close this PR and wait for that to land in Spark.

MaxNevermind · 2026-03-16T21:13:15Z

Closing as recent PR apache/spark#54809 in Spark adds supports for CREATE TABLE.. LIKE..

MaxNevermind added 3 commits October 4, 2025 16:45

add Spark support for CREATE TABLE LIKE

960f770

Merge remote-tracking branch 'refs/remotes/origin/main' into feature/…

e4767c0

…12936-spark-create-table-like

format

b7c4bf3

MaxNevermind changed the title ~~Spark: Support CREATE TABLE LIKE with Spark.~~ Spark: Support CREATE TABLE LIKE with Spark Oct 7, 2025

github-actions bot added the spark label Oct 7, 2025

polish CreateV2TableLikeExec

efbeb06

singhpk234 reviewed Oct 7, 2025

View reviewed changes

anuragmantri reviewed Oct 7, 2025

View reviewed changes

MaxNevermind added 8 commits October 7, 2025 22:36

Merge remote-tracking branch 'origin/main' into feature/12936-spark-c…

5fa0714

…reate-table-like

polish ExtendedDataSourceV2Strategy

dfe4955

remove ResolveTables.scala

58cc12d

Merge remote-tracking branch 'origin/main' into feature/12936-spark-c…

16bbc40

…reate-table-like

Merge remote-tracking branch 'origin/main' into feature/12936-spark-c…

28430fb

…reate-table-like

add testPartitionedTable into testCreateTableLike

c9b46c9

Merge remote-tracking branch 'origin/main' into feature/12936-spark-c…

3ff02b1

…reate-table-like

add docs for CREATE TABLE ... LIKE ...

10fdc7d

github-actions bot added the docs label Oct 13, 2025

MaxNevermind added 5 commits October 14, 2025 21:17

Merge remote-tracking branch 'origin/main' into feature/12936-spark-c…

9f4efa3

…reate-table-like

add tests for CREATE TABLE ... LIKE ...

32a6ff5

format

38554da

Merge remote-tracking branch 'origin/main' into feature/12936-spark-c…

be875cc

…reate-table-like

add testTableExists testPropertiesAreMerged into TestTables.java

a1d059a

MaxNevermind marked this pull request as ready for review October 19, 2025 01:12

MaxNevermind requested review from anuragmantri and singhpk234 October 19, 2025 01:24

github-actions bot added the stale label Nov 28, 2025

github-actions bot removed the stale label Dec 1, 2025

github-actions bot added the stale label Jan 1, 2026

github-actions bot closed this Jan 9, 2026

MaxNevermind added 2 commits March 15, 2026 20:37

docs

6170fed

Merge remote-tracking branch 'refs/remotes/origin/main' into feature/…

ba15e8e

…12936-spark-create-table-like # Conflicts: # spark/v4.0/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/parser/extensions/IcebergSparkSqlExtensionsParser.scala

manuzhang reopened this Mar 16, 2026

docs

ab6a6c6

manuzhang requested review from Copilot and removed request for anuragmantri March 16, 2026 04:35

Copilot started reviewing on behalf of manuzhang March 16, 2026 04:36 View session

Copilot AI reviewed Mar 16, 2026

View reviewed changes

MaxNevermind closed this Mar 16, 2026

		import org.apache.iceberg.SortDirection
		import org.apache.iceberg.SortOrder

		: CREATE TABLE (IF NOT EXISTS)? multipartIdentifier LIKE multipartIdentifier (TBLPROPERTIES '(' tableProperty (',' tableProperty)* ')')? #createTableLike
		\| ALTER TABLE multipartIdentifier ADD PARTITION FIELD transform (AS name=identifier)? #addPartitionField

Conversation

MaxNevermind commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MaxNevermind commented Oct 7, 2025

Uh oh!

singhpk234 left a comment

Choose a reason for hiding this comment

Uh oh!

anuragmantri left a comment

Choose a reason for hiding this comment

Uh oh!

anuragmantri Oct 7, 2025

Choose a reason for hiding this comment

Uh oh!

MaxNevermind Oct 9, 2025

Choose a reason for hiding this comment

Uh oh!

anuragmantri Oct 7, 2025

Choose a reason for hiding this comment

Uh oh!

MaxNevermind Oct 19, 2025

Choose a reason for hiding this comment

Uh oh!

MaxNevermind commented Oct 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Nov 28, 2025

Uh oh!

MaxNevermind commented Nov 30, 2025

Uh oh!

github-actions bot commented Jan 1, 2026

Uh oh!

github-actions bot commented Jan 9, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

anuragmantri commented Mar 16, 2026

Uh oh!

MaxNevermind commented Mar 16, 2026

Uh oh!

anuragmantri commented Mar 16, 2026

Uh oh!

MaxNevermind commented Mar 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

MaxNevermind commented Oct 7, 2025 •

edited

Loading

MaxNevermind commented Oct 28, 2025 •

edited

Loading