EXPORT PARTITION with position matching + extra columns in source

[k-morozov]

Closes: #1717

Problem. EXPORT PART/EXPORT PARTITION matched source and destination columns positionally, exactly like INSERT INTO dest SELECT * FROM src, and required the column count to match exactly. This is unnecessarily strict for a common
real scenario: a MergeTree table's schema grows over time (new trailing columns added), but older partitions still need to be exported to an external table (Iceberg/object storage) whose schema was fixed when it was created. Such exports were rejected outright with NUMBER_OF_COLUMNS_DOESNT_MATCH, with no way to just drop the newer columns and export the rest.

Solution. Added the export_merge_tree_part_schema_mismatch_mode setting (default strict, preserving the old behavior). The new ignore_extra_source_columns_by_position mode allows the source to have more columns than the destination: the extra trailing source columns (by declared/readable position, the same order INSERT SELECT uses) are dropped, both at synchronous validation time (verifyExportSchemaCastable, so a real mismatch still fails immediately at ALTER TABLE ... EXPORT time) and at actual data-export time (ExportPartTask::addExportConvertingActions, via a projection step before the existing positional CAST). The reverse direction — destination has more columns than source — is still always rejected in both modes.

Known follow-up. Ignored columns are still fully read and decompressed from disk before being discarded in the projection step, so no I/O is saved. Skipping the read up front isn't safe as a naive truncation: a kept
ALIAS/MATERIALIZED column can legally forward-reference an ignored one, so it would need a real dependency-graph walk instead. Left as a possible future optimization

Changelog category (leave one):

• Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Added the export_merge_tree_part_schema_mismatch_mode setting. With ignore_extra_source_columns_by_position (default: strict), EXPORT PART/EXPORT PARTITION allows a source table with extra trailing columns — they are simply
ignored instead of failing with NUMBER_OF_COLUMNS_DOESNT_MATCH.

Documentation entry for user-facing changes

...

CI/CD Options

Exclude tests:

• Fast test
• Integration Tests
• Stateless tests
• Stateful tests
• Performance tests
• Aarch64 tests
• All with ASAN
• All with TSAN
• All with MSAN
• All with UBSAN
• All with Coverage
• All Regression
• Disable CI Cache

Regression jobs to run:

• Fast suites (mostly <1h)
• Aggregate Functions (2h)
• Alter (1.5h)
• Benchmark (30m)
• ClickHouse Keeper (1h)
• Iceberg (2h)
• LDAP (1h)
• OAuth (5m)
• Parquet (1.5h)
• RBAC (1.5h)
• SSL Server (1h)
• S3 (2h)
• S3 Export (2h)
• Swarms (30m)
• Tiered Storage (2h)

[github-actions]

Workflow [PR], commit [ad0a935]

[arthurpassos]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. 🎉

Reviewed commit: e74a863681

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[k-morozov]

All current CI failures on this PR are pre-existing / unrelated to this PR's changes:

• Docker server image, Grype Scan: Docker Hub outage (500 Internal Server Error from auth.docker.io) prevented pulling base images — infrastructure issue.
• RegressionTestsRelease / Iceberg (2): almost all failures are EXPORT PARTITION is not implemented for engine MergeTree for the plain merge tree mode — expected, since that support isn't merged into antalya-26.3 yet; a handful of unrelated TABLE_ALREADY_EXISTS failures are known and already tracked.
• RegressionTestsRelease / OAuth: a ModuleNotFoundError in the test suite itself, unrelated to this PR.
• BuzzHouse (amd_debug): an unrelated LOGICAL_ERROR ("Inconsistent AST formatting") hit by a fuzzed query using GROUP BY ... WITH CUBE, unrelated to Iceberg/EXPORT PARTITION.

No failure traces back to the changes in this PR.

[ianton-ru]

I think name is confusing.
I expect that I can say "ignore columns at positions 3, 5 and 7", but actually it ignores last columns.
May be something like ignore_trailing_extra_columns?
/I'm not good in naming anyway/

[k-morozov]

I thought to use 2 modes: ignore_extra_source_columns_by_position and ignore_extra_source_columns_by_name (in the future). Thank you for the example — I agree that the current naming is really confusing.

In the current name, the _by_position suffix meant that all columns from the prefix are matched 1-to-1, and the rest are ignored. I like the name ignore_trailing_extra_columns as the base. I'll suggest adding information that this is about the source. But I also wanted to convey that the columns in the prefix are matched sequentially:

ignore_trailing_extra_source_columns_match_by_sequence

[ianton-ru]

I think it must be a table property in case with several different politics. If you accidentally export one partition with one politic and another partition with different, you get a lot of pain, because same Iceberg column will contain different real data.
And a big question here is wat to do with hybrid tables. When different politics are possible, hybrid must have a knowledge about used to join results from MergeTree and Iceberg tables correctly.

[k-morozov]

I think it must be a table property in case with several different politics. If you accidentally export one partition with one politic and another partition with different, you get a lot of pain, because same Iceberg column will contain different real data.

With the current two modes this cannot corrupt data: both strict and ignore_extra_source_columns_by_position build the exact same positional prefix mapping - destination column N always receives source column N. The only difference is whether the export is allowed at all when the column counts diverge. So exporting one partition under strict and another under ignore_extra yields consistent data in the shared columns. Also note the mode is already pinned in the partition-export manifest, so it cannot change mid-operation.

[k-morozov]

And a big question here is wat to do with hybrid tables. When different politics are possible, hybrid must have a knowledge about used to join results from MergeTree and Iceberg tables correctly.

Hybrid doesn't need to know the export policy: it validates that every segment provides all columns of the declared schema at CREATE/ATTACH and throws BAD_ARGUMENTS (missing column ...) otherwise. So there is no silent-mismerge failure mode - either the schemas are reconcilable (Hybrid reads by name, with auto-cast) or it's a hard error. I've added an integration test test_export_part_ignore_extra_column_breaks_hybrid_over_source_and_destination covering the full lifecycle.

[ianton-ru]

I mean not two current modes, but plans to add third variant.
Anyway, this question can be solved in future, when different modes have been implemented.

[ianton-ru]

LGTM

[DimensionWieldr]

PR CI Triage

Stateless, integration, and S3 export regression all passed. The only substantive red job is Iceberg_2; SQLLogic / Grype / FinishCI are infra noise.

Root Cause Analysis

1. Iceberg_2 plain MergeTree EXPORT PARTITION - export partition on plain merge tree has not yet been merged (Plain merge tree partition export #2032); I just updated xfail versions

2. Iceberg_2 /iceberg/iceberg table engine/feature/drop table - test bug, was fixed last week but not pushed to regression release branch yet

3. Iceberg_2 /iceberg/export partition/no catalog/replicated merge tree/catalogs/drop with purge allows recreating same table - Native CREATE TABLE / DROP TABLE for DataLakeCatalog; optionally prune on DROP #1896 hasn't been merged yet, so this test still failing is expected; I updated the xfail version

4. SQLLogic - job never ran tests: praktika S3 upload failed (Unknown parameter in input: "IfMatch" / botocore). Infra issue.

5. Grype×3 - Grype DB hydrate failed (UNIQUE constraint failed on OS table); report still says 0 high/critical. Infra issue.

Looks good from a CI perspective.

[DimensionWieldr]

Added auto tests in https://github.com/Altinity/clickhouse-regression/blob/main/iceberg/tests/export_partition/settings.py.

No issues found so far.

LGTM