Antalya 26.6: Partition export + cluster functions by zvonand · Pull Request #2146 · Altinity/ClickHouse

zvonand · 2026-08-03T12:37:47Z

Changelog category (leave one):

New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Add support for exporting MergeTree parts and partitions to object storage, including experimental exports to Apache Iceberg, with a command to schedule exports for all partitions at once, automatic type casting as in INSERT ... SELECT, retries with backoff for transient failures, and a persistent history of export tasks in system.replicated_partition_exports. Add alternative syntax for cluster table functions with distributed execution of JOINs on swarm nodes, and speed up object storage queries through caching of S3 ListObjects calls and pre-aggregation on remote nodes.
Add support for exporting MergeTree parts and partitions to object storage, including experimental exports to Apache Iceberg, with a command to schedule exports for all partitions at once, automatic type casting as in INSERT ... SELECT, retries with backoff for transient failures, and a persistent history of export tasks in system.replicated_partition_exports. Add alternative syntax for cluster table functions with distributed execution of JOINs on swarm nodes, and speed up object storage queries through caching of S3 ListObjects calls and pre-aggregation on remote nodes (#1718 by @zvonand, #1646 by @ianton-ru, #1640 by @ianton-ru, #1744 by @zvonand, #1713 by @arthurpassos, #1783 by @ianton-ru, #1741 by @arthurpassos, #1728 by @arthurpassos, #1730 by @arthurpassos, #1813 by @arthurpassos, #1836 by @arthurpassos, #1847 by @arthurpassos, #1782 by @ianton-ru, #1845 by @ianton-ru, #1856 by @ianton-ru, #1863 by @ianton-ru, #1872 by @ianton-ru, #1917 by @arthurpassos, #1779 by @arthurpassos, #1912 by @arthurpassos, #1990 by @arthurpassos, #2004 by @arthurpassos, #1984 by @arthurpassos).

CI/CD Options

Exclude tests:

Regression jobs to run:

Combined port of 23 PR(s) (group auto-grp-pr-1718). Cherry-picked from #1718, #1646, #1640, #1744, #1713, #1783, #1741, #1728, #1730, #1813, #1836, #1847, #1782, #1845, #1856, #1863, #1872, #1917, #1779, #1912, #1990, #2004, #1984.

Antalya 26.3: apassos-3: combined port of 12 PRs #1718 — Antalya 26.3: apassos-3: combined port of 12 PRs
26.3 Antalya port - fixes for s3Cluster distributed calls #1646 — 26.3 Antalya port - fixes for s3Cluster distributed calls
26.3 Antalya port - Alternative syntax for cluster functions #1640 — 26.3 Antalya port - Alternative syntax for cluster functions
Antalya 26.3: Fix export task not being killed during s3 outage #1744 — Antalya 26.3: Fix export task not being killed during s3 outage
Remove lock inside the task export partition #1713 — Remove lock inside the task export partition
Fix incorrect interaction between object_storage_remote_initiator and object_storage_cluster. #1783 — Fix incorrect interaction between object_storage_remote_initiator and object_storage_cluster.
Export partition all #1741 — Export partition all
Make export part and partition respect background tasks memory limit #1728 — Make export part and partition respect background tasks memory limit
improve observability over exceptions #1730 — improve observability over exceptions
dont commit while holding the lock on cleanup thread #1813 — dont commit while holding the lock on cleanup thread
Do not require iceberg insert settings on replicas #1836 — Do not require iceberg insert settings on replicas
Export partition - Introduce commit lock #1847 — Export partition - Introduce commit lock
Antalya 26.3: Cluster Joins part 2 - global mode #1782 — Antalya 26.3: Cluster Joins part 2 - global mode
Do not make 'describe table' query when schema is known #1845 — Do not make 'describe table' query when schema is known
Fix test_remote_initiator_after_non_remote #1856 — Fix test_remote_initiator_after_non_remote
Fix cluster functions with hive partitioning #1863 — Fix cluster functions with hive partitioning
Fix aggregation flow with remote initiator #1872 — Fix aggregation flow with remote initiator
Turn system.replicated_partition_exports table into a history table by removing the TTL #1917 — Turn system.replicated_partition_exports table into a history table by removing the TTL
Auto cast values on export part and partition just like insert select #1779 — Auto cast values on export part and partition just like insert select
Replace the export partition local manifests lock by a multiversion container #1912 — Replace the export partition local manifests lock by a multiversion container
Preserve a few settings to partition export #1990 — Preserve a few settings to partition export
Change export partition log levels #2004 — Change export partition log levels
Partition export per part local backoff policy #1984 — Partition export per part local backoff policy

…next commit) --- Original cherry-pick message follows: Merge pull request #1718 from Altinity/feature/antalya-26.3/apassos-3 Antalya 26.3: apassos-3: combined port of 12 PRs # Conflicts: # ci/jobs/scripts/integration_tests_configs.py # contrib/openssl # src/Common/ErrorCodes.cpp # src/Common/FailPoint.cpp # src/Common/ProfileEvents.cpp # src/Common/setThreadName.h # src/Core/Settings.cpp # src/Core/SettingsEnums.cpp # src/Core/SettingsEnums.h # src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.h # src/Functions/generateSnowflakeID.cpp # src/Interpreters/DDLWorker.cpp # src/Parsers/ASTAlterQuery.cpp # src/Parsers/ASTSystemQuery.cpp # src/Parsers/ParserAlterQuery.cpp # src/Storages/IPartitionStrategy.cpp # src/Storages/IPartitionStrategy.h # src/Storages/MergeTree/IMergeTreeDataPart.cpp # src/Storages/MergeTree/MergeTreeData.cpp # src/Storages/MergeTree/MergeTreeData.h # src/Storages/ObjectStorage/DataLakes/IDataLakeMetadata.h # src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.cpp # src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.h # src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergWrites.cpp # src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergWrites.h # src/Storages/ObjectStorage/DataLakes/Iceberg/MultipleFileWriter.cpp # src/Storages/ObjectStorage/DataLakes/Iceberg/MultipleFileWriter.h # src/Storages/ObjectStorage/DataLakes/Iceberg/Utils.cpp # src/Storages/ObjectStorage/StorageObjectStorage.cpp # src/Storages/ObjectStorage/StorageObjectStorage.h # src/Storages/ObjectStorage/StorageObjectStorageCluster.cpp # src/Storages/ObjectStorage/StorageObjectStorageConfiguration.h # src/Storages/ObjectStorage/StorageObjectStorageSink.h # src/Storages/ObjectStorage/StorageObjectStorageSource.cpp # src/Storages/ObjectStorage/StorageObjectStorageSource.h # src/Storages/StorageMergeTree.cpp # src/Storages/StorageReplicatedMergeTree.cpp # src/Storages/StorageReplicatedMergeTree.h # src/Storages/System/StorageSystemMerges.cpp # src/Storages/System/attachSystemTables.cpp # tests/queries/0_stateless/03413_experimental_settings_cannot_be_enabled_by_default.sql # tests/queries/0_stateless/03745_system_background_schedule_pool.reference

Kept antalya-26.6 side for everything outside the source PR's scope and added the PR's new surfaces (export part/partition, list-objects cache, Iceberg export) on top. Adapted: PENDING_MUTATIONS_NOT_ALLOWED moved to error code 1009 (1005-1008 are taken on antalya-26.6) and END bumped accordingly Adapted: MergeTreeData::writePartLog keeps antalya-26.6's projections_duration_ms parameter and appends the PR's exports_entry; ExportPartTask call sites pass the extra argument Adapted: generateManifestFile keeps antalya-26.6's user_defined_sequence_number / per-file row+byte counts and appends the PR's per_file_stats; the export-commit call site boxes data file paths into Iceberg::IcebergPathFromMetadata Adapted: MultipleFileWriter::getDataFileEntries is implemented on antalya-26.6's existing per-file bookkeeping (data_file_names / data_file_row_counts / data_file_byte_counts / completed_file_stats) instead of the PR's duplicate vectors Source-PR: #1718 (#1718)

…next commit) --- Original cherry-pick message follows: Merge pull request #1646 from Altinity/frontport/antalya-26.3/fix_remote_calls 26.3 Antalya port - fixes for s3Cluster distributed calls # Conflicts: # src/Planner/Planner.cpp # src/Processors/QueryPlan/ObjectFilterStep.cpp # src/Processors/QueryPlan/ObjectFilterStep.h # src/Processors/QueryPlan/QueryPlanStepRegistry.cpp # src/Processors/QueryPlan/ReadFromRemote.cpp # src/QueryPipeline/RemoteQueryExecutor.h # src/Storages/ObjectStorage/StorageObjectStorageCluster.cpp # tests/integration/test_database_iceberg/test.py # tests/integration/test_s3_cluster/test.py

Kept antalya-26.6's already-present, evolved copies of the pieces this frontport re-adds (ObjectFilterStep, its registry entry, the snapshot-based virtuals list in StorageObjectStorageCluster and test_hive_partitioning in test_s3_cluster), and applied the source PR's genuinely new additions on top: the analyzer-side addObjectFilterStep in Planner.cpp, RemoteQueryExecutor::setRemoteFunction/setShardCount plus their ReadFromRemote call sites, and the new integration tests (test_remote_hedged, test_remote_no_hedged, test_joins, _test_cluster_joins). Context-only lines from the 26.3 source that antalya-26.6 no longer has (duplicated_part_uuids, threading/time/HeaderParseError and FloatType imports) were not re-introduced. SettingsChangesHistory: uncommented the existing object_storage_cluster_join_mode row in place and dropped the cherry-pick's duplicate (which had landed inside an unrelated version block). Source-PR: #1646 (#1646)

github-actions · 2026-08-03T12:39:08Z

Workflow [PR], commit [7eba7cb]

…next commit) --- Original cherry-pick message follows: Merge pull request #1640 from Altinity/frontport/antalya-26.3/alternative_syntax 26.3 Antalya port - Alternative syntax for cluster functions # Conflicts: # docs/en/sql-reference/table-functions/iceberg.md # src/Analyzer/FunctionNode.h # src/Common/ErrorCodes.cpp # src/Core/Settings.cpp # src/Databases/DataLake/DatabaseDataLake.cpp # src/Databases/DataLake/DatabaseDataLakeSettings.cpp # src/Databases/DataLake/GlueCatalog.cpp # src/Databases/DataLake/ICatalog.cpp # src/Databases/DataLake/RestCatalog.cpp # src/Databases/DataLake/UnityCatalog.cpp # src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.cpp # src/IO/S3/URI.cpp # src/IO/S3/URI.h # src/IO/S3/getObjectInfo.cpp # src/Interpreters/IcebergMetadataLog.cpp # src/Parsers/FunctionSecretArgumentsFinder.h # src/Server/TCPHandler.cpp # src/Storages/IStorageCluster.h # src/Storages/ObjectStorage/DataLakes/DataLakeConfiguration.h # src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.cpp # src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.h # src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergWrites.cpp # src/Storages/ObjectStorage/DataLakes/Iceberg/PersistentTableComponents.h # src/Storages/ObjectStorage/DataLakes/Iceberg/SchemaProcessor.cpp # src/Storages/ObjectStorage/DataLakes/Iceberg/SchemaProcessor.h # src/Storages/ObjectStorage/DataLakes/Iceberg/Utils.cpp # src/Storages/ObjectStorage/S3/Configuration.cpp # src/Storages/ObjectStorage/StorageObjectStorage.cpp # src/Storages/ObjectStorage/StorageObjectStorageCluster.cpp # src/Storages/ObjectStorage/StorageObjectStorageConfiguration.h # src/Storages/ObjectStorage/StorageObjectStorageSource.cpp # src/Storages/ObjectStorage/registerStorageObjectStorage.cpp # src/Storages/System/StorageSystemTables.cpp # src/TableFunctions/TableFunctionObjectStorage.cpp # tests/integration/compose/docker_compose_iceberg_rest_catalog.yml # tests/integration/helpers/iceberg_utils.py # tests/integration/test_database_delta/test.py # tests/integration/test_database_glue/test.py # tests/integration/test_database_iceberg/test.py # tests/integration/test_mask_sensitive_info/test.py # tests/integration/test_s3_cluster/test.py

Removed the conflict markers left by the cherry-pick and kept the source PR's changes, translated to the shapes antalya-26.6 already has (private configuration members behind getters, renamed assertInitializedDL, ContextPtr threaded through IcebergSchemaProcessor, IcebergPathResolver, table UUIDs in DataLake StorageIDs). Source-PR: #1640 (#1640)

Antalya 26.3: Fix export task not being killed during s3 outage Source-PR: #1744 (#1744)

…n_task Remove lock inside the task export partition Source-PR: #1713 (#1713)

…_ai_issues Fix incorrect interaction between object_storage_remote_initiator and object_storage_cluster. Source-PR: #1783 (#1783)

…next commit) --- Original cherry-pick message follows: Merge pull request #1741 from Altinity/export_partition_all Export partition all # Conflicts: # src/Common/ErrorCodes.cpp # src/Core/Settings.h

Renumbered the two new error codes (EXPORT_PARTITION_ALREADY_EXPORTED, PARTITION_EXPORT_FAILED) to the next free values on antalya-26.6, which already occupies 1005-1009, and bumped END accordingly. Adapted: error code numbering — antalya-26.6 already uses 1005..1009 (STALE_VERSION..PENDING_MUTATIONS_NOT_ALLOWED), so the PR's 1006/1007 became 1010/1011 and END 1007 became 1011 Source-PR: #1741 (#1741)

…next commit) --- Original cherry-pick message follows: Merge pull request #1728 from Altinity/export_part_respect_background_memory_limit Make export part and partition respect background tasks memory limit # Conflicts: # src/Storages/MergeTree/MergeTreeData.cpp

Kept ours' include layout; the only source-PR include addition (Common/MemoryTracker.h) was placed into the relocated <Common/...> include block on antalya-26.6. Source-PR: #1728 (#1728)

…ptions_observability improve observability over exceptions Source-PR: #1730 (#1730)

dont commit while holding the lock on cleanup thread Source-PR: #1813 (#1813)

…tting_madness Do not require iceberg insert settings on replicas Source-PR: #1836 (#1836)

Export partition - Introduce commit lock Source-PR: #1847 (#1847)

…next commit) --- Original cherry-pick message follows: Merge pull request #1782 from Altinity/frontport/antalya-26.3/json_part2 Antalya 26.3: Cluster Joins part 2 - global mode # Conflicts: # src/Storages/buildQueryTreeForShard.cpp # src/Storages/buildQueryTreeForShard.h # tests/integration/test_database_iceberg/test.py # tests/integration/test_s3_cluster/test.py

Kept antalya-26.6 additions outside the PR scope (transfer_overflow_mode setting declaration, forward-declared Block for buildShardCollapseFanOut, per-worker generated-data cleanup in test_s3_cluster, disabled _test_cluster_joins in test_database_iceberg) and applied the PR's changes on top; dropped the unchanged test_hive_partitioning context block the merge commit carried into the conflict region. Source-PR: #1782 (#1782)

…next commit) --- Original cherry-pick message follows: Merge pull request #1845 from Altinity/feature/antalya-26.3/no_useless_describe Do not make 'describe table' query when schema is known # Conflicts: # src/TableFunctions/TableFunctionRemote.cpp

Source-PR: #1845 (#1845)

…emote_initiator_after_non_remote Fix test_remote_initiator_after_non_remote Source-PR: #1856 (#1856)

…next commit) --- Original cherry-pick message follows: Merge pull request #1863 from Altinity/bugfix/antalya-26.3/1855_s3cluster_hive Fix cluster functions with hive partitioning # Conflicts: # src/Storages/ObjectStorage/StorageObjectStorageCluster.cpp # src/Storages/StorageFileCluster.cpp # src/Storages/StorageURLCluster.cpp # tests/integration/test_file_cluster/test.py

Kept antalya-26.6's metadata-snapshot based virtuals plumbing and applied the PR's switch to the filtered hive partition column list. Adapted: IStorageCluster::getHivePartitionColumnsWithoutVirtuals now takes the metadata snapshot, because IStorage::getVirtualsList() no longer exists on antalya-26.6 (virtuals live in StorageInMemoryMetadata::virtuals and are read via metadata->virtuals.getSampleBlock(VirtualsKind::All, VirtualsMaterializationPlace::Reader).getNamesAndTypesList()). Adapted: StorageFileCluster sets virtuals through storage_metadata.setVirtuals() instead of the removed IStorage::setVirtuals(), keeping the PR's new sample_path / PartitionStrategyFactory::StrategyType::NONE arguments. Source-PR: #1863 (#1863)

…next commit) --- Original cherry-pick message follows: Merge pull request #1872 from Altinity/bugfix/antalya-26.3/fix_aggregation_with_remote_initiator Fix aggregation flow with remote initiator # Conflicts: # src/Storages/ObjectStorage/StorageObjectStorageCluster.cpp

…tion_export Preserve a few settings to partition export Source-PR: #1990 (#1990)

…vels Change export partition log levels Source-PR: #2004 (#2004)

…next commit) --- Original cherry-pick message follows: Merge pull request #1984 from Altinity/export-partition-retry-backoff Partition export per part local backoff policy # Conflicts: # src/Core/Settings.cpp # src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.cpp

Settings.cpp: kept both obsolete-setting rows (base branch's allow_experimental_query_deduplication and the PR's export_merge_tree_partition_max_retries). IcebergMetadata.cpp: added only the PR's new error codes METADATA_MISMATCH and UNFINISHED; INCORRECT_DATA was context in the source PR's diff and is unused on antalya-26.6, so it was not imported. Source-PR: #1984 (#1984)

# Conflicts: # src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.h # tests/integration/test_database_iceberg/test.py

# Conflicts: # docs/en/engines/database-engines/datalake.md # src/Common/ProfileEvents.cpp # src/Core/Settings.cpp # src/Parsers/FunctionSecretArgumentsFinder.h # src/Storages/ObjectStorage/DataLakes/Common/AvroForIcebergDeserializer.cpp # src/Storages/ObjectStorage/DataLakes/IDataLakeMetadata.h # src/Storages/ObjectStorage/DataLakes/Iceberg/SchemaProcessor.cpp # src/Storages/ObjectStorage/DataLakes/Iceberg/tests/gtest_iceberg_schema_processor.cpp # src/Storages/ObjectStorage/StorageObjectStorageCluster.cpp # src/Storages/ObjectStorage/StorageObjectStorageConfiguration.cpp # src/Storages/ObjectStorage/StorageObjectStorageSource.cpp # src/Storages/ObjectStorage/registerStorageObjectStorage.cpp # src/Storages/StorageFileCluster.cpp # src/Storages/StorageURLCluster.cpp # tests/integration/test_s3_cluster/test.py # tests/queries/0_stateless/03413_experimental_settings_cannot_be_enabled_by_default.sql

… new Antalya settings in the changes history The merge pushed ASTSystemQuery::Type past 127 values, so magic_enum silently dropped RESET_DDL_WORKER and SYSTEM RESET DDL WORKER became unparsable; and three Antalya settings were missing from SettingsChangesHistory.cpp. Addresses 2 failing test(s) in Fast test on #2146. Still-failing set shrank from 2 -> 0.

zvonand · 2026-08-12T13:59:39Z

RelEasy `analyze-fails` — DONE

run completed at 2026-08-12T13:59:37Z

Head SHA: ea0816eea1 (feature/antalya-26.6/auto-grp-pr-1718)
Tests considered: 2 across 1 CI shard(s)
Outcomes: 1 done · 0 partial · 0 unrelated · 0 unresolved
Commits added by AI: 1 (✅ pushed)
Anthropic cost: $4.0495

Per-shard outcomes

✅ DONE — `Fast test`

2 failed test(s) considered — +1 commit(s) — cost $4.0495
full report

AI narration

I'll start by inspecting the repository state and the PR diff.
Now let me apply both fixes.
Now the settings history entries:

Summary

Triage (Step 1). The PR is a group merge that pulls a large Antalya feature set into antalya-26.6. Both failures are traceable to the merged content:

Root cause A — 02995_new_settings_history: the merge brought in three new settings (lock_object_storage_task_distribution_ms, allow_experimental_iceberg_read_optimization, allow_retries_in_cluster_requests) whose SettingsChangesHistory.cpp entries were left commented out during conflict resolution (the head commit ea0816eea1f un-commented some neighbours but not these). Verified: base branch has neither the settings nor the entries; HEAD has the settings but not the entries.

Root cause B — 04401_system_reset_ddl_worker_access: the merge combined the backport that adds ASTSystemQuery::Type::RESET_DDL_WORKER (last enumerator) with Antalya-only SYSTEM types (STOP/START_SWARM_MODE, DROP_OBJECT_STORAGE_LIST_OBJECTS_CACHE, …). The enum grew from 126 values on antalya-26.6 to 129 on HEAD, so RESET_DDL_WORKER == 128 fell outside magic_enum's default range [-128, 127]. ParserSystemQuery enumerates types via magic_enum::enum_values, so SYSTEM RESET DDL WORKER stopped parsing entirely → no ACCESS_DENIED (and no "grant SYSTEM RESET DDL WORKER" message on the ON CLUSTER path), while CHECK GRANT (which doesn't go through that parser) still reported the privilege — exactly the observed 2-line diff. ASTSystemQuery::typeToString also indexed a magic_enum-sized array out of bounds.

Fix (Step 3, one build). src/Parsers/ASTSystemQuery.h: specialize magic_enum::customize::enum_range<DB::ASTSystemQuery::Type> (0..512) + include base/EnumReflection.h. src/Core/SettingsChangesHistory.cpp: un-comment the three Antalya setting entries. bash .releasy/build.sh succeeded.

Verification (Step 4). The pre-existing local binary was stale (lacked the merged settings), so results only became meaningful after the rebuild. Batch re-run of both failures plus two access-related regression spot-checks:

[1 / 4] 01271_show_privileges:                 [ OK ]
[2 / 4] 02995_new_settings_history:            [ OK ]
[3 / 4] 04401_system_reset_ddl_worker_access:  [ OK ]
[4 / 4] 03247_show_grants_with_implicit:       [ OK ]

Committed as 4b71fda9f2e (append-only); git status --porcelain is empty.

Per-test outcome

04401_system_reset_ddl_worker_access — [fixed] — this PR's merge pushed ASTSystemQuery::Type past magic_enum's default 127 limit so SYSTEM RESET DDL WORKER no longer parsed; range extended, test passes.
02995_new_settings_history — [fixed] — three settings added by this PR's merge had their SettingsChangesHistory.cpp entries left commented out; un-commented, test passes.

DONE

Summary

Triage (Step 1). The PR is a group merge that pulls a large Antalya feature set into antalya-26.6. Both failures are traceable to the merged content:

Root cause A — 02995_new_settings_history: the merge brought in three new settings (lock_object_storage_task_distribution_ms, allow_experimental_iceberg_read_optimization, allow_retries_in_cluster_requests) whose SettingsChangesHistory.cpp entries were left commented out during conflict resolution (the head commit ea0816eea1f un-commented some neighbours but not these). Verified: base branch has neither the settings nor the entries; HEAD has the settings but not the entries.

Root cause B — 04401_system_reset_ddl_worker_access: the merge combined the backport that adds ASTSystemQuery::Type::RESET_DDL_WORKER (last enumerator) with Antalya-only SYSTEM types (STOP/START_SWARM_MODE, DROP_OBJECT_STORAGE_LIST_OBJECTS_CACHE, …). The enum grew from 126 values on antalya-26.6 to 129 on HEAD, so RESET_DDL_WORKER == 128 fell outside magic_enum's default range [-128, 127]. ParserSystemQuery enumerates types via magic_enum::enum_values, so SYSTEM RESET DDL WORKER stopped parsing entirely → no ACCESS_DENIED (and no "grant SYSTEM RESET DDL WORKER" message on the ON CLUSTER path), while CHECK GRANT (which doesn't go through that parser) still reported the privilege — exactly the observed 2-line diff. ASTSystemQuery::typeToString also indexed a magic_enum-sized array out of bounds.

Fix (Step 3, one build). src/Parsers/ASTSystemQuery.h: specialize magic_enum::customize::enum_range<DB::ASTSystemQuery::Type> (0..512) + include base/EnumReflection.h. src/Core/SettingsChangesHistory.cpp: un-comment the three Antalya setting entries. bash .releasy/build.sh succeeded.

Verification (Step 4). The pre-existing local binary was stale (lacked the merged settings), so results only became meaningful after the rebuild. Batch re-run of both failures plus two access-related regression spot-checks:

[1 / 4] 01271_show_privileges:                 [ OK ]
[2 / 4] 02995_new_settings_history:            [ OK ]
[3 / 4] 04401_system_reset_ddl_worker_access:  [ OK ]
[4 / 4] 03247_show_grants_with_implicit:       [ OK ]

Committed as 4b71fda9f2e (append-only); git status --porcelain is empty.

Per-test outcome

04401_system_reset_ddl_worker_access — [fixed] — this PR's merge pushed ASTSystemQuery::Type past magic_enum's default 127 limit so SYSTEM RESET DDL WORKER no longer parsed; range extended, test passes.
02995_new_settings_history — [fixed] — three settings added by this PR's merge had their SettingsChangesHistory.cpp entries left commented out; un-commented, test passes.

DONE

🤖 Posted automatically by releasy analyze-fails. Re-run the command to refresh.

# Conflicts: # src/Common/ProfileEvents.cpp # src/Databases/DataLake/ICatalog.cpp # src/Databases/DataLake/RestCatalog.cpp # src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.cpp

…te node runs them to completion Addresses 2 failing test(s) in Stateless tests (arm_binary, parallel) on #2146. Still-failing set shrank from 2 -> 1. RemoteQueryExecutor marked queries sent by remote()/cluster() table functions with a single shard as INITIAL_QUERY so the remote node can act as an initiator and distribute further (swarm). For intermediate processing stages the remote node returns a header made of internal column identifiers that the initiator matches by name; INITIAL_QUERY enables query tree optimizations there (they are skipped for SECONDARY_QUERY), which rewrote tupleElement(__table1.t, 1) into the subcolumn __table1.`t.1` and broke 04045_merge_function_missing_columns_remote with prefer_localhost_replica=0. Restrict the masquerade to stage == Complete, which is exactly the single-node proxy case the feature needs.

…st, <table function>) Addresses 3 failing test(s) in Integration tests (arm_binary, distributed plan, 3/4) on #2146. Still-failing set shrank from 3 -> 0. Commit 3c7a3ef restricted the 'remote node acts as initiator' masquerade in RemoteQueryExecutor to stage == Complete to fix 04045_merge_function_missing_columns_remote. That also disabled it for the object_storage_remote_initiator flow (remote(host, s3Cluster(...))) whose queries run at an intermediate stage, so the chosen remote initiator handled the query as a plain worker instead of distributing it over the swarm and the tests counted 2 instead of 4 (1 instead of 2) query_log entries. Restore the original condition in RemoteQueryExecutor and instead enable the masquerade only when remote() wraps a table function (table_func_ptr != nullptr), which is exactly the object_storage_remote_initiator case. For remote(host, db, table) - the shape used by 04045_merge_function_missing_columns_remote - the remote node stays a secondary query, so the header of internal column identifiers is not rewritten by query tree optimizations.

…luster in system.tables Addresses 3 failing test(s) in Integration tests (arm_binary, distributed plan, 4/4) on #2146. Still-failing set shrank from 3 -> 0. The cherry-picked 'Expose IcebergS3 partition_key and sorting_key in system.tables' feature looks up the data lake metadata with dynamic_cast<StorageObjectStorage *>. In this branch object storage table engines are instantiated as StorageObjectStorageCluster, which is not derived from StorageObjectStorage, so the cast always failed and system.tables reported empty partition_key/sorting_key for Iceberg tables (test_system_tables_partition_sorting_keys). Handle both storage types via a small helper. The other two failures of the shard (test_remote_initiator, test_writes_multiple_threads) are already fixed by da20658 and pass locally.

… skipping parquet schema conversion The test imported with the upstream reserved-field-id fix expects ICEBERG_SPECIFICATION_VIOLATION for an unmapped non-reserved field id, but the Antalya-only allow_experimental_iceberg_read_optimization (added by this PR, enabled by default) serves the single-row constant column from the manifest statistics, so the data file is never parsed and the check never runs. Disable that optimization in the two SELECTs of the test so it exercises the parquet schema converter as intended. Addresses 2 failing test(s) in Stateless tests (amd_debug, distributed plan, s3 storage, parallel) on #2146. Still-failing set shrank from 2 -> 1 (the other one is an unrelated flake that passes on rerun).

zvonand · 2026-08-14T08:37:18Z

RelEasy `analyze-fails` — DONE

run completed at 2026-08-14T08:37:16Z

Head SHA: e3a3deb1b2 (feature/antalya-26.6/auto-grp-pr-1718)
Tests considered: 3 across 2 CI shard(s)
Outcomes: 1 done · 0 partial · 1 unrelated · 0 unresolved
Commits added by AI: 1 (✅ pushed)
Anthropic cost: $5.3832

Per-shard outcomes

✅ DONE — `Stateless tests (amd_debug, distributed plan, s3 storage, parallel)`

2 failed test(s) considered — +1 commit(s) — cost $5.2133
full report

AI narration

I'll start by inspecting the PR diff and the failing test list.
Let me write a self-contained runner on alternate ports.
Let me verify the root cause quickly with clickhouse-local (cheap, no server).
Root cause confirmed. Now the minimal fix in the test that this PR's new Antalya feature short-circuits:
Both pass. Now let me clean up scratch artifacts and commit only the test change.

Summary

Triage (git diff antalya-26.6..HEAD) — this PR is a large Antalya/upstream sync (1684 files) that imports both the upstream Iceberg v3 reserved-field-id fix (with its new test) and the Antalya-only Iceberg read optimization.

Investigation of 04337_iceberg_v3_row_lineage_reserved_field_id (reproduced locally, deterministic):

The test's second assertion expects ICEBERG_SPECIFICATION_VIOLATION for a data file carrying an unmapped, non-reserved field_id (2147483447).
allow_experimental_iceberg_read_optimization does not exist in antalya-26.6 (git grep on the base found nothing) and is added by this PR with default true. In StorageObjectStorageSource.cpp it folds columns whose manifest hyperrectangle isPoint() into constants and then sets need_only_count, so the parquet file is never opened and Parquet::SchemaConverter::useColumnMapperIfNeeded() (which carries the new reserved-id check) never runs.
Proved with clickhouse local: same table read returns manifest values with the setting on (stale 5 / 77,88 mismatch) and throws ICEBERG_SPECIFICATION_VIOLATION with allow_experimental_iceberg_read_optimization=0.
Fix (test-only, no rebuild needed): disable that optimization in the test's two SELECTs so it exercises the parquet schema converter as intended. Verified: test passes, 3 repeats green.

03357_replacing_min_age_cleanup: passes locally (single run and in the 3-run batch), and CI's own diagnosis reran it 6/6 green with the identical randomized settings — transient, not related to this diff. No code change.

Runner used (the shard's --distributed-plan / --s3-storage are server-install flags per ci/jobs/functional_tests.py; --distributed-plan is not a clickhouse-test argument): .releasy/run_st2.sh --no-sequential <tests> against a locally started server from build/programs/clickhouse.

Per-test result:

04337_iceberg_v3_row_lineage_reserved_field_id — [fixed] — this PR added allow_experimental_iceberg_read_optimization (default on), which answered the query from manifest stats so the newly imported reserved-field-id check never ran; test now pins the setting off (commit 7eba7cb7099).
03357_replacing_min_age_cleanup — [unrelated] — transient flake (CI reran 6/6 pass with the same randomized settings; passes locally too); nothing in the diff touches ReplacingMergeTree cleanup.

Committed: 7eba7cb7099 (appended, no history rewrite, not pushed). Working tree clean.

DONE

Summary

Triage (git diff antalya-26.6..HEAD) — this PR is a large Antalya/upstream sync (1684 files) that imports both the upstream Iceberg v3 reserved-field-id fix (with its new test) and the Antalya-only Iceberg read optimization.

Investigation of 04337_iceberg_v3_row_lineage_reserved_field_id (reproduced locally, deterministic):

The test's second assertion expects ICEBERG_SPECIFICATION_VIOLATION for a data file carrying an unmapped, non-reserved field_id (2147483447).
allow_experimental_iceberg_read_optimization does not exist in antalya-26.6 (git grep on the base found nothing) and is added by this PR with default true. In StorageObjectStorageSource.cpp it folds columns whose manifest hyperrectangle isPoint() into constants and then sets need_only_count, so the parquet file is never opened and Parquet::SchemaConverter::useColumnMapperIfNeeded() (which carries the new reserved-id check) never runs.
Proved with clickhouse local: same table read returns manifest values with the setting on (stale 5 / 77,88 mismatch) and throws ICEBERG_SPECIFICATION_VIOLATION with allow_experimental_iceberg_read_optimization=0.
Fix (test-only, no rebuild needed): disable that optimization in the test's two SELECTs so it exercises the parquet schema converter as intended. Verified: test passes, 3 repeats green.

03357_replacing_min_age_cleanup: passes locally (single run and in the 3-run batch), and CI's own diagnosis reran it 6/6 green with the identical randomized settings — transient, not related to this diff. No code change.

Runner used (the shard's --distributed-plan / --s3-storage are server-install flags per ci/jobs/functional_tests.py; --distributed-plan is not a clickhouse-test argument): .releasy/run_st2.sh --no-sequential <tests> against a locally started server from build/programs/clickhouse.

Per-test result:

04337_iceberg_v3_row_lineage_reserved_field_id — [fixed] — this PR added allow_experimental_iceberg_read_optimization (default on), which answered the query from manifest stats so the newly imported reserved-field-id check never ran; test now pins the setting off (commit 7eba7cb7099).
03357_replacing_min_age_cleanup — [unrelated] — transient flake (CI reran 6/6 pass with the same randomized settings; passes locally too); nothing in the diff touches ReplacingMergeTree cleanup.

Committed: 7eba7cb7099 (appended, no history rewrite, not pushed). Working tree clean.

DONE

⏭️ UNRELATED — `Stateless tests (arm_binary, parallel)`

1 failed test(s) considered — cost $0.1698
full report

AI narration

I'll start by triaging the single failure against the PR diff.

Triage

Diff scope (git diff antalya-26.6..HEAD --stat, 1684 files): the PR is an Antalya group PR centred on Iceberg (Puffin deletion vectors, REST catalog, system.tables for StorageObjectStorageCluster), remote() sub-query masquerading, plus settings-history registration. Recent commits confirm this (Fix CI: iceberg read optimization…, Fix CI: masquerade remote() sub-queries…).

00071_merge_tree_optimize_aio:

The test file itself is untouched by the diff (git diff antalya-26.6..HEAD --stat -- '*00071*' → empty); no stateful test or stateful-CI file is touched either.
The test is a plain MergeTree(EventDate, …) + min_bytes_to_use_direct_io=8192 + OPTIMIZE test over test.hits (-- Tags: stateful) — it exercises local MergeTree direct-IO reads, no Iceberg / object storage / remote() code paths that this PR modifies.
The praktika excerpt's own diagnosis: reproducible only with randomized settings (78/78 passes without randomization), culprit is the randomized settings combination (max_read_buffer_size 545703, max_block_size 61643, parallel formatting/parsing, …) — i.e. a settings-randomization interaction, not a diff regression.
Annotated flaky-elsewhere on 4 other tracked PRs, which the scoping rule treats as near-conclusive evidence it is not this PR's fault.

Classification: NOT-THIS-PR. I cannot write a "this PR broke this because " sentence, so per the hard scoping rule I made no code changes, ran no build, and did not touch the branch (git status --porcelain is empty, HEAD still 7eba7cb7099).

Summary

00071_merge_tree_optimize_aio — [unrelated] — stateful MergeTree direct-IO test untouched by the diff; failure is randomized-settings-induced (passes 78/78 without randomization) and it is already failing on 4 other tracked PRs, so it's a settings-randomization flake, not a regression from this PR's Iceberg/remote() changes.

No commits created, nothing reverted, working tree clean.

UNRELATED

Triage

Diff scope (git diff antalya-26.6..HEAD --stat, 1684 files): the PR is an Antalya group PR centred on Iceberg (Puffin deletion vectors, REST catalog, system.tables for StorageObjectStorageCluster), remote() sub-query masquerading, plus settings-history registration. Recent commits confirm this (Fix CI: iceberg read optimization…, Fix CI: masquerade remote() sub-queries…).

00071_merge_tree_optimize_aio:

The test file itself is untouched by the diff (git diff antalya-26.6..HEAD --stat -- '*00071*' → empty); no stateful test or stateful-CI file is touched either.
The test is a plain MergeTree(EventDate, …) + min_bytes_to_use_direct_io=8192 + OPTIMIZE test over test.hits (-- Tags: stateful) — it exercises local MergeTree direct-IO reads, no Iceberg / object storage / remote() code paths that this PR modifies.
The praktika excerpt's own diagnosis: reproducible only with randomized settings (78/78 passes without randomization), culprit is the randomized settings combination (max_read_buffer_size 545703, max_block_size 61643, parallel formatting/parsing, …) — i.e. a settings-randomization interaction, not a diff regression.
Annotated flaky-elsewhere on 4 other tracked PRs, which the scoping rule treats as near-conclusive evidence it is not this PR's fault.

Classification: NOT-THIS-PR. I cannot write a "this PR broke this because " sentence, so per the hard scoping rule I made no code changes, ran no build, and did not touch the branch (git status --porcelain is empty, HEAD still 7eba7cb7099).

Summary

00071_merge_tree_optimize_aio — [unrelated] — stateful MergeTree direct-IO test untouched by the diff; failure is randomized-settings-induced (passes 78/78 without randomization) and it is already failing on 4 other tracked PRs, so it's a settings-randomization flake, not a regression from this PR's Iceberg/remote() changes.

No commits created, nothing reverted, working tree clean.

UNRELATED

🤖 Posted automatically by releasy analyze-fails. Re-run the command to refresh.

zvonand · 2026-08-14T08:39:24Z

the only test that is related was fixed. this is a test-only fix (no code changes), so no other code was changed. good to be merged.

zvonand added 4 commits August 3, 2026 13:18

zvonand added ai-needs-attention Releasy stopped on a conflict it could not resolve — needs human review releasy Created/managed by RelEasy antalya-26.6 labels Aug 3, 2026

zvonand added the forwardport This is a frontport of code that existed in previous Antalya versions label Aug 4, 2026

zvonand added 21 commits August 5, 2026 17:00

Merge pull request #1744 from Altinity/feature/antalya-26.3/pr-1564

e8c2701

Antalya 26.3: Fix export task not being killed during s3 outage Source-PR: #1744 (#1744)

Merge pull request #1713 from Altinity/export_partition_remove_lock_i…

1906f20

…n_task Remove lock inside the task export partition Source-PR: #1713 (#1713)

Merge pull request #1783 from Altinity/bugfix/antalya-26.3/frontports…

3c13e24

…_ai_issues Fix incorrect interaction between object_storage_remote_initiator and object_storage_cluster. Source-PR: #1783 (#1783)

Cherry-pick of #1741 with unresolved conflict markers (resolution in …

98ed773

…next commit) --- Original cherry-pick message follows: Merge pull request #1741 from Altinity/export_partition_all Export partition all # Conflicts: # src/Common/ErrorCodes.cpp # src/Core/Settings.h

Resolve conflicts in cherry-pick of #1728

bdc7adf

Kept ours' include layout; the only source-PR include addition (Common/MemoryTracker.h) was placed into the relocated <Common/...> include block on antalya-26.6. Source-PR: #1728 (#1728)

Merge pull request #1730 from Altinity/export_partition_refactor_exce…

cda76d4

…ptions_observability improve observability over exceptions Source-PR: #1730 (#1730)

Merge pull request #1813 from Altinity/export_partition_commit_no_lock

c5f4ee9

dont commit while holding the lock on cleanup thread Source-PR: #1813 (#1813)

Merge pull request #1836 from Altinity/export_partition_drop_allow_se…

55ef77e

…tting_madness Do not require iceberg insert settings on replicas Source-PR: #1836 (#1836)

Merge pull request #1847 from Altinity/export-partition-commit-lock

79f0cfd

Export partition - Introduce commit lock Source-PR: #1847 (#1847)

Resolve conflicts in cherry-pick of #1845

19d4882

Source-PR: #1845 (#1845)

Merge pull request #1856 from Altinity/bugfix/antalya-26.3/fix_test_r…

8503183

…emote_initiator_after_non_remote Fix test_remote_initiator_after_non_remote Source-PR: #1856 (#1856)

arthurpassos and others added 4 commits August 5, 2026 19:00

Merge pull request #1990 from Altinity/preserve_few_settings_to_parti…

799819f

…tion_export Preserve a few settings to partition export Source-PR: #1990 (#1990)

Merge pull request #2004 from Altinity/change_export_partition_log_le…

93a5c3f

…vels Change export partition log levels Source-PR: #2004 (#2004)

zvonand added ai-resolved Port conflict auto-resolved by Claude and removed ai-needs-attention Releasy stopped on a conflict it could not resolve — needs human review labels Aug 6, 2026

zvonand marked this pull request as ready for review August 6, 2026 15:54

Merge origin/antalya-26.6 into feature/antalya-26.6/auto-grp-pr-1718

e14a01f

# Conflicts: # src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.h # tests/integration/test_database_iceberg/test.py

zvonand mentioned this pull request Aug 7, 2026

Port graph for antalya-26.6 #2048

Open

54 tasks

zvonand added 3 commits August 12, 2026 11:43

Update SettingsChangesHistory.cpp

ea0816e

zvonand added 3 commits August 13, 2026 05:51

Merge origin/antalya-26.6 into feature/antalya-26.6/auto-grp-pr-1718

e02c797

# Conflicts: # src/Common/ProfileEvents.cpp # src/Databases/DataLake/ICatalog.cpp # src/Databases/DataLake/RestCatalog.cpp # src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.cpp

Update SettingsChangesHistory.cpp

c1c8cdc

Update SettingsChangesHistory.cpp

7564858

ianton-ru mentioned this pull request Aug 13, 2026

Remove object_storage_cluster_join_mode='local' — it is now equivalent to 'allow' #1827

Open

zvonand added 2 commits August 13, 2026 12:16

fix tests

2dd6c7b

This comment was marked as outdated.

Sign in to view

zvonand added 2 commits August 14, 2026 05:26

This comment was marked as outdated.

Sign in to view

zvonand added the verified Approved for release label Aug 14, 2026

zvonand merged commit 5f5903e into antalya-26.6 Aug 14, 2026
89 of 90 checks passed

zvonand added the port-antalya PRs to be ported to all new Antalya releases label Aug 14, 2026

zvonand mentioned this pull request Aug 15, 2026

Antalya 26.6: Iceberg: support external paths in tables #2154

Merged

28 tasks

Navigation Menu

Uh oh!

Antalya 26.6: Partition export + cluster functions - #2146

Antalya 26.6: Partition export + cluster functions#2146
zvonand merged 46 commits into
antalya-26.6from
feature/antalya-26.6/auto-grp-pr-1718

zvonand commented Aug 3, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 3, 2026 •

edited

Loading

Uh oh!

zvonand commented Aug 12, 2026

Summary

Summary

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

zvonand commented Aug 14, 2026

Summary

Summary

Triage

Summary

Triage

Summary

Uh oh!

zvonand commented Aug 14, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

zvonand commented Aug 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

CI/CD Options

Exclude tests:

Regression jobs to run:

Uh oh!

github-actions Bot commented Aug 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zvonand commented Aug 12, 2026

RelEasy analyze-fails — DONE

Per-shard outcomes

✅ DONE — Fast test

Summary

Summary

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

zvonand commented Aug 14, 2026

RelEasy analyze-fails — DONE

Per-shard outcomes

✅ DONE — Stateless tests (amd_debug, distributed plan, s3 storage, parallel)

Summary

Summary

⏭️ UNRELATED — Stateless tests (arm_binary, parallel)

Triage

Summary

Triage

Summary

Uh oh!

zvonand commented Aug 14, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

zvonand commented Aug 3, 2026 •

edited

Loading

github-actions Bot commented Aug 3, 2026 •

edited

Loading

RelEasy `analyze-fails` — DONE

✅ DONE — `Fast test`

RelEasy `analyze-fails` — DONE

✅ DONE — `Stateless tests (amd_debug, distributed plan, s3 storage, parallel)`

⏭️ UNRELATED — `Stateless tests (arm_binary, parallel)`