Antalya 26.6: Cache vended credentials for REST catalogs

[zvonand]

Dropped from this backport: the AI dropped these surfaces rather than pulling in a missing prerequisite. Reviewers: confirm each is genuinely optional.

• catalog auth-state invalidation of the vended credentials cache (cacheCredentials state-snapshot check and the commitSettingsChanges cache clear) - depends on the RestCatalog CatalogState/prepareSettingsChanges machinery not yet on antalya-26.6

Changelog category (leave one):

• Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Add option to cache vended credentials for REST catalogs; add a setting vended_credentials_cache_ttl (seconds). 300 by default. 0 means no caching (ClickHouse#107960 by @zvonand).

Cherry-picked from ClickHouse#107960.

---

Now, new vended credentials are requested on each metadata request. This PR adds an (optional) cache for creds with configurable TTL.

CI/CD Options

Exclude tests:

• Fast test
• Integration Tests
• Stateless tests
• Stateful tests
• Performance tests
• All with ASAN
• All with TSAN
• All with MSAN
• All with UBSAN
• All with Coverage
• All with Aarch64
• All Regression
• Disable CI Cache

Regression jobs to run:

• Fast suites (mostly <1h)
• Aggregate Functions (2h)
• Alter (1.5h)
• Benchmark (30m)
• ClickHouse Keeper (1h)
• Iceberg (2h)
• LDAP (1h)
• Parquet (1.5h)
• RBAC (1.5h)
• SSL Server (1h)
• S3 (2h)
• S3 Export (2h)
• Swarms (30m)
• Tiered Storage (2h)

[github-actions]

Workflow [PR], commit [da60402]

[zvonand]

@blau-ai

[zvonand]

@blau-ai

[blau-ai]

CI triage

Verdict: 11 red checks — 0 caused by this PR. Every failure is pre-existing on antalya-26.6, a known-flaky test, or an infra glitch in the CI tooling. None of them touch the files or code paths this PR changes (REST-catalog vended-credentials caching + Azure buffers). All build flavors that had a valid cache/config compiled the PR's own code cleanly.

Check	Class	PR-caused?
Build (amd_msan), Build (arm_msan)	pre-existing base-branch build break	❌
Unit tests (asan_ubsan), Unit tests (tsan)	pre-existing test failure	❌
Stateless tests (amd_debug, parallel)	flaky (randomized settings)	❌
Regression: iceberg_1, oauth, s3_export_part, s3_export_partition, swarms	pre-existing (test-suite vs build version drift)	❌
Grype Scan clickhouse-server	infra (scanner crashed)	❌

---

Build (amd_msan) / Build (arm_msan) — pre-existing base-branch break

FAILED: .../Iceberg/IcebergWrites.cpp.o
/ClickHouse/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergWrites.cpp:179:28:
error: use of undeclared identifier 'getDecimalScale'

IcebergWrites.cpp is not in this PR's diff. getDecimalScale is declared in DataTypes/DataTypesDecimal.h, which the file does not #include (and none of its included headers pull it transitively). It compiles everywhere else only because ccache serves a stale object; the msan flavor rebuilds from source and hits the real error.

Proof it's pre-existing: the base branch's own MasterCI run 31142308506 (started ~4h before this PR's run) fails with the identical getDecimalScale error at IcebergWrites.cpp:179.

This is what blocks all the msan-dependent jobs (they show as dropped, not run).

Fix (belongs on antalya-26.6, not this PR): add #include <DataTypes/DataTypesDecimal.h> to src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergWrites.cpp. That one line unblocks msan for every PR against the branch. I can open it as a separate blau/* PR if you'd like — just say the word.

Unit tests (asan_ubsan) / Unit tests (tsan) — pre-existing

Both fail on the same gtest throwing Code: 1001, std::runtime_error, e.what() = rebuild failed, in the DiskAccessStorageRecovery access-storage suite. Deterministic across both sanitizers, no connection to DataLake/Azure. Not introduced here.

Stateless tests (amd_debug, parallel) — flaky

Single failure: 00071_merge_tree_optimize_aio timed out ([ FAIL ] 600.03 sec). CI's own minimizer reports "Passes without randomization. Confirmed: the failure is caused by randomized settings" and the runner logs "NOTE: Failed 1 tests - do not block pipeline, exit with 0." MergeTree AIO test, unrelated to this PR. Safe to re-run.

Regression: iceberg_1, oauth, s3_export_part, s3_export_partition, swarms — pre-existing (version drift)

Every one of these fails at the "feature support validation" gate, not in any actual scenario logic:

• oauth → OAuth/JWT authentication is not supported by this build
• swarms → Swarm object_storage_cluster is not supported by this build
• s3_export_part / partition → Export merge tree part is not supported by this build
• iceberg_1 → Swarm object_storage_cluster_join_mode ... not supported and Iceberg partition timezone ... not supported

The shared regression suite is newer than the 26.6 build and probes features not present in this flavour. These would fail identically on the base branch and are independent of vended-credentials caching (the iceberg/oauth runs never reach the credential path — they bail at the feature check).

Grype Scan clickhouse-server — infra

Not a vulnerability finding. The scan step crashed before producing output (jq: error: Could not open file result.json, can't open 'raw.log', preceded by a pip dependency conflict), so the status resolves to "An error occurred." The keeper and -alpine server Grype scans both passed with 0 high/critical. Re-run.

---

PR health check

The PR itself looks healthy. All build configs with a valid cache — amd/arm debug, release, asan_ubsan, tsan, binary — are green, as are Fast test, the targeted integration and AST fuzzer jobs. No failing check touches the changed files (DatabaseDataLake*, RestCatalog*, ICatalog.h, StorageCredentials.h, Azure*, ProfileEvents.cpp) or their code paths. Bottom line: nothing to fix in this PR to make CI reflect its correctness — the board is red for reasons that predate and stand apart from it. The only genuine bug in sight is the base-branch IcebergWrites.cpp missing include above.

One non-CI note for reviewers: the PR description already flags that vended-credentials cache invalidation (auth-state snapshot check + commitSettingsChanges cache clear) was dropped from this backport. That's a functional review item, not a CI item.

[zvonand]

@blau-ai

[blau-ai]

CI triage for #2156

Verdict: CI is red, but none of the failures are caused by this PR. They split into pre‑existing/systemic failures on antalya-26.6 and infra/flaky noise. The diff only touches REST‑catalog vended‑credential caching + Azure blob storage (RestCatalog.*, DatabaseDataLake*, ICatalog.h, ProfileEvents.cpp, Azure *ObjectStorage*, one integration test) — none of the failing test paths exercise that code.

⚠️ I can't build/run ClickHouse or fetch the S3 report.html/build logs from this container (only GitHub Actions logs + the CI status API). Classifications below are from GHA job logs, the commit status API, and base‑branch/sibling‑PR comparison; where I couldn't see a root‑cause log directly I say so and recommend a re‑run.

Not PR‑caused — pre‑existing / systemic on antalya-26.6

Check	Evidence	Classification
Regression iceberg_2 (aarch64 + release)	89/207 features failed — every failing feature is under /iceberg/export partition/glue catalog/… (EXPORT PARTITION, SRS‑047: "50 requirements, 42 unsatisfied 84%"). The catalog started fine (87 features pass). This PR touches REST catalog + Azure, not Glue catalog or EXPORT PARTITION. The base branch's last iceberg_2 run executed only 34 features (4 failed) — the large export‑partition tree comes from a newer regression‑suite ref and is broadly unimplemented/failing on this branch.	Pre‑existing feature gap / test‑suite version, unrelated to the diff
Regression settings (aarch64 + release)	6/1703 scenarios failed; base antalya-26.6 already fails settings (3/1679). The PR adds a DatabaseDataLake setting (vended_credentials_cache_ttl), not a system.settings server setting, so it can't move this suite.	Pre‑existing
Regression S3Export (part) s3_export_part (aarch64 + release)	Red on the base branch and on sibling PRs #2201/#2184/#2199 (both arches). Tracked by open issues #1189 / #1190 (MinIO export‑part).	Pre‑existing / systemic
Grype Scan (GrypeScanServer + server image)	Fails with "An error occurred" = grype can't load the vulnerability DB (pinned version). Matches open tracking issue #2060. Keeper/alpine grype scans passed.	Infra (known issue #2060)

Not PR‑caused — flaky / infra (re‑run should clear)

Check	Evidence	Classification
Build (amd_asan_ubsan)	Status "Failures: 1/3", but the same commit built green for 13/14 variants — amd_debug, amd_release, amd_binary, amd_msan, amd_tsan, arm_asan_ubsan, arm_ubsan, etc. A real compile error would fail every variant (especially arm_asan_ubsan). Base branch builds amd_asan_ubsan green.	Flaky build (likely OOM/timeout on the combined ASAN+UBSAN job). Could not read the build log directly — recommend re‑run to confirm.
Stateless tests (amd_debug, parallel)	Failed: 2 → 00086_array_reduce (359.97s) and 00177_memory_bound_merging (600.04s = hit the 600s timeout). Generic array/merging tests with zero relation to DataLake. Long runtimes on the slow debug build.	Flaky / timeout
Regression S3 (aws_s3, 2) s3_aws_s3_2	Passed on aarch64, failed only on release for the same commit; base branch passes it.	Transient flake
Regression S3 (gcs, 2) s3_gcs_2	"1 module (1 errored) … 1 retry (1 ok)"; passed on release; base passes.	Transient infra flake

Aggregate gates (no independent failure)

PR and FinishCIReport are roll‑ups that just reflect the reds above.

Suggested next steps

• No code change is required for CI. Nothing in the diff maps to a failing test path.
• Re‑run the flaky checks: Build (amd_asan_ubsan), Stateless tests (amd_debug, parallel), Regression S3 (aws_s3, 2), Regression S3 (gcs, 2), and the Grype scans.
• iceberg_2 / settings / s3_export_part: pre‑existing on antalya-26.6; safe to disregard for this PR (track via Segmentation faults after restarting MinIO (TSAN) #1189/Seg faults and logical errors when detaching/attaching partitions with ongoing exports (UBSAN) #1190 and the export‑partition/SRS‑047 gap). Grype is Grype Scan fails: pinned grype v0.92.2 cannot load current vulnerability DB #2060.
• Heads‑up (not a failure): the integration suite — including the test this PR modifies, test_database_iceberg_lakekeeper_catalog — was skipped in this run (Integration tests (…, targeted) = skipping). So the new vended‑credentials caching path isn't positively exercised by CI here; you may want to force that targeted integration job to run before merge to get real coverage of the change.

_{Analysis only — no code changes pushed. Evidence: run 31794598300; iceberg_2 job 94793962453; stateless job 94769988620; commit status API for da60402.}