fix(secrets): worker persistent-secret hygiene + DROP passthrough

[fuziontech]

Problem

A user hit DuckDB errors that persisted for weeks:

Invalid Configuration Error: Ambiguity detected for secret name 'customer_db',
  secret occurs in multiple storage backends.
Invalid Input Error: Ambiguity found for secret name 'customer_warehouse_prod_s3',
  secret occurs in multiple storages: [local_file,memory]

Root cause. They ran CREATE PERSISTENT SECRET ... once, days/weeks ago. A bare CREATE OR REPLACE SECRET defaults to a temporary (memory) secret, and OR REPLACE only replaces within the same storage backend — so the persistent copy was never touched. It was written under the worker's DataDir (/data/.duckdb/stored_secrets, since the worker's HOME is /data), which survives across worker incarnations on a persistent/long-lived /data. Every activation re-creates the same-named in-memory secret, so the name ends up in both local_file and memory → every reference and every DROP is ambiguous.

Two compounding bugs made cleanup impossible:

1. The fix DuckDB suggests — DROP <PERSISTENT|TEMPORARY> SECRET ... — was unrunnable: those forms don't start with "DROP SECRET", so they fell through to the PG transpiler and died with syntax error at or near "PERSISTENT".
2. DROP SECRET ... FROM local_file is unreliable too: once the file is gone but the secret is still in DuckDB's startup-loaded registry, DuckDB throws IO Error: Failed to remove secret file ... (reproduced against real DuckDB).

Changes

1. Passthrough fix (server/conn.go) — add DROP PERSISTENT SECRET / DROP TEMPORARY SECRET (+ IF EXISTS) to the DuckDB utility-command allowlist so they reach DuckDB instead of the PG parser.

2. Pin the secret directory on workers (server/server.go + duckdbservice/duckdb_pair.go) — new Config.PinSecretDirectory, applied in ConfigureMainDB as SET secret_directory = '<DataDir>/secrets', set for every worker via OpenDuckDBPair (the worker-only DB factory; standalone uses openBaseDB and is untouched). Redirecting away from DuckDB's $HOME default means stale secrets in the old location are no longer loaded, so they can't collide with the in-memory secret re-created at activation — and the location is now deterministic and wipeable.

3. Wipe persisted secrets on worker recycle (duckdbservice/service.go) — Warmup wipes both <DataDir>/secrets (pinned) and the legacy <DataDir>/.duckdb/stored_secrets on worker startup (== recycle; a shared-warm worker is single-org-bound for life). Physically clears the historical files that caused the incident so they don't linger on a persistent DataDir.

Why not allow_persistent_secrets = false

An earlier revision disabled persistent secrets outright. CI caught that it breaks the DuckLake ATTACH path (controlplane-tests + k8s-integration-tests): disabling unregisters DuckDB's local_file secret storage backend, which DuckLake depends on (Unknown secret storage found: 'local_file'). Reverted. Pinning leaves local_file registered (just relocated), so DuckLake is unaffected, while the redirect + recycle wipe still keep workers clean.

Tests

• server/conn_test.go::TestIsDuckDBUtilityCommand — DROP variants, quoting, comments, case, word-boundary guards.
• server/persistent_secrets_test.go — TestPinSecretDirectory (a persistent secret still works and lands under the pinned dir); TestPinSecretDirectoryIgnoresLegacyDefault (a secret in the legacy location is not loaded once pinned away); TestSecretDirectory / LegacySecretDirectory derivation.
• duckdbservice/secret_recycle_test.go::TestWipePersistedSecretsOnRecycle — full recycle contract against real DuckDB, non-vacuous control phase (secret survives a plain reopen), asserts both pinned and legacy dirs are wiped.
• Full go test ./server/ + ./duckdbservice/, go vet, and just lint (CI-pinned golangci-lint) clean on touched files.

Why no tests/k8s unit-test for the wipe

Worker DataDir (/data) is an EmptyDir (controlplane/k8s_pool.go:827), destroyed on pod deletion — a pod-kill test would pass even with the wipe removed. The boundary the wipe guards is a process restart over a surviving DataDir (container restart, persistent volume, warm node), modeled directly by the unit test. (Note: the real DuckLake activation path is still covered by the existing controlplane/k8s integration suites, which is what caught the allow_persistent_secrets regression.)

Behavior changes to note

• Standalone is unchanged — pinning is gated behind PinSecretDirectory, which only workers set. Standalone keeps DuckDB's $HOME default and full persistent-secret behavior, so upgrading doesn't relocate existing secrets.
• Workers: persistent secrets still work, but live under <DataDir>/secrets and are wiped on recycle. CREATE/OR REPLACE` (in-memory) secrets — what duckgres and the user's scripts use — are unaffected.

Immediate user remediation (pre-merge)

The DROP loop won't recover from the partially-deleted state. Clear the files and recycle the worker:

kubectl exec <worker-pod> -n duckgres -- rm -rf /data/.duckdb/stored_secrets
# then restart/recycle that worker

If /data is a persistent volume, deleting the pod alone won't help. After this ships, a worker recycle does this automatically. The user can also drop the DROP ... FROM local_file loop from their setup script.

🤖 Generated with Claude Code