From 8fb221a4c7d0731dcc5a67f3b56ca67c4cbfe3ab Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Fri, 24 Jul 2026 23:36:22 +0300 Subject: [PATCH 1/5] docs(dha): revise proposal to HPA-on-scale-subresource after spike The implementation spike disproved the enforced-ownership premise the first revision rested on: SSA field ownership does not hold on the aggregated apps API, admission webhooks cannot fire there, and the fallback HelmRelease webhook is advisory, bypassable, and platform-wide. Revise the mechanism to reuse what Kubernetes already ships: a stock HorizontalPodAutoscaler on the engine operator's scale subresource, a chart conditional that omits the replica field under autoscaling (so Flux and the autoscaler no longer contend), and a thin engine-aware guard for the two brakes HPA lacks (synchronous-quorum floor and replication-lag gate). Record the spike findings and the four alternatives the first Alternatives section skipped. Scope the MVP to PostgreSQL (CNPG); MariaDB follows once its chart supports scale-out; Redis and MongoDB, which expose no scale subresource, are deferred to a thin-shim follow-up. Signed-off-by: Alexey Artamonov --- .../database-horizontal-autoscaling/README.md | 237 +++++++++--------- 1 file changed, 118 insertions(+), 119 deletions(-) diff --git a/design-proposals/database-horizontal-autoscaling/README.md b/design-proposals/database-horizontal-autoscaling/README.md index ddd85c7..ed21bc2 100644 --- a/design-proposals/database-horizontal-autoscaling/README.md +++ b/design-proposals/database-horizontal-autoscaling/README.md @@ -2,14 +2,16 @@ - **Title:** `Database Horizontal Autoscaler for Cozystack` - **Author(s):** `@scooby87` -- **Date:** `2026-07-08` (addressing review by `@IvanHunters`, Gemini, and CodeRabbit) +- **Date:** `2026-07-08`; revised `2026-07-24` after the implementation spike, addressing review by `@IvanHunters`, `@lllamnyp`, Gemini, and CodeRabbit - **Status:** Draft ## Overview -Managed databases in Cozystack (`postgres`, `mariadb`, `redis`, `mongodb`, and others) are scaled only manually today: an operator edits the `replicas` value of the application and waits for the underlying operator to converge. This proposal introduces a dedicated operator, `db-autoscaler`, that automatically adjusts the number of **read replicas** of a managed database in response to load, driven by a new HPA-like custom resource `DatabaseHorizontalAutoscaler` (DHA). +Managed databases in Cozystack (`postgres`, `mariadb`, `redis`, `mongodb`, and others) are scaled only manually today: an operator edits the `replicas` value of the application and waits for the underlying operator to converge. This proposal introduces automatic horizontal scaling of a managed database's **read replicas** in response to load. -The proposal is deliberately scoped to **horizontal scaling of read replicas**, because a stateful database primary cannot be scaled horizontally the way a stateless Deployment can. The autoscaler is topology-aware per engine, respects the synchronous-replica quorum, brakes on replication lag, and applies its decisions by patching the application's `replicas` value (the `Application` `spec`) — the same field a human would edit. Patching that field avoids the engine-CR ownership conflict a stock HPA causes; it does **not** by itself stop a concurrent Flux writer that also declares `replicas` (a non-force writer surfaces an SSA conflict, and a `spec.force: true` writer can seize ownership — see Ownership). +The first revision of this proposal proposed a bespoke `db-autoscaler` operator that owned the application's `replicas` value and enforced that ownership against competing writers. An implementation spike (see [Findings from the implementation spike](#findings-from-the-implementation-spike)) disproved the enforcement premise that design rested on, and surfaced that the same outcome is reachable far more cheaply by reusing the platform Kubernetes already ships. **This revision therefore builds on the stock `HorizontalPodAutoscaler` (HPA) acting on the engine operator's `scale` subresource, combined with a one-line chart change so the autoscaled field is no longer declared in Git.** The only net-new component is a thin, engine-aware guard that adds the database-specific safety brakes HPA does not have (replication-lag gate, synchronous-quorum floor, recommendation/dry-run). + +The proposal is deliberately scoped to **horizontal scaling of read replicas**, because a stateful database primary cannot be scaled horizontally the way a stateless Deployment can. ## Scope and related proposals @@ -18,190 +20,187 @@ This proposal covers **horizontal** autoscaling (read replicas) only. Two siblin - **Vertical autoscaling** — stepping the `resourcesPreset` ladder / in-place pod resize. - **Storage autoscaling** — automatic PVC expansion when a volume fills up. -Write-path scaling that requires data rebalancing (Kafka broker addition with partition reassignment, ClickHouse/MongoDB sharding) is out of scope for this proposal — it is an orchestrated procedure, not a counter change. +Write-path scaling that requires data rebalancing (Kafka broker addition with partition reassignment, ClickHouse/MongoDB sharding) is out of scope — it is an orchestrated procedure, not a counter change. + +**Engine scope of the MVP.** The HPA-on-`scale`-subresource mechanism applies to engines whose operator CR exposes a `scale` subresource: PostgreSQL (CloudNativePG `Cluster.spec.instances`) and MariaDB (`MariaDB.spec.replicas`). The MVP ships **PostgreSQL**; MariaDB follows once its cozystack chart supports on-the-fly scale-out (today it does not, see [Failure and edge cases](#failure-and-edge-cases)). **Redis (spotahome RedisFailover) and MongoDB (Percona) expose no `scale` subresource**, so they cannot be driven by a stock HPA; they are deferred to a follow-up that adds a thin actuation shim for them (see [Alternatives considered](#alternatives-considered)). ## Context -A managed database in Cozystack is an `Application` in the aggregated `apps.cozystack.io` API. That `Application` is a **pure projection of a Flux `HelmRelease`**: `pkg/registry/apps/application/rest.go` converts both ways (`ConvertApplicationToHelmRelease` sets `Values: app.Spec`, and `ConvertHelmReleaseToApplication` does the reverse), with no separate backing store. Flux reconciles the `HelmRelease` values into the engine operator's custom resource (for example a CloudNativePG `Cluster`, where `packages/apps/postgres/templates/db.yaml` maps `instances: {{ .Values.replicas }}`). Every managed database already exposes a horizontal knob in its values — `replicas` (or `kafka.replicas`, etc.) — and Cozystack already runs the observability the autoscaler needs: +A managed database in Cozystack is an `Application` in the aggregated `apps.cozystack.io` API. That `Application` is a **pure projection of a Flux `HelmRelease`**: `pkg/registry/apps/application/rest.go` converts both ways, with no separate backing store. Flux reconciles the `HelmRelease` values into the engine operator's custom resource — for example a CloudNativePG `Cluster`, where `packages/apps/postgres/templates/db.yaml` maps `instances: {{ .Values.replicas }}`. Cozystack already runs the observability the autoscaler needs: -- A per-database `WorkloadMonitor` (`cozystack.io/v1alpha1`, reconciled by `internal/controller/workloadmonitor_controller.go`) reports `status.availableReplicas`, `status.observedReplicas`, and `status.operational`, and already queries VictoriaMetrics over the vmselect Prometheus API. -- Managed-app pods are labeled by the lineage webhook (`internal/lineagecontrollerwebhook/webhook.go`) with `apps.cozystack.io/application.{group,kind,name}` and `internal.cozystack.io/managed-by-cozystack: "true"`. -- VictoriaMetrics (`packages/system/monitoring`) scrapes per-database metrics via `PodMonitor` (for PostgreSQL, `enablePodMonitor: true` on the CNPG `Cluster`). +- A per-database `WorkloadMonitor` (`cozystack.io/v1alpha1`, reconciled by `internal/controller/workloadmonitor_controller.go`) reports `status.availableReplicas`, `status.observedReplicas`, and `status.operational`. +- Managed-app pods are labeled by the lineage webhook (`internal/lineagecontrollerwebhook/webhook.go`) with `apps.cozystack.io/application.{group,kind,name}` and by kube-state-metrics' `kube_pod_labels`, so metric queries can be scoped to a single application's read-serving pods. +- VictoriaMetrics (`packages/system/monitoring`) scrapes per-database metrics; for PostgreSQL, `enablePodMonitor: true` on the CNPG `Cluster` exports `cnpg_*` series, including the replication-lag gauge. ### The problem > "My database is saturated with read traffic during business hours and idle at night, but I have to notice it, hand-edit `replicas`, and hope I picked the right number — and undo it later." -There is no automated way to add or remove read replicas under load. A stock `HorizontalPodAutoscaler` does not fit: it only writes a replica count (for CloudNativePG, `Cluster.spec.instances`) and is blind to database topology. It has nothing to encode the synchronous-commit quorum floor, so it can drive the count below `maxSyncReplicas + 1` — where the operator either rejects the change or loses its write quorum — and it has no gate on replication lag, scaling on the load metric alone while standbys are arbitrarily behind. Which instance to add or remove, and in what order, is the engine operator's decision (CloudNativePG removes the highest-ordinal standby and never the primary); an autoscaler for stateful databases must own the count and the safety guardrails while leaving instance lifecycle to the operator. - -## Goals +There is no automated way to add or remove read replicas under load. A stock HPA is the natural fit for the *decision* — it computes a desired replica count from a metric with stabilization, min/max, and multi-metric semantics — but on its own it is missing two database-specific safety properties: it has no synchronous-commit quorum floor (it can drive the count below `maxSyncReplicas + 1`, where CNPG rejects the change or starves commits), and no replication-lag gate (it would scale on the load metric alone while standbys are arbitrarily behind). This proposal keeps HPA as the decision engine and adds exactly those two brakes — nothing more. -- Automatically scale the number of read replicas for primary-replica engines: PostgreSQL (CNPG), MariaDB, Redis, MongoDB (replica set). -- Apply all decisions by patching the `Application`'s `replicas` value (`spec`) — the Flux-compatible, tenant-facing write path — never the operator CR directly. -- Reuse existing telemetry (VictoriaMetrics + `WorkloadMonitor`); introduce no new exporters. -- Be safe for stateful workloads: respect the replica quorum, brake on replication lag, hand scale-down to the engine operator's graceful instance removal, use long stabilization windows, and honor tenant quotas. -- Provide HPA-like observability: status conditions, events, and a `dryRun` mode. +### Findings from the implementation spike -### Non-goals +The first design rested on one load-bearing claim: the autoscaler could be the *enforced* single owner of the application's `replicas` value, writing it through the aggregated apps API. Building it disproved that claim, step by step. These findings are what motivate the mechanism change in this revision: -- Vertical scaling (resources / presets). -- Autoscaling the write path / the primary. -- Engines that require data rebalancing (Kafka brokers, ClickHouse/MongoDB shards). -- Cluster-node autoscaling (that is cluster-autoscaler's job). +1. **SSA field-level ownership does not hold on the aggregated apps API.** The `Application` spec is an opaque JSON blob and its managed-fields are not round-tripped, so a dedicated field manager cannot claim `.spec.replicas` (`internal/dbautoscaler/reconciler.go` `patchReplicas`). The Open question the first revision flagged — "does the aggregated Patch handler support per-field SSA at all?" — is answered: **no**. +2. **Admission webhooks cannot fire on the aggregated API.** kube-apiserver proxies aggregated-API requests to the extension server, where admission does not run. Enforcement therefore had to move to the backing Flux `HelmRelease`, a CRD served by kube-apiserver. +3. **The HelmRelease webhook is neither cheap nor sufficient.** It must intercept HelmRelease UPDATEs to guard `replicas`; it must allowlist the apps-API extension-server ServiceAccount (or every legitimate tenant edit breaks), which means a tenant edit through the apps API *bypasses* the guard; and it must not hard-fail Flux reconciliation during an outage. What remains is *advisory* ownership plus a platform-wide admission hop — not the enforced guarantee the design promised. +4. **The root cause is self-imposed.** The autoscaler-vs-Flux conflict exists only because our own chart *unconditionally* templates the replica field (`instances: {{ .Values.replicas }}`). Remove that declaration under autoscaling and there is nothing for Flux and the autoscaler to fight over — the entire ownership problem disappears, which is the basis for this revision. ## Design ### 1. Replica model (instances vs read replicas) -The `replicas` value is the **total instance count** of the engine, not the read-replica count. For CNPG, `packages/apps/postgres/templates/db.yaml` sets `instances: {{ .Values.replicas }}` — that is `1` primary plus `replicas − 1` standbys, and read traffic is served only by the standbys via the `-ro` endpoint. The autoscaler therefore separates the two counts explicitly through the adapter's `PrimaryCount()` (CNPG returns `1`): +Unchanged from the first revision, and still relevant because HPA scales the **total** instance count. For CNPG, `instances` is `1` primary plus `replicas − 1` standbys, and read traffic is served only by the standbys via the `-ro` endpoint. The load metric is averaged over the read-serving replicas only: -- read-serving replicas now: `Rcur = currentReplicas − PrimaryCount` -- `desiredRead = ceil(Rcur × currentMetric / targetMetric)` (metric averaged over read-serving replicas only, i.e. divided by `replicas − 1`, never by the total). `targetMetric` must be strictly positive — enforced by CRD schema (`exclusiveMinimum: 0`) and re-checked in the controller, so a zero or negative target is rejected before the division; `Rcur ≥ 1` always holds because `minReplicas ≥ 2`. -- `desiredReplicas = desiredRead + PrimaryCount` +- read-serving replicas now: `Rcur = currentInstances − primaryCount` (CNPG `primaryCount = 1`) +- `desiredRead = ceil(Rcur × currentMetric / targetMetric)` (metric averaged over standbys, never the total; `targetMetric > 0` enforced) +- `desiredInstances = desiredRead + primaryCount` -`minReplicas`/`maxReplicas` in the CRD count **total instances** (they map to the chart's `replicas` field). `minReplicas` must be `≥ QuorumFloor` and, to serve any reads at all, `≥ 2`. +`minReplicas`/`maxReplicas` on the HPA count **total instances** and map to the engine CR's replica field. `minReplicas` must be `≥ maxSyncReplicas + 1` and `≥ 2` to serve any reads. Because a stock HPA divides its target average by the number of pods matching the target's `scale` selector — which includes the primary — the read-serving metric is emitted **pre-averaged over standbys** by the metrics source (§4), so HPA's own averaging is a no-op and the `Rcur = instances − 1` semantics are preserved. ### 2. Data flow ```mermaid flowchart LR - DHA[DatabaseHorizontalAutoscaler CR] -- watch --> OP[db-autoscaler] - OP -- HTTP /api/v1/query --> VM[(VictoriaMetrics
vmselect, illustrative)] - WM[WorkloadMonitor status] -- operational / availableReplicas --> OP - OP -- patch replicas value --> APP[Application spec
apps.cozystack.io] - APP -- projection --> HR[HelmRelease values] - HR -- Flux --> CR[Engine CR
e.g. CNPG Cluster] - CR --> PODS[(replica pods)] + HPA[HorizontalPodAutoscaler] -- scale subresource --> CR[Engine CR
e.g. CNPG Cluster .spec.instances] + HPA -- external metric --> ADAPTER[metrics adapter] + ADAPTER -- HTTP /api/v1/query --> VM[(VictoriaMetrics)] + GUARD[db-scaling guard] -- reads lag / quorum --> VM + GUARD -- gates min/max on the HPA --> HPA + CR -- managed by operator --> PODS[(replica pods)] + NOTE[chart no longer templates replicas under autoscaling] -.-> CR ``` -### 3. Topology adapters +The engine operator owns instance lifecycle: CNPG adds/removes the highest-ordinal standby gracefully, never the primary, and routes reads through `-ro`. The autoscaler never decides *which* instance to remove. -Engine topology differs, so per-`kind` logic is isolated behind an adapter interface. Only primary-replica engines are scalable; sharded modes return `Scalable=false` with a reason. +### 3. Chart change: stop declaring `replicas` under autoscaling -```go -type TopologyAdapter interface { - ReplicasPath() string // "replicas" for pg/mariadb/redis/mongo - PrimaryCount() int32 // CNPG: 1 (non-read-serving instances) - QuorumFloor(appValues map[string]any) int32 // CNPG: quorum.maxSyncReplicas + 1 - DriverQuery(app types.NamespacedName, k DriverKind) string // PromQL for read load (per read replica) - ReplicationLagQuery(app types.NamespacedName) string // e.g. cnpg_pg_replication_lag gauge; write-activity gated - Scalable(appValues map[string]any) (bool, reason string) // false for sharded modes -} +The decisive change. Each autoscalable app chart wraps its replica field in a conditional so that, when autoscaling is enabled for that application, the field is **omitted** from the rendered engine CR: + +```yaml +# packages/apps/postgres/templates/db.yaml (illustrative) +spec: +{{- if not .Values.autoscaling.enabled }} + instances: {{ .Values.replicas }} +{{- end }} ``` -MVP ships the `postgres` (CNPG) adapter. Follow-ups: `mariadb`, `redis`, `mongodb` (only when `sharding: false`). `clickhouse`, `kafka`, and sharded `mongodb` report `Scalable=false`. +With the field absent from the HelmRelease values, Flux neither sets nor reverts it, and the HPA is the sole writer of `.spec.instances` via the `scale` subresource. No ownership annotation, no SSA field manager, no admission webhook, and no terminal-freeze conflict handling are needed — they existed only to win a fight this change prevents. + +Migration cost is real and chart-sized, not operator-sized (see [Upgrade and rollback compatibility](#upgrade-and-rollback-compatibility)). + +### 4. Metric source for the HPA + +The HPA consumes read-load metrics (`ReadConnections`, `ReadCPUUtilization`) from VictoriaMetrics through a **custom/external metrics adapter**. To preserve multi-tenant isolation, the platform-managed adapter configuration injects a mandatory namespace/label matcher into every query and rejects any query it cannot constrain — never raw tenant-supplied PromQL (the same rule the first revision established). Each series is pre-aggregated over the target's standby pods so the value HPA reads is already per-read-replica. -### 4. Reconcile loop +### 5. The thin guard (database-specific brakes) -1. Resolve `targetRef` → load the `Application` values and the linked `WorkloadMonitor`. -2. Ask the adapter `Scalable`? If not → set condition `ScalingActive=False(reason)` and stop. -3. If `operational=false` **or** a scale is still in flight (`availableReplicas != replicas`) → freeze (single-flight) and requeue. -4. Query VictoriaMetrics for the driver metric and the replication lag. -5. Compute `desiredReplicas` per the replica model in §1. -6. Apply guardrails (see below): clamp to `[min,max]`, quorum floor, lag brake, stabilization windows, step limit, tenant quota. -7. If `desiredReplicas != currentReplicas` and the decision passes → patch the `Application`'s `replicas` value (server-side apply, see Ownership). Scale-down is handed to the engine operator, which removes the highest-ordinal standby gracefully and stops routing it in `-ro`. -8. Record convergence in `status.lastConvergedReplicas` **only after observing the autoscaler's own `replicas` write** (matched by its field manager / `managed-by` marker and write generation); if a `spec.force: true` GitOps replacement changed `replicas` in flight, do not record that competing value as converged — keep the freeze and requeue. Then update `status`, set `lastScaleTime`, and emit an Event. +The only net-new controller. It does **not** compute desired counts or write the engine CR — HPA does both. It watches the target and adjusts the **HPA's `minReplicas`/`maxReplicas`** (and surfaces status/events) to encode what HPA cannot: -### 5. Guardrails (normative) +- **Quorum floor** — hold `minReplicas ≥ maxSyncReplicas + 1` for CNPG so HPA can never drive the cluster below a safe synchronous quorum. Because `maxSyncReplicas` is tenant-mutable, the guard reconciles the floor into the HPA's `minReplicas` on change; CNPG independently rejects any unsafe count as a backstop. +- **Replication-lag brake** — when `cnpg_pg_replication_lag` exceeds `maxReplicationLagSeconds` **and the primary is actively writing** (write-activity gated off the exported LSN metrics, so an idle primary does not trip it), the guard pins `maxReplicas = currentInstances` to forbid further scale-up until lag recovers. +- **Recommendation / dry-run** — a mode where the guard computes and reports the recommendation (status, events, metrics, alerts) without ever unpinning the HPA, so operators can validate behavior before enabling actuation. -- `min ≤ desired ≤ max`; at most `behavior.*.step` replicas per decision — **except** that reaching the quorum floor overrides the step limit: `desired` may jump straight to `QuorumFloor` in a single decision, since a safe quorum must never be rate-limited. This is not a freeze; the only freeze in this area is `QuorumExceedsQuota`, when the floor also exceeds the tenant quota. -- `desired ≥ QuorumFloor(app)`. For CNPG the floor is `maxSyncReplicas + 1`: the chart documents `maxSyncReplicas` as "must be less than total replicas", and dropping to/below it makes CNPG cap/reject the change and can starve synchronous commits (writes stall). The floor also never leaves fewer than `minSyncReplicas` standbys available. Pin this to the CNPG version cozystack ships, since the sync-replica API changed across versions. -- **Precedence — quota > quorum floor > min/max.** `maxSyncReplicas` is tenant-mutable after the DHA is created, so at runtime `QuorumFloor` (`maxSyncReplicas + 1`) may exceed `minReplicas`/`maxReplicas`, and may even exceed what the tenant quota permits. The resolution order is fixed and unambiguous: (1) the **tenant quota is a hard ceiling and is never exceeded**; (2) subject to that, the **quorum floor wins** over `minReplicas`/`maxReplicas` — `desired` is clamped *up* to the floor (even above `maxReplicas`), never letting `min`/`max` push the cluster below a safe quorum. When these two rules collide irreconcilably — the quorum floor does not fit the quota (raised `maxSyncReplicas` + tight quota) — the operator does **not** patch and freezes with `ScalingLimited=True` reason `QuorumExceedsQuota` (alert), rather than exceeding quota (which would only hit the StuckScaling path) or scaling below a safe quorum. -- **Lag brake:** replication lag above `maxReplicationLagSeconds` forbids both scale-down and scale-up (`AbleToScale=False`). The signal is the CNPG-exported gauge **`cnpg_pg_replication_lag`** (seconds), already scraped into VictoriaMetrics and used by cozystack's own CNPG dashboards and alerts (`dashboards/db/cloudnativepg.json`, `packages/system/postgres-operator/alerts/`), so no custom query is added. Because that seconds value keeps climbing on a write-idle primary, the brake is **write-activity gated**: it is honoured only while the primary's WAL position is advancing (from CNPG's exported current-vs-`replay_lsn` LSN metrics), so an idle primary does not produce a false freeze during the low-load windows scale-down targets. -- **Cooldown / stabilization:** separate windows for scale-up and (longer) scale-down; scale-down only when the signal held for the whole window. -- **Single-flight with convergence deadline:** one change at a time; the next decision only after `operational=true && availableReplicas == replicas`. Because that gate can never clear if a scale-up cannot converge — a new standby rejected by ResourceQuota admission, an unbindable PVC, or an unschedulable pod — a patched change must reach convergence within `behavior.convergenceDeadlineSeconds` (default a small multiple of the scale-up window). On timeout the operator surfaces `AbleToScale=False` with reason `StuckScaling`, alerts, and **rolls `replicas` back to `status.lastConvergedReplicas`**, releasing single-flight so a subsequent scale-down (which may itself relieve the pressure) is not blocked. `status.lastConvergedReplicas` records the last count that reached `availableReplicas == replicas`. Note the tenant-quota pre-check is advisory (a concurrent allocation can consume quota between check and pod creation), so this stuck path is reachable in practice, not just in theory. `lastConvergedReplicas` is initialized from the observed replica count when the DHA first adopts a target (before any scale), and every rollback target is re-validated against the current quorum floor, `maxSyncReplicas`, and tenant quota; if it is unset or no longer safe, the operator freezes without patching rather than rolling back to a stale or unsafe value. -- **Tenant quota:** the new replica count × preset resources must fit the tenant quota; otherwise `ScalingLimited=True`. -- **Fail-safe freeze:** if vmselect is unreachable or the metric is missing, do not scale (never scale blind); alert. -- **dryRun:** decisions are written to status/events but no patch is applied. +Quota is not re-implemented: because HPA scales the engine CR and pod creation passes through the tenant's `ResourceQuota` admission, an over-quota scale-up simply fails to create pods and is reflected in the CR/HPA status — no separate quota pre-check controller is required. ## User-facing changes -A new namespaced CRD, `DatabaseHorizontalAutoscaler` (group `autoscaling.cozystack.io/v1alpha1`), created by a tenant next to their database application: +A tenant enables autoscaling on their database and creates a standard HPA next to it. The database-specific brakes are configured on a small guard resource (or, equivalently, annotations on the HPA — final form decided in implementation): ```yaml -apiVersion: autoscaling.cozystack.io/v1alpha1 -kind: DatabaseHorizontalAutoscaler +# 1. turn off the static replica declaration for this app +apiVersion: apps.cozystack.io/v1alpha1 +kind: Postgres +metadata: { name: db, namespace: tenant-acme } +spec: + autoscaling: { enabled: true } # chart omits instances; HPA owns it +--- +# 2. stock HPA on the engine CR's scale subresource +apiVersion: autoscaling/v2 +kind: HorizontalPodAutoscaler metadata: { name: db, namespace: tenant-acme } spec: - targetRef: { kind: Postgres, name: db } # apiGroup defaults to apps.cozystack.io - minReplicas: 2 # TOTAL instances (primary + standbys); >= 2 to serve reads + scaleTargetRef: { apiVersion: postgresql.cnpg.io/v1, kind: Cluster, name: postgres-db } + minReplicas: 2 maxReplicas: 6 metrics: - - type: ReadConnections # | ReadCPUUtilization (fixed, safe set) - target: { averageValue: "150" } # per read-serving replica + - type: External + external: + metric: { name: cozystack_read_connections, selector: { matchLabels: { app: postgres-db } } } + target: { type: AverageValue, averageValue: "150" } behavior: - scaleUp: { stabilizationWindowSeconds: 300, step: 1 } - scaleDown: { stabilizationWindowSeconds: 1800, step: 1 } - convergenceDeadlineSeconds: 900 # patched scale must converge within this, else StuckScaling + roll back - constraints: - respectQuorum: true - maxReplicationLagSeconds: 30 - gracefulScaleDown: true # operator-native; DHA does not terminate backends + scaleUp: { stabilizationWindowSeconds: 300 } + scaleDown: { stabilizationWindowSeconds: 1800 } +--- +# 3. database-specific brakes (thin guard) +apiVersion: autoscaling.cozystack.io/v1alpha1 +kind: DatabaseScalingPolicy +metadata: { name: db, namespace: tenant-acme } +spec: + targetRef: { kind: Postgres, name: db } + hpaRef: { name: db } + respectQuorum: true + maxReplicationLagSeconds: 30 dryRun: false -status: - currentReplicas: 3 - desiredReplicas: 4 - lastConvergedReplicas: 3 # last count that reached availableReplicas == replicas - lastScaleTime: "..." - currentMetrics: [ { type: ReadConnections, averageValue: "210" } ] - conditions: [ ScalingActive, AbleToScale, ScalingLimited ] # reasons incl. StuckScaling, QuorumExceedsQuota ``` -When several `metrics` are set, the desired count is the **maximum** of the per-metric desired counts (HPA semantics). The dashboard can surface the DHA status and scaling events like it does for other application sub-resources. When no DHA references an application, nothing changes. +When no HPA references an autoscalable app, and `autoscaling.enabled` is false, nothing changes — the chart templates `replicas` exactly as today. ## Upgrade and rollback compatibility -- **Opt-in and off by default.** The operator ships as an optional platform package, enabled via `bundles.enabledPackages`. Existing clusters, manifests, and APIs are unaffected until a tenant creates a DHA. -- **Ownership (enforced, not advisory).** While a DHA is active, the autoscaler is the **single explicit owner** of the application's `replicas` value. Enforcement: the operator writes `replicas` via **server-side apply with a dedicated field manager (`db-autoscaler`)** and stamps a marker annotation `autoscaling.cozystack.io/managed-by: ` on the `Application`. A competing declarative writer (Flux from a tenant GitOps repo, a human edit) that also claims `replicas` produces an SSA field-manager conflict, which the operator surfaces as a `ScalingLimited`/conflict condition rather than silently fighting. `RetryOnConflict` handles only API-level write races on a single writer — it is **not** the ownership mechanism. **This SSA guarantee holds only against writers that do not force-apply:** a tenant GitOps Flux `Kustomization` with `spec.force: true` (a common setting) takes over the `replicas` managed-fields entry, so the autoscaler's next (non-force) apply is the one that hits the conflict — i.e. the autoscaler loses ownership rather than the competitor. Because SSA alone cannot win against a force-applier, a **validating admission webhook** that rejects conflicting `replicas` writes for DHA-managed applications is **recommended** to close this case deterministically (tracked in Open questions), not merely optional. **Caveat — field-level SSA is not yet confirmed for this API:** `Application` is served by a hand-written `rest.Patcher` (`pkg/registry/apps/application/rest.go`), not a CRD, and its existing conflict test (`rest_conflict_test.go`) covers only `RetryOnConflict` on the backing `HelmRelease` resourceVersion — not per-field managed-fields SSA. If the aggregated Patch handler does not track `.spec.replicas` at field granularity, this ownership model and the `lastConvergedReplicas` rollback silently degrade to advisory. A spike against a real API server (see Testing) must confirm field-level SSA before MVP; **if it does not hold, the admission webhook is mandatory, not merely recommended.** -- **Rollback.** Deleting the DHA stops all autoscaling immediately (and clears the marker), leaving the application at its current `replicas`. Disabling the package removes the operator; no data migration is involved and the change is fully reversible. +- **Opt-in and off by default.** The chart conditional is inert unless `autoscaling.enabled` is set; existing clusters and manifests are unaffected. The guard and metrics adapter ship as optional platform packages. +- **Enabling autoscaling on an existing database (the one real migration).** Flipping `autoscaling.enabled` removes `instances`/`replicas` from the rendered CR. Helm's three-way merge deletes a previously-set field on omission, and CNPG defaults to **1 instance** when `.spec.instances` is unset — so a naive flip would collapse a running cluster to a single instance before the HPA raises it. The rollout must therefore either (a) have the HPA (or the guard) set `.spec.instances` to the current count *before* the field is dropped from the chart, or (b) template a floor via the `scale` subresource so the count never dips below `minReplicas`. This migration path must be exercised on a dev cluster before MVP. +- **Cold start.** Until the HPA takes its first sample it holds at `minReplicas`; a brief window at the floor is expected. +- **Dependent objects.** `WorkloadMonitor` and dashboards that read `.Values.replicas` must switch to the observed instance count (`status.instances` / metrics), since the values field is no longer authoritative under autoscaling. +- **Rollback.** Set `autoscaling.enabled: false` (and delete the HPA): the chart resumes templating `replicas` and Flux reconciles it back. Fully reversible; no data migration. ## Security -- **RBAC.** The operator needs: read DHA and read/patch `applications.apps.cozystack.io`; read `workloadmonitors.cozystack.io`; read `pods`; read `resourcequotas` (core) for the tenant-quota guardrail; and read-only HTTP to vmselect. The `resourcesPreset → resources` mapping is a static table compiled into the operator from the published cozy-lib preset ladder, so no cluster read of preset definitions is required. The operator has **no** write access to Pods, Services, or Endpoints, **no** exec, and **no** direct access to engine operator CRs or Flux `HelmRelease` objects — only the aggregated apps API. -- **Multi-tenancy.** DHA is namespaced and lives in the tenant namespace. Tenant access is granted through the platform's RBAC-aggregation mechanism: the DHA package **ships its own self-contained ClusterRoles** labelled `rbac.cozystack.io/aggregate-to-tenant[-view|-admin|-super-admin]: "true"` (per `packages/system/cozystack-basics/templates/clusterroles.yaml`) — it does **not** edit the shared `cozystack-basics` file, whose write tiers grant apps via a hard-coded per-kind allowlist rather than a wildcard. This gives the tenant ServiceAccount full access, human `view` read-only, and `admin`/`super-admin` write on `databasehorizontalautoscalers.autoscaling.cozystack.io`. A tenant can only autoscale its own applications, and scale-up is validated against the tenant quota. -- **Single active reconciler (HA).** Exactly one instance may act at a time — per-target reconcile state (the `managed-by` marker, single-flight, and the convergence rollback to `lastConvergedReplicas`) assumes a single active writer, never active/active. For availability the operator runs **≥2 replicas with controller-runtime leader-election** (active/standby: the leader reconciles, standbys take over on leader loss), plus pod anti-affinity and a PodDisruptionBudget. `replicas: 1` is a minimum-function default that provides **no** HA (no failover); real HA requires the multi-replica leader-election setup above, which is also what prevents two *active* reconcilers from racing on the annotation write and the rollback decision. -- **Bounded inputs.** All tenant-supplied DHA fields are enumerable and schema-validated: `minReplicas`/`maxReplicas` (integers), a fixed `metrics[].type` enum (`ReadConnections`, `ReadCPUUtilization`), numeric targets, and windows. Arbitrary tenant-supplied PromQL is **not** accepted (see Alternatives), so there is no path for a tenant query to read another tenant's series from shared vmselect. No new secrets are stored or transmitted. +- **RBAC (much reduced vs the first revision).** The guard needs: read its own `DatabaseScalingPolicy`; read/update the referenced `HorizontalPodAutoscaler` (`min/maxReplicas`); read `workloadmonitors`; read-only HTTP to vmselect. It needs **no** write access to `Application`/`HelmRelease`, **no** admission webhook, **no** SSA field manager, and **no** engine-CR writes (HPA does that through the `scale` subresource under the tenant's existing RBAC). The metrics adapter is a standard read-only VictoriaMetrics client. +- **Multi-tenancy.** `DatabaseScalingPolicy` and the HPA are namespaced and live in the tenant namespace; the platform ships self-contained aggregated ClusterRoles (`rbac.cozystack.io/aggregate-to-tenant[-view|-admin]`). The metrics adapter injects a mandatory namespace matcher, so no tenant query can read another tenant's series. +- **Blast radius.** Because there is no cluster-wide admission webhook, enabling this feature adds no admission hop to unrelated Flux reconciliation — a key regression from the first design is gone. ## Failure and edge cases -- vmselect unreachable or metric missing → the autoscaler freezes (no scaling) and surfaces `AbleToScale=False`; alert fires. -- Replication lag above the configured threshold **and the primary is actively writing** → no scale-up and no scale-down until lag recovers. An idle primary does not trip the brake (write-activity gating). -- Scale still in flight (`availableReplicas != replicas`) → single-flight; the next decision waits for convergence, preventing thrashing. -- Scale-up patched but never converges (quota-rejected standby, unbindable PVC, unschedulable pod) → after `convergenceDeadlineSeconds` the operator surfaces `AbleToScale=False(StuckScaling)`, alerts, and rolls `replicas` back to `status.lastConvergedReplicas`, so the autoscaler is not frozen and a relieving scale-down can proceed. -- Target is a sharded engine (e.g. ClickHouse, or MongoDB with `sharding: true`) → `ScalingActive=False` with a clear reason; no action. -- Desired count would drop to/below the quorum floor (`maxSyncReplicas + 1`) → clamped to the floor; `ScalingLimited=True`. -- Tenant quota exceeded on scale-up → clamped; `ScalingLimited=True`. -- Quorum floor exceeds the tenant quota (raised `maxSyncReplicas` + tight quota) → no patch; freeze with `ScalingLimited=True(QuorumExceedsQuota)` and alert — quota is never exceeded and quorum is never violated. -- A competing writer claims `replicas` → SSA conflict surfaced as a condition; the autoscaler does not enter a write war. +- vmselect unreachable or metric missing → HPA has no metric and holds the current count (`ScalingActive=False` on the HPA); the guard alerts. No blind scaling. +- Replication lag above threshold **with an actively-writing primary** → guard pins `maxReplicas = current`, forbidding scale-up until lag recovers. An idle primary does not trip the brake (write-activity gating). +- Desired count would drop to/below the quorum floor → guard holds `minReplicas ≥ maxSyncReplicas + 1`; CNPG rejects an unsafe count as a backstop. +- Tenant quota exceeded on scale-up → pods fail `ResourceQuota` admission; the CR/HPA surface the unmet count; no separate freeze path needed. +- MariaDB target whose chart lacks scale-out support (`replication.replica.bootstrapFrom` unset) → the operator rejects on-the-fly scale-out (`MariaDBScaleOutError`); MariaDB stays out of the enabled set until the chart is fixed. +- Redis / MongoDB target → no `scale` subresource; rejected by the guard with a clear reason (deferred to the shim follow-up). +- Sharded engine (ClickHouse, sharded MongoDB) → out of scope; not autoscalable. ## Testing -- **Unit:** reconcile decisions and each `TopologyAdapter` (including the `PrimaryCount`/`replicas − 1` math and `QuorumFloor = maxSyncReplicas + 1`) with mocked VictoriaMetrics and a mocked Application client (`go test ./internal/controller/...`). -- **Codegen:** `make generate` produces the CRD and deepcopy without errors. -- **envtest (apiserver-backed):** the ownership path is exercised against a real API server, since server-side-apply managed-fields conflict semantics do not exist with a mocked client. A competing writer claims `replicas` both without force and with `force: true`, and the test asserts the `autoscaling.cozystack.io/managed-by` marker, the surfaced conflict condition, and the absence of scaling thrash. This is the only layer that actually verifies the ownership guarantee — the mocked unit tests above cannot. The spike also confirms whether the aggregated `apps.cozystack.io` Patch handler tracks `.spec.replicas` managed-fields **at all** (see the Ownership caveat); if it does not, the ownership guarantee falls back to the admission webhook. -- **Manual (dev cluster, CNPG postgres):** create a DHA with `dryRun: true` → decisions appear in `status`/Events, replicas unchanged. Then disable `dryRun` under read load → the `Application`'s `replicas` grows, CNPG adds a standby, reads route to `-ro`, and Flux does not revert; on load decrease and after the window, scale-down removes a standby gracefully and never drops to/below `maxSyncReplicas + 1`. -- **Negative:** vmselect down → freeze; lag above threshold with active writes → no scaling; idle primary with high lag-seconds → no false freeze; DHA targeting a sharded ClickHouse → `ScalingActive=False`; concurrent GitOps write to `replicas` → SSA conflict condition, no thrash (covered by the envtest above). +- **Unit:** the replica-model math and each engine's quorum/lag logic in the guard, with mocked VictoriaMetrics. +- **Chart:** `helm template` with `autoscaling.enabled: true` omits the replica field; with it false, renders `replicas` exactly as today (regression guard). +- **Migration (dev cluster, CNPG):** flip `autoscaling.enabled` on a running multi-instance cluster and assert it does **not** collapse to 1 instance (the pre-set-before-omit path), then drive load and confirm HPA scales `.spec.instances`, reads route to `-ro`, and Flux does not revert. This replaces the first revision's force-writer ownership envtest, which is no longer meaningful because there is no ownership to enforce. +- **Guard integration:** lag above threshold with active writes pins `maxReplicas`; quorum floor tracks `maxSyncReplicas` changes; `dryRun` reports without pinning. +- **Negative:** vmselect down → no scaling; idle primary with high lag-seconds → no false brake; MariaDB without scale-out → rejected; Redis → rejected. ## Rollout -1. **PoC** — CNPG PostgreSQL on a dev cluster: DHA + `replicas` patch driven by `ReadConnections`; confirm Flux does not revert and reads route to `-ro`. -2. **MVP** — the operator plus the `postgres` adapter, full guardrails (quorum, lag, cooldown, quota), `dryRun`, dashboard surface and events. Shipped as an optional `paas`-bundle package that declares a hard `PackageSource.dependsOn`-class dependency on the monitoring stack (VictoriaMetrics/vmselect + `WorkloadMonitor`) — the decision loop cannot function without it, the same cold-install ordering the platform already applies to cert-manager-dependent charts. The operator Deployment runs **≥2 replicas with controller-runtime leader-election** (active/standby, plus pod anti-affinity and a PodDisruptionBudget) so exactly one instance is active at a time — no active/active race on the `managed-by` annotation, single-flight, or the `lastConvergedReplicas` rollback, while still surviving a node/pod failure. -3. **Adapter expansion** — `mariadb` → `redis` → `mongodb` (replica set). -4. **Observability & policy** — Grafana dashboard of scaling decisions, alerts for "limit reached / freeze". +1. **PoC** — CNPG on a dev cluster: chart conditional + a stock HPA on `.spec.instances` driven by `ReadConnections`; confirm Flux does not revert and reads route to `-ro`. +2. **MVP** — PostgreSQL: the chart change, the metrics adapter (namespace-scoped), the thin guard (quorum floor + lag brake + dry-run), dashboard surface and alerts. Optional packages with a hard dependency on the monitoring stack. +3. **MariaDB** — once the cozystack mariadb chart supports on-the-fly scale-out. +4. **Redis / MongoDB** — a follow-up proposal for a thin actuation shim, since neither exposes a `scale` subresource. ## Open questions -- Adapter order after MVP: `mariadb` → `redis` → `mongodb`? -- Default driver metric: read connections, read QPS, or replica CPU (to be calibrated on real workloads)? -- Is scale-down enabled by default, or scale-up only (down conservative/manual)? -- Ownership enforcement: **does the aggregated `apps.cozystack.io` Patch handler support per-field managed-fields SSA at all** (spike required — see Testing)? If not, the SSA field-manager + marker is insufficient and the validating admission webhook that hard-rejects conflicting `replicas` writes for DHA-managed applications becomes mandatory (it is also the only thing that beats a `spec.force: true` writer). +- Final form of the database-specific brakes: a small `DatabaseScalingPolicy` CRD, or annotations on the HPA? The former is clearer; the latter avoids any new API surface (see Alternative 1). +- Which metrics adapter — prometheus-adapter, a KEDA `ScaledObject` external trigger, or a small purpose-built adapter — best fits VictoriaMetrics with mandatory namespace scoping? +- Migration mechanic for enabling autoscaling on a live cluster: set `.spec.instances` before omitting the chart field, or floor it via the `scale` subresource? +- Default driver metric (read connections vs read QPS vs replica CPU), to be calibrated on real workloads. ## Alternatives considered -- **A controller inside `cozystack-controller`** instead of a standalone operator. It would reuse the existing binary, RBAC, and VictoriaMetrics helper, at the cost of coupling the autoscaler's lifecycle to the platform controller. Rejected in favor of a standalone operator for isolation and an independent release cadence; the logic can be moved later if desired. -- **Patching the engine CR / `HelmRelease` directly.** A direct patch to the operator CR is reverted by Flux. Patching `HelmRelease` values directly is possible but bypasses the supported surface. Note that the `Application` is a pure projection of the `HelmRelease` (rest.go converts both ways, `Values: app.Spec`), so patching the `Application`'s `replicas` **is** writing the same `HelmRelease` values — there is no background regeneration that would clobber it. The reasons to prefer the apps API are validation, label/lineage management, and it being the supported tenant surface, not clobber-avoidance. -- **Stock HPA + KEDA.** Rejected as the primary mechanism: it only writes a replica count and is topology-unaware — no synchronous-commit quorum floor, no replication-lag gate, and it cannot express which instance to remove (that is the engine operator's job). A KEDA/PromQL-style trigger could be reused *as a metric source inside* the operator only if the operator injects a mandatory tenant/namespace label matcher into every query and rejects any query it cannot constrain — never as raw tenant-supplied PromQL against shared vmselect. -- **Scaling the write path via sharding.** Out of scope: it requires data rebalancing (Cruise Control for Kafka, resharding for ClickHouse/MongoDB), which is an orchestrated procedure rather than a replica-count change. +- **A bespoke `db-autoscaler` operator owning `replicas` (the first revision of this proposal).** Rejected after the implementation spike. It required re-drawing HPA's API surface field-for-field (`metrics[].target.averageValue`, `behavior.*.stabilizationWindowSeconds`, the `ScalingActive`/`AbleToScale`/`ScalingLimited` conditions) and re-implementing HPA's decision loop; and its ownership guarantee proved unbuildable on the aggregated apps API ([Findings](#findings-from-the-implementation-spike)) — SSA does not hold, admission cannot fire there, and the fallback HelmRelease webhook is advisory, bypassable by tenant edits, and a platform-wide admission hop. The present design keeps HPA's hardened decision loop and confines net-new code to the two brakes HPA genuinely lacks. +- **HPA writing the `Application`'s `replicas` value (apps API) instead of the engine CR.** This is what the first revision did; it is the source of the whole ownership problem, because the apps values are declared in Git and reverted by Flux. Writing the engine CR's `scale` subresource while the chart omits the field (this design) avoids the conflict at its root. +- **Not creating any new API at all.** The brakes could be expressed as annotations on a stock HPA rather than a `DatabaseScalingPolicy` CRD, eliminating the new API group entirely. Kept as an open question; the CRD is proposed only for clarity, not necessity. +- **A thin actuation shim for engines without a `scale` subresource (Redis, MongoDB).** For these, HPA cannot act directly. A minimal shim that watches a stock HPA's recommendation and propagates it behind the same brakes is the honest path — deferred to a follow-up, since the MVP targets engines that already have a `scale` subresource. +- **Stock HPA + KEDA with tenant-supplied PromQL.** Rejected for the metric-source layer: raw tenant PromQL against shared vmselect breaks tenant isolation. A KEDA/prometheus-adapter trigger is acceptable **only** with a platform-injected, mandatory namespace matcher — the constraint carried over from the first revision. +- **Scaling the write path via sharding.** Out of scope: it requires data rebalancing, an orchestrated procedure rather than a replica-count change. --- From 7e51ebe55a8fab7d8b079a4f18e28c6cec9399c2 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Wed, 29 Jul 2026 15:30:51 +0300 Subject: [PATCH 2/5] =?UTF-8?q?docs(dha):=20address=20review=20=E2=80=94?= =?UTF-8?q?=20HPA=20metric=20encoding,=20guard-owned=20HPA,=20scale-down?= =?UTF-8?q?=20pacing?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Rework the seams flagged in PR #44 review (lllamnyp, IvanHunters): - Metric arithmetic (§1/§4): drop the incorrect 'pre-averaged / External AverageValue' framing. Use a Custom (Pods) metric where each standby reports its read load and the primary reports exactly the target, so stock HPA computes desired = ceil(N*avg/target) = desiredRead + primaryCount. Worked example included; Custom-vs-External made explicit. - HPA ownership (§5): the DatabaseScalingPolicy guard now renders and owns the HPA; the tenant declares only the policy. Removes the min/maxReplicas revert war, the tenant-HPA-RBAC gap, and settles CRD-vs-annotations. - Lag brake moved to the metric layer (clamp series to target while lag high + primary writing) so it freezes both directions, with hysteresis. - Pin scale-down policies (Pods/1 per ~600s) to restore step-of-1 pacing. - Custom-metrics adapter promoted to a decided section (new shared infra; no custom.metrics.k8s.io served today). Two-phase migration mechanism (Helm three-way merge deletes the omitted field regardless of writer). - Add provisioning-latency + feedback-loop, minReplicas>=2 footprint, read-disruption, and quota-alert notes. Lead with the design; move the spike findings to an appendix. - Nits: lineage-label selector, WorkloadMonitor vs engine .status.instances, drop redis/mongo from the overview framing. Signed-off-by: Alexey Artamonov --- .../database-horizontal-autoscaling/README.md | 215 ++++++++++-------- 1 file changed, 115 insertions(+), 100 deletions(-) diff --git a/design-proposals/database-horizontal-autoscaling/README.md b/design-proposals/database-horizontal-autoscaling/README.md index ed21bc2..6be9e8b 100644 --- a/design-proposals/database-horizontal-autoscaling/README.md +++ b/design-proposals/database-horizontal-autoscaling/README.md @@ -2,81 +2,69 @@ - **Title:** `Database Horizontal Autoscaler for Cozystack` - **Author(s):** `@scooby87` -- **Date:** `2026-07-08`; revised `2026-07-24` after the implementation spike, addressing review by `@IvanHunters`, `@lllamnyp`, Gemini, and CodeRabbit +- **Date:** `2026-07-08`; revised `2026-07-24` (mechanism) and `2026-07-29` (addressing @lllamnyp and @IvanHunters review on PR #44), with earlier review by @IvanHunters, Gemini, and CodeRabbit - **Status:** Draft ## Overview -Managed databases in Cozystack (`postgres`, `mariadb`, `redis`, `mongodb`, and others) are scaled only manually today: an operator edits the `replicas` value of the application and waits for the underlying operator to converge. This proposal introduces automatic horizontal scaling of a managed database's **read replicas** in response to load. +This proposal adds automatic horizontal scaling of a managed database's **read replicas** in response to load, using **the stock Kubernetes `HorizontalPodAutoscaler` (HPA) acting on the engine operator's `scale` subresource**, plus a one-line chart change so the replica field is no longer declared in Git, plus a thin engine-aware controller — the **`DatabaseScalingPolicy` guard** — that renders the HPA, encodes the two database-specific brakes HPA lacks (synchronous-quorum floor and replication-lag gate), and drives a custom metric that makes stock HPA arithmetic compute the correct read-replica count. -The first revision of this proposal proposed a bespoke `db-autoscaler` operator that owned the application's `replicas` value and enforced that ownership against competing writers. An implementation spike (see [Findings from the implementation spike](#findings-from-the-implementation-spike)) disproved the enforcement premise that design rested on, and surfaced that the same outcome is reachable far more cheaply by reusing the platform Kubernetes already ships. **This revision therefore builds on the stock `HorizontalPodAutoscaler` (HPA) acting on the engine operator's `scale` subresource, combined with a one-line chart change so the autoscaled field is no longer declared in Git.** The only net-new component is a thin, engine-aware guard that adds the database-specific safety brakes HPA does not have (replication-lag gate, synchronous-quorum floor, recommendation/dry-run). +The proposal is deliberately scoped to **horizontal scaling of read replicas**: a stateful primary cannot be scaled horizontally the way a stateless Deployment can. The MVP targets **PostgreSQL (CloudNativePG)**; see [Scope](#scope-and-related-proposals) for the engine ladder. -The proposal is deliberately scoped to **horizontal scaling of read replicas**, because a stateful database primary cannot be scaled horizontally the way a stateless Deployment can. +### Why this changed -## Scope and related proposals - -This proposal covers **horizontal** autoscaling (read replicas) only. Two sibling axes are explicitly deferred to separate proposals: +An earlier revision proposed a bespoke `db-autoscaler` operator that owned the application's `replicas` value and enforced that ownership against competing writers. An implementation spike disproved the enforcement premise it rested on — SSA field ownership does not hold on the aggregated apps API, admission webhooks cannot fire there, and the fallback HelmRelease webhook is advisory, bypassable, and platform-wide. The spike also showed the whole conflict is self-imposed: it exists only because our own chart unconditionally templates the replica field, so removing that declaration under autoscaling makes the ownership problem disappear rather than needing to be enforced. The full spike findings are preserved in the [Appendix](#appendix-findings-from-the-implementation-spike); this revision builds on their conclusion — reuse HPA, do not reimplement it. -- **Vertical autoscaling** — stepping the `resourcesPreset` ladder / in-place pod resize. -- **Storage autoscaling** — automatic PVC expansion when a volume fills up. +## Scope and related proposals -Write-path scaling that requires data rebalancing (Kafka broker addition with partition reassignment, ClickHouse/MongoDB sharding) is out of scope — it is an orchestrated procedure, not a counter change. +This proposal covers **horizontal** autoscaling (read replicas) only. Two sibling axes are deferred to separate proposals: **vertical autoscaling** (stepping the `resourcesPreset` ladder / in-place resize) and **storage autoscaling** (automatic PVC expansion). Write-path scaling that requires data rebalancing (Kafka broker addition, ClickHouse/MongoDB sharding) is out of scope — it is an orchestrated procedure, not a counter change. -**Engine scope of the MVP.** The HPA-on-`scale`-subresource mechanism applies to engines whose operator CR exposes a `scale` subresource: PostgreSQL (CloudNativePG `Cluster.spec.instances`) and MariaDB (`MariaDB.spec.replicas`). The MVP ships **PostgreSQL**; MariaDB follows once its cozystack chart supports on-the-fly scale-out (today it does not, see [Failure and edge cases](#failure-and-edge-cases)). **Redis (spotahome RedisFailover) and MongoDB (Percona) expose no `scale` subresource**, so they cannot be driven by a stock HPA; they are deferred to a follow-up that adds a thin actuation shim for them (see [Alternatives considered](#alternatives-considered)). +**Engine scope of the MVP.** The HPA-on-`scale`-subresource mechanism applies to engines whose operator CR exposes a `scale` subresource: PostgreSQL (CloudNativePG `Cluster.spec.instances`) and MariaDB (`MariaDB.spec.replicas`). The MVP ships **PostgreSQL**; MariaDB follows once its cozystack chart supports on-the-fly scale-out (today it does not — see [Failure and edge cases](#failure-and-edge-cases)). **Redis (spotahome RedisFailover) and MongoDB (Percona) expose no `scale` subresource**, so a stock HPA cannot drive them; they are deferred to a follow-up that adds a thin actuation shim (see [Alternatives considered](#alternatives-considered)). ## Context -A managed database in Cozystack is an `Application` in the aggregated `apps.cozystack.io` API. That `Application` is a **pure projection of a Flux `HelmRelease`**: `pkg/registry/apps/application/rest.go` converts both ways, with no separate backing store. Flux reconciles the `HelmRelease` values into the engine operator's custom resource — for example a CloudNativePG `Cluster`, where `packages/apps/postgres/templates/db.yaml` maps `instances: {{ .Values.replicas }}`. Cozystack already runs the observability the autoscaler needs: - -- A per-database `WorkloadMonitor` (`cozystack.io/v1alpha1`, reconciled by `internal/controller/workloadmonitor_controller.go`) reports `status.availableReplicas`, `status.observedReplicas`, and `status.operational`. -- Managed-app pods are labeled by the lineage webhook (`internal/lineagecontrollerwebhook/webhook.go`) with `apps.cozystack.io/application.{group,kind,name}` and by kube-state-metrics' `kube_pod_labels`, so metric queries can be scoped to a single application's read-serving pods. -- VictoriaMetrics (`packages/system/monitoring`) scrapes per-database metrics; for PostgreSQL, `enablePodMonitor: true` on the CNPG `Cluster` exports `cnpg_*` series, including the replication-lag gauge. - -### The problem +A managed database in Cozystack is an `Application` in the aggregated `apps.cozystack.io` API — a **pure projection of a Flux `HelmRelease`** (`pkg/registry/apps/application/rest.go` converts both ways, no separate backing store). Flux reconciles the `HelmRelease` values into the engine operator's CR — for CNPG a `Cluster`, where `packages/apps/postgres/templates/db.yaml` maps `instances: {{ .Values.replicas }}`. Cozystack already runs the observability the autoscaler needs: -> "My database is saturated with read traffic during business hours and idle at night, but I have to notice it, hand-edit `replicas`, and hope I picked the right number — and undo it later." - -There is no automated way to add or remove read replicas under load. A stock HPA is the natural fit for the *decision* — it computes a desired replica count from a metric with stabilization, min/max, and multi-metric semantics — but on its own it is missing two database-specific safety properties: it has no synchronous-commit quorum floor (it can drive the count below `maxSyncReplicas + 1`, where CNPG rejects the change or starves commits), and no replication-lag gate (it would scale on the load metric alone while standbys are arbitrarily behind). This proposal keeps HPA as the decision engine and adds exactly those two brakes — nothing more. - -### Findings from the implementation spike - -The first design rested on one load-bearing claim: the autoscaler could be the *enforced* single owner of the application's `replicas` value, writing it through the aggregated apps API. Building it disproved that claim, step by step. These findings are what motivate the mechanism change in this revision: - -1. **SSA field-level ownership does not hold on the aggregated apps API.** The `Application` spec is an opaque JSON blob and its managed-fields are not round-tripped, so a dedicated field manager cannot claim `.spec.replicas` (`internal/dbautoscaler/reconciler.go` `patchReplicas`). The Open question the first revision flagged — "does the aggregated Patch handler support per-field SSA at all?" — is answered: **no**. -2. **Admission webhooks cannot fire on the aggregated API.** kube-apiserver proxies aggregated-API requests to the extension server, where admission does not run. Enforcement therefore had to move to the backing Flux `HelmRelease`, a CRD served by kube-apiserver. -3. **The HelmRelease webhook is neither cheap nor sufficient.** It must intercept HelmRelease UPDATEs to guard `replicas`; it must allowlist the apps-API extension-server ServiceAccount (or every legitimate tenant edit breaks), which means a tenant edit through the apps API *bypasses* the guard; and it must not hard-fail Flux reconciliation during an outage. What remains is *advisory* ownership plus a platform-wide admission hop — not the enforced guarantee the design promised. -4. **The root cause is self-imposed.** The autoscaler-vs-Flux conflict exists only because our own chart *unconditionally* templates the replica field (`instances: {{ .Values.replicas }}`). Remove that declaration under autoscaling and there is nothing for Flux and the autoscaler to fight over — the entire ownership problem disappears, which is the basis for this revision. +- A per-database `WorkloadMonitor` (`cozystack.io/v1alpha1`) reports `status.availableReplicas`, `status.observedReplicas`, and `status.operational`. +- Managed-app pods carry the lineage labels `apps.cozystack.io/application.{group,kind,name}` (via `internal/lineagecontrollerwebhook/webhook.go`), and kube-state-metrics exports `kube_pod_labels` (including CNPG's `cnpg.io/instanceRole` as `label_cnpg_io_instance_role`), so a metric can be scoped to one application's read-serving pods and to the standby role. +- VictoriaMetrics (`packages/system/monitoring`) scrapes per-database metrics; for PostgreSQL `enablePodMonitor: true` exports `cnpg_*` series, including the replication-lag gauge. vmselect is reachable at `vmselect-..svc:8481/select/0/prometheus`. ## Design -### 1. Replica model (instances vs read replicas) +### 1. Replica model and metric encoding -Unchanged from the first revision, and still relevant because HPA scales the **total** instance count. For CNPG, `instances` is `1` primary plus `replicas − 1` standbys, and read traffic is served only by the standbys via the `-ro` endpoint. The load metric is averaged over the read-serving replicas only: +The engine's total instance count is `1` primary plus `replicas − 1` standbys; read traffic is served only by the standbys via `-ro`. The autoscaling target is per read-serving replica: - read-serving replicas now: `Rcur = currentInstances − primaryCount` (CNPG `primaryCount = 1`) -- `desiredRead = ceil(Rcur × currentMetric / targetMetric)` (metric averaged over standbys, never the total; `targetMetric > 0` enforced) +- `desiredRead = ceil(Σ readLoad over standbys / targetPerStandby)` - `desiredInstances = desiredRead + primaryCount` -`minReplicas`/`maxReplicas` on the HPA count **total instances** and map to the engine CR's replica field. `minReplicas` must be `≥ maxSyncReplicas + 1` and `≥ 2` to serve any reads. Because a stock HPA divides its target average by the number of pods matching the target's `scale` selector — which includes the primary — the read-serving metric is emitted **pre-averaged over standbys** by the metrics source (§4), so HPA's own averaging is a no-op and the `Rcur = instances − 1` semantics are preserved. +A stock HPA has no `+ primaryCount` term and no notion of "standbys only" — for a metric it just computes a desired count. The choice of metric *type* is therefore the formula, and the two options are not interchangeable: an **External** metric is a single free-standing value (`desired = ceil(value / target)`, no pod divisor), whereas a **Custom (Pods)** metric (`custom.metrics.k8s.io`, `type: Pods`) is averaged by HPA over the scale target's pods (`desired = ceil(currentPods × avg / target)`). We use the **Custom (Pods)** encoding and synthesize the series so unmodified HPA arithmetic reproduces the model exactly: + +> Each **standby** pod reports its own read load `Lᵢ`; the **primary** pod reports **exactly `targetPerStandby`**. With `N = currentInstances` pods, HPA computes `desired = ceil(N × avg / target) = ceil((target + ΣLᵢ) / target) = 1 + ceil(ΣLᵢ / target) = primaryCount + desiredRead`. + +The `+1` for the primary and the "divide by standbys only" both fall out of the primary reporting the target value — no controller math, no external-metric offset hacks. Worked example, `target = 150` active read connections per standby, a 3-instance cluster (1 primary + 2 standbys): at `ΣLᵢ = 210` → `avg = (150+210)/3 = 120`, `desired = ceil(3×120/150) = ceil(2.4) = 3` (holds); at `ΣLᵢ = 600` → `avg = 250`, `desired = ceil(750/150) = 5` (scales up). Validating this encoding end-to-end against a real HPA is the first thing the PoC must do. + +The two MVP metrics are the same read-load signals the platform already scrapes: active read connections (`cnpg_backends_total{state="active"}`) and read-path CPU (`rate(container_cpu_usage_seconds_total{container="postgres"}[5m])`), each joined to the standby role through `kube_pod_labels{label_cnpg_io_instance_role="replica"}`. ### 2. Data flow ```mermaid flowchart LR - HPA[HorizontalPodAutoscaler] -- scale subresource --> CR[Engine CR
e.g. CNPG Cluster .spec.instances] - HPA -- external metric --> ADAPTER[metrics adapter] - ADAPTER -- HTTP /api/v1/query --> VM[(VictoriaMetrics)] - GUARD[db-scaling guard] -- reads lag / quorum --> VM - GUARD -- gates min/max on the HPA --> HPA + DSP[DatabaseScalingPolicy CR
tenant-declared] -- watch --> GUARD[db-scaling guard] + GUARD -- renders + owns --> HPA[HorizontalPodAutoscaler] + HPA -- custom metric --> ADAPTER[custom-metrics adapter] + ADAPTER -- HTTP /select/0/prometheus --> VM[(VictoriaMetrics
vmselect)] + HPA -- scale subresource --> CR[Engine CR
CNPG Cluster .spec.instances] CR -- managed by operator --> PODS[(replica pods)] - NOTE[chart no longer templates replicas under autoscaling] -.-> CR + NOTE[chart omits replicas under autoscaling] -.-> CR ``` The engine operator owns instance lifecycle: CNPG adds/removes the highest-ordinal standby gracefully, never the primary, and routes reads through `-ro`. The autoscaler never decides *which* instance to remove. ### 3. Chart change: stop declaring `replicas` under autoscaling -The decisive change. Each autoscalable app chart wraps its replica field in a conditional so that, when autoscaling is enabled for that application, the field is **omitted** from the rendered engine CR: +Each autoscalable chart wraps its replica field so that, when autoscaling is enabled for that application, the field is omitted from the rendered engine CR: ```yaml # packages/apps/postgres/templates/db.yaml (illustrative) @@ -86,121 +74,148 @@ spec: {{- end }} ``` -With the field absent from the HelmRelease values, Flux neither sets nor reverts it, and the HPA is the sole writer of `.spec.instances` via the `scale` subresource. No ownership annotation, no SSA field manager, no admission webhook, and no terminal-freeze conflict handling are needed — they existed only to win a fight this change prevents. +With the field absent from the HelmRelease values, Flux neither sets nor reverts it, and the HPA is the sole writer of `.spec.instances` via the `scale` subresource. This is what deletes the entire ownership problem — no marker annotation, SSA field manager, admission webhook, or terminal-freeze conflict handling is needed, because there is no contested field. + +The conditional keys off `autoscaling.enabled`, **not** off presence of the field: the aggregated apps API re-materializes `replicas: 2` from the values-schema default on every round-trip (`packages/apps/postgres/values.schema.json`), so a `hasKey`-style check would always see the field and reopen the conflict. This is harmless only because the chart *ignores* the value under autoscaling — the one sentence here exists to stop a later "simplification" from breaking it. -Migration cost is real and chart-sized, not operator-sized (see [Upgrade and rollback compatibility](#upgrade-and-rollback-compatibility)). +### 4. Custom-metrics adapter (shared platform infrastructure) -### 4. Metric source for the HPA +The HPA's Custom (Pods) metric is served by a **cluster-singleton adapter that registers the `custom.metrics.k8s.io` APIService** and reads from vmselect. Cozystack ships no custom/external metrics API today (only metrics-server's `metrics.k8s.io` resource metrics), so this adapter is **new shared infrastructure** other features will lean on — it warrants its own package and lifecycle, not an afterthought. Whatever backs it (prometheus-adapter, a KEDA metrics apiserver, or a purpose-built adapter), it must: -The HPA consumes read-load metrics (`ReadConnections`, `ReadCPUUtilization`) from VictoriaMetrics through a **custom/external metrics adapter**. To preserve multi-tenant isolation, the platform-managed adapter configuration injects a mandatory namespace/label matcher into every query and rejects any query it cannot constrain — never raw tenant-supplied PromQL (the same rule the first revision established). Each series is pre-aggregated over the target's standby pods so the value HPA reads is already per-read-replica. +- serve the per-pod encoding from §1 (standbys report `Lᵢ`, primary reports `target`), selecting pods by the lineage labels `apps.cozystack.io/application.{group,kind,name}` (not an ad-hoc `app:` label); +- inject a mandatory namespace/label matcher into every query and reject any query it cannot constrain, so no tenant series crosses tenants; +- implement the lag brake as a metric-layer clamp (see §5). -### 5. The thin guard (database-specific brakes) +The Custom-vs-External decision in §1 constrains this choice; it is a design commitment, not an open question. -The only net-new controller. It does **not** compute desired counts or write the engine CR — HPA does both. It watches the target and adjusts the **HPA's `minReplicas`/`maxReplicas`** (and surfaces status/events) to encode what HPA cannot: +### 5. The guard and the `DatabaseScalingPolicy` -- **Quorum floor** — hold `minReplicas ≥ maxSyncReplicas + 1` for CNPG so HPA can never drive the cluster below a safe synchronous quorum. Because `maxSyncReplicas` is tenant-mutable, the guard reconciles the floor into the HPA's `minReplicas` on change; CNPG independently rejects any unsafe count as a backstop. -- **Replication-lag brake** — when `cnpg_pg_replication_lag` exceeds `maxReplicationLagSeconds` **and the primary is actively writing** (write-activity gated off the exported LSN metrics, so an idle primary does not trip it), the guard pins `maxReplicas = currentInstances` to forbid further scale-up until lag recovers. -- **Recommendation / dry-run** — a mode where the guard computes and reports the recommendation (status, events, metrics, alerts) without ever unpinning the HPA, so operators can validate behavior before enabling actuation. +The tenant declares a single namespaced CR, `DatabaseScalingPolicy`; the **guard renders and owns the HPA** as an implementation detail. This is deliberate: a controller must never edit a spec a tenant also declares (that recreates the revert war one level up, on the HPA's `min`/`maxReplicas`). Because the guard is the sole writer of the HPA it creates, there is no second writer to contend with; because the tenant never touches the HPA, no tenant RBAC on `autoscaling/v2` is required (there is none today). The guard encodes the brakes as follows: -Quota is not re-implemented: because HPA scales the engine CR and pod creation passes through the tenant's `ResourceQuota` admission, an over-quota scale-up simply fails to create pods and is reflected in the CR/HPA status — no separate quota pre-check controller is required. +- **Quorum floor** — the guard sets `minReplicas = max(2, maxSyncReplicas + 1)` on its HPA and reconciles it when `maxSyncReplicas` changes, so HPA can never drive the cluster below a safe synchronous quorum; CNPG rejects an unsafe count as a backstop. This is a defaulting/validation rule on a field HPA already has, not a reconcile loop fighting anyone. +- **Replication-lag brake (metric layer)** — while `cnpg_pg_replication_lag` exceeds `maxReplicationLagSeconds` **and the primary is actively writing** (gated on `rate(cnpg_pg_stat_replication_sent_diff_bytes[5m]) > 0`, so an idle primary does not trip it), the adapter clamps every standby's series to exactly `target`, which drives `desired = currentInstances` and **freezes scaling in both directions**. Freezing both ways (not just blocking scale-up, as `maxReplicas`-pinning would) matches the intended brake semantics — scaling down under high lag is equally unsafe. The clamp has **hysteresis**: it releases only after lag falls below a lower recovery threshold (e.g. `0.5 × maxReplicationLagSeconds`) sustained for a cooldown, so the brake does not flap around a single boundary. +- **Scale-down pacing** — the guard pins `behavior.scaleDown.policies: [{type: Pods, value: 1, periodSeconds: ~600}]` on its HPA, so at most one standby is removed per period (restoring the step-of-1 conservatism the design review fought for; the default HPA policy would allow removing 100% of pods in 15s). `periodSeconds` is a deliberate value on the order of minutes — sized against replica provisioning latency (see Failure and edge cases) — and calibrated on real workloads. +- **Dry-run / recommendation** — a mode where the guard computes and reports the recommendation (status, events, metrics, alerts) without creating or actuating the HPA, so behavior can be validated before enabling actuation. + +Quota is not re-implemented: HPA scales the engine CR, pod creation passes through the tenant `ResourceQuota` admission, so an over-quota scale-up simply fails to create pods and is reflected in the CR/HPA status. The guard keeps an **alert on a persistently unmet desired count** so this does not fail silently. ## User-facing changes -A tenant enables autoscaling on their database and creates a standard HPA next to it. The database-specific brakes are configured on a small guard resource (or, equivalently, annotations on the HPA — final form decided in implementation): +A tenant enables autoscaling on the database and creates one `DatabaseScalingPolicy`. The HPA is rendered by the guard and shown here only for reference — the tenant does not author it: ```yaml -# 1. turn off the static replica declaration for this app +# tenant declares: turn on autoscaling + one policy apiVersion: apps.cozystack.io/v1alpha1 kind: Postgres metadata: { name: db, namespace: tenant-acme } spec: - autoscaling: { enabled: true } # chart omits instances; HPA owns it + autoscaling: { enabled: true } # chart omits instances; HPA owns it --- -# 2. stock HPA on the engine CR's scale subresource +apiVersion: autoscaling.cozystack.io/v1alpha1 +kind: DatabaseScalingPolicy +metadata: { name: db, namespace: tenant-acme } +spec: + targetRef: { kind: Postgres, name: db } + minReplicas: 2 # total instances; guard raises to quorum floor if needed + maxReplicas: 6 + metrics: + - type: ReadConnections # | ReadCPUUtilization + target: { averageValue: "150" } # per read-serving replica + maxReplicationLagSeconds: 30 + dryRun: false +``` + +```yaml +# rendered + owned by the guard (reference only): apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler -metadata: { name: db, namespace: tenant-acme } +metadata: { name: db, namespace: tenant-acme, ownerReferences: [DatabaseScalingPolicy/db] } spec: scaleTargetRef: { apiVersion: postgresql.cnpg.io/v1, kind: Cluster, name: postgres-db } - minReplicas: 2 + minReplicas: 3 # max(2, maxSyncReplicas+1) maxReplicas: 6 metrics: - - type: External - external: - metric: { name: cozystack_read_connections, selector: { matchLabels: { app: postgres-db } } } + - type: Pods + pods: + metric: { name: cozystack_db_read_load, selector: { matchLabels: { "apps.cozystack.io/application.name": db } } } target: { type: AverageValue, averageValue: "150" } behavior: scaleUp: { stabilizationWindowSeconds: 300 } - scaleDown: { stabilizationWindowSeconds: 1800 } ---- -# 3. database-specific brakes (thin guard) -apiVersion: autoscaling.cozystack.io/v1alpha1 -kind: DatabaseScalingPolicy -metadata: { name: db, namespace: tenant-acme } -spec: - targetRef: { kind: Postgres, name: db } - hpaRef: { name: db } - respectQuorum: true - maxReplicationLagSeconds: 30 - dryRun: false + scaleDown: { stabilizationWindowSeconds: 1800, policies: [{ type: Pods, value: 1, periodSeconds: 600 }] } ``` -When no HPA references an autoscalable app, and `autoscaling.enabled` is false, nothing changes — the chart templates `replicas` exactly as today. +When `autoscaling.enabled` is false and no policy exists, nothing changes — the chart templates `replicas` exactly as today. ## Upgrade and rollback compatibility -- **Opt-in and off by default.** The chart conditional is inert unless `autoscaling.enabled` is set; existing clusters and manifests are unaffected. The guard and metrics adapter ship as optional platform packages. -- **Enabling autoscaling on an existing database (the one real migration).** Flipping `autoscaling.enabled` removes `instances`/`replicas` from the rendered CR. Helm's three-way merge deletes a previously-set field on omission, and CNPG defaults to **1 instance** when `.spec.instances` is unset — so a naive flip would collapse a running cluster to a single instance before the HPA raises it. The rollout must therefore either (a) have the HPA (or the guard) set `.spec.instances` to the current count *before* the field is dropped from the chart, or (b) template a floor via the `scale` subresource so the count never dips below `minReplicas`. This migration path must be exercised on a dev cluster before MVP. +- **Opt-in and off by default.** The chart conditional is inert unless `autoscaling.enabled` is set; the guard and metrics adapter are optional platform packages. Existing clusters are unaffected. +- **Enabling autoscaling on an existing database — the one real migration, and it needs a deterministic two-phase order.** Flipping `autoscaling.enabled` removes `instances` from the rendered CR, and Helm's three-way merge deletes a key present in the old manifest and absent from the new one **regardless of who last wrote it** — so simply pre-setting `.spec.instances` through the scale subresource does **not** save it: the upgrade deletes the field, CNPG defaults to **1 instance**, and the HPA only re-raises it after CNPG has already begun removing standbys. A safe rollout therefore needs a real two-phase design — e.g. a transition window in which the chart templates `.spec.instances` as a floor (rendered in both the old and new manifest so three-way merge never sees it disappear) while the HPA takes over, then a second phase that drops the floor once the HPA is the established writer. The precise operation order must be worked out and exercised on a dev cluster before MVP; "must be tested" is a gate, not the mechanism. +- **Steady state after migration is correct.** With the field absent from both the previous and the current render, three-way merge leaves the HPA-set `.spec.instances` untouched. - **Cold start.** Until the HPA takes its first sample it holds at `minReplicas`; a brief window at the floor is expected. -- **Dependent objects.** `WorkloadMonitor` and dashboards that read `.Values.replicas` must switch to the observed instance count (`status.instances` / metrics), since the values field is no longer authoritative under autoscaling. -- **Rollback.** Set `autoscaling.enabled: false` (and delete the HPA): the chart resumes templating `replicas` and Flux reconciles it back. Fully reversible; no data migration. +- **Enablement constraint — `minReplicas ≥ 2` changes single-instance footprint.** Enabling autoscaling on a current single-instance Postgres permanently doubles instances (a second replica's PVC and DRBD volume). This is legitimate but must be a conscious enablement decision, not a surprise. +- **Dependent objects.** Consumers that read `.Values.replicas` (dashboards, some tooling) must switch to the observed count. Note the two are distinct: the **engine CR** carries `.status.instances`; the **`WorkloadMonitor`** carries `availableReplicas`/`observedReplicas`/`operational` — do not read a nonexistent `WorkloadMonitor.status.instances`. +- **Rollback.** Set `autoscaling.enabled: false` and delete the policy: the chart resumes templating `replicas` and Flux reconciles it back. Fully reversible; no data migration. ## Security -- **RBAC (much reduced vs the first revision).** The guard needs: read its own `DatabaseScalingPolicy`; read/update the referenced `HorizontalPodAutoscaler` (`min/maxReplicas`); read `workloadmonitors`; read-only HTTP to vmselect. It needs **no** write access to `Application`/`HelmRelease`, **no** admission webhook, **no** SSA field manager, and **no** engine-CR writes (HPA does that through the `scale` subresource under the tenant's existing RBAC). The metrics adapter is a standard read-only VictoriaMetrics client. -- **Multi-tenancy.** `DatabaseScalingPolicy` and the HPA are namespaced and live in the tenant namespace; the platform ships self-contained aggregated ClusterRoles (`rbac.cozystack.io/aggregate-to-tenant[-view|-admin]`). The metrics adapter injects a mandatory namespace matcher, so no tenant query can read another tenant's series. -- **Blast radius.** Because there is no cluster-wide admission webhook, enabling this feature adds no admission hop to unrelated Flux reconciliation — a key regression from the first design is gone. +- **RBAC (much reduced).** The guard needs: read/write its `DatabaseScalingPolicy` and status; create/update/own the rendered `HorizontalPodAutoscaler`; read `workloadmonitors`; read-only HTTP to vmselect. It needs **no** write to `Application`/`HelmRelease`, **no** admission webhook, **no** SSA field manager, and **no** engine-CR writes (the HPA does that through the scale subresource). The tenant needs RBAC only on `databasescalingpolicies`, granted through the platform's aggregated tenant ClusterRoles — **not** on `autoscaling/v2` (which cozystack-basics does not grant, and now need not). +- **Honest note on capability.** An HPA driving a CNPG `Cluster`'s `.spec.instances` scales a resource the tenant has no direct write access to. Because the guard owns the HPA and derives its target from the tenant's own database, this is bounded to the tenant's own workload — but it is a real, if narrow, elevation and is stated here on the record. +- **Multi-tenancy.** The policy and HPA are namespaced and live in the tenant namespace; the metrics adapter injects a mandatory namespace matcher, so no tenant query reads another tenant's series. +- **Blast radius.** No cluster-wide admission webhook — a key regression of the first design is gone; enabling the feature adds no admission hop to unrelated Flux reconciliation. ## Failure and edge cases +- **Replica provisioning latency (stateful reality).** A new CNPG standby does not serve reads immediately: PVC provisioning + base backup/clone + WAL catch-up can take minutes to hours for a large database. `scaleUp.stabilizationWindowSeconds` paces *decisions*, not *readiness*. Worse, cloning a new standby adds WAL-streaming load that *raises* replication lag exactly at scale-up, which can trip the lag brake and freeze further scaling — a feedback loop. The feature is therefore meaningful for read-heavy databases whose working set clones in minutes, not for very large datasets where a clone dominates the load window; during a clone the guard reports the in-progress scale and the brake behavior explicitly rather than issuing more scale-ups. - vmselect unreachable or metric missing → HPA has no metric and holds the current count (`ScalingActive=False` on the HPA); the guard alerts. No blind scaling. -- Replication lag above threshold **with an actively-writing primary** → guard pins `maxReplicas = current`, forbidding scale-up until lag recovers. An idle primary does not trip the brake (write-activity gating). -- Desired count would drop to/below the quorum floor → guard holds `minReplicas ≥ maxSyncReplicas + 1`; CNPG rejects an unsafe count as a backstop. -- Tenant quota exceeded on scale-up → pods fail `ResourceQuota` admission; the CR/HPA surface the unmet count; no separate freeze path needed. -- MariaDB target whose chart lacks scale-out support (`replication.replica.bootstrapFrom` unset) → the operator rejects on-the-fly scale-out (`MariaDBScaleOutError`); MariaDB stays out of the enabled set until the chart is fixed. -- Redis / MongoDB target → no `scale` subresource; rejected by the guard with a clear reason (deferred to the shim follow-up). +- Replication lag above threshold with an actively-writing primary → metric clamp freezes scaling both ways until lag recovers past the hysteresis band; an idle primary does not trip the brake. +- Desired count would drop to/below the quorum floor → `minReplicas` holds it; CNPG rejects an unsafe count as backstop. +- Over-quota scale-up → pods fail `ResourceQuota` admission; the CR/HPA surface the unmet count; the guard alerts on a persistently unmet desired. +- **Read disruption on scale-down.** Removing the highest-ordinal standby gracefully still severs read connections pinned to it through `-ro`. Clients must tolerate reconnection; connection draining / graceful client failover is a known limitation to document for tenants (and a candidate follow-up). +- MariaDB whose chart lacks scale-out support (`replication.replica.bootstrapFrom` unset) → operator rejects on-the-fly scale-out (`MariaDBScaleOutError`); MariaDB stays out of the enabled set until the chart is fixed. +- Redis / MongoDB → no scale subresource; rejected by the guard with a clear reason (deferred to the shim follow-up). - Sharded engine (ClickHouse, sharded MongoDB) → out of scope; not autoscalable. ## Testing -- **Unit:** the replica-model math and each engine's quorum/lag logic in the guard, with mocked VictoriaMetrics. +- **PoC first — validate the metric encoding (§1) against a real HPA:** confirm the standby-`Lᵢ` / primary-`target` Custom (Pods) series makes stock HPA compute `desiredInstances = desiredRead + 1`, and that `ceil` boundaries behave. This gates everything else. +- **Unit:** the replica-model math and the quorum/lag logic in the guard, with mocked VictoriaMetrics. - **Chart:** `helm template` with `autoscaling.enabled: true` omits the replica field; with it false, renders `replicas` exactly as today (regression guard). -- **Migration (dev cluster, CNPG):** flip `autoscaling.enabled` on a running multi-instance cluster and assert it does **not** collapse to 1 instance (the pre-set-before-omit path), then drive load and confirm HPA scales `.spec.instances`, reads route to `-ro`, and Flux does not revert. This replaces the first revision's force-writer ownership envtest, which is no longer meaningful because there is no ownership to enforce. -- **Guard integration:** lag above threshold with active writes pins `maxReplicas`; quorum floor tracks `maxSyncReplicas` changes; `dryRun` reports without pinning. +- **Migration (dev cluster, CNPG):** exercise the two-phase enable on a running multi-instance cluster and assert it does **not** collapse to 1 instance, then drive load and confirm HPA scales `.spec.instances`, reads route to `-ro`, and Flux does not revert. This replaces the first revision's force-writer ownership envtest, which is no longer meaningful — there is no ownership to enforce. +- **Guard integration:** lag above threshold with active writes freezes scaling both ways and releases only past the hysteresis band; quorum floor tracks `maxSyncReplicas`; scale-down removes one standby per `periodSeconds`; `dryRun` reports without creating an HPA. - **Negative:** vmselect down → no scaling; idle primary with high lag-seconds → no false brake; MariaDB without scale-out → rejected; Redis → rejected. ## Rollout -1. **PoC** — CNPG on a dev cluster: chart conditional + a stock HPA on `.spec.instances` driven by `ReadConnections`; confirm Flux does not revert and reads route to `-ro`. -2. **MVP** — PostgreSQL: the chart change, the metrics adapter (namespace-scoped), the thin guard (quorum floor + lag brake + dry-run), dashboard surface and alerts. Optional packages with a hard dependency on the monitoring stack. +1. **PoC** — CNPG on a dev cluster: chart conditional + guard-rendered HPA on `.spec.instances` driven by the synthesized read-load metric; validate the arithmetic and that Flux does not revert. +2. **MVP** — PostgreSQL: the chart change, the custom-metrics adapter (namespace-scoped, lag-clamp), the guard + `DatabaseScalingPolicy` (quorum floor, lag brake, scale-down pacing, dry-run), dashboard surface and alerts. 3. **MariaDB** — once the cozystack mariadb chart supports on-the-fly scale-out. -4. **Redis / MongoDB** — a follow-up proposal for a thin actuation shim, since neither exposes a `scale` subresource. +4. **Redis / MongoDB** — a follow-up proposal for a thin actuation shim, since neither exposes a scale subresource. ## Open questions -- Final form of the database-specific brakes: a small `DatabaseScalingPolicy` CRD, or annotations on the HPA? The former is clearer; the latter avoids any new API surface (see Alternative 1). -- Which metrics adapter — prometheus-adapter, a KEDA `ScaledObject` external trigger, or a small purpose-built adapter — best fits VictoriaMetrics with mandatory namespace scoping? -- Migration mechanic for enabling autoscaling on a live cluster: set `.spec.instances` before omitting the chart field, or floor it via the `scale` subresource? +- Which implementation backs the custom-metrics adapter (prometheus-adapter, a KEDA metrics apiserver, or purpose-built) — constrained by the Custom (Pods) choice in §1 and by the lag-clamp requirement. +- Exact two-phase migration mechanic (chart-templated floor during transition vs a staged operator-driven handover), to be settled on a dev cluster before MVP. - Default driver metric (read connections vs read QPS vs replica CPU), to be calibrated on real workloads. +- `periodSeconds` for scale-down pacing and the hysteresis recovery band — deliberate defaults to be tuned. ## Alternatives considered -- **A bespoke `db-autoscaler` operator owning `replicas` (the first revision of this proposal).** Rejected after the implementation spike. It required re-drawing HPA's API surface field-for-field (`metrics[].target.averageValue`, `behavior.*.stabilizationWindowSeconds`, the `ScalingActive`/`AbleToScale`/`ScalingLimited` conditions) and re-implementing HPA's decision loop; and its ownership guarantee proved unbuildable on the aggregated apps API ([Findings](#findings-from-the-implementation-spike)) — SSA does not hold, admission cannot fire there, and the fallback HelmRelease webhook is advisory, bypassable by tenant edits, and a platform-wide admission hop. The present design keeps HPA's hardened decision loop and confines net-new code to the two brakes HPA genuinely lacks. -- **HPA writing the `Application`'s `replicas` value (apps API) instead of the engine CR.** This is what the first revision did; it is the source of the whole ownership problem, because the apps values are declared in Git and reverted by Flux. Writing the engine CR's `scale` subresource while the chart omits the field (this design) avoids the conflict at its root. -- **Not creating any new API at all.** The brakes could be expressed as annotations on a stock HPA rather than a `DatabaseScalingPolicy` CRD, eliminating the new API group entirely. Kept as an open question; the CRD is proposed only for clarity, not necessity. -- **A thin actuation shim for engines without a `scale` subresource (Redis, MongoDB).** For these, HPA cannot act directly. A minimal shim that watches a stock HPA's recommendation and propagates it behind the same brakes is the honest path — deferred to a follow-up, since the MVP targets engines that already have a `scale` subresource. -- **Stock HPA + KEDA with tenant-supplied PromQL.** Rejected for the metric-source layer: raw tenant PromQL against shared vmselect breaks tenant isolation. A KEDA/prometheus-adapter trigger is acceptable **only** with a platform-injected, mandatory namespace matcher — the constraint carried over from the first revision. -- **Scaling the write path via sharding.** Out of scope: it requires data rebalancing, an orchestrated procedure rather than a replica-count change. +- **A bespoke `db-autoscaler` operator owning `replicas` (the first revision).** Rejected after the implementation spike (see Appendix). It re-drew HPA's API surface field-for-field and re-implemented its decision loop, and its ownership guarantee proved unbuildable on the aggregated apps API. This design keeps HPA's hardened loop and confines net-new code to the brakes HPA lacks. +- **HPA writing the `Application`'s `replicas` value (apps API) instead of the engine CR.** This is what the first revision did; it is the source of the whole ownership problem, because the apps values are declared in Git and reverted by Flux. Writing the engine CR's scale subresource while the chart omits the field avoids the conflict at its root. +- **A guard that pins `min`/`maxReplicas` on a tenant-declared HPA.** Rejected: it relocates the revert war from `replicas` to the HPA spec. Having the guard *own* the HPA (this design) removes the second writer entirely. +- **External metric instead of Custom (Pods).** Rejected: External `AverageValue` has no pod divisor, so it cannot express the read-replica model without off-by-primary errors; the Custom (Pods) encoding makes the model fall out of stock HPA arithmetic. +- **A thin actuation shim for engines without a scale subresource (Redis, MongoDB).** For these, HPA cannot act directly; a minimal shim watching a stock HPA's recommendation behind the same brakes is the honest path — deferred to a follow-up. +- **Stock HPA + KEDA with tenant-supplied PromQL.** Rejected for the metric layer: raw tenant PromQL against shared vmselect breaks isolation. A KEDA/prometheus-adapter trigger is acceptable only with a platform-injected mandatory namespace matcher. +- **Scaling the write path via sharding.** Out of scope: requires data rebalancing, an orchestrated procedure rather than a replica-count change. + +## Appendix: Findings from the implementation spike + +The first revision rested on one load-bearing claim: the autoscaler could be the *enforced* single owner of the application's `replicas` value, writing it through the aggregated apps API. Building it disproved that claim, and these findings are why the mechanism changed: + +1. **SSA field-level ownership does not hold on the aggregated apps API.** The `Application` spec is an opaque JSON blob and its managed-fields are not round-tripped, so a dedicated field manager cannot claim `.spec.replicas`. The first revision's open question — "does the aggregated Patch handler support per-field SSA at all?" — is answered: no. +2. **Admission webhooks cannot fire on the aggregated API.** kube-apiserver proxies aggregated-API requests to the extension server, where admission does not run; enforcement had to move to the backing Flux `HelmRelease`. +3. **The HelmRelease webhook is neither cheap nor sufficient.** It must allowlist the apps-API ServiceAccount (so a tenant edit through the apps API bypasses the guard) and must not hard-fail Flux during an outage. What remains is advisory ownership plus a platform-wide admission hop — not the enforced guarantee promised. +4. **The root cause is self-imposed.** The autoscaler-vs-Flux conflict exists only because the chart unconditionally templates the replica field. Removing that declaration under autoscaling (§3) makes the entire ownership problem disappear. --- From 50dea519df44b21f2c47fea1eabd35b4d72fe0dc Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Wed, 29 Jul 2026 17:44:18 +0300 Subject: [PATCH 3/5] =?UTF-8?q?docs(dha):=20address=20CodeRabbit=20?= =?UTF-8?q?=E2=80=94=20per-policy=20metric=20baseline,=20zero-fill,=20effe?= =?UTF-8?q?ctive=20bounds?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - §4: bind the primary baseline to the policy's targetPerStandby and key series per policy (a shared adapter must not cross targets); require one sample per current pod, zero-filling standby series CNPG omits at zero active connections, and handling Pending/Terminating pods - §5: make the DatabaseScalingPolicy the source of truth for min/maxReplicas and derive effective bounds statelessly each reconcile; when the quorum floor exceeds policy.maxReplicas, quorum wins (raise effectiveMax) and surface a condition, never clamp below quorum Signed-off-by: Alexey Artamonov --- design-proposals/database-horizontal-autoscaling/README.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/design-proposals/database-horizontal-autoscaling/README.md b/design-proposals/database-horizontal-autoscaling/README.md index 6be9e8b..7c03a4a 100644 --- a/design-proposals/database-horizontal-autoscaling/README.md +++ b/design-proposals/database-horizontal-autoscaling/README.md @@ -82,7 +82,8 @@ The conditional keys off `autoscaling.enabled`, **not** off presence of the fiel The HPA's Custom (Pods) metric is served by a **cluster-singleton adapter that registers the `custom.metrics.k8s.io` APIService** and reads from vmselect. Cozystack ships no custom/external metrics API today (only metrics-server's `metrics.k8s.io` resource metrics), so this adapter is **new shared infrastructure** other features will lean on — it warrants its own package and lifecycle, not an afterthought. Whatever backs it (prometheus-adapter, a KEDA metrics apiserver, or a purpose-built adapter), it must: -- serve the per-pod encoding from §1 (standbys report `Lᵢ`, primary reports `target`), selecting pods by the lineage labels `apps.cozystack.io/application.{group,kind,name}` (not an ad-hoc `app:` label); +- serve the per-pod encoding from §1, **keyed per policy**: each standby reports its own read load and the primary reports **that policy's** `targetPerStandby` as its baseline. Because the primary baseline *is* the policy target, the series must be scoped per application/policy (distinct selectors or metric names) so a shared adapter never applies one target's baseline to another; the PoC must exercise multiple `targetPerStandby` values; +- select pods by the lineage labels `apps.cozystack.io/application.{group,kind,name}` (not an ad-hoc `app:` label), and emit **exactly one sample per current pod** — **zero-filling** standby pods whose underlying series is absent (CNPG omits `cnpg_backends_total{state="active"}` when a standby has zero active connections) and handling `Pending`/`Terminating` pods, since a missing standby sample would corrupt the `(target + ΣLᵢ) / N` average and the resulting count; - inject a mandatory namespace/label matcher into every query and reject any query it cannot constrain, so no tenant series crosses tenants; - implement the lag brake as a metric-layer clamp (see §5). @@ -92,7 +93,7 @@ The Custom-vs-External decision in §1 constrains this choice; it is a design co The tenant declares a single namespaced CR, `DatabaseScalingPolicy`; the **guard renders and owns the HPA** as an implementation detail. This is deliberate: a controller must never edit a spec a tenant also declares (that recreates the revert war one level up, on the HPA's `min`/`maxReplicas`). Because the guard is the sole writer of the HPA it creates, there is no second writer to contend with; because the tenant never touches the HPA, no tenant RBAC on `autoscaling/v2` is required (there is none today). The guard encodes the brakes as follows: -- **Quorum floor** — the guard sets `minReplicas = max(2, maxSyncReplicas + 1)` on its HPA and reconciles it when `maxSyncReplicas` changes, so HPA can never drive the cluster below a safe synchronous quorum; CNPG rejects an unsafe count as a backstop. This is a defaulting/validation rule on a field HPA already has, not a reconcile loop fighting anyone. +- **Effective bounds and quorum floor** — the `DatabaseScalingPolicy` is the source of truth for `minReplicas`/`maxReplicas`; the guard never mutates the tenant's configured values, it **derives** the HPA's effective bounds from them on every reconcile. The derivation is stateless, so a controller restart or a policy edit cannot leave stale bounds on the HPA: `effectiveMin = max(policy.minReplicas, 2, maxSyncReplicas + 1)`. `maxSyncReplicas` is tenant-mutable, so the quorum floor can rise above `policy.maxReplicas`; when it does, **quorum wins** — the guard raises `effectiveMax` to the floor as well (never leaving the cluster below a safe synchronous quorum) and surfaces a condition/alert that the configured maximum was overridden, rather than clamping below quorum. CNPG rejects an unsafe count as a final backstop. This is a defaulting/validation rule on fields the HPA already has, not a reconcile loop fighting anyone. - **Replication-lag brake (metric layer)** — while `cnpg_pg_replication_lag` exceeds `maxReplicationLagSeconds` **and the primary is actively writing** (gated on `rate(cnpg_pg_stat_replication_sent_diff_bytes[5m]) > 0`, so an idle primary does not trip it), the adapter clamps every standby's series to exactly `target`, which drives `desired = currentInstances` and **freezes scaling in both directions**. Freezing both ways (not just blocking scale-up, as `maxReplicas`-pinning would) matches the intended brake semantics — scaling down under high lag is equally unsafe. The clamp has **hysteresis**: it releases only after lag falls below a lower recovery threshold (e.g. `0.5 × maxReplicationLagSeconds`) sustained for a cooldown, so the brake does not flap around a single boundary. - **Scale-down pacing** — the guard pins `behavior.scaleDown.policies: [{type: Pods, value: 1, periodSeconds: ~600}]` on its HPA, so at most one standby is removed per period (restoring the step-of-1 conservatism the design review fought for; the default HPA policy would allow removing 100% of pods in 15s). `periodSeconds` is a deliberate value on the order of minutes — sized against replica provisioning latency (see Failure and edge cases) — and calibrated on real workloads. - **Dry-run / recommendation** — a mode where the guard computes and reports the recommendation (status, events, metrics, alerts) without creating or actuating the HPA, so behavior can be validated before enabling actuation. From 63c5ab289f7457d012e6d83614ff3f108b53dc07 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Fri, 31 Jul 2026 18:33:05 +0300 Subject: [PATCH 4/5] =?UTF-8?q?docs(dha):=20collapse=20to=20KEDA=20?= =?UTF-8?q?=E2=80=94=20query=20the=20metric,=20drop=20the=20controller=20a?= =?UTF-8?q?nd=20CRD?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Address @lllamnyp's review: the read-replica metric need not be emitted per-pod — an aggregate value Σ+target with an AverageValue target makes stock HPA compute 1+ceil(Σ/target), so one PromQL query the chart authors replaces the per-pod encoding entirely. With the query in hand the guard controller and DatabaseScalingPolicy CRD dissolve: the chart renders a KEDA ScaledObject; KEDA queries vmselect and manages the HPA on the engine scale subresource. Brakes become static — quorum floor via minReplicaCount template arithmetic, scale-down pacing via a behavior literal, lag brake via a query-side clamp with hysteresis, dry-run by not rendering the object. Flux owns the ScaledObject declaratively, so no runtime writer of its spec exists. - §1: single-value Σ+target metric with worked arithmetic and Σ=0→1 note - §4: metric backend collapsed to the KEDA choice (prometheus-adapter and kube-metrics-adapter weighed) - §5: autoscaling values block + cozy-lib helper replace the controller/CRD - rollback made count-preserving (disable must not shrink a live cluster) - honest note: no active rollback of a stuck scale-up (pending+alert instead) - Alternatives: correct the External-metric entry; add rev3 guard+CRD as rejected; net-new surface rounds down to a helper, a values block, a query Signed-off-by: Alexey Artamonov --- .../database-horizontal-autoscaling/README.md | 190 +++++++++--------- 1 file changed, 97 insertions(+), 93 deletions(-) diff --git a/design-proposals/database-horizontal-autoscaling/README.md b/design-proposals/database-horizontal-autoscaling/README.md index 7c03a4a..7f570fa 100644 --- a/design-proposals/database-horizontal-autoscaling/README.md +++ b/design-proposals/database-horizontal-autoscaling/README.md @@ -2,36 +2,36 @@ - **Title:** `Database Horizontal Autoscaler for Cozystack` - **Author(s):** `@scooby87` -- **Date:** `2026-07-08`; revised `2026-07-24` (mechanism) and `2026-07-29` (addressing @lllamnyp and @IvanHunters review on PR #44), with earlier review by @IvanHunters, Gemini, and CodeRabbit +- **Date:** `2026-07-08`; revised `2026-07-24` (mechanism), `2026-07-29` and `2026-07-31` (addressing @lllamnyp and @IvanHunters review on PR #44), with earlier review by @IvanHunters, Gemini, and CodeRabbit - **Status:** Draft ## Overview -This proposal adds automatic horizontal scaling of a managed database's **read replicas** in response to load, using **the stock Kubernetes `HorizontalPodAutoscaler` (HPA) acting on the engine operator's `scale` subresource**, plus a one-line chart change so the replica field is no longer declared in Git, plus a thin engine-aware controller — the **`DatabaseScalingPolicy` guard** — that renders the HPA, encodes the two database-specific brakes HPA lacks (synchronous-quorum floor and replication-lag gate), and drives a custom metric that makes stock HPA arithmetic compute the correct read-replica count. +This proposal adds automatic horizontal scaling of a managed database's **read replicas** in response to load. The mechanism is **entirely stock**: the application chart renders a **KEDA `ScaledObject`** next to the database; KEDA queries VictoriaMetrics for the read load, computes the desired count with a plain `HorizontalPodAutoscaler` it manages, and drives the engine operator's **`scale` subresource** (CloudNativePG `Cluster.spec.instances`). There is **no bespoke operator and no new CRD** — the net-new surface of this proposal is a Helm helper, one `autoscaling` values block, one PromQL query, and KEDA added as a platform component. The proposal is deliberately scoped to **horizontal scaling of read replicas**: a stateful primary cannot be scaled horizontally the way a stateless Deployment can. The MVP targets **PostgreSQL (CloudNativePG)**; see [Scope](#scope-and-related-proposals) for the engine ladder. ### Why this changed -An earlier revision proposed a bespoke `db-autoscaler` operator that owned the application's `replicas` value and enforced that ownership against competing writers. An implementation spike disproved the enforcement premise it rested on — SSA field ownership does not hold on the aggregated apps API, admission webhooks cannot fire there, and the fallback HelmRelease webhook is advisory, bypassable, and platform-wide. The spike also showed the whole conflict is self-imposed: it exists only because our own chart unconditionally templates the replica field, so removing that declaration under autoscaling makes the ownership problem disappear rather than needing to be enforced. The full spike findings are preserved in the [Appendix](#appendix-findings-from-the-implementation-spike); this revision builds on their conclusion — reuse HPA, do not reimplement it. +This design converged over three revisions, each removing machinery the previous one thought it needed. Rev1 proposed a bespoke `db-autoscaler` operator that *owned* the application's `replicas` value and enforced that ownership; an implementation spike proved the enforcement premise unbuildable on the aggregated apps API, and showed the whole conflict was self-imposed — it exists only because the chart unconditionally templates the replica field (full findings in the [Appendix](#appendix-findings-from-the-implementation-spike)). Rev2/rev3 therefore moved to a stock HPA on the engine's `scale` subresource with the chart omitting the field, keeping only a thin controller and CRD to render the HPA and drive a synthesized metric. Review then showed even that is unnecessary: the metric can be *queried* into existence rather than emitted per-pod, and once the query exists, KEDA renders and manages everything declaratively — so the controller and CRD are gone too. The guiding principle throughout: reuse the platform Kubernetes ships, do not reimplement it. ## Scope and related proposals This proposal covers **horizontal** autoscaling (read replicas) only. Two sibling axes are deferred to separate proposals: **vertical autoscaling** (stepping the `resourcesPreset` ladder / in-place resize) and **storage autoscaling** (automatic PVC expansion). Write-path scaling that requires data rebalancing (Kafka broker addition, ClickHouse/MongoDB sharding) is out of scope — it is an orchestrated procedure, not a counter change. -**Engine scope of the MVP.** The HPA-on-`scale`-subresource mechanism applies to engines whose operator CR exposes a `scale` subresource: PostgreSQL (CloudNativePG `Cluster.spec.instances`) and MariaDB (`MariaDB.spec.replicas`). The MVP ships **PostgreSQL**; MariaDB follows once its cozystack chart supports on-the-fly scale-out (today it does not — see [Failure and edge cases](#failure-and-edge-cases)). **Redis (spotahome RedisFailover) and MongoDB (Percona) expose no `scale` subresource**, so a stock HPA cannot drive them; they are deferred to a follow-up that adds a thin actuation shim (see [Alternatives considered](#alternatives-considered)). +**Engine scope of the MVP.** The mechanism applies to engines whose operator CR exposes a `scale` subresource: PostgreSQL (CloudNativePG `Cluster.spec.instances`) and MariaDB (`MariaDB.spec.replicas`). The MVP ships **PostgreSQL**; MariaDB follows once its cozystack chart supports on-the-fly scale-out (today it does not — see [Failure and edge cases](#failure-and-edge-cases)). **Redis (spotahome RedisFailover) and MongoDB (Percona) expose no `scale` subresource**, so a stock HPA cannot drive them; they are deferred to a follow-up that adds a thin actuation shim (see [Alternatives considered](#alternatives-considered)). ## Context A managed database in Cozystack is an `Application` in the aggregated `apps.cozystack.io` API — a **pure projection of a Flux `HelmRelease`** (`pkg/registry/apps/application/rest.go` converts both ways, no separate backing store). Flux reconciles the `HelmRelease` values into the engine operator's CR — for CNPG a `Cluster`, where `packages/apps/postgres/templates/db.yaml` maps `instances: {{ .Values.replicas }}`. Cozystack already runs the observability the autoscaler needs: - A per-database `WorkloadMonitor` (`cozystack.io/v1alpha1`) reports `status.availableReplicas`, `status.observedReplicas`, and `status.operational`. -- Managed-app pods carry the lineage labels `apps.cozystack.io/application.{group,kind,name}` (via `internal/lineagecontrollerwebhook/webhook.go`), and kube-state-metrics exports `kube_pod_labels` (including CNPG's `cnpg.io/instanceRole` as `label_cnpg_io_instance_role`), so a metric can be scoped to one application's read-serving pods and to the standby role. +- Managed-app pods carry the lineage labels `apps.cozystack.io/application.{group,kind,name}` (via `internal/lineagecontrollerwebhook/webhook.go`), and kube-state-metrics exports `kube_pod_labels` (including CNPG's `cnpg.io/instanceRole` as `label_cnpg_io_instance_role`), so a query can be scoped to one application's read-serving pods and to the standby role. - VictoriaMetrics (`packages/system/monitoring`) scrapes per-database metrics; for PostgreSQL `enablePodMonitor: true` exports `cnpg_*` series, including the replication-lag gauge. vmselect is reachable at `vmselect-..svc:8481/select/0/prometheus`. ## Design -### 1. Replica model and metric encoding +### 1. Replica model and the single-value metric The engine's total instance count is `1` primary plus `replicas − 1` standbys; read traffic is served only by the standbys via `-ro`. The autoscaling target is per read-serving replica: @@ -39,28 +39,35 @@ The engine's total instance count is `1` primary plus `replicas − 1` standbys; - `desiredRead = ceil(Σ readLoad over standbys / targetPerStandby)` - `desiredInstances = desiredRead + primaryCount` -A stock HPA has no `+ primaryCount` term and no notion of "standbys only" — for a metric it just computes a desired count. The choice of metric *type* is therefore the formula, and the two options are not interchangeable: an **External** metric is a single free-standing value (`desired = ceil(value / target)`, no pod divisor), whereas a **Custom (Pods)** metric (`custom.metrics.k8s.io`, `type: Pods`) is averaged by HPA over the scale target's pods (`desired = ceil(currentPods × avg / target)`). We use the **Custom (Pods)** encoding and synthesize the series so unmodified HPA arithmetic reproduces the model exactly: +The key realization is that **the metric need not be emitted per pod — it can be queried into existence.** An HPA only ever consumes the aggregate: for an External (or Object) metric with an `AverageValue` target, `desired = ceil(value / target)`, with no pod divisor. So it is enough to serve a single value `Σ + target`, where `Σ` is the summed standby read load and `target` is the per-standby target folded in as a constant (the chart knows it at render time): -> Each **standby** pod reports its own read load `Lᵢ`; the **primary** pod reports **exactly `targetPerStandby`**. With `N = currentInstances` pods, HPA computes `desired = ceil(N × avg / target) = ceil((target + ΣLᵢ) / target) = 1 + ceil(ΣLᵢ / target) = primaryCount + desiredRead`. +> `desired = ceil((Σ + target) / target) = 1 + ceil(Σ / target) = primaryCount + desiredRead`. -The `+1` for the primary and the "divide by standbys only" both fall out of the primary reporting the target value — no controller math, no external-metric offset hacks. Worked example, `target = 150` active read connections per standby, a 3-instance cluster (1 primary + 2 standbys): at `ΣLᵢ = 210` → `avg = (150+210)/3 = 120`, `desired = ceil(3×120/150) = ceil(2.4) = 3` (holds); at `ΣLᵢ = 600` → `avg = 250`, `desired = ceil(750/150) = 5` (scales up). Validating this encoding end-to-end against a real HPA is the first thing the PoC must do. +Both the `+1` for the primary and the "divide by standbys only" fall out of adding `target` inside the query — no per-pod emission, no controller math, no external-metric offset. The whole expression is one PromQL query the chart authors: -The two MVP metrics are the same read-load signals the platform already scrapes: active read connections (`cnpg_backends_total{state="active"}`) and read-path CPU (`rate(container_cpu_usage_seconds_total{container="postgres"}[5m])`), each joined to the standby role through `kube_pod_labels{label_cnpg_io_instance_role="replica"}`. +```promql +sum(cnpg_backends_total{namespace="tenant-acme",state="active"} + * on(namespace,pod) group_left() kube_pod_labels{namespace="tenant-acme", + label_apps_cozystack_io_application_name="db",label_cnpg_io_instance_role="replica"}) ++ 150 +``` + +Worked example, `target = 150` active read connections per standby, a 3-instance cluster (1 primary + 2 standbys): at `Σ = 210` → `ceil((150+210)/150) = ceil(2.4) = 3` (holds); at `Σ = 600` → `ceil(750/150) = 5` (scales up); at `Σ = 60` → `ceil(210/150) = 2` (scales down). At `Σ = 0` the value is `target` and `desired = 1`, so the `minReplicas ≥ 2` floor (§5) is load-bearing. Validating that this single-value query drives a real HPA to `1 + ceil(Σ/target)` across the `ceil` boundaries is the first thing the PoC must do. The two MVP signals are the ones the platform already scrapes: active read connections (`cnpg_backends_total{state="active"}`) and read-path CPU (`rate(container_cpu_usage_seconds_total{container="postgres"}[5m])`). ### 2. Data flow ```mermaid flowchart LR - DSP[DatabaseScalingPolicy CR
tenant-declared] -- watch --> GUARD[db-scaling guard] - GUARD -- renders + owns --> HPA[HorizontalPodAutoscaler] - HPA -- custom metric --> ADAPTER[custom-metrics adapter] - ADAPTER -- HTTP /select/0/prometheus --> VM[(VictoriaMetrics
vmselect)] + HR[HelmRelease values
autoscaling: enabled] -- Flux renders --> SO[KEDA ScaledObject
query + bounds + behavior] + KEDA[KEDA operator] -- reads --> SO + KEDA -- PromQL /select/0/prometheus --> VM[(VictoriaMetrics
vmselect)] + KEDA -- creates + manages --> HPA[HorizontalPodAutoscaler] HPA -- scale subresource --> CR[Engine CR
CNPG Cluster .spec.instances] CR -- managed by operator --> PODS[(replica pods)] NOTE[chart omits replicas under autoscaling] -.-> CR ``` -The engine operator owns instance lifecycle: CNPG adds/removes the highest-ordinal standby gracefully, never the primary, and routes reads through `-ro`. The autoscaler never decides *which* instance to remove. +The engine operator owns instance lifecycle: CNPG adds/removes the highest-ordinal standby gracefully, never the primary, and routes reads through `-ro`. Nothing in this design decides *which* instance to remove. ### 3. Chart change: stop declaring `replicas` under autoscaling @@ -74,139 +81,136 @@ spec: {{- end }} ``` -With the field absent from the HelmRelease values, Flux neither sets nor reverts it, and the HPA is the sole writer of `.spec.instances` via the `scale` subresource. This is what deletes the entire ownership problem — no marker annotation, SSA field manager, admission webhook, or terminal-freeze conflict handling is needed, because there is no contested field. +With the field absent from the HelmRelease values, Flux neither sets nor reverts it, and the HPA (via the `scale` subresource) is the sole writer of `.spec.instances`. This is what deletes the entire ownership problem — no marker annotation, SSA field manager, admission webhook, or terminal-freeze conflict handling is needed, because there is no contested field. The conditional keys off `autoscaling.enabled`, **not** off presence of the field: the aggregated apps API re-materializes `replicas: 2` from the values-schema default on every round-trip (`packages/apps/postgres/values.schema.json`), so a `hasKey`-style check would always see the field and reopen the conflict. This is harmless only because the chart *ignores* the value under autoscaling — the one sentence here exists to stop a later "simplification" from breaking it. -### 4. Custom-metrics adapter (shared platform infrastructure) +### 4. Metric backend: KEDA -The HPA's Custom (Pods) metric is served by a **cluster-singleton adapter that registers the `custom.metrics.k8s.io` APIService** and reads from vmselect. Cozystack ships no custom/external metrics API today (only metrics-server's `metrics.k8s.io` resource metrics), so this adapter is **new shared infrastructure** other features will lean on — it warrants its own package and lifecycle, not an afterthought. Whatever backs it (prometheus-adapter, a KEDA metrics apiserver, or a purpose-built adapter), it must: +An HPA object cannot carry a query — its metric spec holds only a name and a selector — so the query must live where the metrics-API backend reads it, and the options differ sharply: -- serve the per-pod encoding from §1, **keyed per policy**: each standby reports its own read load and the primary reports **that policy's** `targetPerStandby` as its baseline. Because the primary baseline *is* the policy target, the series must be scoped per application/policy (distinct selectors or metric names) so a shared adapter never applies one target's baseline to another; the PoC must exercise multiple `targetPerStandby` values; -- select pods by the lineage labels `apps.cozystack.io/application.{group,kind,name}` (not an ad-hoc `app:` label), and emit **exactly one sample per current pod** — **zero-filling** standby pods whose underlying series is absent (CNPG omits `cnpg_backends_total{state="active"}` when a standby has zero active connections) and handling `Pending`/`Terminating` pods, since a missing standby sample would corrupt the `(target + ΣLᵢ) / N` average and the resulting count; -- inject a mandatory namespace/label matcher into every query and reject any query it cannot constrain, so no tenant series crosses tenants; -- implement the lag brake as a metric-layer clamp (see §5). +- **prometheus-adapter — ruled out.** Its queries live in one global ConfigMap, so a per-application query means per-application adapter config plus a reload — a registration step for every database. It also speaks to a single upstream URL, while every tenant's metrics live behind a different vmselect. +- **KEDA — recommended.** The query lives inline in a namespaced `ScaledObject` that the chart renders exactly where it would have rendered an HPA; there is no global config and no registration step, and KEDA generates and manages the HPA itself. Everything this design needs passes through: `scaleTargetRef` accepts any CR with a `scale` subresource (CNPG `Cluster` qualifies), `minReplicaCount`/`maxReplicaCount` take the template-computed bounds, `advanced.horizontalPodAutoscalerConfig.behavior` carries the scale-down policies verbatim, and `serverAddress` is per-object — so each tenant's `ScaledObject` points at its own vmselect, which a single-upstream adapter cannot do. +- **kube-metrics-adapter (Zalando)** is the lighter alternative — the query lives in annotations on the HPA — but it is a much smaller project and its per-tenant-server story is weaker. -The Custom-vs-External decision in §1 constrains this choice; it is a design commitment, not an open question. +Because the query is authored by the chart template (the tenant supplies only numbers through values), the mandatory-scoping rule — no raw tenant PromQL against shared vmselect — is satisfied by construction. The cost is that **KEDA becomes a new platform component**: a cluster-singleton that claims the `external.metrics.k8s.io` APIService (nothing serves it in Cozystack today), shared by any future feature that needs custom-metric autoscaling. -### 5. The guard and the `DatabaseScalingPolicy` +### 5. The `autoscaling` values block and the rendered `ScaledObject` -The tenant declares a single namespaced CR, `DatabaseScalingPolicy`; the **guard renders and owns the HPA** as an implementation detail. This is deliberate: a controller must never edit a spec a tenant also declares (that recreates the revert war one level up, on the HPA's `min`/`maxReplicas`). Because the guard is the sole writer of the HPA it creates, there is no second writer to contend with; because the tenant never touches the HPA, no tenant RBAC on `autoscaling/v2` is required (there is none today). The guard encodes the brakes as follows: +There is no controller and no CRD. The tenant sets an `autoscaling` block in the application's own values (validated by `values.schema.json`, like every other cozystack knob), and a cozy-lib Helm helper renders the `ScaledObject`. Each database-specific brake is expressed statically: -- **Effective bounds and quorum floor** — the `DatabaseScalingPolicy` is the source of truth for `minReplicas`/`maxReplicas`; the guard never mutates the tenant's configured values, it **derives** the HPA's effective bounds from them on every reconcile. The derivation is stateless, so a controller restart or a policy edit cannot leave stale bounds on the HPA: `effectiveMin = max(policy.minReplicas, 2, maxSyncReplicas + 1)`. `maxSyncReplicas` is tenant-mutable, so the quorum floor can rise above `policy.maxReplicas`; when it does, **quorum wins** — the guard raises `effectiveMax` to the floor as well (never leaving the cluster below a safe synchronous quorum) and surfaces a condition/alert that the configured maximum was overridden, rather than clamping below quorum. CNPG rejects an unsafe count as a final backstop. This is a defaulting/validation rule on fields the HPA already has, not a reconcile loop fighting anyone. -- **Replication-lag brake (metric layer)** — while `cnpg_pg_replication_lag` exceeds `maxReplicationLagSeconds` **and the primary is actively writing** (gated on `rate(cnpg_pg_stat_replication_sent_diff_bytes[5m]) > 0`, so an idle primary does not trip it), the adapter clamps every standby's series to exactly `target`, which drives `desired = currentInstances` and **freezes scaling in both directions**. Freezing both ways (not just blocking scale-up, as `maxReplicas`-pinning would) matches the intended brake semantics — scaling down under high lag is equally unsafe. The clamp has **hysteresis**: it releases only after lag falls below a lower recovery threshold (e.g. `0.5 × maxReplicationLagSeconds`) sustained for a cooldown, so the brake does not flap around a single boundary. -- **Scale-down pacing** — the guard pins `behavior.scaleDown.policies: [{type: Pods, value: 1, periodSeconds: ~600}]` on its HPA, so at most one standby is removed per period (restoring the step-of-1 conservatism the design review fought for; the default HPA policy would allow removing 100% of pods in 15s). `periodSeconds` is a deliberate value on the order of minutes — sized against replica provisioning latency (see Failure and edge cases) — and calibrated on real workloads. -- **Dry-run / recommendation** — a mode where the guard computes and reports the recommendation (status, events, metrics, alerts) without creating or actuating the HPA, so behavior can be validated before enabling actuation. +- **Quorum floor** — template arithmetic, not a reconcile loop: `minReplicaCount: max(.Values.autoscaling.minReplicas, .Values.quorum.maxSyncReplicas + 1, 2)`. Both values live in the same chart, so a tenant raising `maxSyncReplicas` re-renders the floor atomically in the same values write — strictly better than a controller converging on it. When the floor would exceed `maxReplicas`, the helper raises `maxReplicaCount` to the floor too (quorum wins, never clamp below a safe quorum) and the alert rules flag that the configured maximum was overridden. CNPG rejects an unsafe count as a final backstop. +- **Scale-down pacing** — a literal `behavior.scaleDown.policies: [{type: Pods, value: 1, periodSeconds: ~600}]` in the rendered object, so at most one standby is removed per period (restoring the step-of-1 conservatism; the default HPA policy would allow removing 100% of pods in 15s). `periodSeconds` is a deliberate value on the order of minutes, sized against replica provisioning latency (see [Failure and edge cases](#failure-and-edge-cases)). +- **Replication-lag brake** — a clamp inside the same query: while `cnpg_pg_replication_lag` exceeds the threshold **and the primary is actively writing** (`rate(cnpg_pg_stat_replication_sent_diff_bytes[5m]) > 0`, so an idle primary does not trip it), the query returns `currentInstances × target` (current instance count from a pod count or the HPA status series), which pins `desired = currentInstances` and **freezes scaling in both directions** — safer than `maxReplicas`-pinning, which would block only scale-up while silently allowing scale-down under lag. Hysteresis is expressed query-side: comparing `max_over_time(cnpg_pg_replication_lag[])` against a lower recovery threshold *is* a hysteresis band, so the brake does not flap around a single boundary. +- **Dry-run / recommendation** — render the dashboard and alert rules but not the `ScaledObject` (or use KEDA's pause annotation), so behavior can be observed before actuation is enabled. -Quota is not re-implemented: HPA scales the engine CR, pod creation passes through the tenant `ResourceQuota` admission, so an over-quota scale-up simply fails to create pods and is reflected in the CR/HPA status. The guard keeps an **alert on a persistently unmet desired count** so this does not fail silently. +Quota is not re-implemented: the HPA scales the engine CR and pod creation passes through the tenant `ResourceQuota` admission, so an over-quota scale-up simply fails to create pods and is reflected in the CR/HPA status. An **alert on a persistently unmet desired count** keeps that from failing silently. + +Because the `ScaledObject` is rendered inside the HelmRelease, **Flux owns it declaratively and there is no runtime writer of its spec at all** — which closes the ownership question more completely than any controller-rendered object could. ## User-facing changes -A tenant enables autoscaling on the database and creates one `DatabaseScalingPolicy`. The HPA is rendered by the guard and shown here only for reference — the tenant does not author it: +A tenant turns on autoscaling in the application's own values — nothing else: ```yaml -# tenant declares: turn on autoscaling + one policy apiVersion: apps.cozystack.io/v1alpha1 kind: Postgres metadata: { name: db, namespace: tenant-acme } spec: - autoscaling: { enabled: true } # chart omits instances; HPA owns it ---- -apiVersion: autoscaling.cozystack.io/v1alpha1 -kind: DatabaseScalingPolicy -metadata: { name: db, namespace: tenant-acme } -spec: - targetRef: { kind: Postgres, name: db } - minReplicas: 2 # total instances; guard raises to quorum floor if needed - maxReplicas: 6 - metrics: - - type: ReadConnections # | ReadCPUUtilization - target: { averageValue: "150" } # per read-serving replica - maxReplicationLagSeconds: 30 - dryRun: false + autoscaling: + enabled: true + minReplicas: 2 # total instances; the chart raises to the quorum floor + maxReplicas: 6 + target: 150 # per read-serving replica + maxReplicationLagSeconds: 30 + dryRun: false ``` +The chart renders (reference only — the tenant never authors this): + ```yaml -# rendered + owned by the guard (reference only): -apiVersion: autoscaling/v2 -kind: HorizontalPodAutoscaler -metadata: { name: db, namespace: tenant-acme, ownerReferences: [DatabaseScalingPolicy/db] } +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: { name: db, namespace: tenant-acme } spec: scaleTargetRef: { apiVersion: postgresql.cnpg.io/v1, kind: Cluster, name: postgres-db } - minReplicas: 3 # max(2, maxSyncReplicas+1) - maxReplicas: 6 - metrics: - - type: Pods - pods: - metric: { name: cozystack_db_read_load, selector: { matchLabels: { "apps.cozystack.io/application.name": db } } } - target: { type: AverageValue, averageValue: "150" } - behavior: - scaleUp: { stabilizationWindowSeconds: 300 } - scaleDown: { stabilizationWindowSeconds: 1800, policies: [{ type: Pods, value: 1, periodSeconds: 600 }] } + minReplicaCount: 3 # max(minReplicas, maxSyncReplicas+1, 2) + maxReplicaCount: 6 + advanced: + horizontalPodAutoscalerConfig: + behavior: + scaleUp: { stabilizationWindowSeconds: 300 } + scaleDown: { stabilizationWindowSeconds: 1800, policies: [{ type: Pods, value: 1, periodSeconds: 600 }] } + triggers: + - type: prometheus + metadata: + serverAddress: http://vmselect-shortterm.tenant-root.svc:8481/select/0/prometheus + query: + threshold: "150" # AverageValue ⇒ desired = ceil(value/150) = 1 + ceil(Σ/150) ``` -When `autoscaling.enabled` is false and no policy exists, nothing changes — the chart templates `replicas` exactly as today. +When `autoscaling.enabled` is false, nothing changes — the chart templates `replicas` exactly as today. ## Upgrade and rollback compatibility -- **Opt-in and off by default.** The chart conditional is inert unless `autoscaling.enabled` is set; the guard and metrics adapter are optional platform packages. Existing clusters are unaffected. -- **Enabling autoscaling on an existing database — the one real migration, and it needs a deterministic two-phase order.** Flipping `autoscaling.enabled` removes `instances` from the rendered CR, and Helm's three-way merge deletes a key present in the old manifest and absent from the new one **regardless of who last wrote it** — so simply pre-setting `.spec.instances` through the scale subresource does **not** save it: the upgrade deletes the field, CNPG defaults to **1 instance**, and the HPA only re-raises it after CNPG has already begun removing standbys. A safe rollout therefore needs a real two-phase design — e.g. a transition window in which the chart templates `.spec.instances` as a floor (rendered in both the old and new manifest so three-way merge never sees it disappear) while the HPA takes over, then a second phase that drops the floor once the HPA is the established writer. The precise operation order must be worked out and exercised on a dev cluster before MVP; "must be tested" is a gate, not the mechanism. +- **Opt-in and off by default.** The chart conditional is inert unless `autoscaling.enabled` is set; KEDA and the alert/dashboard bundle are optional platform packages. Existing clusters are unaffected. +- **Enabling autoscaling on an existing database — the one real migration, and it needs a deterministic two-phase order.** Flipping `autoscaling.enabled` removes `instances` from the rendered CR, and Helm's three-way merge deletes a key present in the old manifest and absent from the new one **regardless of who last wrote it** — so simply pre-setting `.spec.instances` through the scale subresource does **not** save it: the upgrade deletes the field, CNPG defaults to **1 instance**, and the HPA only re-raises it after CNPG has already begun removing standbys. A safe rollout therefore needs a real two-phase design — e.g. a transition window in which the chart templates `.spec.instances` as a floor (rendered in both the old and new manifest so three-way merge never sees it disappear) while KEDA takes over, then a second phase that drops the floor once the HPA is the established writer. The precise operation order must be worked out and exercised on a dev cluster before MVP; "must be tested" is a gate, not the mechanism. - **Steady state after migration is correct.** With the field absent from both the previous and the current render, three-way merge leaves the HPA-set `.spec.instances` untouched. -- **Cold start.** Until the HPA takes its first sample it holds at `minReplicas`; a brief window at the floor is expected. -- **Enablement constraint — `minReplicas ≥ 2` changes single-instance footprint.** Enabling autoscaling on a current single-instance Postgres permanently doubles instances (a second replica's PVC and DRBD volume). This is legitimate but must be a conscious enablement decision, not a surprise. +- **Rollback must be count-preserving too.** Setting `autoscaling.enabled: false` re-introduces `instances: {{ .Values.replicas }}` — and `replicas` defaults to `2`, so a naive disable would shrink a live cluster the HPA had grown to, say, 6. The disable path must first observe the current instance count and stage it into `.Values.replicas` (or hold the count via the scale subresource through the transition) before the static field is reintroduced — the mirror image of the enable migration. Only then is it fully reversible; no data migration is involved either way. +- **Cold start.** Until KEDA's HPA takes its first sample it holds at `minReplicaCount`; a brief window at the floor is expected. +- **Enablement constraint — `minReplicas ≥ 2` changes single-instance footprint.** Enabling autoscaling on a current single-instance Postgres permanently doubles instances (a second replica's PVC and DRBD volume). This is legitimate but must be a conscious enablement decision, not a surprise — and it is load-bearing, since at `Σ = 0` the formula yields `desired = 1`. - **Dependent objects.** Consumers that read `.Values.replicas` (dashboards, some tooling) must switch to the observed count. Note the two are distinct: the **engine CR** carries `.status.instances`; the **`WorkloadMonitor`** carries `availableReplicas`/`observedReplicas`/`operational` — do not read a nonexistent `WorkloadMonitor.status.instances`. -- **Rollback.** Set `autoscaling.enabled: false` and delete the policy: the chart resumes templating `replicas` and Flux reconciles it back. Fully reversible; no data migration. ## Security -- **RBAC (much reduced).** The guard needs: read/write its `DatabaseScalingPolicy` and status; create/update/own the rendered `HorizontalPodAutoscaler`; read `workloadmonitors`; read-only HTTP to vmselect. It needs **no** write to `Application`/`HelmRelease`, **no** admission webhook, **no** SSA field manager, and **no** engine-CR writes (the HPA does that through the scale subresource). The tenant needs RBAC only on `databasescalingpolicies`, granted through the platform's aggregated tenant ClusterRoles — **not** on `autoscaling/v2` (which cozystack-basics does not grant, and now need not). -- **Honest note on capability.** An HPA driving a CNPG `Cluster`'s `.spec.instances` scales a resource the tenant has no direct write access to. Because the guard owns the HPA and derives its target from the tenant's own database, this is bounded to the tenant's own workload — but it is a real, if narrow, elevation and is stated here on the record. -- **Multi-tenancy.** The policy and HPA are namespaced and live in the tenant namespace; the metrics adapter injects a mandatory namespace matcher, so no tenant query reads another tenant's series. -- **Blast radius.** No cluster-wide admission webhook — a key regression of the first design is gone; enabling the feature adds no admission hop to unrelated Flux reconciliation. +- **RBAC.** No bespoke controller and no new CRD means no new operator RBAC and no tenant grant on `autoscaling/v2` (cozystack-basics grants none, and none is needed — the tenant edits only its own application values, which it already controls). KEDA ships with its own RBAC to read `ScaledObject`s and to write the engine CRs' `scale` subresource; it is a shared platform component, reviewed once, not per-database. +- **Query scoping by construction.** The PromQL is authored by the chart template with the tenant's namespace and application lineage labels baked in; the tenant supplies only numbers, so there is no path for raw tenant PromQL to read another tenant's series from shared vmselect. +- **Honest note on capability.** Autoscaling a CNPG `Cluster`'s `.spec.instances` moves a knob the tenant has no *direct* write access to; here it is driven only from the tenant's own database load and bounded by the chart-rendered min/max, so the elevation is real but narrow — stated here on the record. +- **Blast radius.** No cluster-wide admission webhook (a key regression of rev1 is gone). The one new platform-wide surface is KEDA claiming the `external.metrics.k8s.io` APIService — a deliberate, reviewed dependency rather than an incidental one. ## Failure and edge cases -- **Replica provisioning latency (stateful reality).** A new CNPG standby does not serve reads immediately: PVC provisioning + base backup/clone + WAL catch-up can take minutes to hours for a large database. `scaleUp.stabilizationWindowSeconds` paces *decisions*, not *readiness*. Worse, cloning a new standby adds WAL-streaming load that *raises* replication lag exactly at scale-up, which can trip the lag brake and freeze further scaling — a feedback loop. The feature is therefore meaningful for read-heavy databases whose working set clones in minutes, not for very large datasets where a clone dominates the load window; during a clone the guard reports the in-progress scale and the brake behavior explicitly rather than issuing more scale-ups. -- vmselect unreachable or metric missing → HPA has no metric and holds the current count (`ScalingActive=False` on the HPA); the guard alerts. No blind scaling. -- Replication lag above threshold with an actively-writing primary → metric clamp freezes scaling both ways until lag recovers past the hysteresis band; an idle primary does not trip the brake. -- Desired count would drop to/below the quorum floor → `minReplicas` holds it; CNPG rejects an unsafe count as backstop. -- Over-quota scale-up → pods fail `ResourceQuota` admission; the CR/HPA surface the unmet count; the guard alerts on a persistently unmet desired. +- **Replica provisioning latency (stateful reality).** A new CNPG standby does not serve reads immediately: PVC provisioning + base backup/clone + WAL catch-up can take minutes to hours for a large database. `scaleUp.stabilizationWindowSeconds` paces *decisions*, not *readiness*. Worse, cloning a new standby adds WAL-streaming load that *raises* replication lag exactly at scale-up, which can trip the lag brake and freeze further scaling — a feedback loop. The feature is therefore meaningful for read-heavy databases whose working set clones in minutes, not for very large datasets where a clone dominates the load window; during a clone the metric/alerts reflect the in-progress scale rather than piling on more scale-ups. +- **Stuck scale-up (unschedulable pod, unbindable PVC, quota-rejected standby).** The HPA keeps `desired` high while the metric stays high; the extra standby sits in `Pending` and an **alert on the persistently unmet desired count** fires for an operator to resolve. Unlike rev1's bespoke operator, there is **no automatic rollback** to the last converged count — a conscious trade: active rollback is genuinely hard to do safely for a database (a slow-but-healthy multi-hour clone is indistinguishable from a stuck one without a fragile deadline), and it was a source of bugs. Pending-plus-alert is the same operator outcome without that machinery. +- vmselect unreachable or metric missing → the HPA has no metric and holds the current count (`ScalingActive=False`); the alert rules fire. No blind scaling. +- Replication lag above threshold with an actively-writing primary → the query clamp freezes scaling both ways until lag recovers past the hysteresis band; an idle primary does not trip the brake. +- Desired count would drop to/below the quorum floor → `minReplicaCount` holds it; CNPG rejects an unsafe count as backstop. - **Read disruption on scale-down.** Removing the highest-ordinal standby gracefully still severs read connections pinned to it through `-ro`. Clients must tolerate reconnection; connection draining / graceful client failover is a known limitation to document for tenants (and a candidate follow-up). - MariaDB whose chart lacks scale-out support (`replication.replica.bootstrapFrom` unset) → operator rejects on-the-fly scale-out (`MariaDBScaleOutError`); MariaDB stays out of the enabled set until the chart is fixed. -- Redis / MongoDB → no scale subresource; rejected by the guard with a clear reason (deferred to the shim follow-up). +- Redis / MongoDB → no scale subresource; the chart does not render a `ScaledObject` for them (deferred to the shim follow-up). - Sharded engine (ClickHouse, sharded MongoDB) → out of scope; not autoscalable. ## Testing -- **PoC first — validate the metric encoding (§1) against a real HPA:** confirm the standby-`Lᵢ` / primary-`target` Custom (Pods) series makes stock HPA compute `desiredInstances = desiredRead + 1`, and that `ceil` boundaries behave. This gates everything else. -- **Unit:** the replica-model math and the quorum/lag logic in the guard, with mocked VictoriaMetrics. -- **Chart:** `helm template` with `autoscaling.enabled: true` omits the replica field; with it false, renders `replicas` exactly as today (regression guard). -- **Migration (dev cluster, CNPG):** exercise the two-phase enable on a running multi-instance cluster and assert it does **not** collapse to 1 instance, then drive load and confirm HPA scales `.spec.instances`, reads route to `-ro`, and Flux does not revert. This replaces the first revision's force-writer ownership envtest, which is no longer meaningful — there is no ownership to enforce. -- **Guard integration:** lag above threshold with active writes freezes scaling both ways and releases only past the hysteresis band; quorum floor tracks `maxSyncReplicas`; scale-down removes one standby per `periodSeconds`; `dryRun` reports without creating an HPA. -- **Negative:** vmselect down → no scaling; idle primary with high lag-seconds → no false brake; MariaDB without scale-out → rejected; Redis → rejected. +- **PoC first — validate the single-value metric (§1) against a real HPA:** confirm the `Σ + target` query with an `AverageValue` threshold drives a KEDA-managed HPA to `1 + ceil(Σ/target)` across the `ceil` boundaries, including `Σ = 0 → 1` clamped up by `minReplicaCount`. This gates everything else. +- **Chart:** `helm template` with `autoscaling.enabled: true` omits the replica field and renders a well-formed `ScaledObject` (bounds = `max(minReplicas, maxSyncReplicas+1, 2)`, scale-down policy present, query scoped to the app's namespace/labels); with it false, renders `replicas` exactly as today (regression guard). +- **Migration (dev cluster, CNPG):** exercise the two-phase enable on a running multi-instance cluster and assert it does **not** collapse to 1 instance; then drive load and confirm the HPA scales `.spec.instances`, reads route to `-ro`, and Flux does not revert. Exercise the disable path and assert it does **not** shrink the live cluster to the default `replicas`. +- **KEDA integration:** lag above threshold with active writes freezes scaling both ways and releases only past the hysteresis band; raising `maxSyncReplicas` re-renders the floor; scale-down removes one standby per `periodSeconds`. +- **Negative:** vmselect down → no scaling; idle primary with high lag-seconds → no false brake; MariaDB without scale-out → no `ScaledObject`; Redis → no `ScaledObject`. ## Rollout -1. **PoC** — CNPG on a dev cluster: chart conditional + guard-rendered HPA on `.spec.instances` driven by the synthesized read-load metric; validate the arithmetic and that Flux does not revert. -2. **MVP** — PostgreSQL: the chart change, the custom-metrics adapter (namespace-scoped, lag-clamp), the guard + `DatabaseScalingPolicy` (quorum floor, lag brake, scale-down pacing, dry-run), dashboard surface and alerts. +1. **PoC** — CNPG on a dev cluster: chart conditional + a `ScaledObject` with the `Σ + target` query; validate the arithmetic, the lag clamp, and that Flux does not revert. +2. **MVP** — PostgreSQL: KEDA added as a platform package, the chart change, the cozy-lib helper that renders the `ScaledObject`, the `autoscaling` values block + schema, and the dashboard/alert bundle. 3. **MariaDB** — once the cozystack mariadb chart supports on-the-fly scale-out. 4. **Redis / MongoDB** — a follow-up proposal for a thin actuation shim, since neither exposes a scale subresource. ## Open questions -- Which implementation backs the custom-metrics adapter (prometheus-adapter, a KEDA metrics apiserver, or purpose-built) — constrained by the Custom (Pods) choice in §1 and by the lag-clamp requirement. -- Exact two-phase migration mechanic (chart-templated floor during transition vs a staged operator-driven handover), to be settled on a dev cluster before MVP. +- Final shape of the lag-clamp query (how `currentInstances` is sourced — pod count vs HPA status series) and the hysteresis recovery band / cooldown — deliberate defaults to be tuned at PoC. +- Exact two-phase enable/disable migration mechanic (chart-templated floor during transition vs a staged handover), to be settled on a dev cluster before MVP. - Default driver metric (read connections vs read QPS vs replica CPU), to be calibrated on real workloads. -- `periodSeconds` for scale-down pacing and the hysteresis recovery band — deliberate defaults to be tuned. +- KEDA packaging in cozystack (version, HA, which APIService/metrics-server coexistence concerns) — it is the one new platform singleton and needs an owner. ## Alternatives considered -- **A bespoke `db-autoscaler` operator owning `replicas` (the first revision).** Rejected after the implementation spike (see Appendix). It re-drew HPA's API surface field-for-field and re-implemented its decision loop, and its ownership guarantee proved unbuildable on the aggregated apps API. This design keeps HPA's hardened loop and confines net-new code to the brakes HPA lacks. -- **HPA writing the `Application`'s `replicas` value (apps API) instead of the engine CR.** This is what the first revision did; it is the source of the whole ownership problem, because the apps values are declared in Git and reverted by Flux. Writing the engine CR's scale subresource while the chart omits the field avoids the conflict at its root. -- **A guard that pins `min`/`maxReplicas` on a tenant-declared HPA.** Rejected: it relocates the revert war from `replicas` to the HPA spec. Having the guard *own* the HPA (this design) removes the second writer entirely. -- **External metric instead of Custom (Pods).** Rejected: External `AverageValue` has no pod divisor, so it cannot express the read-replica model without off-by-primary errors; the Custom (Pods) encoding makes the model fall out of stock HPA arithmetic. -- **A thin actuation shim for engines without a scale subresource (Redis, MongoDB).** For these, HPA cannot act directly; a minimal shim watching a stock HPA's recommendation behind the same brakes is the honest path — deferred to a follow-up. -- **Stock HPA + KEDA with tenant-supplied PromQL.** Rejected for the metric layer: raw tenant PromQL against shared vmselect breaks isolation. A KEDA/prometheus-adapter trigger is acceptable only with a platform-injected mandatory namespace matcher. +- **A bespoke `db-autoscaler` operator owning `replicas` (rev1).** Rejected after the implementation spike (see Appendix): it re-drew HPA's API surface field-for-field, re-implemented its decision loop, and its ownership guarantee proved unbuildable on the aggregated apps API. +- **A thin guard controller + `DatabaseScalingPolicy` CRD rendering the HPA (rev3).** Rejected: even a guard that *owns* the HPA is still a runtime writer of an object's spec, and it re-grew most of the old CRD's fields. Rendering a KEDA `ScaledObject` from the chart is fully declarative (Flux-owned, no runtime spec writer) and needs no controller or API group at all. +- **HPA writing the `Application`'s `replicas` value (apps API) instead of the engine CR.** This is what rev1 did; it is the source of the whole ownership problem, because the apps values are declared in Git and reverted by Flux. Writing the engine CR's scale subresource while the chart omits the field avoids the conflict at its root. +- **prometheus-adapter as the metric backend.** Rejected (§4): global-ConfigMap queries need per-app registration + reload, and a single upstream cannot reach each tenant's vmselect. +- **kube-metrics-adapter (Zalando).** A lighter alternative to KEDA (query in HPA annotations), kept in reserve; smaller project and weaker per-tenant-server support. +- **A per-pod Custom (Pods) metric (rev3).** Correct but needless: it required the adapter to emit one sample per pod (primary = target, zero-filled standbys) purely to make the average equal `(Σ + target)/N`. Serving the aggregate `Σ + target` as an External/Object `AverageValue` is exactly equivalent and needs no per-pod emission — which is why an External metric is the mechanism here, not the off-by-primary hazard an earlier revision ascribed to it. +- **A thin actuation shim for engines without a scale subresource (Redis, MongoDB).** For these, an HPA cannot act directly; a minimal shim watching a stock HPA's recommendation behind the same brakes is the honest path — deferred to a follow-up. - **Scaling the write path via sharding.** Out of scope: requires data rebalancing, an orchestrated procedure rather than a replica-count change. ## Appendix: Findings from the implementation spike From 5f4b6b6d392da055d198b4ca1094a986c81a59b5 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Fri, 31 Jul 2026 18:53:37 +0300 Subject: [PATCH 5/5] =?UTF-8?q?docs(dha):=20address=20branch-review=20?= =?UTF-8?q?=E2=80=94=20commit=20two-phase=20migration,=20consistency=20nit?= =?UTF-8?q?s?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - §Upgrade: commit the enable AND disable migration to an explicit two-phase order in prose (observe live count -> stage -> stand up under a floor -> hand the field off via SSA field-manager release), leaving only the SSA-vs-helm-merge handoff detail as the single PoC item; the disable mirror stages .Values.replicas to the live count so it cannot shrink a running cluster - fix the rendered ScaledObject name (postgres-db, not db) to match its scaleTargetRef and the release name - make the minReplicaCount=3 example reproducible (note maxSyncReplicas=2) - §4: reconcile the per-object serverAddress argument with the shared-root vmselect example (MVP reads the root vmselect scoped by label; per-object serverAddress enables per-tenant stacks later) - PoC checklist: verify spec.subresources.scale on the pinned CNPG version Signed-off-by: Alexey Artamonov --- .../database-horizontal-autoscaling/README.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/design-proposals/database-horizontal-autoscaling/README.md b/design-proposals/database-horizontal-autoscaling/README.md index 7f570fa..a1630a4 100644 --- a/design-proposals/database-horizontal-autoscaling/README.md +++ b/design-proposals/database-horizontal-autoscaling/README.md @@ -90,7 +90,7 @@ The conditional keys off `autoscaling.enabled`, **not** off presence of the fiel An HPA object cannot carry a query — its metric spec holds only a name and a selector — so the query must live where the metrics-API backend reads it, and the options differ sharply: - **prometheus-adapter — ruled out.** Its queries live in one global ConfigMap, so a per-application query means per-application adapter config plus a reload — a registration step for every database. It also speaks to a single upstream URL, while every tenant's metrics live behind a different vmselect. -- **KEDA — recommended.** The query lives inline in a namespaced `ScaledObject` that the chart renders exactly where it would have rendered an HPA; there is no global config and no registration step, and KEDA generates and manages the HPA itself. Everything this design needs passes through: `scaleTargetRef` accepts any CR with a `scale` subresource (CNPG `Cluster` qualifies), `minReplicaCount`/`maxReplicaCount` take the template-computed bounds, `advanced.horizontalPodAutoscalerConfig.behavior` carries the scale-down policies verbatim, and `serverAddress` is per-object — so each tenant's `ScaledObject` points at its own vmselect, which a single-upstream adapter cannot do. +- **KEDA — recommended.** The query lives inline in a namespaced `ScaledObject` that the chart renders exactly where it would have rendered an HPA; there is no global config and no registration step, and KEDA generates and manages the HPA itself. Everything this design needs passes through: `scaleTargetRef` accepts any CR with a `scale` subresource (CNPG `Cluster` qualifies), `minReplicaCount`/`maxReplicaCount` take the template-computed bounds, `advanced.horizontalPodAutoscalerConfig.behavior` carries the scale-down policies verbatim, and `serverAddress` is per-object. In the MVP every `ScaledObject` reads the shared root vmselect (`vmselect-shortterm.tenant-root.svc`) with the query scoped by namespace/lineage labels; the per-object `serverAddress` is the property that lets a tenant with its own isolated monitoring stack point at its own vmselect later without any central reconfiguration — the thing a single-upstream adapter cannot do. - **kube-metrics-adapter (Zalando)** is the lighter alternative — the query lives in annotations on the HPA — but it is a much smaller project and its per-tenant-server story is weaker. Because the query is authored by the chart template (the tenant supplies only numbers through values), the mandatory-scoping rule — no raw tenant PromQL against shared vmselect — is satisfied by construction. The cost is that **KEDA becomes a new platform component**: a cluster-singleton that claims the `external.metrics.k8s.io` APIService (nothing serves it in Cozystack today), shared by any future feature that needs custom-metric autoscaling. @@ -131,10 +131,10 @@ The chart renders (reference only — the tenant never authors this): ```yaml apiVersion: keda.sh/v1alpha1 kind: ScaledObject -metadata: { name: db, namespace: tenant-acme } +metadata: { name: postgres-db, namespace: tenant-acme } # both rendered from the release name spec: scaleTargetRef: { apiVersion: postgresql.cnpg.io/v1, kind: Cluster, name: postgres-db } - minReplicaCount: 3 # max(minReplicas, maxSyncReplicas+1, 2) + minReplicaCount: 3 # max(minReplicas=2, quorum.maxSyncReplicas+1, 2); =3 here with maxSyncReplicas=2 maxReplicaCount: 6 advanced: horizontalPodAutoscalerConfig: @@ -154,9 +154,9 @@ When `autoscaling.enabled` is false, nothing changes — the chart templates `re ## Upgrade and rollback compatibility - **Opt-in and off by default.** The chart conditional is inert unless `autoscaling.enabled` is set; KEDA and the alert/dashboard bundle are optional platform packages. Existing clusters are unaffected. -- **Enabling autoscaling on an existing database — the one real migration, and it needs a deterministic two-phase order.** Flipping `autoscaling.enabled` removes `instances` from the rendered CR, and Helm's three-way merge deletes a key present in the old manifest and absent from the new one **regardless of who last wrote it** — so simply pre-setting `.spec.instances` through the scale subresource does **not** save it: the upgrade deletes the field, CNPG defaults to **1 instance**, and the HPA only re-raises it after CNPG has already begun removing standbys. A safe rollout therefore needs a real two-phase design — e.g. a transition window in which the chart templates `.spec.instances` as a floor (rendered in both the old and new manifest so three-way merge never sees it disappear) while KEDA takes over, then a second phase that drops the floor once the HPA is the established writer. The precise operation order must be worked out and exercised on a dev cluster before MVP; "must be tested" is a gate, not the mechanism. +- **Enabling autoscaling on an existing database — the one real migration.** The hazard: flipping `autoscaling.enabled` removes `instances` from the rendered CR, and Helm's three-way merge deletes a key present in the old manifest and absent from the new one **regardless of who last wrote it**, so a naive flip deletes the field, CNPG defaults to **1 instance**, and the HPA only re-raises it after CNPG has already begun removing standbys. The committed two-phase order that avoids this: **(phase 1 — stand up under a floor, no field removal)** the operator reads the live `.status.instances` (= N) and sets `.Values.replicas = N` (a no-op to the running cluster); then sets `autoscaling.enabled: true` with a transition sub-flag that keeps the chart rendering `instances: {{ .Values.replicas }}` (= N) **alongside** the new `ScaledObject` (whose `minReplicaCount` is pinned to N). The field never leaves the manifest, so three-way merge never deletes it; Flux and the HPA both target N, so neither fights; KEDA comes up healthy and begins observing load. **(phase 2 — hand the field off)** once the `ScaledObject`/HPA is Ready, clear the transition flag so the chart stops rendering `instances`. This is the single present→absent transition, and it is safe only if Flux relinquishes its claim on `.spec.instances` while the HPA keeps writing it — i.e. the handoff must ride on server-side-apply field-manager ownership (Flux drops the field from *its* managed-fields; the HPA's scale-subresource writes keep the value alive), not on Helm's classic three-way delete. Confirming that the platform's Flux/helm-controller path performs this as an SSA release rather than a hard delete — and, if it does not, pinning `.spec.instances` via the scale subresource across the phase-2 apply as a fallback — is the one migration detail the PoC must settle. Steady state after phase 2 is safe: with the field absent from every subsequent render, three-way merge leaves the HPA-managed value untouched. - **Steady state after migration is correct.** With the field absent from both the previous and the current render, three-way merge leaves the HPA-set `.spec.instances` untouched. -- **Rollback must be count-preserving too.** Setting `autoscaling.enabled: false` re-introduces `instances: {{ .Values.replicas }}` — and `replicas` defaults to `2`, so a naive disable would shrink a live cluster the HPA had grown to, say, 6. The disable path must first observe the current instance count and stage it into `.Values.replicas` (or hold the count via the scale subresource through the transition) before the static field is reintroduced — the mirror image of the enable migration. Only then is it fully reversible; no data migration is involved either way. +- **Disabling must be count-preserving too — the mirror sequence.** Setting `autoscaling.enabled: false` re-introduces `instances: {{ .Values.replicas }}`, and `replicas` defaults to `2`, so a naive disable would shrink a live cluster the HPA had grown to, say, 6. The committed order: **(phase 1)** the operator reads the live `.status.instances` (= M, the count the HPA is currently holding) and sets `.Values.replicas = M`; **(phase 2)** clears `autoscaling.enabled` and deletes the `ScaledObject` in the same apply — the chart re-renders `instances: M`, which matches the live count, so Flux reasserts the current value rather than dropping to the default. Only with `.Values.replicas` staged to the live count first is the disable a no-op to the running cluster; no data migration is involved either way. - **Cold start.** Until KEDA's HPA takes its first sample it holds at `minReplicaCount`; a brief window at the floor is expected. - **Enablement constraint — `minReplicas ≥ 2` changes single-instance footprint.** Enabling autoscaling on a current single-instance Postgres permanently doubles instances (a second replica's PVC and DRBD volume). This is legitimate but must be a conscious enablement decision, not a surprise — and it is load-bearing, since at `Σ = 0` the formula yields `desired = 1`. - **Dependent objects.** Consumers that read `.Values.replicas` (dashboards, some tooling) must switch to the observed count. Note the two are distinct: the **engine CR** carries `.status.instances`; the **`WorkloadMonitor`** carries `availableReplicas`/`observedReplicas`/`operational` — do not read a nonexistent `WorkloadMonitor.status.instances`. @@ -182,7 +182,7 @@ When `autoscaling.enabled` is false, nothing changes — the chart templates `re ## Testing -- **PoC first — validate the single-value metric (§1) against a real HPA:** confirm the `Σ + target` query with an `AverageValue` threshold drives a KEDA-managed HPA to `1 + ceil(Σ/target)` across the `ceil` boundaries, including `Σ = 0 → 1` clamped up by `minReplicaCount`. This gates everything else. +- **PoC first — validate the single-value metric (§1) against a real HPA:** confirm the `Σ + target` query with an `AverageValue` threshold drives a KEDA-managed HPA to `1 + ceil(Σ/target)` across the `ceil` boundaries, including `Σ = 0 → 1` clamped up by `minReplicaCount`. Also confirm the pinned CloudNativePG version actually exposes `spec.subresources.scale` on `Cluster.spec.instances` (the assumption the whole mechanism rests on — present in the currently vendored CNPG, but version-sensitive). This gates everything else. - **Chart:** `helm template` with `autoscaling.enabled: true` omits the replica field and renders a well-formed `ScaledObject` (bounds = `max(minReplicas, maxSyncReplicas+1, 2)`, scale-down policy present, query scoped to the app's namespace/labels); with it false, renders `replicas` exactly as today (regression guard). - **Migration (dev cluster, CNPG):** exercise the two-phase enable on a running multi-instance cluster and assert it does **not** collapse to 1 instance; then drive load and confirm the HPA scales `.spec.instances`, reads route to `-ro`, and Flux does not revert. Exercise the disable path and assert it does **not** shrink the live cluster to the default `replicas`. - **KEDA integration:** lag above threshold with active writes freezes scaling both ways and releases only past the hysteresis band; raising `maxSyncReplicas` re-renders the floor; scale-down removes one standby per `periodSeconds`. @@ -198,7 +198,7 @@ When `autoscaling.enabled` is false, nothing changes — the chart templates `re ## Open questions - Final shape of the lag-clamp query (how `currentInstances` is sourced — pod count vs HPA status series) and the hysteresis recovery band / cooldown — deliberate defaults to be tuned at PoC. -- Exact two-phase enable/disable migration mechanic (chart-templated floor during transition vs a staged handover), to be settled on a dev cluster before MVP. +- The two-phase enable/disable order is committed in §Upgrade; the one detail left for the PoC is whether the phase-2 field handoff rides on Flux/helm-controller SSA field-manager release (preferred) or needs the scale-subresource-pin fallback. - Default driver metric (read connections vs read QPS vs replica CPU), to be calibrated on real workloads. - KEDA packaging in cozystack (version, HA, which APIService/metrics-server coexistence concerns) — it is the one new platform singleton and needs an owner.