feat(helm): add HStore deployment chart

[bitflicker64]

Purpose of the PR

• close [Feature] Add an official Helm chart for distributed HStore deployment #3131

Add an official Helm installation path for the distributed HugeGraph HStore
topology on Kubernetes. The chart packages the PD, Store, and Server startup
contract that otherwise has to be reconstructed by each operator.

All six image-side prerequisites have now merged, including
#3130 (2026-08-15). This
branch already contains that merge (84c56fc7) and is not behind current
master. The PR stays a draft until the remaining items under "Remaining
before this leaves draft" are closed.

Main Changes

• Add the first helm/hugegraph chart with chart version 0.1.0:
  • PD and Store StatefulSets with persistent storage and headless Services.
  • Server Deployment and client Service.
  • PD, Store, and optional Server PodDisruptionBudgets.
  • Optional Server HPA and Ingress.
  • A helm test connection hook.
• Add values.schema.json with render-time validation for replica counts,
  storage, PDBs, HPA, authentication, Services, and other value contracts.
• Add default 3+3+3, single-node, and production-oriented values presets.
• Add scheduling controls, per-component ServiceAccounts with token mounting
  disabled by default, configurable pod security contexts, hardened container
  security defaults, graceful termination periods, probes, and extraEnv.
• Track the latest PD, Store, Server, and Hubble images while this PR is a
  draft, using pull policy Always so nodes do not retain an older cached
  image.
• Add Helm lint, render, invalid-value, kubeconform, legacy-values, and
  packaging CI.
• Add chart installation, configuration, upgrade, troubleshooting, and
  limitations documentation, plus a root README installation link.
• Enable Server authentication by default. A kept chart-managed
  <release>-admin Secret supplies the password unless
  server.auth.admin.existingSecret or server.auth.admin.password is set;
  a matching <release>-auth-token Secret shares auth.token_secret across
  Server replicas. Admin password and JWT token each have separate value vs
  Secret keys. Set server.auth.enabled=false only for trusted networks.
• Add an optional Hubble UI component (off by default): a single-replica
  Deployment with a Recreate strategy, ClusterIP Service, optional Ingress
  and persistence, hubble.mode (pd for PD discovery and the operations
  view, direct for the Server Service only), probes, schema validation,
  render-time guards, documentation, and CI coverage. Current Hubble images
  authenticate their login against the cluster, so rendering Hubble without
  server.auth fails unless explicitly overridden.
• Support outside Hubble PD discovery: server.advertiseUrl registers a
  reachable Server URL with PD, and the PD client Service can be exposed as
  NodePort or LoadBalancer. Empty advertiseUrl keeps the in-cluster Service
  URL. Render fails closed when advertiseUrl is set without PD meta mode.
• Fix the auth gap flagged in review: the Server wrapper writes
  auth.admin_pa from the auth Secret alongside usePD and pd.peers, and
  rejects Secret values containing properties-parser metacharacters.
• Run every chart-managed Server in distributed PD metadata mode. The
  wrapper always writes usePD=true, chart-derived pd.peers,
  server.urls_to_pd (the Server client Service URL, or advertiseUrl when
  set), and server.deploy_in_k8s=true. This keeps the graph catalog shared
  across Server replicas and gives discovery clients such as Hubble a
  resolvable Server address.
• The Hubble wrapper writes server.host=0.0.0.0 because current Hubble
  images bind server.host (default localhost) and ignore the legacy
  hubble.host key in the shipped configuration.
• Harden the CI invalid-value step so every rejection case is enforced
  rather than only the last line, and add positive renders for both Hubble
  modes.
• Default partition sharding to store-level HA: fresh installs seed PD with
  partition.default-shard-count derived from the Store count (3 when
  store.replicas is at least 3, otherwise 1, matching PD's odd-only
  constraint), replacing the image default of one shard replica per
  partition. The seed applies at first bootstrap only; PD metadata is
  authoritative afterwards, and the resulting initial partition count
  change is documented.
• Default pd.antiAffinity and store.antiAffinity to preferred so the
  chart schedules on clusters with fewer nodes than replicas;
  values-cluster.yaml keeps required and NOTES warns when PD quorum
  members may co-locate.
• Rewrite the Disaster Recovery documentation around what current PD builds
  actually do (manual /v1/task/patrolPartitions reconciliation, no
  automatic re-replication), and reject JAVA_OPTIONS in extraEnv for
  pd, store, and server because the start scripts drop the chart-managed
  JAVA_OPTS when it is set.

Distributed startup and recovery contracts

• The chart creates no init Job and does not use HG_SERVER_SKIP_INIT. It sets
  HG_SERVER_INIT_STORE_ENABLED=false, relying on the dedicated
  init_store.enabled gate from fix(dist): gate init-store on a dedicated init_store.enabled option #3119 (issue InitStore should be able to skip local init in PD/HStore deployments #3118) so concurrent Server
  replicas do not initialize the same distributed backend.
• Store waits for a configured number of PD REST /v1/health responses before
  starting. That is process liveness, not a Raft-leader check; the remaining
  review thread on this signal is listed below.
• Every Server replica uses PD-backed graph metadata. The chart wrapper
  always writes usePD=true, pd.peers, server.urls_to_pd, and
  server.deploy_in_k8s=true before handing control back to the image
  entrypoint. This is required for distributed HStore so replicas share one
  graph catalog. It does not make a local RocksDB backend shared or close the
  cross-replica CreateGraph readiness window tracked in [Feature] Orchestrate graph creation through PD: new graphs are not consistently available across Server replicas ("Could not rebind [g]") #3137. When
  authentication is enabled (the default), the same wrapper writes
  auth.admin_pa from the mounted Secret. That value applies only at first
  creation and lands in rest-server.properties (mode 600) inside the
  container.
• Resource names reserve suffix and StatefulSet ordinal space before
  truncation, keeping names valid and PVC identities stable across scaling.
• Server startup allows at least 450 seconds for the image's storage wait and
  startup command.

Prerequisites

All six image-side prerequisites have merged:

PR	Required behavior	Merged as
#3105	Bounded port preflight without lsof	1716c774
#3119	Dedicated init_store.enabled gate	39f4f856
#3126	Finite JVM DNS cache TTL, plus startup DNS-policy validation and JDK 24+ security-check handling	de62d97f
#3128	One gRPC stub binding per channel	b026a90a
#3129	Store readiness retries across all configured PD peers	8b2932c7
#3130	Channel and stub refresh after address changes	431f6e6b

This branch includes #3130 via merge(master): bring in #3130 Store channel refresh (84c56fc7) and currently contains all of apache/hugegraph
master. The Kubernetes validation below used the published Docker Hub
:latest PD, Store, Server, and Hubble images.

#3138 (CreateGraph Phase 1 of
#3137) merged on master
(2026-08-12). It is not a chart prerequisite but affects multi-Server behavior
documented in the chart README. #3139
remains open for cluster-wide graph readiness.

#3126's finite DNS TTL is required for #3130 to resolve a replacement Store
address. Neither change provides complete address recovery alone.

#3119 was narrowed before merge to the init_store.enabled option, its gate,
the env mapping, and the init-flag guard. The Docker entrypoint and
auth-bootstrap work that previously shared its branch is tracked separately in
#3133. That split does not change what this chart uses from the merged PR, but
it does mean the entrypoint's PASSWORD and auth.admin_pa handling is a
known-imperfect contract this chart depends on.

Validation baseline and head drift

The runtime matrix below was executed against a composition built from these
exact heads:

PR	Head under test	Status now
#3105	ee8f5559	merged as 1716c774; tested head superseded
#3119	3e505a81	merged as 39f4f856; tested head superseded and scope narrowed
#3126	35e1a240	merged as de62d97f; tested head superseded
#3128	0118e158	merged as b026a90a from this PR head
#3129	575a1122	merged as 8b2932c7 from head 61407213, one commit further
#3130	26218cb3	merged as 431f6e6b; tested head superseded

Only #3128 merged at exactly the head this chart was validated against. The
table below is the original evidence baseline (heads under test at the time).
All six prerequisites are now on master. Results are not a claim about today's
master until the committed matrix rerun completes.

The auth.admin_pa fix and the Hubble component were validated separately
against a newer composition: master at 1716c774 (which already carries
#3105, #3128, and #3129) merged with #3119 at edf07d0f (superseded before
merge), #3126 at b40c42fb (the merged PR head), and #3130 at 198de19e
(a later superseded head). Evidence from that composition:

• Fresh auth-enabled single-node install with Hubble enabled: all Pods
  Ready with zero restarts and init_store.enabled=false throughout (no
  fail-closed refusal).
• Server auth: unauthenticated requests and the default admin:pa both get
  401; the Secret credential gets 200.
• Hubble: the UI serves on 8088; the login fails closed on a wrong password
  and succeeds with the Secret credential in both pd and direct modes;
  an authenticated session reads /graphspaces from the cluster through PD
  discovery; with persistence enabled, the H2 metadata lands on the PVC.
• A fresh non-auth install with Hubble in pd mode confirmed the PD-meta
  properties are written and discovery reaches the Server; it also showed
  current Hubble images cannot complete their login against an auth-less
  cluster (the server rejects /auth/login with "Unconfigured
  authenticator"), which the chart now surfaces as a render-time guard.
• Three independent review passes on the new work found two high-severity
  issues (both fixed and re-verified): the CI invalid-value step enforced
  only its last line, and the PD-meta properties were scoped to unrelated
  settings.

Later chart commits (auth-on by default, auto-generated Secrets, nested
admin/token keys, server.advertiseUrl, and the #3130 master merge) are on
this branch. Additional Kubernetes validation against this PR head is recorded
below. The full multi-node lifecycle matrix rerun remains pending before this
PR leaves draft.

Kubernetes validation (2026-08-17)

Against this PR head (84c56fc7) on Kubernetes (Kind), with authentication
enabled by default, Hubble enabled in pd mode, and the published Docker Hub
images hugegraph/pd:latest, hugegraph/store:latest,
hugegraph/server:latest, and hugegraph/hubble:latest (digests pulled the
same day).

Distributed topology (3 PD + 3 Store + 3 Server + 1 Hubble)

• All ten Pods reached Ready with zero restarts; every component container
  used the :latest tag above; helm test passed.
• Unauthenticated and wrong-password access to protected APIs returned 401;
  the chart admin Secret and JWT login succeeded.
• The same JWT was accepted by all three Server Pods (shared
  auth.token_secret).
• Every Server Pod rendered usePD=true, the three PD peers,
  server.urls_to_pd, server.deploy_in_k8s=true, and
  StandardAuthenticator with auth.admin_pa from the chart Secret.
• REST create/read/update/delete on an HStore graph succeeded.
• Hubble login used the chart admin Secret (bad passwords rejected); graph
  spaces and graphs were visible through PD discovery; vertex create and
  gremlin-query succeeded.
• Hubble cluster overview showed three healthy PD members (one leader) and
  three healthy Stores. The overview lists a single Server endpoint because
  all Server replicas register the shared client Service URL with PD; that is
  the chart registration contract, not a missing replica. The overview may
  report an aggregate PD status of UNKNOWN while individual PD members remain
  UP.

An earlier single-node smoke on the same branch also passed; the authoritative
auth + Hubble check for this update is the 3+3+3+1 run on :latest above.

This does not replace the pending multi-node fault and upgrade lifecycle
matrix.

Remaining before this leaves draft

1. Done: bring refactor(store): recover retries after store replacement #3130 onto this branch (84c56fc7); PR head is not behind
   current master.
2. Rerun the full multi-node 3+3+3 lifecycle matrix against current
   master, covering upgrade, Server/Store replacement, and PD failover for
   the auth-default, Hubble, and advertiseUrl work. Single-node and
   3+3+3+1 Kubernetes smokes are already recorded above.
3. Pin appVersion and component image tags (including Hubble) to the
   next HugeGraph release and switch pull policies to IfNotPresent. There is
   no 1.8.x release tag yet, so the 0.1.0 chart still tracks latest.
4. Remaining review thread: Store's PD wait and PD/Store probes still use
   /v1/health (process liveness, not a Raft-leader/quorum signal). PDB
   quorum-floor and extraEnv reserved-name findings are fixed in the chart.

Resolved on this branch

• auth.admin_pa bootstrap — the Server wrapper writes auth.admin_pa from
  the auth Secret alongside usePD and pd.peers, rejecting values with
  properties-parser metacharacters. Verified with fix(dist): gate init-store on a dedicated init_store.enabled option #3119 on master: auth-enabled
  installs reach Ready with init_store.enabled=false; the Secret credential
  works end to end. auth.admin_pa applies only at first creation and lands in
  a mode-600 file inside the container (documented in the chart README).
• PD PDB quorum floor — when the PD PDB is enabled, render requires
  minAvailable >= floor(replicas/2)+1.
• extraEnv cannot override chart contracts — reserved names such as
  HG_SERVER_INIT_STORE_ENABLED and PASSWORD are rejected at render time.
• Auth on by default with auto-generated kept admin and JWT Secrets.
• Outside Hubble PD discovery via server.advertiseUrl and an exposable
  PD client Service.

Verifying these changes

• Trivial rework / code cleanup without any test coverage. (No Need)
• Already covered by existing tests, such as (please modify tests here).
• Need tests and can be verified as follows:
  • Static chart validation: PASS=239 FAIL=0 SKIP=0.
  • Default stock-values rendering now asserts that the Server wrapper writes
    usePD=true and chart-derived pd.peers; local wrapper execution also
    verified stale local values are replaced while
    HG_SERVER_INIT_STORE_ENABLED=false remains rendered.
  • Edge-case and hardening suite: PASS=37 FAIL=0.
  • Faithful legacy --reuse-values: helm template and
    helm install --dry-run=client passed.
  • Fresh 3+3+3 install: 9 Ready Pods, zero restarts, cluster 61/0, dataset
    30/0.
  • Same-package upgrade: cluster 61/0, dataset 30/0.
  • Server replacement: PASS=91 FAIL=0.
  • Store replacement: PASS=90 FAIL=0 with data preserved.
  • Deterministic PD REST failover: PASS=7 FAIL=0, with restoration verified.
  • Final cluster and dataset gates: 61/0 and 30/0.
  • Independent review passes 6, 7, and 8 reported no unresolved actionable
    findings on the final template and schema implementation.
  • The subsequent chart-version and default-image metadata amendment changed
    no templates or schema and repeated the full static gate at 239/0/0.
  • Kubernetes validation on this PR head (2026-08-17) using published
    hugegraph/{pd,store,server,hubble}:latest images: 3+3+3+1 with auth
    enabled; Ready Pods with zero restarts; helm test pass; REST and Hubble
    CRUD; shared JWT across Server replicas (details above).

The runtime matrix used a four-node Kubernetes cluster and images built from a
composition containing the exact prerequisite heads listed under "Validation
baseline and head drift". Test-only validation harnesses and lab-specific
low-memory values are excluded from this PR. As noted above, those heads have
since moved (and #3130 has merged), so this matrix will be rerun before the PR
is marked ready.

Does this PR potentially affect the following parts?

• Dependencies (add/update license info & regenerate_known_dependencies.sh)
• Modify configurations
• The public API
• Other affects: adds Kubernetes deployment resources and Helm chart CI
• Nope

Documentation Status

• Doc - TODO
• Doc - Done
• Doc - No Need

Documented operational limits

• While this PR is a draft, appVersion and component image tags (PD, Store,
  Server, Hubble) are latest. This is intentionally temporary and is not the
  stable publication configuration; the next release tag will be pinned before
  the PR is marked ready.
• Default values intentionally leave resource requests and limits unset;
  values-cluster.yaml is a production starting point, not a capacity
  guarantee.
• PD and Store anti-affinity defaults to preferred, so the chart schedules
  on any node count; production should pin required (as
  values-cluster.yaml does) so one node failure cannot take out the PD
  quorum or co-locate shard replicas.
• Authentication is on by default. The admin password is applied from the
  Secret only at first creation; the chart cannot rotate an existing cluster's
  admin password. Disable auth only on a trusted network.
• Hubble is optional and disabled by default: single replica, plain HTTP,
  and its login requires server.auth on current images (the chart refuses
  to render it without auth unless explicitly overridden). Its UI connection
  metadata, including graph credentials entered in the UI, lives in an
  embedded H2 database that persists only when hubble.persistence is
  enabled; the PVC is kept on uninstall.
• server.advertiseUrl is empty by default so PD hands back the in-cluster
  Server Service URL. Setting it for an outside Hubble also changes what
  in-cluster discovery clients receive.
• The published images run as root. The chart hardens the container without
  setting runAsNonRoot or readOnlyRootFilesystem, which the current image
  contract cannot support.
• The entrypoint's property handling for mounted and upgraded configs is
  tracked in [Bug] Docker entrypoint auth bootstrap is unsafe for mounted and upgraded configs #3133 and is outside this chart's control.
• TLS, backup and restore, an Operator, multi-cluster orchestration, automatic
  leader transfer, ConfigMap-based component configuration, and a complete
  monitoring stack are outside this PR.
• Automatic partition leader rebalancing after Store restarts is upstream
  work, tracked in #3135;
  the chart documents the manual balance endpoints as the current workaround.
• Shard recovery observability (sync progress, raft lag metrics) is upstream
  work, tracked in #3136.
• The cross-replica propagation window after CreateGraph is documented as a
  chart limitation, tracked in
  #3137. Phase 1 (minimal
  patch) merged: #3138
  (2026-08-12) — the creating Server is consistent at HTTP 200. Other replicas
  may still lag; mitigations remain in the chart README until Phases 2–3 land
  (#3139 and PD-owned
  creation).

TODO (later, needs more research)

• Rework Hubble delivery to the attach model: a small standalone Hubble
  deployment that joins an already-running cluster without recreating PD,
  Store, or Server, with the Hubble image pinned to the matching HugeGraph
  release. A Compose add-on with this shape is in #3149;
  the same pattern fits Kubernetes well (compare TiDB Operator's standalone
  TidbDashboard resource and the documented Kibana attach flow), so
  evaluate exposing Hubble here as an independently upgradable component
  rather than only a bundled sub-chart option.

[bitflicker64]

Ordering note: this chart depends on #3119 and should land after it.

server-deployment.yaml injects HG_SERVER_INIT_STORE_ENABLED=false, and the mapping from that env var onto init_store.enabled only exists once #3119 merges. If this merges first, the chart ships pointing at a variable the image does not understand, and the server runs init-store on every pod start while the chart's README says it does not.

No file overlap between the two, so there is nothing to rebase — it is purely a merge-order dependency.

[bitflicker64]

Blocking, and it comes from a change in #3119 rather than from this PR's own code.

What changed. With init_store.enabled=false, the built-in authenticator's admin is created on the PD startup path from auth.admin_pa, whose default is the public value pa — while Docker PASSWORD is discarded, since init-store reads it from stdin and the disabled path returns first. init-store now fails closed unless an explicit non-empty auth.admin_pa is configured.

What breaks here. This chart sets HG_SERVER_INIT_STORE_ENABLED=false and supplies PASSWORD from server.auth.existingSecret, but its wrapper writes only usePD and pd.peers — never auth.admin_pa. Once #3119 merges, an auth-enabled deployment fails at container start with Refusing to skip init-store: ... no explicit non-empty 'auth.admin_pa' is configured.

Before that change it did not fail — it came up as admin/pa, silently ignoring the Secret. So this is an existing bug here that #3119 makes visible.

What to do. The wrapper should write auth.admin_pa from the Secret it already mounts as PASSWORD, alongside the usePD and pd.peers lines. That also makes this README line true for the first time:

The image entrypoint keeps ownership of PASSWORD handling and auth.admin_pa.

Two notes while you are in there: auth.admin_pa applies only at first creation, so it will not rotate an existing cluster's password, and it lands in a file inside the container. Merge order is unchanged — this still lands after #3119.

[imbajin]

Blocking: yes. In addition to the documented auth.admin_pa gap, the current chart does not actually gate Store startup or rolling progress on PD quorum and permits disruption budgets below the PD majority. Three independent current-head review lanes converged on these findings.

[imbajin]

‼️ This counts responding PD processes, not a PD quorum: the current /v1/health handler returns success unconditionally and does not check for a Raft leader or member catch-up. Two REST endpoints can therefore release every Store while PD has no usable quorum; the PD/Store readiness probes use the same process-liveness signal, so a StatefulSet rolling update can also advance before the restarted member has rejoined safely. Please gate on an actual leader/quorum/catch-up-aware signal (for example, validated membership state) and add no-leader plus rolling-restart scenarios.

[bitflicker64]

Agreed on both counts: /v1/health is process liveness, not quorum, and the probes share the same weakness during rolling updates. The fix I intend is to gate the Store init container and the PD readiness signal on /v1/members (leader present, member state validated) rather than counting healthy responders, and to add the no-leader and PD rolling-restart scenarios you list to the full lifecycle matrix rerun already committed before this PR leaves draft. Landing it together with that rerun keeps the gating change and its runtime evidence in one reviewable step rather than shipping an untested probe contract now.

[bitflicker64]

Addressed in 34234e3. The wrapper now writes auth.admin_pa from the Secret it already mounts as PASSWORD, alongside usePD and pd.peers, rejecting values with properties-parser metacharacters (newline, CR, backslash, leading whitespace). Verified against a composition of current master plus #3119/#3126/#3130: an auth-enabled install starts cleanly with init-store disabled, unauthenticated requests and the default admin:pa both get 401, and the Secret credential authenticates end to end, including through the new Hubble login. Both caveats are documented in the chart README: auth.admin_pa applies only at first creation, so the Secret cannot rotate an existing cluster's password, and the value lands in rest-server.properties (mode 600) inside the container.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 1.53%. Comparing base (431f6e6) to head (84c56fc).

❗ There is a different number of reports uploaded between BASE (431f6e6) and HEAD (84c56fc). Click for more details.

HEAD has 4 uploads less than BASE

Flag	BASE (431f6e6)	HEAD (84c56fc)
	5	1

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #3132       +/-   ##
============================================
- Coverage     41.06%   1.53%   -39.53%     
+ Complexity      519      21      -498     
============================================
  Files           771     749       -22     
  Lines         65962   63447     -2515     
  Branches       8766    8318      -448     
============================================
- Hits          27088     975    -26113     
- Misses        36008   62388    +26380     
+ Partials       2866      84     -2782     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[bitflicker64]

0ebd2dc addresses the deployment-side feedback from testing:

• Fresh installs now seed PD with partition.default-shard-count derived from the Store count (3 with three or more Stores, otherwise 1, matching PD's odd-only constraint), so a default install gets store-level HA instead of one shard replica per partition. Verified on a fresh 1 PD + 3 Store cluster: 12/12 partitions with 3 shards on 3 distinct stores. The seed applies at first bootstrap only; PD metadata is authoritative afterwards, and the initial partition count change (36 to 12 on defaults) is documented.
• pd.antiAffinity and store.antiAffinity default to preferred so single-node and small clusters schedule out of the box; values-cluster.yaml keeps required, and NOTES warns when PD quorum members may co-locate.
• Per-component nodeSelector, tolerations, affinity, and topologySpreadConstraints were already supported; the README now has a Scheduling section making that visible.
• The Disaster Recovery section documents the manual /v1/task/patrolPartitions, /v1/task/balanceLeaders, and /v1/task/balancePartitions triggers, and explains why pd.patrol-interval and store.max-down-time are not exposed: current builds bind but never read them. Periodic leader balancing and recovery metrics are filed as [Feature] PD: optional periodic partition leader balancing in TaskScheduleService #3135 and [Feature] Store exposes no shard recovery metrics (sync progress, raft lag) during disaster recovery #3136, and the CreateGraph propagation window as [Feature] Orchestrate graph creation through PD: new graphs are not consistently available across Server replicas ("Could not rebind [g]") #3137.

[bitflicker64]

Todo (README): auth today is existingSecret-only (create Secret out-of-band; no password in values). Local/Kind docs on my test branch spell the steps. Worth a short Authentication subsection in the main chart README too? Happy to add in a follow-up if maintainers (@imbajin ) wants it.
Optional later: demo-only auto-gen Secret; keep existingSecret for real installs.

[bitflicker64]

Checkpoint (2026-08-17)

Re-validated this PR head (84c56fc7) on Kubernetes against the published Docker Hub images:

• hugegraph/pd:latest
• hugegraph/store:latest
• hugegraph/server:latest
• hugegraph/hubble:latest

Topology: 3 PD + 3 Store + 3 Server + 1 Hubble, authentication enabled (chart defaults), Hubble in pd mode.

Results

• All 10 Pods Ready with zero restarts; helm test passed.
• Protected APIs return 401 without credentials / with a wrong password; chart-managed admin Secret and /auth/login JWT succeed.
• The same JWT is accepted by all three Server Pods (shared auth.token_secret).
• Every Server Pod renders usePD=true, the three PD peers, server.urls_to_pd, server.deploy_in_k8s=true, and StandardAuthenticator with auth.admin_pa from the chart Secret.
• REST create / read / update / delete on an HStore graph succeeded.
• Hubble login (chart admin Secret), PD discovery of graph spaces/graphs, vertex create, and gremlin-query succeeded.
• Loaded a sample demo graph (~100 vertices / ~100 edges) and queried it through Hubble.

Notes (expected)

• Hubble cluster overview lists a single Server endpoint because all Server replicas register the shared client Service URL with PD.
• Overview may show an aggregate PD status of UNKNOWN while individual PD members remain UP.

Still pending before leaving draft: the full multi-node upgrade / replacement / failover lifecycle matrix, and pinning image tags when the next release exists.