[P2] Four-node restart reports online before every coordinator can read and write

[Vonng]

Functional observation

After stopping and restarting a four-node test cluster, all nodes/drives/network links can appear online in all four mcli admin info views while one or more coordinators still return PUT 503 SlowDownWrite and cross-node GET 404 NoSuchKey for objects successfully written through another coordinator.

This occurred with both 0806 -> 0806 and 0806 -> 0903 full-stop/full-start runs. It is not established as a 0903 regression. The environment is four native Darwin/arm64 processes on one host, one directory per node, loopback HTTP and published binaries. Multi-host/Linux behavior remains unverified.

Measured evidence

After setup and full restart, probe PUT through each coordinator and immediately GET each successful object through every coordinator at 0/2/5/15/30/60 seconds after the all-online check.

• The unchanged 0806 restart control returned early write-quorum failures after reporting online.
• An independent 0903 timed run reported online after 2.734 seconds. At 0/2/5 seconds, node 0 PUT failed; six objects successfully written through nodes 1/2 initially returned 404 through node 0.
• At 15/30/60 seconds, all newly submitted PUT/GET combinations passed (12 PUT, 48 GET). Both versions passed single-node-outage read/write at 60 seconds.
• Trace captured the actual HTTP 503/404 responses and storage.ReadVersion file-version-not-found results. Startup logs show the same coordinator missing the remote disk connections involved in the failures.

The original fast-restart functional test remains failed; later successful requests do not retroactively make its acceptance pass.

Data readback

A separate run re-read 12 early PUT-200 objects at 15/30/60 seconds: 144/144 full-byte and VersionId comparisons passed. That run did not reproduce early 404s.

The original nine early PUT-200 objects were then read from their retained directories after one same-version 0903 restart: 36/36 comparisons passed, including 24/24 reads of the six objects that had returned 404. No persistent loss of those objects was observed. Because this included a restart, the original objects' natural recovery time before shutdown remains unknown.

Investigation and acceptance

Admin info aggregates local disk information, while connectDisks installs remote disks and its monitor retries every 15 seconds. This supports an incomplete coordinator disk-connection explanation, but the internal state was not directly instrumented and a product fix is not yet proven.

• Establish the per-coordinator disk/peer state during the early failures and compare both versions.
• Use a bounded per-coordinator PUT plus cross-node GET readiness check for subsequent restart/outage tests, recording time to usable I/O separately from admin/health readiness.
• Re-read the same early acknowledged objects after the window; do not substitute later new-object success for those checks.
• Repeat the focused comparison on independent Linux nodes before generalizing the environment or declaring distributed upgrade acceptance complete.

Keep this investigation small: no mixed-binary rolling-upgrade assumption, unlimited retry loop, fixed 15-second readiness guarantee, or unrelated cluster redesign is required.

Versions and verification

Reproduced in the 2026-09-05 functional review on the published SILO release RELEASE.2026-09-03T13-18-01Z (9b11dc9469e650815b775cb47b039610644f5da4), with RELEASE.2026-08-06T00-00-00Z (3be10fcc1a44f6620ded0bd303461f9d688cca23) as the baseline. The reviewed code remains unchanged on current main f0bd164b92d799c026fb1a31ad8e0df8ecb39388. Product fixes have not been applied.

[Vonng]

Fix plan prepared 2026-09-05 by Claude (Fable 5.1) and reviewed in 2 round(s) of dialogue with the Codex reviewer (gpt-6-astra) that filed this issue. Local paths under /Users/vonng/pgsty/silo-codex-triage-20260905/silo-116 (plan, prototype diff, tests, Codex transcripts) and /private/tmp/silo-mcli-review-20260905.MfP8W9 (review evidence) live on the maintainer's machine, not in the repository.

Fix plan — #116 (Claude ⨯ Codex consensus, 2026-09-05)

1. Verdict

Partially confirmed. The symptom is real and well evidenced: after a four-node full stop/start, a coordinator can answer
S3 with SlowDownWrite (503) and NoSuchKey (404) while every admin info view shows all nodes, drives and network links
online. Two corrections to the issue framing. (a) It is not a 0903 regression and not a SILO change at all. The
0806 -> 0806 control failed identically (logs/distributed-control-old.json, three SlowDownWrite FAILs on
upgrade-data-node-{0,2,3}, run dist-07aaad, started_version: old), and the timed probe's 0806/0903 split is scheduling
luck: its 0806 restart gate itself took 17.484 s and so ran past a reconnect pass, while the 0903 gate returned in 2.734 s
(logs/distributed-readiness-probe-{old,new}.jsonl). Against upstream/master, cmd/healthcheck-handler.go is
byte-identical, and cmd/erasure-sets.go, cmd/prepare-storage.go, cmd/notification.go, cmd/admin-server-info.go and
the Health/StorageInfo region of cmd/erasure-server-pool.go differ only in import paths, comments and one support-link
string. (b) The issue's "use a better readiness gate" premise has no product endpoint to point at: by construction
/minio/health/cluster and /minio/health/cluster/read can return 200 under exactly this condition, because
erasureServerPools.Health reads the peer-aggregated storage view and never this node's installed remote disks (§2.5). That
is inferred from source; it was not measured in the original window. Disposition: a documented startup limitation,
accepted without a server change once H1, D1 and V1 below are done. If V1 leaves an acknowledged object unreadable after
recovery, the finding stays open and gets diagnosed; that evidence would not by itself make a faster reconnect the right fix.

2. Root cause

An inherited startup window with no observable signal. The mechanism is strongly supported by source plus startup logs; the
original run's internal state was not directly instrumented. On main f0bd164b9:

1. cmd/prepare-storage.go:239-326 waitForFormatErasure builds the storage clients once (:257) and returns as soon as
   read quorum of drives loads a format. A remote client is non-nil but carries an empty disk ID
   (cmd/storage-rest-client.go:963-970); the real ID is assigned only after a successful format load
   (cmd/format-erasure.go:331-339). A failed or empty GetDiskID then leaves the installed erasure slot empty
   (cmd/erasure-sets.go:437-450).
2. cmd/erasure-sets.go:284-310 monitorAndConnectEndpoints runs one pre-emptive connectDisks after a 0-1 s sleep, arms
   its timer afterwards and resets it after each pass, so defaultMonitorConnectEndpointInterval (:349) =
   defaultMonitorNewDiskInterval (10 s, cmd/background-newdisks-heal-ops.go:41) + 5 s = 15 s between passes, a floor,
   not an end-to-end recovery bound. Nothing else calls connectDisks, and the grid has no hook into it.
3. The gap is startup-only. Once a slot is installed, storageRESTClient.IsOnline tracks the grid connection
   (cmd/storage-rest-client.go:210-217) and recovers by itself across a peer restart. A never-installed slot has no such path.
4. Writes hit cmd/erasure-object.go:1321-1334: with 2 of 4 installed drives missing, offlineDrives >= (len+1)/2 returns
   errErasureWriteQuorum, mapped by cmd/object-api-errors.go:165 to InsufficientWriteQuorum and by
   cmd/api-errors.go:2346-2347 to ErrSlowDownWrite (503). Reads take a different route: with one of the two installed
   drives lacking the version, findFileInfoInQuorum hits maxCount < quorum and returns
   InsufficientReadQuorum{Type: RQInconsistentMeta} (cmd/erasure-metadata.go:386-388, :425), which
   cmd/erasure-object.go:939-945 converts to errFileVersionNotFound/errFileNotFound, surfacing as 404. A generic
   read-quorum failure would instead be SlowDownRead (cmd/api-errors.go:2348-2349), so the 404 is specific to this
   inconsistent-metadata path. Startup logs match the failure directions exactly: node 0 could not use the node 1 and node 2
   drives, nodes 1 and 2 could not use the node 0 drive, node 3 had all four (logs/dist-af3a2e-{0,1,2,3}-new.log; the same
   lines appear on 0806 in logs/dist-07aaad-*-old.log).
5. Why no probe catches it. ReadinessCheckHandler (cmd/healthcheck-handler.go:132-187) does check request-queue
   overload, KMS and etcd, and checkHealth (:32-53) gates the cluster handlers on object layer, bucket metadata and IAM
   initialisation. None of them consults this node's installed remote disks. ClusterCheckHandler (:56) and
   ClusterReadCheckHandler (:93) call objLayer.Health, which uses z.StorageInfo (cmd/erasure-server-pool.go:2699,
   :733) = globalNotificationSys.StorageInfo (cmd/notification.go:1085-1115): a sum of every peer's own
   LocalStorageInfo, filtered to endpoint.IsLocal (cmd/erasure.go:260-275). admin info
   (cmd/admin-server-info.go:162) uses the same source, and its network map is only isServerResolvable
   (cmd/prepare-storage.go:124-152), a liveness GET that always answers 200. The boot gate inherits the blindness:
   cmd/server-main.go:962-976 waits on HealthyRead from that same aggregated view.

Introducing commit: none in SILO. Present at the fork anchor 27742d469 and unchanged on upstream/master, which keeps the
same 15 s constant and has no faster initial retry.

3. Minimal fix

• Change: none in the server. Three non-product deliverables instead.
  • H1 harness (~25 lines, S). In /private/tmp/silo-mcli-review-20260905.MfP8W9/scripts/distributed.py, keep stable() but record its duration as an observed
    metric only, then add a bounded data-path canary before any restart/outage assertion: one small PUT through each
    coordinator plus a GET of that object through all four. A round passes only when all 4 PUTs and all 16 GETs return
    correct bytes and VersionId. Bound it properly: a hard 60 s deadline, and per-request timeouts sized from the time left,
    because the shared boto config allows one retry with a 30 s read timeout (/private/tmp/silo-mcli-review-20260905.MfP8W9/scripts/functional.py:25-27) and an outer
    deadline check alone does not bound a call. Propagate gate failure directly rather than through record(), which
    swallows exceptions and continues (:34-44), so dependent assertions stop while the finally cleanup still runs.
    Preserve every acknowledged canary key and VersionId and never overwrite one before its scheduled readback. The
    observation logic already exists in /private/tmp/silo-mcli-review-20260905.MfP8W9/scripts/distributed_readiness_probe.py:48-65; it records failures rather than
    asserting, so lift it into a helper and add the assertion, deadline and propagation.
  • D1 docs (~15 lines, S). State that none of the four probes proves that the receiving node can serve the data path,
    and that mcli ready inherits the same blind spot because it calls the same cluster endpoints (/Users/vonng/pgsty/mc
cmd/ready-main.go:123-146). Qualify the canary honestly: it demonstrates successful sampled I/O at that moment for the
    erasure set the key hashes to (cmd/erasure-sets.go:692-698, :734, :740), not per-set completeness, and not
    headroom, since three installed drives of four still admit writes and cross-node reads while leaving none for a further
    node loss (cmd/erasure-object.go:1329-1338). Pages: this repo's docs/metrics/healthcheck/README.md and, on the
    owner's site, content/operations/monitoring/healthcheck-probe.md with its .zh.md twin. Docs are written in the site
    checkout; do not edit them from here.
  • V1 verification (bounded, see §4).
• Behaviour after the fix. Server behaviour is unchanged and stays upstream-compatible, so Kubernetes probes,
  mcli ready and the native silo healthcheck subcommand are untouched. What changes is that our acceptance harness stops
  treating an administrative all-online view as a data-path guarantee, and operators are told the same thing in writing.
• Explicitly out of scope, and why:
  • Shortening the reconnect interval or adding a fast initial backoff. No evidence shows the window failing to close, and
    connectDisks re-opens and re-reads format.json per endpoint, so a faster cadence has a real cost on wide clusters.
    Upstream chose 15 s. This is unproven tuning.
  • Making ReadinessCheckHandler fail on incomplete local disks. Upstream's ready deliberately reports process and
    external-dependency health only; making it data-path aware would flap Kubernetes endpoints for every deployment.
  • Switching Health() to this node's installed disks. It changes the cluster probe and the v3 metrics cache
    (cmd/metrics-v3-cache.go:110) for every deployment, and it changes the boot gate (cmd/server-main.go:963). On this
    4-drive topology boot would not stall, since read quorum is 2 and node 0 had 2; on wider topologies a coordinator missing
    more than parity drives would then block its own boot. That trade is a design change, not a bug fix.
  • A new /minio/health/local endpoint: a new public API surface needing its own design, tests, docs and Helm defaults. Not
    minimal, and this issue does not prove it is needed.
  • Any mixed-binary rolling-upgrade claim, unlimited retry loop, or fixed readiness guarantee.
• Re-open condition. If V1 shows on Linux that the window does not close within the chosen deadline, or leaves an
  acknowledged object unreadable after recovery, keep the finding open and diagnose it before proposing any code change. A
  bounded fast initial reconnect in monitorAndConnectEndpoints is one candidate, not a predetermined answer.

4. Tests

• Regression test(s): none in Go. The behaviour is a multi-process startup race; a cmd unit test would either assert
  upstream aggregation semantics (nothing to guard) or need a real four-node cluster. Codex filed no repro test here, correctly.
• Existing tests that must stay green (a guard only if the re-open condition is ever acted on): TestHealthcheckTarget,
  TestProbeHealthChecksAndVerdicts, TestProbeHealthTLSSkipsVerification, TestProbeHealthUnreachableAndTimeout,
  TestHealthcheckResultLine (cmd/healthcheck-main_test.go), plus go test ./cmd -run 'TestNewErasureSets'
  (cmd/erasure-sets_test.go:159).
• Required live check (V1, bounded). One run per version, four nodes on independent Linux hosts or four containers on one
  Linux host (reuse the pattern in /private/tmp/silo-mcli-review-20260905.MfP8W9/scripts/linux-image-smoke.py), one drive per node, EC:2. For each of 0806 and 0903:
  start, full stop, full start, then run the H1 canary and record restart-to-gate and gate-to-first-usable-I/O as
  separate timings per coordinator; re-read every acknowledged early object at 15/30/60 s comparing bytes and VersionId; keep
  the existing single-node-outage read/write assertion as independent evidence. Pass criterion per version: recovery within
  the deadline and no acknowledged object lost. Report the 0806/0903 comparison descriptively; a single
  scheduling-sensitive run cannot establish "0903 is no worse", and a non-reproduction does not close the original
  recovery-time gap. Budget one afternoon, no lab kept alive afterwards.
• Closure criteria mapped to the issue's four boxes. Box 1 (per-coordinator disk/peer state, both versions) is discharged
  by the startup lines "Unable to use the drive : drive not found, will be retried", which name exactly the peers that
  later fail and appear on both versions (logs/dist-07aaad-*-old.log, logs/dist-af3a2e-*-new.log), plus the code path in
  §2. Box 2 is H1. Box 3 is answered by the supplemental readback and repeated inside V1. Box 4 is V1.

5. Compatibility, risk, rollout

Zero API, wire, on-disk and mixed-version impact, because no server code changes. On data correctness the Codex supplemental
readback found 144/144 fresh re-reads and 36/36 re-reads of the original nine acknowledged objects matching bytes and
VersionId, including all 24 reads of the six objects that had returned 404
(logs/distributed-readiness-supplemental-summary.json). That readback included one same-version restart, so it proves
retention, not the original recovery time; V1 re-tests it. The risk we are declining is the larger one: any change to the
health handlers or the reconnect cadence alters Kubernetes rollout behaviour for every deployment, for a window that so far
always closed on its own.

6. Cost

Effort S. Harness ~25 lines of Python in the review scripts, docs ~15 lines across two site pages plus one in-repo
README, both outside the server product. V1 is one bounded Linux run per version. No server product lines. No dependency on
any other issue in this batch, and it does not gate the 20260903 release, since the behaviour is inherited and pre-existing.
Owner for H1: the review-harness maintainer. Owner for D1: the docs site checkout /Users/vonng/pgsty/silo.pgsty.com. Owner
for V1: server side, before the next distributed upgrade acceptance is declared complete.

7. Consensus with Codex

Rounds used: 2/3. CONSENSUS: YES on all five dimensions. Codex independently confirmed the central finding that both
cluster probes aggregate peer-local storage information rather than the receiving coordinator's installed remote disks, so
pointing the harness at /minio/health/cluster is not a remedy.

Amendments accepted from Codex (all of them; nothing rejected):

1. Disposition reworded: inheritance is provenance, not correctness, so this is "a documented startup limitation accepted
   without a server change" after H1/D1/V1, and a persistent failure in V1 stays open for diagnosis instead of prescribing
   reconnect tuning.
2. Harness rigour: functional.py:25-27 allows one retry at a 30 s read timeout and record() (:34-44) swallows
   exceptions, so H1 needs per-request timeouts from the remaining deadline, an all-4-PUT/all-16-GET pass condition, failure
   propagated outside record(), and acknowledged keys preserved until readback.
3. The 15 s interval is a floor between passes, not an end-to-end recovery bound (cmd/erasure-sets.go:290, :304-307).
4. One scheduling-sensitive run per version cannot establish "0903 is no worse"; report descriptively and keep the outage
   assertion as independent evidence.
5. The remote client is non-nil with an empty disk ID (cmd/storage-rest-client.go:963-970); the empty installed slot comes
   from a failed or empty GetDiskID (cmd/erasure-sets.go:437-450).
6. Three source corrections I had loose or wrong: ready is not a no-op, it checks overload, KMS and etcd; the 404 comes
   from the RQInconsistentMeta rewrite (cmd/erasure-metadata.go:386-388, cmd/erasure-object.go:939-945), not the
   generic SlowDownRead mapping; and switching Health() to installed disks would not have stalled this boot, since read
   quorum was 2 and node 0 had 2.
7. Cluster probes can return 200 under this condition by construction, not measured; credit the checkHealth
   object-layer/bucket-metadata/IAM gate.
8. The canary proves sampled I/O at that moment for the set the key hashes to, not per-set completeness and not headroom.

Open disagreements: none. Codex notes that this approves the plan; H1, D1 and V1 still have to be executed and verified.

[Vonng]

Progress record (2026-09-05): H1 and D1 delivered, V1 open

Status: documentation and harness delivered and reviewed; the issue stays open for V1.

D1, documentation. In-repo: #130 adds a "Startup readiness window" section to docs/metrics/healthcheck/README.md and corrects the readiness-probe description (it also fails on request-queue overload and an unreachable KMS). Site: pgsty/silo.pgsty.com#13 adds the same section to the Healthcheck API page in English and Chinese. Both state that none of the probes, admin info or mcli ready proves the receiving node can serve the data path after a restart, why (the cluster probes aggregate each peer's own local drives; a remote drive that failed to connect at startup stays uninstalled until the next connectDisks pass, about 15 s), the 503/404 symptoms, and the bounded data-path check automation should gate on instead.

H1, harness. The review lab's distributed.py gained canary(tag): after every admin-readiness gate it does one PUT through each coordinator and reads each object through every coordinator, passes only when all 4 PUTs and all 16 GETs return the payload and acknowledged VersionId, sizes per-request timeouts from the remaining 60 s deadline with no retries, raises on the deadline instead of continuing, keeps every acknowledged key/VersionId for later readback, and records time-to-first-usable-I/O separately from the gate. Patch archived locally (followups/h1-distributed-canary.patch).

Smoke run of the patched harness (this Mac, four loopback nodes, published binaries, 0806 → 0903): 9/9 harness tests PASS with the canary at all seven readiness sites, and the canary measured the window directly:

Readiness site	Time to first usable I/O
initial 0806 startup	0.24 s
full stop, restart on 0903	13.89 s
restart node 3 after the outage test	13.62 s
rolling restart node 0 / 1 / 2 / 3	14.77 s / 0.13 s / 0.11 s / 14.63 s

The admin-readiness gate returned in 2.5 s for the initial start and 16.7 s for the full restart in the same run, so the two metrics are now recorded separately as the plan asked. Every acknowledged object was retained.

Adversarial Codex review (gpt-6-astra) of D1 and H1: D1 (documentation) PASS ("D1 now conforms"); H1 (the review-lab harness canary) reached round 3 of 3 without a PASS on hard-deadline enforcement and is escalated to the maintainer (see below)..

V1, still open. One bounded run per version (0806, 0903) on independent Linux nodes with the patched harness, recording restart-to-gate and gate-to-first-usable-I/O per coordinator, re-reading acknowledged early objects at 15/30/60 s, keeping the single-node-outage assertion as independent evidence, and reporting the comparison descriptively. This needs Linux hosts; the Darwin loopback result above is not a substitute. The issue is closed when V1 has run, or re-opened for diagnosis if the window does not close within the deadline there.

H1 escalation (3 Codex rounds, no PASS). The harness canary is correct on structure (four PUTs, sixteen GETs, no SDK retries, unique versioned keys, acknowledged-object ledger, per-phase failure propagation via require_phase, separate admin-gate vs data-path timing, running-version labels). The open findings are all about making the 60-second deadline hard against a Python thread blocked in Body.read(): a ThreadPoolExecutor context join waits for the blocked worker, a completed future is accepted without re-checking the absolute deadline, and a transient setup failure must not mark the bucket ready. Codex's suggested remedy is a main-thread POSIX SIGALRM timer that unwinds a blocked syscall, plus a final remaining() check before accepting success and setup marked complete only after both setup calls succeed. This is a review-lab script (not shipped product, not in any repo), so it does not block the documentation. Maintainer decision requested: (a) accept the harness as a best-effort test tool and proceed to V1, or (b) authorise a fourth hardening pass implementing the SIGALRM-based hard deadline before V1. Either way V1 (the bounded Linux run per version) is still required to close this issue.

[Vonng]

H1 harness: Codex PASS (round 4, owner-authorised)

The review-lab data-path canary passed adversarial Codex review at round 4 (VERDICT: PASS, no blocking findings). Round 3's three MAJORs were closed by rewriting the deadline enforcement:

• The 60-second deadline is now hard, via a main-thread SIGALRM itimer whose handler raises TimeoutError; the signal interrupts a blocked Body.read() with EINTR and unwinds immediately, so a slowly-progressing response cannot outlive the deadline (the previous ThreadPoolExecutor join could not). Codex verified with real SIGALRM tests that both the first and final GET are interrupted near the deadline, and that the timer is cleared and the handler restored afterwards.
• setup_done is set only after both create_bucket and put_bucket_versioning succeed, so a transient setup failure is retried, not skipped.
• The absolute deadline is re-checked after the final GET, so a round that completes late is not accepted as success.
• Evidence records the running version (startup old; a non-simultaneous per-node upgrade mixed until the last node), fixing the mislabel.

Smoke of the four-node loopback harness (0806 → 0903), 10/10 phases: the reconnect window is captured under the hard deadline (one restart gate at 14-15 s / 15 attempts, the rest sub-second), and the final readback re-read every acknowledged object across all nodes with 0 failures. One optional NIT (a --control-old label edge) was applied. The harness patch is archived at /Users/vonng/pgsty/silo-codex-triage-20260905/followups/h1-distributed-canary.py.

Remaining to close this issue: V1, the bounded Linux run per version, which needs your Linux hosts. D1 (docs) is in PR #130 (repo) and pgsty/silo.pgsty.com PR #13 (site).

[Vonng]

Status update — partially addressed, keeping open.

The readiness documentation portion shipped in #130 (merged to main as 5b9959617). What remains before this can be closed is the multi-node Linux validation run confirming that a four-node restart no longer reports online before every coordinator can read and write. Keeping this issue open until that run is completed and its result recorded.