Phase 2: ingest_endpoints routing with gateway fallback + upload fidelity suite

[ehsan6sha]

Phase 2 of the decentralization roadmap (uploads axis; cross-repo with functionland/fula-ota fula-ingest).

Client-side CID (walkable-v8) already ships per-chunk blake3 pre-compute + ETag self-verify + chunk_cids manifest (default ON) - this issue adds the ROUTING so the byte path survives the gateway being down:

Ships:

• Config.ingest_endpoints: Vec<String> (default empty = today's behavior, flag-off-identical). When set, chunk ciphertext bytes go to a fula-ingest node (PUT /v0/block?cid=<precomputed>) instead of the gateway PUT; on ingest failure -> transparent fallback to the existing gateway path. Metadata/index/forest writes unchanged (any federated master).
• Tamper-evidence hardening: ingest declares-and-verifies (server 422s mismatches); client keeps its existing soft self-verify.
• Tests: wiremock matrix (ingest ok / ingest down -> fallback / tampered 422 -> fallback or surface), plus the FULL fidelity suite against the live test master: fxfiles_walkable_v8 upload/walk + dir_index_v8 (60+ files, 35 dirs, nested) + >=1 GB chunked file + walkable_v8_writer on AND off.

Verify: gateway container stopped -> chunk bytes still land via ingest (and are retrievable once metadata commits via a master); client CID == kubo CID; tampered byte rejected.

🤖 Generated with Claude Code

[ehsan6sha]

Design note (locks the v0 wiring — read path must not break)

Chunk objects today are S3 PUTs: the master stores bytes in its kubo AND records the chunk_key -> storage_cid entry in bucket metadata. If chunk bytes go ONLY to a fula-ingest node, the master never learns the mapping and the standard warm read path (GET /bucket/chunk_key) breaks. walkable-v8 CID hints cover offline/cold reads, not the primary path.

v0 wiring (3 parts):

1. fula-ingest (Phase 2: fula-ingest node - verified byte ingress (client-side CID, quota-gated) fula-ota#74, shipped): client streams chunk BYTES to the ingest node — verified blake3 CID, quota-gated, into IPFS.
2. Client (ingest_endpoints in Config): per chunk — bytes to ingest (PUT /v0/block?cid=<precomputed>), then a lightweight master PUT with header x-fula-remote-cid: <cid> and EMPTY body to commit the chunk_key -> cid mapping. Any ingest/remote-PUT failure => transparent fallback to today's full-bytes gateway PUT (flag-off-identical when ingest_endpoints is empty).
3. Gateway (flag FULA_REMOTE_CID_PUT, default OFF): on empty-body PUT with x-fula-remote-cid — validate CID shape (raw+blake3 only), confirm block presence via kubo block/stat (bitswap pulls from the ingest peer; offline=409 retryable), record the object entry with storage_cid=<declared> exactly as a bytes PUT would. Tamper-evidence holds end-to-end: ingest verified the bytes; the CID is self-certifying on every read.

Bytes leave the master's ingress path (the decentralization goal); metadata commits still go to ANY federated master (Stage A); pin request unchanged (per plan, chain arrives Phase 4).

🤖 Generated with Claude Code

[ehsan6sha]

Protocol refinement (prevents a silent-corruption edge)

Hazard found while wiring: if the client used the ingest route against a master with FULA_REMOTE_CID_PUT OFF (or an old build), the empty-body mapping PUT would be stored as a REAL zero-byte chunk object - reads break later, silently. Headers/query params do not save us because an old gateway ignores unknown metadata and happily stores the empty body.

Fix - capability probe:

• Gateway adds GET /fula/capabilities => {"remoteCidPut": true|false} (404 on old builds).
• Client probes ONCE per client instance (cached); the ingest route activates only when ingest_endpoints is non-empty AND the master advertises remoteCidPut:true. Otherwise the client behaves exactly as today (full-bytes PUTs) - misconfiguration degrades to the legacy path, never to corruption.
• Mapping PUT carries x-amz-meta-fula-remote-cid: <cid> + empty body; gateway (flag ON) validates the CID is raw+blake3, confirms block presence via kubo block/stat (bitswap fetches from the ingest peer; absent => 409 so the client falls back to full bytes), records the entry with storage_cid=<cid>, returns ETag=<cid> (client self-verify unchanged).
• Pinning-credential uploads use the same mapping PUT via the pinning variant, preserving per-object pin requests.
• wasm32: ingest route is cfg-gated off (native-only v0).

🤖 Generated with Claude Code