fix(v1): retire dead-tunnel servers from the elastic interception pool

[JannikSt]

The elastic pool never notices when a server's prime tunnel dies. The tunnel service reaps registrations whose frpc connection stayed down >60min (terminal status, frps rejects every reconnect), after which the server's URL 404s forever — but it stays in the pool with load 0 and keeps absorbing rollout assignments. Hit this in prod on a long GLM-4.5-Air SWE run: a network flap killed a third of the tunnels and ~80% of rollouts fast-failed on dead URLs for hours.

• PrimeTunnel.is_alive(): frpc alive + registration not in a terminal status. Status-based on purpose — a terminal registration still exists, so the SDK's check_registered existence probe stays True; transient API failures report alive so a flaky control plane can't mass-retire healthy servers.
• ElasticInterceptionPool health loop (60s): retires dead servers so new rollouts land elsewhere and the pool grows back with fresh tunnels; retired servers drain in-flight rollouts before teardown.
• Also retires zombies (terminal status but still routing on a surviving connection) proactively — they die for good on the next drop.

---

Note

Medium Risk
Changes rollout routing and long-running background teardown in the elastic pool; behavior is scoped to tunnel-required pools but affects remote eval reliability under tunnel failures.

Overview
Fixes the elastic interception pool keeping servers whose prime tunnel has died, which caused rollouts to fast-fail on dead URLs while those servers kept getting picked (load 0, instant failures).

Adds Tunnel.is_alive() (default true for custom tunnels) and PrimeTunnel.is_alive(), which treats frpc as dead when it is not running, the registration is gone, or status is terminal (expired / terminated). Transient tunnel API errors count as alive so a flaky control plane does not mass-retire healthy servers.

ElasticInterceptionPool starts a 60s health loop when tunnels are required: dead servers are removed from the active pool into a draining list, new acquires get healthy servers (and the pool can grow fresh tunnels), and drained servers are torn down with run_shielded only after in-flight rollouts finish. PrimeTunnel.expose() now holds the SDK client on the instance so liveness checks can query registration status.

^{Reviewed by Cursor Bugbot for commit 2742173. Bugbot is set up for automated code reviews on this repo. Configure here.}

[macroscopeapp]

Retire dead-tunnel servers from the elastic interception pool

• Adds a background health loop to ElasticInterceptionPool that runs every 60 seconds when tunnels are required, calling _retire_dead_servers to remove servers with dead tunnels from active rotation.
• Retired servers are moved to a draining list and stopped via run_shielded once their load reaches zero, so in-flight requests are not interrupted.
• Adds PrimeTunnel.is_alive() which checks client state and queries the tunnel service, returning False if the tunnel is missing or in a terminal status (expired, terminated); control-plane errors are treated as alive.
• Adds a base Tunnel.is_alive() that returns True by default, so non-Prime tunnel implementations are unaffected.
• Pool shutdown in stop() now cancels and awaits the health loop task alongside the warm task.

^{Macroscope summarized 2742173.}

[JannikSt]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. More of your lovely PRs please.

Reviewed commit: 28809de410

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[macroscopeapp]

Approvability

Verdict: Needs human review

Unable to check for correctness in 2742173. This PR adds substantial new runtime machinery: a background health check task, server retirement logic, and new state tracking. While framed as a bug fix, it introduces new async lifecycle management that affects how the pool handles servers, warranting human review.

^{You can customize Macroscope's approvability policy. Learn more.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 1eb7f3e. Configure here.}

[JannikSt]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Keep them coming!

Reviewed commit: 2742173830

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".