Skip to content

Stop processes whose heartbeats keep failing past the alive threshold - #778

Merged
rosa merged 2 commits into
rails:mainfrom
wintan1418:heal-pruned-scheduler
Aug 22, 2026
Merged

Stop processes whose heartbeats keep failing past the alive threshold#778
rosa merged 2 commits into
rails:mainfrom
wintan1418:heal-pruned-scheduler

Conversation

@wintan1418

Copy link
Copy Markdown
Contributor

Related to #763 (and the same family as #751).

What I found

The core scenario in #763 — a process whose registration gets pruned while it's still alive — already self-heals on main: the next heartbeat raises RecordNotFound, Registrable#heartbeat clears the process and wakes the run loop, shutting_down? returns true via !registered?, the fork exits, and the supervisor replaces it. I've added a regression test proving this end-to-end for the scheduler (start a supervisor with a scheduler, delete the scheduler's solid_queue_processes row while it's running, assert a fresh registration appears).

The remaining hole

That self-heal only fires when the heartbeat gets a clean RecordNotFound. Heartbeats can also fail without confirming anything about the registration — a dropped DB connection after host sleep/resume, SQLite busy errors (the reporter's setup in #763) — and those errors are only reported via the timer task's observer. The process then keeps running unregistered indefinitely: the supervisor never replaces it because it never exits, and from the outside it's a zombie that stops doing visible work. This matches the reported evidence: an alive, supervised scheduler with no solid_queue_processes row and no "Replaced terminated Scheduler" ever logged.

Fix

Track when heartbeats started failing. Once they've been failing for longer than SolidQueue.process_alive_threshold — the exact point at which every other process considers this one dead and prunable — treat it like finding the registration gone: clear the process and wake the run loop so the process stops and the supervisor starts a replacement that can register afresh. Errors are still re-raised so the existing observer reporting is unchanged, and a successful heartbeat resets the tracking, so transient blips don't accumulate.

Testing

  • New unit test: a worker whose heartbeats persistently raise ActiveRecord::StatementInvalid shuts down its pool once the failures outlast the alive threshold (fails on main, passes with this change).
  • New regression test for the existing pruned-registration replacement path (passes on main and with this change).
  • Full worker/scheduler/supervisor/process-recovery test files green on MySQL.

@wintan1418
wintan1418 force-pushed the heal-pruned-scheduler branch from 389ec4b to ad264b6 Compare July 31, 2026 13:47
wintan1418 and others added 2 commits August 22, 2026 18:35
A process that finds its registration gone already stops itself so the
supervisor can replace it (RecordNotFound on heartbeat). But heartbeats
can also fail without confirming anything about the registration — a
dropped database connection after the host sleeps and resumes, SQLite
busy errors — and those errors were only reported, leaving the process
running unregistered indefinitely: a zombie that polls or schedules
nothing anyone can see, while the supervisor never replaces it because
it never exits.

Once heartbeats have been failing for longer than the alive threshold,
every other process will have considered this one dead and pruned its
registration anyway, so treat it the same as finding the registration
gone: stop, and let the supervisor start a replacement that can register
afresh.

Also add a regression test covering the existing replacement path for a
scheduler whose registration is pruned while it's still alive.

Related to rails#763
Replace the failing-streak tracking with the exact check supervisors
run when pruning, applied from the inside: if the last heartbeat that
actually persisted is older than the alive threshold, this process's
registration is considered prunable and gone by now, so stop and get
replaced — the same remedy the clean RecordNotFound path applies.

For that timestamp to be trustworthy, a failed touch must not leave
the in-memory process claiming a heartbeat that was never persisted:
restore its attributes when the update fails, complementing the
existing restore on entry.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@rosa
rosa force-pushed the heal-pruned-scheduler branch from ad264b6 to e2274b3 Compare August 22, 2026 16:59

@rosa rosa left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @wintan1418! I pushed some mostly style-related changes. Good catch!

@rosa
rosa merged commit 36c2107 into rails:main Aug 22, 2026
57 of 58 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants