Skip to content

ci-queue: Recover when the Redis queue master dies during setup - #98

Draft
kstewartfigma wants to merge 2 commits into
masterfrom
kstewart/recover-ci-queue-master
Draft

ci-queue: Recover when the Redis queue master dies during setup#98
kstewartfigma wants to merge 2 commits into
masterfrom
kstewart/recover-ci-queue-master

Conversation

@kstewartfigma

Copy link
Copy Markdown

Summary

Fixes the longstanding failure mode discussed in Slack: if the worker elected to populate the Redis queue dies while master-status is setup, every surviving worker currently waits until it raises CI::Queue::Redis::LostMaster.

  • replace the permanent setup marker with a renewable, generation-owned Redis lease
  • let surviving workers detect an expired setup lease and retry leader election
  • fence queue publication and chunk metadata writes with compare-and-set Lua scripts so a stale master cannot overwrite its replacement
  • publish the complete queue and ready status atomically
  • bound recovery attempts with configurable defaults

Proof

Added regression coverage for:

  • a surviving worker taking over after the original master dies during setup
  • a stale master being unable to overwrite the replacement queue
  • lease renewal during slow queue population

Validation:

  • 161 tests, 603 assertions, 0 failures across the complete test/ci/queue unit suite
  • gem build succeeds and includes the three new Lua scripts
  • git diff --check passes

Local Ruby 3.4 still has the repository's six existing RSpec output-fixture failures caused by Ruby 3.4 changing backtrace quoting. The repository's CI matrix intentionally runs Ruby 3.0–3.3.

Copy link
Copy Markdown
Author

Codex update: reviewed the failed Ruby matrix and fixed it in 679d4f9.

All four jobs failed in Integration::MinitestRedisTest#test_down_redis: the new follower wait attempted another Redis read after election had already encountered a connection error, bypassing ci-queue's existing graceful behavior when Redis is completely unavailable.

The fix routes only that follower wait through the existing rescue_connection_errors helper. Master-side connection failures, expired setup leases, and stale-master fencing still fail loudly.

Verification:

  • exact formerly failing test: 1 test, 3 assertions, 0 failures
  • complete test/ci/queue suite: 161 tests, 603 assertions, 0 failures
  • GitHub matrix: Ruby 3.0, 3.1, 3.2, and 3.3 all passing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant