Skip to content

wgengine: support recovery from watchdog timeouts - #129

Draft
usman-heracles wants to merge 1 commit into
coder:mainfrom
usman-heracles:usman/recover-watchdog-timeout
Draft

wgengine: support recovery from watchdog timeouts#129
usman-heracles wants to merge 1 commit into
coder:mainfrom
usman-heracles:usman/recover-watchdog-timeout

Conversation

@usman-heracles

@usman-heracles usman-heracles commented Aug 2, 2026

Copy link
Copy Markdown

Problem

A wedged wgengine.Engine operation currently reaches the watchdog's fatal path after 45 seconds. For Coder workspace agents, this terminates the whole process even though the failed tailnet generation can be abandoned and replaced.

Change

  • add NewWatchdogWithTimeoutCallback, which poisons one engine generation and returns ErrWatchdogTimeout instead of terminating the process
  • wake concurrent watchdog callers when a generation is poisoned, quarantine callbacks from the abandoned generation, and preserve deterministic callback removal
  • make Close/Wait safe when the wrapped engine is stuck
  • add focused concurrency, timeout, callback, and cleanup tests

The existing NewWatchdog behavior is unchanged.

Validation

  • go test ./wgengine -count=1
  • go test -race ./wgengine -run 'TestWatchdog' -count=1
  • go test -race -v ./wgengine -run 'TestWatchdog(DefaultTimeoutExitsProcess|DisabledLeavesHungOperationRunning|TimeoutCallback)$' -count=1 demonstrates an actual subprocess exit on the current fatal path, the environment-variable mitigation, and recoverable poison/callback behavior with controlled blocked operations

Integration

A companion Coder change recreates only the agent tailnet generation after the callback, preserving the workspace-agent process, SSH server, reconnecting PTYs, scripts, and metadata loops.

@usman-heracles
usman-heracles force-pushed the usman/recover-watchdog-timeout branch from 27c2322 to 8eaea8f Compare August 2, 2026 16:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant