locking: deal with clock skew

[ThomasWaldmann]

Fixes #9870 (the locking part; the manifest-timestamp part was declared out of scope there — the borg2 manifest is a borg1 remnant and its timestamp check intentionally stays as a whole-repo rollback tripwire).

Two pre-existing fixes found during this PR's review were split out into #10096 (vanished-lock ObjectNotFound race, with-lock epilog); this PR is rebased on top of that merge.

Problem

Lock staleness was judged by comparing a lock's content timestamp (stamped by its writer's clock) against the reader's local clock. A client whose clock runs >30 min ahead stale-kills another client's healthy lock and can then e.g. run compact, deleting chunks the victim still dedup-references for up to ~15 min — a finished archive referencing deleted chunks.

Fix

AND-rule staleness — a lock may only be killed by age if it looks stale in BOTH clock domains:

• writer/local clock domain: local now vs. the lock content timestamp (the pre-existing rule), AND
• store clock domain: store "now" vs. the lock object's store-side mtime (new, via borgstore 0.6.1's ItemInfo.mtime, see ItemInfo: add mtime field (storage-side modification timestamp) borgstore#212).

Store "now" is derived from our own lock object's mtime + elapsed monotonic time, so store-domain comparisons stay entirely within the store's clock domain: neither the clients' nor the store's absolute clock error matters (a third-party-hosted store may have a wrong clock — it only serves as a common reference). A client without an own lock object defers the kill decision until acquire() has created one; the listing right afterwards confirms or vetoes the candidates.

Threat model: store-side mtimes are advisory only — they can veto a kill, never cause one — so a hostile or broken store gains no new capabilities (lock objects are unauthenticated, so it can already delete them or serve fabricated fresh ones). For the same reason the process_alive() check runs first and is never vetoed by store timestamps: if the lock owner is a process on our own machine and it is dead, we know that locally, and a store serving bogus always-fresh mtimes must not be able to keep an abandoned lock alive forever and block us. Backends without store-side mtimes (rclone, old rest servers) keep the previous behavior. This also composes with future AEAD locks: authentication makes the content timestamp trustworthy, but doesn't fix a skewed writer clock, so the advisory mtime cross-check stays the skew-immunity mechanism; all lock parsing remains centralized in _get_locks().

Skew warning — each lock object carries two timestamps of the same write instant (content time = writer clock, mtime = store clock), so writers' clock offsets are comparable with the store's absolute error cancelled. borg checks this on every lock listing and warns (once per lock instance) when another active client's clock differs by more than MAX_MUTUAL_CLOCK_SKEW (5 min) — including for exclusive acquirers, which only ever see the skewed peer in intermediate listings, and before giving up with LockTimeout. Diagnosis only, never an abort, so spoofed store timestamps cannot block backups.

Plus docs (a module docstring in storelocking.py explaining lock objects, acquiring, staleness in both clock domains, store "now", the advisory-only store timestamps, the skew warning and refreshing; the internals section in data-structures.rst keeps a summary and points there; FAQ note recommending synchronized clocks for shared repos) and the borgstore pin bump to ~= 0.6.1.

Hardening after review

An adversarial multi-agent review of this PR produced 15 verified findings; 12 are fixed in the follow-up commits here (kept separate for reviewability), 2 pre-existing ones moved to #10096, one maintainability nit was deliberately left:

• torn-state safety: everything we know about our own lock object (store key, content timestamp, store-side mtime, monotonic anchor) lives in one LockAnchor namedtuple that is only ever replaced as a whole, so an unserialized LockRefresher thread (borg with-lock) can no longer cause KeyError/TypeError or silently pair one lock object's mtime with another's anchor.
• no false alarms: skew warnings are always magnitude-gated; a veto caused by our own lagging store-time estimate (after a suspend, time.monotonic() stood still) no longer produces a bogus "clock skew of ~0s" warning.
• efficiency: the store-clock anchor survives deletion of our transient lock object (it is a clock calibration, not a property of the object), so an acquirer blocked by a healthy-but-skewed lock vetoes the kill on the first listing of every retry (~2 store round-trips) instead of re-running the whole defer/create/veto/delete cycle (~7 round-trips plus lock-object churn). A wall-clock age cap in _store_now() refuses anchors older than the stale timeout (wall-clock age counts suspends; monotonic does not), bounding any suspend-frozen anchor.
• store clock steps: a storage clock stepping BACK by more than the stale timeout during the anchor's lifetime can defeat the veto. Mitigated twice: refresh() now creates the new lock object before listing, so the stale sweep always judges with a seconds-old anchor instead of the previous refresh's (up to 15 min old) — shrinking the exposure to the write/list latency; and the first harvest of each new anchor is compared with the previous anchor's extrapolation, warning once if the storage's clock jumped (only when our own wall and monotonic clocks agree about the elapsed time, so suspends don't false-alarm). The remaining residual is documented at the veto.
• tests: helpers deduplicated — write_raw_lock reuses _create_lock via a new explicit content-timestamp parameter (the lock wire format lives in one place, ahead of AEAD locks), and free_pid is imported from fslocking_test like platform_test already does.

Out of scope, noted for a separate PR: serializing with-lock's LockRefresher against the main thread (pre-existing race class; fuse/hlfuse/webdav already pass a serialization lock).

Verification

• lock testsuite: 26/26 (11 tests new in this PR: refresh judges with a fresh store-time anchor; storage clock step warning; no skew warning about our own old lock object; healthy-but-skewed lock survives incl. exclusive-acquire timeout without killing it; stale-in-both-domains lock killed; warning above / none below threshold; warning during exclusive acquire; no warning when only our store time lags after a suspend; no lock churn while blocked by a skewed lock; dead-process lock killed despite fresh store mtime; mtime==0 legacy fallback)
• full local testsuite: green after the review-hardening commits (2650 passed, 0 failed on the current head, against the released borgstore 0.6.1)
• E2E smoke on a real file: repo with injected foreign locks: skewed-writer lock survives repo-list with Clock skew of ~2400s detected ... 'slowclockhost' logged; genuinely stale lock removed; operations succeed. Also: an abandoned exclusive lock of a dead local PID carrying a fresh store mtime does not block borg create (it is cleaned via process_alive).

borgstore 0.6.1 (with ItemInfo.mtime) is released on PyPI, so the pin bump included here is satisfiable.

History: the feature itself is the first (squashed, rebased onto master) commit; the review hardening follows as separate commits.

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 95.69892% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.18%. Comparing base (92d4cdd) to head (29df5b6).
⚠️ Report is 3 commits behind head on master.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
src/borg/storelocking.py	95.65%	2 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #10077      +/-   ##
==========================================
+ Coverage   87.10%   87.18%   +0.08%     
==========================================
  Files         102      102              
  Lines       18298    18372      +74     
  Branches     2807     2824      +17     
==========================================
+ Hits        15938    16018      +80     
+ Misses       1649     1645       -4     
+ Partials      711      709       -2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.