Recipient subscription stuck in pendingSubscriptions after unclean shutdown until unrelated traffic arrives

[bauhausb]

Environment

• Server (smp-server): Docker image simplexchat/smp-server:v6.5.0, built from simplexmq
  commit ba6af65 (tag v6.5.0, 2026-04-29).
• Client (simplex-chat core library): v6.5.6, extracted from the signed
  simplex-aarch64.apk release (arm64-v8a libsimplex.so/libsupport.so). This version of
  simplex-chat pins its own simplexmq dependency at commit 92598c2 via cabal.project
  (stable branch as of 2026-06-21, not itself a tagged release).
• Version skew, disclosed because it may be relevant to the bug: the client's simplexmq
  dependency (92598c2, 2026-06-21) is 16 commits and about two months newer than the server's
  (ba6af65, 2026-04-29). Reviewed the titles of those 16 commits — none reference subscription
  or agent-connection handling by name, but this was a commit-message review, not a full diff
  audit, so a protocol/agent-state mismatch between server and client versions is not ruled out
  as a contributing factor.
• Client harness: a minimal third-party Android app driving the core directly via C FFI.
  Verified against the v6.5.6 source (apps/multiplatform/common/src/commonMain/cpp/android/ simplex-api.c) that the four functions used —chatMigrateInit, chatSendCmdRetry,
  chatRecvMsgWait, chatCloseStore — are exactly the same functions and signatures the
  official app itself calls (only the JNI package qualifier differs, since this harness uses its
  own package rather than chat.simplex.common.platform). Same command/response protocol
  throughout (bots/api/COMMANDS.md, bots/api/TYPES.md), not a different or reduced API surface.
• Server config: self-hosted, default [INACTIVE_CLIENTS] (ttl=21600,
  check_interval=3600), store_messages = memory, restore_messages = on, store_queues = memory, expire_messages_days = 21. Server and client on the same local network (Android
  emulator via the 10.0.2.2 host alias).

Summary

If a client process is killed uncleanly (no chatCloseStore() call) immediately after receiving
a message, the SMP acknowledgment for that message is never sent. On every subsequent clean
start, the client's subscription to that specific queue is stuck in pendingSubscriptions (per
/get subs details) and never transitions to activeSubscriptions on its own — confirmed idle
for at least 60 minutes with no self-resolution. The subscription only completes when unrelated
new traffic arrives at the same queue (confirmed to be the same queue throughout, by watching a
fixed rcvId transition live — not a distinct queue picking up traffic while an abandoned one
leaks separately), at which point it activates and delivers the entire backlog at once, including
the message from the original unclean shutdown. The same scenario with a clean shutdown
instead of an unclean kill does not reproduce the bug — the subscription completes normally within
seconds either way.

Severity/triage note: in every reproduction here, the message that triggers the stuck state
was already delivered to and durably stored in the recipient's local database before the process
was killed, so it is not lost or invisible to the recipient — the confirmed impact is a
relay-side bookkeeping issue (an indefinitely unacknowledged message occupying a queue slot,
bounded by the 128-message-per-queue quota and 21-day message expiry), not confirmed data loss.
A more severe variant — a kill landing during receipt, before the local database write commits,
which would leave the recipient genuinely unaware a message was even sent — was tested for
statistically (80 randomized-kill cycles) and not observed; see "What we tested" below for why
that's a bound and not a proof.

This is not an edge case for a mobile client: Android routinely kills app processes (low memory,
force-stop, backgrounding) without running any graceful-shutdown callback, so "unclean kill right
after receiving a message" is a normal operating condition, not a rare failure.

Reproduction steps

1. Two users, alice and bob, each with their own ChatCtrl (chatMigrateInit +
   /_create user ... + /_start), both configured (/_servers) against the same self-hosted
   SMP relay.
2. Establish a contact connection between them (/_connect invite from alice, /_connect plan
   • /_connect accept from bob) and confirm both sides reach contactConnected.
3. Stop bob's process cleanly (chatCloseStore + process exit — connection itself is healthy at
   this point).
4. With bob's process not running, alice sends a text message (/_send @<bobContactId> ...).
   Confirmed server-side: the message is queued (relay log on restart:
   messages stored: 1, expired: 0, queues: 2).
5. Start bob's process fresh (chatMigrateInit on the same existing on-disk database +
   /_start). The message is delivered normally — a newChatItems event fires with the correct
   content, i.e. it is now durably stored in bob's local database.
6. Do not call chatCloseStore(). Kill bob's process immediately after step 5's
   newChatItems event (in our repro: adb shell am force-stop, i.e. SIGKILL-equivalent, no
   graceful shutdown hook runs).
7. Restart the relay and check its log: the message is still present
   (messages stored: 1, expired: 0, queues: 2) — confirms the acknowledgment was never sent
   before the kill in step 6.
8. Start bob's process again (same on-disk database, chatMigrateInit + /_start). The message
   is not redelivered, and /get subs details shows the queue's rcvId permanently in
   pendingSubscriptions, never activeSubscriptions:

   {"result":{"type":"agentSubsDetails","agentSubs":{
     "activeSubscriptions":[],
     "pendingSubscriptions":[{"userId":1,"server":"smp://<fp>@<host>","rcvId":"XsuyWw_UHI0YCDYvDZih1nRY3ub9m6pW"}],
     "removedSubscriptions":[]
   }}}

   This persists across repeated fresh-process attempts, and for at least 60 minutes of idle time
   with zero client or network activity in between.
9. Sending a new, unrelated message from alice to the same contact resolves it: the same
   rcvId transitions to activeSubscriptions, and both the new message and the original
   stuck one are delivered together. Confirmed server-side too — the relay's message count drops
   to 0 at that point.

What we tested

Ruled out: INACTIVE_CLIENTS server-side timeout. Hypothesized the server might be holding
the queue "claimed" by bob's dead TCP session until its own inactivity timeout expired. Tested
directly: reproduced the same scenario against a separate relay instance with [INACTIVE_CLIENTS]
ttl=8 / check_interval=5 (vs. the default ttl=21600 / check_interval=3600), confirmed via
the relay's own startup log that the faster settings were active (expiring clients inactive for 8 seconds every 5 seconds), and waited well past several sweep cycles (20+ seconds, then again
90+ seconds). The message stayed stuck on that relay too. This rules out the inactive-client
sweep as the mechanism.

No self-resolution after 60 minutes idle. Left the stuck state completely untouched (no
traffic, no client activity of any kind) for a full hour, then reopened bob. Identical result to
the shorter tests (events_on_reopen=0) — not a delay that eventually resolves on its own.

Specific to unclean shutdown. Reproduced the same scenario but with a genuinely clean
chatCloseStore() before bob goes offline (instead of an unclean kill), then sent a new message
while he's closed. The reopened subscription shows the same brief pendingSubscriptions state
immediately after /_start (normal startup lag), then transitions to activeSubscriptions on
its own within the observation window and delivers normally. A clean shutdown does not reproduce
the stuck-forever state.

Same queue confirmed via live tracking, not inferred. With the subscription already stuck,
watched /get subs details repeatedly across a send of a new message from alice: the same
rcvId was in pendingSubscriptions immediately before and after the send, then present in
activeSubscriptions at the exact snapshot where the new message's newChatItems event fired.
The relay's message count went to 0 at that same point (both messages delivered and acked
together).

Recovery commands tested — neither is a reliable fix. /_resubscribe all
(ResubscribeAllConnections) flips the reported status from pending to active but does not
flush the backlogged message — relay still reports it queued afterward, no delivery within a
20-second window. /reconnect (ReconnectAllServers) had no effect at all — no status change,
no delivery.

Fault-injection test: statistical bound on the more severe "kill mid-receive" variant. The
window between the server delivering a message and the agent's local database write committing
is inside the Haskell core and not deterministically injectable from the client side, so this was
tested statistically: sent a numbered 100-message sequence, then ran 80 cycles of starting bob's
process, letting it receive for a randomized delay (0.5–6 seconds), and externally killing it
(adb shell am force-stop) at that point — never calling chatCloseStore. Aggregated every
message received across all 80 cycles and compared against the sent sequence.

Result, stated precisely: zero gaps and zero duplicates across 80 kill cycles, at a sequence
length (100) kept safely under the relay's 128-message-per-queue quota, after correcting an
initial harness artifact. That artifact: the first attempt used 200 messages sent in one burst
before bob ever started receiving, which silently exceeded the 128-message quota — messages
130–200 were rejected at send time and never entered the queue at all. This initially looked like
a 71-message gap and would have been a dramatic (and wrong) result; caught by checking the
relay's own message count rather than trusting the client-side tally alone, then corrected by
re-running under-quota. Noted here deliberately, since anyone else attempting to reproduce at a
higher message count should watch for the same failure mode.

This is a statistical bound, not a debugger-verified proof: 80 external, randomly-timed kills
against a fast local relay is a real result, not a single clean pass, but it does not establish
that the exact byte-level window between the server sending a message and the local database
write committing is unreachable or safe. A negative result here means "not observed in 80
trials," not "impossible."

Impact

• Not head-of-line blocking. A new message sent while an old one sits stuck is delivered to
  the recipient normally and promptly — confirmed both client-side and via the relay's message
  count (leaked message stays put, new message's slot clears independently).
• Confirmed impact is relay-side, not recipient-visible, in every scenario reproduced here: a
  queue slot occupied indefinitely by an unacknowledged message, bounded in practice by the
  128-message quota (defaultMsgQueueQuota in simplexmq, a compile-time constant not
  configurable via smp-server.ini) and 21-day message expiry.
• Asymmetric failure mode when it does surface: a sender whose send hits a full queue gets an
  immediate, visible QUOTA error; a recipient with a stuck-but-already-delivered message has no
  client-visible indication their queue has an outstanding ack, and neither side can clear it
  from the client (see recovery commands above).
• The more severe variant — genuine recipient-side message loss from a kill during the local
  database write — remains unconfirmed. 80 fault-injection cycles targeting this found no gaps
  or duplicates, a meaningful bound, not proof it cannot happen.

Prior art

Searched simplex-chat/simplexmq and simplex-chat/simplex-chat issues (open and closed) and
GitHub Discussions across a wide range of phrasings (pending subscription, resubscribe,
ResubscribeAllConnections, chatCloseStore, unclean shutdown, force stop, messages stuck, queue quota, Android-lifecycle-specific terms, and more) — no existing report of this
behavior found. One issue reviewed in detail and ruled out as a different mechanism:
simplex-chat/simplex-chat#6487
("Messages not received on groups") describes group membership/read-receipt propagation
reliability across many members, not a single 1:1 queue's subscription getting stuck after one
client's unclean shutdown.

Not covered by this report

• Exact root cause of why the subscription doesn't self-complete on a bare reopen is not
  established — this report describes reproducible behavior and what's been ruled out
  (INACTIVE_CLIENTS, delayed self-resolution, clean-shutdown reproduction), not a diagnosed
  cause in the agent's subscription-handling code.
• The precise byte-level kill-mid-commit window was not deterministically reproduced (not
  possible from outside the Haskell core with the tools available here) — the fault-injection
  test above is a statistical bound over 80 trials, not an exhaustive test or a debugger-verified
  negative result.