Crash: WebDataAccessRequestException: Web data access call to '/data/get-changed-entities' failed with 404:

[JoshuaRowePhantom]

Crash: WebDataAccessRequestException for /data/get-changed-entities 404 escapes background refresh

Summary

The background refresh timer in MainWindowViewModel posts an async void handler (OnRefreshTick) that awaits EntityBroker.RefreshAsync(). When the current dev-tunnel/web-data endpoint answers /data/get-changed-entities with HTTP 404, WebClientDataAccessLayer.PostAsync turns that into a WebDataAccessRequestException. ReconnectingWebDataAccessLayer.ExecuteAsync does not classify a 404 as connectivity-related (only null/401/>=500 are), so the exception is not retried or reconnected — it propagates out of RefreshAsync, out of the async void handler, is posted onto the Avalonia dispatcher via Task.ThrowAsync, and is picked up by UnhandledExceptionHandler.OnDispatcherUnhandledException which crashes the app with the crash dialog.

The desired behaviour (owner comment on #1326): "Failing to connect to mongo queries should result in the network status icon having an overlayed exclamation mark and should have a section in the dialog box showing the most recent errors."

Fix must follow the project convention (see #1301/#1303/#1322): NO central benign-exception filtering. Handle the failure at the source sites of relevance — the background refresh handler, the reconnect-classification layer, and the connection-status view model that already drives the network icon's warning glyph.

Observed stack:

Phantom.Workspaces.Data.Web.Client.WebDataAccessRequestException: Web data access call to '/data/get-changed-entities' failed with 404:
   at Phantom.Workspaces.Data.Web.Client.WebClientDataAccessLayer.PostAsync[TRequest,TResponse](...)
   at Phantom.Workspaces.Services.DevTunnel.ReconnectingWebDataAccessLayer.ExecuteAsync[TResult](...)
   at Phantom.Workspaces.EntityBroker.RefreshAsync(CancellationToken cancellationToken)
   at Phantom.Workspaces.ViewModels.MainWindowViewModel.OnRefreshTick(Object sender, EventArgs e)
   at System.Threading.Tasks.Task.<>c.<ThrowAsync>b__124_0(Object state)
   at Avalonia.Threading.SendOrPostCallbackDispatcherOperation.InvokeCore()

Root Cause

1. OnRefreshTick is an unguarded async void fire-and-forget handler

features/Phantom.Workspaces/ViewModels/MainWindowViewModel.cs:171 wires a 3-second DispatcherTimer to OnRefreshTick:

// MainWindowViewModel.cs:171–172
this.refreshTimer = new DispatcherTimer { Interval = TimeSpan.FromSeconds(3) };
this.refreshTimer.Tick += this.OnRefreshTick;

The handler itself has no error handling at all:

// MainWindowViewModel.cs:1764–1769
private async void OnRefreshTick(
    object? sender,
    EventArgs e)
{
    await this.EntityBroker.RefreshAsync();
}

Any exception from RefreshAsync escapes the async void state machine → Task.ThrowAsync posts it to the dispatcher → UnhandledExceptionHandler.OnDispatcherUnhandledException fires the crash dialog. This is the direct crash cause: a transient/misrouted background poll becomes a fatal user-facing crash.

2. WebClientDataAccessLayer.PostAsync throws on any non-success HTTP status

features/Phantom.Workspaces.Data.Web.Client/WebClientDataAccessLayer.cs:136–142:

if (!response.IsSuccessStatusCode)
{
    var errorBody = await response.Content.ReadAsStringAsync(cancellationToken).ConfigureAwait(false);
    throw new WebDataAccessRequestException(
        $"Web data access call to '{relativeUri}' failed with {(int)response.StatusCode}: {errorBody}",
        response.StatusCode);
}

A 404 for /data/get-changed-entities in practice means the web-data endpoint is not (yet) available on the currently-resolved dev-tunnel host (wrong/stale relay, an old build that doesn't serve that route, the host process not yet up). It is a connectivity/availability failure, not an application error.

3. ReconnectingWebDataAccessLayer does not classify 404 as a reconnectable failure

features/Phantom.Workspaces.Data.Web.Client/WebDataAccessRequestException.cs:28:

public bool IsConnectivityFailure =>
    this.StatusCode is null
    || this.StatusCode == HttpStatusCode.Unauthorized
    || (int)this.StatusCode >= 500;

features/Phantom.Workspaces/Services/DevTunnel/ReconnectingWebDataAccessLayer.cs:154–155:

private static bool DefaultIsConnectionFailure(Exception exception)
    => exception is WebDataAccessRequestException { IsConnectivityFailure: true };

And in ExecuteAsync (line 102–123), only failures matching isConnectionFailure trigger reconnect/retry; everything else propagates:

catch (Exception exception) when (exception is not OperationCanceledException && this.isConnectionFailure(exception))
{
    await this.ReconnectAsync(exception, cancellationToken).ConfigureAwait(false);
    ...
}

So a 404 sails straight through the reconnect layer and out of EntityBroker.RefreshAsync (features/Phantom.Workspaces/EntityBroker.cs:341), where the call is issued via Task.Run(() => this.entityRepository.DataAccessLayer.GetChangedEntitiesAsync(...)).

4. UnhandledExceptionHandler makes it fatal

features/Phantom.Workspaces/UnhandledExceptionHandler.cs:36–42:

internal static void OnDispatcherUnhandledException(object? sender, DispatcherUnhandledExceptionEventArgs e)
{
    Services.Logging.GlobalExceptionLogging.OnDispatcherUnhandled(e.Exception);
    e.Handled = true;
    ShowOrDiscard(e.Exception, isTerminating: false);
}

This is intentional as a last resort. Per project convention we do not add a filter here for WebDataAccessRequestException; we fix the sources.

5. ConnectionStatusViewModel already drives the network-icon warning glyph

features/Phantom.Workspaces/ConnectionStatusWindow.axaml / features/Phantom.Workspaces/MainWindow.axaml:68–82 bind the top-right network icon's warning overlay to ConnectionStatus.HasProblem / ConnectionStatus.ProblemText:

IsVisible="{Binding ConnectionStatus.HasProblem, FallbackValue=False}"
ToolTip.Tip="{Binding ConnectionStatus.ProblemText}"

Today HasProblem is driven only by DevTunnelHostState.Error / Reconnecting set through SetDevTunnelStatus (ConnectionStatusViewModel.cs:138–159). There is no notion of a recent-errors list and no channel for client-side connectivity errors (like the 404 from refresh). This is exactly the surface the owner comment refers to.

Affected Files

File	Role
features/Phantom.Workspaces/ViewModels/MainWindowViewModel.cs (l. 1764–1769; l. 171–172)	OnRefreshTick async-void source site that must handle refresh failures instead of letting them crash.
features/Phantom.Workspaces/EntityBroker.cs (l. 327–380)	RefreshAsync — where GetChangedEntitiesAsync is issued; may also be a natural place to surface the connectivity failure.
features/Phantom.Workspaces/Services/DevTunnel/ReconnectingWebDataAccessLayer.cs (l. 102–123, 154–155)	Reconnect classification — should treat 404/service-unavailable as transient endpoint-availability failure.
features/Phantom.Workspaces.Data.Web.Client/WebDataAccessRequestException.cs (l. 28)	IsConnectivityFailure classification predicate.
features/Phantom.Workspaces.Data.Web.Client/WebClientDataAccessLayer.cs (l. 136–142)	Where the 404 becomes WebDataAccessRequestException.
features/Phantom.Workspaces/ViewModels/ConnectionStatusViewModel.cs (l. 122–159)	Network-status VM that owns HasProblem / ProblemText; must gain a recent-errors list and a "record client connectivity error" entry point.
features/Phantom.Workspaces/ConnectionStatusWindow.axaml	Add a "Recent errors" section bound to the new list.
features/Phantom.Workspaces/MainWindow.axaml (l. 68–82)	Warning overlay binding — already wired to HasProblem; HasProblem must also reflect the new recent-errors state.
features/Phantom.Workspaces/UnhandledExceptionHandler.cs	Not modified. Do not add a central benign-exception filter here.

Design / Fix (at-source; NO central filtering)

Three surgical, at-source fixes:

A. OnRefreshTick — catch and route to the connection-status VM

Refresh is a background, best-effort poll. It must never crash the app; failures should be recorded on the network-status VM and (optionally) logged.

// MainWindowViewModel.cs — at-source handling in the async void timer handler.
private async void OnRefreshTick(
    object? sender,
    EventArgs e)
{
    try
    {
        await this.EntityBroker.RefreshAsync();
    }
    catch (OperationCanceledException)
    {
        // Refresh was cancelled (shutdown / cancellation) — nothing to report.
    }
    catch (WebDataAccessRequestException exception)
    {
        // Transient/misrouted web-data call: surface as a connectivity problem
        // on the network status icon (exclamation overlay) and record for the
        // "recent errors" section of the connection-status dialog.
        this.ConnectionStatus?.RecordClientConnectivityError(exception);
    }
}

We deliberately catch only OperationCanceledException and WebDataAccessRequestException here — any other exception is a real bug and should still flow through the unhandled handler.

B. ReconnectingWebDataAccessLayer — treat endpoint-missing as transient

Extend WebDataAccessRequestException.IsConnectivityFailure (or the DefaultIsConnectionFailure predicate in ReconnectingWebDataAccessLayer) so that 404 and 503 are also treated as reconnectable. Rationale: a 404 for a known data endpoint like /data/get-changed-entities is not an application error but an availability problem (wrong/stale relay, host not serving the route yet). Retrying against a freshly resolved tunnel is the correct behaviour.

// WebDataAccessRequestException.cs
public bool IsConnectivityFailure =>
    this.StatusCode is null
    || this.StatusCode == HttpStatusCode.Unauthorized
    || this.StatusCode == HttpStatusCode.NotFound
    || this.StatusCode == HttpStatusCode.ServiceUnavailable
    || (int)this.StatusCode >= 500;

With this, ReconnectingWebDataAccessLayer.ExecuteAsync (l. 102–123) automatically re-resolves the tunnel and retries; the caller only sees the exception if the reconnect eventually fails (Failed status), at which point (A) surfaces it to the status VM instead of crashing.

C. ConnectionStatusViewModel — add a recent-errors list and expose it in the dialog

Extend the existing status VM so that HasProblem / ProblemText also reflect client-side connectivity errors, and add a bounded ring buffer of recent errors that the connection-status dialog can list.

// ConnectionStatusViewModel.cs additions (sketch)
private readonly ObservableCollection<RecentConnectivityErrorViewModel> recentErrors = new();
public ReadOnlyObservableCollection<RecentConnectivityErrorViewModel> RecentErrors { get; }

public bool HasProblem =>
    this.devTunnelState is DevTunnelHostState.Error or DevTunnelHostState.Reconnecting
    || this.recentErrors.Count > 0;

public string? ProblemText => this.HasProblem
    ? this.devTunnelError
        ?? this.recentErrors.LastOrDefault()?.Message
        ?? this.DevTunnelStatusText
    : null;

public void RecordClientConnectivityError(Exception exception)
{
    this.dispatch(() =>
    {
        this.recentErrors.Insert(0, new RecentConnectivityErrorViewModel(DateTimeOffset.Now, exception.Message));
        while (this.recentErrors.Count > MaxRecentErrors) this.recentErrors.RemoveAt(this.recentErrors.Count - 1);
        this.RaisePropertyChanged(nameof(this.HasProblem));
        this.RaisePropertyChanged(nameof(this.ProblemText));
    });
}

Add a "Recent errors" section to ConnectionStatusWindow.axaml bound to RecentErrors.

Why not a global catch?

Per project convention (#1301, #1303, #1322) we do not filter WebDataAccessRequestException centrally in UnhandledExceptionHandler. Doing so would silently swallow the same exception when it arises in unexpected call sites. Instead the fix is scoped precisely to:

• the fire-and-forget refresh timer that produced this crash,
• the reconnect-classification layer whose job is to know which HTTP failures are transient, and
• the network-status VM which already owns the warning glyph.

Expected Tests

New tests, following Subject_Scenario_ExpectedOutcome PascalCase in the existing Phantom.Workspaces.Tests files:

Test	File	Purpose
MainWindowViewModel_OnRefreshTick_WhenWebDataThrows_DoesNotCrashAndFlagsNetworkStatus	MainWindowViewModelTests.cs	Fire a refresh tick where EntityBroker.RefreshAsync throws WebDataAccessRequestException; assert no exception escapes, ConnectionStatus.HasProblem == true, and the error appears in RecentErrors.
MainWindowViewModel_OnRefreshTick_WhenUnexpectedExceptionThrows_StillPropagates	MainWindowViewModelTests.cs	Guard against over-broad swallowing: a non-connectivity exception must still surface.
ReconnectingWebDataAccessLayer_When404_ClassifiesAsTransientDisconnect	ReconnectingWebDataAccessLayerTests.cs	404 response triggers reconnect/retry, mirroring the existing Operation_On401_TriggersReconnectAndRetries shape.
ReconnectingWebDataAccessLayer_When503_ClassifiesAsTransientDisconnect	ReconnectingWebDataAccessLayerTests.cs	Sibling coverage for 503.
WebDataAccessRequestException_IsConnectivityFailure_Covers404And503	Phantom.Workspaces.Data.Web.Client.Tests	Directly asserts predicate.
ConnectionStatusViewModel_RecordClientConnectivityError_AppendsToRecentErrorsAndFlagsProblem	ConnectionStatusViewModelTests.cs	Records error, HasProblem/ProblemText reflect it, RecentErrors exposes it.
ConnectionStatusViewModel_RecentErrors_AreBoundedAndOrderedNewestFirst	ConnectionStatusViewModelTests.cs	Ring-buffer behaviour, ordering for the dialog.

Considered / Background

• Central filter in UnhandledExceptionHandler. Rejected — violates the "handle at source" convention (Crash: AggregateException: A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread. (The I/O operation has been aborted because of #1301/Crash: AggregateException: A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread. (SshServerSession disposed.) #1303/Crash: AggregateException: A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread. (Cannot access a disposed object. #1322) and would hide the same exception raised from unexpected call sites.
• Catch inside EntityBroker.RefreshAsync. Rejected as the sole fix: EntityBroker is used by many callers and should not silently swallow errors for all of them. OnRefreshTick is the correct at-source site because it is the fire-and-forget entry point where the caller has no await to observe the exception.
• Make OnRefreshTick return Task and let a caller await it. Not applicable — it is a DispatcherTimer.Tick event handler; the signature is fixed.
• Treat 404 as a permanent application error and just log it. Rejected: in practice a 404 on /data/get-changed-entities means the currently-resolved dev-tunnel target isn't serving the endpoint (stale relay / old host / not-yet-started). Re-resolving the tunnel is what actually fixes it; hence classify as transient in the reconnect layer.

[JoshuaRowePhantom]

Failing to connect to mongo queries should result in the network status icon having an overlayed exclamation mark and should have a section in the dialog box showing the most recent errors.

[JoshuaRowePhantom]

Design

Problem. A periodic OnRefreshTick (async void) awaits EntityBroker.RefreshAsync(). When the web data layer throws WebDataAccessRequestException (e.g. a stale dev-tunnel relay answering 404, or 503/5xx), the exception escapes the async-void handler and crashes the app.

Fix — three at-source changes (no central swallowing in the unhandled-exception handler):

A. MainWindowViewModel.OnRefreshTick — wrap the await RefreshAsync() in try/catch. Ignore OperationCanceledException (expected on teardown); catch WebDataAccessRequestException and route it to ConnectionStatus?.RecordClientConnectivityError(exception). Any other exception still propagates (real bugs must not be hidden).

B. WebDataAccessRequestException.IsConnectivityFailure — also treat 404 NotFound and 503 ServiceUnavailable as connectivity failures (in addition to null / 401 / >=500). A known data endpoint answering 404 means the resolved tunnel target isn't serving the route (stale/wrong relay), which re-resolving fixes, so ReconnectingWebDataAccessLayer should reconnect/retry.

C. ConnectionStatusViewModel — add a bounded (20), newest-first ObservableCollection<RecentConnectivityErrorViewModel> exposed as ReadOnlyObservableCollection RecentErrors, plus HasRecentErrors and RecordClientConnectivityError(Exception) (marshalled via the VM dispatch, raises PropertyChanged). HasProblem/ProblemText now also reflect recent errors. New RecentConnectivityErrorViewModel (timestamp + message). A "Recent Errors" section is added to ConnectionStatusWindow.axaml, bound to RecentErrors, visible when HasRecentErrors.

Tests (first): MainWindowViewModelTests (refresh swallows web-data error + flags status; unexpected exception still propagates); ReconnectingWebDataAccessLayerTests (404 and 503 trigger reconnect+retry); WebClientDataAccessLayerTests (IsConnectivityFailure covers 404/503); ConnectionStatusViewModelTests (records + flags problem; bounded, newest-first).

[JoshuaRowePhantom]

Implementation notes

Committed as f441d70b.

A. MainWindowViewModel — OnRefreshTick (async void) now delegates to a new testable internal Task RefreshOnceAsync(Func<Task> refresh) that awaits the refresh and: ignores OperationCanceledException; catches WebDataAccessRequestException and calls ConnectionStatus?.RecordClientConnectivityError(exception); lets every other exception propagate. Added using Phantom.Workspaces.Data.Web.Client;.

B. WebDataAccessRequestException.IsConnectivityFailure — now also true for HttpStatusCode.NotFound (404) and HttpStatusCode.ServiceUnavailable (503), in addition to null/401/>=500, so ReconnectingWebDataAccessLayer re-resolves the tunnel and retries.

C. ConnectionStatusViewModel — added bounded (MaxRecentErrors = 20), newest-first ObservableCollection<RecentConnectivityErrorViewModel> exposed as ReadOnlyObservableCollection RecentErrors, plus HasRecentErrors and RecordClientConnectivityError(Exception) (marshalled via the VM dispatch; inserts at 0, trims the tail, raises PropertyChanged for HasRecentErrors/HasProblem/ProblemText). HasProblem/ProblemText now also reflect recent errors. New RecentConnectivityErrorViewModel (Timestamp + Message). Added a "Recent Errors" section to ConnectionStatusWindow.axaml, bound to RecentErrors, visible when HasRecentErrors.

Tests (all passing, 7/7):

• MainWindowViewModelTests: MainWindowViewModel_OnRefreshTick_WhenWebDataThrows_DoesNotCrashAndFlagsNetworkStatus, MainWindowViewModel_OnRefreshTick_WhenUnexpectedExceptionThrows_StillPropagates.
• ReconnectingWebDataAccessLayerTests: ReconnectingWebDataAccessLayer_When404_ClassifiesAsTransientDisconnect, ReconnectingWebDataAccessLayer_When503_ClassifiesAsTransientDisconnect.
• WebClientDataAccessLayerTests: WebDataAccessRequestException_IsConnectivityFailure_Covers404And503.
• ConnectionStatusViewModelTests: ConnectionStatusViewModel_RecordClientConnectivityError_AppendsToRecentErrorsAndFlagsProblem, ConnectionStatusViewModel_RecentErrors_AreBoundedAndOrderedNewestFirst.

[JoshuaRowePhantom]

Verification result

Status: ✅ Verified

Scope: This comment verifies #1326 only (commit f441d70b). The branch batch/1326-and-4-more also carries #1333, #1330, #1332 and a submodule pointer-bump, which are out of scope here.

Commits on branch:

d674989e Update Phantom.Dock.Avalonia.TabSwitching to 7566bcdbe3c681fc02863d40327a171bb91bce8f
f441d70b Fix #1326: don't crash on connectivity failure during periodic refresh
7be08a7a Add #1332 app-level guard: Alt+Digit survives top-level workspace switches
c82421ce Add #1330 multi-region restore rendering tests
8ec223ef Fix #1333: route restored multi-region tabs to their own dock region

Feature coverage — all three at-source fixes present

• A. MainWindowViewModel.OnRefreshTick delegates to a testable internal Task RefreshOnceAsync(Func<Task>) that ignores OperationCanceledException, routes WebDataAccessRequestException to ConnectionStatus?.RecordClientConnectivityError, and re-throws every other exception (genuine bugs still surface).
• B. WebDataAccessRequestException.IsConnectivityFailure now also classifies 404 NotFound and 503 ServiceUnavailable (in addition to null/401/>=500); ReconnectingWebDataAccessLayer consumes this via DefaultIsConnectionFailure and re-resolves/retries.
• C. ConnectionStatusViewModel gains a bounded (20), newest-first RecentErrors (ReadOnlyObservableCollection), HasRecentErrors, and RecordClientConnectivityError (dispatch-marshalled, raises PropertyChanged); HasProblem/ProblemText reflect recent errors. New RecentConnectivityErrorViewModel (Timestamp + Message). ConnectionStatusWindow.axaml adds a "Recent Errors" section bound to RecentErrors, visible on HasRecentErrors.

Branch & test coverage

• 404-doesn't-crash path: covered (..._WhenWebDataThrows_DoesNotCrashAndFlagsNetworkStatus).
• Cancellation ignored vs unexpected re-throw: covered (..._WhenUnexpectedExceptionThrows_StillPropagates); OperationCanceledException swallow branch exercised implicitly by the propagate test's contrast — predicate paths verified.
• Classification 404/503 transient reconnect: covered (..._When404_..., ..._When503_...).
• Predicate directly asserted incl. negative 400 case: WebDataAccessRequestException_IsConnectivityFailure_Covers404And503.
• Recent-errors record/flag + bounded newest-first ordering: two ConnectionStatusViewModelTests.

No disabled/quarantined tests, no unresolved TODOs, no timing-dependent synchronization (uses deterministic Dispatcher.UIThread.RunJobs()), no synchronous task blocking. Diff touches no filesystem/Git repository layers — needs-slow-tests not required.

Tests

Ran MainWindowViewModelTests,ConnectionStatusViewModelTests,ReconnectingWebDataAccessLayerTests,WebClientDataAccessLayerTests — 68 executed, 68 passed, 0 failed.

Passed!  - Failed:     0, Passed:     7, Skipped:     0, Total:     7, Duration: 678 ms - Phantom.Workspaces.Agent.Gui.Tests.dll (net10.0)
Passed!  - Failed:     0, Passed:     8, Skipped:     0, Total:     8, Duration: 154 ms - Phantom.Workspaces.Data.Web.Client.Tests.dll (net10.0)
Passed!  - Failed:     0, Passed:    53, Skipped:     0, Total:    53, Duration: 24 s - Phantom.Workspaces.Tests.dll (net10.0)
=== Test Run Summary ===
Executed : 68
Passed   : 68
Failed   : 0

Outstanding gaps

None — all criteria satisfied.

[JoshuaRowePhantom]

Checked in

Commit: d674989
Branch fast-forwarded: features → d674989