Make Iseberg NativeAOT-safe with a private PowerShell subprocess

[Marc-André Moreau (mamoreau-devolutions)]

Summary

• Replace in-process SMA with an owned pwsh child that imports a DT-shipped managed binary module and hosts a persistent custom-host runspace.
• Keep the NativeAOT desktop SMA-free using source-generated contracts and authenticated, bounded, bidirectional IPC for parsing, completion, execution, debugger, host input and $psISE callbacks.
• Automatically discover an Iseberg profile alongside detected PowerShell on Windows/Linux/macOS. Use a stable generated identity, preserve existing profiles/defaults and user overrides, honor the existing shell-discovery source's disabled setting, and omit generation from terminal-only builds.
• Ship just Devolutions.Iseberg.PowerShell.dll, directly beside dt.exe and Devolutions.Terminal.exe, not a separate contracts DLL or PowerShell/CoreCLR. Compile canonical SMA-free contract sources into the child module and separately into the parent contracts project. A shared target generates matching build fingerprints and JSON serialization; no assembly merger or duplicated source files.
• Preserve explicit terminal-only builds. macOS app bundles relocate the renamed DLL to Contents/Resources, keeping Contents/MacOS Mach-O-only. MSI installs the module directly under <INSTALLLOCATION>, with no bridge subdirectory.
• Windows release CI includes the module in its existing recursive Authenticode signing pass before MSI/NuGet packaging. Both architecture payload checks explicitly require the module, its trusted timestamped signature, and the same exact DT publisher as the executables. Ordinary PR builds/local publishes remain unsigned, as do the other binaries.
• Preserve documents on child/session loss with explicit reopen guidance, no automatic restart/replay. Fix cancellation/disposal races and asynchronous workbench lifecycle handling.

Latest-head CI: green (1138163)

Both completed workflows target 1138163a5fc78570aed24b5ef56d2bd7acedbf06:

• Pull-request run 37944662672: success.
• Push run 37944655758: success.

The current PR has 48 passed checks, four expected release-only skips, no failures or pending checks. Windows/Linux/macOS managed suites pass across PowerShell 7.4.6, 7.5.11 and 7.6.6. NativeAOT publishing, MSI/MSIX/Linux packaging, Linux ARM64 runtime, NuGet distribution, macOS NuGet signature checks, and browser WASM/E2E gates all pass. Windows signing-selection/verification regressions pass in their managed CI jobs. Protected production release signing/notarization/publication is not claimed by these ordinary PR/push workflows.

Compatibility

The child module and parent contracts project target .NET 8 against SDK 7.4.20. Supported installed runtimes: 7.4.6+ within 7.4.x, 7.5.x and 7.6.x, with bundled .NET 8/9/10. Earlier 7.4 patches cannot bind the patched SDK's SMA 7.4.6.500 reference; recommend current servicing releases. Automatic discovery does not eagerly start PowerShell; compatibility is validated when opening the workbench.

CI runs pinned 7.4.6, 7.5.11 and 7.6.6 managed suites on Windows/Linux/macOS, plus default/terminal-only native publish gates.

Windows CI cleanup repair (1138163)

At 170fba7, the PR's Windows 7.6.6 App run failed only while deleting the isolated module copy: UnauthorizedAccessException from Directory.Delete. That run passed the other 588 App tests and all 665 UI tests. The Linux/macOS managed matrices and Windows/macOS/Linux native publishing/packaging checks also passed.

The isolated-module regression now explicitly verifies session disposal and actual child termination before directory cleanup. Windows access/sharing/lock errors are logged and retried for at most five seconds, then propagated; non-Windows and other filesystem errors are not retried. Engine assertions, per-test hang guard, suite budget and production behavior are unchanged. Copied file attributes are retained in test diagnostics.

After the repair, the complete App suite passes 589 tests, 0 failed, 0 skipped on each of 7.4.6, 7.4.20, 7.5.11 and installed 7.6.6 locally. Successful command: dotnet test tests\Devolutions.Terminal.App.Tests\Devolutions.Terminal.App.Tests.csproj -c Release -p:SkipNativeRestore=true --no-build; DT_ISEBERG_PSHOME selected each isolated runtime. All fresh latest-head CI checks also pass as recorded above.

Renamed module/layout/signing verification

Requirement	Evidence
Devolutions.Iseberg.PowerShell.dll, discovered beside executables, retains private IPC behavior	SingleAssemblyModuleAuthenticatesAndExecutesWithoutContractsDll asserts the exact root discovery path, copies only this DLL, authenticates, checks the child assembly identity Devolutions.Iseberg.PowerShell, verifies no contracts reference/load, executes persistent state, parses incomplete input, checks the SMA-free parent, and verifies disposal/child exit. Full App runs above include this regression.
Native publish and MSI root placement	Release solution build: zero warnings/errors. Fresh win-x64 NativeAOT desktop/CLI publish has the renamed module at the root and neither executable has a CLR header. No old bridge directory/DLL, contracts DLL, SMA or CoreCLR. Actual MSI component generation maps exactly one renamed module directly to INSTALLLOCATION. Published Windows GUI-subsystem dt.exe --help was explicitly waited for and exited 0.
Release signing input includes the module	IsebergModuleIncludedInReleaseSigningInput executes the actual Sign-Packages.ps1 against an inert signer fixture and verifies the complete input-file list, including the root DLL.
Module signatures are mandatory and first-party	TrustedTimestampedRecursivePayload, UnsignedBinaryRejected-Devolutions.Iseberg.PowerShell.dll, MissingTimestampRejected-Devolutions.Iseberg.PowerShell.dll, PublisherMismatchRejected-Devolutions.Iseberg.PowerShell.dll, and MissingRequiredIsebergModuleRejected prove the signature-verification contract. All 17 regressions passed using pwsh -NoLogo -NoProfile -File src\Devolutions.Terminal.Package\Scripts\Test-WindowsPayloadSignaturesRegression.ps1. These use simulated certificate/tool responses, not production signing credentials.
Terminal-only compatibility and strict payload boundary	Fresh feature-disabled native publish contains no bridge/contracts DLL. TerminalOnlyPayloadDoesNotRequireIsebergModule preserves signature validation for explicitly terminal-only payloads. Eight checks against the actual publish target accept the root module and reject a missing module, old root DLL, old bridge directory, nested renamed module, separate contracts DLL, SMA and CoreCLR.

The original renamed-module process/handshake subset passed 14 tests on 7.4.6 and 14 on installed 7.6.6 using dotnet test tests\Devolutions.Terminal.App.Tests\Devolutions.Terminal.App.Tests.csproj -c Release -p:SkipNativeRestore=true --no-build --filter 'FullyQualifiedName~PortableIseProcessTests|FullyQualifiedName~PortableIseHandshakeTests'. Fresh manual-test native output is under artifacts\iseberg-module-renamed\win-x64; the follow-up changes only test cleanup and documentation.

No additional comprehensive native GUI verification was run, as requested. Release-only CA signing requires the existing protected signing environment; normal PR CI exercises its signing-selection/verification regressions rather than claiming signed release artifacts.

Prior automatic-profile verification

• Full Settings suite: 202 passed. Platform fixtures cover Windows/Linux/macOS, absence without pwsh (Windows PowerShell 5 alone is insufficient), exactly one generated Iseberg with multiple PowerShell installations, stable identities, source disabling, and user overrides/defaults preserved through serialization.
• AutomaticIsebergProfileFollowsDetectedPowerShellAndBuildFlag, MissingPowerShellDoesNotGenerateIseberg, DisabledPowerShellSourceAlsoDisablesAutomaticIseberg, and AutomaticIsebergPreservesExistingProfilesDefaultAndUserOverrides provide focused behavioral evidence.
• Terminal-only profile subset: 31 passed, using dotnet test tests\Devolutions.Terminal.Settings.Tests\Devolutions.Terminal.Settings.Tests.csproj -c Release -p:EnablePowerShellIse=false --filter 'FullyQualifiedName~DynamicProfileGeneratorTests|FullyQualifiedName~LinuxRuntimeEnvironmentTests'.
• App suite: 589 passed on installed 7.6.6; focused profile UI/action routing: 42 passed.
• NativeAOT publish succeeded and a newly launched manual-test window showed the generated Iseberg entry in its profile dropdown. Existing manually created Iseberg profiles were preserved separately. This dropdown observation is not a new workbench smoke/parity claim.

Prior single-DLL verification (before the assembly rename/layout change)

• Full App suite: 589 passed on 7.4.6, 7.4.20, 7.5.11 and installed 7.6.6 after assembly consolidation.
• Full UI suite: 665 passed on exact minimum 7.4.6 with the single-DLL module. Complete per-project Settings/App/UI/settings-editor/connection evidence totaled 1,639 passed, 0 failed, 9 existing Unix-only skips.
• The isolated single-assembly test verified contracts belonged to the then-named Iseberg.PowerShell assembly, no Iseberg.Contracts reference/load, persistent execution/parser behavior and the SMA-free parent.
• Default and terminal-only win-x64 NativeAOT publishes succeeded; the former bridge subdirectory contained exactly one DLL, terminal-only contained no bridge, and both executables had no CLR header. MSI component generation mapped the former module into its former install subdirectory.
• Additional comprehensive native GUI smoke for consolidation was skipped at the user's request; prior native GUI evidence remains historical.

Prior native GUI evidence

Before assembly consolidation, delivery-published native GUI automation passed on 7.4.6/.NET 8.0.10 and installed 7.6.6/.NET 10.0.12: editor/native-console shared state, accepted completion, typed Read-Host, Stop/recovery, $psISE reverse callbacks, saved-script breakpoint/Step Over/Continue, resize/RawUI updates and cleanup. A prior compatibility publish also passed on 7.4.20.

Each retained the same child PID and loaded the then-two published net8 bridge/contracts assemblies. Parent enumeration found 80 modules with no SMA/CoreCLR/hostfxr/hostpolicy; all eight smoke-owned processes exited normally. An opening/pre-initialization resize could not be observed through GUI automation; ResizingDuringInitializationDefersIpcUntilTheSessionIsReady covers it through a real child instead.

Cross-platform CI repairs

• Shorten randomized protocol-test endpoints to fit macOS Unix-domain socket limits without reducing GUID entropy.
• Acknowledge and serialize terminal-size updates before subsequent execution; retain startup dimensions locally until authentication and initialization complete.
• Queue background command-catalog refreshes behind execution instead of producing a racing busy-session error.
• Allow 120 minutes for the full sequential Windows test step while retaining its five-minute individual-test hang guard.
• Bound and diagnose Windows-only cleanup of the isolated loaded-module fixture, after independently asserting child termination.

These are representative ISE workflows, not exhaustive Windows PowerShell ISE parity. Boundaries and evidence are documented in docs/iseberg.md.