Repository navigation
Add cross-platform adapter compatibility validation - #1
Conversation
Prepare the shared specimen, pinned profiles, separate stop reporting and Windows/Linux/macOS Actions matrix. Prepared jointly by Neil Colvin and OpenAI Codex.
|
@OsirisTerje @CharliePoole We have completed the Windows, native macOS and Linux VM checks against the published MyGet engine 3.23.0-rc.1. The Linux result needs a qualification. Inputs were SDK 11.0.100, NUnit 4.6.1 and 5.1.0-alpha.0.50, and the same frozen fixtures and package hashes on all three platforms. We used an isolated adapter build, 6.4.0-local.engine323rc1.20261118, based on adapter source
The previous NUnit 5 stop-signature exception is absent from the executed RC probes. The Windows comparison directly confirms six signature failures with engine 3.22 disappearing with 3.23.0-rc.1. On Linux, both .NET 11 VSTest per-test timer controls initially recorded the expected cancellation failure, finally and teardown, but no registered token-callback marker. The harness correctly stopped before attempting runner cancellation. In a focused repeat with unchanged fixture source, NUnit 4 passed its control and reached runner stop; NUnit 5 still failed the control. Linux NUnit 5 / .NET 11 / VSTest runner stop therefore remains unvalidated. The original failed matrix is retained, and the cause of the missing marker is unresolved. Across the runs and focused repeat, 35 distinct runner-stop combinations reached a confirmed test start. All 35 still lacked token-callback, finally and teardown markers before bounded containment. The signature correction and cooperative cancellation remain separate outcomes. Process cleanup audits passed, the shared VM is stopped, and the existing gates are released. These results support the signature correction while leaving the Linux control failure and cooperative-stop behavior open for investigation. Prepared jointly by Neil Colvin and OpenAI Codex. |
Keep the compatibility fixture's observer registered until the real callback has written its receipt. Bound the wait so a missing callback remains a failing control. This prevents cancellation of Task.Delay from letting fixture cleanup unregister an observer that is still pending. Production components, package pins and runner-stop policy are unchanged. Prepared jointly by Neil Colvin and OpenAI Codex.
|
@OsirisTerje @CharliePoole Follow-up: the Linux callback-marker qualification is now traced to a race in our compatibility fixture. The token could cancel the awaited delay before dispatch reached our observer callback. Our using block then disposed the observer registration. In six of twenty real reused-host VSTest cases, the token was already cancelled and cleanup successfully unregistered the still-pending callback. This explains the expected CancelAfter failure and finally/teardown receipts without a callback receipt. The fixture correction retains that registration until the actual callback writes its receipt, with a bounded wait. Twenty equivalent traced controls then recorded all required markers. Independent controls reproduce the registration-lifetime issue without NUnit on .NET 8/10/11; it is not specific to a new .NET 11 or engine defect. The complete corrected-fixture Linux RC matrix now passes its compatibility checks, including the previously blocked NUnit 5/.NET 11 VSTest path. All twelve real runner-stop probes still show the separate cooperative-cancellation gap. Windows and native macOS controls also pass, as do Windows controls using this PR's unchanged historical adapter profile. This is a one-file fixture correction; production engine/adapter/framework code, package pins and workflow policy are unchanged. Original failures and process-cleanup evidence are retained. Prepared jointly by Neil Colvin and OpenAI Codex. |
|
@OsirisTerje We have completed a controlled Windows before/after investigation of the remaining stop gaps, using adapter source 7a4456ac, the exact published NUnit.Engine 3.23.0-rc.1 binaries, and NUnit 4.6.1 / 5.1.0-alpha.0.50. The modified adapters are explicitly local integration packages, not official releases. No engine or Microsoft package was modified. First, changing the normal request from StopRun(true) to StopRun(false) delivers the real NUnit 5 token callback, finally and teardown through both VSTest and MTP on .NET 8, 10 and 11. The original forced-stop runs deliver none of those markers. The RC's modern driver deliberately rejects forced stop for NUnit 5; cooperative stopping and host-owned hard termination are separate responsibilities. We then isolated reporting from that execution fix. The cooperative-stop baseline still discards completion events after cancellation and forces MTP process exit. In a separate local variant we let completion/cleanup events drain, retain suppression of new starts, await MTP execution and dispose normally, and remove the premature exit/synthetic-result paths. Real VSTest cancellation now returns one matching terminal result on all three runtimes, with no duplicates. A queued sentinel does not execute, and a delayed teardown failure retains its actual failure message. MTP revealed an important distinction. The --timeout probe aborts the application/request, and the pinned VSTest bridge rejects late RecordEnd/RecordResult calls once that request token is cancelled. Removing NUnit's process exit alone does not make that a report-preserving graceful stop. Those abort cases still have empty reports; we retain them as unresolved rather than calling the whole lifecycle fixed. The supported alternative is MTP's public graceful-stop capability. We prototyped IGracefulStopTestExecutionResultCapability with execution-owned registrations and AddMaximumFailedTestsService. The real --maximum-failed-tests 1 policy then stops an already-running NUnit test, completes its asynchronous cleanup, and produces a valid TRX on .NET 8, 10 and 11. The public node-update stream contains exactly one terminal update for each of the two started identities, after teardown, and the application exits normally with the policy's documented code 13. The deliberate trigger and framework cancellation retain their actual Failed outcomes. Fourteen focused tests pass on both repository runtimes, and all four standard adapter test runs pass. Six of the seven reporting regressions fail against the preserved baseline; the existing new-start suppression case passes before and after. The new capability also covers overlapping executions, repeated/concurrent stop requests, closed scopes and later-run ownership. A deliberately non-cooperating case confirms that receiving cancellation does not establish execution stopped. Two qualifications for review: the MTP graceful-stop API/option are public experimental contracts in 2.5.2, and these new results are Windows-only. NUnit 4's run-wide token gaps remain recorded separately from its passing timer/ordinary compatibility controls. Our remaining question is how adapter/IDE Stop should select graceful stopping versus request abort. Is the graceful capability the intended integration point for that path as well, or does the bridge need a separate lifetime for publishing terminal results of already-started tests? We have kept the prototype local and can propose the source changes in separate reviewable slices. There is no claim that advertising this capability automatically changes current IDE cancellation semantics. Prepared jointly by Neil Colvin and OpenAI Codex. |
|
@OsirisTerje Follow-up to our controlled Windows results: the same frozen before/final adapter packages now verify on Linux ARM64 and native macOS ARM64 with SDK 11.0.100 and .NET 8/10/11. No adapter, engine, framework or fixture bytes changed for these native runs. Actual runtimes were 8.0.32 / 10.0.13 / 11.0.0 on Linux and 8.0.33 / 10.0.13 / 11.0.0 on macOS. Fourteen focused adapter tests pass on native .NET 8. Both complete compatibility matrices pass on each OS. VSTest cancellation returns one matching terminal result with the final adapter, compared with zero in the cooperative-stop baseline; queued execution is suppressed and the delayed teardown failure is retained. The real MTP maximum-failure graceful-stop policy works on all three target runtimes on each OS. Its public event stream has one terminal update per started identity, after asynchronous teardown, with no duplicates and a valid TRX. Exit 13 is the intentional failure-limit outcome; actual failing/cancelled results remain Failed. The non-cooperating control still requires bounded owner containment and does not acquire a synthetic completion. The distinction in our preceding reply remains: application/request abort still closes the pinned bridge's reporting lifetime, and NUnit 4's run-wide token gaps remain separate from passing timer/compatibility controls. These results do not imply existing IDE Stop requests automatically use the new graceful capability. Two harness portability issues were retained and corrected: the Unix socket test directory needed MTP's supported short-pipe setting, and macOS Python 3.9.6 needed ISO parsing that accepts variable fractional precision. The latter was corrected by re-verifying the recorded data, without rerunning tests or changing results. Process audits and input hashes pass; the VM is stopped and shared gates are released. The source prototype remains local for review. Prepared jointly by Neil Colvin and OpenAI Codex. |
Adds a standalone integration specimen for checking NUnit framework candidates
through the NUnit adapter's direct MTP and VSTest CLI/TranslationLayer paths.
The same explicit profile and command-line runner are used locally and by the
Windows/Linux/macOS Actions matrix.
The runner checks exact-name filtering, escaping, per-test cancellation and actual
target runtime. It reports cooperative runner-stop observations separately; the
historical alpha's existing cancellation gap is preserved as INCOMPLETE. Strict
stop validation is available through the CLI and manual workflow dispatch. Missing
or contradictory evidence is an infrastructure failure, and compatibility failures
always fail the job. No continue-on-error step suppresses failures.
Package versions, public download hashes, SDK, targets and dependency locks are
selected by reviewed profiles. A separate prepare-locks command proposes locks
for another candidate without changing maintained source or running tests. Each
run uses an isolated snapshot and caches, verifies NUnit's C# style, retains its
failures and performs a native process audit. Artifacts omit binaries and caches.
This adds no framework/adapter/engine production changes, public NUnit API,
deployment, package publication or private-machine dependency. It supplements the
framework suite; x86, Framework/Mono, IDE UI and remote execution are outside scope.
Validation before posting: 16 reporting/profile tests pass; five Release projects build with zero warnings/errors; both NUnit C# style checks and actionlint 1.7.12 pass. Windows checks pass 30 raw valid selections, 12 escaped selections and six per-test cancellation controls, while preserving six invalid-filter rejections and all 12 known stop gaps. The strict-stop summary gate fails that gap as expected. A separate NUnit 5.0.0 lock proposal verifies candidate selection without changing the maintained alpha locks.
Hosted validation: the PR-triggered matrix passed on Windows Server 2025 x64, Ubuntu 24.04 x64 and macOS 15 ARM64. All three artifact downloads were independently checked against their Actions SHA-256 digests. Each records 30 valid raw selections, 12 escaped selections, six per-test cancellation controls, six invalid-filter syntax rejections and a passing native-process audit. The 12 known runner-stop gaps remain INCOMPLETE on each platform; green compatibility jobs do not establish complete cooperative stop.
For nunit/nunit#5593. This remains a draft for @OsirisTerje�s review; no merge is proposed.
Prepared jointly by Neil Colvin and OpenAI Codex.