Repository navigation
test(conformance): run the WDIO service through the suite - #433
Merged
Merged
Conversation
Contributor
[Medium impact] The PR appears safe to merge with PR #434’s dependency cleanup. SummaryAdds the WDIO service to the conformance suite with and without BiDi.
Diagram%%{init: {'theme': 'neutral'}}%%
flowchart LR
Harness[Conformance harness] --> CLI[WDIO CLI]
CLI --> Service[Devtools service]
CLI --> Fake[Fake WebDriver and BiDi browser]
Service --> Backend[Recording backend]
Fake --> Truth[Browser actions and documents]
Backend --> Checker[Rules and adapter comparison]
Truth --> Checker
Reviews (5) · Last reviewed commit: "chore(conformance): run the WDIO scenari..." · Reviewed by Greptile |
vishnuv688
force-pushed
the
feat/300-nightwatch-scenario
branch
from
October 19, 2026 11:42
d9ed6a6 to
0e21d24
Compare
vishnuv688
force-pushed
the
feat/300-wdio-scenario
branch
from
October 19, 2026 11:43
4e93f4f to
f3eb951
Compare
The dashboard's per-runner defaults and the declared-over-defaults rule lived in the app's sidebar, where nothing outside it could apply them. The conformance suite needs the same rule to know which controls a stream advertises, so the tables and the resolution move to shared and the app delegates to them. The runner lookup moves with it: the typed runner first, then the older options.framework.
…ard reads them Three rules read the stream in a way only the first two adapters met. Run controls: the dashboard offers every control a runner's defaults allow unless metadata turns it off, so silence advertises all of them, and the WDIO service serves its config through a separate config frame rather than metadata.options.configFile; the rule now applies shared's resolution and accepts either carrier, so a declared set is read over those defaults and a control is absent only when declared false. Fills: the WDIO service also leads a fill with its selector, which reading the text as the trailing argument already handles; a test pins it. Errors: a row whose error has no message is refused where the stream is read, instead of crashing the normaliser.
Once a session offers a BiDi socket, WebdriverIO routes navigation, context lookup, element search and script calls over it. The fake acknowledged those with an empty result, so the browser never left about:blank and the run collapsed. The endpoint now answers every command from the same two-page model the classic routes use, stamps a navigation as the same user action, and refuses /broken with the same message, so an adapter is judged on what it captured rather than on which protocol its client picked.
beforeSession published the config file on the placeholder capturer the plugin starts with, which has no socket, so the frame was dropped on every run. A backend the launcher itself starts inherits the path from the environment, which hid the loss; a dashboard the service attaches to, including the one a rerun reports into, never learned the file. The frame now goes out from before(), on the capturer that connects, and the publish lives beside the detector it uses.
The row stored the thrown Error as is. Its message is non-enumerable, so
the live socket received error: {} and the Errors tab had nothing to
show. The error is now serialized at the row, and webdriver's
decoration, "WebDriverError: <message> when running <endpoint> with
method <M>", is stripped back to the browser's own text, which is what
the other adapters carry. The serialized shape is shared's; core's copy
becomes a re-export.
setValue and addValue rows carried the selector only as a leading argument, which the trace reads positionally, while the other adapters put it on CommandLog.selector. The row now carries both, so a reader of the shared field sees the same target from every adapter.
The service runs under its own CLI from scenarios/wdio, attached to the recording backend through the reuse handshake, in both modes. The CLI resolves its runner, framework and service from the working directory, so a scenario launch may name one. WebdriverIO 10 could not coexist with the version 9 examples under pnpm's hidden hoist, because @wdio/utils resolves plugins from its own location before the project's; the workspace no longer hoists @wdio packages, so each project resolves its own. Pinned at 10.0.1: the registry view pnpm resolves from does not yet list 10.0.2, and the 10.0.2 CLI pins dependencies it cannot resolve.
test:ui and the demo:wdio scripts run the root wdio binary against configs that name the browser runner, the local runner, the mocha and cucumber frameworks, the spec reporter and the devtools service. Those were declared only in packages/app and examples/wdio and reached the CLI through pnpm's hidden hoist, which the workspace no longer applies to @wdio packages; CI's component tests failed with "Couldn't find plugin browser runner". The root now declares what it loads, which is where the CLI's fallback resolution looks.
Given a result, WebdriverIO reads a navigation id from it and then waits for the network events of that navigation, which the fake browser never emits; version 9 waits even when the id is null. With no result at all, neither version waits, and the row the adapter captured is what gets judged.
…rIO 9 A second WebdriverIO major cannot be isolated inside this workspace. @wdio/utils resolves the runner, framework and services from its own location before the project's, and both pnpm's hidden hoist and the root node_modules sit on that path: excluding @wdio from the hoist broke test:ui and the demos, which load their plugins through it from the root, and declaring those plugins at the root handed the version 9 runner back to the version 10 CLI. The scenario therefore runs on the pins the examples use, with expect-webdriverio declared so peer auto-install does not pull a pre-release that drags webdriverio 10 into the graph. The wdio-peer job keeps exercising the service under webdriverio 10.
vishnuv688
force-pushed
the
feat/300-wdio-scenario
branch
from
October 19, 2026 12:08
f3eb951 to
1693dcc
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #432 (Nightwatch scenario); base is
feat/300-nightwatch-scenario. Part of #300.What
The WDIO service now runs through the conformance suite in both modes, from
packages/conformance/scenarios/wdio/, under its own CLI and attached to the recording backend through the reuse handshake. All four adapters are green, with no new allowlist entries.What the first runs found
Adapter bugs, fixed here in
@wdio/devtools-service(changeset included):configframe was published frombeforeSessionon a capturer with no socket, so it was dropped every run; only a backend the launcher itself starts, which inherits the path from the environment, ever learned the file. It is now sent frombefore().error: {}: the rawErrorwas stored and its message is non-enumerable. The row now carries a serialized error with the browser's own message; webdriver'sWebDriverError: <message> when running "<endpoint>" with method "<M>"decoration is stripped, since the other adapters carry the bare message.setValue/addValuerows carried the selector only as a leading argument; they now also stampCommandLog.selectorlike every other adapter.Suite assumptions, fixed in
packages/conformance:browsingContext.navigate,browsingContext.getTree,browsingContext.locateNodes,script.callFunction), from the same two-page model as the classic routes.run-controls-are-servedapplies the dashboard's own defaults (silence advertises every control) throughresolveRunCapabilities, now in@wdio/devtools-sharedwith the app delegating to it, and accepts theconfigframe as what serves them.Toolchain
examples/wdio, withexpect-webdriveriodeclared so pnpm's peer auto-install does not pick a pre-release. A WebdriverIO 10 run cannot coexist in this workspace:@wdio/utilsresolves the runner, framework and services from its own location, and both pnpm's hidden hoist and the rootnode_modulessit on that path, so whichever major is hoisted or declared at the root wins for every project. Moving the repo to 10 is its own change; thewdio-peerjob already exercises the service under webdriverio 10.Verification
pnpm build,pnpm lint,pnpm testare green (201 files, 2596 tests), including the conformance suite for all four adapters in both modes and the cross-adapter comparison.packages/service/tests/index.test.ts), the error on the wire (packages/service/tests/session.test.ts,packages/service/tests/command-error.test.ts), the selector stamp (packages/service/tests/command-selectors.test.ts), the fake's BiDi path (packages/conformance/tests/fakes.test.ts), the rule changes (packages/conformance/tests/checker.test.ts,packages/conformance/tests/agreement.test.ts), shared's resolution (packages/shared/tests/run-capabilities.test.ts).