Repository navigation
Build/Test Tools: Test the whole plugin directory locally against a baseline. - #15382
Draft
adamsilverstein wants to merge 46 commits into
Draft
adamsilverstein wants to merge 46 commits into
adamsilverstein wants to merge 46 commits into
Conversation
Core CI covers core itself, but nothing checks that a new version of WordPress can still boot with popular plugins active. When a plugin's assumptions about core stop holding the result is a fatal error on every request, which is a white screen for real sites and is only discovered after release. Add a workflow that fetches the most popular plugins from the WordPress.org API at run time, then installs and activates each one on its own against the version of WordPress under test. A fatal is caught whether it happens on activation, while WP-CLI loads WordPress, on a front end or login request, or in the debug log, so a white screen with error display turned off is still detected. Failures are reported per plugin in the workflow summary and one broken plugin never stops the rest of the shard from being tested. Plugins that cannot be downloaded are reported as skipped rather than failed so that a network flake does not turn the run red. The run is manual or weekly rather than part of every commit, since a third party plugin breaking should be a signal to release leads, not a red check on unrelated work.
The first run of this workflow reported four failures that were not fatals. Core refuses to activate a plugin whose `Requires Plugins` dependency is missing, which every WooCommerce extension hits when plugins are tested one at a time, and WP-CLI exits non-zero when a plugin redirects while loading. Both are correct behaviour, so record them as skipped and reserve a failure for an actual fatal. A front end request also reported the nonsense status "200000", because the curl fallback appended to output curl had already written. Capture the exit code separately so a stalled transfer is reported as what it is. The stall itself came from WordPress spawning WP-Cron as a loopback request that the single threaded built-in server could not answer while still serving the request that spawned it. Disable WP-Cron and give the server workers so plugin loopback requests cannot deadlock it. Ignore the zizmor unpinned image finding on the database service, which cannot be pinned to a digest while the version is an input.
The WordPress.org API caps `per_page` at 250 and quietly returns 250 for anything larger, so asking for more than that silently tested fewer plugins than requested. Page through the API instead and trim to the requested count, de-duplicating across pages because popularity ordering can shift between two requests. Reject a count above 1000 with a clear message rather than truncating without saying so. Size the shards to the plugin count rather than always splitting into five, so a run of 10 does not spin up five near empty jobs and a run of 250 is not squeezed into the same five. Point pull request and push runs at the latest stable release with a small count. Those runs exist to check that this workflow still works, and a genuine ecosystem fatal against nightly should not sit as a red check on every later change to these files.
A run that builds a single shard logged "across 1 shards".
…sting. Testing each plugin on its own meant core refused to activate anything with a `Requires Plugins` header, so every WooCommerce extension was recorded as skipped without ever being loaded. Those extensions are a large slice of the most popular plugins, which left a gap in exactly the part of the ecosystem the workflow exists to watch. Read the `Requires Plugins` header with WordPress' own parser, install and activate what it names, and only then activate the plugin under test. Chains are followed a level at a time so a dependency declaring its own is installed too, and anything already present is left alone, which terminates a circular declaration. Dependencies mean the plugin under test is no longer alone on the site, so the front page and login screen are checked with only the dependencies active first. A site already broken at that point is the dependency's doing, and the plugin is skipped rather than blamed for it. The debug log is cleared at the same point for the same reason. The results table gains a column naming what else was active so a failure can be read in context. Installing and activating a dependency the size of WooCommerce is not quick, so the job timeout goes from 30 to 45 minutes.
Trying this workflow out meant taking whatever the popularity query returned, so anyone wanting to check one plugin against a release candidate, or to reproduce a failure from an earlier run, had to sit through a full run and read past everything else in the shard. Reviewers could not run it at all: the repository guard skips both jobs everywhere but this repository, and dispatching it here needs write access. Add an optional `plugin-slugs` input that replaces the directory query with a list written by hand. Commas and whitespace both separate slugs so a list can be pasted in however it was written down, and a value outside the character set the directory uses stops the run immediately rather than being reported as an impossible download several minutes later. The list goes through the same sharding as a fetched one, so naming forty plugins still splits across two jobs. Let a dispatched run through the repository guard wherever it is triggered, which is what makes a fork useful for trying this out. Two runs dispatched against the same version of WordPress used to land in the same concurrency group and cancel each other, which is wrong once the plugins being tested can differ between them. Dispatched runs are now grouped by run id.
…dd/plugin-compatibility-workflow
`slack-notifications.yml` no longer accepts `SLACK_GHA_TIMEOUT_WEBHOOK`, and every other caller stopped passing it, so actionlint failed on this workflow once trunk was merged in.
…a terminal. Move the per-plugin checks out of the reusable workflow and into tools/plugin-compatibility/test-plugins.sh, so that reproducing a result from a run no longer means dispatching a workflow and waiting for it. The script provisions a database and a PHP with WP-CLI in throwaway containers by default, so nothing but Docker is needed locally. The workflow passes --no-docker and runs the same code against the runner's own PHP and database service, which keeps one copy of the logic that decides whether a plugin passed.
WP-CLI requires wp-settings.php from inside a method, so a plugin that assigns a variable at file scope and reads it back with global later finds nothing there. eps-301-redirects does exactly that and fatals under WP-CLI on a fresh install, while the front page, the login screen and the debug log are all clean. Failing it reported breakage that no visitor would ever see. Move the WP-CLI boot check after the HTTP and debug log checks and stop letting it fail a plugin on its own. A plugin that is healthy over HTTP now passes with the WP-CLI fatal recorded as a note. Anything that genuinely fatals on load still fails, because it fails the HTTP checks. The reorder also matters because the WP-CLI fatal is written to the debug log, which the log check would otherwise pick up as a failure.
The built-in server forks the workers PHP_CLI_SERVER_WORKERS asks for, and killing the parent alone left them serving on the port. A --no-docker run straight after could then hit the old site or fail its readiness check. The server now starts in its own process group, which is signalled as a whole and waited on until every worker has exited. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01211BLKtDajir1Xg7etxT7R
Reversing the discovery order is not dependency order. A plugin requiring `base, extension`, where extension also requires base, had extension activated first; core refused it and the plugin under test was recorded as SKIPPED without ever running. Each pass now activates whatever core accepts and retries the rest, and only a pass that activates nothing marks the plugin SKIPPED. That still ends a broken or circular dependency set, bounded by the number of dependencies. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01211BLKtDajir1Xg7etxT7R
A dependency that fatals after output has started still answers with HTTP 200, so it passed the baseline requests. The log was then cleared unread, and when the same fatal was logged again with the plugin under test active, that plugin was recorded as FAIL. The baseline log is now checked before it is cleared, and a fatal there marks the plugin SKIPPED against its dependencies, the same as a failed baseline request. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01211BLKtDajir1Xg7etxT7R
A relative --dir broke once the script changed into it, because the server document root, the debug log and the cleanup all resolved the path again from inside it. The directory is now made absolute as soon as it is entered. The database container was removed without its anonymous data volume, so every local run left one behind. It is now removed with --volumes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018vwYjTH76XPsP8BDwweFbg
When the port was already taken, the new PHP server exited with "Address in use" but the readiness request was answered by whatever owned the port, so every check that followed ran against an unrelated site and could pass. Readiness now requires the new server to still be running and to serve a file with a random token written into this install only. A port that is taken stops the run with an error naming the cause. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018vwYjTH76XPsP8BDwweFbg
A file with a syntax error that is included after output has started ends the request with a 200 and logs "PHP Parse error", which neither the response checks nor the log checks looked for, so the plugin passed. Both log checks now match parse errors as well as fatals through one helper, keeping their separate attribution: a dependency's error still skips the plugin, and the plugin's own still fails it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018vwYjTH76XPsP8BDwweFbg
A plugin that calls exit from its activation hook ends WP-CLI with a zero status before core records it as active. The checks then ran against a site without the plugin and passed it. Activation is now confirmed with wp plugin is-active for the plugin under test and for each dependency; a plugin that never became active is skipped, and a dependency that never did is retried in the next activation pass like one that was refused. A dependency that was already installed, such as the Akismet bundled with core, was treated as resolved and never activated or followed, so a plugin requiring it was skipped as having unmet requirements. Dependencies are now tracked by whether they have been resolved rather than whether they are installed: an installed one is not downloaded again but is still activated, baseline checked, and traversed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018vwYjTH76XPsP8BDwweFbg
Removing plugin directories and emptying active_plugins left behind everything else a plugin can write: drop-ins, must-use plugins, wp-config.php constants, files in the install root, tables and options. A must-use loader left requiring a deleted plugin broke every plugin tested after it, which a run against fixtures reproduced as a download SKIPPED and an activation FAIL for two healthy plugins. The fresh install is now recorded once, as an archive of wp-config.php and wp-content, a list of root entries and a copy of each table in the same database, and restored after every plugin. Files go back first so no stale drop-in loads while the tables are rebuilt. Copying tables in SQL avoids depending on a mysqldump that can talk to whichever server version the run uses. If a reset fails, the results so far are reported and the run stops, since anything after it would be measured against leftovers. This replaces the per-plugin install tracking, which was only there to drive the old cleanup. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018vwYjTH76XPsP8BDwweFbg
…unmet. The activation output was matched against "requires PHP version" and "requires WordPress version", which core does not print. Core says "does not meet minimum requirements", so a plugin declaring a newer PHP or WordPress than the run uses was recorded as FAIL instead of SKIPPED, blaming the plugin for core refusing to activate it as designed. Found by a fixture plugin that declares Requires PHP: 99.0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018vwYjTH76XPsP8BDwweFbg
…hecks. The mysql image initialises its data directory with a temporary server that answers on the socket and then restarts. A socket ping could succeed in that window, and the next step failed with a connection refused error. The ping now goes over TCP, which only the real server listens on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018vwYjTH76XPsP8BDwweFbg
…ust catch. Nothing showed that a broken plugin is actually caught: the pull request runs use real plugins that all pass, so a change that stopped detecting fatals would still go green. Each fixture breaks in one specific way - fatal on load, on activation, on the front end, on the login screen, after output, a late parse error, exit during activation, unmet PHP or WordPress requirements, a broken dependency, files and tables left behind for the next plugin - and expected.txt lists the result and reason each one must get. --fixtures installs a plugin from a local directory when one matches the slug, so every check still runs the same code, and --expect compares the results with the file and decides the exit code by whether they match. Weakening the activation check, the parse error check or the reset each makes the fixture run fail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018vwYjTH76XPsP8BDwweFbg
…ty run. The fixtures only prove anything if they run whenever the checks change, and against the version under test, since a core change can alter what a broken plugin looks like from the outside. A fixture-tests job now runs on every event the matrix does, through the same reusable workflow with a new fixtures input, and Slack notifications wait on it too. It needs no downloads beyond WordPress itself and takes about a minute. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018vwYjTH76XPsP8BDwweFbg
The retry job only listed Slack in `needs`, and `needs.*.result` only covers listed jobs. Slack succeeds even when it is reporting a failure, so a failed matrix or fixture run never dispatched its retry. Listing the test jobs alongside Slack matches coding-standards.yml and e2e-tests.yml. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018vwYjTH76XPsP8BDwweFbg
The INT and TERM traps ran cleanup and then let Bash carry on, so an interrupt while waiting for the database left the loop polling a container that had just been removed, and one during the checks left them running against a stopped server and deleted working files. Interrupts now exit with 130 or 143 and cleanup runs once, from the EXIT trap. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018vwYjTH76XPsP8BDwweFbg
…to PHP. The helper always ran `php <wp>`, which works for the phar but prints a shell launcher such as a Composer install's `bin/wp` as text and exits zero. Setup then appeared to succeed without WordPress being installed. A `wp` whose shebang does not name PHP is now run directly, with the memory limit passed through the `WP_CLI_PHP_ARGS` the launcher reads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018vwYjTH76XPsP8BDwweFbg
The `Requires Plugins` parser accepted any run of letters, digits and hyphens, so an entry such as `foo--bar` or `-foo`, which core ignores, could make a plugin skip for a dependency that is not available. It now uses the pattern from WP_Plugin_Dependencies::sanitize_dependency_slugs(). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018vwYjTH76XPsP8BDwweFbg
Core words its refusal "do not meet minimum requirements" when both versions are too old, and the check only matched the singular "does not meet" it uses for one. Such a plugin was reported as a failure and failed its shard, though nothing fataled. The fixture covers the combined case. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018vwYjTH76XPsP8BDwweFbg
The plugin under test is activated through WP-CLI, which loads its active dependencies first. A dependency that is healthy over HTTP but fatals under WP-CLI, which the final check already treats as a note rather than a failure, stopped that activation and the plugin under test was reported as failing. WP-CLI is now checked to load with the dependencies active before activating the plugin under test, and the plugin is skipped when it cannot. The fixture pairs a plugin with the existing WP-CLI only fatal fixture as its dependency. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018vwYjTH76XPsP8BDwweFbg
Co-authored-by: Jonathan Desrosiers <359867+desrosj@users.noreply.github.com>
Co-authored-by: Jonathan Desrosiers <359867+desrosj@users.noreply.github.com>
Co-authored-by: Jonathan Desrosiers <359867+desrosj@users.noreply.github.com>
These are the versions core's own tests report on, and the top 100 plugins pass the same on PHP 8.5 as on 8.3, so the newer version adds coverage without adding noise. The fixture job and the script's local defaults move too, so every run uses the same setup. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018vwYjTH76XPsP8BDwweFbg
…kflow. The checks build nothing and have no platform-specific behavior, so there is nothing to gain from running them on another OS. Hardcoding the runner removes an input that no caller needs to set. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018vwYjTH76XPsP8BDwweFbg
…low tests. The old note said the workflow only tests versions officially available on WordPress.org, which reads as though nightly is out of scope. Nightly is tested, and it comes from trunk. What the workflow never tests is the checkout it runs in. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018vwYjTH76XPsP8BDwweFbg
…ironment. Variables in the environment block read as settings a caller might change, but 250 is the API's own cap and nothing should override it. It now sits next to the paging loop that uses it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018vwYjTH76XPsP8BDwweFbg
adamsilverstein
force-pushed
the
add/plugin-compatibility-full-directory
branch
from
November 3, 2026 18:06
777c924 to
684a4c0
Compare
…ling newline. `read` returns non zero on a final line with no newline, even though it fills in the variables, so the loop dropped that line. A mismatch on the last fixture then went unreported and the run exited 0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018vwYjTH76XPsP8BDwweFbg
…e log. A plugin can start a non-blocking request, such as a loopback that runs a background task. The checks never waited on it, so a fatal in it was logged after the log had been checked and the plugin passed. Worse, it could land after the reset and be blamed on the next plugin. The built-in server logs when it accepts and closes each connection, so the checks now wait, for up to 30 seconds, until every request has closed before reading the log and before resetting the site. A request still running after that is not treated as a failure on its own. The new fixture fatals in a background request three seconds after the page returns; it passed before this change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018vwYjTH76XPsP8BDwweFbg
… active. The activation check runs WP-CLI with regular plugins skipped, but must-use plugins and drop-ins still load. A plugin whose activation writes a must-use plugin that fatals made that check fail, and was then skipped as though its activation had stopped early, while every request to the site was failing. When the plugin does not show as active, the front page and the log are now checked first. A broken site fails the plugin; a healthy one still skips it. The new fixture breaks the site this way; it was skipped before this change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018vwYjTH76XPsP8BDwweFbg
The directory paging lived inline in the workflow, so a run on a developer's machine had no way to build the same list. `list-plugins.sh` now does the paging for both, and adds `--all` and `--updated-since` for lists larger than the workflow's 1,000 plugin cap, which only makes sense to run locally. The matrix job gains a sparse checkout to reach the script. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018vwYjTH76XPsP8BDwweFbg
…nattended. Testing more than the workflow's 1,000 plugins only makes sense on a developer's machine, where a run takes most of a day. Four options make that practical: - `--plugins-file` reads the list from a file, since tens of thousands of slugs do not fit on a command line. Docker runs now always pass the list this way. - `--results` appends each row to a file as it is recorded and skips plugins already in it, so rerunning an interrupted run carries on where it stopped. - `--cache-dir` keeps WP-CLI's download cache outside the container, so a second run against another WordPress version reuses the downloads. - `--timeout`, 300 seconds by default, bounds the download, activation and WP-CLI steps. A plugin that hangs would otherwise stall every plugin after it. An activation that times out fails, as a request running into the same hang would. When a results file or cache is mounted, the container runs as the host user so that it can write to them on Linux. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018vwYjTH76XPsP8BDwweFbg
Across the whole directory, thousands of plugins fail on a current PHP against any version of WordPress, so a single run's failures are mostly noise. The signal before a release is the plugins that pass on the current release and fail on the next. `compare-results.sh` takes the results files of two runs and reports those regressions, plus plugins the candidate fixed, flagging any whose version changed between the runs. It exits 1 when something regressed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018vwYjTH76XPsP8BDwweFbg
Bash holds a signal until the foreground command finishes, so a TERM sent while the checks container ran waited for every remaining plugin to be tested. That matters once several copies run side by side and have to be stopped together. The container is now named, run in the background and waited on, and the EXIT trap removes it. compare-results.sh also refuses an empty results file, which would otherwise be read as the wrong file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018vwYjTH76XPsP8BDwweFbg
… locally. Running every plugin, or every plugin updated recently, ahead of a release takes far more runner time than a shared CI pool should give to one job, but is practical overnight on one machine. test-all-plugins.sh builds the list once, deals it out to several copies of test-plugins.sh running side by side, tests a baseline version and then the candidate, and ends with the compare-results.sh report of plugins that regressed. Results, logs and the download cache live in one output directory, so rerunning the same command after an interruption carries on, and a copy that stopped early is started again while it keeps making progress. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018vwYjTH76XPsP8BDwweFbg
The scripts now cover more than the workflow uses, and the baseline comparison is the part that makes a whole-directory run readable, so it gets explained in one place along with rough run times. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018vwYjTH76XPsP8BDwweFbg
Paging through every plugin by popularity silently drops about a quarter of them. Tens of thousands share the same rounded install count, their order shifts between pages, and the full run returned 56,982 different plugins. Sorting by update date pages cleanly and returns 77,894, which also takes in old plugins the popularity sort leaves out. Lists made with --count still use popularity, which stays stable through the first few thousand. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018vwYjTH76XPsP8BDwweFbg
adamsilverstein
force-pushed
the
add/plugin-compatibility-full-directory
branch
from
November 3, 2026 19:56
684a4c0 to
60c8772
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Claude Code wrote this one up, working from my notes on #13198:
Follow-up to #13198, which needs to land first. This PR includes its commits; the changes here start at d74dd03. Diff of just this PR: adamsilverstein/wordpress-develop@add/plugin-compatibility-workflow...add/plugin-compatibility-full-directory
Trac ticket: https://core.trac.wordpress.org/ticket/65920
#13198 tests the most popular plugins in CI. This makes it practical to test far more - every plugin updated in the last two years, or the whole directory - on a developer's machine or a spare server ahead of a major or security release, without tying up the shared CI runners for days.
Comparing against a baseline
This is the main feature. Across the whole directory, thousands of plugins fail on a current PHP whatever version of WordPress they run on, so the failures from a single run are mostly noise. The useful signal is the plugins that pass on the current release and fail on the next one.
compare-results.shtakes the results of a baseline run and a candidate run and reports:A plugin whose version changed between the two runs is flagged, since the plugin changed too.
Running a large list
This builds the list once, deals it out to 8 copies of
test-plugins.shrunning side by side, tests againstlatestand then the candidate, and writesreport.md. Rerunning the same command after an interruption carries on where it stopped.Rough run times at the ~7 seconds per plugin measured on the most popular plugins, with 8 jobs and both versions:
Changes
list-plugins.sh, which builds the plugin list for both the workflow and local runs. Adds--alland--updated-sincealongside--count. The workflow's 1,000 plugin cap is unchanged.--plugins-file,--results(append results and skip plugins already tested, so runs resume),--cache-dir(share downloads between runs) and--timeout(300 seconds by default, so a plugin that hangs can't stall a run) totest-plugins.sh.compare-results.shfor the baseline comparison.test-all-plugins.sh, which runs the whole thing in parallel and resumably.How has this been tested
test-all-plugins.sh --count=6 --jobs=2: tested against latest and nightly, wrote both results files and a report showing no regressions, in 1m12s.--results,--cache-dirand--plugins-fileon their own: a second run skipped the 4 plugins already recorded and reused cached downloads.--timeout=5: FAIL, "Activation did not finish within 5 seconds".compare-results.shon hand-written results files covering regressed, fixed, fail on both, skipped, missing from one file, and a version change.list-plugins.sh --count=260pages correctly.--updated-since=2026-11-01returned 1,477 plugins and stopped paging at the cutoff.shellcheckandactionlintare clean.Not tested yet: a run anywhere near the full directory. The timings above are estimates, not measurements.
AI Use
Claude Code wrote the code and this description with me steering. I will review and test.