You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Claude Code wrote this description from where the PR stands now:
Adds a workflow that installs popular plugins from the WordPress.org directory one at a time against a version of WordPress and checks that nothing fatals. The goal is to catch the WP Rocket class of breakage - a popular plugin fataling on every request against a new core version - while there is still time to fix core or reach the plugin author.
plugin-compatibility.yml asks the WordPress.org API for the most popular plugins at run time, so there is no list to go stale, and splits them into shards of about 25 (at most 10 shards, at most 1,000 plugins).
Each shard runs tools/plugin-compatibility/test-plugins.sh. The checks live in that script rather than in the YAML, so a red shard can be reproduced from a terminal with the exact same code.
For each plugin the script:
installs it, plus anything named in Requires Plugins, and checks the site with only the dependencies active first, so a broken dependency is not blamed on the plugin under test
activates it and confirms with wp plugin is-active that it really is active
requests / and /wp-login.php, and checks debug.log for fatal and parse errors
boots WordPress through WP-CLI last. A fatal there is only a note, because WP-CLI's function scope breaks some plugins in ways no visitor can hit
restores the fresh install - files, wp-config.php and database tables - so nothing one plugin leaves behind affects the next
A plugin that can't be downloaded, or that core declines to activate because its PHP, WordPress or dependency requirements aren't met, is SKIPPED and doesn't fail the run. Only a fatal does.
Every run writes a results table to the workflow summary.
A separate job runs the fixture plugins in tools/plugin-compatibility/fixtures - plugins built to break in one specific way each - and fails unless every one gets the result listed in expected.txt. That keeps a change to the checks from quietly letting broken plugins pass.
Runs weekly against nightly with the top 100, and on workflow_dispatch, where a release lead can pick a beta or RC, a plugin count, or a list of slugs. pull_request and push runs test 10 plugins against latest, since they only exist to check the workflow itself. This is signal only, not a check on every commit.
Running it yourself
With Docker running, nothing else is needed. A database and a PHP with WP-CLI come up in throwaway containers and are removed afterwards:
WordPress nightly / PHP 8.3: 2 passed, 0 failed, 0 skipped.
PLUGIN VERSION RESULT ALSO ACTIVE DETAILS
woocommerce 11.0.1 PASS
classic-editor 1.7.0 PASS
--help covers the rest, including --no-docker for an existing PHP, WP-CLI and database. From Actions, fill in plugin-slugs on a dispatched run. Dispatched runs work from a fork too:
The fixture job passes on CI and locally, covering fatals on load, on activation, on the front end, on login and after output has started, parse errors, an activation that exits early, unmet requirements, broken dependencies, a dependency that only fatals under WP-CLI, activation that breaks the whole site, a fatal in a background request, and a plugin that leaves files, tables and options behind. Each fixture added for a review fix was checked to fail against the script from before that fix.
Interrupting a run, a port already in use, a relative --dir, and a WP-CLI installed as a shell launcher were each checked locally against the old and new script.
shellcheck, actionlint and zizmor are clean.
The top 100 plugins locally against latest on PHP 8.3 and on PHP 8.5, both with MySQL 9.7: all 100 passed on both.
Not tested: multisite, PHP versions other than 8.3 and 8.5, MariaDB, and the Slack and retry jobs, which follow the other core workflows.
Open questions
Should the PHP version come from .version-support-*.json instead of being pinned to 8.5?
Premium plugins like WP Rocket itself aren't on WordPress.org, so the incident that prompted this wouldn't have been caught. Is a vendor-supplied zip input worth considering later?
instagram-feed fataled on nightly from a cron callback during early testing. WP-Cron is disabled in these runs to avoid a loopback deadlock, so running due cron events through WP-CLI would be a useful follow-up.
AI Use
Code and description both came out of rounds of back and forth with 🤖 Claude Code. I will review and test.
Core CI covers core itself, but nothing checks that a new version of
WordPress can still boot with popular plugins active. When a plugin's
assumptions about core stop holding the result is a fatal error on every
request, which is a white screen for real sites and is only discovered
after release.
Add a workflow that fetches the most popular plugins from the
WordPress.org API at run time, then installs and activates each one on
its own against the version of WordPress under test. A fatal is caught
whether it happens on activation, while WP-CLI loads WordPress, on a
front end or login request, or in the debug log, so a white screen with
error display turned off is still detected.
Failures are reported per plugin in the workflow summary and one broken
plugin never stops the rest of the shard from being tested. Plugins that
cannot be downloaded are reported as skipped rather than failed so that a
network flake does not turn the run red.
The run is manual or weekly rather than part of every commit, since a
third party plugin breaking should be a signal to release leads, not a
red check on unrelated work.
The first run of this workflow reported four failures that were not
fatals. Core refuses to activate a plugin whose `Requires Plugins`
dependency is missing, which every WooCommerce extension hits when
plugins are tested one at a time, and WP-CLI exits non-zero when a plugin
redirects while loading. Both are correct behaviour, so record them as
skipped and reserve a failure for an actual fatal.
A front end request also reported the nonsense status "200000", because
the curl fallback appended to output curl had already written. Capture
the exit code separately so a stalled transfer is reported as what it is.
The stall itself came from WordPress spawning WP-Cron as a loopback
request that the single threaded built-in server could not answer while
still serving the request that spawned it. Disable WP-Cron and give the
server workers so plugin loopback requests cannot deadlock it.
Ignore the zizmor unpinned image finding on the database service, which
cannot be pinned to a digest while the version is an input.
The WordPress.org API caps `per_page` at 250 and quietly returns 250 for
anything larger, so asking for more than that silently tested fewer
plugins than requested. Page through the API instead and trim to the
requested count, de-duplicating across pages because popularity ordering
can shift between two requests. Reject a count above 1000 with a clear
message rather than truncating without saying so.
Size the shards to the plugin count rather than always splitting into
five, so a run of 10 does not spin up five near empty jobs and a run of
250 is not squeezed into the same five.
Point pull request and push runs at the latest stable release with a
small count. Those runs exist to check that this workflow still works,
and a genuine ecosystem fatal against nightly should not sit as a red
check on every later change to these files.
Does any of this overlap with what Plugin Check or Tide already do? Nothing there appears to run plugins against unreleased core, but confirmation from someone closer to that infrastructure would help.
While Tide is still running, its been generally unsupported for years and I've long considered trying to get it shut down as I'm unsure how many entities are even making use of its available audit data set (with the exception of the PHP Compat Checker plugin). If there was interest in having this functionality more formally supported within Tide and then in consuming the resulting data, then that seems great to me but we'd likely want to get someone / a team / a sponsor to help ensure Tide continues to be supported after this implementation completes.
Testing plugins one at a time means every WooCommerce extension is skipped, and those are a large slice of the popular list. Installing WooCommerce alongside them would cover more real sites but breaks the isolation that keeps one broken plugin from masking another. Worth a follow up?
This seems worth a follow-up to me given they make the popular list. So in general, if a plugin has Requires Plugins headers then pre-install the required plugins first to allow the original plugin to be tested.
Claude ran the workflow at full scale and pulled this from the logs:
Proof the check does its job - a full 200 plugin run on a fork caught a real fatal. eps-301-redirects 2.85 installs and activates cleanly, then fatals the moment WP-CLI loads WordPress with the plugin active:
Worth noting how the triage played out: the failure reproduces identically on latest and nightly, so it is a pre-existing plugin bug rather than a core regression. WP-CLI loads WordPress inside a function scope, so the plugin's file-level $EPS_Redirects_Plugin never becomes a real global; when init fires on a plain permalink install, plugin_resources() calls add_admin_message() on that null. That points at a useful follow-up for this workflow: automatically re-check any failure against the latest stable release, so nightly-only failures (likely core regressions) stand out from plugins that were already broken.
Testing plugins one at a time means every WooCommerce extension is skipped, and those are a large slice of the popular list. Installing WooCommerce alongside them would cover more real sites but breaks the isolation that keeps one broken plugin from masking another. Worth a follow up?
This seems worth a follow-up to me given they make the popular list. So in general, if a plugin has Requires Plugins headers then pre-install the required plugins first to allow the original plugin to be tested.
…sting.
Testing each plugin on its own meant core refused to activate anything with a
`Requires Plugins` header, so every WooCommerce extension was recorded as
skipped without ever being loaded. Those extensions are a large slice of the
most popular plugins, which left a gap in exactly the part of the ecosystem the
workflow exists to watch.
Read the `Requires Plugins` header with WordPress' own parser, install and
activate what it names, and only then activate the plugin under test. Chains are
followed a level at a time so a dependency declaring its own is installed too,
and anything already present is left alone, which terminates a circular
declaration.
Dependencies mean the plugin under test is no longer alone on the site, so the
front page and login screen are checked with only the dependencies active first.
A site already broken at that point is the dependency's doing, and the plugin is
skipped rather than blamed for it. The debug log is cleared at the same point for
the same reason. The results table gains a column naming what else was active so
a failure can be read in context.
Installing and activating a dependency the size of WooCommerce is not quick, so
the job timeout goes from 30 to 45 minutes.
Claude took a swing at this one, here is where it landed:
Done in 67005f5. A plugin that declares Requires Plugins now gets those dependencies installed and activated before it is activated itself, so the four WooCommerce extensions that were being recorded as SKIPPED are actually tested.
The header is read with get_plugin_data() rather than a grep, so the same rules apply here as when core decides whether a plugin's requirements are met. Chains are followed a level at a time in case a dependency declares one of its own, and anything already installed is left alone, which is what terminates a circular declaration.
On the isolation worry from the original question: after the dependencies are activated and before the plugin under test is, the front page and login screen are requested on their own. A site already broken at that point is the dependency's doing, so the plugin under test is skipped rather than blamed for it, and the debug log is cleared at the same point for the same reason. The results table gains an "Also active" column so a failure can be read in context.
Tested in a container on PHP 8.3 with MySQL 8.4 and WP-CLI 2.12:
woocommerce-payments, google-listings-and-ads, woocommerce-paypal-payments and woocommerce-gateway-stripe all install woocommerce and pass. Running the previous version of the script against the same setup still reports them as SKIPPED, so that is the change rather than a difference in the environment.
Fixture plugins for the cases that are hard to find in the directory: a two level chain activates deepest first and passes, a dependency that is not on WordPress.org is skipped naming it, a dependency that fatals on load is skipped against the dependency, a dependency that fatals on the front end is caught by the baseline check, two plugins requiring each other terminate instead of looping, and a plugin that fatals on its own with a healthy dependency active still fails.
The four deliberate fatal fixtures still fail for the right reason, and wp-content/plugins and active_plugins were clean after every run.
One thing that turned up while re-testing: instagram-feed 6.12.0 fatals on wp_loaded against 7.1 on a fresh database, same wp_get_image_editor( WP_Error ) trace as before but reached directly rather than through cron. The previous version of the script reports it identically, so it looks like a genuine plugin bug rather than a side effect of this change.
The job timeout went from 30 to 45 minutes, since activating something the size of WooCommerce once per extension is not free. Does that seem like the right trade, or would you rather cap how many dependencies get pulled in?
Trying this workflow out meant taking whatever the popularity query returned,
so anyone wanting to check one plugin against a release candidate, or to
reproduce a failure from an earlier run, had to sit through a full run and read
past everything else in the shard. Reviewers could not run it at all: the
repository guard skips both jobs everywhere but this repository, and dispatching
it here needs write access.
Add an optional `plugin-slugs` input that replaces the directory query with a
list written by hand. Commas and whitespace both separate slugs so a list can be
pasted in however it was written down, and a value outside the character set the
directory uses stops the run immediately rather than being reported as an
impossible download several minutes later. The list goes through the same
sharding as a fetched one, so naming forty plugins still splits across two jobs.
Let a dispatched run through the repository guard wherever it is triggered,
which is what makes a fork useful for trying this out.
Two runs dispatched against the same version of WordPress used to land in the
same concurrency group and cancel each other, which is wrong once the plugins
being tested can differ between them. Dispatched runs are now grouped by run id.
`slack-notifications.yml` no longer accepts `SLACK_GHA_TIMEOUT_WEBHOOK`, and
every other caller stopped passing it, so actionlint failed on this workflow
once trunk was merged in.
…a terminal.
Move the per-plugin checks out of the reusable workflow and into
tools/plugin-compatibility/test-plugins.sh, so that reproducing a result
from a run no longer means dispatching a workflow and waiting for it.
The script provisions a database and a PHP with WP-CLI in throwaway
containers by default, so nothing but Docker is needed locally. The
workflow passes --no-docker and runs the same code against the runner's
own PHP and database service, which keeps one copy of the logic that
decides whether a plugin passed.
WP-CLI requires wp-settings.php from inside a method, so a plugin that
assigns a variable at file scope and reads it back with global later
finds nothing there. eps-301-redirects does exactly that and fatals under
WP-CLI on a fresh install, while the front page, the login screen and the
debug log are all clean. Failing it reported breakage that no visitor
would ever see.
Move the WP-CLI boot check after the HTTP and debug log checks and stop
letting it fail a plugin on its own. A plugin that is healthy over HTTP
now passes with the WP-CLI fatal recorded as a note. Anything that
genuinely fatals on load still fails, because it fails the HTTP checks.
The reorder also matters because the WP-CLI fatal is written to the debug
log, which the log check would otherwise pick up as a failure.
Testing the local runner turned up something that changes the story on eps-301-redirects, which I had held up earlier as proof this catches real fatals. Installing it by hand through wp-admin, nothing broke, so I had Claude dig into why the workflow disagreed.
Claude chased the discrepancy down, here is what came back:
The site is healthy with the plugin active. Front page 200, wp-login.php 200, /wp-admin/ 302, and no debug.log written at all. The only check that failed was the WP-CLI boot step.
WP-CLI requires wp-settings.php from inside WP_CLI\Runner->load_wordpress(), so a plugin's file-scope $var = ... becomes a local of that method rather than a global. A probe must-use plugin reporting from init shows it plainly:
HTTP: global is visible
WP-CLI: global is MISSING
eps-301-redirects assigns $EPS_Redirects_Plugin = new EPS_Redirects_Plugin(); at plugin.php:904 and reads it back with global $EPS_Redirects_Plugin; inside plugin_resources() on init. Under WP-CLI that is null, so line 390 calls ->add_admin_message() on nothing.
Line 390 only runs when permalink_structure is empty, which is what a fresh wp core install leaves behind. Setting /%postname%/ makes wp eval print loaded-ok; setting it back to plain brings the fatal back.
So it is real plugin fragility, but not something a visitor can reach, and not a core regression. A workflow whose job is to catch what breaks real sites should not fail a plugin for it.
The WP-CLI boot check now runs last and records a fatal as a note rather than a failure. A plugin that is healthy over HTTP passes, with the reason in the Details column. Anything that genuinely fatals on load still fails, because it fails the HTTP checks first. The order matters for a second reason: the WP-CLI fatal is written to the debug log, which the log check would otherwise pick up.
Both directions were verified. eps-301-redirects on nightly now passes with the note and exits 0. A fixture must-use plugin that fatals on template_redirect and not under WP-CLI still fails with "The request to / returned HTTP 500" and exits 1.
Sorry for the noise on the earlier claim - the check works, but that particular plugin was the wrong example for it.
Core words its refusal "do not meet minimum requirements" when both versions are too old, and the check only matched the singular "does not meet" it uses for one. Such a plugin was reported as a failure and failed its shard, though nothing fataled. The fixture covers the combined case.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018vwYjTH76XPsP8BDwweFbg
The plugin under test is activated through WP-CLI, which loads its active dependencies first. A dependency that is healthy over HTTP but fatals under WP-CLI, which the final check already treats as a note rather than a failure, stopped that activation and the plugin under test was reported as failing. WP-CLI is now checked to load with the dependencies active before activating the plugin under test, and the plugin is skipped when it cannot. The fixture pairs a plugin with the existing WP-CLI only fatal fixture as its dependency.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018vwYjTH76XPsP8BDwweFbg
The reason will be displayed to describe this comment to others. Learn more.
100 is fine for initial testing, but at some point it would be useful to at least have the capability of running against all plugins in the directory. we could do this before a major release, for example.
The reason will be displayed to describe this comment to others. Learn more.
Moved this into a follow-up so this PR stays focused: #15382
Claude summed up the follow-up:
The big piece is comparing against a baseline. Every plugin is tested against the current release and against the candidate, and the report lists only the ones that pass before and fail after. That is what makes a run across the whole directory, or everything updated in the last two years, readable. It is meant to run on a dev machine or spare server rather than in CI.
Retry: the test jobs are now in failed-workflow.needs, the same pattern as coding-standards.yml and local-docker-environment.yml.
INT/TERM now exit, so the EXIT trap cleans up once and nothing keeps polling the removed container.
A shell launcher now runs directly with WP_CLI_PHP_ARGS; phar and PHP-shebang entry points still go through php.
The slug pattern now matches WP_Plugin_Dependencies::sanitize_dependency_slugs().
Core's plural "do not meet minimum requirements" message is now skipped.
A dependency that WP-CLI cannot load now skips the target instead of failing it.
On this head, all 17 fixture results matched, and actionlint and zizmor pass.
Optional: fixture-requires-newer-php and fixture-requires-newer-wordpress both reach the same "does not meet minimum requirements" branch in the script, so one of them could go.
Landing is still held for the scope and cadence question from Trac comment 8. @aaronjorbin, does the weekly run without blueprints work for you as a starting point?
The reason will be displayed to describe this comment to others. Learn more.
Is there a specific reason why PHP 8.3 is being used here and not 8.5? Is it to target versions that plugins all likely support?
A few follow up questions about the API:
If you query 10 plugins using PHP 8.0 and 10 plugins using PHP 8.5, will the returned list vary because certain plugins set their Required PHP to say 8.2?
Would it be beneficial to run against multiple versions of PHP? It's possible that a plugin has incompatibilities with a given PHP version that present in a unique way.
The reason will be displayed to describe this comment to others. Learn more.
Is there a specific reason why PHP 8.3 is being used here and not 8.5? Is it to target versions that plugins all likely support?
A few follow up questions about the API:
If you query 10 plugins using PHP 8.0 and 10 plugins using PHP 8.5, will the returned list vary because certain plugins set their Required PHP to say 8.2?
I'm not sure about that. I'll do some research on this to try to figure it out.
Would it be beneficial to run against multiple versions of PHP? It's possible that a plugin has incompatibilities with a given PHP version that present in a unique way.
I guess there is some value - even though ideally i'd like to test agains the most recent supported version of PHP possible, we need to make sure the API is returning those plugins and they actually get testing.
My initial goal with this workflow was catching incompatibilities between a new version of core and the most popular plugins. Most popular means most installs, and thus greatest impact if there is a problem. I'm not sure what the threshold should, maybe the top 1000 plugins, or plugins with over 10k installs? While I'd like to have the ability to test all plugins, I don't think its reasonable to do that frequently.
I will do some more research here:
How many plugins have >10k installs? 100k? 1k? i'd love to see the distribution in a table or graph.
What are the php min versions supported in the top 100 or 1000 plugins?
The reason will be displayed to describe this comment to others. Learn more.
I dug into the questions above, short version: the API doesn't filter by PHP version, and the top plugins pass on PHP 8.5 just like 8.3.
Claude pulled the full directory and ran the numbers, here is what came back:
Does the API return a different list depending on PHP version? No. query_plugins with browse=popular returned the identical list with php_version unset, 5.6, 7.4, 8.0 and 8.5 (with wp_version set too). The API filters on neither. A plugin whose Requires PHP isn't met still gets installed, and core's own requirements check marks it SKIPPED.
Install distribution, across all ~73,800 plugins in the directory (install counts are the directory's rounded buckets, so shares are approximate):
Active installs
Plugins
Running total
Share of all installs
1M+
67
67
100k+
394
461
80.5%
10k+
~1,700
~2,160
94.6%
1k+
~4,780
~6,940
98.9%
100+
~10,200
~17,200
under 100
~56,600
~73,800
The top 100 cover about 58% of all installs, and the top 1,000 about 90%. Everything with 10k+ installs is roughly 2,160 plugins and 95% of installs.
Minimum PHP versions:
Requires PHP
Top 100
Top 1,000
5.x
17
204
7.0 - 7.3
16
163
7.4
54
418
8.0 - 8.3
6
53
not declared
7
161
None of the top 1,000 require anything above PHP 8.3, so on 8.3 or 8.5 nothing in that range is skipped for its PHP requirement.
PHP 8.3 vs 8.5: the top 100 were run against latest with MySQL 9.7 on both versions. All 100 passed on both, with no differences between the runs. That covers the top 100 only.
One thing to keep in mind with multiple PHP versions: a plugin that breaks on a new PHP version fails against every version of WordPress, so in a single run it looks the same as a core regression. The baseline comparison in #15382 (test against the current release and the candidate, report only pass-then-fail) is what separates the two.
Given that, moving the default to 8.5 seems fine to me. What do you think about raising the default count to cover everything 10k installs or over (~2k plugins) @desrosj?
The reason will be displayed to describe this comment to others. Learn more.
Given that, moving the default to 8.5 seems fine to me. What do you think about raising the default count to cover everything 10k installs or over (~2k plugins) @desrosj?
I think that's probably fine. @lancewillett I'm wondering if we should check with Systems/Meta about this, though. What do you think?
These are the versions core's own tests report on, and the top 100 plugins pass the same on PHP 8.5 as on 8.3, so the newer version adds coverage without adding noise. The fixture job and the script's local defaults move too, so every run uses the same setup.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018vwYjTH76XPsP8BDwweFbg
…kflow.
The checks build nothing and have no platform-specific behavior, so there is nothing to gain from running them on another OS. Hardcoding the runner removes an input that no caller needs to set.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018vwYjTH76XPsP8BDwweFbg
…low tests.
The old note said the workflow only tests versions officially available on WordPress.org, which reads as though nightly is out of scope. Nightly is tested, and it comes from trunk. What the workflow never tests is the checkout it runs in.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018vwYjTH76XPsP8BDwweFbg
…ironment.
Variables in the environment block read as settings a caller might change, but 250 is the API's own cap and nothing should override it. It now sits next to the paging loop that uses it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018vwYjTH76XPsP8BDwweFbg
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Critical request-tracking and activation-verification gaps can hide fatal plugin failures and contaminate subsequent results, while an expectation without a trailing newline can be silently ignored.
No open issues 🎉
3 previously missed issues
In code that hasn't changed since last review
Verify HTTP health before treating activation failures as inactive
tools/plugin-compatibility/test-plugins.sh:850
If an activation hook writes a broken must-use loader, activation can succeed but is_active can fatal during bootstrap because --skip-plugins still loads must-use plugins. This branch treats that failure as confirmed inactivity and reports SKIPPED without checking HTTP health or logged fatals, hiding a plugin that breaks every request. Distinguish bootstrap failure from confirmed inactivity and check HTTP health on verification failure, reserving this early-activation SKIPPED result for confirmed inactive plugins.
Wait for outstanding requests before checking logs or resetting
tools/plugin-compatibility/test-plugins.sh:870
A plugin can start a nonblocking loopback request that is still running when both checked pages return, so this log check can miss a delayed fatal and report PASS. The outstanding request can also write errors or database changes after the reset, contaminating the next plugin's results. Track outstanding requests and wait for completion with a bounded timeout before checking logs or resetting; handle unfinished requests explicitly, including at the dependency-baseline boundary.
Process the final expectation when the file lacks a trailing newline
tools/plugin-compatibility/test-plugins.sh:988
If a supplied --expect file has no trailing newline, read assigns the final expectation but returns nonzero, so this loop silently skips it. A mismatching final fixture is therefore ignored; a file containing just that expectation reports “Every result matched” and exits 0 even when the fixture failed. Process an unterminated final record as well so every expectation is checked.
…ling newline.
`read` returns non zero on a final line with no newline, even though it fills in the variables, so the loop dropped that line. A mismatch on the last fixture then went unreported and the run exited 0.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018vwYjTH76XPsP8BDwweFbg
…e log.
A plugin can start a non-blocking request, such as a loopback that runs a background task. The checks never waited on it, so a fatal in it was logged after the log had been checked and the plugin passed. Worse, it could land after the reset and be blamed on the next plugin.
The built-in server logs when it accepts and closes each connection, so the checks now wait, for up to 30 seconds, until every request has closed before reading the log and before resetting the site. A request still running after that is not treated as a failure on its own. The new fixture fatals in a background request three seconds after the page returns; it passed before this change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018vwYjTH76XPsP8BDwweFbg
… active.
The activation check runs WP-CLI with regular plugins skipped, but must-use plugins and drop-ins still load. A plugin whose activation writes a must-use plugin that fatals made that check fail, and was then skipped as though its activation had stopped early, while every request to the site was failing.
When the plugin does not show as active, the front page and the log are now checked first. A broken site fails the plugin; a healthy one still skips it. The new fixture breaks the site this way; it was skipped before this change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018vwYjTH76XPsP8BDwweFbg
Copilot's three previously missed issues are addressed in 5f7b4ad (last expectation without a trailing newline), 89e7895 (wait for a plugin's own requests) and a349207 (check the site before skipping an inactive plugin).
The reason will be displayed to describe this comment to others. Learn more.
Approving at a349207. Every finding from my earlier reviews is fixed, and so are Copilot's three. Landing still waits on aaronjorbin's scope answer and desrosj's re-review.
desrosj's seven threads: each matches its commit. PHP 8.5 and MySQL 9.7 are already in the phpunit-tests.yml matrices.
The three Copilot fixes: I ran a fixture subset through the Docker path (nightly, PHP 8.5), once on this head and once with each fix reverted. Each new fixture fails only when its fix is gone:
Run
breaks-site-on-activation
fatal-in-background-request
Other four
This head
FAIL, as expected
FAIL, as expected
match
wait_for_requests made a no-op
match
PASS (mismatch)
match
Inactive-plugin branch reverted to plain SKIPPED
SKIPPED (mismatch)
match
match
The trailing-newline fix: with check_expectations alone, an expect file whose only line is a wrong expectation with no final newline reported "Every result matched" before this change. Now it reports a mismatch and exits 1. No fixture covers it, because the bundled file ends with a newline, so this test was by hand.
CI on this head: the full fixture job matched every expectation (run).
The reason will be displayed to describe this comment to others. Learn more.
I did just have another thought. This is built using the reusable pattern, but that could cause some problems. Over time old branches would still be running these tests against plugins that have moved on from supporting that version of WP.
I don't think it would be a huge deal (the plugins would likely just not activate and be skipped), but the bash file is not being called from trunk and could eventually fail.
Instead, I think we should treat this workflow similar to how we treat test-coverage.yml, upgrade-testing.yml, etc. Can we add these two new workflow files to the workflowFiles list in Grunt.js so that they are deleted when grunt post-branching is run within the newly created branch after branching?
The list of files currently only contains workflow files, but they are specified by the full path. Perhaps we could also include tools/plugin-compatibility in the list to see if those also are cleaned.
The only thing I'm not sure on is whether we would want these new tests to run against the currently supported branch? trunk is probably good enouhg, but I'm not sure on that.
I did just have another thought. This is built using the reusable pattern, but that could cause some problems. Over time old branches would still be running these tests against plugins that have moved on from supporting that version of WP.
I don't think it would be a huge deal (the plugins would likely just not activate and be skipped), but the bash file is not being called from trunk and could eventually fail.
Instead, I think we should treat this workflow similar to how we treat test-coverage.yml, upgrade-testing.yml, etc. Can we add these two new workflow files to the workflowFiles list in Grunt.js so that they are deleted when grunt post-branching is run within the newly created branch after branching?
Yes, will do!
The list of files currently only contains workflow files, but they are specified by the full path. Perhaps we could also include tools/plugin-compatibility in the list to see if those also are cleaned.
👍🏼
The only thing I'm not sure on is whether we would want these new tests to run against the currently supported branch? trunk is probably good enouhg, but I'm not sure on that.
Lets start with trunk and only add other branches as needed.
The workflow and its script are only meant to run from trunk. Leaving them in a branch would pair trunk's reusable workflow with the branch's copy of the script, which drifts over time. Since the workflow tests downloaded builds rather than the checkout, running it from trunk covers any supported version.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018vwYjUy2DYoWVuMJ7UuKDD
The changes in this pull request can previewed and tested using a WordPress Playground instance.
WordPress Playground is an experimental project that creates a full WordPress instance entirely within the browser.
Some things to be aware of
All changes will be lost when closing a tab with a Playground instance.
All changes will be lost when refreshing the page.
A fresh instance is created each time the link below is clicked.
Every time this pull request is updated, a new ZIP file containing all changes is created. If changes are not reflected in the Playground instance,
it's possible that the most recent build failed, or has not completed. Check the list of workflow runs to be sure.
For more details about these limitations and more, check out the Limitations page in the WordPress Playground documentation.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
lancewillett
moved this from In review
to In progress
in WordPress Project Build Tooling
lancewillett
left a comment
desrosj
left a comment
Claude Code wrote this description from where the PR stands now:
Adds a workflow that installs popular plugins from the WordPress.org directory one at a time against a version of WordPress and checks that nothing fatals. The goal is to catch the WP Rocket class of breakage - a popular plugin fataling on every request against a new core version - while there is still time to fix core or reach the plugin author.
Trac ticket: https://core.trac.wordpress.org/ticket/65920
See adamsilverstein#64
How it works
plugin-compatibility.ymlasks the WordPress.org API for the most popular plugins at run time, so there is no list to go stale, and splits them into shards of about 25 (at most 10 shards, at most 1,000 plugins).tools/plugin-compatibility/test-plugins.sh. The checks live in that script rather than in the YAML, so a red shard can be reproduced from a terminal with the exact same code.Requires Plugins, and checks the site with only the dependencies active first, so a broken dependency is not blamed on the plugin under testwp plugin is-activethat it really is active/and/wp-login.php, and checksdebug.logfor fatal and parse errorswp-config.phpand database tables - so nothing one plugin leaves behind affects the nexttools/plugin-compatibility/fixtures- plugins built to break in one specific way each - and fails unless every one gets the result listed inexpected.txt. That keeps a change to the checks from quietly letting broken plugins pass.Runs weekly against nightly with the top 100, and on
workflow_dispatch, where a release lead can pick a beta or RC, a plugin count, or a list of slugs.pull_requestandpushruns test 10 plugins against latest, since they only exist to check the workflow itself. This is signal only, not a check on every commit.Running it yourself
With Docker running, nothing else is needed. A database and a PHP with WP-CLI come up in throwaway containers and are removed afterwards:
--helpcovers the rest, including--no-dockerfor an existing PHP, WP-CLI and database. From Actions, fill inplugin-slugson a dispatched run. Dispatched runs work from a fork too:How has this been tested
--dir, and a WP-CLI installed as a shell launcher were each checked locally against the old and new script.shellcheck,actionlintandzizmorare clean.Not tested: multisite, PHP versions other than 8.3 and 8.5, MariaDB, and the Slack and retry jobs, which follow the other core workflows.
Open questions
.version-support-*.jsoninstead of being pinned to 8.5?instagram-feedfataled on nightly from a cron callback during early testing. WP-Cron is disabled in these runs to avoid a loopback deadlock, so running due cron events through WP-CLI would be a useful follow-up.AI Use
Code and description both came out of rounds of back and forth with 🤖 Claude Code. I will review and test.