Skip to content

fix: keep the mutant --filter under the kernel's per-argument limit - #43

Open
yeapea wants to merge 1 commit into
pestphp:5.xfrom
yeapea:fix/filter-argument-exceeds-max-arg-strlen
Open

fix: keep the mutant --filter under the kernel's per-argument limit#43
yeapea wants to merge 1 commit into
pestphp:5.xfrom
yeapea:fix/filter-argument-exceeds-max-arg-strlen

Conversation

@yeapea

@yeapea yeapea commented Aug 21, 2026

Copy link
Copy Markdown

Closes pestphp/pest#1771.

The defect

A mutant's covering tests are emitted one regex fragment per test and joined into a single --filter= argv element:

'--filter="'.implode('|', $filters).'"',

That element is unbounded. Linux caps one argv element at MAX_ARG_STRLEN = PAGE_SIZE * 32 (include/uapi/linux/binfmts.h) — 131,072 bytes on x86_64, and it is not tunable: ulimit -s moves the total ARG_MAX, never the per-element cap. Past it the child dies with

posix_spawn() failed: Argument list too long

before running a single test, so every mutation in that class is lost. Measured at 172,779 bytes for one mutation in a class with 2,076 covering tests.

macOS has no per-argument cap at all, which is why this reproduces only on Linux — in practice on CI, on the run whose score people actually publish.

The fix

Pest\Mutate\Support\FilterArgument applies two encodings, in order.

1 — Factor the class prefix. Cls::(.*)a|Cls::(.*)b becomes Cls::(.*)(a|b). Lossless, and measured on synthetic sets shaped like the real ones:

naive built
four classes × 300 tests 52,370 34,438 −34.2%, selection identical
one class × 2,076 tests 150,448 32 collapsed, and reported

The inner parentheses are load-bearing. Cls::(.*)a|b parses as (Cls::(.*)a)|(b), so a bare b matches anywhere and the filter runs tests it was never given. There is a test for exactly that.

2 — If it still does not fit, collapse a class to its bare Cls:: prefix, heaviest contributor first, stopping the moment it fits.

That is a strict superset of the original selection, which is precisely why it is safe here: widening can only ever run more tests, so it can never turn a killed mutant into a survivor.

That argument has one precondition worth stating: it holds while the ordinary suite is green. A test already failing for unrelated reasons counts as a kill, so a widened filter can inherit an unrelated red — the same fabrication class as running mutation in parallel.

So a collapse is never silent. widened names each class and how many extra tests it now selects; MutationTest reports it once per process rather than once per mutation in the class. A filter that quietly widened would make the score certify more than the run measured.

And if even a fully collapsed filter does not fit, it throws. Dropping the filter would run the whole suite per mutant and read as a fast green, which is the one outcome worse than the crash.

Why a value object rather than a method

FilterArgument::for() returns argument plus widened, so the encoding stays free of any output concern and is unit-testable without spawning a process. The fwrite(STDERR, …) at the call site is the smallest honest channel for the notice — if you would rather it went through Facade::instance()->emitter(), say so and I will move it; it needs an event class and listener wiring, so I did not add that unasked.

BUDGET_BYTES is 98,304 rather than the kernel's full 131,072: the cap is per element and a suite only grows, so a filter sized to the exact limit is one new test away from failing again.

Proof

pest (whole suite) 328 passed, 649 assertions
new unit test 12 passed, 25 assertions
pint --test passed
phpstan analyse no errors
rector --dry-run done, no changes
pest --type-coverage 100% on both new files, total 100%

Three behaviours were red-proved by reverting the code that guards them:

reverted goes red
the inner parentheses in the factored form 4 arms, including the over-match one
the throw on a still-oversized filter it fails loudly rather than returning a filter that still does not fit
the collapse loop 4 arms — factoring alone does not get a 2,076-test class under the cap

Prior art

This has been running in production in two repositories for weeks — one as a cweagans/composer-patches patch, one as an anchored vendor rewrite — against real mutation runs. This PR is that algorithm, upstreamed, so neither has to re-derive it on every release.

A mutant's covering tests are emitted one regex fragment per test and joined
into a SINGLE `--filter=` argv element. That element is unbounded, and on a
class most of a suite reaches it runs past MAX_ARG_STRLEN — PAGE_SIZE * 32,
131072 bytes on x86_64, a kernel constant `ulimit` does not move. The child then
dies with `posix_spawn() failed: Argument list too long` before running a single
test, so every mutation in that class is lost. Measured at 172,779 bytes for one
mutation in a class with 2,076 covering tests.

macOS has no per-argument cap, so it reproduces only on Linux — in practice on
CI, on the run whose score people publish.

Support/FilterArgument applies two encodings in order:

1. Factor the class prefix: `Cls::(.*)a|Cls::(.*)b` becomes `Cls::(.*)(a|b)`.
   Lossless — both forms select exactly the same tests — and 34% shorter on a
   four-class set. The inner parentheses are load-bearing: without them the
   alternation binds to the whole pattern rather than the tail, and the filter
   selects tests it was never given.

2. If it still does not fit, collapse a class to its bare `Cls::` prefix,
   heaviest first, stopping the moment it fits. That is a strict superset of the
   original selection, so it can only run MORE tests and can never turn a killed
   mutant into a survivor.

A collapse is never silent: `widened` names each class and how many extra tests
it now selects, and MutationTest reports it once per process rather than once
per mutation. A filter that quietly widened would make the score certify more
than the run measured. If even a fully collapsed filter does not fit it throws,
because dropping the filter would run the whole suite per mutant and read as a
fast green.

Closes #1771 (pestphp/pest).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Mutation testing: --filter exceeds the kernel's per-argument limit (and PCRE's compile limit) on widely-covered classes

1 participant