chore(rules): performance improvements and more FPR

[egibs]

Follow-up to #1640.

What

Performance

One rule optimized. sketchy_math_conversions: the five $complex_math_* operand runs change from unbounded [\w\d\s\+\-\*\/\^]+ to {1,64}. Hit set is byte-identical (22927 hits, zero gained or lost) and the rule drops out of the top ten entirely. Full-ruleset effect: 101.55 → 98.91 CPU-seconds median, −2.6%.

Nine were measured and left alone. Every $word_with_spaces variant was equal or worse than baseline 1.27s ({3,16} 1.29, {4,16} 1.30, {2,12} 1.36, dropping fullword 1.42): the pattern contains one literal byte, the space, so there is no usable atom for prefiltering. It is also an upper-bound test (#… < 8), so narrowing it would add detections rather than preserve them. For /\x00[a-z0-9]{3000}/, range-bounding is no cheaper and dropping the \x00 anchor costs 76s. math1, d4_paren_rem, d5_tiny_rem and f1_longmath were worse under every variant tried. The charAt {32,}/{50,} siblings cannot be bounded without making the closing quote unreachable for long base64, which is a semantic change rather than an optimization.

A provably-equivalent 5→1 pattern merge for $complex_math_* was built and measured — all five operators are already members of the character class, so any of ($complex_math*) is equivalent to a single operator alternation — and rejected because it measured no faster.

hash.sha256 and math.entropy gating produces no win. Both are cached per file per argument expression, so one rule costs the same as three. Above the 0.55s measurement floor the true costs are 0.38s for hash.sha256 and 0.24s for math.entropy. Gating a subset is provably worthless: one ungated plus one gated measured 0.92s, identical to fully ungated, while gating both measured 0.55s. The dominant caller, trivy_2026_03, uses nine hashes to identify Homebrew bottles of trivy, and the corpus contains a 240MB linux/clean/trivy, so any bound tight enough to filter anything risks excluding the artifacts a critical rule exists to detect. Entropy gating would save roughly 0.15s, below noise, at a cost of 19 lost detections.

Remaining suppression sites

14 sites applied with zero detections lost anywhere: obfuscate 13→19, http_hardcoded_ip 20→21, encrypt 29→74, pivot_root 6→8, dev_path 21→89, home_path 42→50, download 121→140, daemon 47→47, nmap 13→14, xor_decode_encode unchanged, and base64_shell_decode, cve_mention, high_pdb, rm_f_hardcoded_tmp_path each +1.

Five sites needed a different repair than a plain count guard, and three rules previously classified as needing repair (rootkit_high, unusual_redir, proc_d_exe) turned out already correct, along with office_crypt_archive and bash_persist_persistent.

Compile warnings

make yara-x-compile used a bare -w, which disables all warnings, so 15 were invisible. The target now uses --disable-warnings=text_as_hex and all 15 are fixed.

13 duplicate pattern values removed. Under any of them a duplicate is inert, so five of those deletions are byte-identical in effect. Under a threshold it silently weakens the rule: blocklist/ip.yara uses 2 of them and listed 109.74.154.91 twice, so that one IP satisfied the threshold alone; linux_monitors held "ps" as both $ps and $x_ps, filling one of three monitor slots, and 49 files cleared 3 of ($x*) with the duplicate against 22 without, with 24 of the 27 difference being clean software.

Three were plain typos whose intended pattern never existed: $pcm_alaw = "pcm_mulaw", $http_discord/$http_telegram holding capitalised values, and $eval_request_urllib beginning exec\(. In bitwise.yara, $left and $right held identical values, making (#left > 5 or #right > 5) exactly #left > 5; they are collapsed to one $shift.

Both potentially_slow_loop warnings in exclamations are silenced by bounding the loop with math.min(#exclaim, 256) and math.min(#not_bug, 16), holding the match set at exactly 41.

Why

Two defects recur and account for most of the rule edits.

A leading variable-width character class makes the engine report one match per possible prefix length, which inflates any count-based guard built on it. encrypt's $ref yields 14 matches for the single mention .tempkeyEncrypted, and high_pdb's yields 11 to 20 for one DependencyInjection.pdb. A plain count guard would therefore fire on a file whose only mention is the accepted spelling. All five affected sites use dedicated single-occurrence counters instead, rather than anchoring $ref — anchoring $ref can remove detections, so it does not belong in a count-based repair.

Duplicate pattern values are harmless under any of and harmful under a threshold, because one value fills two slots.

text_as_hex stays suppressed and the hex patterns are unchanged. Rules are embedded in the binary via //go:embed, so a plaintext malware signature would sit as a literal string inside the shipped executable and cause macOS Gatekeeper and XProtect to flag malcontent itself.

Notes

Measurement methodology

--profiling totals are not additive across rules. yara-x deduplicates identical pattern values and searches them once, but charges the full shared cost to every rule declaring the pattern: one rule declaring $word_with_spaces profiles at 3.179s, while three rules declaring the identical value each report 3.455757355s, the same figure to the digit, for one search. The three macho rules in the top ten overlap heavily for this reason.

Isolated pattern benchmarks also overstate in-ruleset gains by roughly 10x — the landed fix measured 10.35s → 1.75s standalone but −2.6% in the full ruleset. Wall-clock timing was unusable at this scale, with a 2.4 to 20.8s spread on an ~18s measurement, so the figures above come from interleaved CPU-time A/B runs.

Goldens

101 mechanical RuleURL line shifts, 100 namespace lines added, 0 removed, plus the pkg/action/testdata/scan_archive fixture. Additions by namespace: fs/path/dev 54, crypto/encrypt 27, net/download 10, fs/path/home 6, obfuscate 2, pivot_root 1. Each traces to a measured recovered detection. Regenerated with make refresh-sample-testdata; none hand-edited.

Verification

make yara-x-compile exit 0 with warnings now enabled and none remaining. make test exit 0, 11 packages. make integration exit 0, 172.8s full run, independently reproduced with a forced -count=1 run. The sample corpus is byte-identical at its pinned commit and no sample was executed.

Not confirmed: the -race integration variant was stopped partway at 56 PASS / 0 FAIL, so that specific configuration has not completed.

Review notes

dev_path moves 21→89 hits, including 46 clean-sample gains at medium. Each is a correct capability report, but it is the largest behavioral change in this diff. If it should be toned down, the lever is adding /dev/urandom, /dev/tty and /dev/std* to the ignore set, which is a new suppression rather than a count-guard fix.

exclamations keeps its nested positional loop rather than moving to a count guard. A count guard gains exactly the six files containing DYNAMIC LINKER BUG!!!, two of which are clean (ld-2.27.so, slirp4netns) — precisely the false positives the guard exists to prevent. $exclaim has a variable-width leading class, so one 21-byte copy of that string yields many overlapping matches, and #not_bug is never greater than 1 anywhere in the corpus.

dev_path keeps no left boundary on $path. Adding a \B boundary fixes a run-tests.php false positive but drops four references the rule reports today, including on malware (2024.GitHub.Clipper/main.exe). Tightening $path can remove detections, so it is left as a separate change rather than folded in here.