Scan character classes with SIMD in the JIT on x86-64

[mattst88]

fast_forward_start_bits() tests the start bitmap one code unit at a time on
every architecture, so a pattern whose first character is a class scans far more
slowly than one starting with a literal, even though the JIT already has a
vectorized scan for the literal case.

This adds a hook for it and implements it for x86-64 with SSE2, both for a class
at the start of the match and for one at a relative offset, so that .{3}[QXZ]
is searched as well as [QXZ].

Commits

Add a helper to split a class bitmap into ranges. optimize_class_ranges()
already reduced a 256 bit bitmap to the code unit values at which membership
changes. That scan moves into extract_class_ranges(), placed before the
include of pcre2_jit_simd_inc.h so the vectorized scan can use it too, and
takes a limit on the number of entries rather than the fixed
MAX_CLASS_RANGE_SIZE. No functional change.

Emit the x86 vector compares through one helper.
fast_forward_char_pair_sse2_compare() assembled each instruction byte by byte,
patching the prefix in place for the AVX2 form, so the same four bytes were put
together seven times over in slightly different ways. The encoding moves into
emit_vector_op(), which takes an opcode and three register indices and picks
the two operand SSE2 form or the three operand AVX2 one from the register width.
It also emits a REX prefix, which the byte-by-byte code had no need for, so that
later users can reach the upper vector registers. The emitted bytes are
unchanged for every combination of compare type, register width, step and
register assignment.

Scan character classes with SIMD in the JIT on x86-64. The bitmap is reduced
to a list of ranges at compile time; the scan then tests each range with the
usual unsigned range idiom, where subtracting the low bound makes the range
start at zero and a saturating subtract of the span leaves zero exactly for the
bytes inside it, since anything below the low bound wraps to a value larger than
the span. That is PSUBB, PSUBUSB and PCMPEQB per range, OR'd together,
with PCMPEQB alone for a single character. Bitmaps needing more than four
ranges keep the existing scan.

Scan for a character class at a relative offset in the JIT. scan_prefix()
already walks the class bitmap at each position, but expands it into the chars
array and keeps at most MAX_DIFF_CHARS characters, so a larger class left
nothing behind but "any character". It now records the bitmap itself alongside
the array. A position keeps its bitmap only while one and the same class reaches
it, since a second alternative, a literal or an unbounded class all make it
describe less than the position accepts.

fast_forward_first_n_chars() then picks the sparsest such position, but only
where it would otherwise give up: a position holding one or two characters gets
a cheaper scan of its own, and a run of them gets the shift table, both of which
beat testing a handful of ranges.

Numbers

Measured over a 4MB subject on an i7-1370P, scanning for a class that does not
occur:

pattern	before	after
[QXZ]	1445 MB/s	25961 MB/s
[0-9]{6}	3337 MB/s	33440 MB/s
.{3}[QXZ]	450 MB/s	25798 MB/s
.{2}[0-9]{6}	647 MB/s	33424 MB/s

Every other pattern measured is unchanged, including a.{4}[QXZ] at 1486 MB/s,
which keeps its single character scan rather than being taken over by the class
scan.

Notes

Vectorizing only pays for a sparse class. The byte-at-a-time loop stops at the
first code unit in the class, so where the class is dense it stops almost
immediately, while a vector loop has already tested a whole block. Measured on
\b\w{12,}\b, whose class accepts 63 code units, the vector scan was 38.6%
slower. Classes accepting more than 48 code units are therefore left alone, and
\b\w{12,}\b and .{3}\w{12} are unchanged.

The bounds of each range live in a vector register of their own, so the scan
needs twelve of them, more than the Win64 ABI leaves as scratch registers. It is
limited to the other x86-64 ABIs, as fast_forward_char_pair_simd() already is,
with a compile time assertion tying the range limit to
SLJIT_NUMBER_OF_SCRATCH_VECTOR_REGISTERS.

SSE4.2's PCMPESTRI was measured as an alternative. It handles up to eight
ranges in one instruction and so is flat in the number of ranges, but it is
slower than this sequence for one or two ranges, which is what real classes
mostly are: 11.5 GB/s against 28.5 GB/s for a single range. It also has no
256-bit form, so it would foreclose widening this loop to AVX2 later.
PCMPISTRI, the faster of the two, cannot be used at all, because it treats a
zero byte as the end of the subject.

RunTest and pcre2_jit_test pass at all three code unit widths, on x86-32, and
with the SIMD scans compiled out.

[mattst88]

This was something that I noticed could be improved when investigating one of the regressions on Alpha.

Not sure if I should add support for necessary ops to sljit first and then use them here or whether the approach this patch takes is okay. Happy to do either.

[zherczeg]

This looks like a good improvement.

The other similar functions also support relative offset, so not only [range] but also .{n}[range] is also supported where n is < 15. I.e. the first few characters are too complex to search, but there is a simple range after them.

[zherczeg]

I think there is a similar function in the code somewhere. Not sure it can be used here.

[mattst88]

It was the scan at the top of optimize_class_ranges(). The first commit of the
series now moves that into extract_class_ranges(), which takes a limit on the
number of entries instead of the fixed MAX_CLASS_RANGE_SIZE, and sits before
the include of pcre2_jit_simd_inc.h so both users see it.

extract_start_bits_ranges() then only has to turn the transition list into
inclusive pairs. optimize_class_ranges() is otherwise unchanged: the
membership of the highest code unit, which its loop used to leave behind in
bit, is read from the bitmap instead.

[zherczeg]

This looks like a good idea

[zherczeg]

Maybe such helpers could be used to improve code readability.

[mattst88]

Done in the second commit of the series. emit_sse2_op() became
emit_vector_op(), taking an opcode and three register indices and picking the
two operand SSE2 form or the three operand AVX2 one from the register width, and
fast_forward_char_pair_sse2_compare() is rewritten on top of it, losing about
60 lines of hand assembled prefix bytes.

The one place the two shapes genuinely differ is the two character compare,
where the 256 bit form does with one instruction what the 128 bit form needs two
for, so that case no longer falls into the shared switch.

Since the AVX2 path is disabled at runtime I could not exercise it the usual way,
so I compiled the old and the new function side by side against a recording stub
and compared their output: 98304 cases, every combination of compare type,
register width, step and register assignment, byte identical.

[NWilson]

This looks nice.

However, I recommend that we bump it to the next release, rather than merge it and release it in a hurry.

I'm just trying to get all the required bugfixes in, and stabilise the branch ahead of the upcoming release.

[mattst88]

Thanks for the review. The two inline comments are answered in their threads;
this covers the rest.

The other similar functions also support relative offset, so not only [range] but also .{n}[range] is also supported where n is < 15.

This is the fourth commit of the series. scan_prefix() already walks the class
bitmap at each position, but expands it into the chars array and keeps at most
MAX_DIFF_CHARS characters, so a larger class left nothing behind but "any
character". It now records the bitmap itself alongside the array, and a position
keeps its bitmap only while one and the same class reaches it, since a second
alternative, a literal or an unbounded class all make it describe less than the
position accepts.

fast_forward_first_n_chars() picks the sparsest such position, but only where
it would otherwise give up. A position holding one or two characters gets a
cheaper scan of its own, and a run of them gets the shift table, and both beat
testing a handful of ranges, so this takes no work away from them. a.{4}[QXZ]
keeps its single character scan, unchanged at 1486 MB/s.

.{3}[QXZ] goes from 450 to 25798 MB/s and .{2}[0-9]{6} from 647 to 33424
MB/s over a 4MB subject on an i7-1370P. A dense class at an offset, .{3}\w{12},
is left alone as before.

Two things worth flagging separately.

The scan as I first posted it was broken on Win64. It holds the bounds of each
range in a vector register of its own, so it needs twelve of them, but
SLJIT_NUMBER_OF_SAVED_FLOAT_REGISTERS is 10 there and only five vector scratch
registers are requested, so it reached registers the ABI requires to be
preserved. It is now limited to the other x86-64 ABIs the same way
fast_forward_char_pair_simd() is, with a compile time assertion tying the range
limit to SLJIT_NUMBER_OF_SCRATCH_VECTOR_REGISTERS so the budget cannot
silently overflow again.

I also tried handing the bound registers out densely rather than two per range,
so that a class of single characters stays below xmm8 and needs no REX prefix.
It saves four bytes on [QXZ], but costs 7.7% on .{3}[QXZ], stable over
repeated runs, purely from the loop body shifting relative to its alignment. Not
worth it, so it is not in the series.

The branch is force-pushed and rebased on current main. Tested at all three code
unit widths, on x86-32, and with the SIMD scans compiled out.

[NWilson]

I am happy, but haven't reviewed in detail.

I did ask GPT 5.6 to dive in and it didn't find any safety or correctness issues.

The branch is ready for merges again.

@zherczeg, please review and merge whenever you are happy with this.

[NWilson]

@zherczeg Matt's benchmarks look convincing, and GPT reviewed the code and didn't find bugs. I read it myself of course, but I don't know x86 SIMD well enough to find any subtle mistakes myself.

Let me know if you would like me to hold the PR (since it affects JIT).

If I don't hear, I think I'll merge it.

[zherczeg]

So the problem is setting up this scan is too costly for dense classes? Can this be computed by extract_start_bits_ranges and return 0?

[mattst88]

It is not the setup: the constants are loaded once, outside the loop. The
cost is in how soon each loop can stop. The code unit at a time loop stops at
the first code unit in the class, and for a dense class that is usually one of
the first few it looks at, while the vector loop has always tested a whole
block of 16 bytes by then. On \b\w{12,}\b, whose class accepts 63 code units,
the vector scan was 38.6% slower.

But yes, it fits better there. extract_start_bits_ranges() now takes a limit
on the number of code units as well as on the number of ranges, and returns 0
when the bitmap accepts more than that, so the caller only checks the count.
The reasoning moved to the comment on X86_START_BITS_MAX_COVERED.

[zherczeg]

Is the code limited to 8 bit characters? In theory, 16/32 bit characters could be compared.

[mattst88]

Not any more. With the PCMPGT form from your other comment the sequence is the
same at every width, using the word and doubleword instructions, so the width
restriction is gone. Above 8 bits, bit 255 of the start bitmap stands for
every code unit from 255 upwards, as in fast_forward_start_bits(), so there
the range which reaches 255 is extended to the largest code unit. A class
bitmap at an offset gets no such extension, since scan_prefix() only keeps
one where it describes the class on its own; the scan takes a flag saying
which kind of bitmap it has.

Scanning a 4M code unit subject for a class that does not occur, in millions
of code units per second on an i7-1370P, [QXZ] goes from 1444 to 13549 at 16
bits and from 1436 to 6339 at 32 bits, and [0-9]{6} from 3593 to 18228 and
from 3152 to 8406. The scan at an offset gets the same: .{3}[QXZ] goes from
455 to 12427 at 16 bits and from 457 to 5852 at 32 bits, and .{2}[0-9]{6}
from 647 to 18297 and from 646 to 8214. Nothing else measured moves.

While testing this I found a separate bug, which predates this series and is
in 10.47: in the 16-bit and 32-bit libraries scan_prefix() only took a
negated class to accept the code units above 255 when bit 255 of its bitmap
was set, so [^\x00-\x40\x42-\xff] fast-forwarded to the next 'A' and missed
\x{100}. The fix is in #980 on its own so that it can be backported, and
this series now builds on it, since the scan at an offset relies on the
same rule for which classes accept the code units above 255.

[zherczeg]

I would propose two optimizations here.

1. You should use PCMPGTB and friends here. The zero point will be SIMM_MAX (127, 32767, ...).
   Let's assume the character is stored in 'c', and the [20-30] range needs to be checked
   c + (SIMM_MAX - 30) > (SIMM_MAX - (30 - 20) - 1)

This check requires two instructions instead of three.

2. The last compare does not need to copy src_ind to a temporary register, it can use it directly.

I don't know the speed of mask move, but using it after every compare could free a vector register, and use the 'or' operation in normal registers.

[mattst88]

Done, thank you. Each range is now PADD and PCMPGT, where the addend moves the
range to the top of the signed code units, so the zero register is gone, and
the last range is tested in the data register itself. At 8 bits that is
5.3% faster on [QXZ] (25752 to 27121 million
code units per second) and 7.8% faster on [0-9]{6} (33312 to 35898), with
.{2}[0-9]{6} 9.0% faster at an offset. The one pattern to lose is
.{3}[QXZ], 2.7% slower (25718 to 25036), steady over repeated runs. It runs
one instruction fewer per block than before and the same class at offset
zero gains 4.6%, so I take it to be the loop alignment again, as with the
dense register packing I mentioned earlier.

I have not tried moving the mask out after each compare. PMOVMSKB only issues
on one port, and it would turn each POR into a PMOVMSKB and an OR, so I would
expect it to be slower, and with the zero register gone the scan uses eleven
of the sixteen vector registers, so there is room to spare.

[zherczeg]

LGTM
Nice improvement!

[zherczeg]

I just remembered something. That is windows. It expects you to save some simd registers. But deciding at function entry time that you will use those registers is a nightmare. So some simd code is disabled on windows at the moment. If you use more than 6 registers, this might be also an issue here.

[NWilson]

I just remembered something. That is windows. It expects you to save some simd registers. But deciding at function entry time that you will use those registers is a nightmare. So some simd code is disabled on windows at the moment. If you use more than 6 registers, this might be also an issue here.

That's a good point. However, it has been considered. The new code is protected by a Windows-check in the #if - JIT_HAS_FAST_FORWARD_START_BITS_SIMD is not provided on Windows. There are some compiler changes, but I think it's all safe, and the entrypoints to the new code are all conditional.

There's also an explicit assertion at the entrypoint:

/* Three registers for the scan itself, and two more for the constants of
   every range it may be asked to handle. */
SLJIT_COMPILE_ASSERT(3 + X86_START_BITS_MAX_RANGES * 2 <= SLJIT_NUMBER_OF_SCRATCH_VECTOR_REGISTERS,
  not_enough_vector_registers);

I didn't check it manually, but GPT is confident that's the right number of registers the code actually uses.

[NWilson]

After merging your negative-class fix, I'll push a merge commit to this branch.

Actually... I'll let you fix the conflict, so you can check it yourself.

[mattst88]

Rebased!

[NWilson]

@mattst88 GitHub doesn't re-run CI when the main branch changes.

I was a sheep and merged this because the CI was green - but it looks like the build fails on x86 when the Alpha and SIMD PRs are both present.

Are you able to fix that?

[mattst88]

Yep, I'll look into it right away.

[NWilson]

@mattst88

GPT is pretty confident this is the fix.

EDIT: GPT was clearly wrong!

It looks like a chunk of code was mis-merged.

diff --git a/src/pcre2_jit_compile.c b/src/pcre2_jit_compile.c
index 62e994a9..4a0a47f5 100644
--- a/src/pcre2_jit_compile.c
+++ b/src/pcre2_jit_compile.c
@@ -7129,16 +7129,6 @@ if (common->match_end_ptr != 0)
   SELECT(SLJIT_GREATER, STR_END, TMP1, 0, STR_END);
   }

-#ifdef JIT_HAS_FAST_FORWARD_START_BITS_SIMD
-if (JIT_HAS_FAST_FORWARD_START_BITS_SIMD && common->mode == PCRE2_JIT_COMPLETE
-    && fast_forward_start_bits_simd(common, start_bits))
-  {
-  if (common->match_end_ptr != 0)
-    OP1(SLJIT_MOV, STR_END, 0, RETURN_ADDR, 0);
-  return;
-  }
-#endif
-
 start = LABEL();

 partial_quit = CMP(SLJIT_GREATER_EQUAL, STR_PTR, 0, STR_END, 0);