Skip to content

feat: add native standalone \X matching - #155

Draft
bric3 wants to merge 2 commits into
DataDog:mainfrom
bric3:bdu/native-grapheme
Draft

bric3 wants to merge 2 commits into
DataDog:mainfrom
bric3:bdu/native-grapheme

Conversation

@bric3

@bric3 bric3 commented Nov 1, 2026 •

Copy link
Copy Markdown

What does this PR do?

Add a stateless native matcher for the exact pattern \X, available through Reggie.compile() and @RegexPattern. Each match consumes one grapheme cluster using the executing JDK's rules.

Area Description Supported Example
Patterns One complete grapheme cluster yes \X on "e\u0301" matches both code points
Patterns Capturing groups around \X no (\X)
Patterns Non-capturing groups around \X no (?:\X)
Patterns Quantified \X no \X+
Patterns Concatenation with \X no a\X
Flags Inline modifiers no (?i)\X
Flags Existing external flags yes Reggie.compile("\\X", ReggieFlags.CASE_INSENSITIVE)
Execution Native runtime compilation yes Reggie.compile("\\X")
Execution Annotation-generated accessors yes @RegexPattern("\\X")
Execution Specialised LTS API admission no ReggieCompiledPattern.tryCompile(...) rejects \X
JDKs Native matching on GA JDKs 21-27 yes JDK 25 selects its Unicode 16 profile
JDKs Native matching on EA or unknown profiles no Strict compilation on JDK 28 EA throws UnsupportedPatternException
Fallback Explicit delegation on unsupported profiles yes Reggie.compileAllowingFallback("\\X") on JDK 28 EA
Allocation Allocation-free bounds and Into scans yes findBoundsFrom(text, 0, reusedBounds)
Allocation Allocation-free rich results no findMatch(text) creates a result and group arrays

Unsupported profiles retain strict refusal and explicit opt-in JDK fallback. The specialised LTS API keeps its existing admission rules.

Motivation

Provide correct grapheme boundaries for combining characters, emoji sequences and CRLF, with no per-cluster allocation during bounds traversal.

Remaining features and implementation path

The dedicated scanner handles exact \X before AST/NFA construction. General composition needs broader changes across the parser and AST, a native execution path that supports variable-width atoms, pattern analysis and strategy selection. Length calculations, capture metadata and structural cache keys must reflect grapheme consumption in both runtime compilation and annotation processing.

These changes must preserve UTF-16 offsets and greedy/lazy behaviour, with JDK differential tests, Unicode conformance tests and benchmarks for mixed patterns. Supporting new JDK profiles also involves Unicode data maintenance; reusable rich results need a public API contract for ownership and lifetime.

Feature Why it is deferred Work needed
Non-capturing groups: (?:\X) Admission currently recognises only exact \X. Add syntax-aware wrapper recognition in runtime compilation and annotation processing.
Capturing groups: (\X) The matcher exposes only group 0. Extend admission, group metadata and UTF-16 capture spans across rich results and caller-provided arrays.
Quantifiers and concatenation: \X+, a\X Existing character transitions consume UTF-16 code units; a grapheme consumes a variable number. Add a grapheme AST atom and a compatible execution path. Update analysis and strategy routing in both compilation paths, preserve greedy/lazy behaviour at cluster boundaries, and benchmark mixed patterns.
Inline flags: (?i)\X The exact-source check excludes inline syntax, even when the flag leaves segmentation unchanged. Recognise and validate inline flags in both compilation paths, preserving flag scope and literal semantics.
Specialised LTS API ReggieCompiledPattern admits its own full-capture linear-token-sequence profile. Extend its full-capture token plan and match state for graphemes while proving native-only, linear-time and interruptible CharSequence guarantees.
Early-access and unknown JDK profiles JDK grapheme rules and Unicode properties can change; parity requires an audited profile. Audit the target GA JDK, update versioned rules/data and fixtures, then run conformance and differential tests through runtime and annotation-generated accessors before admission.
Allocation-free rich results Returned result objects and group arrays retain each match's data. Define an explicit caller-owned reusable result API with clear lifetime and ownership. Existing bounds and Into APIs already support allocation-free scans.

These are follow-up changes. General composition needs correctness and performance checks for mixed character/grapheme transitions, captures and repeated scans.

Related Issue(s)

None.

Change Type

  • Bug fix
  • New feature
  • Performance improvement
  • Refactoring (no functional change)
  • Documentation
  • Test improvement
  • Build/CI change

Checklist

  • I have read the CONTRIBUTING.md guidelines
  • All existing tests pass (./gradlew build)
  • I have added tests for my changes
  • I have updated documentation (if applicable)
  • My commits are signed

Performance Impact

On Java 27 / Apple M4 Max, complete bounds traversal was 1.3x-6x faster than a reused JDK matcher across eight tested corpora. A matching-thread allocation probe measured zero bytes; rich results still allocate.

Measurements and reproduction commands.

Additional Notes

Runtime and Java 21-generated accessors passed the JDK 21-27 matrix. JDK 28 EA covers refusal, fallback and repeated accessor failures. Hosted CI is pending. Unicode conformance fixtures are vendored unchanged.

@codecov-commenter

codecov-commenter commented Nov 1, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.83333% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.2%. Comparing base (57a288f) to head (f87ef8c).

Files with missing lines Patch % Lines
...ava/com/datadoghq/reggie/runtime/GraphemeData.java 90.5% 1 Missing and 4 partials ⚠️
.../com/datadoghq/reggie/runtime/GraphemeScanner.java 97.9% 0 Missing and 1 partial ⚠️
.../com/datadoghq/reggie/runtime/RuntimeCompiler.java 85.7% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@           Coverage Diff            @@
##              main    #155    +/-   ##
========================================
  Coverage     85.1%   85.2%            
- Complexity       1     272   +271     
========================================
  Files          169     172     +3     
  Lines        50037   50205   +168     
  Branches      7712    7760    +48     
========================================
+ Hits         42616   42783   +167     
+ Misses        5406    5400     -6     
- Partials      2015    2022     +7     
Files with missing lines Coverage Δ
...q/reggie/processor/ImplClassBytecodeGenerator.java 95.5% <100.0%> (+0.2%) ⬆️
...adoghq/reggie/processor/RegexPatternProcessor.java 66.8% <100.0%> (+1.0%) ⬆️
...ggie/processor/ReggieMatcherBytecodeGenerator.java 85.6% <100.0%> (+<0.1%) ⬆️
.../com/datadoghq/reggie/runtime/GraphemeMatcher.java 100.0% <100.0%> (ø)
.../com/datadoghq/reggie/runtime/GraphemeScanner.java 97.9% <97.9%> (ø)
.../com/datadoghq/reggie/runtime/RuntimeCompiler.java 83.1% <85.7%> (+0.2%) ⬆️
...ava/com/datadoghq/reggie/runtime/GraphemeData.java 90.5% <90.5%> (ø)

... and 2 files with indirect coverage changes


Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 57a288f...f87ef8c. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@bric3 bric3 changed the title feat: add native standalone \X matching feat: add native standalone \X matching Nov 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants