Skip to content

Add four-symbol shape fallback - #11

Merged
platypii merged 1 commit into
masterfrom
dual-shape-ngrams
Jul 31, 2026
Merged

Add four-symbol shape fallback#11
platypii merged 1 commit into
masterfrom
dual-shape-ngrams

Conversation

@platypii

Copy link
Copy Markdown
Contributor

Summary

  • retain the selective 8-symbol ordinary shape layer as the primary regex projection
  • add a metadata-gated 4-symbol ordinary fallback for patterns too short to produce an 8-symbol shape
  • prefer literal n-grams, then ordinary shape-8, hex shape-8, and finally ordinary shape-4
  • extract both ordinary lengths from one shared ring buffer in the existing source traversal
  • preserve compatibility with indexes and readers that only understand the singular primary shape metadata

Performance

Measured on the local 25k-row WildChat sample against the shape-8 plus hex index from #10:

  • compressed index: 22,065,227 to 22,157,787 bytes (+0.42%)
  • posting rows: 12,534,845 to 12,749,559 (+1.71%)
  • CPU-only full extraction phase: about +20%; this excludes Parquet read/write time
  • MM/DD candidates: 2,500 to 456
  • short letter/digit code candidates: 2,500 to 1,401

Long patterns keep the shape-8 results: slash date 76 blocks, SSN 6, phone 42, card 14, fixed IPv4 37, and UUID 49.

Verification

  • npm test (145 tests)
  • npm run lint

@platypii
platypii merged commit c98ec5e into master Jul 31, 2026
6 checks passed
@platypii
platypii deleted the dual-shape-ngrams branch July 31, 2026 07:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant