Skip to content

Use chunked numeric posting buffers - #14

Merged
platypii merged 1 commit into
masterfrom
perf/chunked-posting-buffers
Aug 1, 2026
Merged

Use chunked numeric posting buffers#14
platypii merged 1 commit into
masterfrom
perf/chunked-posting-buffers

Conversation

@platypii

@platypii platypii commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

Summary

  • replace per-token numeric Maps and JavaScript posting arrays with append-only typed-array chunks
  • group postings by token suffix with a stable counting sort while draining each prefix
  • reuse counting-sort scratch storage and release chunks as prefixes are written
  • retain the existing string-Map path for custom tokenizer configurations

WildChat benchmark

Same 15,587,491,031-byte source, 3,199,860 rows, default index settings, and measurement command on both commits.

Metric master (6f45d8b) this branch (c22ccfa) Change
Total build time 130.01 min 36.75 min 71.7% lower / 3.54x faster
Read, tokenize, accumulate postings ~76.36 min ~20.15 min ~73.6% lower / 3.79x faster
Group, sort, encode, and write ~53.65 min ~16.61 min ~69.0% lower / 3.23x faster
Peak RSS 40.01 GB 21.61 GB 46.0% lower
Output size 3,622,096,587 bytes 3,622,096,587 bytes unchanged

The phase boundary is approximate because the benchmark records one-minute heartbeats rather than explicit internal phase timers. Total wall time and peak RSS come directly from performance timing and macOS /usr/bin/time.

The branch output is byte-for-byte identical to the master artifact.

Test plan

  • npm test (153 tests pass)
  • npm run lint
  • full WildChat index build
  • cmp against the master-generated index artifact

@platypii
platypii merged commit dbfbc55 into master Aug 1, 2026
6 checks passed
@platypii
platypii deleted the perf/chunked-posting-buffers branch August 1, 2026 06:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant