feat(bm25): BM25 ranked text search on standard posting lists

[shaunpatterson]

Supersedes #9652 — reworked per the review feedback there.

What

Adds BM25 ranked full-text search: a new bm25 index tokenizer plus a bm25(predicate, "query", [k], [b]) query function.

How — addressing the #9652 review

The previous version was declined for carrying a parallel storage + retrieval stack alongside the existing one. This rework lands BM25 entirely on Dgraph's standard machinery:

• Term postings are normal index posting lists at IndexKey(attr, IdentBM25‖term), written through addMutation — inheriting MVCC, deltas, rollup, splits, backup, and snapshot. Each posting is a REF posting whose value packs (term-frequency, doc-length).
  • One small core change makes this work: List.encode() now retains REF postings that carry a value through rollup (previously the value was stripped). This mirrors how faceted postings already coexist in both Pack (uid) and Postings (payload). Covered by a forced-rollup regression test.
• No parallel storage stack — bm25block/bm25enc, the LocalCache second write path, the BitBM25Data user-meta, and the dedicated mvcc.go commit branch are gone. MVCC does the delta merging.
• Concurrency: two documents sharing a term now commit concurrently without conflict (standard index conflict key fingerprint(key)^uid), removing the race the previous version mitigated only via Raft serialization.
• Corpus statistics (N, average document length) are sharded across 32 buckets keyed by uid % 32 to avoid a single write-hot key, and summed at read time.
• Scoring is a query-side ranker, not a pseudo-predicate: the bm25 function binds its per-document score to a value variable, so it works through the existing val() system. No bm25_score pseudo-predicate and no new ParentVars channel.
• WAND / Block-Max WAND is kept as an optimization on top of the standard posting-list iterator (per-block max-TF / min-docLen bounds computed from the in-memory list) — not a replacement for it.

One deviation worth your call

The review suggested storing document length "next to the predicate's existing length/count machinery." I instead packed docLen into the posting value alongside TF: a single per-predicate doclen list is a write-conflict hotspot and forces a random read per scored candidate, whereas packing lets scoring read (uid, tf, docLen) in one access. Happy to switch to a dedicated list if you prefer the literal approach.

Usage

description: string @index(bm25) .

{
  results as var(func: bm25(description, "machine learning"))
  q(func: uid(results), orderdesc: val(results), first: 10) {
    description
    score: val(results)
  }
}

Optional BM25 parameters: bm25(description, "...", "1.2", "0.75") (k, b). @index(bm25) is rejected on list predicates (a single document length / corpus stat is not well-defined per list element).

Testing

• Unit: rollup TF/doc-length survival; bucketed-stats accumulation (including cross-transaction and same-bucket read-your-own-writes); value-codec round trip; a 200-trial randomized WAND/Block-Max-WAND vs brute-force correctness check.
• Integration: 34 query BM25 integration tests pass against a multi-node cluster; the full query integration suite passes (no regression from the query-layer changes).

🤖 Generated with Claude Code