Skip to content

fix(index): correct cosine RabitQ query factor - #8393

Merged
Xuanwo merged 1 commit into
mainfrom
gatekeeper/fix-5649-1
Aug 9, 2026
Merged

fix(index): correct cosine RabitQ query factor#8393
Xuanwo merged 1 commit into
mainfrom
gatekeeper/fix-5649-1

Conversation

@lance-gatefixer

Copy link
Copy Markdown
Contributor

Summary

  • treat normalized cosine as L2 when calculating the residual-query RabitQ partition factor
  • preserve the Dot-specific offset
  • compare full cosine and L2 calculator outputs in a regression test

Root cause

Cosine vectors are normalized before IVF_RQ search, so the residual-query estimator operates on squared L2 distances. Its query-factor match still grouped Cosine with Dot and subtracted 1.0, shifting every approximate cosine distance.

Validation

  • cargo test -p lance-index test_residual_query_cosine_uses_l2_query_factor
  • cargo fmt --all -- --check
  • cargo clippy --all --tests --benches -- -D warnings

Fixes #5649

@github-actions github-actions Bot added A-index Vector index, linalg, tokenizer bug Something isn't working labels Aug 7, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gate recommendation: approve.

The residual-query estimator now uses the squared-L2 partition factor required by normalized cosine vectors while preserving the distinct dot-product offset. This is the narrow fix for the incorrect legacy score and bound semantics, and the focused regression covers the affected calculator path.

@cursor

cursor Bot commented Aug 9, 2026

Copy link
Copy Markdown

Review: LGTM — approve & merge

For ResidualQuery, Cosine already works on normalized L2 residuals, so q_factor should match L2 (dist_q_c) rather than Dot (dist_q_c - 1.0). Focused change; test now asserts full distance_all equality between Cosine and L2 residual paths.

@Xuanwo
Xuanwo merged commit f1c2565 into main Aug 9, 2026
56 of 57 checks passed
@Xuanwo
Xuanwo deleted the gatekeeper/fix-5649-1 branch August 9, 2026 09:27
@Xuanwo Xuanwo added the K-approved Latest Gatekeeper recommendation permits acceptance. label Aug 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-index Vector index, linalg, tokenizer bug Something isn't working K-approved Latest Gatekeeper recommendation permits acceptance.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(rust): wrong q_factor for Cosine in RabitQuantizationStorage::dist_calculator

1 participant