Skip to content

feat: make embedding dimension configurable#78

Merged
dcfocus merged 1 commit into
mainfrom
codex/issue-73-configurable-embedding-dim
Jun 12, 2026
Merged

feat: make embedding dimension configurable#78
dcfocus merged 1 commit into
mainfrom
codex/issue-73-configurable-embedding-dim

Conversation

@dcfocus

@dcfocus dcfocus commented Jun 12, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Add per-context embedding dimension configuration, defaulting to 1536 for compatibility.
  • Infer the embedding width from existing dataset schemas on open, and reject mismatched explicit dimensions.
  • Plumb embedding_dim through Python Context.create(...) and REST context creation.
  • Add Rust and Python coverage for non-1536 stores and existing 1536 datasets.

Closes #73

Testing

  • cargo fmt
  • cargo test --workspace
  • cargo clippy --workspace --all-targets -- -D warnings
  • uv run --project python --extra tests --reinstall-package lance-context pytest python/tests/test_persistence.py::test_custom_embedding_dimension_round_trips python/tests/test_persistence.py::test_retrieve_fuses_text_vector_and_filters -q
  • uv run --project python --extra dev ruff format --check python/python/lance_context/api.py python/tests/test_persistence.py
  • uv run --project python --extra dev ruff check python/python/lance_context/api.py python/tests/test_persistence.py
  • cd python && /home/user/.local/bin/uv run --extra dev pyright
  • git diff --check

Note: .codex/skills/ci-pr-helper/scripts/run_ci_checks.sh still points at rust/lance-context/Cargo.toml, which does not exist in the current repo layout, so I ran the equivalent current-layout checks directly.

@dcfocus dcfocus force-pushed the codex/issue-73-configurable-embedding-dim branch from 3e63b8e to c32d2c7 Compare June 12, 2026 02:59
@dcfocus dcfocus merged commit acbf131 into main Jun 12, 2026
9 checks passed
@dcfocus dcfocus mentioned this pull request Jun 12, 2026
25 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make embedding dimension configurable (currently hardcoded to 1536)

1 participant