fix: avoid versioned describe table for namespace opens

[FANNG1]

What

Avoid passing the requested dataset version to namespace describe_table / describeTable when opening a dataset through a namespace client.

The namespace describe call is only used to resolve the table location and storage options. The requested dataset version is still applied by the lower-level dataset open path after namespace resolution.

Why

Some namespace implementations do not support describing a table from a specific version. Passing version to describe_table can fail even though opening the dataset at that version would work once the location and storage options are resolved.

Fixes lance-format/lance-spark#609.

Testing

• JAVA_HOME=/opt/homebrew/Cellar/openjdk@17/17.0.18/libexec/openjdk.jdk/Contents/Home mvn test -Dtest=DirectoryNamespaceTest#testOpenSpecificVersionDoesNotPassVersionToDescribeTable from java/
• UV_HTTP_TIMEOUT=120 uv run pytest python/tests/test_namespace_dir.py::test_dataset_namespace_open_does_not_pass_version_to_describe_table from python/
• JAVA_HOME=/opt/homebrew/Cellar/openjdk@17/17.0.18/libexec/openjdk.jdk/Contents/Home ./mvnw spotless:check from java/
• UV_HTTP_TIMEOUT=120 uv run ruff format --check --diff python from python/
• UV_HTTP_TIMEOUT=120 uv run ruff check python from python/
• git diff --check
• UV_HTTP_TIMEOUT=120 uv run make lint from python/ (fails in pyright because local optional imports tensorflow and torch are not installed; ruff format --check and ruff check passed)

[yanghua]

Can we also add a test for these change?

[yanghua]

Can we also update here?

[FANNG1]

The public behavior of version does not change here: it still opens the requested dataset version.

This change only adjusts the internal namespace resolution flow. We now use describe_table to resolve table metadata/location/storage options, and apply version later in the lower-level dataset open path. I'm not sure this internal detail should be exposed in the public parameter docs.

Do you think the current version doc is misleading for users, or were you suggesting documenting this namespace-specific implementation detail?

[yanghua]

and apply version later in the lower-level dataset open path

Can you share more details about this?

[FANNG1]

Currently some namespace implementations can support describing a specific table version in describe_table, but not all implementations do. For example, REST-backed namespace implementations may only support resolving the current table metadata/location/storage options from describe_table. This is the background of lance-format/lance-spark#609.

This PR avoids requiring every namespace implementation to support versioned describe_table for the dataset-open path. We only use describe_table to resolve the table location and storage options. The requested dataset version is still passed to the actual dataset open path afterward.

So this keeps compatibility with namespace implementations that support only latest-table describe, while preserving the user-facing behavior of version: the opened dataset is still the requested version.

[FANNG1]

Added Java and Python regression tests that verify namespace describe_table / describeTable does not receive the requested dataset version, while the actual dataset open still returns the requested version.