Skip to content

Deprecate FromArrowArray in favour of ArrowSession array imports - #8927

Open
robert3005 wants to merge 7 commits into
developfrom
rk/deprecate-arrow-arrays
Open

Deprecate FromArrowArray in favour of ArrowSession array imports#8927
robert3005 wants to merge 7 commits into
developfrom
rk/deprecate-arrow-arrays

Conversation

@robert3005

@robert3005 robert3005 commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Remove all usages of FromArrowArray and ToArrowArray in favour of using ArrowSession

@codspeed-hq

codspeed-hq Bot commented Jul 23, 2026

Copy link
Copy Markdown

Merging this PR will regress 1 benchmark

⚡ 5 improved benchmarks
❌ 1 regressed benchmark
✅ 1934 untouched benchmarks
🆕 12 new benchmarks
⏩ 89 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation cold_misaligned[(64, 256)] 4.4 ms 5.1 ms -14.19%
Simulation decode_varbin[(1000, 4)] 99.3 µs 60.5 µs +63.93%
Simulation slice_primitive_tight_loop[10000] 502.7 µs 438.1 µs +14.76%
Simulation slice_dict_tight_loop[10000] 837.8 µs 738.5 µs +13.45%
Simulation decompress[u64, (1000, 16)] 72.5 µs 65 µs +11.52%
Simulation compress_fsst[(1000, 64, 8)] 1.2 ms 1 ms +10.52%
🆕 Simulation primitive_mixed_constants[random/n2/nonnull] N/A 420.6 µs N/A
🆕 Simulation primitive_mixed_constants[random/n2/null] N/A 428.7 µs N/A
🆕 Simulation primitive_mixed_constants[random/n64/nonnull] N/A 650.6 µs N/A
🆕 Simulation primitive_mixed_constants[random/n64/null] N/A 715.8 µs N/A
🆕 Simulation primitive_mixed_constants[round_robin/n2/nonnull] N/A 355.9 µs N/A
🆕 Simulation primitive_mixed_constants[round_robin/n2/null] N/A 357.5 µs N/A
🆕 Simulation primitive[random/n2/nonnull] N/A 537.9 µs N/A
🆕 Simulation primitive[random/n2/null] N/A 545.9 µs N/A
🆕 Simulation primitive[random/n64/nonnull] N/A 869.8 µs N/A
🆕 Simulation primitive[random/n64/null] N/A 910.7 µs N/A
🆕 Simulation primitive[round_robin/n2/nonnull] N/A 376.8 µs N/A
🆕 Simulation primitive[round_robin/n2/null] N/A 384.2 µs N/A

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing rk/deprecate-arrow-arrays (6e1dc3e) with develop (e77f520)2

Open in CodSpeed

Footnotes

  1. 89 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

  2. No successful run was found on develop (9157d6c) during the generation of this report, so e77f520 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report.

@robert3005
robert3005 force-pushed the rk/deprecate-arrow-arrays branch from 2bdf0b7 to 154677c Compare July 23, 2026 16:23
@robert3005
robert3005 force-pushed the rk/deprecate-arrow-arrays branch from 154677c to 97e60f3 Compare July 23, 2026 21:49
@robert3005
robert3005 force-pushed the rk/deprecate-arrow-arrays branch 2 times, most recently from ff9c087 to 3c9fce8 Compare July 23, 2026 22:52
@robert3005
robert3005 force-pushed the rk/deprecate-arrow-arrays branch from 3c9fce8 to 7b17f7f Compare July 31, 2026 14:44
@robert3005
robert3005 force-pushed the rk/deprecate-arrow-arrays branch 2 times, most recently from 4ece166 to ffd3548 Compare August 5, 2026 19:18
@robert3005
robert3005 marked this pull request as ready for review August 5, 2026 20:18
@robert3005 robert3005 added the changelog/chore A trivial change label Aug 6, 2026
@robert3005
robert3005 force-pushed the rk/deprecate-arrow-arrays branch 2 times, most recently from d8fe764 to 28f0451 Compare August 10, 2026 13:04
@robert3005
robert3005 force-pushed the rk/deprecate-arrow-arrays branch from 28f0451 to e4dd6aa Compare August 10, 2026 13:48
Base automatically changed from rk/deprecate to develop August 10, 2026 15:50
Comment thread vortex-arrow/src/convert.rs
Comment thread vortex-arrow/src/iter.rs
Follow-up to #8918: `FromArrowArray` is now #[deprecated] and every usage is
replaced with `ArrowSession` methods, making the session the authoritative
Arrow array import path.

- `ArrowSession` gains `from_arrow_array_nullable(&dyn Array, bool)` for
  imports with no Arrow `Field` in hand; nested fields still dispatch
  extension import plugins.
- `ArrowImportVTable::from_arrow_array` now receives the `ArrowSession` so
  plugins convert storage through the session (uuid, geo, json, tensor,
  parquet-variant updated).
- All consumers (datafusion, python, ffi, tui, bench, compat-gen, layout,
  facade doc snippet) migrated to `from_arrow_array` /
  `from_arrow_array_nullable` / `from_arrow_record_batch`.
- `vx_array_from_arrow` now imports through the session using the FFI
  schema's field, so Arrow extension types round-trip.
- `IntoArrowArray` was already deprecated and has no remaining users.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Robert Kruszewski <github@robertk.io>
Comment thread vortex-arrow/src/datum.rs Outdated
robert3005 and others added 3 commits August 10, 2026 16:56
…ions

- `ParquetVariant::from_arrow_variant{,_nullable}` take an `&ArrowSession`
  threaded from vtables/kernels, so shredded storage children resolve
  through the registered import plugins.
- `native_geometry_scalar_from_wkb` takes an `&ArrowSession`; vortex-duckdb
  passes its crate session.
- `vx_array_from_arrow` takes a `const vx_session*` (header regenerated).
- Extract each `FromArrowArray` impl body into a named, invocable
  `vortex_arrow::convert::from_arrow_*` function (primitive, decimal,
  temporal, bytes, byte-view, boolean, struct, list, list-view,
  fixed-size-list, null, dictionary, run-ends, dyn dispatch, record batch);
  the deprecated trait impls are now thin shims so the trait can eventually
  be deleted.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Robert Kruszewski <github@robertk.io>
Array::from_arrow takes a Session; C header regenerated.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Robert Kruszewski <github@robertk.io>
Two `ArrayRef::from_arrow` call sites landed on develop while this branch was
open, so the rebase reintroduced deprecation warnings the PR exists to remove.
Both read a whole Arrow `RecordBatch`, so they convert to
`ArrowSession::from_arrow_record_batch`, matching the other migrated readers.

Signed-off-by: "Robert Kruszewski" <robert@spiraldb.com>
@robert3005
robert3005 force-pushed the rk/deprecate-arrow-arrays branch from e4dd6aa to 51b8065 Compare August 10, 2026 15:57
Signed-off-by: Robert Kruszewski <github@robertk.io>
Signed-off-by: Robert Kruszewski <github@robertk.io>
Signed-off-by: Robert Kruszewski <github@robertk.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/chore A trivial change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants