Skip to content

refactor(array): allocate execution outputs through context - #9671

Merged
gatesn merged 6 commits into
developfrom
ngates/buffer-allocator-execution
Sep 12, 2026
Merged

gatesn merged 6 commits into
developfrom
ngates/buffer-allocator-execution

Conversation

@gatesn

@gatesn gatesn commented Aug 27, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Allocate execution outputs through the execution context.

Changes

  • Propagate allocators through filter, take, interleave, and patches.
  • Use context allocators in dictionary and scalar functions.
  • Add tests for allocator propagation.

@codspeed

codspeed Bot commented Aug 27, 2026 •

Copy link
Copy Markdown

Merging this PR will degrade performance by 5.16%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 3 improved benchmarks
❌ 8 regressed benchmarks
✅ 2186 untouched benchmarks
⏩ 218 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ WallTime mul_u64_nonnull_neon 15.2 µs 20.9 µs -27.21%
❌ WallTime mul_i64_nonnull_neon 17.1 µs 20.1 µs -15.02%
❌ WallTime multiply_shapes_neon[(16384, PerRowPerRow)] 17.1 µs 20.1 µs -14.68%
❌ WallTime dict_canonicalize_gt_u8_avx2[16000000] 7 ms 8.2 ms -14.37%
❌ Simulation take_search[(0.005, 0.05)] 19.8 µs 22.8 µs -13.37%
❌ Simulation take_search[(0.01, 0.05)] 20.5 µs 23.6 µs -13.14%
❌ WallTime dict_canonicalize_gt_u8_avx2[1000000] 427.7 µs 481.1 µs -11.1%
❌ Simulation take_search[(0.005, 0.1)] 26.1 µs 29.1 µs -10.31%
⚡ WallTime arrow_checked_add_u32_neon[16384] 20.4 µs 13.4 µs +52.9%
⚡ WallTime filtered_owned_i64_avx512[OneNullInEight] 26.3 µs 22.1 µs +18.76%
⚡ Simulation allocate_drop_arrow[0] 456.9 ns 402.7 ns +13.45%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing ngates/buffer-allocator-execution (ed0df23) with develop (ad925fd)2

Open in CodSpeed

Footnotes

  1. 218 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

  2. No successful run was found on develop (ed4d347) during the generation of this report, so ad925fd was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

@gatesn gatesn mentioned this pull request Aug 27, 2026
5 tasks
@gatesn
gatesn force-pushed the ngates/buffer-allocator-execution branch from f8000e4 to 9a8b653 Compare August 27, 2026 19:38
@gatesn
gatesn force-pushed the ngates/buffer-allocator-execution branch 2 times, most recently from 6931802 to ed2fbe7 Compare August 27, 2026 21:37
@gatesn
gatesn force-pushed the ngates/buffer-allocator-execution branch from ed2fbe7 to 3c868b7 Compare August 27, 2026 21:38
@gatesn
gatesn force-pushed the ngates/buffer-allocator-execution branch 2 times, most recently from 09d740d to 3e127b9 Compare August 28, 2026 14:56
@gatesn
gatesn force-pushed the ngates/buffer-allocator-execution branch 2 times, most recently from 8d4cbfb to 92397b9 Compare August 28, 2026 17:03
@gatesn
gatesn force-pushed the ngates/buffer-allocator-execution branch from 92397b9 to da81af8 Compare August 28, 2026 17:35
@gatesn
gatesn force-pushed the ngates/buffer-allocator-execution branch from da81af8 to 6210f3b Compare August 28, 2026 18:02
@gatesn
gatesn force-pushed the ngates/buffer-allocator-execution branch 2 times, most recently from 375111f to af6234b Compare August 28, 2026 18:24
@gatesn
gatesn force-pushed the ngates/buffer-allocator-execution branch from af6234b to aa6801b Compare August 28, 2026 20:41
@gatesn
gatesn force-pushed the ngates/buffer-allocator-execution branch 2 times, most recently from 903c310 to 2898fb0 Compare August 28, 2026 20:54
@gatesn
gatesn force-pushed the ngates/buffer-allocator-execution branch from 20a5f31 to 2af2152 Compare September 2, 2026 16:59
@gatesn
gatesn force-pushed the ngates/buffer-allocator-execution branch 2 times, most recently from 2377f06 to a8ffb77 Compare September 2, 2026 17:27
@gatesn
gatesn force-pushed the ngates/buffer-allocator-execution branch from a8ffb77 to 929ff0a Compare September 2, 2026 21:41
@gatesn
gatesn force-pushed the ngates/buffer-allocator-execution branch 3 times, most recently from e099a82 to c8eae21 Compare September 3, 2026 00:11
Base automatically changed from ngates/buffer-allocator-builders to develop September 9, 2026 15:48
@gatesn
gatesn force-pushed the ngates/buffer-allocator-execution branch 5 times, most recently from bfb9ddf to 0264ccd Compare September 9, 2026 18:26
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
@gatesn
gatesn force-pushed the ngates/buffer-allocator-execution branch from 0264ccd to d3fcfb0 Compare September 9, 2026 18:36
Signed-off-by: Nicholas Gates <nick@nickgates.com>

@robert3005 robert3005 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there some follow up for other crates? I think you want to forbid in the codebase the non allocator taking methods which you can do via clippy lint

Comment thread vortex-array/src/scalar_fn/fns/list_contains/mod.rs Outdated
Comment thread vortex-array/src/arrays/fixed_width/filter.rs Outdated
Comment thread vortex-array/src/patches.rs Outdated
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
@gatesn

gatesn commented Sep 11, 2026

Copy link
Copy Markdown
Contributor Author

On workspace-wide enforcement: #9672 was the original follow-up, but its targeted regex checker was the wrong mechanism and was closed. Clippy disallowed-methods is workspace-wide, so enabling it for the convenience constructors now would also reject intentional uses outside allocator-aware execution, including vortex-buffer itself. I agree that a workspace migration followed by a Clippy rule is the right endpoint, but it should remain a separate change. This PR keeps the public convenience APIs while making execution allocations explicit and hiding retained allocator access from buffer callers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants