Skip to content

chore(bigframes): Add substrait based datafusion, acero backends - #17215

Closed
TrevorBergeron wants to merge 29 commits into
mainfrom
tbergeron_substrait
Closed

TrevorBergeron wants to merge 29 commits into
mainfrom
tbergeron_substrait

Conversation

@TrevorBergeron

Copy link
Copy Markdown
Contributor

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

  • Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
  • Ensure the tests and linter pass
  • Code coverage does not decrease (if any source code was changed)
  • Appropriate docs were updated (if necessary)

Fixes #<issue_number_goes_here> 🦕

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a Substrait compilation and execution framework for BigFrames, featuring a compiler that translates logical plans into Substrait format and an executor that utilizes Apache DataFusion for local execution. The implementation handles various relational operations such as joins, aggregations, and window functions, and includes a testing suite with a dedicated test session. Review feedback focuses on enhancing code quality by addressing a potential NameError, removing debug print and stderr statements, replacing hardcoded magic numbers with existing mappings, and optimizing import placement for better performance.

Comment thread packages/bigframes/bigframes/core/compile/substrait/compiler.py Outdated
Comment thread packages/bigframes/bigframes/core/compile/substrait/compiler.py
Comment thread packages/bigframes/bigframes/core/compile/substrait/compiler.py Outdated
Comment thread packages/bigframes/bigframes/core/compile/substrait/compiler.py Outdated
Comment thread packages/bigframes/bigframes/core/compile/substrait/compiler.py Outdated
Comment thread packages/bigframes/bigframes/core/compile/substrait/compiler.py Outdated
Comment thread packages/bigframes/bigframes/session/substrait_executor.py Outdated
@TrevorBergeron TrevorBergeron changed the title feat(bigframes): Add substrait-datafusion backend chore(bigframes): Add substrait based datafusion, acero backends Jun 5, 2026
@TrevorBergeron
TrevorBergeron force-pushed the tbergeron_substrait branch 2 times, most recently from aba16ab to 4a882f4 Compare June 24, 2026 20:17
@TrevorBergeron
TrevorBergeron requested a review from tswast June 24, 2026 22:09
@parthea

parthea commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

I'm going to close this PR due to no activity in the last 2 months. Please feel free to re-open this PR or create a new PR.

@parthea parthea closed this Aug 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants