Skip to content

BE-306: HashQL: PostgreSQL translation#8526

Open
indietyp wants to merge 9 commits intobm/be-457-hashql-mir-execution-pipeline-extensions-for-postgresfrom
bm/be-306-hashql-postgres-translation
Open

BE-306: HashQL: PostgreSQL translation#8526
indietyp wants to merge 9 commits intobm/be-457-hashql-mir-execution-pipeline-extensions-for-postgresfrom
bm/be-306-hashql-postgres-translation

Conversation

@indietyp
Copy link
Copy Markdown
Member

@indietyp indietyp commented Mar 8, 2026

🌟 What is the purpose of this PR?

Implements the postgres compilation backend for HashQL. Takes the MIR control flow graph (after execution analysis has assigned basic blocks to backends and partitioned them into islands) and compiles the Postgres-assigned islands into SQL SELECT statements.

🔍 What does this change?

Postgres compiler (eval/src/postgres/mod.rs):
Top-level entry point. Compiles a GraphRead body island-by-island into a PreparedQuery (a SelectStatement + deduplicated Parameters list). Each Postgres island becomes a CROSS JOIN LATERAL subquery returning a continuation composite value. The continuation carries filter (keep/reject/passthrough), block (next basic block), and locals/values (live-out data for the interpreter to resume from).

Filter compiler (eval/src/postgres/filter/):
Walks the MIR basic blocks within an island and compiles each statement into SQL expressions. Uses an explicit frame stack (not recursion) to handle SwitchInt terminators: each branch becomes a CASE WHEN arm, with the discriminant cast to ::int to avoid boolean/integer type mismatches in PostgreSQL. Out-of-island branches produce continuation values that encode which block to resume and what locals to carry.

Projections (eval/src/postgres/projections.rs):
Maps EntityPath variants to SQL column references or JSONB extraction expressions. Tracks which table joins are needed and only requests them when a path is actually referenced. Handles the split between "column-backed" paths (entity_uuid, web_id, etc.) and "JSONB-backed" paths (properties, type IDs).

Parameters (eval/src/postgres/parameters.rs):
Builds the $1, $2, ... parameter list for the prepared statement. Deduplicates by identity. Each Parameter variant represents a different source: Input (user-provided values), Symbol/Primitive/Int (query literals), Env (closure captures), TemporalAxis (execution context). The CompiledQuery return type exposes which indices correspond to which sources so the interpreter can bind them.

Continuation (eval/src/postgres/continuation.rs):
Builds the ROW(filter, block, locals, values)::continuation composite values that encode island exit state. Handles the three exit cases: passthrough (NULL continuation), filter-only (just a boolean), and full exit (block + live-out locals serialized as parallel int[]/jsonb[] arrays).

Traverse (eval/src/postgres/traverse.rs):
Compiles graph traversal requirements into SQL joins. Reads the island's provides set to determine which entity paths need table joins, then requests them from the database context layer.

Error infrastructure (eval/src/postgres/error.rs):
Diagnostic types for compilation errors (unsupported operations, type mismatches, missing paths) with span-accurate source locations.

Context (eval/src/context.rs):
DatabaseContext trait and implementation that the compiler uses to request table aliases, register joins, and access the schema. Bridges between the HashQL type system and the graph-store query builder.

Compiletest suite (compiletest/src/suite/eval_postgres.rs):
New compiletest suite that runs the full pipeline (parse, type-check, lower to MIR, run execution analysis, compile to SQL) and compares the output against blessed .stdout files. Also emits .aux.mir secondary outputs showing the MIR after execution analysis for debugging.

Pre-Merge Checklist 🚀

🚢 Has this modified a publishable library?

This PR:

  • does not modify any publishable blocks or libraries, or modifications do not need publishing

📜 Does this require a change to the docs?

The changes in this PR:

  • are internal and do not require a docs change

🕸️ Does this require a change to the Turbo Graph?

The changes in this PR:

  • do not affect the execution graph

⚠️ Known issues

OFFSET 0 on lateral subqueries is a workaround for PostgreSQL inlining composites; see continuation.rs doc comments for details.

🛡 What tests cover this?

  • Filter compiler unit tests (filter/tests.rs, ~1000 lines) using insta snapshots covering: straight-line blocks, branching CFGs, diamond merges, island exits, projections, property access, parameter deduplication, lateral subquery generation
  • 30+ compiletest cases under eval/tests/ui/postgres/ covering end-to-end compilation: comparison operators, entity field access, input parameters, let bindings, if-expressions, nested branching, environment captures, list/dict/struct/tuple construction, multiple filters, mixed-source filters

❓ How to test this?

cargo nextest run -p hashql-eval
cargo run --package hashql-compiletest -- run

indietyp added 2 commits March 8, 2026 16:17
feat: checkpoint (II)

feat: checkpoint (III)

feat: snapshot vec

feat: add dedicated filter

feat: checkpoint

feat: filter implementation

feat: filter implementation (mostly) done

chore: environment capture note

chore: always postgres bigint

feat: target clone

feat: simplify lookup

feat: move storage up

feat: eval entity path

chore: checkpoint

chore: checkpoint

chore: find entrypoint

feat: eval context

feat: eval cleanup

chore: cleanup

feat: track index

feat: wire up filter

feat: add error reporting

chore: checkpoint

feat: add traverse, and first postgres compiler outline

feat: traverse bitmap

feat: move traversal out

feat: projections

feat: projections

fix: clippy

feat: subquery projection for lateral

feat: checkpoint

feat: test plan

feat: checkpoint

feat: checkpoint – failing tests ;-;

feat: checkpoint – failing tests ;-;

feat: checkpoint — passing tests

fix: import

fix: entity type

feat: checkpoint

feat: attribute a cost to terminator placement switches

fix: import

feat: checkpoint

feat: checkpoint

chore: lint
@cursor
Copy link
Copy Markdown

cursor Bot commented Mar 8, 2026

PR Summary

Medium Risk
Adds a substantial new SQL compilation path (MIR islands → PostgreSQL SELECT with lateral continuations) and expands the Postgres query AST with a new boolean type; while mostly additive, it affects query generation and could surface as runtime SQL/parameter mismatches.

Overview
Implements a new HashQL PostgreSQL compilation backend that lowers MIR GraphRead filter islands into a prepared PostgreSQL SELECT, emitting CROSS JOIN LATERAL continuation subqueries and a deduplicated $N parameter catalog.

Adds supporting infrastructure in hashql-eval (new EvalContext with live-out analysis, Postgres diagnostics, parameter/projection planning, entity-path→SQL mapping, and a CFG-to-CASE filter compiler with island-exit continuations), plus extensive new snapshot/compiletest coverage and updates workspace deps/docs; also extends the Postgres query builder with a PostgresType::Boolean transpilation.

Reviewed by Cursor Bugbot for commit b399240. Bugbot is set up for automated code reviews on this repo. Configure here.

@vercel
Copy link
Copy Markdown

vercel Bot commented Mar 8, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
hash Ready Ready Preview, Comment May 4, 2026 11:47am
petrinaut Ready Ready Preview May 4, 2026 11:47am
2 Skipped Deployments
Project Deployment Actions Updated (UTC)
hashdotdesign Ignored Ignored Preview May 4, 2026 11:47am
hashdotdesign-tokens Ignored Ignored Preview May 4, 2026 11:47am

Copy link
Copy Markdown
Member Author

indietyp commented Mar 8, 2026

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

This stack of pull requests is managed by Graphite. Learn more about stacking.

@github-actions github-actions Bot removed the area/infra Relates to version control, CI, CD or IaC (area) label Mar 8, 2026
@augmentcode
Copy link
Copy Markdown

augmentcode Bot commented Mar 8, 2026

🤖 Augment PR Summary

Summary: Adds a PostgreSQL compilation backend for HashQL by lowering MIR execution islands into SQL SELECT statements with continuation-based control flow.

Key changes:

  • Introduces eval/src/postgres/ with a top-level PostgresCompiler that compiles GraphRead filters island-by-island into lateral subqueries returning a typed continuation composite.
  • Adds a CFG-to-SQL filter compiler (postgres/filter) that builds CASE WHEN trees iteratively (explicit frame stack) and emits island-exit continuations carrying live-out locals.
  • Implements parameter deduplication (postgres/parameters.rs) and lazy join planning/projection mapping (postgres/projections.rs, postgres/traverse.rs).
  • Adds new eval-level EvalContext + live-out dataflow computation used by the Postgres backend.
  • Extends diagnostics to include Postgres backend categories (postgres/error.rs + eval/src/error.rs integration).
  • Adds a new compiletest suite (compiletest/src/suite/eval_postgres.rs) and a sizable set of end-to-end UI test fixtures under eval/tests/ui/postgres/, plus MIR snapshot tests for hard-to-produce MIR shapes.

Technical notes: Compiled filter islands are materialized with CROSS JOIN LATERAL and OFFSET 0 to prevent Postgres from inlining/duplicating CASE trees; selected entity paths are driven by the island provides set and only trigger joins when referenced.

🤖 Was this summary useful? React with 👍 or 👎

Copy link
Copy Markdown

@augmentcode augmentcode Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review completed. 2 suggestions posted.

Fix All in Augment

Comment augment review to trigger a new review at any time.

Comment thread libs/@local/hashql/eval/src/postgres/mod.rs
Comment thread libs/@local/hashql/eval/src/postgres/traverse.rs Outdated
@codspeed-hq
Copy link
Copy Markdown

codspeed-hq Bot commented Mar 8, 2026

Merging this PR will not alter performance

✅ 80 untouched benchmarks


Comparing bm/be-306-hashql-postgres-translation (b399240) with bm/be-457-hashql-mir-execution-pipeline-extensions-for-postgres (60da845)

Open in CodSpeed

Comment thread libs/@local/hashql/eval/src/postgres/continuation.rs Fixed
Comment thread libs/@local/hashql/eval/src/postgres/filter/mod.rs Fixed
Comment thread libs/@local/hashql/eval/src/postgres/filter/mod.rs Fixed
Comment thread libs/@local/hashql/eval/src/postgres/filter/mod.rs Fixed
Comment thread libs/@local/hashql/eval/src/postgres/mod.rs Fixed
Comment thread libs/@local/hashql/eval/src/postgres/mod.rs Fixed
Comment thread libs/@local/hashql/eval/src/postgres/parameters.rs Fixed
Comment thread libs/@local/hashql/eval/src/postgres/projections.rs Fixed
Comment thread libs/@local/hashql/eval/src/postgres/traverse.rs Fixed
@indietyp indietyp force-pushed the bm/be-306-hashql-postgres-translation branch from 33a9dcc to 8c07a05 Compare March 8, 2026 16:15
@indietyp indietyp force-pushed the bm/be-457-hashql-mir-execution-pipeline-extensions-for-postgres branch from 5d95ba7 to 1aa0f1c Compare March 8, 2026 16:15
@codecov
Copy link
Copy Markdown

codecov Bot commented Mar 8, 2026

Codecov Report

❌ Patch coverage is 94.73985% with 92 lines in your changes missing coverage. Please review.
✅ Project coverage is 63.09%. Comparing base (60da845) to head (b399240).

Files with missing lines Patch % Lines
libs/@local/hashql/eval/src/postgres/filter/mod.rs 89.30% 43 Missing and 5 partials ⚠️
libs/@local/hashql/eval/src/postgres/error.rs 0.00% 22 Missing ⚠️
...bs/@local/hashql/eval/src/postgres/filter/tests.rs 98.72% 4 Missing and 2 partials ⚠️
libs/@local/hashql/eval/src/postgres/mod.rs 96.95% 4 Missing and 1 partial ⚠️
libs/@local/hashql/eval/src/postgres/traverse.rs 96.00% 5 Missing ⚠️
libs/@local/hashql/eval/src/context.rs 95.55% 1 Missing and 1 partial ⚠️
libs/@local/hashql/eval/src/postgres/parameters.rs 98.26% 2 Missing ⚠️
...src/store/postgres/query/expression/conditional.rs 0.00% 1 Missing ⚠️
libs/@local/hashql/eval/src/error.rs 0.00% 1 Missing ⚠️
Additional details and impacted files
@@                                         Coverage Diff                                          @@
##           bm/be-457-hashql-mir-execution-pipeline-extensions-for-postgres    #8526       +/-   ##
====================================================================================================
- Coverage                                                            85.26%   63.09%   -22.18%     
====================================================================================================
  Files                                                                  349     1250      +901     
  Lines                                                                55110   132490    +77380     
  Branches                                                              1339     5409     +4070     
====================================================================================================
+ Hits                                                                 46991    83594    +36603     
- Misses                                                                7672    48030    +40358     
- Partials                                                               447      866      +419     
Flag Coverage Δ
apps.hash-ai-worker-ts 1.41% <ø> (?)
apps.hash-api 0.00% <ø> (?)
blockprotocol.type-system 40.84% <ø> (?)
local.claude-hooks 0.00% <ø> (?)
local.harpc-client 51.24% <ø> (?)
local.hash-backend-utils 0.00% <ø> (?)
local.hash-graph-sdk 9.63% <ø> (?)
local.hash-isomorphic-utils 0.00% <ø> (?)
rust.antsi 0.00% <ø> (?)
rust.error-stack 90.87% <ø> (?)
rust.harpc-codec 84.70% <ø> (?)
rust.harpc-net 96.18% <ø> (?)
rust.harpc-tower 67.03% <ø> (?)
rust.harpc-types 0.00% <ø> (?)
rust.harpc-wire-protocol 92.23% <ø> (?)
rust.hash-codec 72.76% <ø> (?)
rust.hash-graph-api 2.52% <ø> (?)
rust.hash-graph-authorization 62.34% <ø> (?)
rust.hash-graph-postgres-store 26.81% <0.00%> (?)
rust.hash-graph-store 37.76% <ø> (?)
rust.hash-graph-temporal-versioning 47.95% <ø> (?)
rust.hash-graph-types 0.00% <ø> (?)
rust.hash-graph-validation 83.45% <ø> (?)
rust.hashql-ast 87.23% <ø> (?)
rust.hashql-compiletest 29.63% <ø> (ø)
rust.hashql-core 82.44% <ø> (+0.08%) ⬆️
rust.hashql-diagnostics 72.43% <ø> (?)
rust.hashql-eval 81.08% <94.79%> (?)
rust.hashql-hir 89.06% <ø> (?)
rust.hashql-mir 92.08% <ø> (ø)
rust.hashql-syntax-jexpr 94.05% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Comment thread libs/@local/hashql/eval/src/postgres/filter/mod.rs
Comment thread libs/@local/hashql/eval/src/postgres/filter/tests.rs Outdated
Copy link
Copy Markdown
Contributor

@github-advanced-security github-advanced-security AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clippy found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.

indietyp

This comment was marked as duplicate.

@indietyp indietyp force-pushed the bm/be-306-hashql-postgres-translation branch from 02e5f27 to 8b510e5 Compare April 29, 2026 13:37
@indietyp indietyp force-pushed the bm/be-457-hashql-mir-execution-pipeline-extensions-for-postgres branch from a675701 to 3050d4a Compare April 29, 2026 13:37
Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 2 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 8b510e5. Configure here.

Comment thread libs/@local/hashql/eval/src/postgres/filter/mod.rs
ColumnName::from(Identifier::from("b")),
ColumnName::from(Identifier::from("v")),
],
});
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

UNNEST lateral flag contradicts required LATERAL semantics

Medium Severity

The FromItem::Function for the UNNEST call sets lateral: false, but the function body references columns from the preceding "eit" table alias (eit.base_urls, eit.versions). In PostgreSQL, explicit CROSS JOIN requires the LATERAL keyword for the right-hand side to reference columns from the left. The doc comment on line 185 correctly states CROSS JOIN LATERAL UNNEST(...), but the code passes lateral: false, which would produce invalid SQL when the entity type IDs path is accessed.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 8b510e5. Configure here.

@indietyp indietyp force-pushed the bm/be-306-hashql-postgres-translation branch from 8b510e5 to c484cc9 Compare April 29, 2026 15:51
@indietyp indietyp force-pushed the bm/be-457-hashql-mir-execution-pipeline-extensions-for-postgres branch from 3050d4a to 5552cbb Compare April 29, 2026 15:51
@indietyp indietyp force-pushed the bm/be-306-hashql-postgres-translation branch from c484cc9 to b399240 Compare May 4, 2026 11:34
@indietyp indietyp force-pushed the bm/be-457-hashql-mir-execution-pipeline-extensions-for-postgres branch from 5552cbb to 60da845 Compare May 4, 2026 11:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/apps > hash* Affects HASH (a `hash-*` app) area/apps > hash-graph area/apps area/deps Relates to third-party dependencies (area) area/libs Relates to first-party libraries/crates/packages (area) area/tests New or updated tests type/eng > backend Owned by the @backend team

Development

Successfully merging this pull request may close these issues.

3 participants