Skip to content

BE-456: HashQL: Entity type model enrichment and graph store query extensions#8524

Open
indietyp wants to merge 7 commits intographite-base/8524from
bm/be-456-hashql-entity-type-model-enrichment-and-graph-store-query
Open

BE-456: HashQL: Entity type model enrichment and graph store query extensions#8524
indietyp wants to merge 7 commits intographite-base/8524from
bm/be-456-hashql-entity-type-model-enrichment-and-graph-store-query

Conversation

@indietyp
Copy link
Copy Markdown
Member

@indietyp indietyp commented Mar 8, 2026

🌟 What is the purpose of this PR?

Enriches the HashQL entity type model to match the full database schema, extends the graph-store SQL query builder with new expression types needed by the postgres compiler, and adds a database migration for the continuation composite type. This is the foundation layer for the postgres translation stack (BE-457, BE-306).

🔍 What does this change?

hashql-core:

  • Adds IdSnapshotVec, a typed-ID-indexed vector with transactional snapshot/rollback support (used by the postgres compiler for scoped state)
  • Expands the entity type definition with TemporalInterval, EntityTemporalMetadata, Confidence, provenance types, EntityMetadata, and LinkData fields to match the full database schema. Previously only EntityId, EntityRecordId, and partial LinkData were modeled.
  • Adds edition_id and ontology_type_ids symbols

graph-postgres-store query builder:

  • New expression types: FieldAccess (composite field extraction), JsonAgg, Constant::Null, Constant::U128, UnaryOperator::IsNotFalse
  • New cast targets: JsonB, Continuation, Numeric, Int, BigInt
  • Unnest now accepts multiple expressions (UNNEST(a, b, c))
  • SelectStatement gains OFFSET support
  • Convenience methods: .cast(), .not() and .grouped() now take self
  • Alias type changed from &'static str to Identifier for consistency
  • Visibility: mod postgres and mod table made pub for use by the HashQL postgres compiler
  • Unit tests for all new expression types

Database migration (v010__query):

  • Creates the continuation composite type: (filter boolean, block int, locals int[], values jsonb[]). This is the row type returned by compiled filter subqueries; the interpreter reads .filter to decide row inclusion and .block/.locals/.values to resume multi-block CFGs.

hashql-eval:

  • Reworks graph/read/path.rs to handle the expanded entity field structure
  • Scaffolding in lib.rs and error.rs for the postgres module (filled in by BE-306)

Test output updates:

  • ~55 files of updated .jsonc/.stdout/.stderr fixtures across eval (graph/read), HIR (lower/checking, specialization, graph-hoisting), and MIR (reify, inline) test suites, all cascading from the entity type changes

Pre-Merge Checklist 🚀

🚢 Has this modified a publishable library?

This PR:

  • does not modify any publishable blocks or libraries, or modifications do not need publishing

📜 Does this require a change to the docs?

The changes in this PR:

  • are internal and do not require a docs change

🕸️ Does this require a change to the Turbo Graph?

The changes in this PR:

  • do not affect the execution graph

⚠️ Known issues

The entity type model still does not cover every field in the database schema (e.g., property_metadata inner structure is opaque). Fields are added as needed by the postgres compiler; remaining gaps will be filled incrementally.

🛡 What tests cover this?

  • IdSnapshotVec has doc tests and inline unit tests covering push/set/rollback/nested snapshots
  • New graph-store query builder tests for FieldAccess, JsonAgg, Null, U128, IsNotFalse, UNNEST with multiple args, cast types, and OFFSET
  • Existing compiletest suites (eval graph/read, HIR lowering, MIR reify/inline) with blessed output updates

❓ How to test this?

cargo nextest run -p hashql-core
cargo nextest run -p hashql-eval
cargo nextest run -p hash-graph-postgres-store --lib
cargo run --package hashql-compiletest -- run

indietyp added 3 commits March 8, 2026 16:17
feat: checkpoint (II)

feat: checkpoint (III)

feat: snapshot vec

feat: add dedicated filter

feat: checkpoint

feat: filter implementation

feat: filter implementation (mostly) done

chore: environment capture note

chore: always postgres bigint

feat: target clone

feat: simplify lookup

feat: move storage up

feat: eval entity path

chore: checkpoint

chore: checkpoint

chore: find entrypoint

feat: eval context

feat: eval cleanup

chore: cleanup

feat: track index

feat: wire up filter

feat: add error reporting

chore: checkpoint

feat: add traverse, and first postgres compiler outline

feat: traverse bitmap

feat: move traversal out

feat: projections

feat: projections

fix: clippy

feat: subquery projection for lateral

feat: checkpoint

feat: test plan

feat: checkpoint

feat: checkpoint – failing tests ;-;

feat: checkpoint – failing tests ;-;

feat: checkpoint — passing tests

fix: import

fix: entity type

feat: checkpoint

feat: attribute a cost to terminator placement switches

fix: import

feat: checkpoint

feat: checkpoint

chore: lint
@indietyp indietyp requested a review from a team as a code owner March 8, 2026 15:46
@vercel
Copy link
Copy Markdown

vercel Bot commented Mar 8, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
hash Ready Ready Preview, Comment May 4, 2026 11:44am
petrinaut Ready Ready Preview May 4, 2026 11:44am
2 Skipped Deployments
Project Deployment Actions Updated (UTC)
hashdotdesign Ignored Ignored Preview May 4, 2026 11:44am
hashdotdesign-tokens Ignored Ignored Preview May 4, 2026 11:44am

@cursor
Copy link
Copy Markdown

cursor Bot commented Mar 8, 2026

PR Summary

Medium Risk
Medium risk: this changes the Postgres SQL query builder/AST (new expression forms, aliasing, OFFSET, UNNEST semantics) and adds a new database composite type, which could impact query generation and migrations if any downstream assumptions differ.

Overview
Adds a new continuation Postgres composite type via migrations (and a store migration) to support query continuation state.

Extends the graph Postgres query builder with new SQL expression capabilities (e.g. composite FieldAccess, jsonb_agg, NULL/u128 constants, IS NOT FALSE, expanded casts including continuation, multi-arg UNNEST, and SELECT ... OFFSET), updates alias handling to use Identifier, and makes postgres/table modules public for compiler integration.

Enriches HashQL’s entity type model with metadata/temporal/provenance/confidence/encodings fields and updates graph read path resolution plus golden tests to match the new entity.metadata.record_id... access pattern; introduces IdSnapshotVec with snapshot/rollback support for compiler state management.

Reviewed by Cursor Bugbot for commit 1a97257. Bugbot is set up for automated code reviews on this repo. Configure here.

Copy link
Copy Markdown
Member Author

indietyp commented Mar 8, 2026

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

This stack of pull requests is managed by Graphite. Learn more about stacking.

@indietyp indietyp changed the title HashQL: Entity type model enrichment and graph store query extensions BE-456: HashQL: Entity type model enrichment and graph store query extensions Mar 8, 2026
Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 2 potential issues.

Autofix Details

Bugbot Autofix prepared fixes for both issues found in the latest run.

  • ✅ Fixed: Misleading enum variant name for jsonb_agg function
    • Renamed Function::JsonAgg to Function::JsonbAgg and updated all internal usages/tests to match the emitted jsonb_agg(...) SQL.
  • ✅ Fixed: Multiple new symbols defined but never referenced
    • Removed the eight unreferenced symbol definitions from sym.rs to eliminate dead entries and duplicate provenance naming.

Create PR

Or push these changes by commenting:

@cursor push 928f4a2c0b
Preview (928f4a2c0b)
diff --git a/libs/@local/graph/postgres-store/src/store/postgres/query/expression/conditional.rs b/libs/@local/graph/postgres-store/src/store/postgres/query/expression/conditional.rs
--- a/libs/@local/graph/postgres-store/src/store/postgres/query/expression/conditional.rs
+++ b/libs/@local/graph/postgres-store/src/store/postgres/query/expression/conditional.rs
@@ -17,7 +17,7 @@
 pub enum Function {
     Min(Box<Expression>),
     Max(Box<Expression>),
-    JsonAgg(Box<Expression>),
+    JsonbAgg(Box<Expression>),
     JsonExtractText(Box<Expression>),
     JsonExtractAsText(Box<Expression>, PathToken<'static>),
     JsonExtractPath(Vec<Expression>),
@@ -55,7 +55,7 @@
                 expression.transpile(fmt)?;
                 fmt.write_char(')')
             }
-            Self::JsonAgg(expression) => {
+            Self::JsonbAgg(expression) => {
                 fmt.write_str("jsonb_agg(")?;
                 expression.transpile(fmt)?;
                 fmt.write_char(')')
@@ -821,9 +821,9 @@
     }
 
     #[test]
-    fn transpile_json_agg() {
+    fn transpile_jsonb_agg() {
         assert_eq!(
-            Expression::Function(Function::JsonAgg(Box::new(Expression::Parameter(1))))
+            Expression::Function(Function::JsonbAgg(Box::new(Expression::Parameter(1))))
                 .transpile_to_string(),
             "jsonb_agg($1)"
         );

diff --git a/libs/@local/hashql/core/src/symbol/sym.rs b/libs/@local/hashql/core/src/symbol/sym.rs
--- a/libs/@local/hashql/core/src/symbol/sym.rs
+++ b/libs/@local/hashql/core/src/symbol/sym.rs
@@ -36,7 +36,6 @@
     entity,
     entity_edition_id,
     entity_id,
-    entity_type_ids,
     entity_uuid,
     eq,
     Err,
@@ -55,8 +54,6 @@
     left_entity_confidence,
     left_entity_id,
     left_entity_provenance,
-    left_entity_uuid,
-    left_entity_web_id,
     link_data,
     List,
     lt,
@@ -76,10 +73,7 @@
     or,
     pow,
     properties,
-    property_metadata,
     provenance,
-    provenance_edition,
-    provenance_inferred,
     provided,
     r#as: "as",
     r#as_force: "as!",
@@ -101,8 +95,6 @@
     right_entity_confidence,
     right_entity_id,
     right_entity_provenance,
-    right_entity_uuid,
-    right_entity_web_id,
     Some,
     special_form,
     String,
This Bugbot Autofix run was free. To enable autofix for future PRs, go to the Cursor dashboard.

Comment thread libs/@local/hashql/core/src/symbol/sym.rs
@codecov
Copy link
Copy Markdown

codecov Bot commented Mar 8, 2026

Codecov Report

❌ Patch coverage is 66.17954% with 162 lines in your changes missing coverage. Please review.
✅ Project coverage is 60.50%. Comparing base (4503abc) to head (1a97257).

Files with missing lines Patch % Lines
libs/@local/hashql/eval/src/graph/read/path.rs 13.54% 134 Missing ⚠️
libs/@local/hashql/core/src/id/snapshot_vec.rs 79.83% 25 Missing ⚠️
...store/src/store/postgres/query/statement/insert.rs 0.00% 2 Missing ⚠️
...src/store/postgres/query/expression/conditional.rs 98.85% 1 Missing ⚠️
Additional details and impacted files
@@                  Coverage Diff                   @@
##           graphite-base/8524    #8524      +/-   ##
======================================================
- Coverage               67.55%   60.50%   -7.05%     
======================================================
  Files                     897      970      +73     
  Lines                   82970   100617   +17647     
  Branches                 4511     4649     +138     
======================================================
+ Hits                    56049    60880    +4831     
- Misses                  26347    39120   +12773     
- Partials                  574      617      +43     
Flag Coverage Δ
apps.hash-ai-worker-ts 1.41% <ø> (ø)
apps.hash-api 0.00% <ø> (ø)
local.hash-backend-utils 0.00% <ø> (ø)
local.hash-graph-sdk 9.63% <ø> (ø)
local.hash-isomorphic-utils 0.00% <ø> (ø)
rust.hash-graph-api 2.52% <ø> (ø)
rust.hash-graph-postgres-store 26.81% <97.63%> (?)
rust.hashql-ast 87.23% <ø> (ø)
rust.hashql-compiletest 29.63% <ø> (ø)
rust.hashql-core 82.44% <87.30%> (+0.05%) ⬆️
rust.hashql-eval 62.99% <13.54%> (-6.15%) ⬇️
rust.hashql-hir 89.06% <ø> (ø)
rust.hashql-mir 92.45% <ø> (ø)
rust.hashql-syntax-jexpr 94.05% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@augmentcode
Copy link
Copy Markdown

augmentcode Bot commented Mar 8, 2026

🤖 Augment PR Summary

Summary: This PR enriches the HashQL entity type model to better reflect the full graph database schema, and extends the Postgres query-builder/AST to support additional SQL constructs needed by the HashQL→Postgres compiler.

Changes:

  • Adds a new DB migration creating a continuation composite type used as the return row type for compiled filter subqueries.
  • Introduces IdSnapshotVec, a typed-ID-indexed vector with snapshot/rollback support for scoped compiler state.
  • Expands ::graph::types::knowledge::entity with temporal metadata, confidence/provenance, enriched link data, entity metadata, and encodings.
  • Extends the graph Postgres query builder with composite field access, jsonb_agg, additional constants/casts, multi-arg UNNEST, and OFFSET.
  • Updates HashQL eval graph path resolution to traverse the new entity field structure.
  • Blesses updated UI/compiletest fixtures across eval/HIR/MIR to reflect the new entity model.

Technical Notes: Query builder aliasing is standardized on an Identifier type; new expression/unit tests cover the added SQL features.

🤖 Was this summary useful? React with 👍 or 👎

Copy link
Copy Markdown

@augmentcode augmentcode Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review completed. 3 suggestions posted.

Fix All in Augment

Comment augment review to trigger a new review at any time.

Comment thread libs/@local/hashql/eval/src/graph/read/path.rs
Comment thread libs/@local/graph/migrations/graph-migrations/v010__query/down.sql Outdated
Comment thread libs/@local/graph/postgres-store/src/store/mod.rs
@codspeed-hq
Copy link
Copy Markdown

codspeed-hq Bot commented Mar 8, 2026

Merging this PR will not alter performance

✅ 80 untouched benchmarks


Comparing bm/be-456-hashql-entity-type-model-enrichment-and-graph-store-query (1a97257) with bm/be-429-hashql-per-island-entitypath-requirements-and-fetchisland (4503abc)

Open in CodSpeed

@indietyp indietyp force-pushed the bm/be-429-hashql-per-island-entitypath-requirements-and-fetchisland branch from 69a8667 to 40ba97b Compare April 29, 2026 13:08
@indietyp indietyp force-pushed the bm/be-456-hashql-entity-type-model-enrichment-and-graph-store-query branch from bc50eb0 to 8b6ee87 Compare April 29, 2026 13:08
@indietyp indietyp force-pushed the bm/be-429-hashql-per-island-entitypath-requirements-and-fetchisland branch from 40ba97b to a4534ce Compare April 29, 2026 13:37
@indietyp indietyp force-pushed the bm/be-456-hashql-entity-type-model-enrichment-and-graph-store-query branch from 8b6ee87 to 687e4f1 Compare April 29, 2026 13:37
@vercel vercel Bot temporarily deployed to Preview – petrinaut April 29, 2026 15:51 Inactive
@indietyp indietyp force-pushed the bm/be-429-hashql-per-island-entitypath-requirements-and-fetchisland branch from a4534ce to 4503abc Compare May 4, 2026 11:34
@indietyp indietyp force-pushed the bm/be-456-hashql-entity-type-model-enrichment-and-graph-store-query branch from b9e2eb9 to 1a97257 Compare May 4, 2026 11:34
@graphite-app graphite-app Bot changed the base branch from bm/be-429-hashql-per-island-entitypath-requirements-and-fetchisland to graphite-base/8524 May 7, 2026 09:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/libs Relates to first-party libraries/crates/packages (area) area/tests New or updated tests type/eng > backend Owned by the @backend team

Development

Successfully merging this pull request may close these issues.

3 participants