Skip to content

feat: add explicit ToplingDB runtime selection - #3275

Open
imbajin wants to merge 33 commits into
apache:masterfrom
hugegraph:task/topling-core-20261005
Open

imbajin wants to merge 33 commits into
apache:masterfrom
hugegraph:task/topling-core-20261005

Conversation

@imbajin

@imbajin imbajin commented Oct 6, 2026 •

Copy link
Copy Markdown
Member

Purpose of the PR

Make ToplingDB an optional storage engine while preserving HugeGraph APIs. Standard RocksDB remains the default.

How to switch

Switch RocksDB to ToplingDB

  1. Run bash bin/prepare-topling.sh following the setup guide.
  2. Set rocksdb.provider=topling in the standalone graph properties and export TOPLINGDB_ROCKSDB_PROVIDER=topling.
  3. Initialize the fresh graph with bash bin/init-store.sh, then run bash bin/start-hugegraph.sh.

PD and Store use their own configuration and start scripts. ToplingDB offers performance-focused storage and flexible configuration.

Original preparation reference Original preparation reference

Main Changes

  • Prepare and select a component-local ToplingDB runtime before startup.
  • Preserve cache lifetime and release request, native and service resources.

Verifying these changes

  • Trivial rework / code cleanup without any test coverage. (No Need)
  • Covered by existing tests and focused regression tests.
  • Java 17 compilation and installation with Maven 3.10; relevant runtime and lifecycle checks.

Current CI results are shown in the checks below.

Does this PR potentially affect the following parts?

  • Dependencies
  • Modify configurations
  • The public API
  • Other affects: optional native runtime and resource lifecycle
  • Nope

Documentation Status

  • Doc - TODO
  • Doc - Done: repository guide and website PR #510; coordinate both merges.
  • Doc - No Need

@codecov

codecov Bot commented Oct 6, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 57.09970% with 142 lines in your changes missing coverage. Please review.
✅ Project coverage is 41.30%. Comparing base (7398075) to head (bf666f8).
⚠️ Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
...a/org/apache/hugegraph/store/HeartbeatService.java 55.73% 23 Missing and 4 partials ⚠️
...gegraph/backend/store/rocksdb/RocksDBSessions.java 4.54% 21 Missing ⚠️
.../hugegraph/backend/store/rocksdb/RocksDBStore.java 36.36% 20 Missing and 1 partial ⚠️
...he/hugegraph/backend/tx/GraphIndexTransaction.java 45.00% 7 Missing and 4 partials ⚠️
...a/org/apache/hugegraph/pd/TaskScheduleService.java 0.00% 9 Missing ⚠️
.../org/apache/hugegraph/pd/meta/MetadataFactory.java 58.82% 6 Missing and 1 partial ⚠️
...rc/main/java/org/apache/hugegraph/HugeFactory.java 61.11% 6 Missing and 1 partial ⚠️
...graph/backend/cache/CachedSchemaTransactionV2.java 77.41% 2 Missing and 5 partials ⚠️
...ugegraph/backend/cache/CachedGraphTransaction.java 70.00% 3 Missing and 3 partials ⚠️
...raph/backend/store/rocksdb/RocksDBStdSessions.java 54.54% 5 Missing ⚠️
... and 9 more
Additional details and impacted files
@@             Coverage Diff              @@
##             master    #3275      +/-   ##
============================================
+ Coverage     41.18%   41.30%   +0.11%     
- Complexity     6771     6815      +44     
============================================
  Files           766      766              
  Lines         66086    66317     +231     
  Branches       8773     8806      +33     
============================================
+ Hits          27218    27390     +172     
- Misses        35819    35855      +36     
- Partials       3049     3072      +23     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@imbajin
imbajin force-pushed the task/topling-core-20261005 branch from 40994eb to 7ee8766 Compare October 6, 2026 10:26
MrJs133
MrJs133 previously approved these changes Oct 6, 2026
imbajin added 21 commits October 6, 2026 21:39
- prepare trusted JNI in each component runtime directory
- match configured provider to the loaded runtime
- preserve Server column families when clearing TP data
- join PD native owners before metadata release
- document selection and verify standard startup isolation
- return request and task backend leases on their thread
- let the Spring engine own Store database shutdown
- stop and join heartbeat producers and state callbacks
- retain existing Store scan and TTL behavior
- verify commit rollback reuse and shutdown ordering
- specify persistent data WAL and Raft locations
- explain request and task transaction cleanup
- quote the trusted checksum placeholder safely
- remove the trailing separator rejected by TP JNI
- preserve the same physical metadata directory
- verify metadata ownership and standard reopen behavior
- include the newly added runtime utility in the count
- preserve the full scan and class ordering assertions
- verify the complete Commons and RPC unit suites
- move formatting policy to independent PR 268
- restore the master contribution guide
- keep the storage feature review focused
- prepend the selected JNI before inherited preloads
- fall back only when the context loader misses
- preserve provider and linkage failure checks
- cover loader isolation and preload transitions
- scan hidden system relations during deletion
- refresh cached schemas before index updates
- retain normal and OLAP transaction behavior
- cover cleanup regressions on Java 17
- clarify the Java 17 setup requirement
- report missing runtime origins with a clear error
- clear dynamic OLAP tables in their owning database
- cover missing origins and two-database cleanup
- verify EasyMigrate effects before preparing JNI
- bind prepared libraries and reject competing JNI
- coordinate fatal heartbeat shutdown outside workers
- return metrics sessions on success and failure
- check default and explicit providers before returning
- preserve unrelated preloads and selected runtime cleanup
- support empty preload arrays under nounset
- cover provider modes and document launch constraints
- close write batch and options at final detach
- retain owners while another lease is active
- guarantee both disposal attempts and closed state
- cover real JNI owners across repeated worker requests
- select supported auth fixtures by actual graph backend
- run cleanup assertions without a backend system property
- give the OLAP sentinel fixture a valid vertex ID
- assert the intended schema rejection source
- retain schema identity across request cleanup
- keep cache invalidation listeners until graph close
- dispose only the matching graph listener generation
- cover schema append and API update regressions
- validate the preload asset alongside launcher helpers
- include the dependency in the temporary server fixture
- retain security and Java 17 startup contract assertions
- set final runtime directories to 0755
- set regular assets to 0644 without following symlinks
- document service-user and parent access behavior
- route recovery exit outside the state callback lock
- preserve void API and abort invalid-shard restoration
- reuse the single exit gate and producer cleanup
- cover real node shutdown in an isolated JVM
- avoid compile-time imports from the executable Node jar
- use actual reactor classes in the recovery child JVM
- invoke real Node destroy and retain hook assertions
- detach owned store listeners without a cache holder
- preserve reopened listener generation ownership
- clear auth projects using the actual graph backend
- retain reopen failures and clean temporary data
- show trusted JNI preparation and provider selection
- highlight initialize and start with explicit runtime
- distinguish standalone and distributed entry points
- collapse premature wraps within 120 columns
- align multiline calls and native test fixtures
- preserve every Java token and string literal
- limit formatting to changed pull request blocks
- replace the old graphic with the approved A style
- focus on provider selection and user benefits
- show the switching guide in the README
- keep the original graphic as a PR attachment
- derive the Maven revision from the root POM
- preserve version context in Server and PD/Store CI
- cover macOS builds and reject unsupported Maven
- accept forward base updates for the same PR head
- bind selection and reports to the tested merge base
- reject unrelated bases and changed PR identities
- cover queued events with real Git regression tests
- use one Maven version for hosted build workflows
- verify the official archive before executing Maven
- cover Server, PD/Store, Cluster and CodeQL builds
- remove the ineffective revision injection helper

@bitflicker64 bitflicker64 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: yes. Summary: The new current-schema lookup in GraphIndexTransaction throws for vertices and edges whose label was deleted. Those left records can no longer be dropped, and a normal vertex with a left edge can no longer be dropped either. Both deletes worked before this PR. I found nothing else actionable in the launcher selection, the RocksDB provider check, the Topling truncate path, the cache lifetime change or the PD/Store shutdown rework. CI on this head is red, and both failures look unrelated: RateLimiterTest timing in server_memory, and dependency-review flagging the existing netty-handler 4.1.52 in hg-pd-dist. Evidence: static trace of StandardHugeGraph.vertexLabelOrNone/edgeLabelOrNone -> VertexLabel/EdgeLabel.undefined(graph, id); SchemaTransaction.getVertexLabel/getEdgeLabel return null for a missing id; GraphTransaction.removeVertex and prepareDeletions -> updateVertexIndex/updateEdgeIndex -> GraphIndexTransaction.indexLabelIds -> currentSchemaLabel E.checkArgument; GraphTransaction.warnLeftRecord shows these records are returned to callers.

- start the first timer reset after a full window
- retain the public limiter constructor and period
- verify window boundaries with a controlled timer
- use patched Netty and compatible gRPC runtimes
- align Protobuf runtime while retaining generators
- preserve default and unshaded RPC compatibility
- update resolved dependency and license materials
- resolve dependency and license merge conflicts
- retain the ToplingDB and rate limiter fixes
- align secure gRPC and Protobuf runtimes
- preserve upstream TinkerPop and security updates
- resolve shared type and dependency merge conflicts
- preserve current schema lookup for cached elements
- retain the synthetic OLAP schema path
- keep ToplingDB startup and secure runtime fixes
- record packaged Lombok and layertools jars
- retain existing runtime versions and licensing
- validate fail-closed dependency checks
- remove the extra Maven version pin
- reduce README switching guidance to one link
- retain the approved image under a clear filename
- use canonical ASF project links
- allow deletion of records with missing schema labels
- keep current schema lookup for existing labels
- reuse native write batches when resetting sessions
- cover orphan cleanup and repeated native resets
- retain V2 schema caches and listeners until graph close
- protect reopened graphs from stale transaction leases
- preserve inherited preloads when adding jemalloc
- cover cache lifetime and complete Store launch paths
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants