diff --git a/.github/outdated/APACHE_PROPOSAL.md b/.github/outdated/APACHE_PROPOSAL.md index bc3bb84044..b26681d2c1 100644 --- a/.github/outdated/APACHE_PROPOSAL.md +++ b/.github/outdated/APACHE_PROPOSAL.md @@ -34,7 +34,7 @@ Although most of the main development of HugeGraph has been completed, there are #### Meritocracy -HugeGraph was incubated at Baidu in 2016 and open-sourced on [GitHub](https://github.com/hugegraph/hugegraph) in 2018. The project (including sub-projects) now has 30+ contributors from many companies. Some of these contributors become committers, and the project has hundreds of known users around the world. We will follow Apache's Meritocracy way to re-organize the community roles. We have set up the PPMC Team and Committer Team. Of course, contributions are welcomed and highly valued. New contributors are guided and reviewed by existing PMC members. When an active contributor has submitted enough good patches, PMC will start a vote to promote him/her to become a member of the Committer Team or PMC Team. +HugeGraph was incubated at Baidu in 2016 and open-sourced on [GitHub](https://github.com/apache/hugegraph) in 2018. The project (including sub-projects) now has 30+ contributors from many companies. Some of these contributors become committers, and the project has hundreds of known users around the world. We will follow Apache's Meritocracy way to re-organize the community roles. We have set up the PPMC Team and Committer Team. Of course, contributions are welcomed and highly valued. New contributors are guided and reviewed by existing PMC members. When an active contributor has submitted enough good patches, PMC will start a vote to promote him/her to become a member of the Committer Team or PMC Team. #### Community @@ -52,7 +52,7 @@ HugeGraph implements the API of [Apache TinkerPop](https://tinkerpop.apache.org/ #### Project Name -We have checked and believe the name is [suitable](https://github.com/hugegraph/hugegraph/issues/1646) and the project has legal permission to continue using its current name. There are no other projects found using this name through Google Search. +We have checked and believe the name is [suitable](https://github.com/apache/hugegraph/issues/1646) and the project has legal permission to continue using its current name. There are no other projects found using this name through Google Search. #### Relationship with Titan/Janus Graph @@ -94,10 +94,10 @@ HugeGraph documentation is provided on https://hugegraph.github.io/hugegraph-doc This project consists of 2 core sub-projects and 2 other sub-projects, all of which are hosted by [GitHub hugegraph organization](https://github.com/orgs/hugegraph/repositories) since 2018. The codes are already under Apache License Version 2.0. The git address of sub-project repositories are as follows: -1. The graph database repository `hugegraph`, core sub-project, including graph server, graph engine, and graph storage: https://github.com/hugegraph/hugegraph -2. The graph computing repository `hugegraph-computer`, core sub-project, including graph computing and graph algorithms: https://github.com/hugegraph/hugegraph-computer -3. The common functions repository `hugegraph-commons`: https://github.com/hugegraph/hugegraph-commons -4. The ecosystem repository `hugegraph-toolchain`, including `hugegraph-client`, `hugegraph-loader`, `hugegraph-tools`, `hugegraph-hubble`, `hugegraph-test`, `hugegraph-doc`: https://github.com/hugegraph/hugegraph-toolchain +1. The graph database repository `hugegraph`, core sub-project, including graph server, graph engine, and graph storage: https://github.com/apache/hugegraph +2. The graph computing repository `hugegraph-computer`, core sub-project, including graph computing and graph algorithms: https://github.com/apache/hugegraph-computer +3. The common functions repository `hugegraph-commons`: https://github.com/apache/hugegraph/tree/master/hugegraph-commons +4. The ecosystem repository `hugegraph-toolchain`, including `hugegraph-client`, `hugegraph-loader`, `hugegraph-tools`, `hugegraph-hubble`, `hugegraph-test`, `hugegraph-doc`: https://github.com/apache/hugegraph-toolchain ### Source and Intellectual Property Submission Plan @@ -107,7 +107,7 @@ The code is currently under Apache License Version 2.0, and it was verified ther As all dependencies are managed by Apache Maven, none of the external libraries need to be packaged in a source distribution. All dependencies have Apache compatible licenses except for 4 dependencies: MySQL Connector (GPL), word (GPL), JBoss Logging 3 (LGPL), and jnr-posix (LGPL+GPL), we will remove these dependencies in the future. -HugeGraph has the following external [dependencies](https://github.com/hugegraph/hugegraph/issues/1632): +HugeGraph has the following external [dependencies](https://github.com/apache/hugegraph/issues/1632): - Apache License @@ -263,7 +263,7 @@ None - hugegraph-dev: [dev@hugegraph.incubator.apache.org](mailto:dev@hugegraph.incubator.apache.org) for development and users discussions. - hugegraph-private: [private@hugegraph.incubator.apache.org](mailto:private@hugegraph.incubator.apache.org) for PPMC discussions. -- hugegraph-commits: [commits@hugegraph.incubator.apache.org](mailto:commits@hugegraph.incubator.apache.org) for commits / [pull requests](https://github.com/hugegraph/hugegraph/pulls) and other notifications like code review comments. +- hugegraph-commits: [commits@hugegraph.incubator.apache.org](mailto:commits@hugegraph.incubator.apache.org) for commits / [pull requests](https://github.com/apache/hugegraph/pulls) and other notifications like code review comments. #### Subversion Directory @@ -271,18 +271,18 @@ None #### Git Repositories -1. `hugegraph`: https://github.com/hugegraph/hugegraph.git -2. `hugegraph-computer`: https://github.com/hugegraph/hugegraph-computer.git -3. `hugegraph-commons`: https://github.com/hugegraph/hugegraph-commons.git -4. `hugegraph-toolchain`: https://github.com/hugegraph/hugegraph-toolchain.git +1. `hugegraph`: https://github.com/apache/hugegraph.git +2. `hugegraph-computer`: https://github.com/apache/hugegraph-computer.git +3. `hugegraph-commons`: https://github.com/apache/hugegraph.git +4. `hugegraph-toolchain`: https://github.com/apache/hugegraph-toolchain.git #### Issue Tracking -The community would like to continue using [GitHub Issues](https://github.com/hugegraph/hugegraph/issues) (but will move to github.com/apache/). +The community would like to continue using [GitHub Issues](https://github.com/apache/hugegraph/issues) (but will move to github.com/apache/). #### Other Resources -- The community has already chosen [GitHub actions](https://github.com/hugegraph/hugegraph/actions) as continuous integration tools. +- The community has already chosen [GitHub actions](https://github.com/apache/hugegraph/actions) as continuous integration tools. - The community has already used [codecov](https://github.com/marketplace/codecov) to check code coverage. - The community has already used [mvn repository](https://mvnrepository.com/search?q=hugegraph) as binary package release platform. diff --git a/.github/outdated/cla.yml b/.github/outdated/cla.yml index a182b9515b..d5779dcb37 100644 --- a/.github/outdated/cla.yml +++ b/.github/outdated/cla.yml @@ -18,8 +18,8 @@ jobs: PERSONAL_ACCESS_TOKEN : ${{ secrets.PERSONAL_ACCESS_TOKEN }} with: path-to-signatures: '.github/cla-signatures/cla.json' - path-to-document: 'https://github.com/hugegraph/hugegraph-doc/blob/master/CLA.md' # e.g. a CLA or a DCO document - custom-allsigned-prcomment: '**Good**! All Contributors have signed the [CLA](https://github.com/hugegraph/hugegraph-doc/blob/master/CLA.md).' + path-to-document: 'https://www.apache.org/licenses/contributor-agreements.html' # e.g. a CLA or a DCO document + custom-allsigned-prcomment: '**Good**! All Contributors have signed the [CLA](https://www.apache.org/licenses/contributor-agreements.html).' # branch should not be protected branch: 'cla-assist' allowlist: bot* diff --git a/.github/scripts/test_ci_policy.py b/.github/scripts/test_ci_policy.py index 435b62bb3e..437c3f584a 100644 --- a/.github/scripts/test_ci_policy.py +++ b/.github/scripts/test_ci_policy.py @@ -402,6 +402,66 @@ def fetch(path): policy.gate(plan, results) git("checkout", "-q", "feature") + def test_queued_pr_uses_actual_merge_base_and_rejects_changed_identity(self): + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + def git(*args): + return subprocess.check_output(["git", *args], cwd=root, text=True).strip() + git("init", "-q") + git("config", "user.email", "ci@example.invalid") + git("config", "user.name", "CI") + (root / "README.md").write_text("base") + git("add", ".") + git("commit", "-qm", "event base") + event_base = git("rev-parse", "HEAD") + git("checkout", "-qb", "feature") + (root / "hugegraph-server").mkdir() + (root / "hugegraph-server/A.java").write_text("PR source") + git("add", ".") + git("commit", "-qm", "PR head") + head = git("rev-parse", "HEAD") + event_live = self.live_pr() + event_live["head"]["sha"], event_live["base"]["sha"] = head, event_base + event = {"pull_request": dict(event_live, number=7)} + git("checkout", "--detach", "-q", event_base) + (root / "README.md").write_text("target advanced while run was queued") + git("commit", "-qam", "new target base") + tested_base = git("rev-parse", "HEAD") + git("merge", "--no-ff", "-qm", "current PR merge", "feature") + merge = git("rev-parse", "HEAD") + live = dict(event_live, base=dict(event_live["base"], sha=tested_base)) + old = os.getcwd() + try: + os.chdir(root) + os.environ["GITHUB_SHA"] = merge + plan = policy.create_plan("server", event, "apache/server", lambda _: live) + self.assertEqual(tested_base, plan["base"]) + self.assertNotEqual(event_base, plan["base"]) + self.assertEqual(head, plan["head"]) + self.assertEqual(merge, plan["testedMergeSHA"]) + self.assertEqual(["hugegraph-server/A.java"], plan["changedPaths"]) + self.assertIn("server_memory", plan["expected"]) + results = {suite: {"result": "success"} for suite in plan["expected"]} + results["plan"] = {"result": "success"} + self.assertEqual(plan["expected"], policy.gate(plan, results, lambda _: live)["executed"]) + changed = dict(live, head=dict(live["head"], sha=tested_base)) + with self.assertRaises(policy.StaleInputError): + policy.create_plan("server", event, "apache/server", lambda _: changed) + with self.assertRaises(policy.StaleInputError): + policy.gate(plan, results, lambda _: changed) + wrong_merge = git("commit-tree", git("rev-parse", merge + "^{tree}"), + "-p", tested_base, "-p", event_base, "-m", "wrong PR head") + unrelated_base = git("commit-tree", git("rev-parse", tested_base + "^{tree}"), + "-m", "unrelated target history") + unrelated_merge = git("commit-tree", git("rev-parse", merge + "^{tree}"), + "-p", unrelated_base, "-p", head, "-m", "unrelated target merge") + for checkout in [wrong_merge, head, unrelated_merge]: + git("checkout", "--detach", "-q", checkout) + with self.subTest(checkout=checkout), self.assertRaises(policy.StaleInputError): + policy.create_plan("server", event, "apache/server", lambda _: live) + finally: + os.chdir(old) + def test_pure_docs_pr_and_metadata_fail_closed(self): live = self.live_pr() event = {"pull_request": dict(live, number=7)} diff --git a/.github/workflows/pd-store-ci.yml b/.github/workflows/pd-store-ci.yml index 3a238fc4a6..231751e303 100644 --- a/.github/workflows/pd-store-ci.yml +++ b/.github/workflows/pd-store-ci.yml @@ -207,6 +207,11 @@ jobs: done echo "can_run=true" >> "$GITHUB_OUTPUT" + - name: Check explicit Topling launcher selection + run: | + bash $TRAVIS_DIR/test-topling-selection.sh + bash hugegraph-store/hg-store-dist/src/assembly/travis/test-store-preloads.sh + - name: Run PD docker entrypoint secret override tests run: | $TRAVIS_DIR/test-pd-docker-entrypoint.sh diff --git a/README.md b/README.md index 401229df69..fc83184daa 100644 --- a/README.md +++ b/README.md @@ -100,6 +100,8 @@ HugeGraph supports both **standalone** and **distributed** deployments: See the [backend evolution guide](hugegraph-server/README.md#backend-evolution-and-compatibility) for lifecycle and historical compatibility guidance. +To use ToplingDB with the RocksDB backend, follow the [three-step switching guide](docs/toplingdb.md). + ### Deployment Mode Comparison | Mode | Components | Use Case | Data Scale | High Availability | @@ -315,6 +317,7 @@ gremlin> :> g.V().limit(5) For comprehensive documentation, visit the [HugeGraph Documentation](https://hugegraph.apache.org/docs/). + For an existing deployment, read the [RocksDB upgrade guidance](docs/rocksdb-upgrade.md) before upgrading the storage runtime. diff --git a/docs/images/switch-rocksdb-to-toplingdb.png b/docs/images/switch-rocksdb-to-toplingdb.png new file mode 100644 index 0000000000..5f6d92c1cf Binary files /dev/null and b/docs/images/switch-rocksdb-to-toplingdb.png differ diff --git a/docs/toplingdb.md b/docs/toplingdb.md new file mode 100644 index 0000000000..2e12590c97 --- /dev/null +++ b/docs/toplingdb.md @@ -0,0 +1,149 @@ +# Switch RocksDB to ToplingDB + +HugeGraph defaults to standard RocksDB. Server, PD and Store can explicitly select a prepared ToplingDB runtime using their normal distributions. +Prepare and configure each component separately; this guide covers the shipped launch scripts, not custom IDE or embedded launchers. + +| Component | Business provider setting | Start command | +| --- | --- | --- | +| Server with the RocksDB backend | `rocksdb.provider=topling` in each graph properties file | `bin/start-hugegraph.sh` | +| PD | `rocksdb.provider: topling` in `conf/application.yml` | `bin/start-hugegraph-pd.sh` | +| Store | `rocksdb.provider: topling` in `conf/application-pd.yml` | `bin/start-hugegraph-store.sh` | + +For a distributed graph, Server uses `backend=hstore`; PD and Store own its persistent storage. A standalone Server keeps `backend=rocksdb`. +The launcher selection and the business provider must agree. A mismatch fails before opening that component's database. + +## Quick start: standalone Server + +![Switch RocksDB to ToplingDB](images/switch-rocksdb-to-toplingdb.png) + +Use a stopped Server distribution on Linux x86_64 with a Java 17 JDK. If your distribution lacks these scripts, [build it from this source](#build-and-prepare) first. +Prepare the runtime once, select it, then initialize and start the Server. +The graph APIs and normal start/stop commands stay the same; Topling's native options are configured in `conf/toplingdb.yaml`. +Use new, empty, service-writable data and WAL directories outside the distribution for this example. + +**1. Prepare the runtime.** Supply your ToplingDB JNI JAR and its trusted SHA-256: + +```bash +export TOPLING_JNI_JAR=/absolute/path/to/rocksdbjni-topling.jar +export TOPLING_JNI_SHA256='' +bash bin/prepare-topling.sh +``` + +**2. Select Topling.** Edit `conf/graphs/hugegraph.properties` and every other graph loaded from the configured graphs directory: + +```properties +backend=rocksdb +rocksdb.provider=topling +rocksdb.data_path=/srv/hugegraph/topling/server/data +rocksdb.wal_path=/srv/hugegraph/topling/server/wal +``` + +Create those external directories with permissions for the service account. The paths above are examples; choose persistent locations for your deployment. + +**3. Initialize and start.** From the same shell, select the matching JNI runtime and initialize the new store once: + +```bash +export TOPLINGDB_ROCKSDB_PROVIDER=topling +bash bin/init-store.sh +bash bin/start-hugegraph.sh +``` + +Keep `TOPLINGDB_ROCKSDB_PROVIDER=topling` in the service environment for subsequent starts; initialization is only needed for a new store. +For PD and Store, prepare each component separately, then use its YAML provider setting and start command from the table above. +A Server connected to distributed storage keeps `backend=hstore`; PD and Store select their own storage runtime. + +## Build and prepare + +Build the normal distributions from the repository root using Java 17 and Maven 3.6.3 or later: + +```bash +mvn clean package -Dmaven.test.skip=true -Dmaven.javadoc.skip=true +``` + +Obtain a trusted ToplingDB JNI JAR and its SHA-256 through your approved artifact channel. The runtime is not downloaded by the launcher. It must implement the `TOPLINGDB_EASY_MIGRATE_CONF` configuration entry point. +Preparation requires a Java 17 JDK, Linux x86_64, `sha256sum`, `unzip`, `od`, `ldd` and GNU `mv`. The currently exercised JNI requires glibc 2.38 or newer and libaio. +Native dependency requirements depend on the selected JAR; preparation rejects unresolved dependencies. + +In each stopped, unpacked component directory: + +```bash +export TOPLING_JNI_JAR=/absolute/path/to/rocksdbjni-topling.jar +export TOPLING_JNI_SHA256='' +bash bin/prepare-topling.sh +``` + +The script verifies the copied JAR's hash, the Topling Java marker, one Linux x86_64 JNI entry, the ELF architecture and native dependencies. +Before installation, it opens and closes a disposable database with the selected JNI and checks that a probe configuration changes the persisted write-buffer size. +A marker or matching hash alone is insufficient: incompatible builds, including the older `frocksdbjni-8.10.2-topling-1.0.jar` without EasyMigrate support, are rejected. +The probe validates the configuration entry point, not existing-data compatibility or every setting in your production profile. +Its checksum receipt binds the probe to the installed JAR and native library. Launchers reject missing or stale receipts; prepare a fresh distribution when upgrading from an older preparation script. +It installs `topling/rocksdbjni.jar`, its matching native library and optional web resources outside the standard `lib` directory. +On Ubuntu with libaio's t64 name, the compatibility symlink stays inside that component's runtime directory. +The published runtime directories use 0755 and regular files use 0644, so a different service account can read the prepared JNI. +That account also needs traversal permission on the component's parent directories; preparation does not change those parents. + +Preparation refuses an existing `topling` directory. To replace a runtime, stop the service and prepare a fresh distribution; it never changes database files. + +## Select and start + +Set the provider in the component's business configuration. For PD and Store, edit the existing `rocksdb` mapping, retaining its other options: + +```yaml +rocksdb: + provider: topling +``` + +For a standalone Server, edit every selected graph configuration: + +```properties +backend=rocksdb +rocksdb.provider=topling +``` + +Use separate, initially empty data directories for validation. This integration does not convert or certify an existing database for a different provider. + +For persistent use, create service-writable directories outside source checkouts, build outputs and unpacked distributions, and configure all applicable paths: + +| Component | Configuration file | Setting | Example external path | +| --- | --- | --- | --- | +| Server data | Graph properties | `rocksdb.data_path` | `/srv/hugegraph/topling/server/data` | +| Server WAL | Graph properties | `rocksdb.wal_path` | `/srv/hugegraph/topling/server/wal` | +| PD metadata | `conf/application.yml` | `pd.data-path` | `/srv/hugegraph/topling/pd` | +| Store data | `conf/application.yml` | `app.data-path` | `/srv/hugegraph/topling/store/data` | +| Store Raft | `conf/application.yml` | `app.raft-path` | `/srv/hugegraph/topling/store/raft` | + +Set both Server data and WAL paths. Retaining the default distribution-relative WAL path can lose that directory when rebuilding or replacing the distribution. +Preserve the external data, WAL and Raft roots across runtime updates. +Start each component with explicit runtime selection: + +```bash +export TOPLINGDB_ROCKSDB_PROVIDER=topling +# Run the start command for this component from the table above. +``` + +The component uses its own `conf/toplingdb.yaml`. An explicit `TOPLINGDB_EASY_MIGRATE_CONF=/absolute/path/config.yaml` overrides that native configuration. +The shipped profile disables the native HTTP server and keeps `memtable_as_log_index=false`, which is required by the Java write path. +Tune the native profile for your machine before production use; it does not replace the component's business configuration. + +Server's `init-store.sh` and `dump-store.sh` use the same explicit runtime selection. PD and Store put the prepared JNI JAR before their Spring Boot dependencies. +Standard launch remains the default and does not require Topling files or preparation tools. +Launch rejects inherited recognizable RocksDB JNI preloads in default/standard mode and competing JNI in Topling mode. +Unrelated allocator or tracing preloads remain intact; switching from a previously selected Topling runtime removes only that runtime. Store adds verified jemalloc without replacing or duplicating caller preloads; the selected JNI stays first. +Do not preload a second RocksDB JNI alongside the selected runtime. + +## Validate and stop + +Before adopting a JAR, validate each component's real writes, reads, normal stop and restart using a dedicated database directory. +Confirm both the Java classes and the loaded JNI library come from the prepared runtime; a standard-JNI unit test or a successful startup alone is insufficient. +For distributed storage, validate PD and Store together, then perform a Server `hstore` graph operation. + +At request/task completion, explicitly committed writes are preserved; unfinished writes are rolled back and cached backend leases are released. +Shared schema and element caches retain their invalidation listeners until the graph closes. + +Stop incoming work and use the component's normal stop script. PD drains its scheduled metadata work, joins Raft and closes the metadata database and native options. +Do not use a forced kill as evidence of normal resource cleanup. + +To return to standard RocksDB, stop the component, unset `TOPLINGDB_ROCKSDB_PROVIDER` and set its business provider back to `rocksdb`. +Use the corresponding standard-provider data directory; do not test a provider switch by reusing the other provider's database. + +Dedicated Topling distributions, Docker/Compose packaging and broader lifecycle/recovery changes are separate follow-ups. diff --git a/hugegraph-commons/hugegraph-common/README.md b/hugegraph-commons/hugegraph-common/README.md index bfeef03cd2..c2dba551c2 100644 --- a/hugegraph-commons/hugegraph-common/README.md +++ b/hugegraph-commons/hugegraph-common/README.md @@ -5,7 +5,7 @@ [![codecov](https://codecov.io/gh/hugegraph/hugegraph-common/branch/master/graph/badge.svg)](https://codecov.io/gh/hugegraph/hugegraph-common) [![Maven Central](https://maven-badges.herokuapp.com/maven-central/org.apache.hugegraph/hugegraph-common/badge.svg)](https://mvnrepository.com/artifact/org.apache.hugegraph/hugegraph-common) -hugegraph-common is a common module for [HugeGraph](https://github.com/hugegraph/hugegraph) and its peripheral components. +hugegraph-common is a common module for [HugeGraph](https://github.com/apache/hugegraph) and its peripheral components. hugegraph-common encapsulates locks, configurations, events, iterators, rest and some numeric or collection util classes to simplify the development of HugeGraph and its components. diff --git a/hugegraph-commons/hugegraph-common/src/main/java/org/apache/hugegraph/util/RocksDBRuntime.java b/hugegraph-commons/hugegraph-common/src/main/java/org/apache/hugegraph/util/RocksDBRuntime.java new file mode 100644 index 0000000000..d76f1aa84e --- /dev/null +++ b/hugegraph-commons/hugegraph-common/src/main/java/org/apache/hugegraph/util/RocksDBRuntime.java @@ -0,0 +1,68 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with this + * work for additional information regarding copyright ownership. The ASF + * licenses this file to You under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + * License for the specific language governing permissions and limitations + * under the License. + */ + +package org.apache.hugegraph.util; + +import java.net.URL; +import java.security.CodeSource; + +/** Match configuration to the JNI Java classes selected by the launcher. */ +public final class RocksDBRuntime { + + private RocksDBRuntime() { + } + + private static URL origin(Class runtime) { + CodeSource source = runtime.getProtectionDomain().getCodeSource(); + if (source == null || source.getLocation() == null) { + throw new IllegalStateException("Cannot verify RocksDB runtime origin for " + runtime.getName() + + "; use a class loader with a CodeSource location"); + } + return source.getLocation(); + } + + public static void verify(String configuredProvider) { + if (!"rocksdb".equals(configuredProvider) && !"topling".equals(configuredProvider)) { + throw new IllegalArgumentException("Invalid rocksdb.provider: " + configuredProvider); + } + ClassLoader loader = Thread.currentThread().getContextClassLoader(); + try { + Class rocks; + try { + rocks = Class.forName("org.rocksdb.RocksDB", false, loader); + } catch (ClassNotFoundException contextLoaderMiss) { + loader = RocksDBRuntime.class.getClassLoader(); + rocks = Class.forName("org.rocksdb.RocksDB", false, loader); + } + URL origin = origin(rocks); + boolean topling = false; + try { + Class marker = Class.forName("org.rocksdb.SidePluginRepo", false, loader); + topling = origin.equals(origin(marker)); + } catch (ClassNotFoundException standardRuntime) { + // Standard RocksDB has no Topling marker class. + } + if (topling != "topling".equals(configuredProvider)) { + throw new IllegalStateException("Configured rocksdb.provider=" + configuredProvider + + " conflicts with selected RocksDB Java runtime: " + origin + + "; match the launcher opt-in and business configuration"); + } + } catch (ClassNotFoundException e) { + throw new IllegalStateException("RocksDB Java runtime is missing", e); + } + } +} diff --git a/hugegraph-commons/hugegraph-common/src/test/java/org/apache/hugegraph/unit/UnitTestSuite.java b/hugegraph-commons/hugegraph-common/src/test/java/org/apache/hugegraph/unit/UnitTestSuite.java index 7705215e2d..9e5e39c546 100644 --- a/hugegraph-commons/hugegraph-common/src/test/java/org/apache/hugegraph/unit/UnitTestSuite.java +++ b/hugegraph-commons/hugegraph-common/src/test/java/org/apache/hugegraph/unit/UnitTestSuite.java @@ -25,6 +25,7 @@ import org.apache.hugegraph.unit.event.EventHubTest; import org.apache.hugegraph.unit.rest.AbstractRestClientTest; import org.apache.hugegraph.unit.version.VersionTest; +import org.apache.hugegraph.unit.util.RocksDBRuntimeTest; import org.junit.runner.RunWith; import org.junit.runners.Suite; @@ -72,6 +73,7 @@ @RunWith(Suite.class) @Suite.SuiteClasses({ + RocksDBRuntimeTest.class, LockManagerTest.class, LockGroupTest.class, AtomicLockTest.class, diff --git a/hugegraph-commons/hugegraph-common/src/test/java/org/apache/hugegraph/unit/util/ReflectionUtilTest.java b/hugegraph-commons/hugegraph-common/src/test/java/org/apache/hugegraph/unit/util/ReflectionUtilTest.java index f511ddfc7d..6f47aadf4d 100644 --- a/hugegraph-commons/hugegraph-common/src/test/java/org/apache/hugegraph/unit/util/ReflectionUtilTest.java +++ b/hugegraph-commons/hugegraph-common/src/test/java/org/apache/hugegraph/unit/util/ReflectionUtilTest.java @@ -94,7 +94,7 @@ public void testClasses() throws IOException { @SuppressWarnings("unchecked") List classes = IteratorUtils.toList(ReflectionUtil.classes( "org.apache.hugegraph.util")); - Assert.assertEquals(19, classes.size()); + Assert.assertEquals(20, classes.size()); classes.sort(Comparator.comparing(ClassInfo::getName)); Assert.assertEquals("org.apache.hugegraph.util.Bytes", classes.get(0).getName()); diff --git a/hugegraph-commons/hugegraph-common/src/test/java/org/apache/hugegraph/unit/util/RocksDBRuntimeTest.java b/hugegraph-commons/hugegraph-common/src/test/java/org/apache/hugegraph/unit/util/RocksDBRuntimeTest.java new file mode 100644 index 0000000000..6eb9fd1fc8 --- /dev/null +++ b/hugegraph-commons/hugegraph-common/src/test/java/org/apache/hugegraph/unit/util/RocksDBRuntimeTest.java @@ -0,0 +1,188 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with this + * work for additional information regarding copyright ownership. The ASF + * licenses this file to You under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + * License for the specific language governing permissions and limitations + * under the License. + */ + +package org.apache.hugegraph.unit.util; + +import java.io.InputStream; +import java.lang.reflect.InvocationTargetException; +import java.net.URL; +import java.net.URLClassLoader; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.security.ProtectionDomain; + +import javax.tools.ToolProvider; + +import org.apache.hugegraph.testutil.Assert; +import org.apache.hugegraph.util.RocksDBRuntime; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.TemporaryFolder; + +public class RocksDBRuntimeTest { + + @Rule + public TemporaryFolder temporary = new TemporaryFolder(); + + @Test + public void testStandardAndUnrelatedMarkerOrigin() throws Exception { + Path rocks = this.temporary.newFolder("standard").toPath(); + Path marker = this.temporary.newFolder("unrelated").toPath(); + compile(rocks, "RocksDB"); + compile(marker, "SidePluginRepo"); + verifyWithLoader("rocksdb", rocks); + verifyWithLoader("rocksdb", rocks, marker); + Assert.assertThrows(IllegalStateException.class, () -> verifyWithLoader("topling", rocks, marker)); + } + + @Test + public void testToplingRequiresSameOriginAndMatchingConfiguration() throws Exception { + Path runtime = this.temporary.newFolder("topling").toPath(); + compile(runtime, "RocksDB"); + compile(runtime, "SidePluginRepo"); + verifyWithLoader("topling", runtime); + Assert.assertThrows(IllegalStateException.class, () -> verifyWithLoader("rocksdb", runtime)); + Assert.assertThrows(IllegalArgumentException.class, () -> verifyWithLoader("invalid", runtime)); + } + + @Test + public void testMissingContextRuntimeFallsBackToDefiningLoader() throws Exception { + for (String provider : new String[]{"rocksdb", "topling"}) { + Path runtime = this.temporary.newFolder("fallback-" + provider).toPath(); + compile(runtime, "RocksDB"); + if ("topling".equals(provider)) { + compile(runtime, "SidePluginRepo"); + } + verifyWithDefiningLoader(provider, null, runtime); + try (URLClassLoader isolated = new URLClassLoader(new URL[0], null)) { + verifyWithDefiningLoader(provider, isolated, runtime); + String mismatch = "topling".equals(provider) ? "rocksdb" : "topling"; + Assert.assertThrows(IllegalStateException.class, + () -> verifyWithDefiningLoader(mismatch, isolated, runtime)); + } + } + } + + @Test + public void testContextLinkageFailureIsNotHiddenByFallback() throws Exception { + Path runtime = this.temporary.newFolder("fallback-linkage").toPath(); + compile(runtime, "RocksDB"); + LinkageError failure = new NoClassDefFoundError("broken context runtime dependency"); + ClassLoader broken = new ClassLoader(null) { + @Override + protected Class loadClass(String name, boolean resolve) throws ClassNotFoundException { + if ("org.rocksdb.RocksDB".equals(name)) { + throw failure; + } + return super.loadClass(name, resolve); + } + }; + Assert.assertSame(failure, Assert.assertThrows(LinkageError.class, + () -> verifyWithDefiningLoader("rocksdb", broken, runtime))); + } + + @Test + public void testContextProviderMismatchDoesNotSelectDefiningRuntime() throws Exception { + Path standard = this.temporary.newFolder("preferred-context").toPath(); + Path topling = this.temporary.newFolder("unused-defining").toPath(); + compile(standard, "RocksDB"); + compile(topling, "RocksDB"); + compile(topling, "SidePluginRepo"); + try (URLClassLoader context = new URLClassLoader(new URL[]{standard.toUri().toURL()}, null)) { + verifyWithDefiningLoader("rocksdb", context, topling); + Assert.assertThrows(IllegalStateException.class, + () -> verifyWithDefiningLoader("topling", context, topling)); + } + } + + @Test + public void testMissingCodeSourceFailsWithControlledError() throws Exception { + Path runtime = this.temporary.newFolder("no-origin").toPath(); + compile(runtime, "RocksDB"); + byte[] bytes = Files.readAllBytes(runtime.resolve("org/rocksdb/RocksDB.class")); + ClassLoader loader = new ClassLoader(null) { + @Override + protected Class findClass(String name) throws ClassNotFoundException { + if ("org.rocksdb.RocksDB".equals(name)) { + return this.defineClass(name, bytes, 0, bytes.length, new ProtectionDomain(null, null)); + } + throw new ClassNotFoundException(name); + } + }; + ClassLoader previous = Thread.currentThread().getContextClassLoader(); + try { + Thread.currentThread().setContextClassLoader(loader); + for (String provider : new String[]{"rocksdb", "topling"}) { + IllegalStateException failure = Assert.assertThrows(IllegalStateException.class, + () -> RocksDBRuntime.verify(provider)); + Assert.assertTrue(failure.getMessage().contains("Cannot verify RocksDB runtime origin")); + } + } finally { + Thread.currentThread().setContextClassLoader(previous); + } + } + + private static void verifyWithDefiningLoader(String provider, ClassLoader context, Path runtime) throws Exception { + String resource = "org/apache/hugegraph/util/RocksDBRuntime.class"; + Path target = runtime.resolve(resource); + Files.createDirectories(target.getParent()); + try (InputStream bytes = RocksDBRuntime.class.getClassLoader().getResourceAsStream(resource)) { + Files.copy(bytes, target, java.nio.file.StandardCopyOption.REPLACE_EXISTING); + } + ClassLoader previous = Thread.currentThread().getContextClassLoader(); + try (URLClassLoader defining = new URLClassLoader(new URL[]{runtime.toUri().toURL()}, null)) { + Thread.currentThread().setContextClassLoader(context); + Class verifier = Class.forName(RocksDBRuntime.class.getName(), true, defining); + try { + verifier.getMethod("verify", String.class).invoke(null, provider); + } catch (InvocationTargetException e) { + Throwable cause = e.getCause(); + if (cause instanceof Error) { + throw (Error) cause; + } + throw (Exception) cause; + } + } finally { + Thread.currentThread().setContextClassLoader(previous); + } + } + + private static void compile(Path root, String name) throws Exception { + Path source = root.resolve("org/rocksdb/" + name + ".java"); + Files.createDirectories(source.getParent()); + // Loading these classes must not initialize them or native libraries. + String code = "package org.rocksdb; public class " + name + + " { static { if (true) throw new AssertionError(\"initialized\"); } }"; + Files.write(source, code.getBytes(StandardCharsets.UTF_8)); + Assert.assertEquals(0, ToolProvider.getSystemJavaCompiler().run(null, null, null, source.toString())); + } + + private static void verifyWithLoader(String provider, Path... roots) throws Exception { + URL[] urls = new URL[roots.length]; + for (int i = 0; i < roots.length; i++) { + urls[i] = roots[i].toUri().toURL(); + } + ClassLoader previous = Thread.currentThread().getContextClassLoader(); + try (URLClassLoader loader = new URLClassLoader(urls, null)) { + Thread.currentThread().setContextClassLoader(loader); + RocksDBRuntime.verify(provider); + } finally { + Thread.currentThread().setContextClassLoader(previous); + } + } +} diff --git a/hugegraph-commons/hugegraph-rpc/pom.xml b/hugegraph-commons/hugegraph-rpc/pom.xml index 822c983cd7..142092634e 100644 --- a/hugegraph-commons/hugegraph-rpc/pom.xml +++ b/hugegraph-commons/hugegraph-rpc/pom.xml @@ -106,7 +106,7 @@ io.grpc grpc-core - 1.28.1 + ${grpc.version} diff --git a/hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/TaskScheduleService.java b/hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/TaskScheduleService.java index 003a4278dc..c111a6f85c 100644 --- a/hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/TaskScheduleService.java +++ b/hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/TaskScheduleService.java @@ -190,6 +190,22 @@ public void onStoreRaftChanged(Metapb.Store store) { public void shutDown() { executor.shutdownNow(); + boolean interrupted = Thread.interrupted(); + try { + while (!executor.isTerminated()) { + try { + if (!executor.awaitTermination(5, TimeUnit.SECONDS)) { + log.warn("Waiting for PD scheduled tasks to stop"); + } + } catch (InterruptedException e) { + interrupted = true; + } + } + } finally { + if (interrupted) { + Thread.currentThread().interrupt(); + } + } } private boolean isLeader() { diff --git a/hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/meta/MetadataFactory.java b/hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/meta/MetadataFactory.java index 673c313dfc..954a9f9e2e 100644 --- a/hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/meta/MetadataFactory.java +++ b/hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/meta/MetadataFactory.java @@ -28,30 +28,47 @@ */ public class MetadataFactory { - private static HgKVStore store = null; + private static HgKVStore store; + private static boolean shutdown; - public static HgKVStore getStore(PDConfig pdConfig) { + public static synchronized HgKVStore getStore(PDConfig pdConfig) { + if (shutdown) { + throw new IllegalStateException("PD metadata store has shut down"); + } if (store == null) { - synchronized (MetadataFactory.class) { - if (store == null) { - HgKVStore proto = new HgKVStoreImpl(); - //proto.init(pdConfig); - store = pdConfig.getRaft().isEnable() ? - new RaftKVStore(RaftEngine.getInstance(), proto) : - proto; - store.init(pdConfig); + HgKVStore proto = new HgKVStoreImpl(); + HgKVStore candidate = pdConfig.getRaft().isEnable() ? + new RaftKVStore(RaftEngine.getInstance(), proto) : proto; + try { + candidate.init(pdConfig); + store = candidate; + } catch (RuntimeException | Error failure) { + try { + candidate.close(); + } catch (RuntimeException | Error cleanup) { + if (cleanup != failure) { + failure.addSuppressed(cleanup); + } } + throw failure; } } return store; } - public static void closeStore() { + public static synchronized void closeStore() { if (store != null) { store.close(); } } + // Final service shutdown must not allow a late facade to create a new DB. + public static synchronized void shutdownStore() { + shutdown = true; + closeStore(); + store = null; + } + public static StoreInfoMeta newStoreInfoMeta(PDConfig pdConfig) { return new StoreInfoMeta(pdConfig); } diff --git a/hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/raft/RaftEngine.java b/hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/raft/RaftEngine.java index d39384909d..50eb1ca906 100644 --- a/hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/raft/RaftEngine.java +++ b/hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/raft/RaftEngine.java @@ -58,7 +58,6 @@ import com.alipay.sofa.jraft.rpc.RpcServer; import com.alipay.sofa.jraft.rpc.impl.BoltRpcServer; import com.alipay.sofa.jraft.util.Endpoint; -import com.alipay.sofa.jraft.util.internal.ThrowUtil; import io.netty.channel.ChannelHandler; import lombok.extern.slf4j.Slf4j; @@ -195,43 +194,68 @@ public List backChannelHandlers() { } } - public void shutDown() { - if (this.alivePeersRefresher != null) { - this.alivePeersRefresher.shutdownNow(); - try { - // Best effort: shutdownNow only interrupts, and a refresh parked in - // listAlivePeers waits on a lock acquire the interrupt does not break, - // for up to the raft rpc connect timeout per unreachable peer. A refresh - // that outlives this wait may publish one stale count over the reset - // below; acceptable while shutDown has no production caller. - if (!this.alivePeersRefresher.awaitTermination(1, TimeUnit.SECONDS)) { - log.warn("Raft alive-peers refresher still running after shutdown; " + - "hg_raft_alive_peers may briefly report a stale value"); + // Call while Raft can still complete metadata work already issued by a listener. + public void stopLeaderCallbacks() { + this.stateMachine.stopLeaderCallbacks(); + } + + public synchronized void shutDown() { + boolean interrupted = Thread.interrupted(); + try { + if (this.alivePeersRefresher != null) { + this.alivePeersRefresher.shutdownNow(); + while (!this.alivePeersRefresher.isTerminated()) { + try { + if (!this.alivePeersRefresher.awaitTermination(5, TimeUnit.SECONDS)) { + log.warn("Waiting for PD Raft alive-peers refresher to stop"); + } + } catch (InterruptedException e) { + interrupted = true; + } } - } catch (InterruptedException e) { - Thread.currentThread().interrupt(); + this.alivePeersRefresher = null; } - this.alivePeersRefresher = null; - } - this.alivePeerCount = -1; - if (this.raftGroupService != null) { - this.raftGroupService.shutdown(); - try { - this.raftGroupService.join(); - } catch (final InterruptedException e) { + this.alivePeerCount = -1; + if (this.raftGroupService != null) { + this.raftGroupService.shutdown(); + boolean joined = false; + while (!joined) { + try { + this.raftGroupService.join(); + joined = true; + } catch (InterruptedException e) { + interrupted = true; + } + } + this.raftGroupService = null; + this.raftNode = null; + } else if (this.raftNode != null) { + this.raftNode.shutdown(); + boolean joined = false; + while (!joined) { + try { + this.raftNode.join(); + joined = true; + } catch (InterruptedException e) { + interrupted = true; + } + } this.raftNode = null; - ThrowUtil.throwException(e); } - this.raftGroupService = null; - } - if (this.rpcServer != null) { - this.rpcServer.shutdown(); - this.rpcServer = null; - } - if (this.raftNode != null) { - this.raftNode.shutdown(); + if (this.rpcServer != null) { + this.rpcServer.shutdown(); + this.rpcServer = null; + } + if (this.raftRpcClient != null) { + this.raftRpcClient.shutdown(); + this.raftRpcClient = null; + } + log.info("PD Raft shutdown joined"); + } finally { + if (interrupted) { + Thread.currentThread().interrupt(); + } } - this.raftNode = null; } public boolean isLeader() { diff --git a/hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/raft/RaftRpcClient.java b/hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/raft/RaftRpcClient.java index 8c7398a53a..18602101f0 100644 --- a/hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/raft/RaftRpcClient.java +++ b/hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/raft/RaftRpcClient.java @@ -45,6 +45,13 @@ public synchronized boolean init(final RpcOptions rpcOptions) { return this.rpcClient.init(null); } + public synchronized void shutdown() { + if (this.rpcClient != null) { + this.rpcClient.shutdown(); + this.rpcClient = null; + } + } + /** * Request a snapshot */ diff --git a/hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/raft/RaftStateMachine.java b/hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/raft/RaftStateMachine.java index 65451297ca..cc9caaa900 100644 --- a/hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/raft/RaftStateMachine.java +++ b/hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/raft/RaftStateMachine.java @@ -80,6 +80,9 @@ static final class ProbeView { } private List taskHandlers; private List stateListeners; + private final Object callbackMonitor = new Object(); + private int callbacks; + private boolean stoppingCallbacks; public RaftStateMachine() { this.taskHandlers = new CopyOnWriteArrayList<>(); @@ -150,6 +153,54 @@ public void onShutdown() { super.onShutdown(); } + private void notifyStateListeners() { + synchronized (this.callbackMonitor) { + if (this.stoppingCallbacks || CollectionUtils.isEmpty(this.stateListeners)) { + return; + } + this.callbacks++; + } + try { + Utils.runInThread(() -> { + try { + this.stateListeners.forEach(RaftStateListener::onRaftLeaderChanged); + } finally { + callbackFinished(); + } + }); + } catch (RuntimeException | Error failure) { + callbackFinished(); + throw failure; + } + } + + private void callbackFinished() { + synchronized (this.callbackMonitor) { + this.callbacks--; + this.callbackMonitor.notifyAll(); + } + } + + void stopLeaderCallbacks() { + boolean interrupted = false; + synchronized (this.callbackMonitor) { + this.stoppingCallbacks = true; + while (this.callbacks != 0) { + try { + this.callbackMonitor.wait(5000L); + } catch (InterruptedException e) { + interrupted = true; + } + if (this.callbacks != 0) { + log.warn("Waiting for PD leader callbacks to finish"); + } + } + } + if (interrupted) { + Thread.currentThread().interrupt(); + } + } + @Override public void onLeaderStart(final long term) { this.leaderTerm.set(term); @@ -157,11 +208,7 @@ public void onLeaderStart(final long term) { super.onLeaderStart(term); log.info("Raft becomes leader"); - Utils.runInThread(() -> { - if (!CollectionUtils.isEmpty(stateListeners)) { - stateListeners.forEach(RaftStateListener::onRaftLeaderChanged); - } - }); + notifyStateListeners(); } @Override @@ -182,11 +229,7 @@ public void onLeaderStop(final Status status) { public void onStartFollowing(final LeaderChangeContext ctx) { this.probeView = new ProbeView(State.STATE_FOLLOWER, true); super.onStartFollowing(ctx); - Utils.runInThread(() -> { - if (!CollectionUtils.isEmpty(stateListeners)) { - stateListeners.forEach(RaftStateListener::onRaftLeaderChanged); - } - }); + notifyStateListeners(); } @Override diff --git a/hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/store/HgKVStoreImpl.java b/hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/store/HgKVStoreImpl.java index fd5b9c6031..f33648e0eb 100644 --- a/hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/store/HgKVStoreImpl.java +++ b/hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/store/HgKVStoreImpl.java @@ -62,12 +62,11 @@ public class HgKVStoreImpl implements HgKVStore { @Override public void init(PDConfig config) { - dbOptions = new Options().setCreateIfMissing(true); - final Lock writeLock = this.readWriteLock.writeLock(); writeLock.lock(); try { - this.dbPath = config.getDataPath() + "/rocksdb/"; + this.dbOptions = new Options().setCreateIfMissing(true); + this.dbPath = config.getDataPath() + "/rocksdb"; File file = new File(this.dbPath); if (!file.exists()) { try { @@ -77,13 +76,14 @@ public void init(PDConfig config) { } } openRocksDB(dbPath); - } catch (PDException e) { - // TODO: retry the open and then fail PD startup instead of logging: a held RocksDB LOCK - // leaves this PD running uninitialized (/v1/ready answers 503 STATE_UNINITIALIZED - // while /v1/health answers 200), and with several PDs nothing restarts it. The Helm - // chart (helm/hugegraph) documents this as a limitation; drop that entry once fixed. + } catch (PDException | RuntimeException | Error e) { + // TODO: retry opening a held RocksDB LOCK before failing PD startup. // https://github.com/apache/hugegraph/issues/3226 - log.error("Failed to open data file,{}", e); + if (this.dbOptions != null) { + this.dbOptions.close(); + this.dbOptions = null; + } + throw new IllegalStateException("Failed to open PD metadata at " + this.dbPath, e); } finally { writeLock.unlock(); } @@ -119,10 +119,10 @@ public byte[] get(byte[] key) throws PDException { public List scanPrefix(byte[] prefix) { final Lock readLock = this.readWriteLock.readLock(); readLock.lock(); - try (ReadOptions options = new ReadOptions() - .setIterateLowerBound(new Slice(prefix))) { + try (Slice bound = new Slice(prefix); + ReadOptions options = new ReadOptions().setIterateLowerBound(bound); + RocksIterator iterator = db.newIterator(options)) { List kvs = new ArrayList<>(); - RocksIterator iterator = db.newIterator(options); iterator.seekToFirst(); while (iterator.isValid() && 0 == Bytes.indexOf(iterator.key(), prefix)) { kvs.add(new KV(iterator.key(), iterator.value())); @@ -152,9 +152,9 @@ public long remove(byte[] key) throws PDException { public long removeByPrefix(byte[] prefix) throws PDException { final Lock readLock = this.readWriteLock.readLock(); readLock.lock(); - try (ReadOptions options = new ReadOptions() - .setIterateLowerBound(new Slice(prefix))) { - RocksIterator iterator = db.newIterator(options); + try (Slice bound = new Slice(prefix); + ReadOptions options = new ReadOptions().setIterateLowerBound(bound); + RocksIterator iterator = db.newIterator(options)) { iterator.seekToFirst(); while (iterator.isValid()) { @@ -309,11 +309,11 @@ public void loadSnapshot(String snapshotPath) throws PDException { public List scanRange(byte[] start, byte[] end) { final Lock readLock = this.readWriteLock.readLock(); readLock.lock(); - try (ReadOptions options = new ReadOptions() - .setIterateLowerBound(new Slice(start)) - .setIterateUpperBound(new Slice(end))) { + try (Slice lower = new Slice(start); + Slice upper = new Slice(end); + ReadOptions options = new ReadOptions().setIterateLowerBound(lower).setIterateUpperBound(upper); + RocksIterator iterator = db.newIterator(options)) { List kvs = new ArrayList<>(); - RocksIterator iterator = db.newIterator(options); iterator.seekToFirst(); while (iterator.isValid()) { kvs.add(new KV(iterator.key(), iterator.value())); @@ -327,7 +327,17 @@ public List scanRange(byte[] start, byte[] end) { @Override public void close() { - closeRocksDB(); + Lock writeLock = this.readWriteLock.writeLock(); + writeLock.lock(); + try { + closeRocksDB(); + if (this.dbOptions != null) { + this.dbOptions.close(); + this.dbOptions = null; + } + } finally { + writeLock.unlock(); + } } private void closeRocksDB() { diff --git a/hugegraph-pd/hg-pd-dist/src/assembly/descriptor/server-assembly.xml b/hugegraph-pd/hg-pd-dist/src/assembly/descriptor/server-assembly.xml index c43a868352..ca6439ae7b 100644 --- a/hugegraph-pd/hg-pd-dist/src/assembly/descriptor/server-assembly.xml +++ b/hugegraph-pd/hg-pd-dist/src/assembly/descriptor/server-assembly.xml @@ -41,6 +41,24 @@ + + + ${project.basedir}/../../hugegraph-server/hugegraph-dist/src/assembly/static/bin/prepare-topling.sh + bin + 755 + + + ${project.basedir}/../../hugegraph-server/hugegraph-dist/src/assembly/static/bin/preload-topling.sh + bin + 755 + + + ${project.basedir}/../../hugegraph-server/hugegraph-dist/src/assembly/static/conf/toplingdb.yaml + conf + 644 + + + diff --git a/hugegraph-pd/hg-pd-dist/src/assembly/static/bin/start-hugegraph-pd.sh b/hugegraph-pd/hg-pd-dist/src/assembly/static/bin/start-hugegraph-pd.sh index 82a92dc9f9..2647deee52 100755 --- a/hugegraph-pd/hg-pd-dist/src/assembly/static/bin/start-hugegraph-pd.sh +++ b/hugegraph-pd/hg-pd-dist/src/assembly/static/bin/start-hugegraph-pd.sh @@ -173,15 +173,26 @@ fi JVM_OPTIONS="-Dlog4j.configurationFile=${CONF}/log4j2.xml -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager" +source "$BIN/preload-topling.sh" || exit 1 +BOOT_JARS=("${LIB}"/hg-pd-service-*.jar) +if [ "${#BOOT_JARS[@]}" -ne 1 ] || [ ! -f "${BOOT_JARS[0]}" ]; then + echo "Error: expected one component executable JAR in $LIB" >&2 + exit 1 +fi +JAVA_MAIN=(-jar "${BOOT_JARS[0]}") +if [ -n "${TOPLING_RUNTIME_CLASSPATH:-}" ]; then + JAVA_MAIN=(-cp "${TOPLING_RUNTIME_CLASSPATH}:${BOOT_JARS[0]}" org.springframework.boot.loader.JarLauncher) +fi + # Turn on security check if [[ $DAEMON == "true" ]]; then echo "Starting HugeGraphPDServer in daemon mode..." if [[ "${STDOUT_MODE:-false}" == "true" ]]; then - exec ${JAVA} -Dname="HugeGraphPD" ${JVM_OPTIONS} ${JAVA_OPTIONS} -jar \ - -Dspring.config.location=${CONF}/application.yml ${LIB}/hg-pd-service-*.jar & + exec ${JAVA} -Dname="HugeGraphPD" ${JVM_OPTIONS} ${JAVA_OPTIONS} \ + -Dspring.config.location=${CONF}/application.yml "${JAVA_MAIN[@]}" "$@" & else - exec ${JAVA} -Dname="HugeGraphPD" ${JVM_OPTIONS} ${JAVA_OPTIONS} -jar \ - -Dspring.config.location=${CONF}/application.yml ${LIB}/hg-pd-service-*.jar >> ${OUTPUT} 2>&1 & + exec ${JAVA} -Dname="HugeGraphPD" ${JVM_OPTIONS} ${JAVA_OPTIONS} \ + -Dspring.config.location=${CONF}/application.yml "${JAVA_MAIN[@]}" "$@" >> ${OUTPUT} 2>&1 & fi PID="$!" # Write pid to file @@ -193,10 +204,10 @@ else echo "$$" > "$PID_FILE" echo "[+pid] $$" if [[ "${STDOUT_MODE:-false}" == "true" ]]; then - exec ${JAVA} -Dname="HugeGraphPD" ${JVM_OPTIONS} ${JAVA_OPTIONS} -jar \ - -Dspring.config.location=${CONF}/application.yml ${LIB}/hg-pd-service-*.jar + exec ${JAVA} -Dname="HugeGraphPD" ${JVM_OPTIONS} ${JAVA_OPTIONS} \ + -Dspring.config.location=${CONF}/application.yml "${JAVA_MAIN[@]}" "$@" else - exec ${JAVA} -Dname="HugeGraphPD" ${JVM_OPTIONS} ${JAVA_OPTIONS} -jar \ - -Dspring.config.location=${CONF}/application.yml ${LIB}/hg-pd-service-*.jar >> ${OUTPUT} 2>&1 + exec ${JAVA} -Dname="HugeGraphPD" ${JVM_OPTIONS} ${JAVA_OPTIONS} \ + -Dspring.config.location=${CONF}/application.yml "${JAVA_MAIN[@]}" "$@" >> ${OUTPUT} 2>&1 fi fi diff --git a/hugegraph-pd/hg-pd-dist/src/assembly/static/conf/application.yml b/hugegraph-pd/hg-pd-dist/src/assembly/static/conf/application.yml index ada9791922..73bcae9069 100644 --- a/hugegraph-pd/hg-pd-dist/src/assembly/static/conf/application.yml +++ b/hugegraph-pd/hg-pd-dist/src/assembly/static/conf/application.yml @@ -96,3 +96,7 @@ partition: # The default maximum number of replicas per machine # the initial number of partitions= store-max-shard-count * store-number / default-shard-count store-max-shard-count: 12 + +# Match the explicit launcher opt-in when enabling Topling. +rocksdb: + provider: rocksdb diff --git a/hugegraph-pd/hg-pd-grpc/pom.xml b/hugegraph-pd/hg-pd-grpc/pom.xml index be8ccc44fd..4c6b77abbd 100644 --- a/hugegraph-pd/hg-pd-grpc/pom.xml +++ b/hugegraph-pd/hg-pd-grpc/pom.xml @@ -33,7 +33,8 @@ 1.7.1 - 1.39.0 + + 1.39.0 3.17.2 0.6.1 @@ -90,7 +91,7 @@ grpc-java - io.grpc:protoc-gen-grpc-java:${grpc.version}:exe:${os.detected.classifier} + io.grpc:protoc-gen-grpc-java:${grpc.codegen.version}:exe:${os.detected.classifier} ${project.basedir}/src/main/proto ${project.basedir}/src/main/java @@ -138,14 +139,13 @@ - 1.47.0 - 3.21.7 + 1.47.0 com.google.protobuf protobuf-java - ${riscv64.protobuf.java.version} + ${protobuf.runtime.version} diff --git a/hugegraph-pd/hg-pd-service/pom.xml b/hugegraph-pd/hg-pd-service/pom.xml index 18f03dff2e..7332c1210b 100644 --- a/hugegraph-pd/hg-pd-service/pom.xml +++ b/hugegraph-pd/hg-pd-service/pom.xml @@ -150,7 +150,7 @@ com.google.protobuf protobuf-java-util - 3.17.2 + ${protobuf.runtime.version} org.apache.hugegraph diff --git a/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/boot/HugePDServer.java b/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/boot/HugePDServer.java index 54d2761df8..2ca7f7540c 100644 --- a/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/boot/HugePDServer.java +++ b/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/boot/HugePDServer.java @@ -17,6 +17,7 @@ package org.apache.hugegraph.pd.boot; +import org.apache.hugegraph.util.RocksDBRuntime; import org.springframework.boot.SpringApplication; import org.springframework.boot.autoconfigure.SpringBootApplication; import org.springframework.context.annotation.ComponentScan; @@ -37,8 +38,10 @@ public static void main(String[] args) { System.setProperty("logging.path", "logs"); System.setProperty("com.alipay.remoting.client.log.level", "error"); } - Runtime.getRuntime().addShutdownHook(new ShutdownHook(Thread.currentThread())); - SpringApplication.run(HugePDServer.class); + SpringApplication application = new SpringApplication(HugePDServer.class); + application.addInitializers(context -> RocksDBRuntime.verify( + context.getEnvironment().getProperty("rocksdb.provider", "rocksdb"))); + application.run(args); System.out.println("Hugegraph-pd started."); } } diff --git a/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/boot/ShutdownHook.java b/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/boot/ShutdownHook.java index 371235f341..a8cf28abbf 100644 --- a/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/boot/ShutdownHook.java +++ b/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/boot/ShutdownHook.java @@ -15,75 +15,120 @@ * limitations under the License. */ - package org.apache.hugegraph.pd.boot; +import java.util.concurrent.ThreadPoolExecutor; +import java.util.concurrent.TimeUnit; + +import org.apache.hugegraph.pd.TaskScheduleService; +import org.apache.hugegraph.pd.meta.MetadataFactory; +import org.apache.hugegraph.pd.raft.RaftEngine; +import org.apache.hugegraph.pd.pulse.PDPulseSubject; +import org.apache.hugegraph.pd.service.KvServiceGrpcImpl; import org.apache.hugegraph.pd.service.MetadataService; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; +import org.apache.hugegraph.pd.service.PDService; +import org.springframework.beans.factory.DisposableBean; +import org.springframework.context.ApplicationListener; +import org.lognet.springboot.grpc.context.GRpcServerInitializedEvent; -import java.util.concurrent.ThreadPoolExecutor; +import io.grpc.Server; +import org.springframework.stereotype.Component; + +import lombok.extern.slf4j.Slf4j; -public class ShutdownHook extends Thread { +/** + * Spring stops its transports before destroying singleton beans. The PDService + * dependency keeps its metadata owners available until this cleanup completes. + */ +@Slf4j +@Component +public class ShutdownHook implements DisposableBean, ApplicationListener { - private static Logger log = LoggerFactory.getLogger(ShutdownHook.class); - private static String msg = "there are still uninterruptible jobs that have not been completed and" + - " will wait for them to complete"; - private Thread main; + private final PDService service; + private final KvServiceGrpcImpl kvService; + private Server grpcServer; - public ShutdownHook(Thread main) { - super(); - this.main = main; + public ShutdownHook(PDService service, KvServiceGrpcImpl kvService) { + this.service = service; + this.kvService = kvService; } @Override - public void run() { - log.info("shutdown signal received"); - main.interrupt(); - waitForShutdown(); - try { - main.join(); - } catch (InterruptedException e) { - } - log.info("shutdown completed"); + public void onApplicationEvent(GRpcServerInitializedEvent event) { + this.grpcServer = event.getServer(); } - private void waitForShutdown() { - checkUninterruptibleJobs(); + @Override + public void destroy() { + // The starter's default stop initiates shutdown without waiting. + awaitGrpcTermination(); + this.kvService.stopScheduling(); + closeOwners(this.service); } - private void checkUninterruptibleJobs() { - ThreadPoolExecutor jobs = MetadataService.getUninterruptibleJobs(); + public static void closeOwners(PDService service) { + log.info("Closing PD background work and native owners"); + TaskScheduleService tasks = service.getTaskService(); + if (tasks != null) { + tasks.shutDown(); + } + PDPulseSubject.stopScheduling(); + RaftEngine raft = RaftEngine.getInstance(); + raft.stopLeaderCallbacks(); + // Raft may still submit snapshot saves while its shutdown is joining. + raft.shutDown(); + awaitUninterruptibleJobs(); + // A failed join must not release the database out from under Raft. + MetadataFactory.shutdownStore(); + log.info("PD native owners closed"); + } + + private void awaitGrpcTermination() { + if (this.grpcServer == null) { + return; + } + // End long-lived watch/pulse streams as well as accepting no new calls. + this.grpcServer.shutdownNow(); + boolean interrupted = Thread.interrupted(); try { - if (jobs != null) { - long lastPrint = System.currentTimeMillis() - 5000; - log.info("check for ongoing background jobs that cannot be interrupted, active:{}, queue:{}.", - jobs.getActiveCount(), jobs.getQueue().size()); - while (jobs.getActiveCount() != 0 || jobs.getQueue().size() != 0) { - synchronized (ShutdownHook.class) { - if (System.currentTimeMillis() - lastPrint > 5000) { - log.warn(msg); - lastPrint = System.currentTimeMillis(); - } - try { - ShutdownHook.class.wait(200); - } catch (InterruptedException e) { - log.error("close jobs with error:", e); - } + while (!this.grpcServer.isTerminated()) { + try { + if (!this.grpcServer.awaitTermination(5, TimeUnit.SECONDS)) { + log.warn("Waiting for PD gRPC transport to terminate"); } + } catch (InterruptedException e) { + interrupted = true; } - log.info("all ongoing background jobs have been completed and the shutdown will continue"); } + } finally { + if (interrupted) { + Thread.currentThread().interrupt(); + } + } + } - } catch (Exception e) { - log.error("close jobs with error:", e); + private static void awaitUninterruptibleJobs() { + ThreadPoolExecutor jobs = MetadataService.getUninterruptibleJobs(); + if (jobs == null) { + return; } + jobs.shutdown(); + boolean interrupted = Thread.interrupted(); try { - if (jobs != null) { - jobs.shutdownNow(); + while (!jobs.isTerminated()) { + try { + if (!jobs.awaitTermination(5, TimeUnit.SECONDS)) { + log.warn("Waiting for PD metadata jobs, active: {}, queued: {}", + jobs.getActiveCount(), jobs.getQueue().size()); + } + } catch (InterruptedException e) { + interrupted = true; + } + } + } finally { + if (interrupted) { + Thread.currentThread().interrupt(); } - } catch (Exception e) { - log.error("close jobs with error:", e); } } } diff --git a/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/pulse/PDPulseSubject.java b/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/pulse/PDPulseSubject.java index 86b95cb880..58d886abee 100644 --- a/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/pulse/PDPulseSubject.java +++ b/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/pulse/PDPulseSubject.java @@ -89,6 +89,27 @@ public class PDPulseSubject { TimeUnit.SECONDS); } + /** Stop periodic queue work and wait while Raft can still serve its writes. */ + public static void stopScheduling() { + scheduledExecutor.shutdown(); + boolean interrupted = Thread.interrupted(); + try { + while (!scheduledExecutor.isTerminated()) { + try { + if (!scheduledExecutor.awaitTermination(5, TimeUnit.SECONDS)) { + log.warn("Waiting for PD pulse scheduler to stop"); + } + } catch (InterruptedException e) { + interrupted = true; + } + } + } finally { + if (interrupted) { + Thread.currentThread().interrupt(); + } + } + } + private static void doSchedule() { appendQueue(); expireQueue(); diff --git a/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/service/KvServiceGrpcImpl.java b/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/service/KvServiceGrpcImpl.java index e49a4dc48b..94b41a0f0a 100644 --- a/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/service/KvServiceGrpcImpl.java +++ b/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/service/KvServiceGrpcImpl.java @@ -26,6 +26,7 @@ import java.util.concurrent.atomic.AtomicLong; import javax.annotation.PostConstruct; +import javax.annotation.PreDestroy; import org.apache.hugegraph.pd.KvService; import org.apache.hugegraph.pd.common.PDException; @@ -86,6 +87,32 @@ public void init() { }, 0, KvWatchSubject.WATCH_TTL * 1 / 3, TimeUnit.MILLISECONDS); } + /** Stop periodic watch work and wait while Raft can still serve its writes. */ + @PreDestroy + public void stopScheduling() { + ScheduledExecutorService current = this.executor; + if (current == null) { + return; + } + current.shutdown(); + boolean interrupted = Thread.interrupted(); + try { + while (!current.isTerminated()) { + try { + if (!current.awaitTermination(5, TimeUnit.SECONDS)) { + log.warn("Waiting for PD KV watch scheduler to stop"); + } + } catch (InterruptedException e) { + interrupted = true; + } + } + } finally { + if (interrupted) { + Thread.currentThread().interrupt(); + } + } + } + /** * Ordinary put * diff --git a/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/service/PDService.java b/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/service/PDService.java index 94d136a844..f40b2f5801 100644 --- a/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/service/PDService.java +++ b/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/service/PDService.java @@ -53,6 +53,7 @@ import org.apache.hugegraph.pd.StoreNodeService; import org.apache.hugegraph.pd.StoreStatusListener; import org.apache.hugegraph.pd.TaskScheduleService; +import org.apache.hugegraph.pd.boot.ShutdownHook; import org.apache.hugegraph.pd.common.KVPair; import org.apache.hugegraph.pd.common.PDException; import org.apache.hugegraph.pd.config.PDConfig; @@ -177,6 +178,22 @@ public void setInitConfig(PDConfig pdConfig) { */ @PostConstruct public void init() throws PDException { + try { + initService(); + } catch (PDException | RuntimeException | Error failure) { + // Failed @PostConstruct beans are not registered for destruction. + try { + ShutdownHook.closeOwners(this); + } catch (RuntimeException | Error cleanup) { + if (cleanup != failure) { + failure.addSuppressed(cleanup); + } + } + throw failure; + } + } + + private void initService() throws PDException { log.info("PDService init………… {}", pdConfig); configService = new ConfigService(pdConfig); diff --git a/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/core/PDCoreSuiteTest.java b/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/core/PDCoreSuiteTest.java index ddc1fad7f9..1c73176703 100644 --- a/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/core/PDCoreSuiteTest.java +++ b/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/core/PDCoreSuiteTest.java @@ -33,6 +33,7 @@ RaftStateMachineTest.class, MetadataKeyHelperTest.class, HgKVStoreImplTest.class, + PDNativeOwnerTest.class, PDConfigTest.class, ConfigServiceTest.class, IdServiceTest.class, diff --git a/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/core/PDNativeOwnerTest.java b/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/core/PDNativeOwnerTest.java new file mode 100644 index 0000000000..e7c6c1d050 --- /dev/null +++ b/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/core/PDNativeOwnerTest.java @@ -0,0 +1,226 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hugegraph.pd.core; + +import java.lang.reflect.Constructor; +import java.lang.reflect.Field; +import java.nio.file.Path; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.FutureTask; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.TimeoutException; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.atomic.AtomicReference; +import java.util.concurrent.locks.ReadWriteLock; + +import org.apache.hugegraph.pd.config.PDConfig; +import org.apache.hugegraph.pd.meta.MetadataFactory; +import org.apache.hugegraph.pd.raft.RaftEngine; +import org.apache.hugegraph.pd.raft.RaftRpcClient; +import org.apache.hugegraph.pd.raft.RaftStateMachine; +import org.apache.hugegraph.pd.store.HgKVStore; +import org.apache.hugegraph.pd.store.HgKVStoreImpl; +import org.apache.hugegraph.testutil.Whitebox; +import org.junit.Assert; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.TemporaryFolder; +import org.rocksdb.Options; +import org.rocksdb.RocksDB; + +import com.alipay.sofa.jraft.Node; +import com.alipay.sofa.jraft.RaftGroupService; +import com.alipay.sofa.jraft.rpc.RpcServer; + +import static org.mockito.Mockito.doAnswer; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.never; +import static org.mockito.Mockito.times; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +public class PDNativeOwnerTest { + + @Rule + public TemporaryFolder temporary = new TemporaryFolder(); + + @Test(timeout = 10000) + public void testCloseWaitsForReadersAndReleasesNativeOwners() throws Exception { + RocksDB.loadLibrary(); + Path path = this.temporary.newFolder().toPath(); + PDConfig config = new PDConfig(); + config.setDataPath(path.toString()); + HgKVStoreImpl store = new HgKVStoreImpl(); + store.init(config); + RocksDB db = Whitebox.getInternalState(store, "db"); + Options options = Whitebox.getInternalState(store, "dbOptions"); + ReadWriteLock lock = Whitebox.getInternalState(store, "readWriteLock"); + store.put(new byte[]{1}, new byte[]{2}); + store.put(new byte[]{3}, new byte[]{4}); + Assert.assertEquals(1, store.scanPrefix(new byte[]{1}).size()); + Assert.assertEquals(2, store.scanRange(new byte[]{0}, new byte[]{4}).size()); + store.removeByPrefix(new byte[]{1}); + Assert.assertEquals(0, store.scanPrefix(new byte[]{1}).size()); + CountDownLatch closing = new CountDownLatch(1); + FutureTask close = new FutureTask<>(() -> { + closing.countDown(); + store.close(); + return null; + }); + Thread closer = new Thread(close, "pd-owner-close"); + lock.readLock().lock(); + try { + closer.start(); + Assert.assertTrue(closing.await(1, TimeUnit.SECONDS)); + Assert.assertThrows(TimeoutException.class, () -> close.get(50, TimeUnit.MILLISECONDS)); + Assert.assertTrue(db.isOwningHandle()); + Assert.assertTrue(options.isOwningHandle()); + } finally { + lock.readLock().unlock(); + } + close.get(2, TimeUnit.SECONDS); + store.close(); + Assert.assertFalse(db.isOwningHandle()); + Assert.assertFalse(options.isOwningHandle()); + HgKVStoreImpl reopened = new HgKVStoreImpl(); + reopened.init(config); + try { + Assert.assertArrayEquals(new byte[]{4}, reopened.get(new byte[]{3})); + } finally { + reopened.close(); + } + } + + @Test + public void testInitializationFailureReleasesOptions() { + RocksDB.loadLibrary(); + HgKVStoreImpl store = new HgKVStoreImpl(); + PDConfig config = mock(PDConfig.class); + AtomicReference allocated = new AtomicReference<>(); + IllegalStateException injected = new IllegalStateException("injected configuration failure"); + when(config.getDataPath()).thenAnswer(invocation -> { + allocated.set(Whitebox.getInternalState(store, "dbOptions")); + throw injected; + }); + IllegalStateException failure = Assert.assertThrows(IllegalStateException.class, () -> store.init(config)); + Assert.assertSame(injected, failure.getCause()); + Assert.assertFalse(allocated.get().isOwningHandle()); + store.close(); + } + + @Test + public void testTerminalFactoryCloseRejectsLazyReopen() throws Exception { + Field storeField = MetadataFactory.class.getDeclaredField("store"); + Field shutdownField = MetadataFactory.class.getDeclaredField("shutdown"); + storeField.setAccessible(true); + shutdownField.setAccessible(true); + synchronized (MetadataFactory.class) { + Object previous = storeField.get(null); + boolean shutdown = shutdownField.getBoolean(null); + HgKVStore owned = mock(HgKVStore.class); + try { + storeField.set(null, owned); + shutdownField.setBoolean(null, false); + MetadataFactory.shutdownStore(); + MetadataFactory.shutdownStore(); + verify(owned, times(1)).close(); + Assert.assertThrows(IllegalStateException.class, () -> MetadataFactory.getStore(null)); + } finally { + storeField.set(null, previous); + shutdownField.setBoolean(null, shutdown); + } + } + } + + @Test + public void testInterruptedRaftJoinRetainsOwnerAndClosesRpcClient() throws Exception { + RaftEngine engine = newEngine(); + RaftGroupService group = mock(RaftGroupService.class); + Node node = mock(Node.class); + RpcServer server = mock(RpcServer.class); + RaftRpcClient client = mock(RaftRpcClient.class); + Whitebox.setInternalState(engine, "raftGroupService", group); + Whitebox.setInternalState(engine, "raftNode", node); + Whitebox.setInternalState(engine, "rpcServer", server); + Whitebox.setInternalState(engine, "raftRpcClient", client); + AtomicInteger attempts = new AtomicInteger(); + doAnswer(invocation -> { + Assert.assertSame(node, engine.getRaftNode()); + if (attempts.incrementAndGet() == 1) { + throw new InterruptedException("injected join interruption"); + } + return null; + }).when(group).join(); + try { + engine.shutDown(); + Assert.assertTrue(Thread.interrupted()); + Assert.assertEquals(2, attempts.get()); + Assert.assertNull(engine.getRaftNode()); + engine.shutDown(); + verify(group, times(1)).shutdown(); + verify(node, never()).shutdown(); + verify(server, times(1)).shutdown(); + verify(client, times(1)).shutdown(); + } finally { + Thread.interrupted(); + } + } + + @Test(timeout = 10000) + public void testLeaderCallbacksDrainBeforeStoppingRaft() throws Exception { + RaftEngine engine = newEngine(); + RaftStateMachine state = Whitebox.getInternalState(engine, "stateMachine"); + CountDownLatch entered = new CountDownLatch(1); + CountDownLatch release = new CountDownLatch(1); + AtomicInteger calls = new AtomicInteger(); + state.addStateListener(() -> { + calls.incrementAndGet(); + entered.countDown(); + try { + Assert.assertTrue(release.await(3, TimeUnit.SECONDS)); + } catch (InterruptedException e) { + throw new AssertionError(e); + } + }); + state.onLeaderStart(1); + Assert.assertTrue(entered.await(1, TimeUnit.SECONDS)); + FutureTask stop = new FutureTask<>(() -> { + engine.stopLeaderCallbacks(); + return null; + }); + Thread stopper = new Thread(stop, "pd-leader-drain"); + try { + stopper.start(); + Assert.assertThrows(TimeoutException.class, () -> stop.get(50, TimeUnit.MILLISECONDS)); + release.countDown(); + stop.get(2, TimeUnit.SECONDS); + state.onLeaderStart(2); + engine.stopLeaderCallbacks(); + Assert.assertEquals(1, calls.get()); + } finally { + release.countDown(); + stopper.join(3000); + } + } + + private static RaftEngine newEngine() throws Exception { + Constructor constructor = RaftEngine.class.getDeclaredConstructor(); + constructor.setAccessible(true); + return constructor.newInstance(); + } +} diff --git a/hugegraph-server/hugegraph-api/pom.xml b/hugegraph-server/hugegraph-api/pom.xml index e5a81be208..c6aeffa306 100644 --- a/hugegraph-server/hugegraph-api/pom.xml +++ b/hugegraph-server/hugegraph-api/pom.xml @@ -44,9 +44,7 @@ io.grpc grpc-netty-shaded - - - 1.47.0 + ${grpc.version} diff --git a/hugegraph-server/hugegraph-api/src/main/java/org/apache/hugegraph/auth/HugeGraphAuthProxy.java b/hugegraph-server/hugegraph-api/src/main/java/org/apache/hugegraph/auth/HugeGraphAuthProxy.java index 702abfe3b5..9d23e82556 100644 --- a/hugegraph-server/hugegraph-api/src/main/java/org/apache/hugegraph/auth/HugeGraphAuthProxy.java +++ b/hugegraph-server/hugegraph-api/src/main/java/org/apache/hugegraph/auth/HugeGraphAuthProxy.java @@ -40,6 +40,7 @@ import javax.security.sasl.AuthenticationException; import org.apache.commons.configuration2.Configuration; +import org.apache.hugegraph.HugeFactory; import org.apache.hugegraph.HugeGraph; import org.apache.hugegraph.HugeGraphSupplier; import org.apache.hugegraph.auth.HugeAuthenticator.RolePerm; @@ -1303,7 +1304,13 @@ public void run() { try { this.runner.run(); } finally { - resetContext(); + try { + HugeFactory.closeCurrentThreadTransactions(); + } catch (Throwable e) { + LOG.error("Failed to close Gremlin worker transactions", e); + } finally { + resetContext(); + } } } } diff --git a/hugegraph-server/hugegraph-api/src/main/java/org/apache/hugegraph/server/ApplicationConfig.java b/hugegraph-server/hugegraph-api/src/main/java/org/apache/hugegraph/server/ApplicationConfig.java index 9bbed7f486..5cc3e0eb33 100644 --- a/hugegraph-server/hugegraph-api/src/main/java/org/apache/hugegraph/server/ApplicationConfig.java +++ b/hugegraph-server/hugegraph-api/src/main/java/org/apache/hugegraph/server/ApplicationConfig.java @@ -17,12 +17,14 @@ package org.apache.hugegraph.server; +import org.apache.hugegraph.HugeFactory; import org.apache.hugegraph.config.HugeConfig; import org.apache.hugegraph.core.GraphManager; import org.apache.hugegraph.define.WorkLoad; import org.apache.hugegraph.event.EventHub; import org.apache.hugegraph.exception.HugeException; import org.apache.hugegraph.util.E; +import org.apache.hugegraph.util.Log; import org.apache.hugegraph.version.CoreVersion; import org.apache.tinkerpop.gremlin.server.util.MetricManager; import org.glassfish.hk2.api.Factory; @@ -35,6 +37,7 @@ import org.glassfish.jersey.server.monitoring.ApplicationEventListener; import org.glassfish.jersey.server.monitoring.RequestEvent; import org.glassfish.jersey.server.monitoring.RequestEventListener; +import org.slf4j.Logger; import com.codahale.metrics.MetricRegistry; import com.codahale.metrics.jersey3.InstrumentedResourceMethodApplicationListener; @@ -64,6 +67,9 @@ ) @ApplicationPath("/") public class ApplicationConfig extends ResourceConfig { + + private static final Logger LOG = Log.logger(ApplicationConfig.class); + @Context private ServletConfig servletConfig; @@ -195,7 +201,15 @@ public void onEvent(ApplicationEvent event) { @Override public RequestEventListener onRequest(RequestEvent event) { - return null; + return request -> { + if (request.getType() == RequestEvent.Type.FINISHED) { + try { + HugeFactory.closeCurrentThreadTransactions(); + } catch (RuntimeException e) { + LOG.error("Failed to release REST request transactions", e); + } + } + }; } }); } diff --git a/hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/HugeFactory.java b/hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/HugeFactory.java index cac015b0f6..451455ba0f 100644 --- a/hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/HugeFactory.java +++ b/hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/HugeFactory.java @@ -21,7 +21,9 @@ import java.io.File; import java.net.URL; +import java.util.ArrayList; import java.util.HashMap; +import java.util.List; import java.util.Map; import java.util.concurrent.TimeoutException; import java.util.concurrent.atomic.AtomicBoolean; @@ -46,7 +48,7 @@ public class HugeFactory { public static final String SYS_GRAPH = Graph.Hidden.hide("sys_graph"); private static final Logger LOG = Log.logger(HugeFactory.class); private static final String NAME_REGEX = "^[A-Za-z][A-Za-z0-9_]{0,47}$"; - private static final Map GRAPHS = new HashMap<>(); + private static final Map GRAPHS = new HashMap<>(); private static final AtomicBoolean SHUT_DOWN = new AtomicBoolean(false); private static final Thread SHUT_DOWN_HOOK = new Thread(() -> { LOG.info("HugeGraph is shutting down"); @@ -85,7 +87,7 @@ public static synchronized HugeGraph open(HugeConfig config) { String graphSpace = config.get(CoreOptions.GRAPH_SPACE); name = name.toLowerCase(); String spaceGraphName = graphSpace + "-" + name; - HugeGraph graph = GRAPHS.get(spaceGraphName); + StandardHugeGraph graph = GRAPHS.get(spaceGraphName); if (graph == null || graph.closed()) { graph = new StandardHugeGraph(config); GRAPHS.put(spaceGraphName, graph); @@ -106,11 +108,33 @@ public static HugeGraph open(URL url) { return open(getRemoteConfig(url)); } - public static void remove(HugeGraph graph) { + public static synchronized void remove(HugeGraph graph) { String spaceGraphName = graph.spaceGraphName(); GRAPHS.remove(spaceGraphName); } + public static void closeCurrentThreadTransactions() { + List graphs; + synchronized (HugeFactory.class) { + graphs = new ArrayList<>(GRAPHS.values()); + } + Throwable failure = null; + for (StandardHugeGraph graph : graphs) { + try { + graph.closeCurrentThreadTransaction(); + } catch (Throwable e) { + if (failure == null) { + failure = e; + } else if (failure != e) { + failure.addSuppressed(e); + } + } + } + if (failure != null) { + throw new HugeException("Failed to close current thread transactions", failure); + } + } + public static void checkGraphName(String name, String configFile) { E.checkArgument(SYS_GRAPH.equals(name) || name.matches(NAME_REGEX), "Invalid graph name '%s' in %s, " + diff --git a/hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/StandardHugeGraph.java b/hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/StandardHugeGraph.java index b55ad34e19..d9b838e897 100644 --- a/hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/StandardHugeGraph.java +++ b/hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/StandardHugeGraph.java @@ -385,6 +385,19 @@ public boolean closed() { return this.closed; } + void closeCurrentThreadTransaction() { + try { + if (this.tx.isOpen()) { + // Request/task cleanup must never commit unfinished writes. + this.tx.rollback(); + } + } finally { + this.tx.clearTransactionListeners(); + this.tx.resetState(); + this.tx.destroyTransaction(); + } + } + private void closeTx() { try { if (this.tx.isOpen()) { @@ -1101,8 +1114,23 @@ public synchronized void close() throws Exception { this.closeTx(); } finally { this.closed = true; - this.storeProvider.close(); - LockUtil.destroy(this.spaceGraphName()); + try { + CachedGraphTransaction.closeGraph(this.params); + } finally { + try { + try { + CachedSchemaTransaction.closeGraph(this.params); + } finally { + CachedSchemaTransactionV2.closeGraph(this.params); + } + } finally { + try { + this.storeProvider.close(); + } finally { + LockUtil.destroy(this.spaceGraphName()); + } + } + } } // Make sure that all transactions are closed in all threads diff --git a/hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/backend/cache/CacheListenerHolder.java b/hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/backend/cache/CacheListenerHolder.java index b4133679bb..f0c7c738eb 100644 --- a/hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/backend/cache/CacheListenerHolder.java +++ b/hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/backend/cache/CacheListenerHolder.java @@ -19,23 +19,37 @@ import org.apache.hugegraph.event.EventHub; import org.apache.hugegraph.event.EventListener; +import org.apache.hugegraph.util.Events; /* - * Listener lifetime must cover all active transactions for the graph. - * The holder is removed from the registry and unregistered from EventHub - * only when the last transaction releases it. + * Caches and listeners belong to the graph, including between request leases. + * Transaction close releases a lease; graph close disposes the holder. */ final class CacheListenerHolder { final EventListener listener; final EventHub hub; + final Runnable cleanup; // Must only be read or written inside ConcurrentMap.compute() for the // enclosing registry; ConcurrentHashMap.compute() serialises per-key access. int refCount; CacheListenerHolder(EventListener listener, EventHub hub) { + this(listener, hub, () -> { }); + } + + CacheListenerHolder(EventListener listener, EventHub hub, Runnable cleanup) { this.listener = listener; this.hub = hub; + this.cleanup = cleanup; this.refCount = 1; } + + void close() { + try { + this.hub.unlisten(Events.CACHE, this.listener); + } finally { + this.cleanup.run(); + } + } } diff --git a/hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/backend/cache/CachedGraphTransaction.java b/hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/backend/cache/CachedGraphTransaction.java index e8e579e15e..d814f52f55 100644 --- a/hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/backend/cache/CachedGraphTransaction.java +++ b/hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/backend/cache/CachedGraphTransaction.java @@ -66,9 +66,8 @@ public final class CachedGraphTransaction extends GraphTransaction { private static final long AVG_EDGE_ENTRY_SIZE = 100L; /* - * Listener lifetime must cover all active transactions for the graph. - * The holder is removed from the registry and unregistered from EventHub - * only when the last transaction releases it. + * Retain graph caches and invalidation listeners between request leases. + * Only graph close disposes the holder. */ private static final ConcurrentMap GRAPH_CACHE_EVENT_LISTENERS = new ConcurrentHashMap<>(); @@ -158,10 +157,11 @@ private void listenChanges() { return false; }; BackendStoreProvider provider = this.store().provider(); + EventHub graphEventHub = this.params().graphEventHub(); String graphName = this.params().spaceGraphName(); StoreListenerHolder storeAcquired = STORE_EVENT_LISTENERS.compute( graphName, (key, existing) -> { - if (existing == null || existing.provider != provider) { + if (existing == null || existing.provider != provider || existing.hub != graphEventHub) { // Graph close/reopen creates a new provider for the // same graph name; replace the stale holder. Old // transactions skip decrement via identity check. @@ -169,7 +169,7 @@ private void listenChanges() { existing.provider.unlisten(existing.listener); } provider.listen(storeListener); - return new StoreListenerHolder(storeListener, provider); + return new StoreListenerHolder(storeListener, provider, graphEventHub); } existing.refCount++; return existing; @@ -223,7 +223,6 @@ private void listenChanges() { } return false; }; - EventHub graphEventHub = this.params().graphEventHub(); CacheListenerHolder acquired = GRAPH_CACHE_EVENT_LISTENERS.compute( graphName, (key, existing) -> { if (existing == null || existing.hub != graphEventHub) { @@ -231,11 +230,10 @@ private void listenChanges() { // same graph name; replace the stale holder. Old // transactions skip decrement via identity check. if (existing != null) { - existing.hub.unlisten(Events.CACHE, - existing.listener); + existing.close(); } graphEventHub.listen(Events.CACHE, listener); - return new CacheListenerHolder(listener, graphEventHub); + return new CacheListenerHolder(listener, graphEventHub, () -> this.clearCache(null, false)); } existing.refCount++; return existing; @@ -253,10 +251,6 @@ private void unlistenChanges() { return existing; } existing.refCount--; - if (existing.refCount == 0) { - existing.hub.unlisten(Events.CACHE, existing.listener); - return null; - } return existing; }); this.holder = null; @@ -269,16 +263,35 @@ private void unlistenChanges() { return existing; } existing.refCount--; - if (existing.refCount == 0) { - existing.provider.unlisten(existing.listener); - return null; - } return existing; }); this.storeHolder = null; } } + public static void closeGraph(HugeGraphParams params) { + String graphName = params.spaceGraphName(); + EventHub graphEventHub = params.graphEventHub(); + try { + GRAPH_CACHE_EVENT_LISTENERS.computeIfPresent(graphName, (key, existing) -> { + if (existing.hub != graphEventHub) { + return existing; + } + existing.close(); + return null; + }); + } finally { + STORE_EVENT_LISTENERS.computeIfPresent(graphName, (key, storeHolder) -> { + // The hub owns this graph generation even if the provider is pooled. + if (storeHolder.hub != graphEventHub) { + return storeHolder; + } + storeHolder.provider.unlisten(storeHolder.listener); + return null; + }); + } + } + private void notifyChanges(String action, HugeType type, Id[] ids) { EventHub graphEventHub = this.params().graphEventHub(); graphEventHub.notifyExcept(Events.CACHE, this.cacheEventListener, @@ -533,24 +546,24 @@ public void removeIndex(IndexLabel indexLabel) { } /* - * Listener lifetime must cover all active transactions for the graph. - * The holder is removed from the registry and unregistered from the - * BackendStoreProvider only when the last transaction releases it. - * Mirror of CacheListenerHolder for the store event path. + * Store invalidation outlives request leases just like the graph cache. + * The hub identifies the graph generation even when a provider is pooled. */ private static final class StoreListenerHolder { final EventListener listener; final BackendStoreProvider provider; + final EventHub hub; // Must only be read or written inside ConcurrentMap.compute() for the // enclosing registry; ConcurrentHashMap.compute() serialises per-key // access. int refCount; StoreListenerHolder(EventListener listener, - BackendStoreProvider provider) { + BackendStoreProvider provider, EventHub hub) { this.listener = listener; this.provider = provider; + this.hub = hub; this.refCount = 1; } } diff --git a/hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/backend/cache/CachedSchemaTransaction.java b/hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/backend/cache/CachedSchemaTransaction.java index 8e48218e30..b3441f81d8 100644 --- a/hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/backend/cache/CachedSchemaTransaction.java +++ b/hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/backend/cache/CachedSchemaTransaction.java @@ -44,9 +44,8 @@ public final class CachedSchemaTransaction extends SchemaTransaction { /* - * Listener lifetime must cover all active transactions for the graph. - * The holder is removed from the registry and unregistered from EventHub - * only when the last transaction releases it. + * Retain graph caches and invalidation listeners between request leases. + * Only graph close disposes the holder. */ private static final ConcurrentMap SCHEMA_CACHE_EVENT_LISTENERS = new ConcurrentHashMap<>(); @@ -93,7 +92,6 @@ public void close() { try { super.close(); } finally { - this.clearCache(false); this.unlistenChanges(); } } @@ -118,7 +116,7 @@ private void listenChanges() { } return false; }; - this.store().provider().listen(this.storeEventListener); + // Listen cache event: "cache"(invalid cache item) EventListener listener = event -> { @@ -150,12 +148,17 @@ private void listenChanges() { // same graph name; replace the stale holder. Old // transactions skip decrement via identity check. if (existing != null) { - existing.hub.unlisten(Events.CACHE, - existing.listener); + existing.close(); } schemaEventHub.listen(Events.CACHE, listener); - return new CacheListenerHolder(listener, - schemaEventHub); + this.store().provider().listen(this.storeEventListener); + return new CacheListenerHolder(listener, schemaEventHub, () -> { + try { + this.store().provider().unlisten(this.storeEventListener); + } finally { + this.clearCache(false); + } + }); } existing.refCount++; return existing; @@ -165,9 +168,6 @@ private void listenChanges() { } private void unlistenChanges() { - // Unlisten store event - this.store().provider().unlisten(this.storeEventListener); - // Unlisten cache event CacheListenerHolder ours = this.holder; if (ours != null) { @@ -177,11 +177,6 @@ private void unlistenChanges() { return existing; } existing.refCount--; - if (existing.refCount == 0) { - existing.hub.unlisten(Events.CACHE, - existing.listener); - return null; - } return existing; }); this.holder = null; @@ -189,6 +184,16 @@ private void unlistenChanges() { } } + public static void closeGraph(HugeGraphParams params) { + SCHEMA_CACHE_EVENT_LISTENERS.computeIfPresent(params.spaceGraphName(), (key, existing) -> { + if (existing.hub != params.schemaEventHub()) { + return existing; + } + existing.close(); + return null; + }); + } + private void notifyChanges(String action, HugeType type, Id id) { EventHub graphEventHub = this.params().schemaEventHub(); graphEventHub.notifyExcept(Events.CACHE, this.cacheEventListener, diff --git a/hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/backend/cache/CachedSchemaTransactionV2.java b/hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/backend/cache/CachedSchemaTransactionV2.java index 6ce6ae7db9..89c390718d 100644 --- a/hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/backend/cache/CachedSchemaTransactionV2.java +++ b/hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/backend/cache/CachedSchemaTransactionV2.java @@ -23,6 +23,7 @@ import java.util.Set; import java.util.UUID; import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.ConcurrentMap; import java.util.concurrent.atomic.AtomicBoolean; import java.util.function.Consumer; @@ -76,6 +77,10 @@ public class CachedSchemaTransactionV2 extends SchemaTransactionV2 { private static final String SCHEMA_CACHE_CLEAR_SOURCE = UUID.randomUUID().toString(); + private static final ConcurrentMap + SCHEMA_CACHE_EVENT_LISTENERS = new ConcurrentHashMap<>(); + + private CacheListenerHolder holder; private final Cache idCache; private final Cache nameCache; @@ -113,8 +118,8 @@ private static Id generateId(HugeType type, String name) { return IdGenerator.of(type.string() + "-" + name); } + @Override public void close() { - this.clearCache(false); this.unlistenChanges(); } @@ -166,10 +171,9 @@ private void listenChanges() { } return false; }; - this.graphParams().loadGraphStore().provider().listen(this.storeEventListener); // Listen cache event: "cache"(invalid cache item) - this.cacheEventListener = event -> { + EventListener listener = event -> { LOG.debug("Graph {} received schema cache event: {}", this.graph(), event); Object[] args = event.args(); @@ -203,11 +207,28 @@ private void listenChanges() { return false; }; EventHub schemaEventHub = this.graphParams().schemaEventHub(); - if (!schemaEventHub.containsListener(Events.CACHE)) { - schemaEventHub.listen(Events.CACHE, this.cacheEventListener); - } - listenSchemaCacheClear(); + CacheListenerHolder acquired = SCHEMA_CACHE_EVENT_LISTENERS.compute( + this.graphParams().spaceGraphName(), (key, existing) -> { + if (existing == null || existing.hub != schemaEventHub) { + if (existing != null) { + existing.close(); + } + schemaEventHub.listen(Events.CACHE, listener); + this.graphParams().loadGraphStore().provider().listen(this.storeEventListener); + return new CacheListenerHolder(listener, schemaEventHub, () -> { + try { + this.graphParams().loadGraphStore().provider().unlisten(this.storeEventListener); + } finally { + this.clearCache(false); + } + }); + } + existing.refCount++; + return existing; + }); + this.holder = acquired; + this.cacheEventListener = acquired.listener; } private static void listenSchemaCacheClear() { @@ -278,13 +299,27 @@ private void resetCachedAllIfReachedCapacity() { } private void unlistenChanges() { - // Unlisten store event - this.graphParams().loadGraphStore().provider() - .unlisten(this.storeEventListener); + CacheListenerHolder ours = this.holder; + if (ours != null) { + SCHEMA_CACHE_EVENT_LISTENERS.compute(this.graphParams().spaceGraphName(), (key, existing) -> { + if (existing == ours) { + existing.refCount--; + } + return existing; + }); + this.holder = null; + this.cacheEventListener = null; + } + } - // Unlisten cache event - EventHub schemaEventHub = this.graphParams().schemaEventHub(); - schemaEventHub.unlisten(Events.CACHE, this.cacheEventListener); + public static void closeGraph(HugeGraphParams params) { + SCHEMA_CACHE_EVENT_LISTENERS.computeIfPresent(params.spaceGraphName(), (key, existing) -> { + if (existing.hub != params.schemaEventHub()) { + return existing; + } + existing.close(); + return null; + }); } private CachedTypes cachedTypes() { diff --git a/hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/backend/tx/GraphIndexTransaction.java b/hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/backend/tx/GraphIndexTransaction.java index b333c929df..5b5289dd59 100644 --- a/hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/backend/tx/GraphIndexTransaction.java +++ b/hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/backend/tx/GraphIndexTransaction.java @@ -148,14 +148,24 @@ public void updateVertexIndex(HugeVertex vertex, boolean removed) { return; } // Update index(only property, no edge) of a vertex - IndexBuilder.forEachIndexLabelId(vertex.element(), this.graph()::edgeLabel, - id -> this.updateIndex(id, vertex, removed)); + for (Id id : this.indexLabelIds(vertex.schemaLabel())) { + this.updateIndex(id, vertex, removed); + } } @Watched(prefix = "index") public void updateEdgeIndex(HugeEdge edge, boolean removed) { - IndexBuilder.forEachIndexLabelId(edge.element(), this.graph()::edgeLabel, - id -> this.updateIndex(id, edge, removed)); + // Update index of an edge + for (Id id : this.indexLabelIds(edge.schemaLabel())) { + this.updateIndex(id, edge, removed); + } + + EdgeLabel label = edge.schemaLabel(); + if (label.hasFather()) { + for (Id id : graph().edgeLabel(label.fatherId()).indexLabels()) { + this.updateIndex(id, edge, removed); + } + } } private void updateVertexOlapIndex(HugeVertex vertex, boolean removed) { @@ -177,6 +187,9 @@ protected void updateIndex(Id ilId, HugeElement element, boolean removed) { E.checkArgument(indexLabel != null, "Not exist index label with id '%s'", ilId); + if (!(element instanceof HugeVertex && ((HugeVertex) element).olap())) { + element.element().schemaLabel(this.currentSchemaLabel(element.schemaLabel())); + } this.indexBuilder.forEachIndex(element.element(), indexLabel, index -> { Object value = index.fieldValues(); if (indexLabel.indexType().isUnique() && !removed && @@ -1386,9 +1399,23 @@ private static void validateIndexLabel(IndexLabel indexLabel) { indexLabel, indexLabel.status()); } - private static Set relatedIndexLabels(HugeElement element) { + Set indexLabelIds(SchemaLabel label) { + return this.currentSchemaLabel(label).indexLabels(); + } + + private SchemaLabel currentSchemaLabel(SchemaLabel label) { + // Elements can outlive schema cache eviction. Resolve the current index + // membership rather than using the schema object retained by an element. + ISchemaTransaction schema = this.params().schemaTransaction(); + SchemaLabel current = label.type() == HugeType.VERTEX_LABEL ? + schema.getVertexLabel(label.id()) : schema.getEdgeLabel(label.id()); + // Missing schema is represented by an undefined label so orphan records remain removable. + return current != null ? current : label; + } + + private Set relatedIndexLabels(HugeElement element) { Set indexLabels = InsertionOrderUtil.newSet(); - Set indexLabelIds = element.schemaLabel().indexLabels(); + Set indexLabelIds = this.indexLabelIds(element.schemaLabel()); for (Id id : indexLabelIds) { IndexLabel indexLabel = element.graph().indexLabel(id); @@ -1643,7 +1670,7 @@ private long processSecondaryOrSearchIndexLeft(ConditionQuery query, // Delete unused index long count = 0; Set incorrectPkIds; - for (IndexLabel il : relatedIndexLabels(deletion)) { + for (IndexLabel il : this.tx.relatedIndexLabels(deletion)) { incorrectPkIds = incorrectPKs.keySet().stream() .map(PropertyKey::id) .collect(Collectors.toSet()); diff --git a/hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/backend/tx/GraphTransaction.java b/hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/backend/tx/GraphTransaction.java index e807b74fb3..70487f159e 100644 --- a/hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/backend/tx/GraphTransaction.java +++ b/hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/backend/tx/GraphTransaction.java @@ -370,7 +370,12 @@ protected void prepareDeletions(Map removedVertices, Map removedEdges) { // Remove related edges of each vertex for (HugeVertex v : removedVertices.values()) { - if (!v.schemaLabel().existsLinkLabel()) { + // System edge labels may be absent from schema enumeration after + // request cleanup, so always scan auth/system vertex relations. + // OLAP shares its ID with a base vertex and must keep its original + // path rather than scan that base vertex's ordinary edges. + VertexLabel label = v.schemaLabel(); + if ((!label.system() || label.olap()) && !label.existsLinkLabel()) { continue; } // Query all edges of the vertex and remove them @@ -638,7 +643,7 @@ public HugeVertex addVertex(HugeVertex vertex) { this.locksTable.lockReads(LockUtil.VERTEX_LABEL_DELETE, vertex.schemaLabel().id()); this.locksTable.lockReads(LockUtil.INDEX_LABEL_DELETE, - vertex.schemaLabel().indexLabels()); + this.indexTx.indexLabelIds(vertex.schemaLabel())); // Ensure vertex label still exists from vertex-construct to lock this.graph().vertexLabel(vertex.schemaLabel().id()); /* @@ -711,7 +716,7 @@ public void removeVertex(HugeVertex vertex) { // Override vertices in local `addedVertices` this.addedVertices.remove(vertex.id()); // Force load vertex to ensure all properties are loaded (refer to #2181) - if (!vertex.schemaLabel().indexLabels().isEmpty()) { + if (!this.indexTx.indexLabelIds(vertex.schemaLabel()).isEmpty()) { vertex.forceLoad(); } // Collect the removed vertex @@ -905,7 +910,7 @@ public HugeEdge addEdge(HugeEdge edge) { this.locksTable.lockReads(LockUtil.EDGE_LABEL_DELETE, edge.schemaLabel().id()); this.locksTable.lockReads(LockUtil.INDEX_LABEL_DELETE, - edge.schemaLabel().indexLabels()); + this.indexTx.indexLabelIds(edge.schemaLabel())); // Ensure edge label still exists from edge-construct to lock this.graph().edgeLabel(edge.schemaLabel().id()); /* @@ -1844,7 +1849,7 @@ private void lockForUpdateProperty(SchemaLabel schemaLabel, Id pkey = prop.propertyKey().id(); Set indexIds = new HashSet<>(); - for (Id il : schemaLabel.indexLabels()) { + for (Id il : this.indexTx.indexLabelIds(schemaLabel)) { if (graph().indexLabel(il).indexFields().contains(pkey)) { indexIds.add(il); } diff --git a/hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/util/FixedTimerWindowRateLimiter.java b/hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/util/FixedTimerWindowRateLimiter.java index 0163656d5b..7b08204bbb 100644 --- a/hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/util/FixedTimerWindowRateLimiter.java +++ b/hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/util/FixedTimerWindowRateLimiter.java @@ -34,7 +34,11 @@ public class FixedTimerWindowRateLimiter implements RateLimiter { private final int limit; public FixedTimerWindowRateLimiter(int limitPerSecond) { - this.timer = new Timer("RateAuditLog", true); + this(limitPerSecond, new Timer("RateAuditLog", true)); + } + + FixedTimerWindowRateLimiter(int limitPerSecond, Timer timer) { + this.timer = timer; this.count = new LongAdder(); this.limit = limitPerSecond; // Count will be reset if hit limit (run once per 1000ms) @@ -45,7 +49,7 @@ public void run() { count.reset(); } } - }, 0L, RESET_PERIOD); + }, RESET_PERIOD, RESET_PERIOD); } @Override diff --git a/hugegraph-server/hugegraph-dist/src/assembly/static/bin/dump-store.sh b/hugegraph-server/hugegraph-dist/src/assembly/static/bin/dump-store.sh index 14b6ac1f79..f612557e82 100755 --- a/hugegraph-server/hugegraph-dist/src/assembly/static/bin/dump-store.sh +++ b/hugegraph-server/hugegraph-dist/src/assembly/static/bin/dump-store.sh @@ -51,5 +51,6 @@ echo "Dumping HugeGraph Store($conf)..." dump_store_ext_jar_path=$LIB/hugegraph-dist-*.jar for i in $LIB/*.jar; do dump_store_ext_jar_path=$dump_store_ext_jar_path:$i; export dump_store_ext_jar_path; done -exec "$JAVA" -cp "$dump_store_ext_jar_path" \ +source "$BIN/preload-topling.sh" || exit 1 +exec "$JAVA" -cp "${TOPLING_RUNTIME_CLASSPATH:+$TOPLING_RUNTIME_CLASSPATH:}$dump_store_ext_jar_path" \ org.apache.hugegraph.cmd.StoreDumper "$@" diff --git a/hugegraph-server/hugegraph-dist/src/assembly/static/bin/hugegraph-server.sh b/hugegraph-server/hugegraph-dist/src/assembly/static/bin/hugegraph-server.sh index 2c93aa085c..b0ffaa1f43 100644 --- a/hugegraph-server/hugegraph-dist/src/assembly/static/bin/hugegraph-server.sh +++ b/hugegraph-server/hugegraph-dist/src/assembly/static/bin/hugegraph-server.sh @@ -86,7 +86,8 @@ CP="$CP":$(find -L $PLUGINS -name '*.jar' | sort | tr '\n' ':') # (Cygwin only) Use ; classpath separator and reformat paths for Windows ("C:\foo") [[ $(uname) = CYGWIN* ]] && CP="$(cygpath -p -w "$CP")" -export CLASSPATH="${CLASSPATH:-}:$CP" +source "$BIN/preload-topling.sh" || exit 1 +export CLASSPATH="${TOPLING_RUNTIME_CLASSPATH:+$TOPLING_RUNTIME_CLASSPATH:}${CLASSPATH:-}:$CP" # Change to $BIN's parent cd "${TOP}" || exit 1 @@ -259,12 +260,12 @@ fi # Turn on security check if [[ "${STDOUT_MODE:-false}" == "true" ]]; then exec ${JAVA} @"${JVM_MODULE_OPTIONS}" -Dname="HugeGraphServer" ${JVM_OPTIONS} ${JAVA_OPTIONS} \ - ${SECURITY_MANAGER_OPTION} -cp ${CLASSPATH}: \ + ${SECURITY_MANAGER_OPTION} -cp "${CLASSPATH}:" \ org.apache.hugegraph.bootstrap.HugeGraphServerBootstrap \ ${OPEN_SECURITY_CHECK} ${GREMLIN_SERVER_CONF} ${REST_SERVER_CONF} else exec ${JAVA} @"${JVM_MODULE_OPTIONS}" -Dname="HugeGraphServer" ${JVM_OPTIONS} ${JAVA_OPTIONS} \ - ${SECURITY_MANAGER_OPTION} -cp ${CLASSPATH}: \ + ${SECURITY_MANAGER_OPTION} -cp "${CLASSPATH}:" \ org.apache.hugegraph.bootstrap.HugeGraphServerBootstrap \ ${OPEN_SECURITY_CHECK} ${GREMLIN_SERVER_CONF} ${REST_SERVER_CONF} \ >> ${LOGS}/hugegraph-server-stdout.log 2>&1 diff --git a/hugegraph-server/hugegraph-dist/src/assembly/static/bin/init-store.sh b/hugegraph-server/hugegraph-dist/src/assembly/static/bin/init-store.sh index d934192fad..b1eb3438fb 100755 --- a/hugegraph-server/hugegraph-dist/src/assembly/static/bin/init-store.sh +++ b/hugegraph-server/hugegraph-dist/src/assembly/static/bin/init-store.sh @@ -56,6 +56,8 @@ echo "Initializing HugeGraph Store..." CP=$(find -L "${LIB}" -name 'hugegraph*.jar' | sort | tr '\n' ':') CP="$CP":$(find -L "${LIB}" -name '*.jar' \! -name 'hugegraph*' | sort | tr '\n' ':') CP="$CP":$(find -L "${PLUGINS}" -name '*.jar' | sort | tr '\n' ':') +source "$BIN/preload-topling.sh" || exit 1 +CP="${TOPLING_RUNTIME_CLASSPATH:+$TOPLING_RUNTIME_CLASSPATH:}$CP" "${JAVA}" @"${JVM_MODULE_OPTIONS}" -cp "$CP" \ org.apache.hugegraph.cmd.InitStore "${CONF}"/rest-server.properties INIT_STORE_STATUS=$? diff --git a/hugegraph-server/hugegraph-dist/src/assembly/static/bin/preload-topling.sh b/hugegraph-server/hugegraph-dist/src/assembly/static/bin/preload-topling.sh new file mode 100755 index 0000000000..3f88216368 --- /dev/null +++ b/hugegraph-server/hugegraph-dist/src/assembly/static/bin/preload-topling.sh @@ -0,0 +1,102 @@ +#!/bin/bash +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# Sourced by launchers. Do not change the caller's shell flags or business config. +topling_remove_path() { + local value="$1" remove="$2" preserve_empty="${3:-false}" entry result="" separator="" + while true; do + entry="${value%%:*}" + if [ "$entry" != "$remove" ] && { [ -n "$entry" ] || [ "$preserve_empty" = true ]; }; then + result="$result$separator$entry" + separator=: + fi + [[ "$value" = *:* ]] || break + value="${value#*:}" + done + printf '%s\n' "$result" +} + +topling_select_runtime() { + local top provider native jar conf inherited entry + local -a preloads + top="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" || return 1 + provider="${TOPLINGDB_ROCKSDB_PROVIDER:-rocksdb}" + case "$provider" in rocksdb | topling) ;; *) + echo "Error: invalid TOPLINGDB_ROCKSDB_PROVIDER '$provider'" >&2; return 1 ;; + esac + if [ -n "${TOPLING_ACTIVE_NATIVE:-}" ]; then + LD_PRELOAD=$(topling_remove_path "${LD_PRELOAD:-}" "$TOPLING_ACTIVE_NATIVE") + LD_LIBRARY_PATH=$(topling_remove_path "${LD_LIBRARY_PATH:-}" "$(dirname "$TOPLING_ACTIVE_NATIVE")") + export LD_PRELOAD LD_LIBRARY_PATH + fi + if [ -n "${TOPLING_RUNTIME_CLASSPATH:-}" ] && [ "${CLASSPATH+x}" = x ]; then + CLASSPATH=$(topling_remove_path "$CLASSPATH" "$TOPLING_RUNTIME_CLASSPATH" true) + export CLASSPATH + fi + unset TOPLING_ACTIVE_NATIVE TOPLING_RUNTIME_CLASSPATH + if [ -n "${TOPLING_AUTO_MIGRATE_CONF:-}" ] && + [ "${TOPLINGDB_EASY_MIGRATE_CONF:-}" = "$TOPLING_AUTO_MIGRATE_CONF" ]; then + unset TOPLINGDB_EASY_MIGRATE_CONF + fi + unset TOPLING_AUTO_MIGRATE_CONF + native="$top/topling/librocksdbjni-linux64.so" + # Two RocksDB JNI libraries can conflict before the JVM even starts. + inherited="${LD_PRELOAD:-}" + inherited="${inherited//:/ }" + inherited="${inherited//$'\n'/ }" + read -r -a preloads <<< "$inherited" + for entry in "${preloads[@]-}"; do + case "${entry##*/}" in librocksdbjni*.so* | libfrocksdbjni*.so*) + if [ "$provider" != topling ] || [ "$entry" != "$native" ]; then + echo "Error: competing RocksDB JNI in LD_PRELOAD: $entry" >&2 + return 1 + fi ;; + esac + done + [ "$provider" = topling ] || return 0 + [ "$(uname -s)/$(uname -m)" = Linux/x86_64 ] || { + echo 'Error: Topling requires Linux x86_64' >&2; return 1; + } + jar="$top/topling/rocksdbjni.jar" + [ -r "$jar" ] && [ -r "$native" ] || { + echo "Error: run $top/bin/prepare-topling.sh before enabling Topling" >&2; return 1; + } + [ -r "$top/topling/easy-migrate.sha256" ] && + (cd "$top/topling" && sha256sum -c easy-migrate.sha256 >/dev/null) || { + echo "Error: unverified EasyMigrate runtime; prepare a fresh distribution with $top/bin/prepare-topling.sh" >&2 + return 1 + } + conf="${TOPLINGDB_EASY_MIGRATE_CONF:-}" + if [ -z "$conf" ]; then + conf="$top/conf/toplingdb.yaml" + export TOPLING_AUTO_MIGRATE_CONF="$conf" + fi + [ -r "$conf" ] || { echo "Error: unreadable Easy Migrate configuration: $conf" >&2; return 1; } + export TOPLINGDB_EASY_MIGRATE_CONF="$conf" + export LD_LIBRARY_PATH="$top/topling${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}" + # The selected JNI must precede inherited preloads during native symbol lookup. + export LD_PRELOAD="$native${LD_PRELOAD:+:$LD_PRELOAD}" + export TOPLING_ACTIVE_NATIVE="$native" TOPLING_RUNTIME_CLASSPATH="$jar" + echo "Selected Topling runtime: $jar; configuration: $conf" +} + +if topling_select_runtime; then + unset -f topling_select_runtime topling_remove_path +else + unset -f topling_select_runtime topling_remove_path + return 1 2>/dev/null || exit 1 +fi diff --git a/hugegraph-server/hugegraph-dist/src/assembly/static/bin/prepare-topling.sh b/hugegraph-server/hugegraph-dist/src/assembly/static/bin/prepare-topling.sh new file mode 100755 index 0000000000..847924c464 --- /dev/null +++ b/hugegraph-server/hugegraph-dist/src/assembly/static/bin/prepare-topling.sh @@ -0,0 +1,126 @@ +#!/bin/bash +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# Run only after stopping this component. Runtime preparation never changes data. +set -Eeuo pipefail +TOP="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +[ "$(uname -s)/$(uname -m)" = Linux/x86_64 ] || { + echo 'Error: Topling requires Linux x86_64' >&2; exit 1; +} +[[ "${TOPLING_JNI_JAR:-}" = /* ]] && [ -r "$TOPLING_JNI_JAR" ] || { + echo 'Error: TOPLING_JNI_JAR must be an absolute readable file' >&2; exit 1; +} +[[ "${TOPLING_JNI_SHA256:-}" =~ ^[[:xdigit:]]{64}$ ]] || { + echo 'Error: TOPLING_JNI_SHA256 must contain 64 hexadecimal characters' >&2; exit 1; +} +[ ! -e "$TOP/topling" ] && [ ! -L "$TOP/topling" ] || { + echo 'Error: runtime already prepared; use a fresh stopped distribution to replace it' >&2; exit 1; +} +staging=$(mktemp -d "$TOP/.topling-prepare.XXXXXX") +trap 'rm -rf "$staging"' EXIT +cp "$TOPLING_JNI_JAR" "$staging/rocksdbjni.jar" +expected=$(printf '%s' "$TOPLING_JNI_SHA256" | tr 'A-F' 'a-f') +actual=$(sha256sum "$staging/rocksdbjni.jar" | awk '{print $1}') +[ "$actual" = "$expected" ] || { echo 'Error: Topling JNI SHA-256 mismatch' >&2; exit 1; } +entries=$(unzip -Z1 "$staging/rocksdbjni.jar") +[ "$(grep -cx 'org/rocksdb/SidePluginRepo.class' <<< "$entries" || true)" = 1 ] && +[ "$(grep -cx 'librocksdbjni-linux64.so' <<< "$entries" || true)" = 1 ] && +[ "$(grep -Ec '(^|/)librocksdbjni-linux64[.]so$' <<< "$entries" || true)" = 1 ] || { + echo 'Error: expected Topling Java marker and exactly one Linux x86_64 JNI entry' >&2; exit 1; +} +unzip -p "$staging/rocksdbjni.jar" librocksdbjni-linux64.so > "$staging/librocksdbjni-linux64.so" +header=$(od -An -tx1 -N20 "$staging/librocksdbjni-linux64.so" | tr -d ' \n') +[[ "$header" = 7f454c460201* ]] && [ "${header:36:4}" = 3e00 ] || { + echo 'Error: JNI is not a little-endian Linux x86_64 ELF library' >&2; exit 1; +} +# Ubuntu's t64 transition renamed libaio; keep compatibility inside this component. +if [ ! -e /usr/lib/x86_64-linux-gnu/libaio.so.1 ] && + [ -e /usr/lib/x86_64-linux-gnu/libaio.so.1t64 ]; then + ln -s /usr/lib/x86_64-linux-gnu/libaio.so.1t64 "$staging/libaio.so.1" +fi +ldd_output=$(LD_LIBRARY_PATH="$staging${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}" \ + ldd "$staging/librocksdbjni-linux64.so") +if grep -q 'not found' <<< "$ldd_output"; then + printf 'Error: install the missing native dependencies first:\n%s\n' "$ldd_output" >&2 + exit 1 +fi +# Verify EasyMigrate behavior in a disposable DB before touching business data. +# Class markers and hashes identify a library, but do not prove this capability. +cat > "$staging/probe.yaml" <<'YAML' +http: {auto_start_http: false} +CFOptions: + default: + write_buffer_size: 17M +DBOptions: + default: + create_if_missing: true + memtable_as_log_index: false +YAML +cat > "$staging/ToplingRuntimeProbe.java" <<'JAVA' +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.regex.Pattern; +import org.rocksdb.Options; +import org.rocksdb.RocksDB; + +class ToplingRuntimeProbe { + public static void main(String[] args) throws Exception { + if (!"17".equals(System.getProperty("java.specification.version"))) { + throw new IllegalStateException("Prepare Topling with Java 17"); + } + RocksDB.loadLibrary(); + Path database = Path.of(args[0]); + try (Options options = new Options().setCreateIfMissing(true); + RocksDB db = RocksDB.open(options, database.toString())) { + db.put(new byte[]{1}, new byte[]{2}); + if (db.get(new byte[]{1})[0] != 2) { + throw new IllegalStateException("Topling capability probe could not read its write"); + } + } + Pattern configured = Pattern.compile("(?m)^\\s*write_buffer_size\\s*=\\s*17825792\\s*$"); + boolean applied = false; + try (var paths = Files.list(database)) { + for (Path path : paths.filter(p -> p.getFileName().toString().startsWith("OPTIONS-")).toList()) { + applied |= configured.matcher(Files.readString(path)).find(); + } + } + if (!applied) { + throw new IllegalStateException("Selected JNI does not apply EasyMigrate configuration"); + } + } +} +JAVA +if ! LD_LIBRARY_PATH="$staging${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}" \ + LD_PRELOAD="$staging/librocksdbjni-linux64.so" \ + TOPLINGDB_EASY_MIGRATE_CONF="$staging/probe.yaml" \ + java --add-opens=java.base/java.nio=ALL-UNNAMED --source 17 \ + --class-path "$staging/rocksdbjni.jar" "$staging/ToplingRuntimeProbe.java" "$staging/probe-db"; then + echo 'Error: Java 17 JDK / Topling EasyMigrate capability probe failed' >&2 + exit 1 +fi +native_hash=$(sha256sum "$staging/librocksdbjni-linux64.so" | awk '{print $1}') +printf '%s rocksdbjni.jar\n%s librocksdbjni-linux64.so\n' "$actual" "$native_hash" > "$staging/easy-migrate.sha256" +rm -rf "$staging/probe-db" "$staging/probe.yaml" "$staging/ToplingRuntimeProbe.java" +mkdir "$staging/rocksdb_resource" +status=0 +unzip -j "$staging/rocksdbjni.jar" '*.html' '*.css' -d "$staging/rocksdb_resource" >/dev/null 2>&1 || status=$? +[ "$status" = 0 ] || [ "$status" = 11 ] || { echo 'Error: cannot extract native web resources' >&2; exit 1; } +# Publish readable runtime assets for a service UID different from the preparer. +find "$staging" -type d -exec chmod 0755 {} + +find "$staging" -type f -exec chmod 0644 {} + +mv -T "$staging" "$TOP/topling" +echo "Prepared component-local Topling runtime: $TOP/topling" diff --git a/hugegraph-server/hugegraph-dist/src/assembly/static/conf/toplingdb.yaml b/hugegraph-server/hugegraph-dist/src/assembly/static/conf/toplingdb.yaml new file mode 100644 index 0000000000..3067b378f1 --- /dev/null +++ b/hugegraph-server/hugegraph-dist/src/assembly/static/conf/toplingdb.yaml @@ -0,0 +1,157 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# common parameters +http: + # normally parent path of db path + document_root: ./topling/rocksdb_resource + listening_ports: '127.0.0.1:2011' + auto_start_http: false +Cache: + lru_cache: + class: LRUCache + params: + capacity: 8G + num_shard_bits: -1 + strict_capacity_limit: false + high_pri_pool_ratio: 0.5 + use_adaptive_mutex: false + metadata_charge_policy: kFullChargeCacheMetadata +Statistics: + stat: + class: default + params: + discard_tickers: + - rocksdb.block.cache + - rocksdb.block.cachecompressed + - rocksdb.block + - rocksdb.memtable.payload.bytes.at.flush + - rocksdb.memtable.garbage.bytes.at.flush + - rocksdb.txn + - rocksdb.blobdb + - rocksdb.row.cache + - rocksdb.number.block + - rocksdb.bloom.filter + - rocksdb.persistent + - rocksdb.sim.block.cache + discard_histograms: + # comment: .... + - rocksdb.blobdb + - rocksdb.bytes.compressed + - rocksdb.bytes.decompressed + - rocksdb.num.index.and.filter.blocks.read.per.level + - rocksdb.num.data.blocks.read.per.level + - rocksdb.compression.times.nanos + - rocksdb.decompression.times.nanos + - rocksdb.read.block.get.micros + - rocksdb.write.raw.block.micros + # comment end of array + #stats_level: kAll + stats_level: kDisableAll +MemTableRepFactory: + cspp: + class: cspp + params: + mem_cap: 16G + use_vm: false + token_use_idle: true + chunk_size: 16K + convert_to_sst: kFileMmap + sync_sst_file: false + skiplist: + class: SkipList + params: + lookahead: 0 +TableFactory: + cspp_memtab_sst: + class: CSPPMemTabTable + params: # empty params + bb: + class: BlockBasedTable + params: + checksum: kCRC32c + block_size: 4K + block_restart_interval: 16 + index_block_restart_interval: 1 + metadata_block_size: 4K + enable_index_compression: true + block_cache: "${lru_cache}" + readers: + BlockBasedTable: bb + CSPPMemTabTable: cspp_memtab_sst + block_cache_compressed: + persistent_cache: + filter_policy: + dispatch: + class: DispatcherTable + params: + default: bb + readers: + BlockBasedTable: bb + CSPPMemTabTable: cspp_memtab_sst + level_writers: [ bb, bb, bb, bb, bb, bb ] +CFOptions: + default: + max_write_buffer_number: 6 + memtable_factory: "${cspp}" + write_buffer_size: 128M + # set target_file_size_base as small as 512K is to make many SST files, + # thus key prefix cache can present efficiency + target_file_size_base: 64M + target_file_size_multiplier: 1 + table_factory: dispatch + max_bytes_for_level_base: 512M + max_bytes_for_level_multiplier: 10 + level_compaction_dynamic_level_bytes: false + level0_slowdown_writes_trigger: 20 + level0_stop_writes_trigger: 36 + level0_file_num_compaction_trigger: 2 + merge_operator: uint64add # support merge + level_compaction_dynamic_file_size: true + optimize_filters_for_hits: true + allow_merge_memtables: true + min_write_buffer_number_to_merge: 2 + compression_per_level: + - kNoCompression + - kNoCompression + - kSnappyCompression + - kSnappyCompression + - kSnappyCompression + - kSnappyCompression + - kSnappyCompression +DBOptions: + log: + create_if_missing: true + create_missing_column_families: true + default: + create_if_missing: true + create_missing_column_families: false # this is important, must be false to hugegraph + max_background_compactions: -1 + max_subcompactions: 4 + max_level1_subcompactions: 0 + inplace_update_support: false + WAL_size_limit_MB: 0 + statistics: "${stat}" + max_manifest_file_size: 100M + max_background_jobs: 8 + # Java org.rocksdb.WriteBatch has no Topling mmap WAL. With + # memtable_as_log_index enabled, every batch write fails: + # "WriteBatch has no mmap wal". Keep the option false for this client. + # convert_to_sst is still Topling-specific. Do not open data written by + # this profile with the standard RocksDB provider; restore a snapshot + # taken before migration instead. + compaction_readahead_size: 0 + memtable_as_log_index: false diff --git a/hugegraph-server/hugegraph-dist/src/assembly/travis/test-java17-upgrade-contracts.sh b/hugegraph-server/hugegraph-dist/src/assembly/travis/test-java17-upgrade-contracts.sh index b371003713..926a96de22 100755 --- a/hugegraph-server/hugegraph-dist/src/assembly/travis/test-java17-upgrade-contracts.sh +++ b/hugegraph-server/hugegraph-dist/src/assembly/travis/test-java17-upgrade-contracts.sh @@ -166,10 +166,11 @@ MODULE_OPTIONS="${SERVER_ROOT}/bin/jvm-module.options" SERVER_SCRIPT="${SERVER_ROOT}/bin/hugegraph-server.sh" INIT_STORE_SCRIPT="${SERVER_ROOT}/bin/init-store.sh" UTIL_SCRIPT="${SERVER_ROOT}/bin/util.sh" +PRELOAD_SCRIPT="${SERVER_ROOT}/bin/preload-topling.sh" CONF_SOURCE="${SERVER_ROOT}/conf" for source_file in "$MODULE_OPTIONS" "$SERVER_SCRIPT" \ - "$INIT_STORE_SCRIPT" "$UTIL_SCRIPT"; do + "$INIT_STORE_SCRIPT" "$UTIL_SCRIPT" "$PRELOAD_SCRIPT"; do [[ -f "$source_file" ]] || fail "runtime asset is missing: $source_file" done [[ -d "$CONF_SOURCE" ]] || fail "server conf is missing: $CONF_SOURCE" @@ -182,7 +183,7 @@ MOCK_JAVA_HOME="${TEMP_DIR}/mock-java-home" mkdir -p "${DIST_ROOT}"/{bin,conf,ext,lib,logs,plugins} \ "${MOCK_JAVA_HOME}/bin" cp "$MODULE_OPTIONS" "$SERVER_SCRIPT" "$INIT_STORE_SCRIPT" "$UTIL_SCRIPT" \ - "${DIST_ROOT}/bin/" + "$PRELOAD_SCRIPT" "${DIST_ROOT}/bin/" cp -R "${CONF_SOURCE}/." "${DIST_ROOT}/conf/" # Model a full pre-Phase-2 conf/ directory: it has no module argfile. Both diff --git a/hugegraph-server/hugegraph-dist/src/assembly/travis/test-topling-selection.sh b/hugegraph-server/hugegraph-dist/src/assembly/travis/test-topling-selection.sh new file mode 100755 index 0000000000..de9624cf1f --- /dev/null +++ b/hugegraph-server/hugegraph-dist/src/assembly/travis/test-topling-selection.sh @@ -0,0 +1,128 @@ +#!/bin/bash +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# Selection-only fixtures; real JNI preparation and live services are separate gates. +set -Eeuo pipefail +SOURCE_BIN="$(cd "$(dirname "${BASH_SOURCE[0]}")/../static/bin" && pwd)" +TEST_ROOT=$(mktemp -d) +trap 'rm -rf "$TEST_ROOT"' EXIT +mkdir -p "$TEST_ROOT/fake-bin" +printf '%s\n' '#!/bin/sh' 'case "$1" in -s) echo Linux ;; -m) echo x86_64 ;; esac' > "$TEST_ROOT/fake-bin/uname" +chmod +x "$TEST_ROOT/fake-bin/uname" +for component in server pd store; do + mkdir -p "$TEST_ROOT/$component/bin" "$TEST_ROOT/$component/conf" + cp "$SOURCE_BIN/preload-topling.sh" "$TEST_ROOT/$component/bin/" + # Default startup works without prepared assets and never selects optional JNI. + env -u TOPLINGDB_ROCKSDB_PROVIDER -u LD_PRELOAD -u TOPLING_ACTIVE_NATIVE \ + -u TOPLING_RUNTIME_CLASSPATH bash -euc 'source "$1"; test -z "${TOPLING_RUNTIME_CLASSPATH:-}"' \ + _ "$TEST_ROOT/$component/bin/preload-topling.sh" + mkdir "$TEST_ROOT/$component/topling" + touch "$TEST_ROOT/$component/topling/rocksdbjni.jar" "$TEST_ROOT/$component/topling/librocksdbjni-linux64.so" + (cd "$TEST_ROOT/$component/topling" && sha256sum rocksdbjni.jar librocksdbjni-linux64.so > easy-migrate.sha256) + printf '%s\n' 'http: {auto_start_http: false}' > "$TEST_ROOT/$component/conf/toplingdb.yaml" +done +for order in 'pd store' 'store pd' 'server pd'; do + read -r first second <<< "$order" + PATH="$TEST_ROOT/fake-bin:$PATH" bash -ec ' + export TOPLINGDB_ROCKSDB_PROVIDER=topling + unset TOPLINGDB_EASY_MIGRATE_CONF TOPLING_AUTO_MIGRATE_CONF + export LD_LIBRARY_PATH=/unrelated CLASSPATH=unrelated.jar + source "$1/$2/bin/preload-topling.sh" + bash -ec '"'"'source "$1/$2/bin/preload-topling.sh" + test "$TOPLINGDB_EASY_MIGRATE_CONF" = "$1/$2/conf/toplingdb.yaml" + '"'"' _ "$1" "$3" + # Reproduce the Server launcher exporting its selected JAR to children. + export CLASSPATH="$TOPLING_RUNTIME_CLASSPATH::unrelated.jar::" + TOPLINGDB_ROCKSDB_PROVIDER=rocksdb bash -ec '"'"'source "$1/$2/bin/preload-topling.sh" + test "$CLASSPATH" = :unrelated.jar:: + test -z "${TOPLING_RUNTIME_CLASSPATH:-}" + '"'"' _ "$1" "$3" + source "$1/$3/bin/preload-topling.sh" + test "$CLASSPATH" = :unrelated.jar:: + test "$TOPLING_RUNTIME_CLASSPATH" = "$1/$3/topling/rocksdbjni.jar" + test "$TOPLINGDB_EASY_MIGRATE_CONF" = "$1/$3/conf/toplingdb.yaml" + export TOPLINGDB_EASY_MIGRATE_CONF="$1/$2/conf/toplingdb.yaml" + source "$1/$3/bin/preload-topling.sh" + test "$TOPLINGDB_EASY_MIGRATE_CONF" = "$1/$2/conf/toplingdb.yaml" + export CLASSPATH="$TOPLING_RUNTIME_CLASSPATH:$CLASSPATH" + export TOPLINGDB_ROCKSDB_PROVIDER=rocksdb + source "$1/$2/bin/preload-topling.sh" + test -z "${TOPLING_RUNTIME_CLASSPATH:-}" + test "$TOPLINGDB_EASY_MIGRATE_CONF" = "$1/$2/conf/toplingdb.yaml" + test "$LD_LIBRARY_PATH" = /unrelated + test "$CLASSPATH" = :unrelated.jar:: + ' _ "$TEST_ROOT" "$first" "$second" +done +# LD_PRELOAD accepts colon and space separators. Keep caller entries after our JNI. +for inherited in '/unrelated/liballoc.so:/unrelated/libtrace.so' \ + '/unrelated/liballoc.so /unrelated/libtrace.so'; do + PATH="$TEST_ROOT/fake-bin:$PATH" bash -ec ' + unset TOPLING_ACTIVE_NATIVE TOPLING_RUNTIME_CLASSPATH + export TOPLINGDB_ROCKSDB_PROVIDER=topling LD_PRELOAD="$2" + source "$1/server/bin/preload-topling.sh" + test "$LD_PRELOAD" = "$1/server/topling/librocksdbjni-linux64.so:$2" + # Switching components must replace only the runtime we previously selected. + source "$1/pd/bin/preload-topling.sh" + test "$LD_PRELOAD" = "$1/pd/topling/librocksdbjni-linux64.so:$2" + export TOPLINGDB_ROCKSDB_PROVIDER=rocksdb + source "$1/store/bin/preload-topling.sh" + test "$LD_PRELOAD" = "$2" + ' _ "$TEST_ROOT" "$inherited" +done +# Reject competing JNI before adding the selected runtime; unrelated entries survive. +for provider in unset rocksdb topling; do + for inherited in '/other/librocksdbjni.so:/unrelated/libtrace.so' \ + '/unrelated/libtrace.so /other/librocksdbjni-linux64.so' \ + '/other/libfrocksdbjni.so' "$TEST_ROOT/server/topling/librocksdbjni-linux64.so"; do + if [ "$provider" = topling ] && [ "$inherited" = "$TEST_ROOT/server/topling/librocksdbjni-linux64.so" ]; then + continue + fi + if PATH="$TEST_ROOT/fake-bin:$PATH" bash -ec ' + unset TOPLING_ACTIVE_NATIVE TOPLING_RUNTIME_CLASSPATH + export LD_PRELOAD="$2" + if [ "$3" = unset ]; then + unset TOPLINGDB_ROCKSDB_PROVIDER + else + export TOPLINGDB_ROCKSDB_PROVIDER="$3" + fi + source "$1/server/bin/preload-topling.sh" + ' _ "$TEST_ROOT" "$inherited" "$provider" > "$TEST_ROOT/error" 2>&1; then + echo "FAIL: competing RocksDB JNI was accepted in $provider mode" >&2; exit 1 + fi + grep -q 'competing RocksDB JNI in LD_PRELOAD' "$TEST_ROOT/error" + done +done +# Older or modified preparations must be verified again before launch. +for state in missing stale; do + receipt="$TEST_ROOT/server/topling/easy-migrate.sha256" + if [ "$state" = missing ]; then + mv "$receipt" "$receipt.saved" + else + printf 'changed' >> "$TEST_ROOT/server/topling/rocksdbjni.jar" + fi + if PATH="$TEST_ROOT/fake-bin:$PATH" TOPLINGDB_ROCKSDB_PROVIDER=topling \ + bash -ec 'source "$1"' _ "$TEST_ROOT/server/bin/preload-topling.sh" > "$TEST_ROOT/error" 2>&1; then + echo 'FAIL: unverified preparation was accepted' >&2; exit 1 + fi + grep -q 'unverified EasyMigrate runtime' "$TEST_ROOT/error" + [ "$state" != missing ] || mv "$receipt.saved" "$receipt" +done +if TOPLINGDB_ROCKSDB_PROVIDER=invalid bash -ec 'source "$1"' _ \ + "$TEST_ROOT/server/bin/preload-topling.sh" > "$TEST_ROOT/error" 2>&1; then + echo 'FAIL: invalid provider was accepted' >&2; exit 1 +fi +echo 'PASS: standard default, explicit selection, component and inherited configuration isolation' diff --git a/hugegraph-server/hugegraph-rocksdb/src/main/java/org/apache/hugegraph/backend/store/rocksdb/RocksDBOptions.java b/hugegraph-server/hugegraph-rocksdb/src/main/java/org/apache/hugegraph/backend/store/rocksdb/RocksDBOptions.java index 17148e0944..0f0d18f1a3 100644 --- a/hugegraph-server/hugegraph-rocksdb/src/main/java/org/apache/hugegraph/backend/store/rocksdb/RocksDBOptions.java +++ b/hugegraph-server/hugegraph-rocksdb/src/main/java/org/apache/hugegraph/backend/store/rocksdb/RocksDBOptions.java @@ -61,6 +61,14 @@ public static synchronized RocksDBOptions instance() { "rocksdb-data/data" ); + public static final ConfigOption PROVIDER = + new ConfigOption<>( + "rocksdb.provider", + "The RocksDB-compatible runtime provider.", + allowValues("rocksdb", "topling"), + "rocksdb" + ); + public static final ConfigListOption DATA_DISKS = new ConfigListOption<>( "rocksdb.data_disks", diff --git a/hugegraph-server/hugegraph-rocksdb/src/main/java/org/apache/hugegraph/backend/store/rocksdb/RocksDBSessions.java b/hugegraph-server/hugegraph-rocksdb/src/main/java/org/apache/hugegraph/backend/store/rocksdb/RocksDBSessions.java index 7d0576e369..7c06081f7b 100644 --- a/hugegraph-server/hugegraph-rocksdb/src/main/java/org/apache/hugegraph/backend/store/rocksdb/RocksDBSessions.java +++ b/hugegraph-server/hugegraph-rocksdb/src/main/java/org/apache/hugegraph/backend/store/rocksdb/RocksDBSessions.java @@ -17,6 +17,9 @@ package org.apache.hugegraph.backend.store.rocksdb; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collection; import java.util.List; import java.util.Set; @@ -59,6 +62,39 @@ public RocksDBSessions(HugeConfig config, String database, String store) { public abstract void forceCloseRocksDB(); + public boolean databaseOpened() { + return this.opened(); + } + + public void clearTables(Collection tables) { + Session session = (Session) this.useSession(); + try { + List>> ranges = new ArrayList<>(); + for (String table : tables) { + Pair range = session.keyRange(table); + if (range != null) { + ranges.add(Pair.of(table, range)); + } + } + + for (Pair> entry : ranges) { + String table = entry.getLeft(); + byte[] first = entry.getRight().getLeft(); + byte[] last = entry.getRight().getRight(); + if (!Arrays.equals(first, last)) { + session.deleteRange(table, first, last); + } + session.delete(table, last); + } + session.commit(); + } catch (Throwable e) { + session.rollback(); + throw e; + } finally { + this.close(); + } + } + @Override public abstract Session session(); diff --git a/hugegraph-server/hugegraph-rocksdb/src/main/java/org/apache/hugegraph/backend/store/rocksdb/RocksDBStdSessions.java b/hugegraph-server/hugegraph-rocksdb/src/main/java/org/apache/hugegraph/backend/store/rocksdb/RocksDBStdSessions.java index 467de5a273..7dca2fc0ec 100644 --- a/hugegraph-server/hugegraph-rocksdb/src/main/java/org/apache/hugegraph/backend/store/rocksdb/RocksDBStdSessions.java +++ b/hugegraph-server/hugegraph-rocksdb/src/main/java/org/apache/hugegraph/backend/store/rocksdb/RocksDBStdSessions.java @@ -43,6 +43,7 @@ import org.apache.hugegraph.util.Bytes; import org.apache.hugegraph.util.E; import org.apache.hugegraph.util.Log; +import org.apache.hugegraph.util.RocksDBRuntime; import org.apache.hugegraph.util.StringEncoding; import org.rocksdb.BlockBasedTableConfig; import org.rocksdb.BloomFilter; @@ -367,6 +368,7 @@ private void ingestExternalFile() throws RocksDBException { private static OpenedRocksDB openRocksDB(HugeConfig config, String dataPath, String walPath) throws RocksDBException { + RocksDBRuntime.verify(config.get(RocksDBOptions.PROVIDER)); // Init options Options options = new Options(); RocksDBStdSessions.initOptions(config, options, options, options, options); @@ -385,6 +387,7 @@ private static OpenedRocksDB openRocksDB(HugeConfig config, String dataPath, private static OpenedRocksDB openRocksDB(HugeConfig config, List cfNames, String dataPath, String walPath) throws RocksDBException { + RocksDBRuntime.verify(config.get(RocksDBOptions.PROVIDER)); // Old CFs should always be opened Set mergedCFs = RocksDBStdSessions.mergeOldCFs(dataPath, cfNames); @@ -731,7 +734,15 @@ public void open() { @Override public void close() { assert this.closeable(); - this.opened = false; + try { + this.batch.close(); + } finally { + try { + this.writeOptions.close(); + } finally { + this.opened = false; + } + } } @Override @@ -741,7 +752,7 @@ public boolean closed() { @Override public void reset() { - this.batch = new WriteBatch(); + this.batch.clear(); } /** @@ -782,14 +793,18 @@ public Pair keyRange(String table) { RocksIterator iter = rocksdb().newIterator(cf.get())) { iter.seekToFirst(); if (!iter.isValid()) { + iter.status(); return null; } startKey = iter.key(); iter.seekToLast(); if (!iter.isValid()) { - return Pair.of(startKey, null); + iter.status(); + throw new BackendException("Missing last key in table '%s'", table); } endKey = iter.key(); + } catch (RocksDBException e) { + throw new BackendException("Failed to read key range of table '%s'", e, table); } return Pair.of(startKey, endKey); } diff --git a/hugegraph-server/hugegraph-rocksdb/src/main/java/org/apache/hugegraph/backend/store/rocksdb/RocksDBStore.java b/hugegraph-server/hugegraph-rocksdb/src/main/java/org/apache/hugegraph/backend/store/rocksdb/RocksDBStore.java index eb7abcf6bc..535101c805 100644 --- a/hugegraph-server/hugegraph-rocksdb/src/main/java/org/apache/hugegraph/backend/store/rocksdb/RocksDBStore.java +++ b/hugegraph-server/hugegraph-rocksdb/src/main/java/org/apache/hugegraph/backend/store/rocksdb/RocksDBStore.java @@ -60,6 +60,7 @@ import org.apache.hugegraph.config.CoreOptions; import org.apache.hugegraph.config.HugeConfig; import org.apache.hugegraph.exception.ConnectionException; +import org.apache.hugegraph.exception.NotSupportException; import org.apache.hugegraph.type.HugeType; import org.apache.hugegraph.util.Consumers; import org.apache.hugegraph.util.E; @@ -103,6 +104,7 @@ public abstract class RocksDBStore extends AbstractBackendStore tableDBMap = this.tableDBMapping(); + List mainTables = new ArrayList<>(this.tableNames()); + for (String table : tableDBMap.keySet()) { + if (table.equals(HugeType.OLAP.string())) { + mainTables.removeAll(this.olapTables()); + } else { + mainTables.remove(table); + } + } + this.sessions.clearTables(mainTables); + for (Map.Entry entry : + tableDBMap.entrySet()) { + Collection tables; + if (entry.getKey().equals(HugeType.OLAP.string())) { + tables = this.olapTables(); + } else { + tables = Collections.singletonList(entry.getKey()); + } + entry.getValue().clearTables(tables); + } + } + @Override public void beginTx() { Lock readLock = this.storeLock.readLock(); @@ -1104,10 +1139,16 @@ public synchronized void init() { Lock writeLock = this.storeLock().writeLock(); writeLock.lock(); try { - RocksDBSessions.Session session = super.session(HugeType.META); - String driverVersion = this.provider().driverVersion(); - this.meta.writeVersion(session, driverVersion); - LOG.info("Write down the backend version: {}", driverVersion); + RocksDBSessions sessions = super.db(HugeType.META); + sessions.useSession(); + try { + RocksDBSessions.Session session = super.session(HugeType.META); + String driverVersion = this.provider().driverVersion(); + this.meta.writeVersion(session, driverVersion); + LOG.info("Write down the backend version: {}", driverVersion); + } finally { + sessions.close(); + } } finally { writeLock.unlock(); } diff --git a/hugegraph-server/hugegraph-test/src/main/java/org/apache/hugegraph/auth/BackendLeaseCleanupTest.java b/hugegraph-server/hugegraph-test/src/main/java/org/apache/hugegraph/auth/BackendLeaseCleanupTest.java new file mode 100644 index 0000000000..9c8909d19f --- /dev/null +++ b/hugegraph-server/hugegraph-test/src/main/java/org/apache/hugegraph/auth/BackendLeaseCleanupTest.java @@ -0,0 +1,147 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hugegraph.auth; + +import java.util.Map; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicInteger; + +import org.apache.hugegraph.HugeFactory; +import org.apache.hugegraph.auth.HugeGraphAuthProxy.Context; +import org.apache.hugegraph.auth.HugeGraphAuthProxy.ContextTask; +import org.apache.hugegraph.backend.store.rocksdb.RocksDBSessions; +import org.apache.hugegraph.backend.store.rocksdb.RocksDBStore; +import org.apache.hugegraph.core.BaseCoreTest; +import org.apache.hugegraph.testutil.Assert; +import org.apache.hugegraph.testutil.Whitebox; +import org.apache.tinkerpop.gremlin.structure.T; +import org.apache.tinkerpop.gremlin.structure.Transaction; +import org.junit.After; +import org.junit.Assume; +import org.junit.Before; +import org.junit.Test; + +public class BackendLeaseCleanupTest extends BaseCoreTest { + + @Before + public void initSchema() { + graph().schema().vertexLabel("lifecycle").useCustomizeNumberId().create(); + HugeFactory.closeCurrentThreadTransactions(); + } + + @After + public void cleanup() { + try { + HugeFactory.closeCurrentThreadTransactions(); + } finally { + HugeGraphAuthProxy.resetContext(); + } + } + + @Test + public void testFailedTaskRollsBackDespiteCommitOnClose() { + AtomicInteger rollbackEvents = new AtomicInteger(); + HugeGraphAuthProxy.setContext(Context.admin()); + IllegalStateException failure = new IllegalStateException("task failed after writing"); + ContextTask task = new ContextTask(() -> { + Transaction tx = graph().tx(); + tx.onClose(Transaction.CLOSE_BEHAVIOR.COMMIT); + tx.onReadWrite(Transaction.READ_WRITE_BEHAVIOR.MANUAL); + tx.open(); + tx.addTransactionListener(status -> rollbackEvents.incrementAndGet()); + graph().addVertex(T.label, "lifecycle", T.id, 1L); + throw failure; + }); + HugeGraphAuthProxy.resetContext(); + + Assert.assertSame(failure, Assert.assertThrows(IllegalStateException.class, task::run)); + Assert.assertFalse(graph().tx().isOpen()); + ThreadLocal backend = Whitebox.getInternalState(graph().tx(), "transactions"); + Assert.assertNull(backend.get()); + Assert.assertNull(HugeGraphAuthProxy.getContext()); + Assert.assertEquals(1, rollbackEvents.get()); + // Reuse the worker thread: AUTO and default rollback must be restored. + Assert.assertEquals(0L, graph().traversal().V().count().next().longValue()); + graph().addVertex(T.label, "lifecycle", T.id, 2L); + graph().tx().close(); + Assert.assertEquals(0L, graph().traversal().V().count().next().longValue()); + Assert.assertEquals(1, rollbackEvents.get()); + } + + @Test + public void testSuccessfulTaskKeepsExplicitCommitAndResetsThreadState() { + AtomicInteger commitEvents = new AtomicInteger(); + HugeGraphAuthProxy.setContext(Context.admin()); + ContextTask task = new ContextTask(() -> { + Transaction tx = graph().tx(); + tx.onClose(Transaction.CLOSE_BEHAVIOR.COMMIT); + tx.onReadWrite(Transaction.READ_WRITE_BEHAVIOR.MANUAL); + tx.open(); + tx.addTransactionListener(status -> commitEvents.incrementAndGet()); + graph().addVertex(T.label, "lifecycle", T.id, 1L); + tx.commit(); + }); + HugeGraphAuthProxy.resetContext(); + + task.run(); + + Assert.assertFalse(graph().tx().isOpen()); + ThreadLocal backend = Whitebox.getInternalState(graph().tx(), "transactions"); + Assert.assertNull(backend.get()); + Assert.assertNull(HugeGraphAuthProxy.getContext()); + Assert.assertEquals(1L, graph().traversal().V().count().next().longValue()); + graph().addVertex(T.label, "lifecycle", T.id, 2L); + graph().tx().close(); + Assert.assertEquals(1L, graph().traversal().V().count().next().longValue()); + Assert.assertEquals(1, commitEvents.get()); + } + @Test + public void testCommittedRocksDBWorkerReturnsBackendLeases() throws Exception { + Assume.assumeTrue("rocksdb".equals(graph().backend())); + ExecutorService worker = Executors.newSingleThreadExecutor(); + try { + worker.submit(() -> { + try { + graph().addVertex(T.label, "lifecycle", T.id, 1L); + graph().tx().commit(); + RocksDBStore store = Whitebox.getInternalState(params().graphTransaction(), "store"); + Map databases = Whitebox.getInternalState(store, "dbs"); + long threadId = Thread.currentThread().getId(); + for (RocksDBSessions sessions : databases.values()) { + Map leases = Whitebox.getInternalState(sessions, "sessions"); + Assert.assertTrue(leases.containsKey(threadId)); + } + HugeFactory.closeCurrentThreadTransactions(); + for (RocksDBSessions sessions : databases.values()) { + Map leases = Whitebox.getInternalState(sessions, "sessions"); + Assert.assertFalse(leases.containsKey(threadId)); + } + // Reuse this exact worker and confirm the committed row survives. + Assert.assertEquals(1L, graph().traversal().V().count().next().longValue()); + } finally { + HugeFactory.closeCurrentThreadTransactions(); + } + }).get(30, TimeUnit.SECONDS); + } finally { + worker.shutdownNow(); + } + } + +} diff --git a/hugegraph-server/hugegraph-test/src/main/java/org/apache/hugegraph/backend/tx/GraphIndexTransactionTest.java b/hugegraph-server/hugegraph-test/src/main/java/org/apache/hugegraph/backend/tx/GraphIndexTransactionTest.java index ed945554ac..62325f0ae9 100644 --- a/hugegraph-server/hugegraph-test/src/main/java/org/apache/hugegraph/backend/tx/GraphIndexTransactionTest.java +++ b/hugegraph-server/hugegraph-test/src/main/java/org/apache/hugegraph/backend/tx/GraphIndexTransactionTest.java @@ -29,6 +29,7 @@ import org.apache.hugegraph.backend.serializer.AbstractSerializer; import org.apache.hugegraph.backend.store.BackendEntry; import org.apache.hugegraph.backend.store.BackendStore; +import org.apache.hugegraph.id.Id; import org.apache.hugegraph.id.IdGenerator; import org.apache.hugegraph.id.EdgeId; import org.apache.hugegraph.query.ConditionQuery; @@ -230,6 +231,10 @@ private static List captureEngineIndexes(FakeObjects objects, List objects.graph().vertexLabel((Id) call.getArgument(0))); + Mockito.when(schema.getEdgeLabel(Mockito.any(Id.class))) + .thenAnswer(call -> objects.graph().edgeLabel((Id) call.getArgument(0))); Mockito.when(schema.getIndexLabels()).thenReturn(labels); Mockito.when(params.analyzer()).thenReturn(analyzer); for (IndexLabel label : labels) { diff --git a/hugegraph-server/hugegraph-test/src/main/java/org/apache/hugegraph/backend/tx/GraphTransactionTest.java b/hugegraph-server/hugegraph-test/src/main/java/org/apache/hugegraph/backend/tx/GraphTransactionTest.java index a41ee5ffc0..d5ed38aa40 100644 --- a/hugegraph-server/hugegraph-test/src/main/java/org/apache/hugegraph/backend/tx/GraphTransactionTest.java +++ b/hugegraph-server/hugegraph-test/src/main/java/org/apache/hugegraph/backend/tx/GraphTransactionTest.java @@ -19,6 +19,7 @@ import java.util.ArrayList; import java.util.Arrays; +import java.util.Collections; import java.util.List; import org.apache.hugegraph.HugeFactory; @@ -49,6 +50,23 @@ public class GraphTransactionTest { + @Test + public void testOlapDeletionDoesNotScanBaseVertexEdges() { + GraphTransaction transaction = Mockito.mock(GraphTransaction.class, Mockito.CALLS_REAL_METHODS); + HugeVertex vertex = Mockito.mock(HugeVertex.class); + Mockito.when(vertex.schemaLabel()).thenReturn(VertexLabel.OLAP_VL); + Id id = IdGenerator.of(1L); + Mockito.when(vertex.id()).thenReturn(id); + + // Preserve OLAP's original rejection before any ordinary edge scan. + IllegalStateException error = Assert.assertThrows(IllegalStateException.class, () -> { + transaction.prepareDeletions(Collections.singletonMap(id, vertex), Collections.emptyMap()); + }); + Assert.assertTrue(error.getMessage().startsWith("Graph is null of schema")); + Mockito.verify(transaction, Mockito.never()).queryEdgesFromBackend(Mockito.any()); + Mockito.verify(transaction, Mockito.never()).doRemove(Mockito.any()); + } + @Test public void testBatchDecisionsRemainFixedAfterOriginChanges() { ConditionQuery root = new ConditionQuery(HugeType.VERTEX); diff --git a/hugegraph-server/hugegraph-test/src/main/java/org/apache/hugegraph/core/AuthTest.java b/hugegraph-server/hugegraph-test/src/main/java/org/apache/hugegraph/core/AuthTest.java index 12afccd1f8..66680e70cc 100644 --- a/hugegraph-server/hugegraph-test/src/main/java/org/apache/hugegraph/core/AuthTest.java +++ b/hugegraph-server/hugegraph-test/src/main/java/org/apache/hugegraph/core/AuthTest.java @@ -27,7 +27,7 @@ import javax.security.sasl.AuthenticationException; import org.apache.commons.collections.CollectionUtils; -import org.apache.hugegraph.exception.HugeException; +import org.apache.hugegraph.HugeFactory; import org.apache.hugegraph.HugeGraph; import org.apache.hugegraph.HugeGraphParams; import org.apache.hugegraph.auth.AuthConstant; @@ -45,6 +45,8 @@ import org.apache.hugegraph.auth.TokenGenerator; import org.apache.hugegraph.auth.UserWithRole; import org.apache.hugegraph.backend.cache.Cache; +import org.apache.hugegraph.exception.HugeException; +import org.apache.hugegraph.exception.NotFoundException; import org.apache.hugegraph.id.Id; import org.apache.hugegraph.id.IdGenerator; import org.apache.hugegraph.struct.schema.VertexLabel; @@ -141,8 +143,7 @@ public void clearAll() { } //FIXME: support project in hstore - boolean isHstore = Objects.equals("hstore", System.getProperty("backend")) || - (System.getProperty("backend") == null); + boolean isHstore = Objects.equals("hstore", graph.backend()); if (!isHstore) { for (HugeProject project : authManager.listAllProject(-1)) { @@ -1778,6 +1779,90 @@ public void testDelProject() { }); } + @Test + public void testProjectCleanupRepeatsWithoutBackendSystemProperty() { + Assume.assumeTrue("skip this test for hstore", !Objects.equals("hstore", graph().backend())); + String previous = System.getProperty("backend"); + try { + System.clearProperty("backend"); + for (int attempt = 0; attempt < 2; attempt++) { + this.testDeleteProjectAfterRequestCleanup(); + this.clearAll(); + Assert.assertEquals(0, graph().authManager().listAllProject(-1).size()); + } + } finally { + if (previous == null) { + System.clearProperty("backend"); + } else { + System.setProperty("backend", previous); + } + } + } + + @Test + public void testDeleteProjectAfterRequestCleanup() { + Assume.assumeTrue("skip this test for hstore", !Objects.equals("hstore", graph().backend())); + AuthManager authManager = graph().authManager(); + HugeProject project = makeProject("cleanup_project", ""); + Id projectId = authManager.createProject(project); + HugeProject saved = authManager.getProject(projectId); + Id retainedId = authManager.createProject(makeProject("retained_project", "")); + HugeProject retained = authManager.getProject(retainedId); + Assert.assertEquals(3, authManager.listAccessByTarget(saved.targetId(), -1).size()); + Id userId = authManager.createUser(makeUser("membership_user", "pass")); + authManager.createBelong(makeBelong(userId, saved.adminGroupId())); + Id retainedBelongId = authManager.createBelong(makeBelong(userId, retained.adminGroupId())); + + // Match FINISHED cleanup between REST creation and deletion requests. + HugeFactory.closeCurrentThreadTransactions(); + authManager.deleteProject(projectId); + HugeFactory.closeCurrentThreadTransactions(); + + Assert.assertEquals(0, authManager.listAccessByGroup(saved.adminGroupId(), -1).size()); + Assert.assertEquals(0, authManager.listAccessByGroup(saved.opGroupId(), -1).size()); + Assert.assertEquals(0, authManager.listAccessByTarget(saved.targetId(), -1).size()); + Assert.assertEquals(3, authManager.listAccessByTarget(retained.targetId(), -1).size()); + Assert.assertEquals(0, authManager.listBelongByGroup(saved.adminGroupId(), -1).size()); + Assert.assertEquals(1, authManager.listBelongByUser(userId, -1).size()); + Assert.assertEquals(retainedBelongId, authManager.getBelong(retainedBelongId).id()); + Assert.assertEquals(userId, authManager.getUser(userId).id()); + Assert.assertEquals(projectId, authManager.createProject(makeProject("cleanup_project", ""))); + Assert.assertEquals(0, authManager.listBelongByGroup(saved.adminGroupId(), -1).size()); + } + + @Test + public void testDeleteUserWithRelationsAfterRequestCleanup() { + Assume.assumeTrue("skip this test for hstore", !Objects.equals("hstore", graph().backend())); + AuthManager authManager = graph().authManager(); + Id deletedUser = authManager.createUser(makeUser("deleted_member", "pass")); + Id retainedUser = authManager.createUser(makeUser("retained_member", "pass")); + Id group = authManager.createGroup(makeGroup("shared_group")); + authManager.createBelong(makeBelong(deletedUser, group)); + Id retainedBelong = authManager.createBelong(makeBelong(retainedUser, group)); + + HugeFactory.closeCurrentThreadTransactions(); + authManager.deleteUser(deletedUser); + HugeFactory.closeCurrentThreadTransactions(); + + Assert.assertThrows(NotFoundException.class, () -> authManager.getUser(deletedUser)); + Assert.assertEquals(0, authManager.listBelongByUser(deletedUser, -1).size()); + Assert.assertEquals(1, authManager.listBelongByGroup(group, -1).size()); + Assert.assertEquals(retainedBelong, authManager.getBelong(retainedBelong).id()); + Assert.assertEquals(retainedUser, authManager.getUser(retainedUser).id()); + } + + @Test + public void testDeleteUserWithoutRelationsAfterRequestCleanup() { + Assume.assumeTrue("skip this test for hstore", !Objects.equals("hstore", graph().backend())); + AuthManager authManager = graph().authManager(); + Id userId = authManager.createUser(makeUser("isolated_user", "pass")); + HugeFactory.closeCurrentThreadTransactions(); + authManager.deleteUser(userId); + HugeFactory.closeCurrentThreadTransactions(); + Assert.assertThrows(NotFoundException.class, () -> authManager.getUser(userId)); + Assert.assertEquals(0, authManager.listBelongByUser(userId, -1).size()); + } + @Test public void testUpdateProject() { Assume.assumeTrue("skip this test for hstore", diff --git a/hugegraph-server/hugegraph-test/src/main/java/org/apache/hugegraph/core/CoreTestSuite.java b/hugegraph-server/hugegraph-test/src/main/java/org/apache/hugegraph/core/CoreTestSuite.java index 5d33f92811..9b45f40a64 100644 --- a/hugegraph-server/hugegraph-test/src/main/java/org/apache/hugegraph/core/CoreTestSuite.java +++ b/hugegraph-server/hugegraph-test/src/main/java/org/apache/hugegraph/core/CoreTestSuite.java @@ -18,6 +18,7 @@ package org.apache.hugegraph.core; import org.apache.hugegraph.HugeGraph; +import org.apache.hugegraph.auth.BackendLeaseCleanupTest; import org.apache.hugegraph.constant.ServiceConstant; import org.apache.hugegraph.dist.RegisterUtil; import org.apache.hugegraph.masterelection.GlobalMasterInfo; @@ -34,6 +35,7 @@ @RunWith(Suite.class) @Suite.SuiteClasses({ + BackendLeaseCleanupTest.class, PropertyKeyCoreTest.class, VertexLabelCoreTest.class, EdgeLabelCoreTest.class, diff --git a/hugegraph-server/hugegraph-test/src/main/java/org/apache/hugegraph/core/EdgeCoreTest.java b/hugegraph-server/hugegraph-test/src/main/java/org/apache/hugegraph/core/EdgeCoreTest.java index d626df4a1a..c69ef7254f 100644 --- a/hugegraph-server/hugegraph-test/src/main/java/org/apache/hugegraph/core/EdgeCoreTest.java +++ b/hugegraph-server/hugegraph-test/src/main/java/org/apache/hugegraph/core/EdgeCoreTest.java @@ -213,6 +213,48 @@ protected void initStrikeIndex() { .by("place", "tool", "reason").create(); } + @Test + public void testDeleteEdgeWithMissingSchemaLabel() throws InterruptedException, ExecutionException { + HugeGraph graph = graph(); + Vertex author = graph.addVertex(T.label, "author", "id", 1, "name", "James", + "age", 62, "lived", "Canadian"); + Vertex book = graph.addVertex(T.label, "book", "name", "Java"); + Edge edge = author.addEdge("authored", book, "score", 3); + graph.tx().commit(); + + // Preserve the backend edge while removing its schema metadata. + params().schemaTransaction().removeSchema(graph.edgeLabel("authored")); + params().graphEventHub().notify(Events.CACHE, "clear", null).get(); + Edge orphan = graph.edges(edge.id()).next(); + Assert.assertEquals("~undefined", orphan.label()); + + orphan.remove(); + graph.tx().commit(); + Assert.assertFalse(graph.edges(edge.id()).hasNext()); + Assert.assertFalse(graph.vertices(author.id()).next().edges(Direction.OUT).hasNext()); + Assert.assertFalse(graph.vertices(book.id()).next().edges(Direction.IN).hasNext()); + } + + @Test + public void testDeleteVertexAdjacentToEdgeWithMissingSchemaLabel() throws InterruptedException, ExecutionException { + HugeGraph graph = graph(); + Vertex author = graph.addVertex(T.label, "author", "id", 1, "name", "James", + "age", 62, "lived", "Canadian"); + Vertex book = graph.addVertex(T.label, "book", "name", "Java"); + Edge edge = author.addEdge("authored", book, "score", 3); + graph.tx().commit(); + + params().schemaTransaction().removeSchema(graph.edgeLabel("authored")); + params().graphEventHub().notify(Events.CACHE, "clear", null).get(); + Assert.assertEquals("~undefined", graph.edges(edge.id()).next().label()); + + graph.vertices(author.id()).next().remove(); + graph.tx().commit(); + Assert.assertFalse(graph.vertices(author.id()).hasNext()); + Assert.assertFalse(graph.edges(edge.id()).hasNext()); + Assert.assertFalse(graph.vertices(book.id()).next().edges(Direction.IN).hasNext()); + } + @Test public void testAddEdge() { HugeGraph graph = graph(); diff --git a/hugegraph-server/hugegraph-test/src/main/java/org/apache/hugegraph/core/IndexLabelCoreTest.java b/hugegraph-server/hugegraph-test/src/main/java/org/apache/hugegraph/core/IndexLabelCoreTest.java index c9988db513..6a7fdb60a9 100644 --- a/hugegraph-server/hugegraph-test/src/main/java/org/apache/hugegraph/core/IndexLabelCoreTest.java +++ b/hugegraph-server/hugegraph-test/src/main/java/org/apache/hugegraph/core/IndexLabelCoreTest.java @@ -22,6 +22,7 @@ import org.apache.hugegraph.exception.HugeException; import org.apache.hugegraph.HugeGraph; +import org.apache.hugegraph.HugeGraphParams; import org.apache.hugegraph.exception.ExistedException; import org.apache.hugegraph.exception.NoIndexException; import org.apache.hugegraph.exception.NotFoundException; @@ -31,10 +32,12 @@ import org.apache.hugegraph.struct.schema.Userdata; import org.apache.hugegraph.struct.schema.VertexLabel; import org.apache.hugegraph.testutil.Assert; +import org.apache.hugegraph.testutil.Whitebox; import org.apache.hugegraph.type.HugeType; import org.apache.hugegraph.type.define.IndexType; import org.apache.hugegraph.type.define.WriteType; import org.apache.hugegraph.util.DateUtil; +import org.apache.hugegraph.util.Events; import org.apache.tinkerpop.gremlin.process.traversal.P; import org.apache.tinkerpop.gremlin.structure.Edge; import org.apache.tinkerpop.gremlin.structure.T; @@ -1297,6 +1300,76 @@ public void testRemoveIndexLabelOfVertex() { }); } + @Test + public void testUpdateCachedVertexAfterIndexRemoval() throws Exception { + super.initPropertyKeys(); + SchemaManager schema = graph().schema(); + schema.vertexLabel("person").properties("name", "city").primaryKeys("name").create(); + schema.indexLabel("personByCity").onV("person").secondary().by("city").create(); + Vertex original = graph().addVertex(T.label, "person", "name", "cache-test", "city", "old"); + graph().tx().commit(); + Vertex cached = graph().vertices(original.id()).next(); + graph().tx().commit(); + // Schema cache eviction must not make a retained vertex's index list authoritative. + HugeGraphParams params = Whitebox.getInternalState(graph(), "params"); + params.schemaEventHub().notify(Events.CACHE, "clear", null).get(); + + schema.indexLabel("personByCity").remove(); + Assert.assertThrows(NotFoundException.class, () -> schema.getIndexLabel("personByCity")); + cached.property("city", "new"); + graph().tx().commit(); + Assert.assertEquals("new", graph().vertices(original.id()).next().value("city")); + graph().tx().commit(); + schema.indexLabel("personByCity").onV("person").secondary().by("city").create(); + schema.propertyKey("extra").asText().create(); + schema.vertexLabel("person").properties("extra").nullableKeys("extra").append(); + schema.indexLabel("personByExtra").onV("person").secondary().by("extra").create(); + graph().taskScheduler().waitUntilAllTasksCompleted(30); + cached.property("city", "newer"); + graph().tx().commit(); + Assert.assertFalse(graph().traversal().V().has("city", "new").hasNext()); + Assert.assertEquals(original.id(), graph().traversal().V().has("city", "newer").next().id()); + graph().tx().commit(); + cached.remove(); + graph().tx().commit(); + Assert.assertFalse(graph().vertices(original.id()).hasNext()); + } + + @Test + public void testUpdateCachedEdgeAfterIndexRemoval() throws Exception { + super.initPropertyKeys(); + SchemaManager schema = graph().schema(); + schema.vertexLabel("author").properties("id", "name").primaryKeys("id").create(); + schema.vertexLabel("book").properties("name").primaryKeys("name").create(); + schema.edgeLabel("authored").singleTime().link("author", "book").properties("contribution").create(); + schema.indexLabel("authoredByContri").onE("authored").secondary().by("contribution").create(); + Vertex author = graph().addVertex(T.label, "author", "id", 1, "name", "author"); + Vertex book = graph().addVertex(T.label, "book", "name", "book"); + Edge original = author.addEdge("authored", book, "contribution", "old"); + graph().tx().commit(); + Edge cached = graph().edges(original.id()).next(); + graph().tx().commit(); + HugeGraphParams params = Whitebox.getInternalState(graph(), "params"); + params.schemaEventHub().notify(Events.CACHE, "clear", null).get(); + + schema.indexLabel("authoredByContri").remove(); + Assert.assertThrows(NotFoundException.class, () -> schema.getIndexLabel("authoredByContri")); + cached.property("contribution", "new"); + graph().tx().commit(); + Assert.assertEquals("new", graph().edges(original.id()).next().value("contribution")); + graph().tx().commit(); + schema.indexLabel("authoredByContri").onE("authored").secondary().by("contribution").create(); + graph().taskScheduler().waitUntilAllTasksCompleted(30); + cached.property("contribution", "newer"); + graph().tx().commit(); + Assert.assertFalse(graph().traversal().E().has("contribution", "new").hasNext()); + Assert.assertEquals(original.id(), graph().traversal().E().has("contribution", "newer").next().id()); + graph().tx().commit(); + cached.remove(); + graph().tx().commit(); + Assert.assertFalse(graph().edges(original.id()).hasNext()); + } + @Test public void testRemoveIndexLabelOfEdge() { super.initPropertyKeys(); diff --git a/hugegraph-server/hugegraph-test/src/main/java/org/apache/hugegraph/core/VertexCoreTest.java b/hugegraph-server/hugegraph-test/src/main/java/org/apache/hugegraph/core/VertexCoreTest.java index b6657ae150..5add98f071 100644 --- a/hugegraph-server/hugegraph-test/src/main/java/org/apache/hugegraph/core/VertexCoreTest.java +++ b/hugegraph-server/hugegraph-test/src/main/java/org/apache/hugegraph/core/VertexCoreTest.java @@ -31,6 +31,7 @@ import java.util.Random; import java.util.Set; import java.util.UUID; +import java.util.concurrent.ExecutionException; import java.util.concurrent.atomic.AtomicInteger; import org.apache.hugegraph.exception.HugeException; @@ -72,6 +73,7 @@ import org.apache.hugegraph.util.Blob; import org.apache.hugegraph.util.CollectionUtil; import org.apache.hugegraph.util.DateUtil; +import org.apache.hugegraph.util.Events; import org.apache.hugegraph.util.LongEncoding; import org.apache.tinkerpop.gremlin.process.traversal.P; import org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.GraphTraversal; @@ -198,6 +200,23 @@ protected void initComputerIndex() { .create(); } + @Test + public void testDeleteVertexWithMissingSchemaLabel() throws InterruptedException, ExecutionException { + HugeGraph graph = graph(); + Vertex vertex = graph.addVertex(T.label, "author", "id", 1, "name", "James"); + graph.tx().commit(); + + // Simulate schema loss while the committed backend record remains. + params().schemaTransaction().removeSchema(graph.vertexLabel("author")); + params().graphEventHub().notify(Events.CACHE, "clear", null).get(); + Vertex orphan = graph.vertices(vertex.id()).next(); + Assert.assertEquals("~undefined", orphan.label()); + + orphan.remove(); + graph.tx().commit(); + Assert.assertFalse(graph.vertices(vertex.id()).hasNext()); + } + @Test public void testAddVertex() { HugeGraph graph = graph(); diff --git a/hugegraph-server/hugegraph-test/src/main/java/org/apache/hugegraph/unit/UnitTestSuite.java b/hugegraph-server/hugegraph-test/src/main/java/org/apache/hugegraph/unit/UnitTestSuite.java index 617440b668..0c9a0af01d 100644 --- a/hugegraph-server/hugegraph-test/src/main/java/org/apache/hugegraph/unit/UnitTestSuite.java +++ b/hugegraph-server/hugegraph-test/src/main/java/org/apache/hugegraph/unit/UnitTestSuite.java @@ -44,6 +44,7 @@ import org.apache.hugegraph.unit.cache.CachedGraphTransactionTest; import org.apache.hugegraph.unit.cache.CachedSchemaTransactionTest; import org.apache.hugegraph.unit.cache.RamTableTest; +import org.apache.hugegraph.unit.cache.RequestCacheLifetimeTest; import org.apache.hugegraph.unit.cmd.InitStoreConfigTest; import org.apache.hugegraph.unit.config.GremlinConfigCompatibilityTest; import org.apache.hugegraph.unit.config.GraphBinarySchemaCompatibilityTest; @@ -148,6 +149,7 @@ MetaManagerSchemaCacheClearEventTest.class, EtcdMetaDriverTest.class, CachedGraphTransactionTest.class, + RequestCacheLifetimeTest.class, CacheManagerTest.class, RamTableTest.class, diff --git a/hugegraph-server/hugegraph-test/src/main/java/org/apache/hugegraph/unit/cache/CachedGraphTransactionTest.java b/hugegraph-server/hugegraph-test/src/main/java/org/apache/hugegraph/unit/cache/CachedGraphTransactionTest.java index 8708d78479..a18bcbec6d 100644 --- a/hugegraph-server/hugegraph-test/src/main/java/org/apache/hugegraph/unit/cache/CachedGraphTransactionTest.java +++ b/hugegraph-server/hugegraph-test/src/main/java/org/apache/hugegraph/unit/cache/CachedGraphTransactionTest.java @@ -357,7 +357,65 @@ public void testClosingNonOwnerKeepsStoreListenerRegistered() } @Test - public void testLastCloseRemovesStoreListener() throws Exception { + public void testGraphCloseRemovesStoreListenerWithoutCacheHolder() throws Exception { + String name = this.params.spaceGraphName(); + Object cacheHolder = graphCacheEventListeners().remove(name); + Assert.assertNotNull(cacheHolder); + Whitebox.invoke(cacheHolder.getClass(), "close", cacheHolder); + Object storeHolder = storeEventListeners().get(name); + EventListener listener = holderListener(storeHolder); + BackendStoreProvider provider = holderProvider(storeHolder); + CachedGraphTransaction.closeGraph(this.params); + Assert.assertFalse(storeEventListeners().containsKey(name)); + Assert.assertFalse(provider.storeEventHub().listeners(EventHub.ANY_EVENT).contains(listener)); + } + + @Test + public void testGraphCloseCleansStoreAfterCacheRegistrationFailure() throws Exception { + String name = this.params.spaceGraphName(); + EventHub hub = Mockito.spy(new EventHub("cache-registration-failure")); + RuntimeException failure = new IllegalStateException("injected cache registration failure"); + Mockito.doThrow(failure).when(hub).listen(Mockito.eq(Events.CACHE), Mockito.any(EventListener.class)); + HugeGraphParams failedParams = Mockito.spy(this.params); + Mockito.doReturn(hub).when(failedParams).graphEventHub(); + try { + new CachedGraphTransaction(failedParams, this.params.loadGraphStore()); + Assert.fail("Cache registration must reach the injected failure"); + } catch (RuntimeException expected) { + Assert.assertSame(failure, expected); + } + Object storeHolder = storeEventListeners().get(name); + Assert.assertSame(hub, Whitebox.getInternalState(storeHolder, "hub")); + BackendStoreProvider provider = holderProvider(storeHolder); + EventListener listener = holderListener(storeHolder); + Assert.assertTrue(provider.storeEventHub().listeners(EventHub.ANY_EVENT).contains(listener)); + CachedGraphTransaction.closeGraph(failedParams); + Assert.assertFalse(storeEventListeners().containsKey(name)); + Assert.assertFalse(provider.storeEventHub().listeners(EventHub.ANY_EVENT).contains(listener)); + } + + @Test + public void testOldGraphCloseKeepsReopenedStoreGeneration() throws Exception { + HugeGraphParams oldParams = this.params; + this.cache.close(); + this.cache = null; + this.graph.clearBackend(); + this.graph.close(); + this.graph = HugeFactory.open(FakeObjects.newConfig()); + this.params = Whitebox.getInternalState(this.graph, "params"); + this.cache = new CachedGraphTransaction(this.params, this.params.loadGraphStore()); + String name = this.params.spaceGraphName(); + Object cacheHolder = graphCacheEventListeners().get(name); + Object storeHolder = storeEventListeners().get(name); + CachedGraphTransaction.closeGraph(oldParams); + Assert.assertSame(cacheHolder, graphCacheEventListeners().get(name)); + Assert.assertSame(storeHolder, storeEventListeners().get(name)); + Assert.assertTrue(holderProvider(storeHolder).storeEventHub().listeners(EventHub.ANY_EVENT) + .contains(holderListener(storeHolder))); + } + + @Test + public void testGraphCloseRemovesStoreListener() throws Exception { ConcurrentMap storeListeners = storeEventListeners(); String graphName = this.params.spaceGraphName(); @@ -374,6 +432,11 @@ public void testLastCloseRemovesStoreListener() throws Exception { this.cache = null; this.params.graphTransaction().close(); + Assert.assertSame(holder, storeListeners.get(graphName)); + Assert.assertEquals(0, holderRefCount(holder)); + this.graph.clearBackend(); + this.graph.close(); + this.graph = null; Assert.assertFalse(storeListeners.containsKey(graphName)); } @@ -473,14 +536,16 @@ public void testReopenGraphReRegistersStoreListener() throws Exception { this.cache = null; this.params.graphTransaction().close(); - // Last close drops the registry entry and unregisters the listener. + // Request leases may end while the graph caches remain warm. + Assert.assertTrue(storeListeners.containsKey(graphName)); + Assert.assertTrue(provider.storeEventHub().listeners(EventHub.ANY_EVENT).contains(registered)); + + this.graph.clearBackend(); + this.graph.close(); Assert.assertFalse(storeListeners.containsKey(graphName)); Assert.assertFalse(provider.storeEventHub() .listeners(EventHub.ANY_EVENT) .contains(registered)); - - this.graph.clearBackend(); - this.graph.close(); this.graph = null; HugeGraph reopened = HugeFactory.open(FakeObjects.newConfig()); @@ -505,7 +570,7 @@ public void testReopenGraphReRegistersStoreListener() throws Exception { } @Test - public void testLastCloseRemovesGraphCacheListener() throws Exception { + public void testGraphCloseRemovesGraphCacheListener() throws Exception { ConcurrentMap cacheListeners = graphCacheEventListeners(); String graphName = this.params.spaceGraphName(); @@ -523,13 +588,15 @@ public void testLastCloseRemovesGraphCacheListener() throws Exception { this.cache = null; this.params.graphTransaction().close(); + Assert.assertTrue(cacheListeners.containsKey(graphName)); + Assert.assertTrue(this.params.graphEventHub().listeners(Events.CACHE).contains(registered)); + + this.graph.clearBackend(); + this.graph.close(); Assert.assertFalse(cacheListeners.containsKey(graphName)); Assert.assertFalse(this.params.graphEventHub() .listeners(Events.CACHE) .contains(registered)); - - this.graph.clearBackend(); - this.graph.close(); this.graph = null; HugeGraph reopened = HugeFactory.open(FakeObjects.newConfig()); diff --git a/hugegraph-server/hugegraph-test/src/main/java/org/apache/hugegraph/unit/cache/CachedSchemaTransactionTest.java b/hugegraph-server/hugegraph-test/src/main/java/org/apache/hugegraph/unit/cache/CachedSchemaTransactionTest.java index a4ea489817..ccdec12dd6 100644 --- a/hugegraph-server/hugegraph-test/src/main/java/org/apache/hugegraph/unit/cache/CachedSchemaTransactionTest.java +++ b/hugegraph-server/hugegraph-test/src/main/java/org/apache/hugegraph/unit/cache/CachedSchemaTransactionTest.java @@ -19,6 +19,7 @@ import java.lang.reflect.Constructor; import java.lang.reflect.Field; +import java.util.ArrayList; import java.util.Arrays; import java.util.Collections; import java.util.List; @@ -45,14 +46,21 @@ import org.apache.hugegraph.event.EventListener; import org.apache.hugegraph.meta.MetaDriver; import org.apache.hugegraph.meta.MetaManager; +import org.apache.hugegraph.meta.PdMetaDriver; import org.apache.hugegraph.meta.managers.GraphMetaManager; import org.apache.hugegraph.struct.schema.SchemaElement; +import org.apache.hugegraph.struct.schema.PropertyKey; +import org.apache.hugegraph.struct.schema.VertexLabel; +import org.apache.hugegraph.structure.HugeVertex; import org.apache.hugegraph.testutil.Assert; import org.apache.hugegraph.testutil.Whitebox; import org.apache.hugegraph.type.HugeType; +import org.apache.hugegraph.type.define.IdStrategy; import org.apache.hugegraph.unit.BaseUnitTest; import org.apache.hugegraph.unit.FakeObjects; import org.apache.hugegraph.util.Events; +import org.apache.hugegraph.util.JsonUtil; +import org.apache.tinkerpop.gremlin.structure.VertexProperty; import org.junit.After; import org.junit.Before; import org.junit.Test; @@ -297,7 +305,7 @@ public void testCacheListenerSurvivesOwnerClose() throws Exception { } @Test - public void testLastCloseRemovesSchemaCacheListener() throws Exception { + public void testGraphCloseRemovesSchemaCacheListener() throws Exception { ConcurrentMap registry = schemaCacheEventListeners(); String graphName = this.params.spaceGraphName(); CachedSchemaTransaction owner = this.cache(); @@ -314,13 +322,15 @@ public void testLastCloseRemovesSchemaCacheListener() throws Exception { this.cache = null; this.params.schemaTransaction().close(); + Assert.assertTrue(registry.containsKey(graphName)); + Assert.assertTrue(this.params.schemaEventHub().listeners(Events.CACHE).contains(registered)); + + this.graph.clearBackend(); + this.graph.close(); Assert.assertFalse(registry.containsKey(graphName)); Assert.assertFalse(this.params.schemaEventHub() .listeners(Events.CACHE) .contains(registered)); - - this.graph.clearBackend(); - this.graph.close(); this.graph = null; HugeGraph reopened = HugeFactory.open(FakeObjects.newConfig()); @@ -1002,4 +1012,196 @@ public void testResetCachedAllIfReachedCapacity() throws Exception { Whitebox.setInternalState(cache, "idCache.capacity", old); } } + + @Test + public void testV2WarmRequestsRetainSchemaWithoutMetadataReadsAndReceiveInvalidations() throws Exception { + AtomicBoolean registered = metaListenerFlag(); + boolean previous = registered.getAndSet(true); + List requests = new ArrayList<>(); + Object originalGraphManager = null; + boolean graphManagerReplaced = false; + try { + FakeObjects objects = new FakeObjects("v2-request-schema"); + HugeGraphParams requestParams = v2RequestParams(objects.graph()); + VertexLabel original = objects.newVertexLabel(IdGenerator.of(31), "retained", + IdStrategy.CUSTOMIZE_NUMBER); + AtomicReference metadata = new AtomicReference<>(JsonUtil.toJson(original.asMap())); + AtomicInteger reads = new AtomicInteger(); + AtomicInteger scans = new AtomicInteger(); + PdMetaDriver driver = Mockito.mock(PdMetaDriver.class); + Mockito.when(driver.get(Mockito.anyString())).thenAnswer(call -> { + reads.incrementAndGet(); + return metadata.get(); + }); + Mockito.when(driver.scanWithPrefix(Mockito.anyString())).thenAnswer(call -> { + scans.incrementAndGet(); + return ImmutableMap.of("label", metadata.get()); + }); + originalGraphManager = swapGraphMetaManager(new GraphMetaManager(driver, "test")); + graphManagerReplaced = true; + CachedSchemaTransactionV2 first = new CachedSchemaTransactionV2(driver, "test", requestParams); + requests.add(first); + VertexLabel retainedLabel = first.getVertexLabel(original.id()); + Assert.assertEquals(1, reads.get()); + HugeVertex retained = new HugeVertex(objects.graph(), IdGenerator.of(32), retainedLabel); + first.close(); + for (int request = 0; request < 10; request++) { + CachedSchemaTransactionV2 next = new CachedSchemaTransactionV2(driver, "test", requestParams); + requests.add(next); + Assert.assertSame(retainedLabel, next.getVertexLabel(original.id())); + Assert.assertSame(retainedLabel, next.getVertexLabel(original.name())); + next.close(); + } + Assert.assertEquals("Warm request cleanup must not re-read metadata", 1, reads.get()); + PropertyKey appended = objects.newPropertyKey(IdGenerator.of(33), "appended"); + retainedLabel.properties(appended.id()); + retainedLabel.nullableKeys(appended.id()); + Assert.assertEquals("accepted", retained.property(VertexProperty.Cardinality.single, + appended.name(), "accepted").value()); + Object holder = v2CacheEventListeners().get(this.params.spaceGraphName()); + Assert.assertNotNull(holder); + Assert.assertEquals(0, holderRefCount(holder)); + Assert.assertTrue(this.params.schemaEventHub().listeners(Events.CACHE).contains(holderListener(holder))); + VertexLabel updated = objects.newVertexLabel(original.id(), "changed", IdStrategy.CUSTOMIZE_NUMBER); + metadata.set(JsonUtil.toJson(updated.asMap())); + this.params.schemaEventHub().notify(Events.CACHE, Cache.ACTION_INVALID, + HugeType.VERTEX_LABEL, original.id()).get(); + CachedSchemaTransactionV2 afterInvalidation = new CachedSchemaTransactionV2(driver, "test", requestParams); + requests.add(afterInvalidation); + Assert.assertEquals("changed", afterInvalidation.getVertexLabel(original.id()).name()); + Assert.assertEquals("Invalidation must refresh metadata even with no request lease", 2, reads.get()); + Assert.assertNotSame(retainedLabel, afterInvalidation.getVertexLabel(original.id())); + afterInvalidation.close(); + VertexLabel afterStore = objects.newVertexLabel(original.id(), "store-cleared", + IdStrategy.CUSTOMIZE_NUMBER); + metadata.set(JsonUtil.toJson(afterStore.asMap())); + EventHub storeHub = this.params.loadGraphStore().provider().storeEventHub(); + EventListener v2StoreListener = Whitebox.getInternalState(first, "storeEventListener"); + List otherStoreListeners = new ArrayList<>(storeHub.listeners(EventHub.ANY_EVENT)); + Assert.assertTrue("The V2 Store listener must remain registered without a request lease", + otherStoreListeners.remove(v2StoreListener)); + // Isolate the Store path from V1 listeners that asynchronously emit another CACHE clear. + for (EventListener listener : otherStoreListeners) { + storeHub.unlisten(EventHub.ANY_EVENT, listener); + } + try { + Assert.assertEquals(1, storeHub.notify(Events.STORE_CLEAR, + this.params.loadGraphStore().provider()).get().intValue()); + } finally { + for (EventListener listener : otherStoreListeners) { + storeHub.listen(EventHub.ANY_EVENT, listener); + } + } + Mockito.verify(driver).put(Mockito.anyString(), Mockito.anyString()); + CachedSchemaTransactionV2 afterStoreEvent = new CachedSchemaTransactionV2(driver, "test", requestParams); + requests.add(afterStoreEvent); + Assert.assertEquals("store-cleared", afterStoreEvent.getVertexLabel(original.id()).name()); + Assert.assertEquals("Store events must refresh metadata with no active request", 3, reads.get()); + afterStoreEvent.close(); + VertexLabel allLabel = null; + for (int request = 0; request < 4; request++) { + CachedSchemaTransactionV2 next = new CachedSchemaTransactionV2(driver, "test", requestParams); + requests.add(next); + List all = next.getVertexLabels(); + Assert.assertEquals(1, all.size()); + Assert.assertEquals("store-cleared", all.get(0).name()); + if (allLabel == null) { + allLabel = all.get(0); + } else { + Assert.assertSame(allLabel, all.get(0)); + } + next.close(); + } + Assert.assertEquals("Warm all-schema requests must retain the cached-all flag", 1, scans.get()); + } finally { + for (CachedSchemaTransactionV2 request : requests) { + request.close(); + } + CachedSchemaTransactionV2.closeGraph(this.params); + if (graphManagerReplaced) { + swapGraphMetaManager(originalGraphManager); + } + registered.set(previous); + } + } + + @Test + public void testV2GraphCloseReopenKeepsNewHubSafeFromOldLeases() throws Exception { + AtomicBoolean registered = metaListenerFlag(); + boolean previous = registered.getAndSet(true); + HugeGraphParams oldParams = this.params; + CachedSchemaTransactionV2 oldLease = null; + CachedSchemaTransactionV2 newLease = null; + try { + FakeObjects objects = new FakeObjects("v2-generation-schema"); + VertexLabel original = objects.newVertexLabel(IdGenerator.of(41), "generation", + IdStrategy.CUSTOMIZE_NUMBER); + AtomicInteger reads = new AtomicInteger(); + PdMetaDriver driver = Mockito.mock(PdMetaDriver.class); + Mockito.when(driver.get(Mockito.anyString())).thenAnswer(call -> { + reads.incrementAndGet(); + return JsonUtil.toJson(original.asMap()); + }); + oldLease = new CachedSchemaTransactionV2(driver, "test", oldParams); + VertexLabel oldLabel = oldLease.getVertexLabel(original.id()); + Object oldHolder = v2CacheEventListeners().get(oldParams.spaceGraphName()); + EventListener oldListener = holderListener(oldHolder); + this.cache.close(); + this.cache = null; + this.graph.close(); + this.graph = null; + Assert.assertFalse(v2CacheEventListeners().containsKey(oldParams.spaceGraphName())); + Assert.assertFalse(oldParams.schemaEventHub().listeners(Events.CACHE).contains(oldListener)); + Assert.assertEquals(0L, Whitebox.invoke(oldLease, "idCache", "size")); + this.graph = HugeFactory.open(FakeObjects.newConfig()); + this.params = Whitebox.getInternalState(this.graph, "params"); + Assert.assertNotSame(oldParams.schemaEventHub(), this.params.schemaEventHub()); + newLease = new CachedSchemaTransactionV2(driver, "test", this.params); + VertexLabel newLabel = newLease.getVertexLabel(original.id()); + Object newHolder = v2CacheEventListeners().get(this.params.spaceGraphName()); + Assert.assertNotSame(oldHolder, newHolder); + Assert.assertNotSame(oldLabel, newLabel); + Assert.assertEquals(2, reads.get()); + int newRefCount = holderRefCount(newHolder); + oldLease.close(); + CachedSchemaTransactionV2.closeGraph(oldParams); + Assert.assertSame(newHolder, v2CacheEventListeners().get(this.params.spaceGraphName())); + Assert.assertEquals(newRefCount, holderRefCount(newHolder)); + Assert.assertTrue(this.params.schemaEventHub().listeners(Events.CACHE) + .contains(holderListener(newHolder))); + Assert.assertSame(newLabel, newLease.getVertexLabel(original.id())); + Assert.assertEquals(2, reads.get()); + } finally { + if (oldLease != null) { + oldLease.close(); + } + if (newLease != null) { + newLease.close(); + } + CachedSchemaTransactionV2.closeGraph(oldParams); + CachedSchemaTransactionV2.closeGraph(this.params); + registered.set(previous); + } + } + + private HugeGraphParams v2RequestParams(HugeGraph supplier) { + Mockito.when(supplier.name()).thenReturn(this.params.name()); + Mockito.when(supplier.graphSpace()).thenReturn(this.graph.graphSpace()); + Mockito.when(supplier.spaceGraphName()).thenReturn(this.params.spaceGraphName()); + HugeGraphParams params = Mockito.mock(HugeGraphParams.class); + Mockito.when(params.graph()).thenReturn(supplier); + Mockito.when(params.name()).thenReturn(this.params.name()); + Mockito.when(params.spaceGraphName()).thenReturn(this.params.spaceGraphName()); + Mockito.when(params.configuration()).thenReturn(this.params.configuration()); + Mockito.when(params.schemaEventHub()).thenReturn(this.params.schemaEventHub()); + Mockito.when(params.loadGraphStore()).thenReturn(this.params.loadGraphStore()); + return params; + } + + @SuppressWarnings("unchecked") + private static ConcurrentMap v2CacheEventListeners() throws Exception { + Field field = CachedSchemaTransactionV2.class.getDeclaredField("SCHEMA_CACHE_EVENT_LISTENERS"); + field.setAccessible(true); + return (ConcurrentMap) field.get(null); + } } diff --git a/hugegraph-server/hugegraph-test/src/main/java/org/apache/hugegraph/unit/cache/RequestCacheLifetimeTest.java b/hugegraph-server/hugegraph-test/src/main/java/org/apache/hugegraph/unit/cache/RequestCacheLifetimeTest.java new file mode 100644 index 0000000000..812343be30 --- /dev/null +++ b/hugegraph-server/hugegraph-test/src/main/java/org/apache/hugegraph/unit/cache/RequestCacheLifetimeTest.java @@ -0,0 +1,187 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hugegraph.unit.cache; + +import java.lang.reflect.Field; +import java.lang.reflect.Method; +import java.util.Collections; +import java.nio.file.Files; +import java.nio.file.Path; + +import org.apache.hugegraph.HugeFactory; +import org.apache.hugegraph.HugeGraph; +import org.apache.hugegraph.HugeGraphParams; +import org.apache.hugegraph.api.graph.VertexAPI; +import org.apache.hugegraph.api.graph.EdgeAPI; +import org.apache.hugegraph.core.GraphManager; +import org.apache.hugegraph.util.JsonUtil; +import org.apache.hugegraph.backend.cache.CachedGraphTransaction; +import org.apache.hugegraph.backend.cache.CachedSchemaTransaction; +import org.apache.hugegraph.util.Events; +import org.apache.hugegraph.dist.RegisterUtil; +import org.apache.hugegraph.testutil.Assert; +import org.apache.hugegraph.testutil.Whitebox; +import org.apache.hugegraph.unit.FakeObjects; +import org.apache.hugegraph.config.HugeConfig; +import org.apache.tinkerpop.gremlin.structure.Direction; +import org.apache.tinkerpop.gremlin.structure.Edge; +import org.apache.tinkerpop.gremlin.structure.T; +import org.apache.tinkerpop.gremlin.structure.Vertex; +import org.junit.Test; +import sun.misc.Unsafe; + +public class RequestCacheLifetimeTest { + + @Test + public void testVertexSchemaAppendAfterRequestCleanupWithWarmElementCaches() throws Exception { + this.checkSchemaAppendAfterRequestCleanup(false); + } + + @Test + public void testEdgeSchemaAppendAfterRequestCleanupWithWarmElementCaches() throws Exception { + this.checkSchemaAppendAfterRequestCleanup(true); + } + + @Test + public void testFailedReopenPreservesCauseAndDeletesDirectory() throws Exception { + Path path = Files.createTempDirectory("request-cache-reopen-failure"); + RuntimeException failure = null; + try { + this.checkSchemaAppendAfterRequestCleanup(false, path, true); + Assert.fail("The invalid backend must fail during reopen"); + } catch (RuntimeException expected) { + failure = expected; + } + System.out.println("REOPEN_FAILURE=" + failure + "; DIRECTORY_EXISTS=" + Files.exists(path)); + Assert.assertNotNull(failure); + Assert.assertTrue(failure instanceof org.apache.hugegraph.exception.HugeException); + Assert.assertEquals("Failed to load backend store provider", failure.getMessage()); + Assert.assertFalse(Files.exists(path)); + } + + private void checkSchemaAppendAfterRequestCleanup(boolean edgeFirst) throws Exception { + Path path = Files.createTempDirectory("request-cache-lifetime"); + this.checkSchemaAppendAfterRequestCleanup(edgeFirst, path, false); + } + + private void checkSchemaAppendAfterRequestCleanup(boolean edgeFirst, Path path, + boolean failReopen) throws Exception { + RegisterUtil.registerBackends(); + HugeConfig config = FakeObjects.newConfig(); + config.setProperty("backend", "rocksdb"); + config.setProperty("serializer", "binary"); + config.setProperty("store", edgeFirst ? "request_cache_lifetime_edge" : "request_cache_lifetime_vertex"); + config.setProperty("rocksdb.data_path", path.toString()); + config.setProperty("rocksdb.wal_path", path.toString()); + HugeGraph graph = HugeFactory.open(config); + try { + graph.initBackend(); + graph.schema().vertexLabel("person").useCustomizeNumberId().create(); + graph.schema().edgeLabel("knows").sourceLabel("person").targetLabel("person").create(); + Vertex first = graph.addVertex(T.label, "person", T.id, 1L); + Vertex second = graph.addVertex(T.label, "person", T.id, 2L); + Edge edge = first.addEdge("knows", second); + Object edgeId = edge.id(); + graph.tx().commit(); + Vertex warm = graph.vertices(1L).next(); + warm.edges(Direction.OUT, "knows").next(); + HugeGraphParams params = Whitebox.getInternalState(graph, "params"); + CachedGraphTransaction tx = (CachedGraphTransaction) params.graphTransaction(); + Assert.assertTrue((long) Whitebox.invoke(tx, "verticesCache", "size") > 0L); + Assert.assertTrue((long) Whitebox.invoke(tx, "edgesCache", "size") > 0L); + HugeFactory.closeCurrentThreadTransactions(); + + graph.schema().propertyKey("added").asText().create(); + graph.schema().vertexLabel("person").properties("added").nullableKeys("added").append(); + graph.schema().edgeLabel("knows").properties("added").nullableKeys("added").append(); + HugeFactory.closeCurrentThreadTransactions(); + + Vertex refetched = graph.vertices(1L).next(); + Edge refetchedEdge = refetched.edges(Direction.OUT, "knows").next(); + if (edgeFirst) { + refetchedEdge.property("added", "edge-value"); + refetched.property("added", "vertex-value"); + } else { + refetched.property("added", "vertex-value"); + refetchedEdge.property("added", "edge-value"); + } + graph.tx().commit(); + HugeFactory.closeCurrentThreadTransactions(); + Assert.assertEquals("vertex-value", graph.vertices(1L).next().value("added")); + Assert.assertEquals("edge-value", graph.edges(edgeId).next().value("added")); + HugeFactory.closeCurrentThreadTransactions(); + updateThroughApi(graph, VertexAPI.class, "JsonVertex", "1", "vertex-api-value"); + HugeFactory.closeCurrentThreadTransactions(); + updateThroughApi(graph, EdgeAPI.class, "JsonEdge", edgeId.toString(), "edge-api-value"); + HugeFactory.closeCurrentThreadTransactions(); + Assert.assertEquals("vertex-api-value", graph.vertices(1L).next().value("added")); + Assert.assertEquals("edge-api-value", graph.edges(edgeId).next().value("added")); + CachedGraphTransaction idleGraph = (CachedGraphTransaction) params.graphTransaction(); + CachedSchemaTransaction idleSchema = (CachedSchemaTransaction) params.schemaTransaction(); + HugeFactory.closeCurrentThreadTransactions(); + Assert.assertTrue((long) Whitebox.invoke(idleGraph, "verticesCache", "size") > 0L); + Assert.assertTrue((long) Whitebox.invoke(idleSchema, "idCache", "size") > 0L); + params.loadGraphStore().provider().storeEventHub() + .notify(Events.STORE_CLEAR, params.loadGraphStore().provider()).get(); + Assert.assertEquals(0L, Whitebox.invoke(idleGraph, "verticesCache", "size")); + Assert.assertEquals(0L, Whitebox.invoke(idleGraph, "edgesCache", "size")); + Assert.assertEquals(0L, Whitebox.invoke(idleSchema, "idCache", "size")); + graph.close(); + graph = null; + if (failReopen) { + config.setProperty("backend", "invalid_reopen_backend"); + } + graph = HugeFactory.open(config); + Assert.assertEquals("vertex-api-value", graph.vertices(1L).next().value("added")); + Assert.assertEquals("edge-api-value", graph.edges(edgeId).next().value("added")); + } finally { + try { + HugeFactory.closeCurrentThreadTransactions(); + if (graph != null) { + try { + graph.clearBackend(); + } finally { + graph.close(); + } + } + } finally { + try (java.util.stream.Stream files = Files.walk(path)) { + for (Path file : files.sorted(java.util.Comparator.reverseOrder()).toArray(Path[]::new)) { + Files.deleteIfExists(file); + } + } + } + } + } + + private static void updateThroughApi(HugeGraph graph, Class api, String bodyClass, + String id, String value) throws Exception { + // Match existing API fixtures: avoid constructor-started RPC/PD services. + Field field = Unsafe.class.getDeclaredField("theUnsafe"); + field.setAccessible(true); + GraphManager manager = (GraphManager) ((Unsafe) field.get(null)).allocateInstance(GraphManager.class); + Whitebox.setInternalState(manager, "graphs", Collections.singletonMap(graph.spaceGraphName(), graph)); + Class bodyType = Class.forName(api.getName() + "$" + bodyClass); + Object body = JsonUtil.fromJson("{\"properties\":{\"added\":\"" + value + "\"}}", bodyType); + Method update = api.getMethod("update", GraphManager.class, String.class, String.class, + String.class, String.class, bodyType); + String response = (String) update.invoke(api.getConstructor().newInstance(), manager, + "DEFAULT", graph.name(), id, "append", body); + Assert.assertTrue(response.contains(value)); + } +} diff --git a/hugegraph-server/hugegraph-test/src/main/java/org/apache/hugegraph/unit/rocksdb/RocksDBSessionsTest.java b/hugegraph-server/hugegraph-test/src/main/java/org/apache/hugegraph/unit/rocksdb/RocksDBSessionsTest.java index 5dbc96c5c6..5c8b14c71f 100644 --- a/hugegraph-server/hugegraph-test/src/main/java/org/apache/hugegraph/unit/rocksdb/RocksDBSessionsTest.java +++ b/hugegraph-server/hugegraph-test/src/main/java/org/apache/hugegraph/unit/rocksdb/RocksDBSessionsTest.java @@ -19,24 +19,378 @@ import java.io.File; import java.io.IOException; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicReference; import org.apache.commons.io.FileUtils; import org.apache.hugegraph.backend.store.rocksdb.RocksDBMetrics; import org.apache.hugegraph.backend.store.rocksdb.RocksDBOptions; import org.apache.hugegraph.backend.store.rocksdb.RocksDBSessions; import org.apache.hugegraph.backend.store.rocksdb.RocksDBStdSessions; +import org.apache.hugegraph.exception.BackendException; +import org.apache.hugegraph.backend.store.rocksdb.RocksDBStore; +import org.apache.hugegraph.backend.store.rocksdb.RocksDBStoreProvider; import org.apache.hugegraph.backend.store.rocksdbsst.RocksDBSstSessions; import org.apache.hugegraph.config.HugeConfig; import org.apache.hugegraph.testutil.Assert; +import org.apache.hugegraph.testutil.Whitebox; +import org.apache.hugegraph.type.HugeType; import org.apache.hugegraph.unit.FakeObjects; import org.junit.Test; +import org.mockito.AdditionalAnswers; +import org.mockito.Mockito; +import org.rocksdb.ColumnFamilyHandle; +import org.rocksdb.RocksDB; +import org.rocksdb.RocksIterator; import org.rocksdb.RocksDBException; +import org.rocksdb.WriteBatch; +import org.rocksdb.WriteOptions; import com.google.common.collect.ImmutableList; import com.google.common.collect.ImmutableSet; public class RocksDBSessionsTest extends BaseRocksDBUnitTest { + @Test + public void testResetDiscardsPendingWritesWithoutLeakingNativeBatch() throws Exception { + RocksDBSessions.Session session = this.rocks.session(); + WriteBatch previous = Whitebox.getInternalState(session, "batch"); + for (int i = 0; i < 32; i++) { + session.put(TABLE, getBytes("pending"), getBytes("value")); + this.rocks.forceResetSessions(); + WriteBatch current = Whitebox.getInternalState(session, "batch"); + Assert.assertFalse(session.hasChanges()); + Assert.assertTrue(current.isOwningHandle()); + Assert.assertTrue(current == previous || !previous.isOwningHandle()); + previous = current; + } + session.put(TABLE, getBytes("retained"), getBytes("value")); + session.commit(); + Assert.assertNull(this.get("pending")); + Assert.assertEquals("value", this.get("retained")); + } + + @Test + public void testFinalDetachDisposesNativeOwners() throws Exception { + RocksDBSessions.Session session = this.rocks.session(); + WriteBatch batch = Whitebox.getInternalState(session, "batch"); + WriteOptions options = Whitebox.getInternalState(session, "writeOptions"); + // Keep the DB alive while this worker releases its final request lease. + CountDownLatch attached = new CountDownLatch(1); + CountDownLatch release = new CountDownLatch(1); + AtomicReference failure = new AtomicReference<>(); + Thread keeper = new Thread(() -> { + try { + this.rocks.session(); + attached.countDown(); + release.await(); + } catch (Throwable e) { + failure.set(e); + } finally { + attached.countDown(); + try { + this.rocks.close(); + } catch (Throwable e) { + failure.compareAndSet(null, e); + } + } + }); + keeper.start(); + try { + Assert.assertTrue(attached.await(30, TimeUnit.SECONDS)); + Assert.assertNull(failure.get()); + Assert.assertSame(session, this.rocks.useSession()); + Assert.assertFalse(this.rocks.close()); + Assert.assertTrue(batch.isOwningHandle()); + Assert.assertTrue(options.isOwningHandle()); + session.put(TABLE, getBytes("lease"), getBytes("retained")); + session.commit(); + Assert.assertFalse(this.rocks.close()); + Assert.assertFalse(batch.isOwningHandle()); + Assert.assertFalse(options.isOwningHandle()); + // Native close is idempotent even after the pool removed this session. + session.close(); + Assert.assertFalse(batch.isOwningHandle()); + Assert.assertFalse(options.isOwningHandle()); + for (int i = 0; i < 32; i++) { + RocksDBSessions.Session request = this.rocks.session(); + Assert.assertNotSame(session, request); + WriteBatch requestBatch = Whitebox.getInternalState(request, "batch"); + WriteOptions requestOptions = Whitebox.getInternalState(request, "writeOptions"); + Assert.assertTrue(requestBatch.isOwningHandle()); + Assert.assertTrue(requestOptions.isOwningHandle()); + Assert.assertArrayEquals(getBytes("retained"), request.get(TABLE, getBytes("lease"))); + request.put(TABLE, getBytes("request"), getBytes("value")); + request.commit(); + Assert.assertFalse(this.rocks.close()); + Assert.assertFalse(requestBatch.isOwningHandle()); + Assert.assertFalse(requestOptions.isOwningHandle()); + } + } finally { + // Leave the fixture's main-worker session open for ordinary teardown. + this.rocks.session(); + release.countDown(); + keeper.join(TimeUnit.SECONDS.toMillis(30)); + } + Assert.assertFalse(keeper.isAlive()); + Assert.assertNull(failure.get()); + Assert.assertEquals("retained", this.get("lease")); + } + + @Test + public void testAdapterToplingTruncateWithMultipleKeys() throws Exception { + this.assertAdapterTruncate(true); + } + + @Test + public void testAdapterStandardTruncateWithMultipleKeys() throws Exception { + this.assertAdapterTruncate(false); + } + + @Test + public void testAdapterToplingTruncateDiscardsPendingWritesInEmptyTable() throws Exception { + this.assertAdapterTruncateDiscardsPendingWrites(true, false); + } + + @Test + public void testAdapterToplingTruncateDiscardsPendingWritesOutsideRange() throws Exception { + this.assertAdapterTruncateDiscardsPendingWrites(true, true); + } + + @Test + public void testAdapterStandardTruncateDiscardsPendingWritesInEmptyTable() throws Exception { + this.assertAdapterTruncateDiscardsPendingWrites(false, false); + } + + @Test + public void testAdapterStandardTruncateDiscardsPendingWritesOutsideRange() throws Exception { + this.assertAdapterTruncateDiscardsPendingWrites(false, true); + } + + @Test + public void testAdapterToplingTruncatePreservesBackendVersion() throws Exception { + this.assertAdapterTruncatePreservesBackendVersion(true); + } + + @Test + public void testAdapterStandardTruncatePreservesBackendVersion() throws Exception { + this.assertAdapterTruncatePreservesBackendVersion(false); + } + + @Test + public void testToplingTruncateRoutesDynamicOlapTablesToTheirDatabase() throws Exception { + String dynamic = "graph+ap_123"; + String olapPath = DB_PATH + "/independent-olap"; + RocksDBStdSessions olap = new RocksDBStdSessions(FakeObjects.newConfig(), "db", "olap", olapPath, olapPath); + try { + olap.createTable(dynamic); + olap.session().put(dynamic, getBytes("key"), getBytes("olap value")); + olap.session().commit(); + this.put("main", "main value"); + RocksDBStore store = new RocksDBStore.RocksDBGraphStore(null, "db", "graph") { + @Override + protected List tableNames() { + return ImmutableList.of(TABLE, dynamic); + } + + @Override + protected List olapTables() { + return ImmutableList.of(dynamic); + } + + @Override + protected Map tableDBMapping() { + return java.util.Collections.singletonMap(HugeType.OLAP.string(), olap); + } + }; + Whitebox.setInternalState(store, "sessions", this.rocks); + Map databases = Whitebox.getInternalState(store, "dbs"); + databases.put(DB_PATH, this.rocks); + databases.put(olapPath, olap); + Whitebox.setInternalState(store, "toplingProvider", true); + // Real standard JNI exercises the Topling Java routing branch. + Assert.assertFalse(this.rocks.existsTable(dynamic)); + store.truncate(); + Assert.assertNull(this.get("main")); + Assert.assertTrue(olap.existsTable(dynamic)); + Assert.assertNull(olap.session().get(dynamic, getBytes("key"))); + olap.session().put(dynamic, getBytes("next"), getBytes("after truncate")); + olap.session().commit(); + Assert.assertArrayEquals(getBytes("after truncate"), olap.session().get(dynamic, getBytes("next"))); + } finally { + olap.close(); + } + } + + private void assertAdapterTruncatePreservesBackendVersion(boolean topling) throws Exception { + RocksDBStoreProvider provider = new RocksDBStoreProvider(); + RocksDBStore store = new RocksDBStore.RocksDBSystemStore(provider, "db", "store") { + @Override + protected List tableNames() { + return ImmutableList.builder().addAll(super.tableNames()).add(TABLE).build(); + } + }; + Whitebox.setInternalState(store, "sessions", this.rocks); + Map databases = Whitebox.getInternalState(store, "dbs"); + databases.put(DB_PATH, this.rocks); + Whitebox.setInternalState(store, "toplingProvider", topling); + store.init(); + this.put("old", "before truncate"); + Assert.assertEquals(provider.driverVersion(), store.storedVersion()); + + store.truncate(); + + Assert.assertNull(this.get("old")); + Assert.assertEquals(provider.driverVersion(), store.storedVersion()); + this.put("new", "after truncate"); + this.rocks.close(); + this.rocks = new RocksDBStdSessions(FakeObjects.newConfig(), "db", "store", + DB_PATH, DB_PATH, ImmutableList.of(TABLE)); + this.rocks.session(); + Whitebox.setInternalState(store, "sessions", this.rocks); + databases.clear(); + databases.put(DB_PATH, this.rocks); + // Reopening must read the persisted version without init repairing it. + Assert.assertEquals(provider.driverVersion(), store.storedVersion()); + Assert.assertEquals("after truncate", this.get("new")); + } + + private void assertAdapterTruncateDiscardsPendingWrites(boolean topling, boolean committedRange) throws Exception { + if (committedRange) { + this.put("m", "committed first"); + this.put("n", "committed last"); + } + RocksDBSessions.Session session = this.rocks.session(); + // Both inserts fall outside the committed range; an empty CF has no range at all. + session.put(TABLE, getBytes("a"), getBytes("pending first")); + session.put(TABLE, getBytes("z"), getBytes("pending last")); + Assert.assertTrue(session.hasChanges()); + + RocksDBStore store = this.adapterStore(topling, ImmutableList.of(TABLE)); + store.truncate(); + + Assert.assertFalse(session.hasChanges()); + Assert.assertNull(session.keyRange(TABLE)); + Assert.assertNull(session.get(TABLE, getBytes("a"))); + Assert.assertNull(session.get(TABLE, getBytes("z"))); + // The retained session must not replay pending writes after truncate either. + session.commit(); + Assert.assertNull(session.keyRange(TABLE)); + this.put("new", "after truncate"); + Assert.assertEquals("after truncate", this.get("new")); + } + + private void assertAdapterTruncate(boolean topling) throws Exception { + List tables = ImmutableList.of(TABLE, "single", "empty", "multiple"); + this.rocks.createTable("single", "empty", "multiple"); + // Deliberately unordered, including empty and unsigned byte boundaries. + byte[][] keys = {new byte[]{(byte) 0xff}, new byte[]{0}, new byte[]{}, + new byte[]{(byte) 0x80}, new byte[]{0, (byte) 0xff}, new byte[]{0x7f}}; + for (String table : ImmutableList.of(TABLE, "multiple")) { + for (byte[] key : keys) { + this.rocks.session().put(table, key, new byte[]{42}); + } + } + this.rocks.session().put("single", new byte[]{(byte) 0xff}, new byte[]{43}); + this.commit(); + Object opened = Whitebox.getInternalState(this.rocks, "rocksdb"); + Map handles = Whitebox.getInternalState(opened, "cfHandles"); + Map before = new HashMap<>(handles); + RocksDBStore store = this.adapterStore(topling, tables); + store.truncate(); + for (String table : tables) { + Assert.assertTrue(this.rocks.existsTable(table)); + Assert.assertNull(this.rocks.session().keyRange(table)); + for (byte[] key : keys) { + Assert.assertNull(this.rocks.session().get(table, key)); + } + if (topling) { + Assert.assertSame(before.get(table), handles.get(table)); + } else { + Assert.assertNotSame(before.get(table), handles.get(table)); + } + } + for (String table : tables) { + this.rocks.session().put(table, new byte[]{(byte) 0xff}, new byte[]{44}); + } + this.commit(); + for (String table : tables) { + Assert.assertArrayEquals(new byte[]{44}, this.rocks.session().get(table, new byte[]{(byte) 0xff})); + } + // A second truncate covers the single-key path for every formerly empty CF. + store.truncate(); + for (String table : tables) { + Assert.assertNull(this.rocks.session().keyRange(table)); + } + } + + @Test + public void testTruncatePropagatesFirstKeyReadFailure() throws Exception { + this.assertTruncateKeyReadFailure(false); + } + + @Test + public void testTruncatePropagatesLastKeyReadFailure() throws Exception { + this.assertTruncateKeyReadFailure(true); + } + + private void assertTruncateKeyReadFailure(boolean lastKey) throws Exception { + String brokenTable = "broken"; + this.rocks.createTable(brokenTable); + this.put("healthy", "retained"); + this.rocks.session().put(brokenTable, getBytes("a"), getBytes("first")); + this.rocks.session().put(brokenTable, getBytes("z"), getBytes("last")); + this.commit(); + RocksDBStore store = this.adapterStore(true, ImmutableList.of(TABLE, brokenTable)); + Object opened = Whitebox.getInternalState(this.rocks, "rocksdb"); + RocksDB realDB = Whitebox.getInternalState(opened, "rocksdb"); + Map handles = Whitebox.getInternalState(opened, "cfHandles"); + ColumnFamilyHandle brokenHandle = Whitebox.getInternalState(handles.get(brokenTable), "handle"); + RocksIterator brokenIterator = Mockito.mock(RocksIterator.class); + Mockito.when(brokenIterator.isValid()).thenReturn(lastKey, false); + Mockito.when(brokenIterator.key()).thenReturn(getBytes("a")); + RocksDBException readFailure = new RocksDBException("injected iterator read failure"); + Mockito.doThrow(readFailure).when(brokenIterator).status(); + // Keep a real DB and healthy iterator; replace only the failing native + // boundary. The production truncate -> clearTables -> keyRange chain runs. + RocksDB faultDB = Mockito.mock(RocksDB.class, AdditionalAnswers.delegatesTo(realDB)); + Mockito.doReturn(brokenIterator).when(faultDB).newIterator(brokenHandle); + Whitebox.setInternalState(opened, "rocksdb", faultDB); + try { + Throwable failure = Assert.assertThrows(BackendException.class, store::truncate); + Assert.assertSame(readFailure, failure.getCause()); + Mockito.verify(brokenIterator).status(); + Mockito.verify(brokenIterator).close(); + } finally { + Whitebox.setInternalState(opened, "rocksdb", realDB); + } + Assert.assertEquals("retained", this.get("healthy")); + Assert.assertArrayEquals(getBytes("first"), this.rocks.session().get(brokenTable, getBytes("a"))); + Assert.assertArrayEquals(getBytes("last"), this.rocks.session().get(brokenTable, getBytes("z"))); + Assert.assertFalse(this.rocks.session().hasChanges()); + Assert.assertSame(brokenHandle, Whitebox.getInternalState(handles.get(brokenTable), "handle")); + } + + private RocksDBStore adapterStore(boolean topling, List tables) { + RocksDBStore store = new RocksDBStore.RocksDBGraphStore(new RocksDBStoreProvider(), "db", "store") { + @Override + protected List tableNames() { + return tables; + } + }; + Whitebox.setInternalState(store, "sessions", this.rocks); + Map databases = Whitebox.getInternalState(store, "dbs"); + databases.put(DB_PATH, this.rocks); + // Select the actual adapter branch, independently of the loaded JNI. + // This fixture also runs unchanged with the externally loaded TP JNI on Linux. + Whitebox.setInternalState(store, "toplingProvider", topling); + return store; + } + @Test public void testTable() throws RocksDBException { final String TABLE2 = "test-table2"; diff --git a/hugegraph-server/hugegraph-test/src/main/java/org/apache/hugegraph/unit/util/RateLimiterTest.java b/hugegraph-server/hugegraph-test/src/main/java/org/apache/hugegraph/unit/util/RateLimiterTest.java index 111b639421..4ace252175 100644 --- a/hugegraph-server/hugegraph-test/src/main/java/org/apache/hugegraph/unit/util/RateLimiterTest.java +++ b/hugegraph-server/hugegraph-test/src/main/java/org/apache/hugegraph/unit/util/RateLimiterTest.java @@ -17,6 +17,9 @@ package org.apache.hugegraph.unit.util; +import java.lang.reflect.Constructor; +import java.util.Timer; +import java.util.TimerTask; import java.util.concurrent.atomic.LongAdder; import org.apache.hugegraph.testutil.Assert; @@ -74,6 +77,55 @@ public RateLimiter newRateLimiter(int rate) { return new FixedTimerWindowRateLimiter(rate); } + @Test + public void testInitialTimerWindow() throws Exception { + try (RecordingTimer timer = new RecordingTimer()) { + Constructor constructor = + FixedTimerWindowRateLimiter.class.getDeclaredConstructor(int.class, Timer.class); + constructor.setAccessible(true); + RateLimiter limiter = constructor.newInstance(400, timer); + + // The initial reset must wait for a complete window, even if the timer thread starts late. + Assert.assertEquals(RateLimiter.RESET_PERIOD, timer.delay); + Assert.assertEquals(RateLimiter.RESET_PERIOD, timer.period); + for (int i = 0; i < 400; i++) { + Assert.assertTrue(limiter.tryAcquire()); + } + LongAdder count = Whitebox.getInternalState(limiter, "count"); + Assert.assertEquals(400, count.intValue()); + Assert.assertFalse(limiter.tryAcquire()); + + timer.task.run(); + Assert.assertEquals(0, count.intValue()); + Assert.assertTrue(limiter.tryAcquire()); + timer.task.run(); + Assert.assertEquals(1, count.intValue()); + } + } + + private static class RecordingTimer extends Timer implements AutoCloseable { + + private TimerTask task; + private long delay; + private long period; + + private RecordingTimer() { + super("RateLimiterTest", true); + } + + @Override + public void schedule(TimerTask task, long delay, long period) { + this.task = task; + this.delay = delay; + this.period = period; + } + + @Override + public void close() { + this.cancel(); + } + } + @Test public void testTryAcquireAfterPeriod() throws Exception { int rateLimit = 400; diff --git a/hugegraph-server/pom.xml b/hugegraph-server/pom.xml index 13a2d5361c..586ab10b04 100644 --- a/hugegraph-server/pom.xml +++ b/hugegraph-server/pom.xml @@ -51,7 +51,6 @@ 3.21.0-GA 3.1.2 8.45 - 1.47.0 3.21.7 1.36 3.7.1 @@ -239,7 +238,7 @@ com.google.protobuf protobuf-java - ${protobuf.version} + ${protobuf.runtime.version} diff --git a/hugegraph-store/hg-store-client/pom.xml b/hugegraph-store/hg-store-client/pom.xml index d3d5eae41c..ae5089af8e 100644 --- a/hugegraph-store/hg-store-client/pom.xml +++ b/hugegraph-store/hg-store-client/pom.xml @@ -76,7 +76,7 @@ com.google.protobuf protobuf-java-util - 3.17.2 + ${protobuf.runtime.version} diff --git a/hugegraph-store/hg-store-core/src/main/java/org/apache/hugegraph/store/HeartbeatService.java b/hugegraph-store/hg-store-core/src/main/java/org/apache/hugegraph/store/HeartbeatService.java index fe48248c0c..73dbfceb7a 100644 --- a/hugegraph-store/hg-store-core/src/main/java/org/apache/hugegraph/store/HeartbeatService.java +++ b/hugegraph-store/hg-store-core/src/main/java/org/apache/hugegraph/store/HeartbeatService.java @@ -23,6 +23,7 @@ import java.util.Collections; import java.util.List; import java.util.Map; +import java.util.concurrent.atomic.AtomicBoolean; import org.apache.hugegraph.pd.common.PDException; import org.apache.hugegraph.pd.common.PDRuntimeException; @@ -63,11 +64,15 @@ public class HeartbeatService implements Lifecycle, Partit private final List stateListeners; private final Object partitionThreadLock = new Object(); private final Object storeThreadLock = new Object(); + private final Object stateChangeLock = new Object(); private int heartbeatFailCount = 0; private int reportErrCount = 0; // Thread sleep time private volatile int timerNextDelay = 1000; private volatile boolean terminated = false; + private final AtomicBoolean fatalExitRequested = new AtomicBoolean(); + private Thread storeHeartbeatThread; + private Thread partitionHeartbeatThread; public HeartbeatService(HgStoreEngine storeEngine) { this.storeEngine = storeEngine; @@ -90,8 +95,10 @@ public boolean init(HgStoreEngineOptions opts) { storeInfo.setDeployPath(HeartbeatService.class.getResource("/").getPath()); storeInfo.setDataPath(options.getDataPath()); this.pdProvider = options.getPdProvider(); - new Thread(() -> doStoreHeartbeat(), PoolNames.HEARTBEAT).start(); - new Thread(() -> doPartitionHeartbeat(), PoolNames.P_HEARTBEAT).start(); + storeHeartbeatThread = new Thread(this::doStoreHeartbeat, PoolNames.HEARTBEAT); + partitionHeartbeatThread = new Thread(this::doPartitionHeartbeat, PoolNames.P_HEARTBEAT); + storeHeartbeatThread.start(); + partitionHeartbeatThread.start(); return true; } @@ -137,7 +144,9 @@ protected void doStoreHeartbeat() { } synchronized (storeThreadLock) { - storeThreadLock.wait(timerNextDelay); + if (!terminated) { + storeThreadLock.wait(timerNextDelay); + } } } catch (Throwable e) { if (e instanceof PDRuntimeException && @@ -145,6 +154,9 @@ protected void doStoreHeartbeat() { log.error("store heartbeat error: PD UNREACHABLE"); synchronized (storeThreadLock) { try { + if (terminated) { + break; + } if (timerNextDelay < 10000) { storeThreadLock.wait(timerNextDelay); } else { @@ -171,7 +183,9 @@ protected void doPartitionHeartbeat() { } try { synchronized (partitionThreadLock) { - partitionThreadLock.wait(options.getPartitionHBInterval() * 1000L); + if (!terminated) { + partitionThreadLock.wait(options.getPartitionHBInterval() * 1000L); + } } } catch (InterruptedException e) { log.error("doPartitionHeartbeat error: ", e); @@ -215,15 +229,18 @@ protected void registerStore() { if (exceptCode == ErrorType.STORE_ID_NOT_EXIST_VALUE) { log.error("The store ID {} does not match the PD. Check that the correct PD is " + "connected, " + "and then delete the store ID!!!", storeInfo.getId()); - System.exit(-1); + requestFatalExit(); + return; } else if (exceptCode == ErrorType.STORE_HAS_BEEN_REMOVED_VALUE) { log.error("The store ID {} has been removed, please delete all data and restart!", storeInfo.getId()); - System.exit(-1); + requestFatalExit(); + return; } else if (exceptCode == ErrorType.STORE_PROHIBIT_DUPLICATE_VALUE) { log.error("The store ID {} maybe duplicated, please check out store raft address " + "and restart later!", storeInfo.getId()); - System.exit(-1); + requestFatalExit(); + return; } } } @@ -240,11 +257,13 @@ protected void storeHeartbeat() { if (exceptCode == ErrorType.STORE_ID_NOT_EXIST_VALUE) { log.error("The store ID {} does not match the PD. Check that the correct PD is " + "connected, and then delete the store ID!!!", storeInfo.getId()); - System.exit(-1); + requestFatalExit(); + return; } else if (exceptCode == ErrorType.STORE_HAS_BEEN_REMOVED_VALUE) { log.error("The store ID {} has been removed, please delete all data and restart!", storeInfo.getId()); - System.exit(-1); + requestFatalExit(); + return; } } if (clusterStats == null || clusterStats.getState() == null) { @@ -279,11 +298,19 @@ protected void storeHeartbeat() { } protected synchronized void onStateChanged(Metapb.StoreState newState) { + if (terminated) { + return; + } Utils.runInThread(() -> { - Metapb.StoreState oldState = this.storeInfo.getState(); - this.storeInfo.setState(newState); - stateListeners.forEach((e) -> - e.stateChanged(this.storeInfo, oldState, newState)); + synchronized (stateChangeLock) { + if (terminated) { + return; + } + Metapb.StoreState oldState = this.storeInfo.getState(); + this.storeInfo.setState(newState); + stateListeners.forEach((e) -> + e.stateChanged(this.storeInfo, oldState, newState)); + } }); } @@ -371,9 +398,54 @@ public void monitorMemory() { @Override public void shutdown() { log.info("HeartbeatService shutdown"); + stopProducers(); + joinHeartbeat(storeHeartbeatThread); + joinHeartbeat(partitionHeartbeatThread); + // Producers may have submitted callbacks that create partition owners. + // Never hold this monitor while joining the producer threads. + synchronized (stateChangeLock) { + if (Thread.currentThread().isInterrupted()) { + throw new IllegalStateException("Interrupted while stopping store state callbacks"); + } + } + } + + private void requestFatalExit() { + requestExit(-1); + } + + void requestExit(int status) { + if (!fatalExitRequested.compareAndSet(false, true)) { + return; + } + stopProducers(); + // Hooks join producers and wait for state callbacks to release their lock. + // Let the requesting producer or callback return before shutdown waits. + new Thread(() -> System.exit(status), "store-fatal-exit").start(); + } + + private void stopProducers() { terminated = true; + synchronized (storeThreadLock) { + storeThreadLock.notifyAll(); + } synchronized (partitionThreadLock) { - partitionThreadLock.notify(); + partitionThreadLock.notifyAll(); + } + } + + private static void joinHeartbeat(Thread thread) { + if (thread == null) { + return; + } + if (thread == Thread.currentThread()) { + throw new IllegalStateException("Cannot close the store from its heartbeat thread"); + } + try { + thread.join(); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new IllegalStateException("Interrupted while stopping store heartbeat", e); } } diff --git a/hugegraph-store/hg-store-core/src/main/java/org/apache/hugegraph/store/HgStoreEngine.java b/hugegraph-store/hg-store-core/src/main/java/org/apache/hugegraph/store/HgStoreEngine.java index 948b1e8f43..4753c85264 100644 --- a/hugegraph-store/hg-store-core/src/main/java/org/apache/hugegraph/store/HgStoreEngine.java +++ b/hugegraph-store/hg-store-core/src/main/java/org/apache/hugegraph/store/HgStoreEngine.java @@ -254,7 +254,13 @@ public void stateChanged(Store store, Metapb.StoreState oldState, Metapb.StoreSt if (newState == Metapb.StoreState.Up) { // Status changes to online, record store information partitionManager.setStore(store); - partitionManager.loadPartition(); + try { + partitionManager.loadPartition(); + } catch (PartitionManager.InvalidShardException e) { + // Never wait for shutdown hooks while owning the state callback lock. + heartbeatService.requestExit(0); + return; + } restoreLocalPartitionEngine(); } } diff --git a/hugegraph-store/hg-store-core/src/main/java/org/apache/hugegraph/store/meta/PartitionManager.java b/hugegraph-store/hg-store-core/src/main/java/org/apache/hugegraph/store/meta/PartitionManager.java index d078466c68..9325ff71d9 100644 --- a/hugegraph-store/hg-store-core/src/main/java/org/apache/hugegraph/store/meta/PartitionManager.java +++ b/hugegraph-store/hg-store-core/src/main/java/org/apache/hugegraph/store/meta/PartitionManager.java @@ -100,6 +100,15 @@ public void load() { deletedFileManager.load(); } + public static final class InvalidShardException extends IllegalStateException { + + private static final long serialVersionUID = 1L; + + private InvalidShardException() { + super("Local partition does not belong to this store"); + } + } + public void loadPartition() { loadPartitions(); } @@ -299,7 +308,7 @@ private void loadPartitions() { // businessHandler.dbCompaction(graph, partId); log.error("partition {}-{} is illegal. store id {} not in valid shard group:{}", graph, partId, getStore().getId(), shards2Peers(shards)); - System.exit(0); + throw new InvalidShardException(); } } diff --git a/hugegraph-store/hg-store-core/src/main/java/org/apache/hugegraph/store/metric/SystemMetricService.java b/hugegraph-store/hg-store-core/src/main/java/org/apache/hugegraph/store/metric/SystemMetricService.java index c75d32e343..fbae045518 100644 --- a/hugegraph-store/hg-store-core/src/main/java/org/apache/hugegraph/store/metric/SystemMetricService.java +++ b/hugegraph-store/hg-store-core/src/main/java/org/apache/hugegraph/store/metric/SystemMetricService.java @@ -163,8 +163,10 @@ private void loadRocksDbInfo(Map map) { Set names = dbFactory.getGraphNames(); if (names != null) { for (String name : names) { - try { - RocksDBSession session = dbFactory.queryGraphDB(name); + try (RocksDBSession session = dbFactory.queryGraphDB(name)) { + if (session == null) { + continue; + } Statistics statistics = session.getRocksDbStats(); map.put( "rocksdb.graph." + name + "." + diff --git a/hugegraph-store/hg-store-dist/src/assembly/descriptor/server-assembly.xml b/hugegraph-store/hg-store-dist/src/assembly/descriptor/server-assembly.xml index 522a93c917..2e761200ef 100644 --- a/hugegraph-store/hg-store-dist/src/assembly/descriptor/server-assembly.xml +++ b/hugegraph-store/hg-store-dist/src/assembly/descriptor/server-assembly.xml @@ -46,6 +46,24 @@ + + + ${project.basedir}/../../hugegraph-server/hugegraph-dist/src/assembly/static/bin/prepare-topling.sh + bin + 755 + + + ${project.basedir}/../../hugegraph-server/hugegraph-dist/src/assembly/static/bin/preload-topling.sh + bin + 755 + + + ${project.basedir}/../../hugegraph-server/hugegraph-dist/src/assembly/static/conf/toplingdb.yaml + conf + 644 + + + diff --git a/hugegraph-store/hg-store-dist/src/assembly/static/bin/start-hugegraph-store.sh b/hugegraph-store/hg-store-dist/src/assembly/static/bin/start-hugegraph-store.sh index 15006bb777..d20b9e6ec6 100755 --- a/hugegraph-store/hg-store-dist/src/assembly/static/bin/start-hugegraph-store.sh +++ b/hugegraph-store/hg-store-dist/src/assembly/static/bin/start-hugegraph-store.sh @@ -39,6 +39,17 @@ PID_FILE="$BIN/pid" . "$BIN"/util.sh +# Keep caller preloads visible to the runtime selector, including conflicting JNI. +function preload_jemalloc() { + local inherited="${LD_PRELOAD//:/ }" entry + local -a preloads + read -r -a preloads <<< "${inherited//$'\n'/ }" + for entry in "${preloads[@]-}"; do + [ "$entry" != "$1" ] || return 0 + done + export LD_PRELOAD="$1${LD_PRELOAD:+:$LD_PRELOAD}" +} + arch=$(uname -m) echo "Current arch: $arch" @@ -47,7 +58,7 @@ if [[ $arch == "aarch64" || $arch == "arm64" ]]; then download_url="${GITHUB}/apache/hugegraph-doc/raw/binary-1.5/dist/server/libjemalloc_aarch64.so" expected_md5="2a631d2f81837f9d5864586761c5e380" if download_and_verify "$download_url" "$lib_file" "$expected_md5"; then - export LD_PRELOAD="$lib_file" + preload_jemalloc "$lib_file" else echo "Failed to verify or download $lib_file, skip it" fi @@ -56,7 +67,7 @@ elif [[ $arch == "x86_64" ]]; then download_url="${GITHUB}/apache/hugegraph-doc/raw/binary-1.5/dist/server/libjemalloc.so" expected_md5="fd61765eec3bfea961b646c269f298df" if download_and_verify "$download_url" "$lib_file" "$expected_md5"; then - export LD_PRELOAD="$lib_file" + preload_jemalloc "$lib_file" else echo "Failed to verify or download $lib_file, skip it" fi @@ -229,17 +240,28 @@ fi echo "Starting HG-StoreServer..." +source "$BIN/preload-topling.sh" || exit 1 +BOOT_JARS=("${LIB}"/hg-store-node-*.jar) +if [ "${#BOOT_JARS[@]}" -ne 1 ] || [ ! -f "${BOOT_JARS[0]}" ]; then + echo "Error: expected one component executable JAR in $LIB" >&2 + exit 1 +fi +JAVA_MAIN=(-jar "${BOOT_JARS[0]}") +if [ -n "${TOPLING_RUNTIME_CLASSPATH:-}" ]; then + JAVA_MAIN=(-cp "${TOPLING_RUNTIME_CLASSPATH}:${BOOT_JARS[0]}" org.springframework.boot.loader.JarLauncher) +fi + # Turn on security check if [[ $DAEMON == "true" ]]; then echo "Starting HugeGraphStoreServer in daemon mode..." if [[ "${STDOUT_MODE:-false}" == "true" ]]; then - exec ${JAVA} -Dname="HugeGraphStore" ${JVM_OPTIONS} ${JAVA_OPTIONS} -jar \ + exec ${JAVA} -Dname="HugeGraphStore" ${JVM_OPTIONS} ${JAVA_OPTIONS} \ -Dspring.config.location=${CONF}/application.yml \ - ${LIB}/hg-store-node-*.jar & + "${JAVA_MAIN[@]}" "$@" & else - exec ${JAVA} -Dname="HugeGraphStore" ${JVM_OPTIONS} ${JAVA_OPTIONS} -jar \ + exec ${JAVA} -Dname="HugeGraphStore" ${JVM_OPTIONS} ${JAVA_OPTIONS} \ -Dspring.config.location=${CONF}/application.yml \ - ${LIB}/hg-store-node-*.jar >> ${OUTPUT} 2>&1 & + "${JAVA_MAIN[@]}" "$@" >> ${OUTPUT} 2>&1 & fi PID="$!" # Write pid to file @@ -251,12 +273,12 @@ else echo "$$" > "$PID_FILE" echo "[+pid] $$" if [[ "${STDOUT_MODE:-false}" == "true" ]]; then - exec ${JAVA} -Dname="HugeGraphStore" ${JVM_OPTIONS} ${JAVA_OPTIONS} -jar \ + exec ${JAVA} -Dname="HugeGraphStore" ${JVM_OPTIONS} ${JAVA_OPTIONS} \ -Dspring.config.location=${CONF}/application.yml \ - ${LIB}/hg-store-node-*.jar + "${JAVA_MAIN[@]}" "$@" else - exec ${JAVA} -Dname="HugeGraphStore" ${JVM_OPTIONS} ${JAVA_OPTIONS} -jar \ + exec ${JAVA} -Dname="HugeGraphStore" ${JVM_OPTIONS} ${JAVA_OPTIONS} \ -Dspring.config.location=${CONF}/application.yml \ - ${LIB}/hg-store-node-*.jar >> ${OUTPUT} 2>&1 + "${JAVA_MAIN[@]}" "$@" >> ${OUTPUT} 2>&1 fi fi diff --git a/hugegraph-store/hg-store-dist/src/assembly/static/conf/application-pd.yml b/hugegraph-store/hg-store-dist/src/assembly/static/conf/application-pd.yml index a4a06386d5..56cbdd8e63 100644 --- a/hugegraph-store/hg-store-dist/src/assembly/static/conf/application-pd.yml +++ b/hugegraph-store/hg-store-dist/src/assembly/static/conf/application-pd.yml @@ -26,6 +26,8 @@ management: include: "*" rocksdb: + # Match the explicit launcher opt-in when enabling Topling. + provider: rocksdb # rocksdb total memory usage, force flush to disk when reaching this value # TODO: default this lower, or size it from the container memory limit, or map it from an # HG_STORE_* variable in the Docker entrypoint. A default of 0 only makes AppConfig fall back diff --git a/hugegraph-store/hg-store-dist/src/assembly/travis/test-store-preloads.sh b/hugegraph-store/hg-store-dist/src/assembly/travis/test-store-preloads.sh new file mode 100755 index 0000000000..e3fdc42c21 --- /dev/null +++ b/hugegraph-store/hg-store-dist/src/assembly/travis/test-store-preloads.sh @@ -0,0 +1,143 @@ +#!/bin/bash +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# Complete launcher fixtures with Java/download substitutes, not business JVM or JNI execution. +set -Eeuo pipefail +SOURCE_BIN="$(cd "$(dirname "${BASH_SOURCE[0]}")/../static/bin" && pwd)" +REPO_ROOT="$(cd "$SOURCE_BIN/../../../../../.." && pwd)" +SHARED_BIN="$REPO_ROOT/hugegraph-server/hugegraph-dist/src/assembly/static/bin" +TEST_ROOT=$(mktemp -d) +TEST_ROOT=$(cd "$TEST_ROOT" && pwd -P) +trap 'rm -rf "$TEST_ROOT"' EXIT +mkdir -p "$TEST_ROOT/fake-bin" "$TEST_ROOT/java/bin" +cat > "$TEST_ROOT/fake-bin/uname" <<'MOCK' +#!/bin/bash +case "$1" in -m) echo "$FIXTURE_ARCH" ;; *) echo Linux ;; esac +MOCK +cat > "$TEST_ROOT/fake-bin/ps" <<'MOCK' +#!/bin/bash +exit 0 +MOCK +cat > "$TEST_ROOT/java/bin/java" <<'MOCK' +#!/bin/bash +if [ "$1" = -version ]; then + printf '%s\n' "${LD_PRELOAD:-}" > "$VERSION_CAPTURE" + echo 'openjdk version "17.0.1"' >&2 +else + printf '%s\n' "${LD_PRELOAD:-}" > "$MAIN_CAPTURE" + printf '%s\n' "$@" > "$MAIN_CAPTURE.args" +fi +MOCK +chmod +x "$TEST_ROOT/fake-bin/"* "$TEST_ROOT/java/bin/java" + +run_launcher() { + local arch="$1" download="$2" provider="$3" inherited="$4" name="$5" + local top="$TEST_ROOT/$name" + local -a provider_env=(-u TOPLINGDB_ROCKSDB_PROVIDER) + [ "$provider" = default ] || provider_env=("TOPLINGDB_ROCKSDB_PROVIDER=$provider") + mkdir -p "$top/bin" "$top/conf" "$top/lib" "$top/topling" + cp "$SOURCE_BIN/start-hugegraph-store.sh" "$top/bin/" + cp "$SHARED_BIN/preload-topling.sh" "$top/bin/" + cat > "$top/bin/util.sh" <<'MOCK' +# Substitute only utility calls needed by this launcher fixture. +ensure_path_writable() { mkdir -p "$1"; } +download_and_verify() { + [ "$FIXTURE_DOWNLOAD" = success ] || return 1 + touch "$2" +} +MOCK + touch "$top/lib/hg-store-node-fixture.jar" "$top/topling/rocksdbjni.jar" \ + "$top/topling/librocksdbjni-linux64.so" + (cd "$top/topling" && sha256sum rocksdbjni.jar librocksdbjni-linux64.so > easy-migrate.sha256) + printf '%s\n' 'http: {auto_start_http: false}' > "$top/conf/toplingdb.yaml" + VERSION_CAPTURE="$top/version-preloads" MAIN_CAPTURE="$top/main-preloads" \ + FIXTURE_ARCH="$arch" FIXTURE_DOWNLOAD="$download" JAVA_HOME="$TEST_ROOT/java" \ + JAVA_OPTIONS=-Xmx64m STDOUT_MODE=true LD_PRELOAD="$inherited" \ + PATH="$TEST_ROOT/fake-bin:$PATH" \ + env -u TOPLING_ACTIVE_NATIVE -u TOPLING_RUNTIME_CLASSPATH -u TOPLINGDB_EASY_MIGRATE_CONF \ + -u TOPLING_AUTO_MIGRATE_CONF "${provider_env[@]}" bash "$top/bin/start-hugegraph-store.sh" -d false \ + > "$top/launcher.log" 2>&1 +} + +# Both allocation-download paths preserve tracing/allocator entries before the selector runs. +for arch in x86_64 aarch64 arm64; do + for download in success failure; do + for separator in colon space; do + inherited=/caller/libtrace.so:/caller/liballoc.so + [ "$separator" != space ] || inherited='/caller/libtrace.so /caller/liballoc.so' + name="$arch-$download-$separator" + top="$TEST_ROOT/$name" + jemalloc="$top/bin/libjemalloc.so" + [ "$arch" = x86_64 ] || jemalloc="$top/bin/libjemalloc_aarch64.so" + expected="$inherited" + [ "$download" != success ] || expected="$jemalloc:$inherited" + run_launcher "$arch" "$download" rocksdb "$inherited" "$name" || { + cat "$top/launcher.log" >&2; exit 1; + } + test "$(cat "$top/version-preloads")" = "$expected" + test "$(cat "$top/main-preloads")" = "$expected" + grep -qx -- '-jar' "$top/main-preloads.args" + + default_top="$TEST_ROOT/$name-default" + default_expected="${expected//$top/$default_top}" + run_launcher "$arch" "$download" default "$inherited" "$name-default" + test "$(cat "$default_top/main-preloads")" = "$default_expected" + + # A successful allocator download must not hide recognizable competing JNI. + for provider in default rocksdb topling; do + bad="$name-conflict-$provider" + if run_launcher "$arch" "$download" "$provider" \ + "$inherited:/other/librocksdbjni.so" "$bad"; then + echo "FAIL: competing JNI escaped the full $arch/$download launcher" >&2 + exit 1 + fi + grep -q 'competing RocksDB JNI in LD_PRELOAD' "$TEST_ROOT/$bad/launcher.log" + test ! -e "$TEST_ROOT/$bad/main-preloads" + grep -q '/caller/libtrace.so' "$TEST_ROOT/$bad/version-preloads" + grep -q '/caller/liballoc.so' "$TEST_ROOT/$bad/version-preloads" + done + done + done + + # A caller already using the same allocator keeps its order without a duplicate entry. + name="$arch-already-loaded" + jemalloc="$TEST_ROOT/$name/bin/libjemalloc.so" + [ "$arch" = x86_64 ] || jemalloc="$TEST_ROOT/$name/bin/libjemalloc_aarch64.so" + inherited="/caller/libtrace.so:$jemalloc:/caller/liballoc.so" + run_launcher "$arch" success rocksdb "$inherited" "$name" + test "$(cat "$TEST_ROOT/$name/main-preloads")" = "$inherited" +done + +# Topling is supported only on x86_64; selected JNI precedes allocator and caller libraries. +for download in success failure; do + name="topling-$download" + top="$TEST_ROOT/$name" + inherited=/caller/libtrace.so:/caller/liballoc.so + expected="$inherited" + [ "$download" != success ] || expected="$top/bin/libjemalloc.so:$expected" + run_launcher x86_64 "$download" topling "$inherited" "$name" + test "$(cat "$top/main-preloads")" = "$top/topling/librocksdbjni-linux64.so:$expected" + grep -qx -- '-cp' "$top/main-preloads.args" + grep -qx 'org.springframework.boot.loader.JarLauncher' "$top/main-preloads.args" + name="arm-topling-$download" + if run_launcher aarch64 "$download" topling "$inherited" "$name"; then + echo 'FAIL: Topling was accepted on unsupported ARM' >&2; exit 1 + fi + grep -q 'Topling requires Linux x86_64' "$TEST_ROOT/$name/launcher.log" + test ! -e "$TEST_ROOT/$name/main-preloads" +done +echo 'PASS: complete Store launcher preserves preloads and rejects competing JNI across allocator paths' diff --git a/hugegraph-store/hg-store-grpc/pom.xml b/hugegraph-store/hg-store-grpc/pom.xml index 75f4594a0d..edf96a82a9 100644 --- a/hugegraph-store/hg-store-grpc/pom.xml +++ b/hugegraph-store/hg-store-grpc/pom.xml @@ -32,7 +32,8 @@ 1.6.2 - 1.39.0 + + 1.39.0 3.17.2 0.6.1 @@ -89,7 +90,7 @@ grpc-java - io.grpc:protoc-gen-grpc-java:${grpc.version}:exe:${os.detected.classifier} + io.grpc:protoc-gen-grpc-java:${grpc.codegen.version}:exe:${os.detected.classifier} ${project.basedir}/src/main/proto diff --git a/hugegraph-store/hg-store-node/pom.xml b/hugegraph-store/hg-store-node/pom.xml index a551aeb68e..56cff46dba 100644 --- a/hugegraph-store/hg-store-node/pom.xml +++ b/hugegraph-store/hg-store-node/pom.xml @@ -99,7 +99,7 @@ com.google.protobuf protobuf-java-util - 3.17.2 + ${protobuf.runtime.version} org.apache.hugegraph diff --git a/hugegraph-store/hg-store-node/src/main/java/org/apache/hugegraph/store/node/StoreNodeApplication.java b/hugegraph-store/hg-store-node/src/main/java/org/apache/hugegraph/store/node/StoreNodeApplication.java index c793ed96f2..2f96a34d0c 100644 --- a/hugegraph-store/hg-store-node/src/main/java/org/apache/hugegraph/store/node/StoreNodeApplication.java +++ b/hugegraph-store/hg-store-node/src/main/java/org/apache/hugegraph/store/node/StoreNodeApplication.java @@ -17,8 +17,8 @@ package org.apache.hugegraph.store.node; -import org.apache.hugegraph.store.node.listener.ContextClosedListener; import org.apache.hugegraph.store.node.listener.PdConfigureListener; +import org.apache.hugegraph.util.RocksDBRuntime; import org.springframework.boot.SpringApplication; import org.springframework.boot.autoconfigure.SpringBootApplication; import org.springframework.context.ConfigurableApplicationContext; @@ -31,14 +31,15 @@ @SpringBootApplication public class StoreNodeApplication { - //TODO Is this OK? - private final AppShutdownHook shutdownHook = new AppShutdownHook(Thread.currentThread()); - public static void main(String[] args) { - start(); + start(args); } public static void start() { + start(new String[0]); + } + + public static void start(String... args) { // Set the log location for the slot usage String logPath = System.getProperty("logging.path"); if (StringUtils.isBlank(logPath)) { @@ -55,10 +56,11 @@ public static void start() { } SpringApplication application = new SpringApplication(StoreNodeApplication.class); PdConfigureListener listener = new PdConfigureListener(); - ContextClosedListener closedListener = new ContextClosedListener(); application.addListeners(listener); - application.addListeners(closedListener); - ConfigurableApplicationContext context = application.run(); + // PD environment overrides are applied before context initialization. + application.addInitializers(context -> RocksDBRuntime.verify( + context.getEnvironment().getProperty("rocksdb.provider", "rocksdb"))); + ConfigurableApplicationContext context = application.run(args); listener.setContext(context); System.out.println("StoreNodeApplication started."); } diff --git a/hugegraph-store/hg-store-node/src/main/java/org/apache/hugegraph/store/node/grpc/HgStoreNodeService.java b/hugegraph-store/hg-store-node/src/main/java/org/apache/hugegraph/store/node/grpc/HgStoreNodeService.java index 17cec42e71..93268cd64f 100644 --- a/hugegraph-store/hg-store-node/src/main/java/org/apache/hugegraph/store/node/grpc/HgStoreNodeService.java +++ b/hugegraph-store/hg-store-node/src/main/java/org/apache/hugegraph/store/node/grpc/HgStoreNodeService.java @@ -35,6 +35,7 @@ import org.apache.hugegraph.store.node.AppConfig; import org.apache.hugegraph.store.options.HgStoreEngineOptions; import org.apache.hugegraph.store.options.RaftRocksdbOptions; +import org.apache.hugegraph.util.RocksDBRuntime; import org.apache.hugegraph.store.raft.RaftClosure; import org.apache.hugegraph.store.raft.RaftOperation; import org.apache.hugegraph.store.raft.RaftTaskHandler; @@ -120,6 +121,7 @@ public void init() { setJobConfig(appConfig.getJobOptions()); }}; + RocksDBRuntime.verify(String.valueOf(options.getRocksdbConfig().getOrDefault("rocksdb.provider", "rocksdb"))); RaftRocksdbOptions.initRocksdbGlobalConfig(options.getRocksdbConfig()); options.getLabels().put("rest.port", Integer.toString(appConfig.getRestPort())); diff --git a/hugegraph-store/hg-store-node/src/main/java/org/apache/hugegraph/store/node/listener/ContextClosedListener.java b/hugegraph-store/hg-store-node/src/main/java/org/apache/hugegraph/store/node/listener/ContextClosedListener.java index df8d084e22..285ec3b968 100644 --- a/hugegraph-store/hg-store-node/src/main/java/org/apache/hugegraph/store/node/listener/ContextClosedListener.java +++ b/hugegraph-store/hg-store-node/src/main/java/org/apache/hugegraph/store/node/listener/ContextClosedListener.java @@ -20,7 +20,6 @@ import java.util.concurrent.ScheduledExecutorService; import java.util.concurrent.ThreadPoolExecutor; -import org.apache.hugegraph.store.HgStoreEngine; import org.apache.hugegraph.store.node.grpc.HgStoreStreamImpl; import org.apache.hugegraph.store.node.task.TTLCleaner; import org.springframework.beans.factory.annotation.Autowired; @@ -28,9 +27,6 @@ import org.springframework.context.event.ContextClosedEvent; import org.springframework.stereotype.Service; -import com.alipay.sofa.jraft.Status; -import com.alipay.sofa.jraft.entity.PeerId; - import lombok.extern.slf4j.Slf4j; @Slf4j @@ -45,22 +41,6 @@ public class ContextClosedListener implements ApplicationListener { - try { - Status status = - leader.getRaftNode().transferLeadershipTo(PeerId.ANY_PEER); - log.info("partition {} transfer leader status: {}", - leader.getGroupId(), status); - } catch (Exception e) { - log.info("partition {} transfer leader error: ", - leader.getGroupId(), e); - } - }); - HgStoreEngine.getInstance().getPartitionEngines().forEach( - ((integer, partitionEngine) -> partitionEngine.getRaftNode() - .shutdown()) - ); - } catch (Exception e) { - log.error("transfer leader failed: " + e.getMessage()); - } - } } diff --git a/hugegraph-store/hg-store-test/pom.xml b/hugegraph-store/hg-store-test/pom.xml index cfe8a83cfd..9804435aba 100644 --- a/hugegraph-store/hg-store-test/pom.xml +++ b/hugegraph-store/hg-store-test/pom.xml @@ -130,7 +130,7 @@ com.google.protobuf protobuf-java-util - 3.17.2 + ${protobuf.runtime.version} compile diff --git a/hugegraph-store/hg-store-test/src/main/java/org/apache/hugegraph/store/HeartbeatShutdownTest.java b/hugegraph-store/hg-store-test/src/main/java/org/apache/hugegraph/store/HeartbeatShutdownTest.java new file mode 100644 index 0000000000..d746299873 --- /dev/null +++ b/hugegraph-store/hg-store-test/src/main/java/org/apache/hugegraph/store/HeartbeatShutdownTest.java @@ -0,0 +1,406 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hugegraph.store; + +import java.io.File; +import java.lang.reflect.Constructor; +import java.lang.reflect.Field; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.List; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.Future; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.TimeoutException; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.atomic.AtomicInteger; + +import org.apache.hugegraph.pd.common.PDException; +import org.apache.hugegraph.pd.grpc.Metapb; +import org.apache.hugegraph.pd.grpc.Pdpb.ErrorType; +import org.apache.hugegraph.store.meta.Partition; +import org.apache.hugegraph.store.meta.PartitionManager; +import org.apache.hugegraph.store.meta.Store; +import org.apache.hugegraph.store.meta.StoreMetadata; +import org.apache.hugegraph.store.metric.HgMetricService; +import org.apache.hugegraph.store.options.HgStoreEngineOptions; +import org.apache.hugegraph.store.pd.PdProvider; +import org.apache.hugegraph.store.util.PartitionMetaStoreWrapper; +import org.junit.Assert; +import org.junit.Test; +import org.mockito.Mockito; + +import com.alipay.sofa.jraft.rpc.RpcServer; + +public class HeartbeatShutdownTest { + + @Test + public void testInvalidRecoveredShardExitsAfterNodeDestroy() throws Exception { + assertFatalExit("recovery", 0); + } + + @Test + public void testFatalRegistrationExitsAfterShutdownHooks() throws Exception { + for (int code : new int[]{ErrorType.STORE_ID_NOT_EXIST_VALUE, + ErrorType.STORE_HAS_BEEN_REMOVED_VALUE, + ErrorType.STORE_PROHIBIT_DUPLICATE_VALUE}) { + assertFatalExit("register", code); + } + } + + @Test + public void testFatalHeartbeatExitsAfterShutdownHooks() throws Exception { + for (int code : new int[]{ErrorType.STORE_ID_NOT_EXIST_VALUE, ErrorType.STORE_HAS_BEEN_REMOVED_VALUE}) { + assertFatalExit("heartbeat", code); + } + } + + private static void assertFatalExit(String phase, int code) throws Exception { + Path directory = Files.createTempDirectory(Path.of("target"), "heartbeat-fatal-"); + Path marker = directory.resolve("hook-complete"); + Path output = directory.resolve("child.log"); + String classpath = System.getProperty("surefire.test.class.path", System.getProperty("java.class.path")); + if ("recovery".equals(phase)) { + // Node's packaged artifact stores classes under Spring Boot's BOOT-INF. + Path nodeClasses = Path.of("../hg-store-node/target/classes").toAbsolutePath(); + Assert.assertTrue("Missing Node reactor classes: " + nodeClasses, + Files.isRegularFile(nodeClasses.resolve( + "org/apache/hugegraph/store/node/grpc/HgStoreNodeService.class"))); + classpath += File.pathSeparator + nodeClasses; + } + Process process = new ProcessBuilder(new File(System.getProperty("java.home"), "bin/java").toString(), + "-cp", classpath, HeartbeatShutdownTest.class.getName(), + phase, Integer.toString(code), marker.toString()) + .redirectErrorStream(true).redirectOutput(output.toFile()).start(); + try { + boolean exited = process.waitFor(20, TimeUnit.SECONDS); + if (!exited) { + new ProcessBuilder(new File(System.getProperty("java.home"), "bin/jstack").toString(), + Long.toString(process.pid())) + .redirectErrorStream(true).redirectOutput(directory.resolve("thread-dump.log").toFile()) + .start().waitFor(5, TimeUnit.SECONDS); + } + Assert.assertTrue("Fatal heartbeat did not exit; child log: " + output + "\n" + Files.readString(output), + exited); + Assert.assertEquals(Files.readString(output), "recovery".equals(phase) ? 0 : 255, process.exitValue()); + Assert.assertEquals("joined", Files.readString(marker)); + if ("recovery".equals(phase)) { + String log = Files.readString(output); + Assert.assertTrue(log, log.contains("not in valid shard group")); + Assert.assertTrue(log, log.contains("NODE_DESTROY_COMPLETE")); + } + } finally { + if (process.isAlive()) { + process.destroyForcibly(); + Assert.assertTrue(process.waitFor(5, TimeUnit.SECONDS)); + } + } + } + + public static void main(String[] args) throws Exception { + String phase = args[0]; + int code = Integer.parseInt(args[1]); + Path marker = Path.of(args[2]); + if ("recovery".equals(phase)) { + runInvalidShardRecovery(marker); + return; + } + CountDownLatch producersStarted = new CountDownLatch(1); + PdProvider provider = Mockito.mock(PdProvider.class); + Mockito.when(provider.getClusterStats()).thenReturn(Metapb.ClusterStats.newBuilder() + .setState(Metapb.ClusterState.Cluster_OK).build()); + Mockito.when(provider.registerStore(Mockito.any())).thenAnswer(invocation -> { + producersStarted.await(); + if ("register".equals(phase)) { + System.out.println("FATAL_REGISTER " + code); + throw new PDException(code, "fatal registration fixture"); + } + return 1L; + }); + Mockito.when(provider.storeHeartbeat(Mockito.any())).thenAnswer(invocation -> { + System.out.println("FATAL_HEARTBEAT " + code); + throw new PDException(code, "fatal heartbeat fixture"); + }); + HeartbeatService service = new HeartbeatService(null); + service.setStoreMetadata(Mockito.mock(StoreMetadata.class)); + HgStoreEngineOptions options = Mockito.mock(HgStoreEngineOptions.class); + Mockito.when(options.getPdProvider()).thenReturn(provider); + Mockito.when(options.getGrpcAddress()).thenReturn("127.0.0.1:8500"); + Mockito.when(options.getRaftAddress()).thenReturn("127.0.0.1:8510"); + Mockito.when(options.getStoreHBInterval()).thenReturn(1); + Mockito.when(options.getPartitionHBInterval()).thenReturn(60); + Runtime.getRuntime().addShutdownHook(new Thread(() -> { + System.out.println("SHUTDOWN_HOOK_JOIN"); + service.shutdown(); + try { + Files.writeString(marker, "joined"); + System.out.println("SHUTDOWN_HOOK_COMPLETE"); + } catch (Exception e) { + throw new AssertionError(e); + } + }, "fatal-fixture-shutdown")); + service.init(options); + producersStarted.countDown(); + } + + private static void runInvalidShardRecovery(Path marker) throws Exception { + Path data = marker.getParent().resolve("data"); + Path raft = marker.getParent().resolve("raft"); + Files.createDirectories(data.resolve(HgStoreEngineOptions.DB_Path_Prefix).resolve("00001")); + Files.createDirectories(raft.resolve(HgStoreEngineOptions.Raft_Path_Prefix).resolve("00001/log")); + HgStoreEngineOptions options = new HgStoreEngineOptions(); + options.setDataPath(data.toString()); + options.setRaftPath(raft.toString()); + PdProvider provider = Mockito.mock(PdProvider.class); + Metapb.Partition partition = Metapb.Partition.newBuilder().setId(1).setGraphName("graph").build(); + Mockito.when(provider.getShardGroup(1)).thenReturn(Metapb.ShardGroup.newBuilder() + .setId(1).addShards(Metapb.Shard.newBuilder().setStoreId(2) + .setRole(Metapb.ShardRole.Leader)).build()); + Mockito.when(provider.getPartitionByID("graph", 1)).thenReturn(new Partition(partition)); + Store remote = new Store(); + remote.setRaftAddress("127.0.0.1:8511"); + Mockito.when(provider.getStoreByID(2L)).thenReturn(remote); + PartitionManager manager = new PartitionManager(provider, options); + StoreMetadata metadata = Mockito.mock(StoreMetadata.class); + Store local = new Store(); + local.setId(1); + Mockito.when(metadata.getStore()).thenReturn(local); + Mockito.when(metadata.getPartitionStore(1)).thenReturn(Metapb.PartitionStore.newBuilder() + .setPartitionId(1).setStoreLocation(data.toString()).build()); + Mockito.when(metadata.getPartitionRaft(1)).thenReturn(Metapb.PartitionRaft.newBuilder() + .setPartitionId(1).setRaftLocation(raft.toString()).build()); + setField(PartitionManager.class, manager, "storeMetadata", metadata); + PartitionMetaStoreWrapper wrapper = Mockito.mock(PartitionMetaStoreWrapper.class); + Mockito.when(wrapper.scan(Mockito.eq(1), Mockito.eq(Metapb.Partition.parser()), Mockito.any())) + .thenReturn(List.of(partition)); + setField(PartitionManager.class, manager, "wrapper", wrapper); + Constructor constructor = HgStoreEngine.class.getDeclaredConstructor(); + constructor.setAccessible(true); + HgStoreEngine engine = constructor.newInstance(); + Fixture service = new Fixture(false); + RpcServer rpc = Mockito.mock(RpcServer.class); + HgMetricService metrics = Mockito.mock(HgMetricService.class); + setField(HgStoreEngine.class, engine, "partitionManager", manager); + setField(HgStoreEngine.class, engine, "heartbeatService", service); + setField(HgStoreEngine.class, engine, "rpcServer", rpc); + setField(HgStoreEngine.class, engine, "metricService", metrics); + Class nodeClass = Class.forName("org.apache.hugegraph.store.node.grpc.HgStoreNodeService"); + Class configClass = Class.forName("org.apache.hugegraph.store.node.AppConfig"); + Object node = nodeClass.getConstructor(configClass).newInstance((Object) null); + setField(nodeClass, node, "storeEngine", engine); + service.addStateListener(engine); + Runtime.getRuntime().addShutdownHook(new Thread(() -> { + System.out.println("NODE_DESTROY_ENTERED"); + try { + nodeClass.getMethod("destroy").invoke(node); + Mockito.verify(metrics).shutdown(); + Mockito.verify(rpc).shutdown(); + Files.writeString(marker, "joined"); + System.out.println("NODE_DESTROY_COMPLETE"); + } catch (Exception e) { + throw new AssertionError(e); + } + }, "recovery-fixture-shutdown")); + service.start(); + Assert.assertTrue(service.entered.await(5, TimeUnit.SECONDS)); + service.onStateChanged(Metapb.StoreState.Up); + } + + private static void setField(Class owner, Object object, String name, Object value) throws Exception { + Field field = owner.getDeclaredField(name); + field.setAccessible(true); + field.set(object, value); + } + + @Test + public void testShutdownJoinsBothInFlightProducersBeforeReturning() throws Exception { + Fixture service = new Fixture(true); + ExecutorService closer = Executors.newSingleThreadExecutor(); + try { + service.start(); + Assert.assertTrue(service.entered.await(5, TimeUnit.SECONDS)); + Future shutdown = closer.submit(service::shutdown); + assertStillClosing(shutdown); + service.storeRelease.countDown(); + assertStillClosing(shutdown); + service.partitionRelease.countDown(); + shutdown.get(5, TimeUnit.SECONDS); + Assert.assertFalse(service.storeThread.isAlive()); + Assert.assertFalse(service.partitionThread.isAlive()); + } finally { + service.storeRelease.countDown(); + service.partitionRelease.countDown(); + service.shutdown(); + closer.shutdownNow(); + } + } + + @Test + public void testShutdownWakesBothIdleProducers() throws Exception { + Fixture service = new Fixture(false); + ExecutorService closer = Executors.newSingleThreadExecutor(); + try { + service.start(); + Assert.assertTrue(service.entered.await(5, TimeUnit.SECONDS)); + awaitSleeping(service.storeThread); + awaitSleeping(service.partitionThread); + closer.submit(service::shutdown).get(5, TimeUnit.SECONDS); + Assert.assertFalse(service.storeThread.isAlive()); + Assert.assertFalse(service.partitionThread.isAlive()); + } finally { + service.shutdown(); + closer.shutdownNow(); + } + } + + @Test + public void testShutdownWaitsForDerivedStateCallbackAndRejectsLateUpdates() throws Exception { + Fixture service = new Fixture(false); + ExecutorService closer = Executors.newSingleThreadExecutor(); + CountDownLatch callbackEntered = new CountDownLatch(1); + CountDownLatch callbackRelease = new CountDownLatch(1); + AtomicInteger calls = new AtomicInteger(); + service.addStateListener((store, previous, current) -> { + calls.incrementAndGet(); + callbackEntered.countDown(); + try { + callbackRelease.await(); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new AssertionError(e); + } + }); + try { + service.start(); + Assert.assertTrue(service.entered.await(5, TimeUnit.SECONDS)); + service.onStateChanged(Metapb.StoreState.Up); + Assert.assertTrue(callbackEntered.await(5, TimeUnit.SECONDS)); + Future shutdown = closer.submit(service::shutdown); + assertStillClosing(shutdown); + callbackRelease.countDown(); + shutdown.get(5, TimeUnit.SECONDS); + service.onStateChanged(Metapb.StoreState.Offline); + Assert.assertEquals(1, calls.get()); + Assert.assertEquals(Metapb.StoreState.Up, service.getStoreInfo().getState()); + } finally { + callbackRelease.countDown(); + service.shutdown(); + closer.shutdownNow(); + } + } + + @Test + public void testInterruptedShutdownDoesNotAllowDatabaseClose() throws Exception { + Fixture service = new Fixture(true); + CountDownLatch closing = new CountDownLatch(1); + AtomicBoolean databaseClosed = new AtomicBoolean(); + AtomicBoolean interruptedFailure = new AtomicBoolean(); + Thread closer = new Thread(() -> { + closing.countDown(); + try { + service.shutdown(); + // HgStoreEngine only closes native owners after this call succeeds. + databaseClosed.set(true); + } catch (IllegalStateException expected) { + interruptedFailure.set(Thread.currentThread().isInterrupted()); + } + }); + try { + service.start(); + Assert.assertTrue(service.entered.await(5, TimeUnit.SECONDS)); + closer.start(); + Assert.assertTrue(closing.await(5, TimeUnit.SECONDS)); + closer.interrupt(); + closer.join(5000); + Assert.assertFalse(closer.isAlive()); + Assert.assertTrue(interruptedFailure.get()); + Assert.assertFalse(databaseClosed.get()); + } finally { + service.storeRelease.countDown(); + service.partitionRelease.countDown(); + service.shutdown(); + closer.join(5000); + } + } + + private static void assertStillClosing(Future shutdown) throws Exception { + try { + shutdown.get(100, TimeUnit.MILLISECONDS); + Assert.fail("Shutdown returned while heartbeat still owns work"); + } catch (TimeoutException expected) { + // A successful return would let HgStoreEngine close the DB too soon. + } + } + + private static void awaitSleeping(Thread thread) throws Exception { + long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(5); + while (thread.getState() != Thread.State.TIMED_WAITING && System.nanoTime() < deadline) { + Thread.sleep(5); + } + Assert.assertEquals(Thread.State.TIMED_WAITING, thread.getState()); + } + + private static final class Fixture extends HeartbeatService { + + private final CountDownLatch entered = new CountDownLatch(2); + private final CountDownLatch storeRelease; + private final CountDownLatch partitionRelease; + private volatile Thread storeThread; + private volatile Thread partitionThread; + + private Fixture(boolean blocked) { + super(null); + this.storeRelease = new CountDownLatch(blocked ? 1 : 0); + this.partitionRelease = new CountDownLatch(blocked ? 1 : 0); + } + + private void start() throws Exception { + this.setStoreMetadata(Mockito.mock(StoreMetadata.class)); + HgStoreEngineOptions options = Mockito.mock(HgStoreEngineOptions.class); + Mockito.when(options.getPartitionHBInterval()).thenReturn(60); + Field delay = HeartbeatService.class.getDeclaredField("timerNextDelay"); + delay.setAccessible(true); + delay.setInt(this, 60000); + this.init(options); + } + + @Override + protected void registerStore() { + this.storeThread = Thread.currentThread(); + this.awaitWork(this.storeRelease); + } + + @Override + protected void partitionHeartbeat() { + this.partitionThread = Thread.currentThread(); + this.awaitWork(this.partitionRelease); + } + + private void awaitWork(CountDownLatch release) { + this.entered.countDown(); + try { + release.await(); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new AssertionError(e); + } + } + } +} diff --git a/hugegraph-store/hg-store-test/src/main/java/org/apache/hugegraph/store/core/CoreSuiteTest.java b/hugegraph-store/hg-store-test/src/main/java/org/apache/hugegraph/store/core/CoreSuiteTest.java index 830c348244..c2b3d68e69 100644 --- a/hugegraph-store/hg-store-test/src/main/java/org/apache/hugegraph/store/core/CoreSuiteTest.java +++ b/hugegraph-store/hg-store-test/src/main/java/org/apache/hugegraph/store/core/CoreSuiteTest.java @@ -17,6 +17,7 @@ package org.apache.hugegraph.store.core; +import org.apache.hugegraph.store.HeartbeatShutdownTest; import org.apache.hugegraph.store.StoreIdChangeTest; import org.apache.hugegraph.store.core.snapshot.HgSnapshotHandlerTest; import org.junit.runner.RunWith; @@ -45,6 +46,8 @@ // HgBusinessImplTest.class @RunWith(Suite.class) @Suite.SuiteClasses({ + HeartbeatShutdownTest.class, + MetricSessionTest.class, HgSnapshotHandlerTest.class, StoreIdChangeTest.class }) diff --git a/hugegraph-store/hg-store-test/src/main/java/org/apache/hugegraph/store/core/MetricSessionTest.java b/hugegraph-store/hg-store-test/src/main/java/org/apache/hugegraph/store/core/MetricSessionTest.java new file mode 100644 index 0000000000..8bafb25c43 --- /dev/null +++ b/hugegraph-store/hg-store-test/src/main/java/org/apache/hugegraph/store/core/MetricSessionTest.java @@ -0,0 +1,112 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hugegraph.store.core; + +import java.lang.reflect.Field; +import java.lang.reflect.Method; +import java.nio.file.Files; +import java.util.HashMap; +import java.util.Map; +import java.util.UUID; + +import org.apache.hugegraph.rocksdb.access.RocksDBFactory; +import org.apache.hugegraph.rocksdb.access.RocksDBSession; +import org.apache.hugegraph.store.HgStoreEngine; +import org.apache.hugegraph.store.business.BusinessHandler; +import org.apache.hugegraph.store.metric.SystemMetricService; +import org.apache.hugegraph.store.rocksdb.BaseRocksDbTest; +import org.junit.Assert; +import org.junit.Test; +import org.mockito.Mockito; +import org.rocksdb.MemoryUsageType; +import org.rocksdb.RocksDB; +import org.rocksdb.Statistics; +import org.rocksdb.TickerType; + +public class MetricSessionTest extends BaseRocksDbTest { + + @Test + public void testRepeatedMetricsReturnBorrowedSessions() throws Exception { + assertMetricsReturnSessions(false); + } + + @Test + public void testFailedMetricsReturnBorrowedSessions() throws Exception { + assertMetricsReturnSessions(true); + } + + private static void assertMetricsReturnSessions(boolean failStatistics) throws Exception { + RocksDBFactory factory = RocksDBFactory.getInstance(); + String name = "metric-session-" + UUID.randomUUID(); + RocksDBSession borrowed = factory.createGraphDB(Files.createTempDirectory(name).toString(), name); + Field sessionsField = RocksDBFactory.class.getDeclaredField("dbSessionMap"); + sessionsField.setAccessible(true); + @SuppressWarnings("unchecked") + Map sessions = (Map) sessionsField.get(factory); + RocksDBSession owner = sessions.get(name); + Field statisticsField = RocksDBSession.class.getDeclaredField("rocksDbStats"); + statisticsField.setAccessible(true); + RocksDB nativeOwner = owner.getDB(); + Statistics original = owner.getRocksDbStats(); + Statistics failing = Mockito.mock(Statistics.class); + Mockito.when(failing.getTickerCount(Mockito.any(TickerType.class))) + .thenThrow(new IllegalStateException("statistics fixture")); + try { + if (failStatistics) { + statisticsField.set(owner, failing); + } + HgStoreEngine engine = Mockito.mock(HgStoreEngine.class); + BusinessHandler business = Mockito.mock(BusinessHandler.class); + Mockito.when(engine.getBusinessHandler()).thenReturn(business); + Map memory = new HashMap<>(); + for (MemoryUsageType type : MemoryUsageType.values()) { + memory.put(type, 0L); + } + Mockito.when(business.getApproximateMemoryUsageByType(null)).thenReturn(memory); + SystemMetricService service = new SystemMetricService(); + service.setStoreEngine(engine); + Method collect = SystemMetricService.class.getDeclaredMethod("loadRocksDbInfo", Map.class); + collect.setAccessible(true); + int baseline = owner.getRefCount(); + for (int i = 0; i < 3; i++) { + Map metrics = new HashMap<>(); + collect.invoke(service, metrics); + if (!failStatistics) { + Assert.assertTrue(metrics.containsKey("rocksdb.graph." + name + ".number_keys_written")); + } + Assert.assertEquals("metric collection must return its borrowed session", baseline, + owner.getRefCount()); + } + if (failStatistics) { + Mockito.verify(failing, Mockito.times(3)).getTickerCount(TickerType.NUMBER_KEYS_WRITTEN); + } + statisticsField.set(owner, original); + borrowed.close(); + factory.releaseGraphDB(name); + Assert.assertFalse("the final owner release must close native RocksDB", nativeOwner.isOwningHandle()); + } finally { + statisticsField.set(owner, original); + borrowed.close(); + factory.releaseGraphDB(name); + // Also release the native fixture if the old implementation leaked its borrowed references. + Method shutdown = RocksDBSession.class.getDeclaredMethod("shutdown"); + shutdown.setAccessible(true); + shutdown.invoke(owner); + } + } +} diff --git a/install-dist/release-docs/LICENSE b/install-dist/release-docs/LICENSE index 4008742f82..0e337f4dbf 100644 --- a/install-dist/release-docs/LICENSE +++ b/install-dist/release-docs/LICENSE @@ -258,7 +258,9 @@ Third party Apache 2.0 licenses The following components are provided under the Apache 2.0 License. See project link for details. Per-component LICENSE-[project].txt files are included only when additional non-Apache-2.0 license text is required for a bundled component. -The Netty 4.1.137.Final modules share licenses/LICENSE-netty-4.1.137.Final.txt. +The Netty modules share licenses/LICENSE-netty-4.1.137.Final.txt; tcnative +modules use licenses/LICENSE-netty-tcnative-2.0.81.Final.txt. +The shaded gRPC transport uses licenses/LICENSE-grpc-netty-shaded-1.55.3.txt. Paths referenced by upstream licenses are reproduced as labeled sections in the corresponding per-component license file. @@ -315,16 +317,19 @@ the corresponding per-component license file. https://central.sonatype.com/artifact/com.google.android/annotations/4.1.1.4 -> Apache 2.0 https://central.sonatype.com/artifact/com.google.api.grpc/proto-google-common-protos/1.17.0 -> Apache 2.0 https://central.sonatype.com/artifact/com.google.api.grpc/proto-google-common-protos/2.0.1 -> Apache 2.0 + https://central.sonatype.com/artifact/com.google.api.grpc/proto-google-common-protos/2.9.0 -> Apache 2.0 https://central.sonatype.com/artifact/com.google.auto.service/auto-service-annotations/1.0 -> Apache 2.0 https://central.sonatype.com/artifact/com.google.code.findbugs/jsr305/3.0.1 -> Apache 2.0 https://central.sonatype.com/artifact/com.google.code.findbugs/jsr305/3.0.2 -> Apache 2.0 https://central.sonatype.com/artifact/com.google.code.gson/gson/2.8.6 -> Apache 2.0 https://central.sonatype.com/artifact/com.google.code.gson/gson/2.8.9 -> Apache 2.0 + https://central.sonatype.com/artifact/com.google.code.gson/gson/2.9.0 -> Apache 2.0 https://central.sonatype.com/artifact/com.google.errorprone/error_prone_annotations/2.1.3 -> Apache 2.0 https://central.sonatype.com/artifact/com.google.errorprone/error_prone_annotations/2.10.0 -> Apache 2.0 https://central.sonatype.com/artifact/com.google.errorprone/error_prone_annotations/2.18.0 -> Apache 2.0 https://central.sonatype.com/artifact/com.google.errorprone/error_prone_annotations/2.3.4 -> Apache 2.0 https://central.sonatype.com/artifact/com.google.errorprone/error_prone_annotations/2.4.0 -> Apache 2.0 + https://central.sonatype.com/artifact/com.google.errorprone/error_prone_annotations/2.11.0 -> Apache 2.0 https://central.sonatype.com/artifact/com.google.errorprone/error_prone_annotations/2.48.0 -> Apache 2.0 https://central.sonatype.com/artifact/com.google.guava/failureaccess/1.0.1 -> Apache 2.0 https://central.sonatype.com/artifact/com.google.guava/guava/25.1-jre -> Apache 2.0 @@ -333,6 +338,8 @@ the corresponding per-component license file. https://central.sonatype.com/artifact/com.google.guava/guava/30.1-android -> Apache 2.0 https://central.sonatype.com/artifact/com.google.guava/guava/31.0.1-android -> Apache 2.0 https://central.sonatype.com/artifact/com.google.guava/guava/32.0.1-android -> Apache 2.0 + https://central.sonatype.com/artifact/com.google.guava/guava/31.1-android -> Apache 2.0 + https://central.sonatype.com/artifact/com.google.guava/guava/31.1-jre -> Apache 2.0 https://central.sonatype.com/artifact/com.google.guava/listenablefuture/9999.0-empty-to-avoid-conflict-with-guava -> Apache 2.0 https://central.sonatype.com/artifact/com.google.j2objc/j2objc-annotations/1.1 -> Apache 2.0 https://central.sonatype.com/artifact/com.google.j2objc/j2objc-annotations/1.3 -> Apache 2.0 @@ -401,29 +408,16 @@ the corresponding per-component license file. https://central.sonatype.com/artifact/io.github.hakky54/ayza-for-netty/10.0.0 -> Apache 2.0 https://central.sonatype.com/artifact/io.github.hakky54/ayza/10.0.0 -> Apache 2.0 https://central.sonatype.com/artifact/io.github.lognet/grpc-spring-boot-starter/4.5.5 -> Apache 2.0 - https://central.sonatype.com/artifact/io.grpc/grpc-api/1.28.1 -> Apache 2.0 - https://central.sonatype.com/artifact/io.grpc/grpc-api/1.39.0 -> Apache 2.0 - https://central.sonatype.com/artifact/io.grpc/grpc-api/1.47.0 -> Apache 2.0 - https://central.sonatype.com/artifact/io.grpc/grpc-context/1.28.1 -> Apache 2.0 - https://central.sonatype.com/artifact/io.grpc/grpc-context/1.39.0 -> Apache 2.0 - https://central.sonatype.com/artifact/io.grpc/grpc-context/1.47.0 -> Apache 2.0 - https://central.sonatype.com/artifact/io.grpc/grpc-core/1.28.1 -> Apache 2.0 - https://central.sonatype.com/artifact/io.grpc/grpc-core/1.39.0 -> Apache 2.0 - https://central.sonatype.com/artifact/io.grpc/grpc-core/1.47.0 -> Apache 2.0 - https://central.sonatype.com/artifact/io.grpc/grpc-grpclb/1.39.0 -> Apache 2.0 - https://central.sonatype.com/artifact/io.grpc/grpc-netty-shaded/1.28.0 -> Apache 2.0 - https://central.sonatype.com/artifact/io.grpc/grpc-netty-shaded/1.39.0 -> Apache 2.0 - https://central.sonatype.com/artifact/io.grpc/grpc-netty-shaded/1.47.0 -> Apache 2.0 - https://central.sonatype.com/artifact/io.grpc/grpc-netty/1.39.0 -> Apache 2.0 - https://central.sonatype.com/artifact/io.grpc/grpc-netty/1.47.0 -> Apache 2.0 - https://central.sonatype.com/artifact/io.grpc/grpc-protobuf-lite/1.28.0 -> Apache 2.0 - https://central.sonatype.com/artifact/io.grpc/grpc-protobuf-lite/1.39.0 -> Apache 2.0 - https://central.sonatype.com/artifact/io.grpc/grpc-protobuf/1.28.0 -> Apache 2.0 - https://central.sonatype.com/artifact/io.grpc/grpc-protobuf/1.39.0 -> Apache 2.0 - https://central.sonatype.com/artifact/io.grpc/grpc-services/1.39.0 -> Apache 2.0 - https://central.sonatype.com/artifact/io.grpc/grpc-stub/1.28.0 -> Apache 2.0 - https://central.sonatype.com/artifact/io.grpc/grpc-stub/1.39.0 -> Apache 2.0 - https://central.sonatype.com/artifact/io.grpc/grpc-stub/1.47.0 -> Apache 2.0 + https://central.sonatype.com/artifact/io.grpc/grpc-api/1.55.3 -> Apache 2.0 + https://central.sonatype.com/artifact/io.grpc/grpc-context/1.55.3 -> Apache 2.0 + https://central.sonatype.com/artifact/io.grpc/grpc-core/1.55.3 -> Apache 2.0 + https://central.sonatype.com/artifact/io.grpc/grpc-grpclb/1.55.3 -> Apache 2.0 + https://central.sonatype.com/artifact/io.grpc/grpc-netty/1.55.3 -> Apache 2.0 + https://central.sonatype.com/artifact/io.grpc/grpc-netty-shaded/1.55.3 -> Apache 2.0 + https://central.sonatype.com/artifact/io.grpc/grpc-protobuf/1.55.3 -> Apache 2.0 + https://central.sonatype.com/artifact/io.grpc/grpc-protobuf-lite/1.55.3 -> Apache 2.0 + https://central.sonatype.com/artifact/io.grpc/grpc-services/1.55.3 -> Apache 2.0 + https://central.sonatype.com/artifact/io.grpc/grpc-stub/1.55.3 -> Apache 2.0 https://central.sonatype.com/artifact/io.jsonwebtoken/jjwt-api/0.11.2 -> Apache 2.0 https://central.sonatype.com/artifact/io.jsonwebtoken/jjwt-api/0.11.5 -> Apache 2.0 https://central.sonatype.com/artifact/io.jsonwebtoken/jjwt-impl/0.11.2 -> Apache 2.0 @@ -756,6 +750,8 @@ The text of each license is also included in licenses/LICENSE-[project].txt. https://central.sonatype.com/artifact/org.codehaus.mojo/animal-sniffer-annotations/1.17 -> MIT https://central.sonatype.com/artifact/org.codehaus.mojo/animal-sniffer-annotations/1.18 -> MIT https://central.sonatype.com/artifact/org.codehaus.mojo/animal-sniffer-annotations/1.19 -> MIT + https://central.sonatype.com/artifact/org.codehaus.mojo/animal-sniffer-annotations/1.21 -> MIT + https://central.sonatype.com/artifact/org.mockito/mockito-core/3.3.3 -> MIT https://central.sonatype.com/artifact/org.jruby.jcodings/jcodings/1.0.58 -> MIT https://central.sonatype.com/artifact/org.jruby.joni/joni/2.2.1 -> MIT https://central.sonatype.com/artifact/org.mockito/mockito-core/3.9.0 -> MIT @@ -821,10 +817,12 @@ The following components are provided under the BSD-3-Clause License. See projec The text of each license is also included in licenses/LICENSE-[project].txt. https://central.sonatype.com/artifact/com.google.protobuf/protobuf-java-util/3.17.2 -> BSD-3-Clause + https://central.sonatype.com/artifact/com.google.protobuf/protobuf-java-util/3.22.3 -> BSD-3-Clause https://central.sonatype.com/artifact/com.google.protobuf/protobuf-java/3.11.0 -> BSD-3-Clause https://central.sonatype.com/artifact/com.google.protobuf/protobuf-java/3.17.2 -> BSD-3-Clause https://central.sonatype.com/artifact/com.google.protobuf/protobuf-java/3.21.7 -> BSD-3-Clause https://central.sonatype.com/artifact/com.google.protobuf/protobuf-java/3.5.1 -> BSD-3-Clause + https://central.sonatype.com/artifact/com.google.protobuf/protobuf-java/3.22.3 -> BSD-3-Clause https://central.sonatype.com/artifact/com.sun.activation/jakarta.activation/2.0.0 -> BSD-3-Clause https://central.sonatype.com/artifact/org.abego.treelayout/org.abego.treelayout.core/1.0.3 -> BSD-3-Clause https://central.sonatype.com/artifact/org.antlr/antlr4-runtime/4.9.1 -> BSD-3-Clause diff --git a/install-dist/release-docs/NOTICE b/install-dist/release-docs/NOTICE index 81b0ed929b..911718b90d 100644 --- a/install-dist/release-docs/NOTICE +++ b/install-dist/release-docs/NOTICE @@ -1354,6 +1354,8 @@ Copyright 2015-2026 The Apache Software Foundation. This product includes software developed at The Apache Software Foundation (http://www.apache.org/). +This product includes/uses ANTLR (http://www.antlr2.org/) +developed by Terence Parr 1989-2006 ------------------------------------------------------------------------ Activiti ------------------------------------------------------------------------ @@ -2435,6 +2437,120 @@ decompression for Java., which can be obtained at: The Netty license paths above are reproduced as sections in licenses/LICENSE-netty-4.1.137.Final.txt. + +======================================================================== + +Netty tcnative 2.0.81.Final NOTICE + +======================================================================== + + The Netty Project + ================= + +Please visit the Netty web site for more information: + + * http://netty.io/ + +Copyright 2016 The Netty Project + +The Netty Project licenses this file to you under the Apache License, +version 2.0 (the "License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at: + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +License for the specific language governing permissions and limitations +under the License. + +------------------------------------------------------------------------------- +This product contains a forked and modified version of Tomcat Native + + * LICENSE: + * license/LICENSE.tomcat-native.txt (Apache License 2.0) + * HOMEPAGE: + * http://tomcat.apache.org/native-doc/ + * https://svn.apache.org/repos/asf/tomcat/native/ + +This product contains the Maven wrapper scripts from 'Maven Wrapper', that provides an easy way to ensure a user has everything necessary to run the Maven build. + + * LICENSE: + * license/LICENSE.mvn-wrapper.txt (Apache License 2.0) + * HOMEPAGE: + * https://github.com/takari/maven-wrapper + +This product contains code from boringssl. + + * LICENSE + * license/LICENSE.boringssl.txt (Apache License 2.0) + * HOMEPAGE: + * https://boringssl.googlesource.com/boringssl/ + +The referenced license paths are reproduced in +licenses/LICENSE-netty-tcnative-2.0.81.Final.txt. + +======================================================================== + +grpc-netty-shaded 1.55.3 NOTICE + +======================================================================== + + The Netty Project + ================= + +Please visit the Netty web site for more information: + + * http://netty.io/ + +Copyright 2016 The Netty Project + +The Netty Project licenses this file to you under the Apache License, +version 2.0 (the "License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at: + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +License for the specific language governing permissions and limitations +under the License. + +------------------------------------------------------------------------------- +This product contains a forked and modified version of Tomcat Native + + * LICENSE: + * license/LICENSE.tomcat-native.txt (Apache License 2.0) + * HOMEPAGE: + * http://tomcat.apache.org/native-doc/ + * https://svn.apache.org/repos/asf/tomcat/native/ + +This product contains the Maven wrapper scripts from 'Maven Wrapper', that provides an easy way to ensure a user has everything necessary to run the Maven build. + + * LICENSE: + * license/LICENSE.mvn-wrapper.txt (Apache License 2.0) + * HOMEPAGE: + * https://github.com/takari/maven-wrapper + +This product contains small piece of code to support AIX, taken from netbsd. + + * LICENSE: + * license/LICENSE.aix-netbsd.txt (OpenSSL License) + * HOMEPAGE: + * https://ftp.netbsd.org/pub/NetBSD/NetBSD-current/src/crypto/external/bsd/openssl/dist + + +This product contains code from boringssl. + + * LICENSE (Combination ISC and OpenSSL license) + * license/LICENSE.boringssl.txt (Combination ISC and OpenSSL license) + * HOMEPAGE: + * https://boringssl.googlesource.com/boringssl/ + +The referenced license paths are reproduced in +licenses/LICENSE-grpc-netty-shaded-1.55.3.txt. ======================================================================== spring-boot-jarmode-layertools-2.7.1 NOTICE diff --git a/install-dist/release-docs/licenses/LICENSE-animal-sniffer-annotations-1.21.txt b/install-dist/release-docs/licenses/LICENSE-animal-sniffer-annotations-1.21.txt new file mode 100644 index 0000000000..016a46a9ab --- /dev/null +++ b/install-dist/release-docs/licenses/LICENSE-animal-sniffer-annotations-1.21.txt @@ -0,0 +1,21 @@ +The MIT License + +Copyright (c) 2008 Kohsuke Kawaguchi and codehaus.org. + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. diff --git a/install-dist/release-docs/licenses/LICENSE-grpc-netty-shaded-1.55.3.txt b/install-dist/release-docs/licenses/LICENSE-grpc-netty-shaded-1.55.3.txt new file mode 100644 index 0000000000..3889f0c1e5 --- /dev/null +++ b/install-dist/release-docs/licenses/LICENSE-grpc-netty-shaded-1.55.3.txt @@ -0,0 +1,973 @@ +======================================================================== +LICENSE.txt +======================================================================== + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +======================================================================== +license/LICENSE.aix-netbsd.txt +======================================================================== + + + LICENSE ISSUES + ============== + + The OpenSSL toolkit stays under a double license, i.e. both the conditions of + the OpenSSL License and the original SSLeay license apply to the toolkit. + See below for the actual license texts. + + OpenSSL License + --------------- + +/* ==================================================================== + * Copyright (c) 1998-2019 The OpenSSL Project. All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * + * 3. All advertising materials mentioning features or use of this + * software must display the following acknowledgment: + * "This product includes software developed by the OpenSSL Project + * for use in the OpenSSL Toolkit. (http://www.openssl.org/)" + * + * 4. The names "OpenSSL Toolkit" and "OpenSSL Project" must not be used to + * endorse or promote products derived from this software without + * prior written permission. For written permission, please contact + * openssl-core@openssl.org. + * + * 5. Products derived from this software may not be called "OpenSSL" + * nor may "OpenSSL" appear in their names without prior written + * permission of the OpenSSL Project. + * + * 6. Redistributions of any form whatsoever must retain the following + * acknowledgment: + * "This product includes software developed by the OpenSSL Project + * for use in the OpenSSL Toolkit (http://www.openssl.org/)" + * + * THIS SOFTWARE IS PROVIDED BY THE OpenSSL PROJECT ``AS IS'' AND ANY + * EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR + * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE OpenSSL PROJECT OR + * ITS CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT + * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; + * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, + * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED + * OF THE POSSIBILITY OF SUCH DAMAGE. + * ==================================================================== + * + * This product includes cryptographic software written by Eric Young + * (eay@cryptsoft.com). This product includes software written by Tim + * Hudson (tjh@cryptsoft.com). + * + */ + + Original SSLeay License + ----------------------- + +/* Copyright (C) 1995-1998 Eric Young (eay@cryptsoft.com) + * All rights reserved. + * + * This package is an SSL implementation written + * by Eric Young (eay@cryptsoft.com). + * The implementation was written so as to conform with Netscapes SSL. + * + * This library is free for commercial and non-commercial use as long as + * the following conditions are aheared to. The following conditions + * apply to all code found in this distribution, be it the RC4, RSA, + * lhash, DES, etc., code; not just the SSL code. The SSL documentation + * included with this distribution is covered by the same copyright terms + * except that the holder is Tim Hudson (tjh@cryptsoft.com). + * + * Copyright remains Eric Young's, and as such any Copyright notices in + * the code are not to be removed. + * If this package is used in a product, Eric Young should be given attribution + * as the author of the parts of the library used. + * This can be in the form of a textual message at program startup or + * in documentation (online or textual) provided with the package. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * 3. All advertising materials mentioning features or use of this software + * must display the following acknowledgement: + * "This product includes cryptographic software written by + * Eric Young (eay@cryptsoft.com)" + * The word 'cryptographic' can be left out if the rouines from the library + * being used are not cryptographic related :-). + * 4. If you include any Windows specific code (or a derivative thereof) from + * the apps directory (application code) you must include an acknowledgement: + * "This product includes software written by Tim Hudson (tjh@cryptsoft.com)" + * + * THIS SOFTWARE IS PROVIDED BY ERIC YOUNG ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * The licence and distribution terms for any publically available version or + * derivative of this code cannot be changed. i.e. this code cannot simply be + * copied and put under another distribution licence + * [including the GNU Public Licence.] + */ + + +======================================================================== +license/LICENSE.boringssl.txt +======================================================================== + +BoringSSL is a fork of OpenSSL. As such, large parts of it fall under OpenSSL +licensing. Files that are completely new have a Google copyright and an ISC +license. This license is reproduced at the bottom of this file. +Contributors to BoringSSL are required to follow the CLA rules for Chromium: +https://cla.developers.google.com/clas +Files in third_party/ have their own licenses, as described therein. The MIT +license, for third_party/fiat, which, unlike other third_party directories, is +compiled into non-test libraries, is included below. +The OpenSSL toolkit stays under a dual license, i.e. both the conditions of the +OpenSSL License and the original SSLeay license apply to the toolkit. See below +for the actual license texts. Actually both licenses are BSD-style Open Source +licenses. In case of any license issues related to OpenSSL please contact +openssl-core@openssl.org. +The following are Google-internal bug numbers where explicit permission from +some authors is recorded for use of their work. (This is purely for our own +record keeping.) + 27287199 + 27287880 + 27287883 + OpenSSL License + --------------- +/* ==================================================================== + * Copyright (c) 1998-2011 The OpenSSL Project. All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * + * 3. All advertising materials mentioning features or use of this + * software must display the following acknowledgment: + * "This product includes software developed by the OpenSSL Project + * for use in the OpenSSL Toolkit. (http://www.openssl.org/)" + * + * 4. The names "OpenSSL Toolkit" and "OpenSSL Project" must not be used to + * endorse or promote products derived from this software without + * prior written permission. For written permission, please contact + * openssl-core@openssl.org. + * + * 5. Products derived from this software may not be called "OpenSSL" + * nor may "OpenSSL" appear in their names without prior written + * permission of the OpenSSL Project. + * + * 6. Redistributions of any form whatsoever must retain the following + * acknowledgment: + * "This product includes software developed by the OpenSSL Project + * for use in the OpenSSL Toolkit (http://www.openssl.org/)" + * + * THIS SOFTWARE IS PROVIDED BY THE OpenSSL PROJECT ``AS IS'' AND ANY + * EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR + * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE OpenSSL PROJECT OR + * ITS CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT + * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; + * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, + * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED + * OF THE POSSIBILITY OF SUCH DAMAGE. + * ==================================================================== + * + * This product includes cryptographic software written by Eric Young + * (eay@cryptsoft.com). This product includes software written by Tim + * Hudson (tjh@cryptsoft.com). + * + */ + Original SSLeay License + ----------------------- +/* Copyright (C) 1995-1998 Eric Young (eay@cryptsoft.com) + * All rights reserved. + * + * This package is an SSL implementation written + * by Eric Young (eay@cryptsoft.com). + * The implementation was written so as to conform with Netscapes SSL. + * + * This library is free for commercial and non-commercial use as long as + * the following conditions are aheared to. The following conditions + * apply to all code found in this distribution, be it the RC4, RSA, + * lhash, DES, etc., code; not just the SSL code. The SSL documentation + * included with this distribution is covered by the same copyright terms + * except that the holder is Tim Hudson (tjh@cryptsoft.com). + * + * Copyright remains Eric Young's, and as such any Copyright notices in + * the code are not to be removed. + * If this package is used in a product, Eric Young should be given attribution + * as the author of the parts of the library used. + * This can be in the form of a textual message at program startup or + * in documentation (online or textual) provided with the package. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * 3. All advertising materials mentioning features or use of this software + * must display the following acknowledgement: + * "This product includes cryptographic software written by + * Eric Young (eay@cryptsoft.com)" + * The word 'cryptographic' can be left out if the rouines from the library + * being used are not cryptographic related :-). + * 4. If you include any Windows specific code (or a derivative thereof) from + * the apps directory (application code) you must include an acknowledgement: + * "This product includes software written by Tim Hudson (tjh@cryptsoft.com)" + * + * THIS SOFTWARE IS PROVIDED BY ERIC YOUNG ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * The licence and distribution terms for any publically available version or + * derivative of this code cannot be changed. i.e. this code cannot simply be + * copied and put under another distribution licence + * [including the GNU Public Licence.] + */ +ISC license used for completely new code in BoringSSL: +/* Copyright (c) 2015, Google Inc. + * + * Permission to use, copy, modify, and/or distribute this software for any + * purpose with or without fee is hereby granted, provided that the above + * copyright notice and this permission notice appear in all copies. + * + * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES + * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF + * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY + * SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES + * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION + * OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN + * CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. */ +The code in third_party/fiat carries the MIT license: +Copyright (c) 2015-2016 the fiat-crypto authors (see +https://github.com/mit-plv/fiat-crypto/blob/master/AUTHORS). +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. +Licenses for support code +------------------------- +Parts of the TLS test suite are under the Go license. This code is not included +in BoringSSL (i.e. libcrypto and libssl) when compiled, however, so +distributing code linked against BoringSSL does not trigger this license: +Copyright (c) 2009 The Go Authors. All rights reserved. +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + * Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above +copyright notice, this list of conditions and the following disclaimer +in the documentation and/or other materials provided with the +distribution. + * Neither the name of Google Inc. nor the names of its +contributors may be used to endorse or promote products derived from +this software without specific prior written permission. +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +BoringSSL uses the Chromium test infrastructure to run a continuous build, +trybots etc. The scripts which manage this, and the script for generating build +metadata, are under the Chromium license. Distributing code linked against +BoringSSL does not trigger this license. +Copyright 2015 The Chromium Authors. All rights reserved. +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + * Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above +copyright notice, this list of conditions and the following disclaimer +in the documentation and/or other materials provided with the +distribution. + * Neither the name of Google Inc. nor the names of its +contributors may be used to endorse or promote products derived from +this software without specific prior written permission. +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +======================================================================== +license/LICENSE.mvn-wrapper.txt +======================================================================== + + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +======================================================================== +license/LICENSE.tomcat-native.txt +======================================================================== + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/install-dist/release-docs/licenses/LICENSE-netty-tcnative-2.0.81.Final.txt b/install-dist/release-docs/licenses/LICENSE-netty-tcnative-2.0.81.Final.txt new file mode 100644 index 0000000000..2439050582 --- /dev/null +++ b/install-dist/release-docs/licenses/LICENSE-netty-tcnative-2.0.81.Final.txt @@ -0,0 +1,861 @@ +======================================================================== +LICENSE.txt +======================================================================== + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +======================================================================== +license/LICENSE.boringssl.txt +======================================================================== + + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +Licenses for support code +------------------------- + +Parts of the TLS test suite are under the Go license. This code is not included +in BoringSSL (i.e. libcrypto and libssl) when compiled, however, so +distributing code linked against BoringSSL does not trigger this license: + +Copyright (c) 2009 The Go Authors. All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above +copyright notice, this list of conditions and the following disclaimer +in the documentation and/or other materials provided with the +distribution. + * Neither the name of Google Inc. nor the names of its +contributors may be used to endorse or promote products derived from +this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +======================================================================== +license/LICENSE.mvn-wrapper.txt +======================================================================== + + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +======================================================================== +license/LICENSE.tomcat-native.txt +======================================================================== + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/install-dist/release-docs/licenses/LICENSE-protobuf-java-3.22.3.txt b/install-dist/release-docs/licenses/LICENSE-protobuf-java-3.22.3.txt new file mode 100644 index 0000000000..19b305b000 --- /dev/null +++ b/install-dist/release-docs/licenses/LICENSE-protobuf-java-3.22.3.txt @@ -0,0 +1,32 @@ +Copyright 2008 Google Inc. All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above +copyright notice, this list of conditions and the following disclaimer +in the documentation and/or other materials provided with the +distribution. + * Neither the name of Google Inc. nor the names of its +contributors may be used to endorse or promote products derived from +this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +Code generated by the Protocol Buffer compiler is owned by the owner +of the input file used when generating it. This code is not +standalone and requires a support library to be linked with it. This +support library is itself covered by the above license. diff --git a/install-dist/release-docs/licenses/LICENSE-protobuf-java-util-3.22.3.txt b/install-dist/release-docs/licenses/LICENSE-protobuf-java-util-3.22.3.txt new file mode 100644 index 0000000000..19b305b000 --- /dev/null +++ b/install-dist/release-docs/licenses/LICENSE-protobuf-java-util-3.22.3.txt @@ -0,0 +1,32 @@ +Copyright 2008 Google Inc. All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above +copyright notice, this list of conditions and the following disclaimer +in the documentation and/or other materials provided with the +distribution. + * Neither the name of Google Inc. nor the names of its +contributors may be used to endorse or promote products derived from +this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +Code generated by the Protocol Buffer compiler is owned by the owner +of the input file used when generating it. This code is not +standalone and requires a support library to be linked with it. This +support library is itself covered by the above license. diff --git a/install-dist/scripts/dependency/known-dependencies.txt b/install-dist/scripts/dependency/known-dependencies.txt index dec9bf0e5c..30fb50a89a 100644 --- a/install-dist/scripts/dependency/known-dependencies.txt +++ b/install-dist/scripts/dependency/known-dependencies.txt @@ -5,8 +5,7 @@ accessors-smart-1.2.jar android-json-0.0.20131108.vaadin1.jar animal-sniffer-annotations-1.14.jar animal-sniffer-annotations-1.17.jar -animal-sniffer-annotations-1.18.jar -animal-sniffer-annotations-1.19.jar +animal-sniffer-annotations-1.21.jar annotations-13.0.jar annotations-24.0.1.jar annotations-4.1.1.4.jar @@ -37,7 +36,6 @@ byte-buddy-agent-1.10.20.jar byte-buddy-agent-1.11.6.jar byte-buddy-agent-1.17.6.jar caffeine-2.3.1.jar -checker-compat-qual-2.5.5.jar checker-qual-2.0.0.jar checker-qual-2.5.2.jar checker-qual-3.12.0.jar @@ -85,10 +83,9 @@ eclipse-collections-11.1.0.jar eclipse-collections-api-10.4.0.jar eclipse-collections-api-11.1.0.jar error_prone_annotations-2.1.3.jar -error_prone_annotations-2.10.0.jar +error_prone_annotations-2.11.0.jar error_prone_annotations-2.18.0.jar error_prone_annotations-2.3.4.jar -error_prone_annotations-2.4.0.jar error_prone_annotations-2.48.0.jar exp4j-0.4.8.jar expressions-9.0-9.0.20190305.jar @@ -125,36 +122,24 @@ groovy-jsr223-4.0.25.jar groovy-swing-4.0.25.jar groovy-templates-4.0.25.jar groovy-xml-4.0.25.jar -grpc-api-1.28.1.jar -grpc-api-1.39.0.jar -grpc-api-1.47.0.jar -grpc-context-1.28.1.jar -grpc-context-1.39.0.jar -grpc-context-1.47.0.jar -grpc-core-1.28.1.jar -grpc-core-1.39.0.jar -grpc-core-1.47.0.jar -grpc-grpclb-1.39.0.jar -grpc-netty-1.47.0.jar -grpc-netty-shaded-1.28.0.jar -grpc-netty-shaded-1.39.0.jar -grpc-netty-shaded-1.47.0.jar -grpc-protobuf-1.28.0.jar -grpc-protobuf-1.39.0.jar -grpc-protobuf-lite-1.28.0.jar -grpc-protobuf-lite-1.39.0.jar -grpc-services-1.39.0.jar +grpc-api-1.55.3.jar +grpc-context-1.55.3.jar +grpc-core-1.55.3.jar +grpc-grpclb-1.55.3.jar +grpc-netty-1.55.3.jar +grpc-netty-shaded-1.55.3.jar +grpc-protobuf-1.55.3.jar +grpc-protobuf-lite-1.55.3.jar +grpc-services-1.55.3.jar grpc-spring-boot-starter-4.5.5.jar -grpc-stub-1.28.0.jar -grpc-stub-1.39.0.jar -grpc-stub-1.47.0.jar -gson-2.8.6.jar +grpc-stub-1.55.3.jar gson-2.8.9.jar +gson-2.9.0.jar guava-25.1-jre.jar guava-27.1-jre.jar guava-30.0-jre.jar -guava-30.1-android.jar -guava-31.0.1-android.jar +guava-31.1-android.jar +guava-31.1-jre.jar guava-32.0.1-android.jar hamcrest-2.2.jar hamcrest-3.0.jar @@ -446,18 +431,12 @@ osgi-resource-locator-1.0.3.jar parboiled-core-1.2.0.jar parboiled-scala_2.12-1.2.0.jar parser-9.0-9.0.20190305.jar -perfmark-api-0.19.0.jar -perfmark-api-0.23.0.jar perfmark-api-0.25.0.jar picocli-4.7.6.jar pinyin4j-2.5.0.jar -proto-google-common-protos-1.17.0.jar -proto-google-common-protos-2.0.1.jar -protobuf-java-3.11.0.jar -protobuf-java-3.17.2.jar -protobuf-java-3.21.7.jar -protobuf-java-3.5.1.jar -protobuf-java-util-3.17.2.jar +proto-google-common-protos-2.9.0.jar +protobuf-java-3.22.3.jar +protobuf-java-util-3.22.3.jar protostuff-api-1.6.0.jar protostuff-collectionschema-1.6.0.jar protostuff-core-1.6.0.jar diff --git a/pom.xml b/pom.xml index 32a4d7ef3f..d5c3820371 100644 --- a/pom.xml +++ b/pom.xml @@ -91,6 +91,8 @@ 4.0.25 2.15.2 4.1.137.Final + 1.55.3 + 3.22.3 2.2 2.5.15 5.3.27 @@ -130,6 +132,20 @@ + + io.grpc + grpc-bom + ${grpc.version} + pom + import + + + com.google.protobuf + protobuf-bom + ${protobuf.runtime.version} + pom + import + com.fasterxml.jackson