From 4dcec5abd6c179e45617ae392b43372c29c98b8b Mon Sep 17 00:00:00 2001 From: Bob den Os Date: Tue, 25 Feb 2025 14:10:25 +0100 Subject: [PATCH 1/7] Initial additions to the node databases --- node.js/databases.md | 173 ++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 164 insertions(+), 9 deletions(-) diff --git a/node.js/databases.md b/node.js/databases.md index 0e7bbb9524..529cba66fd 100644 --- a/node.js/databases.md +++ b/node.js/databases.md @@ -12,18 +12,56 @@ status: released [[toc]] -## cds.**DatabaseService** class { #cds-db-service} +## cds.**DatabaseService** class {#cds-db-service } ### class cds.**DatabaseService** extends cds.Service - ### srv.begin () → this {#db-begin } -In case of database services this actually starts the transaction by acquiring a physical connection from the connection pool, and optionally sends a command to the database like `BEGIN TRANSACTION`. +Creates a transaction out of the DatabaseService. Is called by `cds.tx` automatically. + +### srv.commit () → this {#db-commit } + +Commits all the write operations executed on the transaction. Is called by `cds.tx` automatically. + +### srv.rollback () → this {#db-rollback } + +Reverts all the write operations executed on the transaction. Is called by `cds.tx` automatically. + +### srv.factory () → this {#db-factory } + +The factory property provides an object which implements the pool APIs. Allowing the current DatabaseService to re use the physical database connections improving overall performance. + +### srv.acquire () → this {#db-acquire } + +Creates a connection for the current DatabaseService. Is called by [begin](#db-begin) automatically. + +### srv.release () → this {#db-release } + +Releases the connection for the current DatabaseService. Allowing the connection to be used again for future [acquire](#db-acquire) calls. Is Called by [commit](#db-commit) and [rollback](#db-rollback) automatically. + +### srv.destroy () → this {#db-destroy } + +Destroys the connection for the current DatabaseService. Can be called to completely remove the connection from the DatabaseService. Is required to be called when the connection transaction is in an `unknown` state. + +### srv.disconnect () → this {#db-disconnect } + +Disconnects all connections of the current DatabaseService. This is required for database connections that leverage network connections. As the connections are kept alive while inside the `pool`. Which prevents the process from reaching the `idle` state required for the processes to grasefully shutdown. Is called by `cds.on('shutdown')` automatically. + +### srv.infer () → this {#db-infer } + +Uses the current DatabaseService model to `infer` a provided `cds.ql.Query`. It is best to use `cds.infer` instead. (deprecated?) + +### srv.set () → this {#db-set } + +Sets the provided key value pairs as variables for the current connection. The values can be access by using the `session_context('')` function inside any Queries. Is called by [begin](#db-begin) automatically. + +### srv.run () → this {#db-run } + +Runs the provided `cds.ql.Query` using the current DatabaseService. -This method is called automatically by the framework on the first query, so **you never have to call it** in application coding. There are only very rare cases where you'd want to do so, for example to reuse a `tx` object to start subsequent physical transactions after a former `commit` or `rollback`. But this is not considered good practice. ## cds.DatabaseService — Consumption {#databaseservice-consumption } @@ -129,10 +167,6 @@ Even though we provide a default pool configuration, we expect that each applica The main use case of upsert is data replication. [Upsert](../cds/cqn.md#upsert) updates existing entity records from the given data or inserts new ones if they don't exist in the database. -::: warning -Even if an entity doesn't exist in the database:
→ Upsert is **not** equivalent to Insert. -::: - `UPSERT` statements can be created with the [UPSERT](cds-ql#upsert) query API: ```js @@ -142,7 +176,6 @@ UPSERT.into('db.Books') `UPSERT` queries are translated into DB native upsert statements, more specifically they unfold to an [UPSERT SQL statement](https://help.sap.com/docs/HANA_CLOUD_DATABASE/c1d3f60099654ecfb3fe36ac93c121bb/ea8b6773be584203bcd99da76844c5ed.html) on SAP HANA and to an [INSERT ON CONFLICT SQL statement](https://www.sqlite.org/lang_upsert.html) on SQLite. -- The rows to be upserted need to have the same structure, that is, all rows needs to specify the same named values. - The upsert data must contain all key elements of the entity. - If upsert data is incomplete only the given values are updated or inserted, which means the `UPSERT` statement has "PATCH semantics". - `UPSERT` statements don't have a where clause. The key values of the entity that is upserted are extracted from the data. @@ -155,6 +188,128 @@ The following actions are *not* performed on upsert: In contrast to the Java runtime, deep upserts and delta payloads are not yet supported. ::: +## `@cap-js/db-service` + +The `node.js` DatabaseService core class is implemented by the `@cap-js/db-service` module. With more database specific implementation in `@cap-js/hana`, `@cap-js/sqlite` and `@cap-js/postgres`. Which can be used interchangably depening on the underlying database used. + +### Architecture + +The core principle of the `@cap-js` database services is "don't look at the data". As the database services are the foundational service of all CAP applications the performance of these services is especially important. The heaviest work the database service has to do is handling the `data`. + +#### JSON + +In CAP applications all the `data` uses the `JSON` format. It would be nice if the databases could understand the same format. As this would allow the CAP applications to not transform the `data` between different formats. While `SQL` doesn't specify how `data` should be stored by the database implementing the specification. It does provide certain paradigms which require computationally heavy operations. Which has most implementations pick heavily optimized internal `data` formats that allow for improved performance. Over time the `JSON` format has gained wide popularity and has resulted in many modern databases implement the specification. Which allows CAP applications to convey its intentions to the database through these `JSON` APIs. Removing the need to transform the `data` when reading or writing. + +##### Transform {#databaseservice-architecture-transform } + +It is important to understand the special challenges that come with using javascript. As most database protocols use their internal `data` format to communicate with clients. It is required for the javascript client to convert the javascript types into the database native binary format. + +Probably the most simplistic data type for all programming languages and databases will be an `integer`. For javascript this type doesn't actually exist it is a subset of the `Number` type. Therefor when the database driver has to convert the `Number` type into an `integer` it has to do more work then you might expect. To give a real reference the publicly available `hdb` driver will be used. Which has an implementation for the [`int32`](https://github.com/SAP/node-hdb/blob/6f38a473278730c5edce969a87891420ce4baecb/lib/protocol/data/Int32.js#L35), [`int64`](https://github.com/SAP/node-hdb/blob/6f38a473278730c5edce969a87891420ce4baecb/lib/util/bignum.js#L379) and [`int128`](https://github.com/SAP/node-hdb/blob/6f38a473278730c5edce969a87891420ce4baecb/lib/util/bignum.js#L600) types. Here is a breakdown of the amount of objects and operations each type require before they can be send to the database. + +`int32` +- Objects: 1 (Buffer) +- Operations: 1 (Function) + +`int64` +- Objects: 14 (dynamic Numbers) 40 (static Numbers) 1 (String) +- Operations: 90 (operators) 11 (Functions) + +`int128` +- Objects: 22 (dynamic Numbers) 80 (static Numbers) 1 (String) +- Operations: 192 (operators) 18 (Functions) + +As for comparison when these types are used in a compiled language there are no operations required it will be a pointer. With the only exception being when the database and client use a different endianness. In which case one of them has to swap the bytes around. + +##### Read + +When reading `data` from the database the new implementations rely on the database responding in the `JSON` format. Allowing the CAP application to not have to do any postprocessing on the response. It is possible for the ODataService to take the result of the DatabaseService as is. The way this is achieved is by using output converts which are baked into the `SQL` statement. Allowing the database to convert the internal format into the OData specified format of that type. The output converter is a function attached to the element of any entity or query. Which enables protocol adapters to generate queries with protocol specific converters. The output converters are database specific so depending on the database internal `data` structure the output converter might be more or less computationally intensive or be completely omitted when the database is OData compliant. + +Another big benefit that reading in the `JSON` format directly from the database enables. Is the ability to read deep nested data structures. While `SQL` only allows a single scalar value to be selected by sub queries. By converting a multi column / row result into a single `JSON` result it is possible to create database native `expand` queries. Which means that the database can optimize the execution better. Where in the past it was required for the application to send multiple requests for the different levels of `expand` queries or convert flattened to-one `expand` queries. + +When the CAP applications know that there is no post processing required (e.g. no `after` handlers). It can skip the `JSON.parse` and `JSON.stringify` which are required to grand javascript access to the result for manipulations. With the ultimate goal of not having to load the whole result into memory, but instead stream the database result chunks directly into the http response connection. Allowing CAP applications to handle much larger `data` sets while using significantly less memory and cpu. + +##### Write + +Using the `JSON` format for writing operations comes with many cascading improvements. + +When using the standard way of executing `INSERT` and `UPSERT` queries. It is required to match the `SQL` to the data structure. Which goes against the main principle of the new database services. + +A simple example of the impact is as follows: + +```javascript +await INSERT([ + {ID:1}, + {ID:2, name: ''}, + {ID:3, descr: ''} +]) +``` + +Which will actually `prepare` and `execute` the following queries: + +```SQL +INSERT INTO (ID) VALUES (?) +INSERT INTO (ID,name) VALUES (?,?) +INSERT INTO (ID,descr) VALUES (?,?) +``` + +Where the usage of the `JSON` format allows the database services to only `prepare` and `execute` the following query: + +```SQL +INSERT INTO (ID,name,descr,...) AS SELECT ID, name, descr,... FROM JSON_EACH(?) +``` + +Where the placeholder will be provided with the `JSON.stringify` of the provided `entries`. Which means that the query will only be executed once. Reducing the number of network round trips required to do the same amount of work. Greatly improving through put as multiple rows can fit within a single network packet and the transformation to a `JSON` string is much cheaper then the native transformation steps as mentioned in the [transform](#databaseservice-architecture-transform) section. + +By having a single `JSON` placeholder it is possible to stream the dataset through the CAP application. By taking the `req` object which natively is a `Readable` stream and providing it as the `entries` of the `INSERT` statement. + +```javascript +app.post('/upload', async (req,res) => { + try{ + await INSERT(req).into('entity') + res.status(201) + res.end() + } catch (err) { + res.status(400) + res.end(err.message) + } +}) +``` + +As the `JSON` is converted into an intermediate table it is also possible to improve the way that `UPSERT` statements are handled. It has all the same benefits as `INSERT` query have and a bit more. + +```SQL +UPSERT INTO (ID,name,createdAt,modifiedAt,...) AS +SELECT + ID, + new.name ? new.name : old.name, -- only update name when provided + old.ID ? old.createdAt : $now, -- only apply @cds.on.insert when no OLD entry exists + old.ID ? $now : null -- only apply @cds.on.update when an OLD entry exists +FROM JSON_EACH(?) AS NEW +JOIN OLD ON OLD.ID = NEW.ID +``` + +##### Match + +For `@cap-js/hana` specifically there is a `JSON` optimization that assists HANA in re-using execution plans. As HANA has a very advanced execution plan optimizer it is very valuable to be able to re-use the already existing execution plans. One kind of query was always preventing HANA from using the existing execution plans as the `SQL` query would always change based upon the data provided. + +```javascript +const IDs = [{val:1},{val:2},...] +cds.ql`SELECT * FROM ${entity} WHERE ID in ${IDs}` +``` + +Which would create a slightly different variant of the following query based upon then number of `val`s provided: + +```SQL +SELECT * FROM entity WHERE ID IN (?,?,...) +``` + +Where now this query will always prodice the same `SQL` statement. Allowing HANA to use the existing execution plan. + +```SQL +SELECT * FROM entity WHERE ID IN (SELECT VAL FROM JSON_TABLE(?,'$' COLUMNS(VAL DOUBLE PATH '$.val'))) +``` + + ## More to Come This documentation is not complete yet, or the APIs are not released for general availability. From 9abffd4db1f1552d79642dca0d2ee6afded1be8d Mon Sep 17 00:00:00 2001 From: Bob den Os Date: Mon, 5 May 2025 13:15:49 +0800 Subject: [PATCH 2/7] Add unification section --- node.js/databases.md | 37 ++++++++++++++++++++++++++++++++++++- 1 file changed, 36 insertions(+), 1 deletion(-) diff --git a/node.js/databases.md b/node.js/databases.md index 529cba66fd..7d78ad496c 100644 --- a/node.js/databases.md +++ b/node.js/databases.md @@ -196,7 +196,7 @@ The `node.js` DatabaseService core class is implemented by the `@cap-js/db-servi The core principle of the `@cap-js` database services is "don't look at the data". As the database services are the foundational service of all CAP applications the performance of these services is especially important. The heaviest work the database service has to do is handling the `data`. -#### JSON +#### JSON {#databaseservice-json } In CAP applications all the `data` uses the `JSON` format. It would be nice if the databases could understand the same format. As this would allow the CAP applications to not transform the `data` between different formats. While `SQL` doesn't specify how `data` should be stored by the database implementing the specification. It does provide certain paradigms which require computationally heavy operations. Which has most implementations pick heavily optimized internal `data` formats that allow for improved performance. Over time the `JSON` format has gained wide popularity and has resulted in many modern databases implement the specification. Which allows CAP applications to convey its intentions to the database through these `JSON` APIs. Removing the need to transform the `data` when reading or writing. @@ -310,6 +310,41 @@ SELECT * FROM entity WHERE ID IN (SELECT VAL FROM JSON_TABLE(?,'$' COLUMNS(VAL D ``` +#### Unification + +One of the primary features that had to be supported by all new `@cap-js` database services is path expressions. One of the most powerfull features was not being supported by the development focused `SQLite` database service. Which resulted in many applications developing around the missing feature. Allowing the usage of `SQLite`, but remove the possibility of using a very important fundamental feature. When extrapolating this requirement into a more general requirement it becomes unification. + +Therefor the following features are added to the new `@cap-js` database services to greatly improve the behavior of all the database services. + +##### Path expressions + +One of the most complex parts of the `@cap-js` database services. It allows for executing all `cqn` queries natively on all the database services. Providing improved overall performance, query definition simplification, inherent query optimizations. + +###### Expand + +When a `cqn` query contains an `expand` column this column is converted into an `sub select` which has the `on` conditions as its `where` clause referencing directly to the outer query. Allowing the database to create internal relationships between the different data sources and removes the inherent data duplication that comes with using `to-many` join conditions. + +###### Property navigation + +When a `cqn` query contains one or more property access that goes through an path expression these are translated into a `join`. The column name behavior is by default the path expression with the `.` replaced with `_`, but these can be overwritten by providing an `as` alias. + +###### Entity navigation + +When a `cqn` query references an entity through a path expression this is translated into a direct access into the final entity with a `where` clause that uses `exists` clauses to reduce the resulting data to match the restrictions applied by the path expression. This provides an innate benefit over using a `join` between the entities that it won't return duplicate data. + +##### Functions + +One of the features most databases provide, but are not extensively specified by the `SQL` standard are functions. So while there are a few functions which are universally supported by all databases. Most databases provide more functionalities that can be used, but will behave differently between databases or might just have a different function name. Therefor the `@cap-js` database services provide a set of universal functions that are supported by all the database services. The primary focus of the functions is to allow the `OData` protocol to be executed natively on the database. + +There are some functions that are provided by `HANA` which are not supported by other databases, but the primary goal of the application might be to run `HANA` in production. Therefor it is impossible or impractical to use `SQLite` for development. To improve this primary use case the `@cap-js` services are able to leverage the function framework to provide `HANA` function APIs. Not all functions will be available so when your application requires a `HANA` specific function [contributions](https://github.com/cap-js/cds-dbs/pulls) are always welcome. There are already some functions implemented of varied complexity from [`current_timestamp`](https://github.com/cap-js/cds-dbs/blob/7c6b2f5a6837afbeb1e24daef9a49e25cf7e92f0/db-service/lib/cql-functions.js#L186) to [`months_between`](https://github.com/cap-js/cds-dbs/blob/7c6b2f5a6837afbeb1e24daef9a49e25cf7e92f0/sqlite/lib/cql-functions.js#L129C3-L129C17) to [`hierarchy`](https://github.com/cap-js/cds-dbs/blob/7c6b2f5a6837afbeb1e24daef9a49e25cf7e92f0/db-service/lib/cql-functions.js#L200) allowing anyone with `cql` knowledge to implement any missing functions. + +##### Data structure + +A very common issue has been so far that applications where getting different results based upon which database or even which database driver they where using. Causing many un expected side effects when writing custom handlers. One of the primary root causes where `cds.Int64` and `cds.Decimal` which would either return as a `Number` or as a `String`. Which resulted in many application developers to write a custom handler that expected a `Number` to be returned for their productive system to receive a `String` which caused their computational logic to produce the wrong result. As javascript is very lenient it doesn't have a problem doing `1 + 1` or `'1' + 1`, but the produced result will very drastically. + +The way that the unified data structures is achieved is by leveraging the [`JSON`](#databaseservice-json) format as return type. Which results in all the database drivers only every handling arbitrary string contents. Which prevents the drivers from influencing the data structure. + + ## More to Come This documentation is not complete yet, or the APIs are not released for general availability. From 81d21d39d08cc1bb41b97cbaf5feadca02286ce6 Mon Sep 17 00:00:00 2001 From: I543501 Date: Wed, 24 Jun 2026 11:13:15 +0200 Subject: [PATCH 3/7] Document known HANA limitations Adds a "SAP HANA - Known Limitations" section to node.js/databases.md, collected for BLI cap/cdsnode#2725. The section is grouped: - General (driver-independent) - hdb driver - @sap/hana-client driver Covers virtual elements in expressions, BLOB defaults, date/time ranges, draft + LargeBinary handling (with move_media_data_in_db and binary_draft_compat flags), binary filter limitations on hdb, HanaLobStream / for-await on hana-client, and related driver quirks. --- node.js/databases.md | 149 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 149 insertions(+) diff --git a/node.js/databases.md b/node.js/databases.md index 7d78ad496c..406ebf1748 100644 --- a/node.js/databases.md +++ b/node.js/databases.md @@ -349,3 +349,152 @@ The way that the unified data structures is achieved is by leveraging the [`JSON This documentation is not complete yet, or the APIs are not released for general availability. Stay tuned to upcoming releases for further updates. + + +## SAP HANA — Known Limitations {#hana-limitations} + +The following limitations apply when running on SAP HANA. Some apply regardless of which client driver is used, others depend on the chosen driver: [`hdb`](https://www.npmjs.com/package/hdb), [`@sap/hana-client`](https://www.npmjs.com/package/@sap/hana-client), or [`@cap-js/hana`](https://www.npmjs.com/package/@cap-js/hana). + +Please report additional limitations you encounter at [cap-js/cds-dbs](https://github.com/cap-js/cds-dbs/issues). + +### General {#hana-limitations-general} + +These limitations apply to SAP HANA independently of the chosen driver. + +#### Virtual elements in expressions + +Virtual elements (`@cds.virtual`/`virtual`) **must not appear** in `WHERE`, `SELECT`, `ORDER BY`, or any other expression. Such queries are rejected with *"Virtual elements are not allowed in expressions"*. + +The runtime cannot transparently replace virtual conditions with `1 = 1` because it does not have enough information about expression boundaries (`xpr`). + +#### `BLOB` columns cannot have default values + +SAP HANA does not support default values on `BLOB` types. Provide values explicitly on `INSERT`/`UPSERT` instead. + +#### Date/time range and formatting + +- `DATE`, `SECONDDATE`, and `TIMESTAMP` reject years ≤ 0 and years ≥ 10000. +- The upper bound is `9999-12-31 23:59:59` — `24:00:00` is **not** accepted (it is on some other databases). +- Truncated forms are not auto-expanded; date components must be fully specified when a time is included. + +#### `DateTime` / timestamp precision + +When using the [`hdb`](#hana-limitations-hdb) driver, only millisecond precision (3 fractional digits) is supported. See the [driver-specific section](#hana-limitations-hdb) below. + +#### `UNION`-based queries + +Queries with a top-level `UNION` (`SELECT.SET.op === 'union'`) are not supported. + +#### `HIERARCHY`, `HIERARCHY_DESCENDANTS`, and `HIERARCHY_ANCESTORS` + +These HANA functions are currently not exposed via the unified CQL function framework. + +#### `Binary` input restrictions + +Binary values are converted with `HEXTOBIN(...)` and therefore only work when the input is a string literal or a `NEW.` reference. Arbitrary binary expressions cannot be embedded in DML. + +#### `Int64` and `Decimal` are returned as strings + +To prevent precision loss in JavaScript, `cds.Int64` and `cds.Decimal` values are returned as strings. Custom handlers must not assume `Number`. See the [Data structure](#databaseservice-data-structure) section above. + +#### Streaming binary content uses `NCLOB` + +`NVARCHAR` parameters quickly hit size limits when streaming. The runtime therefore interprets the parameter as `NCLOB` so that values up to 2 GB per row can be streamed. + +#### HANA Express (version ≤ 2) + +Older HANA Express versions cannot process large JSON documents (limit between 64 KB and 128 KB). The runtime falls back to inserting entries one at a time on such servers. + +#### `not not` requires HANA ≥ 4 + +Double-negation (`not not `) is not supported on HANA servers older than version 4. + +#### `JSON_TABLE` does not produce `BOOLEAN` + +When working at SQL level: `JSON_TABLE` returns booleans via `NVARCHAR` + `CASE` workaround. + +#### Filtering on structured / JSON-array-valued elements + +Filters that compare against a structured element whose value is a JSON array return no results on HANA. + +#### `groupBy` with same-named fields from different associations + +A `groupBy` that includes elements with identical names from different paths (e.g. `author.ID` and `ID`) does not produce the expected result. See [cap/cdsnode#2366](https://github.tools.sap/cap/cdsnode/issues/2366). + +#### `any()` over unmanaged-association navigation *(on `@cap-js/hana`)* + +OData `$filter` expressions using `any()` that navigate through an **unmanaged** association are currently not supported by the new `@cap-js/hana` driver. The legacy `hdb`-based stack supports this case. + +--- + +### `hdb` driver {#hana-limitations-hdb} + +The following limitations are specific to the [`hdb`](https://www.npmjs.com/package/hdb) driver. + +#### Timestamp precision is limited to 3 fractional digits + +`hdb` does not yet support timestamps with more than 3 fractional digits of second precision (milliseconds, not microseconds). See [cap/cdsnode#2368](https://github.tools.sap/cap/cdsnode/issues/2368). + +#### `$filter` with binary literals fails + +Queries such as `?$filter=binary eq binary'AB12…'` fail with `"Cannot set parameter at row: 1. Wrong input for BINARY type"`. See [cap/cdsnode#2357](https://github.tools.sap/cap/cdsnode/issues/2357). + +#### Large strings may be returned as streams + +`hdb` occasionally returns large `NCLOB`/`NVARCHAR` values as `Readable` streams instead of plain strings. Consumer code that reads such values directly must be prepared to drain a stream. + +#### Drafts: `cds.LargeBinary` is not auto-copied from draft to active + +When activating a draft (`draftActivate`), unchanged LOB columns (`cds.LargeBinary`) are not automatically copied from the draft table to the active table. + +To opt-in to the database-side move (`UPDATE ... FROM SELECT`), enable the feature flag: + +```jsonc +// package.json +"cds": { + "features": { /* … */ }, + "fiori": { "move_media_data_in_db": true } +} +``` + +The legacy flag `cds.features.binary_draft_compat` restores the pre-optimization behavior (every BLOB copied into the draft entity). + +#### Draft + binary streaming combined is not supported + +End-to-end tests for streamed binaries inside drafts are currently excluded on the `hdb` driver. + +--- + +### `@sap/hana-client` driver {#hana-limitations-hana-client} + +The following limitations are specific to the [`@sap/hana-client`](https://www.npmjs.com/package/@sap/hana-client) driver. + +#### `communicationTimeout` must be set explicitly + +Without an explicit `communicationTimeout` (in milliseconds), queries can hang for over ten minutes. The runtime sets `60000` by default — applications can override via pool/connection configuration. + +#### `LOB`s cannot be streamed deferred + +The `ResultSet` API of `@sap/hana-client` only allows `getData()` on the current row and is forward-only. As a consequence, all LOB content of a result row is buffered in memory before processing continues — true deferred streaming is not possible with this driver. + +#### Result-set size ceiling (~4 GB) + +A consequence of the above limitation: result sets are capped at approximately `0xFFFFFFFB` bytes (~4 GB). + +#### `HanaLobStream` cannot be consumed with `for await` + +The `HanaLobStream` returned by `@sap/hana-client` hangs when iterated with `for await ... of`. The runtime detects this case (via `result[key].constructor.name === 'HanaLobStream'`) and uses a `PassThrough`-based reader instead. + +#### UPDATE with streams does not report affected rows + +When an `UPDATE` includes streamed parameters (LOBs), `@sap/hana-client` returns `0` even if rows were modified. The runtime treats `streams.length && changes === 0` as success. As a side effect, ETag/affected-row checks during streaming UPDATEs are unreliable. + +#### APM decorators (e.g. Dynatrace) cannot wrap internal methods + +`@sap/hana-client` does not allow wrapping/decorating `client.prepare()` and other internals. APM integrations that rely on method wrapping have reduced visibility on this driver. + +--- + +### Reporting additional limitations + +If you hit a HANA-specific issue not listed here, please open an issue at [cap-js/cds-dbs](https://github.com/cap-js/cds-dbs/issues) (for new-stack issues) or at the [internal `cdsnode` tracker](https://github.tools.sap/cap/cdsnode/issues) (for legacy-stack issues). Driver name (`hdb`, `@sap/hana-client`, or `@cap-js/hana`), HANA version, and a minimal reproduction help triage. From 70ad3e7fbf2a52ba166a5779f96d65d5cb2f3126 Mon Sep 17 00:00:00 2001 From: I543501 Date: Wed, 24 Jun 2026 15:52:24 +0200 Subject: [PATCH 4/7] Reclassify HANA limitations by driver scope - Move timestamp precision from hdb-only to General, since the runtime workaround applies to every HANA driver (originally reported on hdb, BLI #2368). - Move "any() over unmanaged-association navigation" out of General into a new @cap-js/hana driver subsection (hdb supports it). - Clarify that the UNION limitation applies to all databases. --- node.js/databases.md | 24 ++++++++++++++---------- 1 file changed, 14 insertions(+), 10 deletions(-) diff --git a/node.js/databases.md b/node.js/databases.md index 406ebf1748..5a28a5e939 100644 --- a/node.js/databases.md +++ b/node.js/databases.md @@ -379,11 +379,13 @@ SAP HANA does not support default values on `BLOB` types. Provide values explici #### `DateTime` / timestamp precision -When using the [`hdb`](#hana-limitations-hdb) driver, only millisecond precision (3 fractional digits) is supported. See the [driver-specific section](#hana-limitations-hdb) below. +Timestamps with more than 3 fractional digits of second precision (i.e. sub-millisecond / microsecond precision) are not reliably supported on SAP HANA. Values are truncated to milliseconds. + +This was originally reported for the [`hdb`](#hana-limitations-hdb) driver — see [cap/cdsnode#2368](https://github.tools.sap/cap/cdsnode/issues/2368) — but the runtime currently applies the same truncation for every HANA driver until higher precision is confirmed. #### `UNION`-based queries -Queries with a top-level `UNION` (`SELECT.SET.op === 'union'`) are not supported. +Queries with a top-level `UNION` (`SELECT.SET.op === 'union'`) are not supported by the database service layer — this applies to **all** databases, not only HANA, and is listed here for completeness. #### `HIERARCHY`, `HIERARCHY_DESCENDANTS`, and `HIERARCHY_ANCESTORS` @@ -421,20 +423,12 @@ Filters that compare against a structured element whose value is a JSON array re A `groupBy` that includes elements with identical names from different paths (e.g. `author.ID` and `ID`) does not produce the expected result. See [cap/cdsnode#2366](https://github.tools.sap/cap/cdsnode/issues/2366). -#### `any()` over unmanaged-association navigation *(on `@cap-js/hana`)* - -OData `$filter` expressions using `any()` that navigate through an **unmanaged** association are currently not supported by the new `@cap-js/hana` driver. The legacy `hdb`-based stack supports this case. - --- ### `hdb` driver {#hana-limitations-hdb} The following limitations are specific to the [`hdb`](https://www.npmjs.com/package/hdb) driver. -#### Timestamp precision is limited to 3 fractional digits - -`hdb` does not yet support timestamps with more than 3 fractional digits of second precision (milliseconds, not microseconds). See [cap/cdsnode#2368](https://github.tools.sap/cap/cdsnode/issues/2368). - #### `$filter` with binary literals fails Queries such as `?$filter=binary eq binary'AB12…'` fail with `"Cannot set parameter at row: 1. Wrong input for BINARY type"`. See [cap/cdsnode#2357](https://github.tools.sap/cap/cdsnode/issues/2357). @@ -495,6 +489,16 @@ When an `UPDATE` includes streamed parameters (LOBs), `@sap/hana-client` returns --- +### `@cap-js/hana` driver {#hana-limitations-cap-js-hana} + +The following limitations are specific to the new [`@cap-js/hana`](https://www.npmjs.com/package/@cap-js/hana) driver. + +#### `any()` over unmanaged-association navigation + +OData `$filter` expressions using `any()` that navigate through an **unmanaged** association are currently not supported. The legacy `hdb`-based stack supports this case. + +--- + ### Reporting additional limitations If you hit a HANA-specific issue not listed here, please open an issue at [cap-js/cds-dbs](https://github.com/cap-js/cds-dbs/issues) (for new-stack issues) or at the [internal `cdsnode` tracker](https://github.tools.sap/cap/cdsnode/issues) (for legacy-stack issues). Driver name (`hdb`, `@sap/hana-client`, or `@cap-js/hana`), HANA version, and a minimal reproduction help triage. From adc5155eb7c3838967c293f6fbd1037068555d82 Mon Sep 17 00:00:00 2001 From: I543501 Date: Wed, 24 Jun 2026 16:02:14 +0200 Subject: [PATCH 5/7] Move timestamp precision back to hdb section The source comment (cds/tests/.../timestamp.test.js:26) explicitly names hdb. The runtime line `timestamp.slice(0, 23) + 'Z'` is a test-side simplification, not evidence the limitation applies to other HANA drivers. Per the classification rule "comment names a driver -> not General", this belongs under hdb. --- node.js/databases.md | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/node.js/databases.md b/node.js/databases.md index 5a28a5e939..8087c42489 100644 --- a/node.js/databases.md +++ b/node.js/databases.md @@ -377,12 +377,6 @@ SAP HANA does not support default values on `BLOB` types. Provide values explici - The upper bound is `9999-12-31 23:59:59` — `24:00:00` is **not** accepted (it is on some other databases). - Truncated forms are not auto-expanded; date components must be fully specified when a time is included. -#### `DateTime` / timestamp precision - -Timestamps with more than 3 fractional digits of second precision (i.e. sub-millisecond / microsecond precision) are not reliably supported on SAP HANA. Values are truncated to milliseconds. - -This was originally reported for the [`hdb`](#hana-limitations-hdb) driver — see [cap/cdsnode#2368](https://github.tools.sap/cap/cdsnode/issues/2368) — but the runtime currently applies the same truncation for every HANA driver until higher precision is confirmed. - #### `UNION`-based queries Queries with a top-level `UNION` (`SELECT.SET.op === 'union'`) are not supported by the database service layer — this applies to **all** databases, not only HANA, and is listed here for completeness. @@ -429,6 +423,10 @@ A `groupBy` that includes elements with identical names from different paths (e. The following limitations are specific to the [`hdb`](https://www.npmjs.com/package/hdb) driver. +#### Timestamp precision limited to 3 fractional digits + +`hdb` does not yet support timestamps with more than 3 fractional digits of second precision (milliseconds, not microseconds). See [cap/cdsnode#2368](https://github.tools.sap/cap/cdsnode/issues/2368). + #### `$filter` with binary literals fails Queries such as `?$filter=binary eq binary'AB12…'` fail with `"Cannot set parameter at row: 1. Wrong input for BINARY type"`. See [cap/cdsnode#2357](https://github.tools.sap/cap/cdsnode/issues/2357). From 9d1a33adefa88c7cf58655c8d9e77f51452277fa Mon Sep 17 00:00:00 2001 From: I543501 Date: Thu, 25 Jun 2026 15:21:32 +0200 Subject: [PATCH 6/7] Update databases.md --- node.js/databases.md | 36 ++++++++++-------------------------- 1 file changed, 10 insertions(+), 26 deletions(-) diff --git a/node.js/databases.md b/node.js/databases.md index 8087c42489..a11f5631d1 100644 --- a/node.js/databases.md +++ b/node.js/databases.md @@ -338,19 +338,12 @@ One of the features most databases provide, but are not extensively specified by There are some functions that are provided by `HANA` which are not supported by other databases, but the primary goal of the application might be to run `HANA` in production. Therefor it is impossible or impractical to use `SQLite` for development. To improve this primary use case the `@cap-js` services are able to leverage the function framework to provide `HANA` function APIs. Not all functions will be available so when your application requires a `HANA` specific function [contributions](https://github.com/cap-js/cds-dbs/pulls) are always welcome. There are already some functions implemented of varied complexity from [`current_timestamp`](https://github.com/cap-js/cds-dbs/blob/7c6b2f5a6837afbeb1e24daef9a49e25cf7e92f0/db-service/lib/cql-functions.js#L186) to [`months_between`](https://github.com/cap-js/cds-dbs/blob/7c6b2f5a6837afbeb1e24daef9a49e25cf7e92f0/sqlite/lib/cql-functions.js#L129C3-L129C17) to [`hierarchy`](https://github.com/cap-js/cds-dbs/blob/7c6b2f5a6837afbeb1e24daef9a49e25cf7e92f0/db-service/lib/cql-functions.js#L200) allowing anyone with `cql` knowledge to implement any missing functions. -##### Data structure +##### Data structure {#databaseservice-data-structure } A very common issue has been so far that applications where getting different results based upon which database or even which database driver they where using. Causing many un expected side effects when writing custom handlers. One of the primary root causes where `cds.Int64` and `cds.Decimal` which would either return as a `Number` or as a `String`. Which resulted in many application developers to write a custom handler that expected a `Number` to be returned for their productive system to receive a `String` which caused their computational logic to produce the wrong result. As javascript is very lenient it doesn't have a problem doing `1 + 1` or `'1' + 1`, but the produced result will very drastically. The way that the unified data structures is achieved is by leveraging the [`JSON`](#databaseservice-json) format as return type. Which results in all the database drivers only every handling arbitrary string contents. Which prevents the drivers from influencing the data structure. - -## More to Come - -This documentation is not complete yet, or the APIs are not released for general availability. -Stay tuned to upcoming releases for further updates. - - ## SAP HANA — Known Limitations {#hana-limitations} The following limitations apply when running on SAP HANA. Some apply regardless of which client driver is used, others depend on the chosen driver: [`hdb`](https://www.npmjs.com/package/hdb), [`@sap/hana-client`](https://www.npmjs.com/package/@sap/hana-client), or [`@cap-js/hana`](https://www.npmjs.com/package/@cap-js/hana). @@ -374,17 +367,13 @@ SAP HANA does not support default values on `BLOB` types. Provide values explici #### Date/time range and formatting - `DATE`, `SECONDDATE`, and `TIMESTAMP` reject years ≤ 0 and years ≥ 10000. -- The upper bound is `9999-12-31 23:59:59` — `24:00:00` is **not** accepted (it is on some other databases). -- Truncated forms are not auto-expanded; date components must be fully specified when a time is included. +- The upper bound is `9999-12-31 23:59:59` — `24:00:00` is **not** accepted. +- Truncated forms are not auto-expanded. Date components must be fully specified when a time is included. #### `UNION`-based queries Queries with a top-level `UNION` (`SELECT.SET.op === 'union'`) are not supported by the database service layer — this applies to **all** databases, not only HANA, and is listed here for completeness. -#### `HIERARCHY`, `HIERARCHY_DESCENDANTS`, and `HIERARCHY_ANCESTORS` - -These HANA functions are currently not exposed via the unified CQL function framework. - #### `Binary` input restrictions Binary values are converted with `HEXTOBIN(...)` and therefore only work when the input is a string literal or a `NEW.` reference. Arbitrary binary expressions cannot be embedded in DML. @@ -415,7 +404,7 @@ Filters that compare against a structured element whose value is a JSON array re #### `groupBy` with same-named fields from different associations -A `groupBy` that includes elements with identical names from different paths (e.g. `author.ID` and `ID`) does not produce the expected result. See [cap/cdsnode#2366](https://github.tools.sap/cap/cdsnode/issues/2366). +A `groupBy` that includes elements with identical names from different paths (e.g. `author.ID` and `ID`) does not produce the expected result. --- @@ -425,11 +414,11 @@ The following limitations are specific to the [`hdb`](https://www.npmjs.com/pack #### Timestamp precision limited to 3 fractional digits -`hdb` does not yet support timestamps with more than 3 fractional digits of second precision (milliseconds, not microseconds). See [cap/cdsnode#2368](https://github.tools.sap/cap/cdsnode/issues/2368). +`hdb` does not yet support timestamps with more than 3 fractional digits of second precision (milliseconds, not microseconds). #### `$filter` with binary literals fails -Queries such as `?$filter=binary eq binary'AB12…'` fail with `"Cannot set parameter at row: 1. Wrong input for BINARY type"`. See [cap/cdsnode#2357](https://github.tools.sap/cap/cdsnode/issues/2357). +Queries such as `?$filter=binary eq binary'AB12…'` fail with `"Cannot set parameter at row: 1. Wrong input for BINARY type"`. #### Large strings may be returned as streams @@ -485,18 +474,13 @@ When an `UPDATE` includes streamed parameters (LOBs), `@sap/hana-client` returns `@sap/hana-client` does not allow wrapping/decorating `client.prepare()` and other internals. APM integrations that rely on method wrapping have reduced visibility on this driver. ---- - -### `@cap-js/hana` driver {#hana-limitations-cap-js-hana} - -The following limitations are specific to the new [`@cap-js/hana`](https://www.npmjs.com/package/@cap-js/hana) driver. - #### `any()` over unmanaged-association navigation -OData `$filter` expressions using `any()` that navigate through an **unmanaged** association are currently not supported. The legacy `hdb`-based stack supports this case. +OData `$filter` expressions using `any()` that navigate through an **unmanaged** association are currently not supported. --- -### Reporting additional limitations +## More to Come -If you hit a HANA-specific issue not listed here, please open an issue at [cap-js/cds-dbs](https://github.com/cap-js/cds-dbs/issues) (for new-stack issues) or at the [internal `cdsnode` tracker](https://github.tools.sap/cap/cdsnode/issues) (for legacy-stack issues). Driver name (`hdb`, `@sap/hana-client`, or `@cap-js/hana`), HANA version, and a minimal reproduction help triage. +This documentation is not complete yet, or the APIs are not released for general availability. +Stay tuned to upcoming releases for further updates. \ No newline at end of file From b5fc866f8c5020d51c52b758c7c6c7392487f271 Mon Sep 17 00:00:00 2001 From: I543501 Date: Tue, 30 Jun 2026 13:43:52 +0200 Subject: [PATCH 7/7] feedback --- node.js/databases.md | 136 ------------------------------------------- 1 file changed, 136 deletions(-) diff --git a/node.js/databases.md b/node.js/databases.md index 6c273227d3..eed76c6646 100644 --- a/node.js/databases.md +++ b/node.js/databases.md @@ -325,142 +325,6 @@ A very common issue has been so far that applications where getting different re The way that the unified data structures is achieved is by leveraging the [`JSON`](#databaseservice-json) format as return type. Which results in all the database drivers only every handling arbitrary string contents. Which prevents the drivers from influencing the data structure. -## SAP HANA — Known Limitations {#hana-limitations} - -The following limitations apply when running on SAP HANA. Some apply regardless of which client driver is used, others depend on the chosen driver: [`hdb`](https://www.npmjs.com/package/hdb), [`@sap/hana-client`](https://www.npmjs.com/package/@sap/hana-client), or [`@cap-js/hana`](https://www.npmjs.com/package/@cap-js/hana). - -Please report additional limitations you encounter at [cap-js/cds-dbs](https://github.com/cap-js/cds-dbs/issues). - -### General {#hana-limitations-general} - -These limitations apply to SAP HANA independently of the chosen driver. - -#### Virtual elements in expressions - -Virtual elements (`@cds.virtual`/`virtual`) **must not appear** in `WHERE`, `SELECT`, `ORDER BY`, or any other expression. Such queries are rejected with *"Virtual elements are not allowed in expressions"*. - -The runtime cannot transparently replace virtual conditions with `1 = 1` because it does not have enough information about expression boundaries (`xpr`). - -#### `BLOB` columns cannot have default values - -SAP HANA does not support default values on `BLOB` types. Provide values explicitly on `INSERT`/`UPSERT` instead. - -#### Date/time range and formatting - -- `DATE`, `SECONDDATE`, and `TIMESTAMP` reject years ≤ 0 and years ≥ 10000. -- The upper bound is `9999-12-31 23:59:59` — `24:00:00` is **not** accepted. -- Truncated forms are not auto-expanded. Date components must be fully specified when a time is included. - -#### `UNION`-based queries - -Queries with a top-level `UNION` (`SELECT.SET.op === 'union'`) are not supported by the database service layer — this applies to **all** databases, not only HANA, and is listed here for completeness. - -#### `Binary` input restrictions - -Binary values are converted with `HEXTOBIN(...)` and therefore only work when the input is a string literal or a `NEW.` reference. Arbitrary binary expressions cannot be embedded in DML. - -#### `Int64` and `Decimal` are returned as strings - -To prevent precision loss in JavaScript, `cds.Int64` and `cds.Decimal` values are returned as strings. Custom handlers must not assume `Number`. See the [Data structure](#databaseservice-data-structure) section above. - -#### Streaming binary content uses `NCLOB` - -`NVARCHAR` parameters quickly hit size limits when streaming. The runtime therefore interprets the parameter as `NCLOB` so that values up to 2 GB per row can be streamed. - -#### HANA Express (version ≤ 2) - -Older HANA Express versions cannot process large JSON documents (limit between 64 KB and 128 KB). The runtime falls back to inserting entries one at a time on such servers. - -#### `not not` requires HANA ≥ 4 - -Double-negation (`not not `) is not supported on HANA servers older than version 4. - -#### `JSON_TABLE` does not produce `BOOLEAN` - -When working at SQL level: `JSON_TABLE` returns booleans via `NVARCHAR` + `CASE` workaround. - -#### Filtering on structured / JSON-array-valued elements - -Filters that compare against a structured element whose value is a JSON array return no results on HANA. - -#### `groupBy` with same-named fields from different associations - -A `groupBy` that includes elements with identical names from different paths (e.g. `author.ID` and `ID`) does not produce the expected result. - ---- - -### `hdb` driver {#hana-limitations-hdb} - -The following limitations are specific to the [`hdb`](https://www.npmjs.com/package/hdb) driver. - -#### Timestamp precision limited to 3 fractional digits - -`hdb` does not yet support timestamps with more than 3 fractional digits of second precision (milliseconds, not microseconds). - -#### `$filter` with binary literals fails - -Queries such as `?$filter=binary eq binary'AB12…'` fail with `"Cannot set parameter at row: 1. Wrong input for BINARY type"`. - -#### Large strings may be returned as streams - -`hdb` occasionally returns large `NCLOB`/`NVARCHAR` values as `Readable` streams instead of plain strings. Consumer code that reads such values directly must be prepared to drain a stream. - -#### Drafts: `cds.LargeBinary` is not auto-copied from draft to active - -When activating a draft (`draftActivate`), unchanged LOB columns (`cds.LargeBinary`) are not automatically copied from the draft table to the active table. - -To opt-in to the database-side move (`UPDATE ... FROM SELECT`), enable the feature flag: - -```jsonc -// package.json -"cds": { - "features": { /* … */ }, - "fiori": { "move_media_data_in_db": true } -} -``` - -The legacy flag `cds.features.binary_draft_compat` restores the pre-optimization behavior (every BLOB copied into the draft entity). - -#### Draft + binary streaming combined is not supported - -End-to-end tests for streamed binaries inside drafts are currently excluded on the `hdb` driver. - ---- - -### `@sap/hana-client` driver {#hana-limitations-hana-client} - -The following limitations are specific to the [`@sap/hana-client`](https://www.npmjs.com/package/@sap/hana-client) driver. - -#### `communicationTimeout` must be set explicitly - -Without an explicit `communicationTimeout` (in milliseconds), queries can hang for over ten minutes. The runtime sets `60000` by default — applications can override via pool/connection configuration. - -#### `LOB`s cannot be streamed deferred - -The `ResultSet` API of `@sap/hana-client` only allows `getData()` on the current row and is forward-only. As a consequence, all LOB content of a result row is buffered in memory before processing continues — true deferred streaming is not possible with this driver. - -#### Result-set size ceiling (~4 GB) - -A consequence of the above limitation: result sets are capped at approximately `0xFFFFFFFB` bytes (~4 GB). - -#### `HanaLobStream` cannot be consumed with `for await` - -The `HanaLobStream` returned by `@sap/hana-client` hangs when iterated with `for await ... of`. The runtime detects this case (via `result[key].constructor.name === 'HanaLobStream'`) and uses a `PassThrough`-based reader instead. - -#### UPDATE with streams does not report affected rows - -When an `UPDATE` includes streamed parameters (LOBs), `@sap/hana-client` returns `0` even if rows were modified. The runtime treats `streams.length && changes === 0` as success. As a side effect, ETag/affected-row checks during streaming UPDATEs are unreliable. - -#### APM decorators (e.g. Dynatrace) cannot wrap internal methods - -`@sap/hana-client` does not allow wrapping/decorating `client.prepare()` and other internals. APM integrations that rely on method wrapping have reduced visibility on this driver. - -#### `any()` over unmanaged-association navigation - -OData `$filter` expressions using `any()` that navigate through an **unmanaged** association are currently not supported. - ---- - ## More to Come This documentation is not complete yet, or the APIs are not released for general availability.