diff --git a/website/docs/apis/cpp/error-handling.md b/website/docs/apis/cpp/error-handling.md index 7447a264c7..864ae08c95 100644 --- a/website/docs/apis/cpp/error-handling.md +++ b/website/docs/apis/cpp/error-handling.md @@ -100,7 +100,7 @@ See `fluss::ErrorCode` in `fluss.hpp` for the full list of named constants. ## Retry Logic -Some errors are transient, where the server may be temporarily unavailable, mid-election, or under load. `IsRetriable()` can be used for deciding to to retry an operation rather than treating the error as permanent. +Some errors are transient, where the server may be temporarily unavailable, mid-election, or under load. `IsRetriable()` can be used for deciding to retry an operation rather than treating the error as permanent. `ErrorCode::IsRetriable(int32_t code)` is a static helper available directly on the error code: diff --git a/website/docs/concepts/architecture.md b/website/docs/concepts/architecture.md index 74f616fa77..f5ee5a8266 100644 --- a/website/docs/concepts/architecture.md +++ b/website/docs/concepts/architecture.md @@ -52,7 +52,7 @@ In upcoming releases, **ZooKeeper will be replaced** by **KvStore** for metadata - **Hierarchical Storage for LogStores:** By offloading LogStore data, it reduces storage costs and accelerates scaling operations. - **Persistent Storage for KvStores:** It ensures durable storage for KvStore data and collaborates with LogStore to enable fault recovery. -Additionally, **Remote Storage** allows clients to perform bulk read operations on Log and Kv data, enhancing data analysis efficiency and reduce the overhead on Fluss servers. In the future, it will also support bulk write operations, optimizing data import workflows for greater scalability and performance. +Additionally, **Remote Storage** allows clients to perform bulk read operations on Log and Kv data, enhancing data analysis efficiency and reducing the overhead on Fluss servers. In the future, it will also support bulk write operations, optimizing data import workflows for greater scalability and performance. ## Client Fluss clients/SDKs support streaming reads/writes, batch reads/writes, DDL and point queries. Currently, the main implementation of client is Flink Connector. Users can use Flink SQL to easily operate Fluss tables and data. \ No newline at end of file diff --git a/website/docs/engine-flink/delta-joins.md b/website/docs/engine-flink/delta-joins.md index 23690de45f..843e9ffbe9 100644 --- a/website/docs/engine-flink/delta-joins.md +++ b/website/docs/engine-flink/delta-joins.md @@ -17,7 +17,7 @@ Starting with **Apache Fluss 0.8**, streaming join jobs running on **Flink 2.1 o ## How Delta Join Works -Traditional streaming joins in Flink require maintaining both input sides entirely in state to match records across streams. Delta join, by contrast, uses a **index-key lookup mechanism** to transform the behavior of querying data from the state into querying data from the Fluss source table, thereby avoiding redundant storage of the same data in both the Fluss source table and the state. This drastically reduces state size and improves performance for many streaming analytics and enrichment workloads. +Traditional streaming joins in Flink require maintaining both input sides entirely in state to match records across streams. Delta join, by contrast, uses an **index-key lookup mechanism** to transform the behavior of querying data from the state into querying data from the Fluss source table, thereby avoiding redundant storage of the same data in both the Fluss source table and the state. This drastically reduces state size and improves performance for many streaming analytics and enrichment workloads. ![](../assets/delta_join.png) @@ -49,7 +49,7 @@ CREATE TABLE `fluss_catalog`.`my_db`.`left_src` ( ) WITH ( -- prefix key 'bucket.key' = 'city_id', - -- in Flink 2.1, delta join only support append-only source + -- in Flink 2.1, delta join only supports append-only source 'table.merge-engine' = 'first_row' ); ``` @@ -61,7 +61,7 @@ CREATE TABLE `fluss_catalog`.`my_db`.`right_src` ( `city_name` VARCHAR NOT NULL, PRIMARY KEY (city_id) NOT ENFORCED ) WITH ( - -- in Flink 2.1, delta join only support append-only source + -- in Flink 2.1, delta join only supports append-only source 'table.merge-engine' = 'first_row' ); ``` @@ -120,7 +120,7 @@ Sink(table=[fluss_catalog.my_db.snk], fields=[city_id, order_id, content, city_n Delta Join relies on performing lookups by the join key (e.g., the `city_id` in the above example) on the Fluss source tables. This requires that the Fluss source tables are defined with appropriate index on the join key to enable efficient lookups. -Currently, Fluss only supports to defines a single index key per table, which is also referred to as the **prefix key**. The general secondary index which allows define multiple indexes with arbitrary fields is planned to be supported in the near future releases. +Currently, Fluss only supports defining a single index key per table, which is also referred to as the **prefix key**. The general secondary index which allows defining multiple indexes with arbitrary fields is planned to be supported in the near future releases. A prefix key defines the portion of a table’s primary key that can be used for efficient key-based lookups or index pruning. @@ -157,7 +157,7 @@ There is a known issue ([FLINK-38399](https://issues.apache.org/jira/browse/FLIN #### Limitations - The primary key or the prefix key of the tables must be included as part of the equivalence conditions in the join. -- The join must be a INNER join. +- The join must be an INNER join. - The downstream node of the join must support idempotent updates, typically it's an upsert sink and should not have a `SinkUpsertMaterializer` node before it. - Flink planner automatically inserts a `SinkUpsertMaterializer` when the sink’s primary key does not fully cover the upstream update key. - You can learn more details about `SinkUpsertMaterializer` by reading this [blog](https://www.ververica.com/blog/flink-sql-secrets-mastering-the-art-of-changelog-events). diff --git a/website/docs/engine-flink/lookups.md b/website/docs/engine-flink/lookups.md index dd7532808f..a423625d09 100644 --- a/website/docs/engine-flink/lookups.md +++ b/website/docs/engine-flink/lookups.md @@ -135,8 +135,8 @@ For more details about Fluss partitioned table, see [Partitioned Tables](table-d ### Instructions -- Use a primary key table as a dimension table, and the join condition must a prefix subset of the primary keys of the dimension table. -- The bucket key of Fluss dimension table need to set as the join key when creating Fluss table. +- Use a primary key table as a dimension table, and the join condition must be a prefix subset of the primary keys of the dimension table. +- The bucket key of Fluss dimension table needs to be set as the join key when creating Fluss table. - Fluss prefix lookup join is in asynchronous mode by default for higher throughput. You can change the mode of prefix lookup join as synchronous mode by setting the SQL Hint `'lookup.async' = 'false'`. diff --git a/website/docs/engine-flink/options.md b/website/docs/engine-flink/options.md index 329ea94f13..f99ea35702 100644 --- a/website/docs/engine-flink/options.md +++ b/website/docs/engine-flink/options.md @@ -35,7 +35,7 @@ You can dynamically apply the options via SQL hints. The dynamically applied opt They will only be used for the current query and will not affect other queries on the table. The dynamically applied options have a higher priority than the options stored in the metadata of the table. :::note -The [Storage Options](#storage-options) doesn't supported to be dynamically configured via SQL hints, because the storage behavior is determined when the table is created. +The [Storage Options](#storage-options) cannot be dynamically configured via SQL hints, because the storage behavior is determined when the table is created. ::: For example, the following SQL statements temporarily disables the CRC check for the log reading and ignores deletes for writing: @@ -65,7 +65,7 @@ See more details about [ALTER TABLE ... SET](engine-flink/ddl.md#set-properties) |-----------------------------------------|----------|-------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | auto-increment.fields | String | (None) | Defines the auto increment columns. The auto increment column can only be used in primary-key table. With an auto increment column in the table, whenever a new row is inserted into the table, the new row will be assigned with the next available value from the auto-increment sequence. The data type of the auto increment column must be INT or BIGINT. Currently a table can have only one auto-increment column. Adding an auto increment column to an existing table is not supported. | | bucket.num | int | The bucket number of Fluss cluster. | The number of buckets of a Fluss table. | -| bucket.key | String | (None) | Specific the distribution policy of the Fluss table. Data will be distributed to each bucket according to the hash value of bucket-key (It must be a subset of the primary keys excluding partition keys of the primary key table). If you specify multiple fields, delimiter is `,`. If the table has a primary key and a bucket key is not specified, the bucket key will be used as primary key(excluding the partition key). If the table has no primary key and the bucket key is not specified, the data will be distributed to each bucket randomly. | +| bucket.key | String | (None) | Specific the distribution policy of the Fluss table. Data will be distributed to each bucket according to the hash value of bucket-key (It must be a subset of the primary keys excluding partition keys of the primary key table). If you specify multiple fields, delimiter is `,`. If the table has a primary key and a bucket key is not specified, the primary key will be used as bucket key (excluding the partition key). If the table has no primary key and the bucket key is not specified, the data will be distributed to each bucket randomly. | | table.log.ttl | Duration | 7 days | The time to live for log segments. The configuration controls the maximum time we will retain a log before we will delete old segments to free up space. If set to -1, the log will not be deleted. | | table.auto-partition.enabled | Boolean | false | Whether enable auto partition for the table. Disable by default. When auto partition is enabled, the partitions of the table will be created automatically. | | table.auto-partition.key | String | (None) | This configuration defines the time-based partition key to be used for auto-partitioning when a table is partitioned with multiple keys. Auto-partitioning utilizes a time-based partition key to handle partitions automatically, including creating new ones and removing outdated ones, by comparing the time value of the partition with the current system time. In the case of a table using multiple partition keys (such as a composite partitioning strategy), this feature determines which key should serve as the primary time dimension for making auto-partitioning decisions. And If the table has only one partition key, this config is not necessary. Otherwise, it must be specified. | @@ -81,15 +81,15 @@ See more details about [ALTER TABLE ... SET](engine-flink/ddl.md#set-properties) | table.kv.format | Enum | COMPACTED | The format of the kv records in kv store. The default value is `COMPACTED`. The supported formats are `COMPACTED` and `INDEXED`. | | table.kv.format-version | Integer | (None) | The version of the kv format. Automatically set by the coordinator during table creation if not configured by users.

**Note:** The datalake encoding and bucketing strategy mentioned below only takes effect when `datalake.format` is configured at cluster level.

**Version Behaviors:**

(1) **Version 1**: Tables created before `table.kv.format-version` was introduced are treated as version 1. Uses datalake's encoder (e.g., Paimon/Iceberg) for both primary key and bucket key encoding. This may not support prefix lookup properly because some datalake encoders (like Paimon) don't guarantee that encoded bucket key bytes are a prefix of encoded primary key bytes.

(2) **Version 2** (current): New tables use Fluss's default encoder for primary key encoding when bucket key differs from primary key, which ensures proper prefix lookup support. When bucket key equals primary key (default bucket key), it still uses datalake's encoder for optimization. Bucket key encoding always uses datalake's encoder to align with datalake bucket calculation. | | table.kv.standby-replica.enabled | Boolean | (None) | Whether to enable standby replicas for primary key tables. Standby replicas maintain recent KV snapshots for fast leader promotion. Automatically set to `true` by the coordinator during table creation for new PK tables. Tables created before this option was introduced are treated as disabled. Can be dynamically enabled via `ALTER TABLE SET ('table.kv.standby-replica.enabled' = 'true')`. | -| table.log.tiered.local-segments | Integer | 2 | The number of log segments to retain in local for each table when log tiered storage is enabled. It must be greater that 0. The default is 2. | -| table.datalake.enabled | Boolean | false | Whether enable lakehouse storage for the table. Disabled by default. When this option is set to ture and the datalake tiering service is up, the table will be tiered and compacted into datalake format stored on lakehouse storage. | +| table.log.tiered.local-segments | Integer | 2 | The number of log segments to retain in local for each table when log tiered storage is enabled. It must be greater than 0. The default is 2. | +| table.datalake.enabled | Boolean | false | Whether enable lakehouse storage for the table. Disabled by default. When this option is set to true and the datalake tiering service is up, the table will be tiered and compacted into datalake format stored on lakehouse storage. | | table.datalake.format | Enum | (None) | The data lake format of the table specifies the tiered Lakehouse storage format. Currently, supported formats are `paimon`, `iceberg`, and `lance`. In the future, more kinds of data lake format will be supported, such as DeltaLake or Hudi. Once the `table.datalake.format` property is configured, Fluss adopts the key encoding and bucketing strategy used by the corresponding data lake format. This ensures consistency in key encoding and bucketing, enabling seamless **Union Read** functionality across Fluss and Lakehouse. The `table.datalake.format` can be pre-defined before enabling `table.datalake.enabled`. This allows the data lake feature to be dynamically enabled on the table without requiring table recreation. If `table.datalake.format` is not explicitly set during table creation, the table will default to the format specified by the `datalake.format` configuration in the Fluss cluster. | | table.datalake.freshness | Duration | 3min | It defines the maximum amount of time that the datalake table's content should lag behind updates to the Fluss table. Based on this target freshness, the Fluss service automatically moves data from the Fluss table and updates to the datalake table, so that the data in the datalake table is kept up to date within this target. If the data does not need to be as fresh, you can specify a longer target freshness time to reduce costs. | | table.datalake.auto-compaction | Boolean | false | If true, compaction will be triggered automatically when tiering service writes to the datalake. It is disabled by default. | | table.datalake.auto-expire-snapshot | Boolean | false | If true, snapshot expiration will be triggered automatically when tiering service commits to the datalake. It is disabled by default. | -| table.merge-engine | Enum | (None) | Defines the merge engine for the primary key table. By default, primary key table uses the [default merge engine(last_row)](table-design/merge-engines/default.md). It also supports two merge engines are `first_row`, `versioned` and `aggregation`. The [first_row merge engine](table-design/merge-engines/first-row.md) will keep the first row of the same primary key. The [versioned merge engine](table-design/merge-engines/versioned.md) will keep the row with the largest version of the same primary key. The `aggregation` merge engine will aggregate rows with the same primary key using field-level aggregate functions. | +| table.merge-engine | Enum | (None) | Defines the merge engine for the primary key table. By default, primary key table uses the [default merge engine(last_row)](table-design/merge-engines/default.md). It also supports three other merge engines: `first_row`, `versioned` and `aggregation`. The [first_row merge engine](table-design/merge-engines/first-row.md) will keep the first row of the same primary key. The [versioned merge engine](table-design/merge-engines/versioned.md) will keep the row with the largest version of the same primary key. The `aggregation` merge engine will aggregate rows with the same primary key using field-level aggregate functions. | | table.merge-engine.versioned.ver-column | String | (None) | The column name of the version column for the `versioned` merge engine. If the merge engine is set to `versioned`, the version column must be set. | -| table.delete.behavior | Enum | ALLOW | Controls the behavior of delete operations on primary key tables. Three modes are supported: `ALLOW` (default for default merge engine) - allows normal delete operations; `IGNORE` - silently ignores delete requests without errors; `DISABLE` - rejects delete requests and throws explicit errors. This configuration provides system-level guarantees for some downstream pipelines (e.g., Flink Delta Join) that must not receive any delete events in the changelog of the table. For tables with `first_row` or `versioned` or `aggregation` merge engines, this option is automatically set to `IGNORE` and cannot be overridden. Note: For `aggregation` merge engine, when set to `allow`, delete operations will remove the entire record. This configuration only applicable to primary key tables. | +| table.delete.behavior | Enum | ALLOW | Controls the behavior of delete operations on primary key tables. Three modes are supported: `ALLOW` (default for default merge engine) - allows normal delete operations; `IGNORE` - silently ignores delete requests without errors; `DISABLE` - rejects delete requests and throws explicit errors. This configuration provides system-level guarantees for some downstream pipelines (e.g., Flink Delta Join) that must not receive any delete events in the changelog of the table. For tables with `first_row` or `versioned` or `aggregation` merge engines, this option is automatically set to `IGNORE` and cannot be overridden. Note: For `aggregation` merge engine, when set to `allow`, delete operations will remove the entire record. This configuration is only applicable to primary key tables. | | table.changelog.image | Enum | FULL | Defines the changelog image mode for primary key tables. This configuration is inspired by similar settings in database systems like MySQL's `binlog_row_image` and PostgreSQL's `replica identity`. Two modes are supported: `FULL` (default) - produces both UPDATE_BEFORE and UPDATE_AFTER records for update operations, capturing complete information about updates and allowing tracking of previous values; `WAL` - does not produce UPDATE_BEFORE records. Only INSERT, UPDATE_AFTER (and DELETE if allowed) records are emitted. When WAL mode is enabled, the default merge engine is used (no merge engine configured), updates are full row updates (not partial update), and there is no auto-increment column, an optimization is applied to skip looking up old values, and in this case INSERT operations are converted to UPDATE_AFTER events. This mode reduces storage and transmission costs but loses the ability to track previous values. Only applicable to primary key tables. | | table.auto-inc.batch-size | Long | 100000L | The batch size of auto-increment IDs fetched from the distributed counter each time. This value determines the length of the locally cached ID segment. Default: 100000. A larger batch size may cause significant auto-increment ID gaps, especially when unused cached ID segments are discarded due to TabletServer restarts or abnormal terminations. Conversely, a smaller batch size increases the frequency of ID fetch requests to the distributed counter, introducing extra network overhead and reducing write throughput and performance. | | table.statistics.columns | String | (None) | Configures column-level statistics collection for the table. By default this option is not set and no column statistics are collected. The value `*` means collect statistics for all supported columns. A comma-separated list of column names means collect statistics only for the specified columns (recommended for minimal overhead). Supported types: BOOLEAN, TINYINT, SMALLINT, INTEGER, BIGINT, FLOAT, DOUBLE, STRING, CHAR, DECIMAL, DATE, TIME, TIMESTAMP, TIMESTAMP_LTZ. Unsupported types (BYTES, BINARY, ARRAY, MAP, ROW) are automatically excluded. Example: `'col1,col2'` to collect statistics only for columns used in filter conditions. Note: enabling column statistics requires the V1 batch format. Downstream consumers must be upgraded to Fluss v1.0+ before enabling this option. | @@ -105,8 +105,8 @@ See more details about [ALTER TABLE ... SET](engine-flink/ddl.md#set-properties) | scan.kv.snapshot.lease.duration | Duration | 1day | The time period how long to wait before expiring the kv snapshot lease to avoid kv snapshot blocking to delete. | | client.scanner.log.check-crc | Boolean | true | Automatically check the CRC3 of the read records for LogScanner. This ensures no on-the-wire or on-disk corruption to the messages occurred. This check adds some overhead, so it may be disabled in cases seeking extreme performance. | | client.scanner.log.max-poll-records | Integer | 500 | The maximum number of records returned in a single call to poll() for LogScanner. Note that this config doesn't impact the underlying fetching behavior. The Scanner will cache the records from each fetch request and returns them incrementally from each poll. | -| client.scanner.log.fetch.max-bytes | MemorySize | 16mb | The maximum amount of data the server should return for a fetch request from client. Records are fetched in batches, and if the first record batch in the first non-empty bucket of the fetch is larger than this value, the record batch will still be returned to ensure that the fetch can make progress. As such, this is not a absolute maximum. | -| client.scanner.log.fetch.max-bytes-for-bucket | MemorySize | 1mb | The maximum amount of data the server should return for a table bucket in fetch request fom client. Records are fetched in batches, and the max bytes size is config by this option. | +| client.scanner.log.fetch.max-bytes | MemorySize | 16mb | The maximum amount of data the server should return for a fetch request from client. Records are fetched in batches, and if the first record batch in the first non-empty bucket of the fetch is larger than this value, the record batch will still be returned to ensure that the fetch can make progress. As such, this is not an absolute maximum. | +| client.scanner.log.fetch.max-bytes-for-bucket | MemorySize | 1mb | The maximum amount of data the server should return for a table bucket in fetch request from client. Records are fetched in batches, and the max bytes size is configured by this option. | | client.scanner.log.fetch.min-bytes | MemorySize | 1b | The minimum bytes expected for each fetch log request from client to response. If not enough bytes, wait up to client.scanner.log.fetch-wait-max-time time to return. | | client.scanner.log.fetch.wait-max-time | Duration | 500ms | The maximum time to wait for enough bytes to be available for a fetch log request from client to response. | | client.scanner.io.tmpdir | String | System.getProperty("java.io.tmpdir") + "/fluss" | Local directory that is used by client for storing the data files (like kv snapshot, log segment files) to read temporarily | @@ -145,12 +145,12 @@ See more details about [ALTER TABLE ... SET](engine-flink/ddl.md#set-properties) | client.writer.batch-size | MemorySize | 2mb | The writer or walBuilder will attempt to batch records together into one batch for the same bucket. This helps performance on both the client and the server. | | client.writer.dynamic-batch-size.enabled | Boolean | true | Controls whether the client writer dynamically adjusts the batch size based on actual write throughput. Enabled by default. With dynamic batch sizing enabled, the writer adapts memory allocation per batch according to historical write sizes for the target table or partition. This ensures better memory utilization and performance under varying throughput conditions. The dynamic batch size is bounded: it will not exceed `client.writer.batch-size`, nor fall below `client.writer.buffer.page-size`. When disabled, the writer uses a fixed batch size (`client.writer.batch-size`) for all batches, this may lead to frequent memory waits and suboptimal write performance if the incoming data rate is inconsistent across partitions. | | client.writer.buffer.wait-timeout | Duration | 2^(63)-1ns | Defines how long the writer will block when waiting for segments to become available. | -| client.writer.batch-timeout | Duration | 100ms | The writer groups ay rows that arrive in between request sends into a single batched request. Normally this occurs only under load when rows arrive faster than they can be sent out. However in some circumstances the writer may want to reduce the number of requests even under moderate load. This setting accomplishes this by adding a small amount of artificial delay, that is, rather than immediately sending out a row, the writer will wait for up to the given delay to allow other records to be sent so that the sends can be batched together. This can be thought of as analogous to Nagle's algorithm in TCP. This setting gives the upper bound on the delay for batching: once we get client.writer.batch-size worth of rows for a bucket it will be sent immediately regardless of this setting, however if we have fewer than this many bytes accumulated for this bucket we will delay for the specified time waiting for more records to show up. | +| client.writer.batch-timeout | Duration | 100ms | The writer groups any rows that arrive in between request sends into a single batched request. Normally this occurs only under load when rows arrive faster than they can be sent out. However in some circumstances the writer may want to reduce the number of requests even under moderate load. This setting accomplishes this by adding a small amount of artificial delay, that is, rather than immediately sending out a row, the writer will wait for up to the given delay to allow other records to be sent so that the sends can be batched together. This can be thought of as analogous to Nagle's algorithm in TCP. This setting gives the upper bound on the delay for batching: once we get client.writer.batch-size worth of rows for a bucket it will be sent immediately regardless of this setting, however if we have fewer than this many bytes accumulated for this bucket we will delay for the specified time waiting for more records to show up. | | client.writer.bucket.no-key-assigner | Enum | STICKY | The bucket assigner for no key table. For table with bucket key or primary key, we choose a bucket based on a hash of the key. For these table without bucket key and primary key, we can use this option to specify bucket assigner, the candidate assigner is ROUND_ROBIN, STICKY, the default assigner is STICKY.
ROUND_ROBIN: this strategy will assign the bucket id for the input row by round robin.
STICKY: this strategy will assign new bucket id only if the batch changed in record accumulator, otherwise the bucket id will be the same as the front record. | -| client.writer.acks | String | all | The number of acknowledgments the writer requires the leader to have received before considering a request complete. This controls the durability of records that are sent. The following settings are allowed:
acks=0: If set to 0, then the writer will not wait for any acknowledgment from the server at all. No guarantee can be mode that the server has received the record in this case.
acks=1: This will mean the leader will write the record to its local log but will respond without awaiting full acknowledge the record but before the followers have replicated it then the record will be lost.
acks=-1 (all): This will mean the leader will wait for the full ser of in-sync replicas to acknowledge the record. This guarantees that the record will not be lost as long as at least one in-sync replica remains alive, This is the strongest available guarantee. | +| client.writer.acks | String | all | The number of acknowledgments the writer requires the leader to have received before considering a request complete. This controls the durability of records that are sent. The following settings are allowed:
acks=0: If set to 0, then the writer will not wait for any acknowledgment from the server at all. No guarantee can be made that the server has received the record in this case.
acks=1: This will mean the leader will write the record to its local log but will respond without awaiting full acknowledge the record but before the followers have replicated it then the record will be lost.
acks=-1 (all): This will mean the leader will wait for the full set of in-sync replicas to acknowledge the record. This guarantees that the record will not be lost as long as at least one in-sync replica remains alive, This is the strongest available guarantee. | | client.writer.request-max-size | MemorySize | 10mb | The maximum size of a request in bytes. This setting will limit the number of record batches the writer will send in a single request to avoid sending huge requests. Note that this retry is no different than if the writer resent the row upon receiving the error. | | client.writer.retries | Integer | Integer.MAX_VALUE | Setting a value greater than zero will cause the client to resend any record whose send fails with a potentially transient error. | -| client.writer.enable-idempotence | Boolean | true | Writer idempotence is enabled by default if no conflicting config are set. If conflicting config are set and writer idempotence is not explicitly enabled, idempotence is disabled. If idempotence is explicitly enabled and conflicting config are set, a ConfigException is thrown | +| client.writer.enable-idempotence | Boolean | true | Writer idempotence is enabled by default if no conflicting configs are set. If conflicting configs are set and writer idempotence is not explicitly enabled, idempotence is disabled. If idempotence is explicitly enabled and conflicting configs are set, a ConfigException is thrown | | client.writer.max-inflight-requests-per-bucket | Integer | 5 | The maximum number of unacknowledged requests per bucket for writer. This configuration can work only if `client.writer.enable-idempotence` is set to true. When the number of inflight requests per bucket exceeds this setting, the writer will wait for the inflight requests to complete before sending out new requests. | | client.writer.dynamic-create-partition.enabled | Boolean | true | Whether to enable dynamic partition creation for the client writer. When enabled, new partitions are automatically created if they don't already exist during data writes. | diff --git a/website/docs/engine-flink/procedures.md b/website/docs/engine-flink/procedures.md index 24baa9c08e..1fee3d1f64 100644 --- a/website/docs/engine-flink/procedures.md +++ b/website/docs/engine-flink/procedures.md @@ -567,7 +567,7 @@ Fluss provides procedures to manage KV snapshot leases, allowing you to drop lea ### drop_kv_snapshot_lease -Drop KV snapshots leased under a specified leaseId. This is typically used for handle the scenario of lease +Drop KV snapshots leased under a specified leaseId. This is typically used to handle the scenario of lease remnants. After an abnormal job termination (e.g., crash or forced cancellation), the registered lease may not be released automatically and could require manual cleanup. diff --git a/website/docs/install-deploy/deploying-local-cluster.md b/website/docs/install-deploy/deploying-local-cluster.md index 1277d33531..0334c4d9f0 100644 --- a/website/docs/install-deploy/deploying-local-cluster.md +++ b/website/docs/install-deploy/deploying-local-cluster.md @@ -49,7 +49,7 @@ After that, the Fluss local cluster is started. ## Interacting with Fluss -After Fluss local cluster is started, you can use **Fluss Client** (Currently, only support Flink SQL Client) to interact with Fluss. +After Fluss local cluster is started, you can use **Fluss Client** (Currently, only Flink SQL Client is supported) to interact with Fluss. The following subsections will show you how to use Flink SQL Client to interact with Fluss. ### Flink SQL Client diff --git a/website/docs/install-deploy/deploying-with-helm.md b/website/docs/install-deploy/deploying-with-helm.md index 949ff92b18..142740a5be 100644 --- a/website/docs/install-deploy/deploying-with-helm.md +++ b/website/docs/install-deploy/deploying-with-helm.md @@ -171,7 +171,7 @@ The following table lists the configurable parameters of the Fluss chart, and th | Parameter | Description | Default | |-----------|-------------|---------| | `image.registry` | Container image registry | `""` | -| `image.repository` | Container image repository | `fluss` | +| `image.repository` | Container image repository | `apache/fluss` | | `image.tag` | Container image tag | `$FLUSS_VERSION$` | | `image.pullPolicy` | Container image pull policy | `IfNotPresent` | | `image.pullSecrets` | Container image pull secrets | `[]` | diff --git a/website/docs/install-deploy/overview.md b/website/docs/install-deploy/overview.md index 48784adb14..bb26dd721a 100644 --- a/website/docs/install-deploy/overview.md +++ b/website/docs/install-deploy/overview.md @@ -72,7 +72,7 @@ We have listed them in the table below the figure. TabletServer

- TabletServers are the actual node to manage and store data. + TabletServers are the actual nodes to manage and store data.

diff --git a/website/docs/maintenance/configuration.md b/website/docs/maintenance/configuration.md index a8f6d966f8..6ef2d52b47 100644 --- a/website/docs/maintenance/configuration.md +++ b/website/docs/maintenance/configuration.md @@ -35,7 +35,7 @@ during the Fluss cluster working. | `security.${protocol}.*` | String | (none) | Protocol-specific configuration properties. For example, security.sasl.jaas.config for SASL authentication settings. | | default.bucket.number | Integer | 1 | The default number of buckets for a table in Fluss cluster. It's a cluster-level parameter and all the tables without specifying bucket number in the cluster will use the value as the bucket number. | | default.replication.factor | Integer | 1 | The default replication factor for the log of a table in Fluss cluster. It's a cluster-level parameter, and all the tables without specifying replication factor in the cluster will use the value as replication factor. | -| remote.data.dir | String | (None) | The directory used for storing the kv snapshot data files and remote log for log tiered storage in a Fluss supported filesystem. When upgrading to `remote.data.dirs`, please ensure this value is placed as the first entry in the new configuration.For new clusters, it is recommended to use `remote.data.dirs` instead. If `remote.data.dirs` is configured, this value will be ignored. | +| remote.data.dir | String | (None) | The directory used for storing the kv snapshot data files and remote log for log tiered storage in a Fluss supported filesystem. When upgrading to `remote.data.dirs`, please ensure this value is placed as the first entry in the new configuration. For new clusters, it is recommended to use `remote.data.dirs` instead. If `remote.data.dirs` is configured, this value will be ignored. | | remote.data.dirs | List<String> | (None) | A comma-separated list of directories in Fluss supported filesystems for storing the kv snapshot data files and remote log files of tables/partitions. If configured, when a new table or a new partition is created, one of the directories from this list will be selected according to the strategy specified by `remote.data.dirs.strategy` (`ROUND_ROBIN` by default). If not configured, the system uses `remote.data.dir` as the sole remote data directory for all data. | | remote.data.dirs.strategy | Enum | ROUND_ROBIN | The strategy for selecting the remote data directory from `remote.data.dirs`. The candidate strategies are: [ROUND_ROBIN, WEIGHTED_ROUND_ROBIN], the default strategy is ROUND_ROBIN.
ROUND_ROBIN: this strategy employs a round-robin approach to select one from the available remote directories.
WEIGHTED_ROUND_ROBIN: this strategy selects one of the available remote directories based on the weights configured in `remote.data.dirs.weights`. | | remote.data.dirs.weights | List<Integer>| (None) | The weights of the remote data directories. This is a list of weights corresponding to the `remote.data.dirs` in the same order. When `remote.data.dirs.strategy` is set to `WEIGHTED_ROUND_ROBIN`, this must be configured, and its size must be equal to `remote.data.dirs`; otherwise, it will be ignored. | @@ -73,7 +73,7 @@ during the Fluss cluster working. | tablet-server.rack | String | (None) | The rack for the TabletServer. This will be used in rack aware bucket assignment for fault tolerance. Examples: `RACK1`, `cn-hangzhou-server10` | | data.dir | String | /tmp/fluss-data | This configuration controls the directory where Fluss will store its data. The default value is /tmp/fluss-data | | server.writer-id.expiration-time | Duration | 7d | The time that the tablet server will wait without receiving any write request from a client before expiring the related status. The default value is 7 days. | -| server.writer-id.expiration-check-interval | Duration | 10min | The interval at which to remove writer ids that have expired due to `server.writer-id.expiration-time passing. The default value is 10 minutes. | +| server.writer-id.expiration-check-interval | Duration | 10min | The interval at which to remove writer ids that have expired due to `server.writer-id.expiration-time` passing. The default value is 10 minutes. | | server.background.threads | Integer | 10 | The number of threads to use for various background processing tasks. The default value is 10. | | server.buffer.memory-size | MemorySize | 256mb | The total bytes of memory the server can use, e.g, buffer write-ahead-log rows. | | server.buffer.page-size | MemorySize | 128kb | Size of every page in memory buffers (`server.buffer.memory-size`). | @@ -127,8 +127,8 @@ during the Fluss cluster working. | log.replica.fetch-operation-purge-number | Integer | 1000 | The purge number (in number of requests) of the fetch log operation manager, the default value is 1000. | | log.replica.fetcher-number | Integer | 1 | Number of fetcher threads used to replicate log records from each source tablet server. The total number of fetchers on each tablet server is bound by this parameter multiplied by the number of tablet servers in the cluster. Increasing this value can increase the degree of I/O parallelism in the follower and leader tablet server at the cost of higher CPU and memory utilization. | | log.replica.fetch.backoff-interval | Duration | 1s | The amount of time to sleep when fetch bucket error occurs. | -| log.replica.fetch.max-bytes | MemorySize | 16mb | The maximum amount of data the server should return for a fetch request from follower. Records are fetched in batches, and if the first record batch in the first non-empty bucket of the fetch is larger than this value, the record batch will still be returned to ensure that the fetch can make progress. As such, this is not a absolute maximum. Note that the fetcher performs multiple fetches in parallel. | -| log.replica.fetch.max-bytes-for-bucket | MemorySize | 1mb | The maximum amount of data the server should return for a table bucket in fetch request fom follower. Records are fetched in batches, and the max bytes size is config by this option. | +| log.replica.fetch.max-bytes | MemorySize | 16mb | The maximum amount of data the server should return for a fetch request from follower. Records are fetched in batches, and if the first record batch in the first non-empty bucket of the fetch is larger than this value, the record batch will still be returned to ensure that the fetch can make progress. As such, this is not an absolute maximum. Note that the fetcher performs multiple fetches in parallel. | +| log.replica.fetch.max-bytes-for-bucket | MemorySize | 1mb | The maximum amount of data the server should return for a table bucket in fetch request from follower. Records are fetched in batches, and the max bytes size is configured by this option. | | log.replica.fetch.min-bytes | MemorySize | 1b | The minimum bytes expected for each fetch log request from the follower to response. If not enough bytes, wait up to log.replica.fetch-wait-max-time time to return. | | log.replica.fetch.wait-max-time | Duration | 500ms | The maximum time to wait for enough bytes to be available for a fetch log request from the follower to response. This value should always be less than the `log.replica.max-lag-time` at all times to prevent frequent shrinking of ISR for low throughput tables | | log.replica.min-in-sync-replicas-number | Integer | 1 | When a writer set `client.writer.acks` to all (-1), this configuration specifies the minimum number of replicas that must acknowledge a write for the write to be considered successful. If this minimum cannot be met, then the writer will raise an exception (NotEnoughReplicas). when used together, this config and `client.writer.acks` allow you to enforce greater durability guarantees. A typical scenario would be to create a table with a replication factor of 3. set this conf to 2, and write with acks = -1. This will ensure that the writer raises an exception if a majority of replicas don't receive a write. | diff --git a/website/docs/maintenance/observability/monitor-metrics.md b/website/docs/maintenance/observability/monitor-metrics.md index 50ae66ea35..631163a3b8 100644 --- a/website/docs/maintenance/observability/monitor-metrics.md +++ b/website/docs/maintenance/observability/monitor-metrics.md @@ -16,18 +16,18 @@ Fluss supports different metric types: **Counters**, **Gauges**, **Histograms**, - `Meter`: The gauge exports the meter's rate. Fluss client also has supported built-in metrics to measure operations of **write to**, **read from** fluss cluster, -which can be bridged to Flink use Flink connector standard metrics. +which can be bridged to Flink using Flink connector standard metrics. ## Scope Every metric is assigned an identifier and a set of key-value pairs under which the metric will be reported. The identifier is delimited by `metrics.scope.delimiter`. Currently, the `metrics.scope.delimiter` is not configurable, -it determined by the metric reporter. Take prometheus as example, the scope will delimited by `_`, so the scope like `A_B_C`, +it is determined by the metric reporter. Take prometheus as example, the scope will be delimited by `_`, so the scope like `A_B_C`, while Fluss metrics will always begin with `fluss`, as `fluss_A_B_C`. The key-value pairs are called **variables** and are used to filter metrics. There are no restrictions on the -number of order of variables. Variables are case-sensitive. +number or order of variables. Variables are case-sensitive. ## Reporter diff --git a/website/docs/maintenance/operations/upgrade-notes-0.8.md b/website/docs/maintenance/operations/upgrade-notes-0.8.md index 1c0f32e499..9ad35bc81e 100644 --- a/website/docs/maintenance/operations/upgrade-notes-0.8.md +++ b/website/docs/maintenance/operations/upgrade-notes-0.8.md @@ -160,10 +160,10 @@ This change prioritizes **tiering service stability and performance**: - **Consider manual compaction** for large tables during maintenance windows ## SASL Plain Authorization jaas configuration -Due to Fluss being donated to the Apache Software Foundation, the package namespace has been changed from `org.alibaba.fluss` to `org.apache.fluss`. +Due to Fluss being donated to the Apache Software Foundation, the package namespace has been changed from `com.alibaba.fluss` to `org.apache.fluss`. The `security.sasl.plain.jaas.config` configuration has been updated accordingly: -* Server side: `security.sasl.plain.jaas.config` is changed from `org.alibaba.fluss.security.auth.sasl.plain.PlainLoginModule` to `org.apache.fluss.security.auth.sasl.plain.PlainLoginModule`. -* Client side: `client.security.sasl.jaas.config` is changed from `org.alibaba.fluss.security.auth.sasl.plain.PlainLoginModule` to `org.apache.fluss.security.auth.sasl.plain.PlainLoginModule`. If you are using client.security.sasl.username and client.security.sasl.password configurations, no changes are required as these remain compatible. +* Server side: `security.sasl.plain.jaas.config` is changed from `com.alibaba.fluss.security.auth.sasl.plain.PlainLoginModule` to `org.apache.fluss.security.auth.sasl.plain.PlainLoginModule`. +* Client side: `client.security.sasl.jaas.config` is changed from `com.alibaba.fluss.security.auth.sasl.plain.PlainLoginModule` to `org.apache.fluss.security.auth.sasl.plain.PlainLoginModule`. If you are using client.security.sasl.username and client.security.sasl.password configurations, no changes are required as these remain compatible. ## Flink Catalog required Default database. -Fluss catalog used to not need defalut database even though a wrong defalut database won't be checked. From 0.8, it's required to specify a default database. \ No newline at end of file +Fluss catalog used to not need default database even though a wrong default database won't be checked. From 0.8, it's required to specify a default database. \ No newline at end of file diff --git a/website/docs/maintenance/operations/upgrading.md b/website/docs/maintenance/operations/upgrading.md index b1312f033b..b4123ac0e2 100644 --- a/website/docs/maintenance/operations/upgrading.md +++ b/website/docs/maintenance/operations/upgrading.md @@ -132,7 +132,7 @@ Since Fluss 0.8, the auto-compaction feature during datalake tiering is **disabl | Version | Behavior | |----------------------------------------|--------------------------------------------------------------------------------------------------------------------------------| | **Previous Version(v0.7 and earlier)** | Auto-compaction enabled during tiering | -| **Fluss 0.8+** | **Auto-compaction disabled by default**. Tiering service focus solely on data movement; compaction must be explicitly enabled. | +| **Fluss 0.8+** | **Auto-compaction disabled by default**. Tiering service focuses solely on data movement; compaction must be explicitly enabled. | ### How to Enable Compaction To maintain the previous behavior and enable automatic compaction, you must manually configure `table.datalake.auto-compaction = true` for each table in table option. diff --git a/website/docs/maintenance/tiered-storage/filesystems/azure.md b/website/docs/maintenance/tiered-storage/filesystems/azure.md index c0f997a21d..7a0ed0a564 100644 --- a/website/docs/maintenance/tiered-storage/filesystems/azure.md +++ b/website/docs/maintenance/tiered-storage/filesystems/azure.md @@ -45,7 +45,7 @@ Azure Blob Storage support is not included in the default Fluss distribution. To ## Configurations setup -To enabled Azure Blob Storage as remote storage, there are some required configurations that must be added to Fluss' `server.yaml`: +To enable Azure Blob Storage as remote storage, there are some required configurations that must be added to Fluss' `server.yaml`: ```yaml # The dir that used to be as the remote storage of Fluss, use the Azure Data Lake Storage URI diff --git a/website/docs/maintenance/tiered-storage/filesystems/hdfs.md b/website/docs/maintenance/tiered-storage/filesystems/hdfs.md index 3dac3c7cda..e50f36f8b3 100644 --- a/website/docs/maintenance/tiered-storage/filesystems/hdfs.md +++ b/website/docs/maintenance/tiered-storage/filesystems/hdfs.md @@ -29,7 +29,7 @@ Verify downloaded JARs against the [KEYS file](https://downloads.apache.org/incu ## Configurations setup -To enabled HDFS as remote storage, you need to define the hdfs path as remote storage in Fluss' `server.yaml`: +To enable HDFS as remote storage, you need to define the hdfs path as remote storage in Fluss' `server.yaml`: ```yaml title="conf/server.yaml" # The dir that used to be as the remote storage of Fluss remote.data.dir: hdfs://namenode:50010/path/to/remote/storage diff --git a/website/docs/maintenance/tiered-storage/filesystems/obs.md b/website/docs/maintenance/tiered-storage/filesystems/obs.md index e701b2ee5f..ebe2785bf4 100644 --- a/website/docs/maintenance/tiered-storage/filesystems/obs.md +++ b/website/docs/maintenance/tiered-storage/filesystems/obs.md @@ -45,7 +45,7 @@ HuaweiCloud OBS support is not included in the default Fluss distribution. To en ## Configurations setup -To enabled HuaweiCloud OBS as remote storage, there are some required configurations that must be added to Fluss' `server.yaml`: +To enable HuaweiCloud OBS as remote storage, there are some required configurations that must be added to Fluss' `server.yaml`: ```yaml # The dir that used to be as the remote storage of Fluss diff --git a/website/docs/maintenance/tiered-storage/filesystems/oss.md b/website/docs/maintenance/tiered-storage/filesystems/oss.md index 330f0f8e0e..e11fce1a9e 100644 --- a/website/docs/maintenance/tiered-storage/filesystems/oss.md +++ b/website/docs/maintenance/tiered-storage/filesystems/oss.md @@ -32,7 +32,7 @@ Verify downloaded JARs against the [KEYS file](https://downloads.apache.org/incu ## Configurations setup -To enabled OSS as remote storage, there are some required configurations that must be added to Fluss' `server.yaml`: +To enable OSS as remote storage, there are some required configurations that must be added to Fluss' `server.yaml`: ```yaml # The dir that used to be as the remote storage of Fluss @@ -79,5 +79,5 @@ such as `acs:ram::123456789012:role/testrole`. Since client will use the STS tok See more detail in [AssumeRole](https://help.aliyun.com/zh/ram/developer-reference/api-sts-2015-04-01-assumerole). Apart from the above configurations, you can also define the configuration keys mentioned in the [Hadoop OSS documentation](http://hadoop.apache.org/docs/current/hadoop-aliyun/tools/hadoop-aliyun/index.html) -in the Fluss' `server.yaml`. These configurations defines in Hadoop OSS documentation are advanced configurations which are usually used by performance tuning. +in the Fluss' `server.yaml`. These configurations defined in Hadoop OSS documentation are advanced configurations which are usually used by performance tuning. diff --git a/website/docs/maintenance/tiered-storage/filesystems/overview.md b/website/docs/maintenance/tiered-storage/filesystems/overview.md index c02abbda67..af32189e12 100644 --- a/website/docs/maintenance/tiered-storage/filesystems/overview.md +++ b/website/docs/maintenance/tiered-storage/filesystems/overview.md @@ -36,7 +36,7 @@ The Fluss project supports the following file systems: - **[AWS S3](s3.md)** is supported by `fluss-fs-s3` and registered under the `s3://` URI scheme. S3 filesystem is included in default Fluss binary distribution, so you can use it directly without manual installation. -- **[Azure Blob Storage](azure.md)** is supported by `fluss-fs-azure` and registered under the `abfs://`,`abfss://`,`wasb://`,`wasbs://`, URI schemes. Please make sure to [manually install the OBS plugin](azure.md#install-azure-fs-plugin-manually). +- **[Azure Blob Storage](azure.md)** is supported by `fluss-fs-azure` and registered under the `abfs://`,`abfss://`,`wasb://`,`wasbs://`, URI schemes. Please make sure to [manually install the Azure plugin](azure.md#install-azure-fs-plugin-manually). - **[HuaweiCloud OBS](obs.md)** is supported by `fluss-fs-obs` and registered under the `obs://` URI scheme. Please make sure to [manually install the OBS plugin](obs.md#install-obs-plugin-manually). diff --git a/website/docs/maintenance/tiered-storage/filesystems/s3.md b/website/docs/maintenance/tiered-storage/filesystems/s3.md index 311573a591..2acebd7589 100644 --- a/website/docs/maintenance/tiered-storage/filesystems/s3.md +++ b/website/docs/maintenance/tiered-storage/filesystems/s3.md @@ -29,7 +29,7 @@ Verify downloaded JARs against the [KEYS file](https://downloads.apache.org/incu ## Configurations setup -To enabled S3 as remote storage, there are some required configurations that must be added to Fluss' `server.yaml`: +To enable S3 as remote storage, there are some required configurations that must be added to Fluss' `server.yaml`: ```yaml # The dir that used to be as the remote storage of Fluss diff --git a/website/docs/maintenance/tiered-storage/remote-storage.md b/website/docs/maintenance/tiered-storage/remote-storage.md index 1d00a537d3..2edbbbe613 100644 --- a/website/docs/maintenance/tiered-storage/remote-storage.md +++ b/website/docs/maintenance/tiered-storage/remote-storage.md @@ -34,7 +34,7 @@ Below is the list for all configurations to control the log segments tiered beha ### Table configurations about remote log When local log segments are copied to remote storage, the local log segments will be deleted to reduce local disk cost. -But sometimes, we want to keep the several latest log segments retain in local, although they have been coped to remote storage for better read performance. +But sometimes, we want to keep the several latest log segments retain in local, although they have been copied to remote storage for better read performance. You can control how many log segments to retain in local by setting the configuration `table.log.tiered.local-segments`(default is 2) per table. ## Remote snapshot of primary key table diff --git a/website/docs/quickstart/flink.md b/website/docs/quickstart/flink.md index edb92924d0..74b6c40c53 100644 --- a/website/docs/quickstart/flink.md +++ b/website/docs/quickstart/flink.md @@ -277,7 +277,7 @@ END; ``` Fluss primary-key tables support high QPS point lookup queries on primary keys. Performing a [lookup join](https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/dev/table/sql/queries/joins/#lookup-join) is really efficient and you can use it to enrich -the `fluss_orders` table with information from the `fluss_customer` and `fluss_nation` primary-key tables. +the `fluss_order` table with information from the `fluss_customer` and `fluss_nation` primary-key tables. ```sql title="Flink SQL" INSERT INTO enriched_orders diff --git a/website/docs/quickstart/security.md b/website/docs/quickstart/security.md index 603dad8b74..91089b066e 100644 --- a/website/docs/quickstart/security.md +++ b/website/docs/quickstart/security.md @@ -362,7 +362,7 @@ SELECT * FROM `consumer_catalog`.`fluss`.`fluss_order` LIMIT 10; ``` -Attempting to read data using the `developer` user also get the same result: +Attempting to read data using the `developer` user also gets the same result: ```sql SET 'execution.runtime-mode' = 'batch'; -- use tableau result mode diff --git a/website/docs/security/authentication.md b/website/docs/security/authentication.md index f4a661bd09..e78e897ed7 100644 --- a/website/docs/security/authentication.md +++ b/website/docs/security/authentication.md @@ -59,7 +59,7 @@ Clients must specify the appropriate security protocol and authentication mechan |----------------------------------|--------|---------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | client.security.protocol | String | PLAINTEXT | The security protocol used to communicate with brokers. Currently, only `PLAINTEXT` and `SASL` are supported, the configuration value is case insensitive. | | client.security.sasl.mechanism | String | PLAIN | The SASL mechanism used for authentication. Only support PLAIN now, but will support more mechanisms in the future. | -| client.security.sasl.username | String | (none) | The password to use for client-side SASL JAAS authentication. This is used when the client connects to the Fluss cluster with SASL authentication enabled. If not provided, the username will be read from the JAAS configuration string specified by `client.security.sasl.jaas.config`. | +| client.security.sasl.username | String | (none) | The username to use for client-side SASL JAAS authentication. This is used when the client connects to the Fluss cluster with SASL authentication enabled. If not provided, the username will be read from the JAAS configuration string specified by `client.security.sasl.jaas.config`. | | client.security.sasl.password | String | (none) | The password to use for client-side SASL JAAS authentication. This is used when the client connects to the Fluss cluster with SASL authentication enabled. If not provided, the password will be read from the JAAS configuration string specified by `client.security.sasl.jaas.config`. | | client.security.sasl.jaas.config | String | (none) | JAAS configuration for SASL. If not set, fallback to system property `-Djava.security.auth.login.config`. | diff --git a/website/docs/security/authorization.md b/website/docs/security/authorization.md index dd748e66d9..58e12d97d7 100644 --- a/website/docs/security/authorization.md +++ b/website/docs/security/authorization.md @@ -155,9 +155,9 @@ CALL [catalog].sys.drop_acl( | Parameter | Required | Description | |------------|----------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | resource | NO | The resource to apply the ACL to (e.g., `cluster`, `cluster.db1`, `cluster.db1.table1`). If not specified, it will filter all the resource (same as `ANY`) | -| permission | NO | The permission to grant or deny to the principal on the resource (e.g., `ALLOW`, `DENY`). If If not specified, it will filter all the permission(same as `ANY`) | -| principal | NO | The principal to apply the ACL to (e.g., `User:alice`, `Role:admin`). If If not specified, it will filter all the principal(same as `ANY`) | -| operation | NO | The operation to allow or deny for the principal on the resource (e.g., `READ`, `WRITE`, `CREATE`, `DELETE`, `ALTER`, `DESCRIBE`, `ANY`, `ALL`). If If not specified, it will filter all the operation(same as `ANY`) | +| permission | NO | The permission to grant or deny to the principal on the resource (e.g., `ALLOW`, `DENY`). If not specified, it will filter all the permission(same as `ANY`) | +| principal | NO | The principal to apply the ACL to (e.g., `User:alice`, `Role:admin`). If not specified, it will filter all the principal(same as `ANY`) | +| operation | NO | The operation to allow or deny for the principal on the resource (e.g., `READ`, `WRITE`, `CREATE`, `DELETE`, `ALTER`, `DESCRIBE`, `ANY`, `ALL`). If not specified, it will filter all the operation(same as `ANY`) | | host | NO | The host to apply the ACL to (e.g., `127.0.0.1`). If not specified, the ACL applies to all hosts( same as `ANY`) | ### List ACL @@ -165,7 +165,7 @@ List ACL will return a list of ACLs that match the specified criteria. The general syntax is: ```sql title="Flink SQL" -- Recommended, use named argument (only supported since Flink 1.19) -CALL [catalog].sys.drop_acl( +CALL [catalog].sys.list_acl( resource => '[resource]', permission => 'ALLOW', principal => '[principal_type:principal_name]', @@ -174,7 +174,7 @@ CALL [catalog].sys.drop_acl( ); -- Use indexed argument -CALL [catalog].sys.drop_acl( +CALL [catalog].sys.list_acl( '[resource]', '[permission]', '[principal_type:principal_name]', @@ -184,10 +184,10 @@ CALL [catalog].sys.drop_acl( ``` | Parameter | Required | Description | |------------|----------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| resource | NO | The resource to apply the ACL to (e.g., `cluster`, `cluster.db1`, `cluster.db1.table1`). If If not specified, it will filter all the resource(same as `ANY`) | -| permission | NO | The permission to grant or deny to the principal on the resource (e.g., `ALLOW`, `DENY`). If If not specified, it will filter all the permission(same as `ANY`) | -| principal | NO | The principal to apply the ACL to (e.g., `User:alice`, `Role:admin`). If If not specified, it will filter all the principal(same as `ANY`) | -| operation | NO | The operation to allow or deny for the principal on the resource (e.g., `READ`, `WRITE`, `CREATE`, `DELETE`, `ALTER`, `DESCRIBE`, `ANY`, `ALL`). If If not specified, it will filter all the operation(same as `ANY`) | +| resource | NO | The resource to apply the ACL to (e.g., `cluster`, `cluster.db1`, `cluster.db1.table1`). If not specified, it will filter all the resource(same as `ANY`) | +| permission | NO | The permission to grant or deny to the principal on the resource (e.g., `ALLOW`, `DENY`). If not specified, it will filter all the permission(same as `ANY`) | +| principal | NO | The principal to apply the ACL to (e.g., `User:alice`, `Role:admin`). If not specified, it will filter all the principal(same as `ANY`) | +| operation | NO | The operation to allow or deny for the principal on the resource (e.g., `READ`, `WRITE`, `CREATE`, `DELETE`, `ALTER`, `DESCRIBE`, `ANY`, `ALL`). If not specified, it will filter all the operation(same as `ANY`) | | host | NO | The host to apply the ACL to (e.g., `127.0.0.1`). If not specified, the ACL applies to all hosts( same as `ANY`) | diff --git a/website/docs/security/overview.md b/website/docs/security/overview.md index aecbf87a51..2752fa9e68 100644 --- a/website/docs/security/overview.md +++ b/website/docs/security/overview.md @@ -35,9 +35,9 @@ Server side listener configurations: | Option | Type | Default Value | Description | | --- | --- | --- | --- | | bind.listeners | String | FLUSS://localhost:9123 | The network address and port to which the server binds for accepting connections. This defines the interface and port where the server will listen for incoming requests. The format is `listener_name://host:port`, and multiple addresses can be specified, separated by commas. Use `0.0.0.0` for the `host` to bind to all available interfaces which is dangerous on production and not suggested for production usage. The `listener_name` serves as an identifier for the address in the configuration. For example, `internal.listener.name` specifies the address used for internal server communication. If multiple addresses are configured, ensure that the `listener_name` values are unique. | -| advertised.listeners | (none) | String | The externally advertised address and port for client connections. Required in distributed environments when the bind address is not publicly reachable. Format matches `bind.listeners` (listener_name://host:port). Defaults to the value of `bind.listeners` if not explicitly configured. | -| internal.listener.name | FLUSS | String |The listener name used for internal server communication. | -| security.protocol.map | Map | (none) | A map defining the authentication protocol for each listener. The format is `listenerName1:protocol1,listenerName2:protocol2`, e.g.,`CLIENT:SASL, INTERNAL:PLAINTEXT`.A map defining the authentication protocol for each listener. The format is `listenerName1:protocol1,listenerName2:protocol2`, e.g., `INTERNAL:PLAINTEXT,CLIENT:GSSAPI`. Each listener can be associated with a specific authentication protocol. Listeners not included in the map will use PLAINTEXT by default, which does not require authentication. Currently, only PLAINTEXT and SASL/PLAIN are supported. | +| advertised.listeners | String | (none) | The externally advertised address and port for client connections. Required in distributed environments when the bind address is not publicly reachable. Format matches `bind.listeners` (listener_name://host:port). Defaults to the value of `bind.listeners` if not explicitly configured. | +| internal.listener.name | String | FLUSS |The listener name used for internal server communication. | +| security.protocol.map | Map | (none) | A map defining the authentication protocol for each listener. The format is `listenerName1:protocol1,listenerName2:protocol2`, e.g., `INTERNAL:PLAINTEXT,CLIENT:GSSAPI`. Each listener can be associated with a specific authentication protocol. Listeners not included in the map will use PLAINTEXT by default, which does not require authentication. Currently, only PLAINTEXT and SASL/PLAIN are supported. | Here is an example server side configuration: @@ -85,7 +85,7 @@ Example usage: * `new FlussPrincipal("admins", "Group")` – A group-based principal for authorization. ### Enable Authorization and Assign Super Users -Fluss provides a pluggable authorization framework that uses Access Control Lists (ACLs) to determine whether a given Fluss Principal is allowed to perform an operation on a specific resource.To enable authorization, you need to configure the following properties: +Fluss provides a pluggable authorization framework that uses Access Control Lists (ACLs) to determine whether a given Fluss Principal is allowed to perform an operation on a specific resource. To enable authorization, you need to configure the following properties: | Option | Type | Default Value | Description | |--------------------|---------|---------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | authorizer.enabled | Boolean | false | Specifies whether to enable the authorization feature. | @@ -114,13 +114,13 @@ security.protocol.map: CLIENT:SASL, INTERNAL:PLAINTEXT The client presents its credentials using the selected authentication method. For SASL/PLAIN, this typically includes a username and password. - If the credentials are valid and the authentication is successful, and the client is allowed to proceed with further operations. Otherwise, the connection is rejected, and an authentication error is returned. + If the credentials are valid and the authentication is successful, the client is allowed to proceed with further operations. Otherwise, the connection is rejected, and an authentication error is returned. 4. Fluss Principal is created Upon successful authentication, Fluss creates a FlussPrincipal, representing the identity of the authenticated user -5. Authorization check occur +5. Authorization check occurs Before allowing any operation (like producing or consuming data), Fluss checks whether the Fluss Principal has permission to perform that action based on configured ACL rules. diff --git a/website/docs/streaming-lakehouse/overview.md b/website/docs/streaming-lakehouse/overview.md index d86ab3e573..80552e4aba 100644 --- a/website/docs/streaming-lakehouse/overview.md +++ b/website/docs/streaming-lakehouse/overview.md @@ -13,7 +13,7 @@ The well-known data lake formats such as [Apache Iceberg](https://iceberg.apache facilitating a harmonious balance between data storage, reliability, and analytical capabilities within a single, unified platform. Lakehouse, as a modern architecture, is effective in addressing the complex needs of data management and analytics. -However, they struggle to meet real-time analytics scenarios that require sub-second-level data freshness due to limitations in their implementation. +However, it struggles to meet real-time analytics scenarios that require sub-second-level data freshness due to limitations in its implementation. With these data lake formats, you will get into a contradictory situation: 1. If you require low latency, then you must write and commit frequently, resulting in many small Parquet files. This becomes inefficient for @@ -39,7 +39,7 @@ Some powerful features it provides are: - **Unified Metadata**: Fluss provides unified table metadata for both data in Stream and Lakehouse. Users only need to manage one table and can access real-time streaming data, historical data, or both combined. - **Union Reads**: Compute engines that perform queries on the table will read the union of the real-time streaming data and Lakehouse data. Currently, Flink and Spark support union reads, with more engines on the roadmap. -- **Real-Time Lakehouse**: The union reads help Lakehouse evolving from near-real-time analytics to truly real-time analytics. This empowers businesses to gain more valuable insights from real-time data. +- **Real-Time Lakehouse**: The union reads help Lakehouse evolve from near-real-time analytics to truly real-time analytics. This empowers businesses to gain more valuable insights from real-time data. - **Analytical Streams**: The union reads help data streams to have the powerful analytics capabilities. This reduces complexity when developing streaming applications, simplifies debugging, and allows for immediate access to live data insights. - **Connect to Lakehouse Ecosystem**: Fluss keeps the table metadata in sync with data lake catalogs while compacting data into Lakehouse. As a result, external engines like Spark, StarRocks, Flink, and Trino can read the data directly. They simply connect to the data lake catalog. diff --git a/website/docs/table-design/data-distribution/bucketing.md b/website/docs/table-design/data-distribution/bucketing.md index 615c1cb0d0..9cc6d069a6 100644 --- a/website/docs/table-design/data-distribution/bucketing.md +++ b/website/docs/table-design/data-distribution/bucketing.md @@ -23,7 +23,7 @@ Primary-Key Tables use primary key (excluding partition key) as the bucket key b ## Sticky Bucketing **Sticky Bucketing** enables larger batches and reduces latency when writing records into Log Tables. After sending a batch, the sticky bucket changes. Over time, the records are spread out evenly among all the buckets. -Sticky Bucketing is the default bucketing strategy for Log Tables. This is quite important because Log Tables uses Apache Arrow as the underling data format which is efficient for large batches. +Sticky Bucketing is the default bucketing strategy for Log Tables. This is quite important because Log Tables use Apache Arrow as the underlying data format which is efficient for large batches. **Usage**: setting `'client.writer.bucket.no-key-assigner'='sticky'` property for the table to enable this strategy. PrimaryKey Tables do not support this strategy. diff --git a/website/docs/table-design/data-distribution/partitioning.md b/website/docs/table-design/data-distribution/partitioning.md index 69255bd0fb..b39c593334 100644 --- a/website/docs/table-design/data-distribution/partitioning.md +++ b/website/docs/table-design/data-distribution/partitioning.md @@ -9,8 +9,8 @@ sidebar_position: 2 In Fluss, a **Partitioned Table** organizes data based on one or more partition keys, providing a way to improve query performance and manageability for large datasets. Partitions allow the system to divide data into distinct segments, each corresponding to specific values of the partition keys. For partitioned tables, Fluss supports three strategies of managing partitions. - - **Manual management partitions**, user can create new partitions or drop exists partitions. Learn how to create or drop partitions please refer to [Add Partition](engine-flink/ddl.md#add-partition) and [Drop Partition](engine-flink/ddl.md#drop-partition). - - **Auto management partitions**, the partitions will be created based on the auto partitioning rules configured at the time of table creation, and expired partitions are automatically removed, ensuring data not expanding unlimited. See [Auto Partitioning](table-design/data-distribution/partitioning.md#auto-partitioning). + - **Manual management partitions**, user can create new partitions or drop existing partitions. Learn how to create or drop partitions please refer to [Add Partition](engine-flink/ddl.md#add-partition) and [Drop Partition](engine-flink/ddl.md#drop-partition). + - **Auto management partitions**, the partitions will be created based on the auto partitioning rules configured at the time of table creation, and expired partitions are automatically removed, ensuring data does not expand indefinitely. See [Auto Partitioning](table-design/data-distribution/partitioning.md#auto-partitioning). - **Dynamic create partitions**, the partitions will be created automatically based on the data being written to the table. See [Dynamic Partitioning](table-design/data-distribution/partitioning.md#dynamic-partitioning). These three strategies are orthogonal and can coexist on the same table. @@ -22,7 +22,7 @@ Partitioned tables (either primary-key table or log table) support configuring p For example, in an `Order` primary key table, the partition key can be defined as `(date, region)`. Data will then be stored in partitions corresponding to specific combinations such as `date=2025-04-05, region=US`. Users can leverage partition pruning during streaming queries — such as filtering by `region=US` — to improve read performance through partition pushdown. ### Key Benefits of Partitioned Tables -- **Improved Query Performance:** By narrowing down the query scope to specific partitions, the system reads fewer data, reducing query execution time. +- **Improved Query Performance:** By narrowing down the query scope to specific partitions, the system reads less data, reducing query execution time. - **Data Organization:** Partitions help in logically organizing data, making it easier to manage and query. - **Scalability:** Partitioning large datasets distributes the data across smaller, manageable chunks, improving scalability. diff --git a/website/docs/table-design/table-types/log-table.md b/website/docs/table-design/table-types/log-table.md index 405aa272e3..ef9d297dda 100644 --- a/website/docs/table-design/table-types/log-table.md +++ b/website/docs/table-design/table-types/log-table.md @@ -54,7 +54,7 @@ When a query specifies only a subset of columns, Fluss skips reading irrelevant This enables efficient column pruning during query execution and ensures that only the required columns are read, minimizing I/O overhead and improving overall system efficiency. -During query execution, query engines like Flink analyzes the query to identify the columns required for processing and tells Fluss to only read the necessary columns. +During query execution, query engines like Flink analyze the query to identify the columns required for processing and tell Fluss to only read the necessary columns. For example the following streaming query: ```sql diff --git a/website/docs/table-design/table-types/pk-table.md b/website/docs/table-design/table-types/pk-table.md index 54e22d5020..6a8335ac89 100644 --- a/website/docs/table-design/table-types/pk-table.md +++ b/website/docs/table-design/table-types/pk-table.md @@ -37,7 +37,7 @@ partition key. ## Bucket Assigning For primary key tables, Fluss always determines which bucket the data belongs to based on the hash value of the bucket -key (It must be a subset of the primary keys excluding partition keys of the primary key table) for each record. If the bucket key is not specified, the bucket key will be used as the primary key (excluding the partition key). +key (It must be a subset of the primary keys excluding partition keys of the primary key table) for each record. If the bucket key is not specified, the primary key (excluding the partition key) will be used as the bucket key. Data with the same hash value will be distributed to the same bucket. ## Partial Update