From c04c3b77553f59888dec51329f8199435fad5fa1 Mon Sep 17 00:00:00 2001 From: Rich Loveland Date: Thu, 30 Apr 2026 13:54:23 -0400 Subject: [PATCH 1/5] Add more (high-level) details about Pebble compression features Fixes DOC-16758 --- src/current/v26.1/architecture/storage-layer.md | 12 ++++++++++++ .../v26.1/cockroach-debug-pebble-db-analyze-data.md | 12 ++++++++++++ src/current/v26.2/architecture/storage-layer.md | 12 ++++++++++++ .../v26.2/cockroach-debug-pebble-db-analyze-data.md | 12 ++++++++++++ 4 files changed, 48 insertions(+) diff --git a/src/current/v26.1/architecture/storage-layer.md b/src/current/v26.1/architecture/storage-layer.md index 9fe05b3e55c..a4674f498f3 100644 --- a/src/current/v26.1/architecture/storage-layer.md +++ b/src/current/v26.1/architecture/storage-layer.md @@ -67,6 +67,18 @@ SSTs are an on-disk representation of sorted lists of key-value pairs. Conceptua SST files are immutable; they are never modified, even during the [compaction process](#compaction). +##### SST compression + +Pebble compresses SSTable and blob value data to reduce physical storage use. The default profile, `fastest`, is optimized for low CPU overhead and is appropriate for most workloads. + +For advanced storage tuning, CockroachDB exposes the `storage.sstable.compression_algorithm` [cluster setting]({% link {{ page.version.version }}/cluster-settings.md %}). The profile values are ordered by increasing compression effort: `fastest`, `fast`, `balanced`, and `good`. Higher-effort profiles can improve compression for some workloads, but can also increase CPU usage for writes, compactions, and reads that decompress data. Most users do not need to tune this setting. Work with [Cockroach Labs Support](https://support.cockroachlabs.com/) before changing this setting in production. + +Changing `storage.sstable.compression_algorithm` does not immediately recompress existing SST files. SSTs are immutable, so a new setting applies as Pebble writes new SSTs or rewrites existing SSTs during compaction, ingestion, restore, or other SST-writing work. During a transition, a store can contain SSTs compressed with multiple algorithms. + +To evaluate the compression behavior of an existing store or backup, use [`cockroach debug pebble db analyze-data`]({% link {{ page.version.version }}/cockroach-debug-pebble-db-analyze-data.md %}). Compare the compression ratio with the compression and decompression throughput for representative data before changing the cluster setting. + +You can monitor overall storage compression with the `storage.compression.cr` metric. + ##### LSM levels The levels of the LSM are organized from L0 to L6. L0 is the top-most level. L6 is the bottom-most level. New data is added into L0 (e.g., using [`INSERT`]({% link {{ page.version.version }}/insert.md %}) or [`IMPORT INTO`]({% link {{ page.version.version }}/import-into.md %})) and then merged down into lower levels over time. diff --git a/src/current/v26.1/cockroach-debug-pebble-db-analyze-data.md b/src/current/v26.1/cockroach-debug-pebble-db-analyze-data.md index b7dbc59a9bb..6edcb65300c 100644 --- a/src/current/v26.1/cockroach-debug-pebble-db-analyze-data.md +++ b/src/current/v26.1/cockroach-debug-pebble-db-analyze-data.md @@ -55,6 +55,17 @@ This data can help Cockroach Labs evaluate compression defaults and can help you The output CSV file is periodically rewritten while the command is running. Even if the command is interrupted, you can still use the most recently written output. +### Interpret results + +The command groups sampled blocks into buckets and reports compression results for each bucket. Use the results to compare algorithms for the data that is actually stored in the analyzed store or backup. + +- `CR` is the compression ratio. Higher values mean that compressed data uses less storage. +- `Comp` is compression throughput. Lower throughput means higher CPU cost when Pebble writes or rewrites SSTs. +- `Decomp` is decompression throughput. Lower throughput means higher CPU cost when CockroachDB reads compressed data. +- `Snappy`, `MinLZ1`, `Zstd1`, `Auto1/30`, `Auto1/15`, and `Zstd3` are compression experiments. These names are not a one-to-one list of valid values for [`storage.sstable.compression_algorithm`]({% link {{ page.version.version }}/cluster-settings.md %}); use them to compare the general speed and space tradeoffs of lower-effort, adaptive, and higher-effort compression. + +Most users do not need to tune SSTable compression settings. Use these results to understand whether compression tradeoffs are relevant for a representative store, and consult [Cockroach Labs Support](https://support.cockroachlabs.com/) before changing SSTable compression settings in production. For more information about Pebble SSTable compression, see [SST compression]({% link {{ page.version.version }}/architecture/storage-layer.md %}#sst-compression). + ## Examples ### View help output @@ -155,4 +166,5 @@ Sampled 4 files, 2.4MB (100.00%) ## See also - [`cockroach` Commands Overview]({% link {{ page.version.version }}/cockroach-commands.md %}) +- [Storage Layer: SST compression]({% link {{ page.version.version }}/architecture/storage-layer.md %}#sst-compression) - [Troubleshooting Overview]({% link {{ page.version.version }}/troubleshooting-overview.md %}) diff --git a/src/current/v26.2/architecture/storage-layer.md b/src/current/v26.2/architecture/storage-layer.md index 914152ea1d2..a2816f3328d 100644 --- a/src/current/v26.2/architecture/storage-layer.md +++ b/src/current/v26.2/architecture/storage-layer.md @@ -67,6 +67,18 @@ SSTs are an on-disk representation of sorted lists of key-value pairs. Conceptua SST files are immutable; they are never modified, even during the [compaction process](#compaction). +##### SST compression + +Pebble compresses SSTable and blob value data to reduce physical storage use. The default profile, `fastest`, is optimized for low CPU overhead and is appropriate for most workloads. + +For advanced storage tuning, CockroachDB exposes the `storage.sstable.compression_algorithm` [cluster setting]({% link {{ page.version.version }}/cluster-settings.md %}). The profile values are ordered by increasing compression effort: `fastest`, `fast`, `balanced`, and `good`. Higher-effort profiles can improve compression for some workloads, but can also increase CPU usage for writes, compactions, and reads that decompress data. Most users do not need to tune this setting. Work with [Cockroach Labs Support](https://support.cockroachlabs.com/) before changing this setting in production. + +Changing `storage.sstable.compression_algorithm` does not immediately recompress existing SST files. SSTs are immutable, so a new setting applies as Pebble writes new SSTs or rewrites existing SSTs during compaction, ingestion, restore, or other SST-writing work. During a transition, a store can contain SSTs compressed with multiple algorithms. + +To evaluate the compression behavior of an existing store or backup, use [`cockroach debug pebble db analyze-data`]({% link {{ page.version.version }}/cockroach-debug-pebble-db-analyze-data.md %}). Compare the compression ratio with the compression and decompression throughput for representative data before changing the cluster setting. + +You can monitor overall storage compression with the `storage.compression.cr` metric. + ##### LSM levels The levels of the LSM are organized from L0 to L6. L0 is the top-most level. L6 is the bottom-most level. New data is added into L0 (e.g., using [`INSERT`]({% link {{ page.version.version }}/insert.md %}) or [`IMPORT INTO`]({% link {{ page.version.version }}/import-into.md %})) and then merged down into lower levels over time. diff --git a/src/current/v26.2/cockroach-debug-pebble-db-analyze-data.md b/src/current/v26.2/cockroach-debug-pebble-db-analyze-data.md index b7dbc59a9bb..6edcb65300c 100644 --- a/src/current/v26.2/cockroach-debug-pebble-db-analyze-data.md +++ b/src/current/v26.2/cockroach-debug-pebble-db-analyze-data.md @@ -55,6 +55,17 @@ This data can help Cockroach Labs evaluate compression defaults and can help you The output CSV file is periodically rewritten while the command is running. Even if the command is interrupted, you can still use the most recently written output. +### Interpret results + +The command groups sampled blocks into buckets and reports compression results for each bucket. Use the results to compare algorithms for the data that is actually stored in the analyzed store or backup. + +- `CR` is the compression ratio. Higher values mean that compressed data uses less storage. +- `Comp` is compression throughput. Lower throughput means higher CPU cost when Pebble writes or rewrites SSTs. +- `Decomp` is decompression throughput. Lower throughput means higher CPU cost when CockroachDB reads compressed data. +- `Snappy`, `MinLZ1`, `Zstd1`, `Auto1/30`, `Auto1/15`, and `Zstd3` are compression experiments. These names are not a one-to-one list of valid values for [`storage.sstable.compression_algorithm`]({% link {{ page.version.version }}/cluster-settings.md %}); use them to compare the general speed and space tradeoffs of lower-effort, adaptive, and higher-effort compression. + +Most users do not need to tune SSTable compression settings. Use these results to understand whether compression tradeoffs are relevant for a representative store, and consult [Cockroach Labs Support](https://support.cockroachlabs.com/) before changing SSTable compression settings in production. For more information about Pebble SSTable compression, see [SST compression]({% link {{ page.version.version }}/architecture/storage-layer.md %}#sst-compression). + ## Examples ### View help output @@ -155,4 +166,5 @@ Sampled 4 files, 2.4MB (100.00%) ## See also - [`cockroach` Commands Overview]({% link {{ page.version.version }}/cockroach-commands.md %}) +- [Storage Layer: SST compression]({% link {{ page.version.version }}/architecture/storage-layer.md %}#sst-compression) - [Troubleshooting Overview]({% link {{ page.version.version }}/troubleshooting-overview.md %}) From 5f4a18472787a4ad70d34929eb1def51ec442088 Mon Sep 17 00:00:00 2001 From: Rich Loveland Date: Mon, 4 May 2026 13:16:49 -0400 Subject: [PATCH 2/5] Update with RaduBerinde feedback (1) --- src/current/v26.1/architecture/storage-layer.md | 6 +++--- src/current/v26.1/cockroach-debug-pebble-db-analyze-data.md | 6 +++--- src/current/v26.2/architecture/storage-layer.md | 6 +++--- src/current/v26.2/cockroach-debug-pebble-db-analyze-data.md | 6 +++--- 4 files changed, 12 insertions(+), 12 deletions(-) diff --git a/src/current/v26.1/architecture/storage-layer.md b/src/current/v26.1/architecture/storage-layer.md index a4674f498f3..d70787691b6 100644 --- a/src/current/v26.1/architecture/storage-layer.md +++ b/src/current/v26.1/architecture/storage-layer.md @@ -69,13 +69,13 @@ SST files are immutable; they are never modified, even during the [compaction pr ##### SST compression -Pebble compresses SSTable and blob value data to reduce physical storage use. The default profile, `fastest`, is optimized for low CPU overhead and is appropriate for most workloads. +Pebble compresses SSTable and blob value data to reduce physical storage use. The default profile, `fastest`, uses MinLZ1 on `amd64` and `arm64` platforms, is optimized for low CPU overhead, and is appropriate for most workloads. -For advanced storage tuning, CockroachDB exposes the `storage.sstable.compression_algorithm` [cluster setting]({% link {{ page.version.version }}/cluster-settings.md %}). The profile values are ordered by increasing compression effort: `fastest`, `fast`, `balanced`, and `good`. Higher-effort profiles can improve compression for some workloads, but can also increase CPU usage for writes, compactions, and reads that decompress data. Most users do not need to tune this setting. Work with [Cockroach Labs Support](https://support.cockroachlabs.com/) before changing this setting in production. +For advanced storage tuning, CockroachDB exposes the `storage.sstable.compression_algorithm` [cluster setting]({% link {{ page.version.version }}/cluster-settings.md %}). The profile values are ordered by increasing compression effort: `fastest`, `fast`, `balanced`, and `good`. These profiles enable selective use of Zstd1 depending on the block type, LSM level, and compression benefit. Higher-effort profiles use Zstd1 more frequently and can improve compression for some workloads, but can also increase CPU usage for writes, compactions, and reads that decompress data. Most users do not need to tune this setting. Work with [Cockroach Labs Support](https://support.cockroachlabs.com/) before changing this setting in production. Changing `storage.sstable.compression_algorithm` does not immediately recompress existing SST files. SSTs are immutable, so a new setting applies as Pebble writes new SSTs or rewrites existing SSTs during compaction, ingestion, restore, or other SST-writing work. During a transition, a store can contain SSTs compressed with multiple algorithms. -To evaluate the compression behavior of an existing store or backup, use [`cockroach debug pebble db analyze-data`]({% link {{ page.version.version }}/cockroach-debug-pebble-db-analyze-data.md %}). Compare the compression ratio with the compression and decompression throughput for representative data before changing the cluster setting. +To evaluate the CPU usage vs. size tradeoff for your particular data, use [`cockroach debug pebble db analyze-data`]({% link {{ page.version.version }}/cockroach-debug-pebble-db-analyze-data.md %}). Review the results with [Cockroach Labs Support](https://support.cockroachlabs.com/) before changing the cluster setting. You can monitor overall storage compression with the `storage.compression.cr` metric. diff --git a/src/current/v26.1/cockroach-debug-pebble-db-analyze-data.md b/src/current/v26.1/cockroach-debug-pebble-db-analyze-data.md index 6edcb65300c..004bc247b29 100644 --- a/src/current/v26.1/cockroach-debug-pebble-db-analyze-data.md +++ b/src/current/v26.1/cockroach-debug-pebble-db-analyze-data.md @@ -55,16 +55,16 @@ This data can help Cockroach Labs evaluate compression defaults and can help you The output CSV file is periodically rewritten while the command is running. Even if the command is interrupted, you can still use the most recently written output. -### Interpret results +### Review results with Cockroach Labs -The command groups sampled blocks into buckets and reports compression results for each bucket. Use the results to compare algorithms for the data that is actually stored in the analyzed store or backup. +The command groups sampled blocks into buckets and reports compression results for each bucket. - `CR` is the compression ratio. Higher values mean that compressed data uses less storage. - `Comp` is compression throughput. Lower throughput means higher CPU cost when Pebble writes or rewrites SSTs. - `Decomp` is decompression throughput. Lower throughput means higher CPU cost when CockroachDB reads compressed data. - `Snappy`, `MinLZ1`, `Zstd1`, `Auto1/30`, `Auto1/15`, and `Zstd3` are compression experiments. These names are not a one-to-one list of valid values for [`storage.sstable.compression_algorithm`]({% link {{ page.version.version }}/cluster-settings.md %}); use them to compare the general speed and space tradeoffs of lower-effort, adaptive, and higher-effort compression. -Most users do not need to tune SSTable compression settings. Use these results to understand whether compression tradeoffs are relevant for a representative store, and consult [Cockroach Labs Support](https://support.cockroachlabs.com/) before changing SSTable compression settings in production. For more information about Pebble SSTable compression, see [SST compression]({% link {{ page.version.version }}/architecture/storage-layer.md %}#sst-compression). +Most users do not need to tune SSTable compression settings. These results are intended to support review with Cockroach Labs rather than direct tuning decisions by end users. To understand whether compression tradeoffs are relevant for a representative store, consult [Cockroach Labs Support](https://support.cockroachlabs.com/) before changing SSTable compression settings in production. For more information about Pebble SSTable compression, see [SST compression]({% link {{ page.version.version }}/architecture/storage-layer.md %}#sst-compression). ## Examples diff --git a/src/current/v26.2/architecture/storage-layer.md b/src/current/v26.2/architecture/storage-layer.md index a2816f3328d..59a04408276 100644 --- a/src/current/v26.2/architecture/storage-layer.md +++ b/src/current/v26.2/architecture/storage-layer.md @@ -69,13 +69,13 @@ SST files are immutable; they are never modified, even during the [compaction pr ##### SST compression -Pebble compresses SSTable and blob value data to reduce physical storage use. The default profile, `fastest`, is optimized for low CPU overhead and is appropriate for most workloads. +Pebble compresses SSTable and blob value data to reduce physical storage use. The default profile, `fastest`, uses MinLZ1 on `amd64` and `arm64` platforms, is optimized for low CPU overhead, and is appropriate for most workloads. -For advanced storage tuning, CockroachDB exposes the `storage.sstable.compression_algorithm` [cluster setting]({% link {{ page.version.version }}/cluster-settings.md %}). The profile values are ordered by increasing compression effort: `fastest`, `fast`, `balanced`, and `good`. Higher-effort profiles can improve compression for some workloads, but can also increase CPU usage for writes, compactions, and reads that decompress data. Most users do not need to tune this setting. Work with [Cockroach Labs Support](https://support.cockroachlabs.com/) before changing this setting in production. +For advanced storage tuning, CockroachDB exposes the `storage.sstable.compression_algorithm` [cluster setting]({% link {{ page.version.version }}/cluster-settings.md %}). The profile values are ordered by increasing compression effort: `fastest`, `fast`, `balanced`, and `good`. These profiles enable selective use of Zstd1 depending on the block type, LSM level, and compression benefit. Higher-effort profiles use Zstd1 more frequently and can improve compression for some workloads, but can also increase CPU usage for writes, compactions, and reads that decompress data. Most users do not need to tune this setting. Work with [Cockroach Labs Support](https://support.cockroachlabs.com/) before changing this setting in production. Changing `storage.sstable.compression_algorithm` does not immediately recompress existing SST files. SSTs are immutable, so a new setting applies as Pebble writes new SSTs or rewrites existing SSTs during compaction, ingestion, restore, or other SST-writing work. During a transition, a store can contain SSTs compressed with multiple algorithms. -To evaluate the compression behavior of an existing store or backup, use [`cockroach debug pebble db analyze-data`]({% link {{ page.version.version }}/cockroach-debug-pebble-db-analyze-data.md %}). Compare the compression ratio with the compression and decompression throughput for representative data before changing the cluster setting. +To evaluate the CPU usage vs. size tradeoff for your particular data, use [`cockroach debug pebble db analyze-data`]({% link {{ page.version.version }}/cockroach-debug-pebble-db-analyze-data.md %}). Review the results with [Cockroach Labs Support](https://support.cockroachlabs.com/) before changing the cluster setting. You can monitor overall storage compression with the `storage.compression.cr` metric. diff --git a/src/current/v26.2/cockroach-debug-pebble-db-analyze-data.md b/src/current/v26.2/cockroach-debug-pebble-db-analyze-data.md index 6edcb65300c..004bc247b29 100644 --- a/src/current/v26.2/cockroach-debug-pebble-db-analyze-data.md +++ b/src/current/v26.2/cockroach-debug-pebble-db-analyze-data.md @@ -55,16 +55,16 @@ This data can help Cockroach Labs evaluate compression defaults and can help you The output CSV file is periodically rewritten while the command is running. Even if the command is interrupted, you can still use the most recently written output. -### Interpret results +### Review results with Cockroach Labs -The command groups sampled blocks into buckets and reports compression results for each bucket. Use the results to compare algorithms for the data that is actually stored in the analyzed store or backup. +The command groups sampled blocks into buckets and reports compression results for each bucket. - `CR` is the compression ratio. Higher values mean that compressed data uses less storage. - `Comp` is compression throughput. Lower throughput means higher CPU cost when Pebble writes or rewrites SSTs. - `Decomp` is decompression throughput. Lower throughput means higher CPU cost when CockroachDB reads compressed data. - `Snappy`, `MinLZ1`, `Zstd1`, `Auto1/30`, `Auto1/15`, and `Zstd3` are compression experiments. These names are not a one-to-one list of valid values for [`storage.sstable.compression_algorithm`]({% link {{ page.version.version }}/cluster-settings.md %}); use them to compare the general speed and space tradeoffs of lower-effort, adaptive, and higher-effort compression. -Most users do not need to tune SSTable compression settings. Use these results to understand whether compression tradeoffs are relevant for a representative store, and consult [Cockroach Labs Support](https://support.cockroachlabs.com/) before changing SSTable compression settings in production. For more information about Pebble SSTable compression, see [SST compression]({% link {{ page.version.version }}/architecture/storage-layer.md %}#sst-compression). +Most users do not need to tune SSTable compression settings. These results are intended to support review with Cockroach Labs rather than direct tuning decisions by end users. To understand whether compression tradeoffs are relevant for a representative store, consult [Cockroach Labs Support](https://support.cockroachlabs.com/) before changing SSTable compression settings in production. For more information about Pebble SSTable compression, see [SST compression]({% link {{ page.version.version }}/architecture/storage-layer.md %}#sst-compression). ## Examples From 7db21ec4a226ef55d16d66e3c19875ed37013e26 Mon Sep 17 00:00:00 2001 From: Rich Loveland Date: Mon, 11 May 2026 17:03:22 -0400 Subject: [PATCH 3/5] Address Pebble compression review feedback --- src/current/v26.1/cockroach-debug-pebble-db-analyze-data.md | 2 +- src/current/v26.2/cockroach-debug-pebble-db-analyze-data.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/src/current/v26.1/cockroach-debug-pebble-db-analyze-data.md b/src/current/v26.1/cockroach-debug-pebble-db-analyze-data.md index 004bc247b29..55f5701ba91 100644 --- a/src/current/v26.1/cockroach-debug-pebble-db-analyze-data.md +++ b/src/current/v26.1/cockroach-debug-pebble-db-analyze-data.md @@ -57,7 +57,7 @@ The output CSV file is periodically rewritten while the command is running. Even ### Review results with Cockroach Labs -The command groups sampled blocks into buckets and reports compression results for each bucket. +The command groups sampled blocks into buckets (by block type, block size, data compressibility) and reports compression results for each bucket. - `CR` is the compression ratio. Higher values mean that compressed data uses less storage. - `Comp` is compression throughput. Lower throughput means higher CPU cost when Pebble writes or rewrites SSTs. diff --git a/src/current/v26.2/cockroach-debug-pebble-db-analyze-data.md b/src/current/v26.2/cockroach-debug-pebble-db-analyze-data.md index 004bc247b29..55f5701ba91 100644 --- a/src/current/v26.2/cockroach-debug-pebble-db-analyze-data.md +++ b/src/current/v26.2/cockroach-debug-pebble-db-analyze-data.md @@ -57,7 +57,7 @@ The output CSV file is periodically rewritten while the command is running. Even ### Review results with Cockroach Labs -The command groups sampled blocks into buckets and reports compression results for each bucket. +The command groups sampled blocks into buckets (by block type, block size, data compressibility) and reports compression results for each bucket. - `CR` is the compression ratio. Higher values mean that compressed data uses less storage. - `Comp` is compression throughput. Lower throughput means higher CPU cost when Pebble writes or rewrites SSTs. From faba647558596d43fe7717df3bda878bac1bdd93 Mon Sep 17 00:00:00 2001 From: Rich Loveland Date: Tue, 12 May 2026 16:40:04 -0400 Subject: [PATCH 4/5] Update with taroface feedback (1) --- src/current/v26.1/architecture/storage-layer.md | 8 ++++++-- .../v26.1/cockroach-debug-pebble-db-analyze-data.md | 4 ++-- src/current/v26.2/architecture/storage-layer.md | 8 ++++++-- .../v26.2/cockroach-debug-pebble-db-analyze-data.md | 4 ++-- 4 files changed, 16 insertions(+), 8 deletions(-) diff --git a/src/current/v26.1/architecture/storage-layer.md b/src/current/v26.1/architecture/storage-layer.md index d70787691b6..5fa72c2aa92 100644 --- a/src/current/v26.1/architecture/storage-layer.md +++ b/src/current/v26.1/architecture/storage-layer.md @@ -71,13 +71,17 @@ SST files are immutable; they are never modified, even during the [compaction pr Pebble compresses SSTable and blob value data to reduce physical storage use. The default profile, `fastest`, uses MinLZ1 on `amd64` and `arm64` platforms, is optimized for low CPU overhead, and is appropriate for most workloads. -For advanced storage tuning, CockroachDB exposes the `storage.sstable.compression_algorithm` [cluster setting]({% link {{ page.version.version }}/cluster-settings.md %}). The profile values are ordered by increasing compression effort: `fastest`, `fast`, `balanced`, and `good`. These profiles enable selective use of Zstd1 depending on the block type, LSM level, and compression benefit. Higher-effort profiles use Zstd1 more frequently and can improve compression for some workloads, but can also increase CPU usage for writes, compactions, and reads that decompress data. Most users do not need to tune this setting. Work with [Cockroach Labs Support](https://support.cockroachlabs.com/) before changing this setting in production. +For advanced storage tuning, CockroachDB exposes the [`storage.sstable.compression_algorithm`]({% link {{ page.version.version }}/cluster-settings.md %}#setting-storage-sstable-compression-algorithm) cluster setting. The profile values are ordered by increasing compression effort: `fastest`, `fast`, `balanced`, and `good`. These profiles enable selective use of Zstd1 depending on the block type, LSM level, and compression benefit. Higher-effort profiles use Zstd1 more frequently and can improve compression for some workloads, but can also increase CPU usage for writes, compactions, and reads that decompress data. + +{{site.data.alerts.callout_info}} +Most users do not need to tune this setting. Work with [Cockroach Labs Support](https://support.cockroachlabs.com/) before changing this setting in production. +{{site.data.alerts.end}} Changing `storage.sstable.compression_algorithm` does not immediately recompress existing SST files. SSTs are immutable, so a new setting applies as Pebble writes new SSTs or rewrites existing SSTs during compaction, ingestion, restore, or other SST-writing work. During a transition, a store can contain SSTs compressed with multiple algorithms. To evaluate the CPU usage vs. size tradeoff for your particular data, use [`cockroach debug pebble db analyze-data`]({% link {{ page.version.version }}/cockroach-debug-pebble-db-analyze-data.md %}). Review the results with [Cockroach Labs Support](https://support.cockroachlabs.com/) before changing the cluster setting. -You can monitor overall storage compression with the `storage.compression.cr` metric. +You can monitor overall storage compression with the `storage.compression.cr` [metric]({% link cockroachcloud/metrics.md %}). ##### LSM levels diff --git a/src/current/v26.1/cockroach-debug-pebble-db-analyze-data.md b/src/current/v26.1/cockroach-debug-pebble-db-analyze-data.md index 55f5701ba91..9277df0b01a 100644 --- a/src/current/v26.1/cockroach-debug-pebble-db-analyze-data.md +++ b/src/current/v26.1/cockroach-debug-pebble-db-analyze-data.md @@ -62,9 +62,9 @@ The command groups sampled blocks into buckets (by block type, block size, data - `CR` is the compression ratio. Higher values mean that compressed data uses less storage. - `Comp` is compression throughput. Lower throughput means higher CPU cost when Pebble writes or rewrites SSTs. - `Decomp` is decompression throughput. Lower throughput means higher CPU cost when CockroachDB reads compressed data. -- `Snappy`, `MinLZ1`, `Zstd1`, `Auto1/30`, `Auto1/15`, and `Zstd3` are compression experiments. These names are not a one-to-one list of valid values for [`storage.sstable.compression_algorithm`]({% link {{ page.version.version }}/cluster-settings.md %}); use them to compare the general speed and space tradeoffs of lower-effort, adaptive, and higher-effort compression. +- `Snappy`, `MinLZ1`, `Zstd1`, `Auto1/30`, `Auto1/15`, and `Zstd3` are compression experiments. These names are not a one-to-one list of valid values for [`storage.sstable.compression_algorithm`]({% link {{ page.version.version }}/cluster-settings.md %}). Use them to compare the general speed and space tradeoffs of lower-effort, adaptive, and higher-effort compression. -Most users do not need to tune SSTable compression settings. These results are intended to support review with Cockroach Labs rather than direct tuning decisions by end users. To understand whether compression tradeoffs are relevant for a representative store, consult [Cockroach Labs Support](https://support.cockroachlabs.com/) before changing SSTable compression settings in production. For more information about Pebble SSTable compression, see [SST compression]({% link {{ page.version.version }}/architecture/storage-layer.md %}#sst-compression). +Most users do not need to tune SSTable compression settings. These results are intended to support review with Cockroach Labs rather than direct tuning decisions by end users. To understand whether compression tradeoffs are relevant for a representative store, consult [Cockroach Labs Support](https://support.cockroachlabs.com/) before changing SSTable compression settings in production. For more information about Pebble SSTable compression, refer to [SST compression]({% link {{ page.version.version }}/architecture/storage-layer.md %}#sst-compression). ## Examples diff --git a/src/current/v26.2/architecture/storage-layer.md b/src/current/v26.2/architecture/storage-layer.md index 59a04408276..28dcd2c1591 100644 --- a/src/current/v26.2/architecture/storage-layer.md +++ b/src/current/v26.2/architecture/storage-layer.md @@ -71,13 +71,17 @@ SST files are immutable; they are never modified, even during the [compaction pr Pebble compresses SSTable and blob value data to reduce physical storage use. The default profile, `fastest`, uses MinLZ1 on `amd64` and `arm64` platforms, is optimized for low CPU overhead, and is appropriate for most workloads. -For advanced storage tuning, CockroachDB exposes the `storage.sstable.compression_algorithm` [cluster setting]({% link {{ page.version.version }}/cluster-settings.md %}). The profile values are ordered by increasing compression effort: `fastest`, `fast`, `balanced`, and `good`. These profiles enable selective use of Zstd1 depending on the block type, LSM level, and compression benefit. Higher-effort profiles use Zstd1 more frequently and can improve compression for some workloads, but can also increase CPU usage for writes, compactions, and reads that decompress data. Most users do not need to tune this setting. Work with [Cockroach Labs Support](https://support.cockroachlabs.com/) before changing this setting in production. +For advanced storage tuning, CockroachDB exposes the [`storage.sstable.compression_algorithm`]({% link {{ page.version.version }}/cluster-settings.md %}#setting-storage-sstable-compression-algorithm) cluster setting. The profile values are ordered by increasing compression effort: `fastest`, `fast`, `balanced`, and `good`. These profiles enable selective use of Zstd1 depending on the block type, LSM level, and compression benefit. Higher-effort profiles use Zstd1 more frequently and can improve compression for some workloads, but can also increase CPU usage for writes, compactions, and reads that decompress data. + +{{site.data.alerts.callout_info}} +Most users do not need to tune this setting. Work with [Cockroach Labs Support](https://support.cockroachlabs.com/) before changing this setting in production. +{{site.data.alerts.end}} Changing `storage.sstable.compression_algorithm` does not immediately recompress existing SST files. SSTs are immutable, so a new setting applies as Pebble writes new SSTs or rewrites existing SSTs during compaction, ingestion, restore, or other SST-writing work. During a transition, a store can contain SSTs compressed with multiple algorithms. To evaluate the CPU usage vs. size tradeoff for your particular data, use [`cockroach debug pebble db analyze-data`]({% link {{ page.version.version }}/cockroach-debug-pebble-db-analyze-data.md %}). Review the results with [Cockroach Labs Support](https://support.cockroachlabs.com/) before changing the cluster setting. -You can monitor overall storage compression with the `storage.compression.cr` metric. +You can monitor overall storage compression with the `storage.compression.cr` [metric]({% link cockroachcloud/metrics.md %}). ##### LSM levels diff --git a/src/current/v26.2/cockroach-debug-pebble-db-analyze-data.md b/src/current/v26.2/cockroach-debug-pebble-db-analyze-data.md index 55f5701ba91..9277df0b01a 100644 --- a/src/current/v26.2/cockroach-debug-pebble-db-analyze-data.md +++ b/src/current/v26.2/cockroach-debug-pebble-db-analyze-data.md @@ -62,9 +62,9 @@ The command groups sampled blocks into buckets (by block type, block size, data - `CR` is the compression ratio. Higher values mean that compressed data uses less storage. - `Comp` is compression throughput. Lower throughput means higher CPU cost when Pebble writes or rewrites SSTs. - `Decomp` is decompression throughput. Lower throughput means higher CPU cost when CockroachDB reads compressed data. -- `Snappy`, `MinLZ1`, `Zstd1`, `Auto1/30`, `Auto1/15`, and `Zstd3` are compression experiments. These names are not a one-to-one list of valid values for [`storage.sstable.compression_algorithm`]({% link {{ page.version.version }}/cluster-settings.md %}); use them to compare the general speed and space tradeoffs of lower-effort, adaptive, and higher-effort compression. +- `Snappy`, `MinLZ1`, `Zstd1`, `Auto1/30`, `Auto1/15`, and `Zstd3` are compression experiments. These names are not a one-to-one list of valid values for [`storage.sstable.compression_algorithm`]({% link {{ page.version.version }}/cluster-settings.md %}). Use them to compare the general speed and space tradeoffs of lower-effort, adaptive, and higher-effort compression. -Most users do not need to tune SSTable compression settings. These results are intended to support review with Cockroach Labs rather than direct tuning decisions by end users. To understand whether compression tradeoffs are relevant for a representative store, consult [Cockroach Labs Support](https://support.cockroachlabs.com/) before changing SSTable compression settings in production. For more information about Pebble SSTable compression, see [SST compression]({% link {{ page.version.version }}/architecture/storage-layer.md %}#sst-compression). +Most users do not need to tune SSTable compression settings. These results are intended to support review with Cockroach Labs rather than direct tuning decisions by end users. To understand whether compression tradeoffs are relevant for a representative store, consult [Cockroach Labs Support](https://support.cockroachlabs.com/) before changing SSTable compression settings in production. For more information about Pebble SSTable compression, refer to [SST compression]({% link {{ page.version.version }}/architecture/storage-layer.md %}#sst-compression). ## Examples From ced529e12a1e567d9c0063910569968a63dd003b Mon Sep 17 00:00:00 2001 From: Rich Loveland Date: Wed, 13 May 2026 10:34:39 -0400 Subject: [PATCH 5/5] Apply Pebble compression Support wording --- src/current/v26.1/cockroach-debug-pebble-db-analyze-data.md | 4 +++- src/current/v26.2/cockroach-debug-pebble-db-analyze-data.md | 4 +++- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/src/current/v26.1/cockroach-debug-pebble-db-analyze-data.md b/src/current/v26.1/cockroach-debug-pebble-db-analyze-data.md index 9277df0b01a..fa82120db89 100644 --- a/src/current/v26.1/cockroach-debug-pebble-db-analyze-data.md +++ b/src/current/v26.1/cockroach-debug-pebble-db-analyze-data.md @@ -64,7 +64,9 @@ The command groups sampled blocks into buckets (by block type, block size, data - `Decomp` is decompression throughput. Lower throughput means higher CPU cost when CockroachDB reads compressed data. - `Snappy`, `MinLZ1`, `Zstd1`, `Auto1/30`, `Auto1/15`, and `Zstd3` are compression experiments. These names are not a one-to-one list of valid values for [`storage.sstable.compression_algorithm`]({% link {{ page.version.version }}/cluster-settings.md %}). Use them to compare the general speed and space tradeoffs of lower-effort, adaptive, and higher-effort compression. -Most users do not need to tune SSTable compression settings. These results are intended to support review with Cockroach Labs rather than direct tuning decisions by end users. To understand whether compression tradeoffs are relevant for a representative store, consult [Cockroach Labs Support](https://support.cockroachlabs.com/) before changing SSTable compression settings in production. For more information about Pebble SSTable compression, refer to [SST compression]({% link {{ page.version.version }}/architecture/storage-layer.md %}#sst-compression). +These results are intended for review with Cockroach Labs, rather than to direct your own tuning decisions. Before changing SSTable compression settings in production, contact [Cockroach Labs Support](https://support.cockroachlabs.com/) to understand whether compression tradeoffs are relevant for a representative store. Most users do not need to tune SSTable compression settings. + +For more information about Pebble SSTable compression, refer to [SST compression]({% link {{ page.version.version }}/architecture/storage-layer.md %}#sst-compression). ## Examples diff --git a/src/current/v26.2/cockroach-debug-pebble-db-analyze-data.md b/src/current/v26.2/cockroach-debug-pebble-db-analyze-data.md index 9277df0b01a..fa82120db89 100644 --- a/src/current/v26.2/cockroach-debug-pebble-db-analyze-data.md +++ b/src/current/v26.2/cockroach-debug-pebble-db-analyze-data.md @@ -64,7 +64,9 @@ The command groups sampled blocks into buckets (by block type, block size, data - `Decomp` is decompression throughput. Lower throughput means higher CPU cost when CockroachDB reads compressed data. - `Snappy`, `MinLZ1`, `Zstd1`, `Auto1/30`, `Auto1/15`, and `Zstd3` are compression experiments. These names are not a one-to-one list of valid values for [`storage.sstable.compression_algorithm`]({% link {{ page.version.version }}/cluster-settings.md %}). Use them to compare the general speed and space tradeoffs of lower-effort, adaptive, and higher-effort compression. -Most users do not need to tune SSTable compression settings. These results are intended to support review with Cockroach Labs rather than direct tuning decisions by end users. To understand whether compression tradeoffs are relevant for a representative store, consult [Cockroach Labs Support](https://support.cockroachlabs.com/) before changing SSTable compression settings in production. For more information about Pebble SSTable compression, refer to [SST compression]({% link {{ page.version.version }}/architecture/storage-layer.md %}#sst-compression). +These results are intended for review with Cockroach Labs, rather than to direct your own tuning decisions. Before changing SSTable compression settings in production, contact [Cockroach Labs Support](https://support.cockroachlabs.com/) to understand whether compression tradeoffs are relevant for a representative store. Most users do not need to tune SSTable compression settings. + +For more information about Pebble SSTable compression, refer to [SST compression]({% link {{ page.version.version }}/architecture/storage-layer.md %}#sst-compression). ## Examples