fix[sparse]: ensure cached patches are updated and use PatchesData#7811
fix[sparse]: ensure cached patches are updated and use PatchesData#7811joseph-isaacs merged 4 commits intodevelopfrom
PatchesData#7811Conversation
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
PatchesDataPatchesData
Merging this PR will degrade performance by 23.4%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ⚡ | Simulation | take_10k_random |
239.7 µs | 194.9 µs | +22.97% |
| ❌ | Simulation | new_alp_prim_test_between[f32, 32768] |
155.3 µs | 184.3 µs | -15.72% |
| ❌ | Simulation | new_alp_prim_test_between[f64, 16384] |
126.4 µs | 155.9 µs | -18.96% |
| ⚡ | Simulation | new_bp_prim_test_between[i64, 32768] |
236.6 µs | 177.7 µs | +33.13% |
| ❌ | Simulation | new_alp_prim_test_between[f64, 32768] |
192.5 µs | 251.3 µs | -23.4% |
| ⚡ | Simulation | patched_take_10k_contiguous_patches |
258.9 µs | 228.6 µs | +13.24% |
| ⚡ | Simulation | patched_take_10k_random |
271.3 µs | 241 µs | +12.57% |
| ⚡ | Simulation | take_10k_contiguous |
315.3 µs | 270.8 µs | +16.41% |
| ⚡ | Simulation | decompress_rd[f32, (10000, 0.01)] |
91.1 µs | 82.6 µs | +10.29% |
| ⚡ | Simulation | decompress_rd[f64, (10000, 0.0)] |
139.4 µs | 125.3 µs | +11.22% |
| ⚡ | Simulation | decompress_rd[f64, (100000, 0.0)] |
1,021.9 µs | 801.2 µs | +27.55% |
| ⚡ | Simulation | decompress_rd[f32, (10000, 0.0)] |
91.8 µs | 83.2 µs | +10.35% |
| ⚡ | Simulation | decompress_rd[f32, (10000, 0.1)] |
91.1 µs | 82.6 µs | +10.33% |
| ⚡ | Simulation | decompress_rd[f64, (10000, 0.01)] |
139.5 µs | 125.3 µs | +11.27% |
| ⚡ | Simulation | decompress_rd[f32, (100000, 0.0)] |
584.3 µs | 454.2 µs | +28.63% |
| ❌ | Simulation | decompress_rd[f64, (100000, 0.1)] |
843.3 µs | 978.9 µs | -13.85% |
| ⚡ | Simulation | decompress_rd[f64, (10000, 0.1)] |
139.3 µs | 121.9 µs | +14.26% |
| ❌ | Simulation | decompress_rd[f64, (100000, 0.01)] |
843.3 µs | 978.8 µs | -13.85% |
Comparing ji/sparse-array-patches (edd2c4a) with develop (a2edb2f)1
Footnotes
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals ProfilingVortex (geomean): 0.991x ➖ datafusion / vortex-file-compressed (0.991x ➖, 0↑ 0↓)
|
File Sizes: PolarSignals ProfilingNo file size changes detected. |
Benchmarks: FineWeb NVMeVerdict: No clear signal (low confidence) datafusion / vortex-file-compressed (1.004x ➖, 1↑ 2↓)
datafusion / vortex-compact (1.002x ➖, 0↑ 0↓)
datafusion / parquet (0.985x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed (1.004x ➖, 0↑ 1↓)
duckdb / vortex-compact (0.993x ➖, 0↑ 0↓)
duckdb / parquet (1.009x ➖, 0↑ 1↓)
Full attributed analysis
|
File Sizes: FineWeb NVMeNo file size changes detected. |
Benchmarks: TPC-H SF=1 on NVMEVerdict: No clear signal (environment too noisy confidence) datafusion / vortex-file-compressed (0.856x ✅, 20↑ 0↓)
datafusion / vortex-compact (0.886x ✅, 18↑ 0↓)
datafusion / parquet (0.857x ✅, 18↑ 0↓)
datafusion / arrow (0.852x ✅, 16↑ 0↓)
duckdb / vortex-file-compressed (0.841x ✅, 21↑ 0↓)
duckdb / vortex-compact (0.879x ✅, 15↑ 0↓)
duckdb / parquet (0.916x ➖, 7↑ 0↓)
duckdb / duckdb (0.875x ✅, 13↑ 0↓)
Full attributed analysis
|
File Sizes: TPC-H SF=1 on NVMEFile Size Changes (195 files changed, -98.4% overall, 0↑ 195↓)
Totals:
|
Benchmarks: TPC-DS SF=1 on NVMEVerdict: No clear signal (low confidence) datafusion / vortex-file-compressed (0.964x ➖, 5↑ 1↓)
datafusion / vortex-compact (0.970x ➖, 2↑ 0↓)
datafusion / parquet (0.979x ➖, 1↑ 2↓)
duckdb / vortex-file-compressed (0.985x ➖, 3↑ 1↓)
duckdb / vortex-compact (0.971x ➖, 5↑ 0↓)
duckdb / parquet (0.990x ➖, 3↑ 2↓)
duckdb / duckdb (0.952x ➖, 7↑ 0↓)
Full attributed analysis
|
File Sizes: TPC-DS SF=1 on NVMENo file size changes detected. |
Benchmarks: FineWeb S3Verdict: No clear signal (environment too noisy confidence) datafusion / vortex-file-compressed (0.963x ➖, 0↑ 0↓)
datafusion / vortex-compact (1.039x ➖, 0↑ 0↓)
datafusion / parquet (0.961x ➖, 1↑ 0↓)
duckdb / vortex-file-compressed (0.972x ➖, 0↑ 0↓)
duckdb / vortex-compact (1.002x ➖, 0↑ 0↓)
duckdb / parquet (0.995x ➖, 0↑ 0↓)
Full attributed analysis
|
Benchmarks: Random AccessVortex (geomean): 0.959x ➖ unknown / unknown (0.967x ➖, 5↑ 0↓)
|
Benchmarks: Statistical and Population GeneticsVerdict: No clear signal (low confidence) duckdb / vortex-file-compressed (1.026x ➖, 0↑ 1↓)
duckdb / vortex-compact (0.988x ➖, 0↑ 0↓)
duckdb / parquet (1.008x ➖, 0↑ 0↓)
Full attributed analysis
|
File Sizes: Statistical and Population GeneticsNo file size changes detected. |
Benchmarks: TPC-H SF=10 on NVMEVerdict: No clear signal (low confidence) datafusion / vortex-file-compressed (0.923x ➖, 5↑ 0↓)
datafusion / vortex-compact (0.856x ✅, 20↑ 0↓)
datafusion / parquet (0.979x ➖, 0↑ 0↓)
datafusion / arrow (0.929x ➖, 8↑ 0↓)
duckdb / vortex-file-compressed (0.887x ✅, 16↑ 0↓)
duckdb / vortex-compact (0.906x ➖, 11↑ 0↓)
duckdb / parquet (0.943x ➖, 1↑ 0↓)
duckdb / duckdb (0.930x ➖, 3↑ 0↓)
Full attributed analysis
|
File Sizes: TPC-H SF=10 on NVMENo file size changes detected. |
Benchmarks: TPC-H SF=1 on S3Verdict: No clear signal (environment too noisy confidence) datafusion / vortex-file-compressed (0.915x ➖, 1↑ 0↓)
datafusion / vortex-compact (0.873x ➖, 4↑ 1↓)
datafusion / parquet (0.954x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed (0.979x ➖, 0↑ 0↓)
duckdb / vortex-compact (0.958x ➖, 0↑ 0↓)
duckdb / parquet (0.986x ➖, 0↑ 0↓)
Full attributed analysis
|
Benchmarks: Clickbench on NVMEVerdict: No clear signal (low confidence) datafusion / vortex-file-compressed (0.948x ➖, 0↑ 0↓)
datafusion / parquet (0.941x ➖, 3↑ 0↓)
duckdb / vortex-file-compressed (0.948x ➖, 4↑ 0↓)
duckdb / parquet (0.966x ➖, 0↑ 0↓)
duckdb / duckdb (0.975x ➖, 2↑ 1↓)
Full attributed analysis
|
File Sizes: Clickbench on NVMEFile Size Changes (1 files changed, -0.0% overall, 0↑ 1↓)
Totals:
|
Benchmarks: CompressionVortex (geomean): 1.003x ➖ unknown / unknown (1.003x ➖, 2↑ 3↓)
|
Benchmarks: TPC-H SF=10 on S3Verdict: No clear signal (environment too noisy confidence) datafusion / vortex-file-compressed (1.004x ➖, 1↑ 1↓)
datafusion / vortex-compact (0.979x ➖, 0↑ 0↓)
datafusion / parquet (0.975x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed (0.901x ➖, 0↑ 0↓)
duckdb / vortex-compact (0.952x ➖, 0↑ 0↓)
duckdb / parquet (0.938x ➖, 0↑ 0↓)
Full attributed analysis
|
Sparse array help on to a cached via over the slots that form a Patch, make this read only.
Include
PatchesDatato hold the non children/slot part of Patches.