Skip to content

[SPARK-56802][SQL] Add bulk read+widen path for FLOAT to Double Parquet vector updater #55816

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
LuciferYang wants to merge 4 commits into from
Closed

[SPARK-56802][SQL] Add bulk read+widen path for FLOAT to Double Parquet vector updater #55816

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
051b94b
[SPARK-56802][SQL] Add bulk read+widen path for FLOAT to Double Parqu…
LuciferYang May 12, 2026
fcf4b24
Benchmark results for org.apache.spark.sql.execution.datasources.parq…
LuciferYang May 12, 2026
68fc576
Benchmark results for org.apache.spark.sql.execution.datasources.parq…
LuciferYang May 12, 2026
7b05162
Benchmark results for org.apache.spark.sql.execution.datasources.parq…
LuciferYang May 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 23 additions & 23 deletions sql/core/benchmarks/ParquetVectorUpdaterBenchmark-jdk21-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,14 +6,14 @@ OpenJDK 64-Bit Server VM 21.0.11+10-LTS on Linux 6.17.0-1010-azure
AMD EPYC 7763 64-Core Processor
Identity Updaters: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
BooleanUpdater 0 0 0 16954.6 0.1 1.0X
ByteUpdater (INT32 -> Byte) 0 0 0 3764.8 0.3 0.2X
ShortUpdater (INT32 -> Short) 1 1 0 1677.4 0.6 0.1X
IntegerUpdater 0 0 0 10269.9 0.1 0.6X
LongUpdater 0 0 0 5120.9 0.2 0.3X
FloatUpdater 0 0 0 10252.8 0.1 0.6X
DoubleUpdater 0 0 0 5133.4 0.2 0.3X
BinaryUpdater 15 15 3 71.0 14.1 0.0X
BooleanUpdater 0 0 0 17004.4 0.1 1.0X
ByteUpdater (INT32 -> Byte) 0 0 0 3758.6 0.3 0.2X
ShortUpdater (INT32 -> Short) 1 1 0 1678.4 0.6 0.1X
IntegerUpdater 0 0 0 10276.1 0.1 0.6X
LongUpdater 0 0 0 5112.7 0.2 0.3X
FloatUpdater 0 0 0 10293.3 0.1 0.6X
DoubleUpdater 0 0 0 3870.8 0.3 0.2X
BinaryUpdater 15 15 1 71.3 14.0 0.0X


================================================================================================
Expand All @@ -24,11 +24,11 @@ OpenJDK 64-Bit Server VM 21.0.11+10-LTS on Linux 6.17.0-1010-azure
AMD EPYC 7763 64-Core Processor
Type-converting Updaters: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
IntegerToLongUpdater 0 0 0 6220.9 0.2 1.0X
IntegerToDoubleUpdater 0 0 0 6215.0 0.2 1.0X
FloatToDoubleUpdater 2 2 0 489.5 2.0 0.1X
DateToTimestampNTZUpdater 29 29 1 36.1 27.7 0.0X
DowncastLongUpdater (INT64 -> Decimal(9,2)) 2 2 0 455.2 2.2 0.1X
IntegerToLongUpdater 0 0 0 5145.1 0.2 1.0X
IntegerToDoubleUpdater 0 0 0 5120.5 0.2 1.0X
FloatToDoubleUpdater 0 0 0 2527.2 0.4 0.5X
DateToTimestampNTZUpdater 29 29 0 36.3 27.5 0.0X
DowncastLongUpdater (INT64 -> Decimal(9,2)) 2 2 0 455.6 2.2 0.1X


================================================================================================
Expand All @@ -39,9 +39,9 @@ OpenJDK 64-Bit Server VM 21.0.11+10-LTS on Linux 6.17.0-1010-azure
AMD EPYC 7763 64-Core Processor
Rebase Updaters: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------------------------------------
IntegerWithRebaseUpdater (DATE legacy) 0 0 0 3645.1 0.3 1.0X
LongWithRebaseUpdater (TIMESTAMP_MICROS legacy) 0 0 0 2663.5 0.4 0.7X
LongAsMicrosUpdater (TIMESTAMP_MILLIS) 2 3 0 419.9 2.4 0.1X
IntegerWithRebaseUpdater (DATE legacy) 0 0 0 3647.4 0.3 1.0X
LongWithRebaseUpdater (TIMESTAMP_MICROS legacy) 0 0 0 2668.6 0.4 0.7X
LongAsMicrosUpdater (TIMESTAMP_MILLIS) 2 3 0 420.3 2.4 0.1X


================================================================================================
Expand All @@ -52,8 +52,8 @@ OpenJDK 64-Bit Server VM 21.0.11+10-LTS on Linux 6.17.0-1010-azure
AMD EPYC 7763 64-Core Processor
Unsigned Updaters: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-----------------------------------------------------------------------------------------------------------------------------
UnsignedIntegerUpdater (UINT32 -> Long) 0 0 0 5846.6 0.2 1.0X
UnsignedLongUpdater (UINT64 -> Decimal(20,0)) 17 18 0 60.3 16.6 0.0X
UnsignedIntegerUpdater (UINT32 -> Long) 0 0 0 5089.6 0.2 1.0X
UnsignedLongUpdater (UINT64 -> Decimal(20,0)) 17 18 1 60.2 16.6 0.0X


================================================================================================
Expand All @@ -64,9 +64,9 @@ OpenJDK 64-Bit Server VM 21.0.11+10-LTS on Linux 6.17.0-1010-azure
AMD EPYC 7763 64-Core Processor
Decimal Updaters: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
IntegerToDecimalUpdater 0 0 0 10296.2 0.1 1.0X
LongToDecimalUpdater 0 0 0 5145.8 0.2 0.5X
FixedLenByteArrayToDecimalUpdater 21 21 1 50.2 19.9 0.0X
IntegerToDecimalUpdater 0 0 0 7746.5 0.1 1.0X
LongToDecimalUpdater 0 0 0 3874.1 0.3 0.5X
FixedLenByteArrayToDecimalUpdater 21 21 0 50.7 19.7 0.0X


================================================================================================
Expand All @@ -77,8 +77,8 @@ OpenJDK 64-Bit Server VM 21.0.11+10-LTS on Linux 6.17.0-1010-azure
AMD EPYC 7763 64-Core Processor
FixedLenByteArray Updaters: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------------------
FixedLenByteArrayUpdater (len=16 -> Binary) 20 21 1 52.7 19.0 1.0X
FixedLenByteArrayUpdater (len=16 -> Binary) 19 20 2 54.1 18.5 1.0X
FixedLenByteArrayAsIntUpdater (len=4 -> Decimal(9,2)) 7 7 0 160.1 6.2 3.0X
FixedLenByteArrayAsLongUpdater (len=8 -> Decimal(18,4)) 8 8 0 132.7 7.5 2.5X
FixedLenByteArrayAsLongUpdater (len=8 -> Decimal(18,4)) 8 8 0 133.2 7.5 2.5X


46 changes: 23 additions & 23 deletions sql/core/benchmarks/ParquetVectorUpdaterBenchmark-jdk25-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,14 +6,14 @@ OpenJDK 64-Bit Server VM 25.0.3+9-LTS on Linux 6.17.0-1010-azure
AMD EPYC 7763 64-Core Processor
Identity Updaters: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
BooleanUpdater 0 0 0 17171.8 0.1 1.0X
ByteUpdater (INT32 -> Byte) 0 0 0 3675.3 0.3 0.2X
ShortUpdater (INT32 -> Short) 1 1 0 1661.2 0.6 0.1X
IntegerUpdater 0 0 0 10301.4 0.1 0.6X
LongUpdater 0 0 0 5147.9 0.2 0.3X
FloatUpdater 0 0 0 10294.3 0.1 0.6X
DoubleUpdater 0 0 0 5113.4 0.2 0.3X
BinaryUpdater 16 16 1 66.9 14.9 0.0X
BooleanUpdater 0 0 0 17138.1 0.1 1.0X
ByteUpdater (INT32 -> Byte) 0 0 0 3678.2 0.3 0.2X
ShortUpdater (INT32 -> Short) 1 1 0 1662.8 0.6 0.1X
IntegerUpdater 0 0 0 10231.8 0.1 0.6X
LongUpdater 0 0 0 5103.2 0.2 0.3X
FloatUpdater 0 0 0 10203.9 0.1 0.6X
DoubleUpdater 0 0 0 5105.4 0.2 0.3X
BinaryUpdater 17 17 0 62.8 15.9 0.0X


================================================================================================
Expand All @@ -24,11 +24,11 @@ OpenJDK 64-Bit Server VM 25.0.3+9-LTS on Linux 6.17.0-1010-azure
AMD EPYC 7763 64-Core Processor
Type-converting Updaters: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------
IntegerToLongUpdater 0 0 0 6351.2 0.2 1.0X
IntegerToDoubleUpdater 0 0 0 6423.7 0.2 1.0X
FloatToDoubleUpdater 2 2 0 483.3 2.1 0.1X
DateToTimestampNTZUpdater 29 29 1 36.6 27.3 0.0X
DowncastLongUpdater (INT64 -> Decimal(9,2)) 2 2 0 455.4 2.2 0.1X
IntegerToLongUpdater 0 0 0 6239.9 0.2 1.0X
IntegerToDoubleUpdater 0 0 0 6382.6 0.2 1.0X
FloatToDoubleUpdater 0 0 0 2570.7 0.4 0.4X
DateToTimestampNTZUpdater 26 26 0 40.5 24.7 0.0X
DowncastLongUpdater (INT64 -> Decimal(9,2)) 2 2 0 637.7 1.6 0.1X


================================================================================================
Expand All @@ -39,9 +39,9 @@ OpenJDK 64-Bit Server VM 25.0.3+9-LTS on Linux 6.17.0-1010-azure
AMD EPYC 7763 64-Core Processor
Rebase Updaters: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------------------------------------------
IntegerWithRebaseUpdater (DATE legacy) 0 0 0 3653.2 0.3 1.0X
LongWithRebaseUpdater (TIMESTAMP_MICROS legacy) 0 0 0 2659.6 0.4 0.7X
LongAsMicrosUpdater (TIMESTAMP_MILLIS) 3 3 0 371.1 2.7 0.1X
IntegerWithRebaseUpdater (DATE legacy) 0 0 0 3660.5 0.3 1.0X
LongWithRebaseUpdater (TIMESTAMP_MICROS legacy) 0 0 0 2663.6 0.4 0.7X
LongAsMicrosUpdater (TIMESTAMP_MILLIS) 2 2 0 521.3 1.9 0.1X


================================================================================================
Expand All @@ -52,8 +52,8 @@ OpenJDK 64-Bit Server VM 25.0.3+9-LTS on Linux 6.17.0-1010-azure
AMD EPYC 7763 64-Core Processor
Unsigned Updaters: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-----------------------------------------------------------------------------------------------------------------------------
UnsignedIntegerUpdater (UINT32 -> Long) 0 0 0 6324.8 0.2 1.0X
UnsignedLongUpdater (UINT64 -> Decimal(20,0)) 17 17 0 60.4 16.5 0.0X
UnsignedIntegerUpdater (UINT32 -> Long) 0 0 0 6196.3 0.2 1.0X
UnsignedLongUpdater (UINT64 -> Decimal(20,0)) 17 18 1 60.3 16.6 0.0X


================================================================================================
Expand All @@ -64,8 +64,8 @@ OpenJDK 64-Bit Server VM 25.0.3+9-LTS on Linux 6.17.0-1010-azure
AMD EPYC 7763 64-Core Processor
Decimal Updaters: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
IntegerToDecimalUpdater 0 0 0 10258.9 0.1 1.0X
LongToDecimalUpdater 0 0 0 5144.1 0.2 0.5X
IntegerToDecimalUpdater 0 0 0 10247.8 0.1 1.0X
LongToDecimalUpdater 0 0 0 5125.7 0.2 0.5X
FixedLenByteArrayToDecimalUpdater 21 21 0 50.7 19.7 0.0X


Expand All @@ -77,8 +77,8 @@ OpenJDK 64-Bit Server VM 25.0.3+9-LTS on Linux 6.17.0-1010-azure
AMD EPYC 7763 64-Core Processor
FixedLenByteArray Updaters: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
---------------------------------------------------------------------------------------------------------------------------------------
FixedLenByteArrayUpdater (len=16 -> Binary) 21 22 1 50.2 19.9 1.0X
FixedLenByteArrayAsIntUpdater (len=4 -> Decimal(9,2)) 7 7 0 152.5 6.6 3.0X
FixedLenByteArrayAsLongUpdater (len=8 -> Decimal(18,4)) 8 8 0 127.5 7.8 2.5X
FixedLenByteArrayUpdater (len=16 -> Binary) 21 21 1 51.0 19.6 1.0X
FixedLenByteArrayAsIntUpdater (len=4 -> Decimal(9,2)) 7 7 0 152.6 6.6 3.0X
FixedLenByteArrayAsLongUpdater (len=8 -> Decimal(18,4)) 8 8 0 127.7 7.8 2.5X


Loading