Is your feature request related to a problem or challenge?
octet_length only accepts string types. When we use a binary column, the plan fails:
Function 'octet_length' requires String, but received Binary (DataType: BinaryView)
A workaround for SUM(octet_length(col)) over binary is:
SUM(length(encode(col, 'hex')) / 2)
But this materializes 2x the column's bytes just to count it.
Describe the solution you'd like
Extend octet_length to natively accept Binary, LargeBinary, and BinaryView, returning the byte length.
- SQL:1999 defines
OCTET_LENGTH for string values, including binary strings.
- PostgreSQL, DuckDB, Spark, MySQL, and SQLite accept binary values.
utf8_to_int_type already maps Binary -> Int32, LargeBinary -> Int64, and BinaryView -> Int32.
We've implemented the solution internally and happy to submit PR.
datafusion/functions/src/string/octet_length.rs
Before:
impl OctetLengthFunc {
pub fn new() -> Self {
Self {
signature: Signature::coercible(
vec![
Coercion::new_exact(TypeSignatureClass::Native(logical_string()))
.with_encoding_preservation(EncodingPreservation::dictionary()),
],
Volatility::Immutable,
),
}
}
}
After: the two-branch string|binary select that md5 uses:
impl OctetLengthFunc {
pub fn new() -> Self {
Self {
signature: Signature::one_of(
vec![
TypeSignature::Coercible(vec![
Coercion::new_exact(TypeSignatureClass::Native(logical_string()))
.with_encoding_preservation(
EncodingPreservation::dictionary(),
),
]),
TypeSignature::Coercible(vec![
Coercion::new_exact(TypeSignatureClass::Native(logical_binary()))
.with_encoding_preservation(
EncodingPreservation::dictionary(),
),
]),
],
Volatility::Immutable,
),
}
}
}
Describe alternatives you've considered
length(encode(col, 'hex')) / 2 // works but doubles the data to count it
Additional context
bit_length has the same gap and could be part in the same PR.
Is your feature request related to a problem or challenge?
octet_lengthonly accepts string types. When we use a binary column, the plan fails:A workaround for
SUM(octet_length(col))over binary is:But this materializes 2x the column's bytes just to count it.
Describe the solution you'd like
Extend
octet_lengthto natively acceptBinary,LargeBinary, andBinaryView, returning the byte length.OCTET_LENGTHfor string values, including binary strings.utf8_to_int_typealready mapsBinary -> Int32,LargeBinary -> Int64, andBinaryView -> Int32.We've implemented the solution internally and happy to submit PR.
datafusion/functions/src/string/octet_length.rsBefore:
After: the two-branch string|binary select that
md5uses:Describe alternatives you've considered
Additional context
bit_lengthhas the same gap and could be part in the same PR.