Reading a Parquet file with datafusion.execution.parquet.coerce_int96 set is supposed to only change the time unit that INT96 timestamp columns decode at. However it also empties the metadata of every struct, list and map field in the file.
Cause
In datafusion/datasource-parquet/src/schema_coercion.rs, leaf fields are cloned:
// :423
fn field_with_new_type(field: &FieldRef, new_type: DataType) -> FieldRef {
Arc::new(field.as_ref().clone().with_data_type(new_type))
}
while container fields are constructed fresh, and Field::new* starts from empty metadata:
// :327
let processed_struct = Field::new_struct(
current_field.name(),
processed_children.as_slice(),
current_field.is_nullable(),
);
// :360
let processed_list = Field::new_list(
current_field.name(),
Arc::clone(&processed_children[0]),
current_field.is_nullable(),
);
// :392
DataType::Map(Arc::clone(&processed_children[0]), *sorted),
Name, type and nullability are carried across; current_field.metadata() is never read.
A test that demonstrates the issue: https://github.com/ryzhyk/coerce_int96_bug/tree/main
Reading a Parquet file with
datafusion.execution.parquet.coerce_int96set is supposed to only change the time unit that INT96 timestamp columns decode at. However it also empties the metadata of every struct, list and map field in the file.Cause
In
datafusion/datasource-parquet/src/schema_coercion.rs, leaf fields are cloned:while container fields are constructed fresh, and
Field::new*starts from empty metadata:Name, type and nullability are carried across;
current_field.metadata()is never read.A test that demonstrates the issue: https://github.com/ryzhyk/coerce_int96_bug/tree/main