Skip to content

Int96Coercer drops the metadata of struct, list and map fields. #24786

Description

@ryzhyk

Reading a Parquet file with datafusion.execution.parquet.coerce_int96 set is supposed to only change the time unit that INT96 timestamp columns decode at. However it also empties the metadata of every struct, list and map field in the file.

Cause

In datafusion/datasource-parquet/src/schema_coercion.rs, leaf fields are cloned:

// :423
fn field_with_new_type(field: &FieldRef, new_type: DataType) -> FieldRef {
    Arc::new(field.as_ref().clone().with_data_type(new_type))
}

while container fields are constructed fresh, and Field::new* starts from empty metadata:

// :327
let processed_struct = Field::new_struct(
    current_field.name(),
    processed_children.as_slice(),
    current_field.is_nullable(),
);
// :360
let processed_list = Field::new_list(
    current_field.name(),
    Arc::clone(&processed_children[0]),
    current_field.is_nullable(),
);
// :392
DataType::Map(Arc::clone(&processed_children[0]), *sorted),

Name, type and nullability are carried across; current_field.metadata() is never read.

A test that demonstrates the issue: https://github.com/ryzhyk/coerce_int96_bug/tree/main

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions