Regression: DataFusion 55 no longer prunes row groups for `col = <literal>` when statistics show the column is entirely NULL

[jensholdgaard]

Note

This report was investigated and written with AI assistance (Claude Code), posted with the account owner's review and consent.

Describe the bug

DataFusion 54 prunes a Parquet row group from statistics alone when the predicate is col = <literal> and the row group's statistics record null_count == row_count for that column (equality with a non-NULL literal cannot match any row). DataFusion 55.0.0 builds the same pruning predicate — including the <col>_null_count@N != row_count@M clause — but no longer prunes the row group; it is scanned instead.

Query results are unaffected (0 rows on both versions). The regression is scan work: for workloads where a selective equality column is sparsely populated (in our case, a body column that is NULL for the overwhelming majority of rows), row groups that 54 skipped from the footer are now read.

To Reproduce

Self-contained reproducer (~70 lines, inlined below): writes a single-row-group Parquet file with one nullable Binary column, all 100 values NULL, default WriterProperties and default SessionContext, registers it as a ListingTable, and filters with the DataFrame API:

df.filter(col("body").eq(lit(ScalarValue::Binary(Some(b"x".to_vec())))))?

Output, datafusion = "54":

rows = 0 (correct on both versions)
PruningMetrics { name: "row_groups_pruned_statistics", pruning_metrics: PruningMetrics { pruned: 1, matched: 0, fully_matched: 0 } }

Output, datafusion = "55" (only the dependency line changed):

rows = 0 (correct on both versions)
PruningMetrics { name: "row_groups_pruned_statistics", pruning_metrics: PruningMetrics { pruned: 0, matched: 1, fully_matched: 0 } }

Cargo.toml + src/main.rs (complete)

[package]
name = "df-pruning-repro"
version = "0.0.0"
edition = "2021"

[dependencies]
# Flip to "54" and the row group is pruned; on "55" it is scanned.
datafusion = "55"
tokio = { version = "1", features = ["rt-multi-thread", "macros"] }
tempfile = "3"

//! DataFusion 54 prunes a row group whose statistics say a column is
//! entirely NULL when the predicate is `col = <literal>`; DataFusion 55
//! scans it. Default writer properties, default SessionContext.
use std::sync::Arc;

use datafusion::arrow::array::{BinaryArray, RecordBatch};
use datafusion::arrow::datatypes::{DataType, Field, Schema};
use datafusion::common::ScalarValue;
use datafusion::datasource::file_format::parquet::ParquetFormat;
use datafusion::datasource::listing::{
    ListingOptions, ListingTable, ListingTableConfig, ListingTableUrl,
};
use datafusion::parquet::arrow::ArrowWriter;
use datafusion::prelude::*;

#[tokio::main]
async fn main() -> datafusion::error::Result<()> {
    // One row group, one nullable Binary column, every value NULL —
    // the footer statistics record null_count == num_rows, no min/max.
    let schema = Arc::new(Schema::new(vec![Field::new(
        "body",
        DataType::Binary,
        true,
    )]));
    let batch = RecordBatch::try_new(
        schema.clone(),
        vec![Arc::new(BinaryArray::from(vec![None::<&[u8]>; 100]))],
    )
    .unwrap();
    let dir = tempfile::tempdir().unwrap();
    let path = dir.path().join("all_null.parquet");
    let file = std::fs::File::create(&path).unwrap();
    let mut w = ArrowWriter::try_new(file, schema, None).unwrap();
    w.write(&batch).unwrap();
    w.close().unwrap();

    let ctx = SessionContext::new();
    let url = ListingTableUrl::parse(format!("file://{}", path.display())).unwrap();
    let options =
        ListingOptions::new(Arc::new(ParquetFormat::default())).with_file_extension(".parquet");
    let schema = options.infer_schema(&ctx.state(), &url).await?;
    let table = ListingTable::try_new(
        ListingTableConfig::new(url)
            .with_listing_options(options)
            .with_schema(schema),
    )?;
    // `body = X` can match nothing when every value is NULL, so the row
    // group is prunable from statistics alone.
    let df = ctx
        .read_table(Arc::new(table))?
        .filter(col("body").eq(lit(ScalarValue::Binary(Some(b"x".to_vec())))))?;

    let plan = df.create_physical_plan().await?;
    let batches = datafusion::physical_plan::collect(plan.clone(), ctx.task_ctx()).await?;
    println!(
        "rows = {} (correct on both versions)",
        batches.iter().map(|b| b.num_rows()).sum::<usize>()
    );

    fn walk(p: &Arc<dyn datafusion::physical_plan::ExecutionPlan>) {
        if let Some(m) = p.metrics() {
            for metric in m.iter() {
                if metric.value().name() == "row_groups_pruned_statistics" {
                    println!("{:?}", metric.value());
                }
            }
        }
        for c in p.children() {
            walk(c);
        }
    }
    walk(&plan);
    Ok(())
}

Expected behavior

The row group is pruned on 55 as it was on 54: the file's statistics prove body = 'x' cannot match (null_count == row_count), and the physical plan's pruning_predicate (identical on both versions in our larger application, including the body_null_count != row_count conjunct) already expresses that proof.

Additional context

• Found upgrading a Parquet log store (ourios) from DF 54.0.0 → 55.0.0: three pruning-assertion tests went red with pruned: 0 where 54 gave pruned: 2; written files byte-identical across the upgrade (parquet 58 vs 59 emit the same statistics for these columns), so this is read-path only.
• Triage hint: the regression reproduces through ListingTable + the DataFrame API filter. In our first attempt we could NOT reproduce via register_parquet + a SQL string (WHERE body = X'6E6F7065') — that path does not prune on either 54 or 55 — so the SQL literal/rewrite path seems to sit on a different guarantee/pruning route and may mask the regression during triage.
• EnabledStatistics::Page vs Chunk and dictionary on/off for the column make no difference; defaults reproduce.

[jensholdgaard]

Proposed fix: #24770.

Root cause turned out to be an interaction rather than a pruning bug. constant_columns_from_stats folds an all-NULL column to a NULL literal; the predicate then simplifies to a bare constant, at which point build_pruning_predicates returns None (no column references remain) and prune_row_groups falls through with no pruning. So for exactly the files whose statistics carry the most information, the substitution is strictly counterproductive — before it, the pruning predicate's own col_null_count != row_count conjunct proved the row group empty.

Bisected to #22969 (removing ListingOptions::collect_stat in favour of execution.collect_statistics, default true) — correct in itself, it just began feeding the substitution on paths that previously had no file statistics. Confirming detail: setting execution.collect_statistics = false restores pruning on 55.

The PR skips the remaining row groups when the collapsed predicate is false/NULL, with a regression test that fails on current main. One correction to the report above: the standalone reproducer uses the DataFrame API, and I noted the SQL path did not reproduce — that is because the SQL literal path did not trigger the substitution in my attempt, not because pruning differs there.