[Spark] REPLACE PARTITIONED BY: rewrite data files that do not store partition columns

[sezruby]

Stacked on #17 (ALTER TABLE ... REPLACE PARTITIONED BY WITH CLUSTER BY). Review that first; this PR only contains the follow-up commit.

Which Delta project/connector is this regarding?

• Spark

Description

#17 makes dropping the partitioning a metadata-only operation, but it requires every data file to already store the partition column values in Parquet (e.g. written with delta.writePartitionColumnsToParquet = true). By default, Delta does not write partition values into data files, so existing tables usually fail that check.

With this change, the command rewrites only the files that are missing the partition columns, in the same commit as the metadata change:

• Files that already contain the partition columns are re-added with partitionValues = {} (no rewrite), as in [Spark] Support ALTER TABLE ... REPLACE PARTITIONED BY WITH CLUSTER BY #17.
• Files that don't are read and rewritten to new files that include all columns. The old files are removed. All actions use dataChange = false.
• The rewrite follows the OPTIMIZE approach:
  • deletion vectors are applied, so rewritten files have no DV;
  • materialized row IDs and row commit versions are preserved when row tracking is enabled;
  • optimized write and auto compaction are not triggered.
• New operation metrics: numRewrittenFiles, numRewrittenBytes, numAddedFilesFromRewrite.
• New internal conf spark.databricks.delta.alterTable.replacePartitionedBy.rewriteNonMaterializedFiles (default true). When set to false, the command keeps the [Spark] Support ALTER TABLE ... REPLACE PARTITIONED BY WITH CLUSTER BY #17 behavior: it fails with DELTA_REPLACE_PARTITIONED_BY_PARTITION_COLUMNS_NOT_MATERIALIZED. The error message now names this conf.

As a result, the cost of dropping the partitioning is proportional to the data that lacks partition values. Tables that already materialize them stay metadata-only.

How was this patch tested?

ReplacePartitionedByWithClusterBySuite:

• Mixed table: only the non-materialized files are rewritten. Materialized files keep their paths. Removes and adds use dataChange = false. Metrics are reported. Results, including null partition values and filters, are unchanged.
• DV + row tracking: rewritten files have no DV, deleted rows stay deleted, and _metadata.row_id / row_commit_version are preserved.
• Rewrite disabled: the error is raised, it includes the conf name, and the table is unchanged.
• The existing metadata-only test now also asserts there are no RemoveFiles and numRewrittenFiles = 0.

Also ran DeltaErrorsSuite, DeltaThrowableSuite and scalastyle.

Does this PR introduce any user-facing changes?

Yes. ALTER TABLE ... REPLACE PARTITIONED BY no longer fails on tables whose data files don't store partition values; it rewrites those files instead. The previous behavior is available via the conf above.