[RLC] Case 2 — same-file deletion-vector merge (concurrent DML ⟂ DML)

[sezruby]

Part of the row-level-concurrency umbrella (#3) — Case 2. Full design: #5. Addresses delta-io#7057.

Summary

POC implementing deletion-vector-based row-level concurrency for OSS Delta (Spark). Two concurrent DML operations that modify the same physical file but touch disjoint rows no longer abort the loser. A new config-gated phase in ConflictChecker decodes both transactions' deletion vectors, and when the newly-deleted rows are disjoint, merges them (dv_win ∪ dv_cur), writes a new DV file, and rebases the losing transaction onto the winner's post-image.

This is the sound same-file DV union. It is deliberately narrow: it reconciles conflicts that are provable from DV positions alone, and falls back to today's abort for everything else — including a rewrite-only DML winner's new image files, which are handed to conflict-time data skipping (Case 1, #4 / #8). See Correctness: UPDATE image files below.

What it does

• New phase resolveRowLevelConflicts() in ConflictChecker, run before the file-level checks.
• For a file both transactions modified with a DV, decode dv_win, dv_cur, and their common base dv_base (from the current txn's pre-image RemoveFile).
• Disjointness rule: (dv_win ∩ dv_cur) \ dv_base empty → merge the DVs, persist a new DV file, and rebase the current AddFile(P) onto the winner's post-image (its RemoveFile(P) now tombstones the winner's AddFile(P)). Non-empty → genuine same-row conflict, aborts as before.
• The delete/read, delete/delete, and append checks skip the resolved same-file paths (rowLevelResolvedPaths) — re-checking the winner's re-added AddFile(P) (whose DV we already folded in) would be a false conflict.
• DV-only — row tracking is not required. Physical DV positions index one immutable Parquet file, so they are version-independent; when row tracking is enabled, the existing reassignOverlappingRowIds phase (already run) keeps IDs consistent since the merged file is the same physical file with an unchanged base row ID.
• Gated by spark.databricks.delta.rowLevelConcurrency.enabled (internal, default off). No protocol / action / reader changes — the output is a standard DV action.

Correctness: UPDATE image files (deferred to Case 1 / #4)

An UPDATE produces two things per touched file: the same-path DV pair (handled here, soundly) and a new image file at a fresh path carrying the updated row values. This PR does not skip those image files in the append check. That is intentional:

An UPDATE can move a row into the loser's predicate — winner UPDATE SET x = 15 WHERE id = r, loser DELETE WHERE x > 10, where r started at x = 5. Under a winner→loser serial order the loser would delete r, so this is a genuine write-skew conflict. The DV union cannot detect it (r is a different row from the ones the loser touched), so any op-agnostic "suppress a rewrite-only DML winner's added files" shortcut would be unsound — it would let the loser commit with r surviving at x = 15, unmatched by its own predicate, strictly weaker than base WriteSerializable.

Instead, image files remain ordinary non-blind changed-data files and flow into the standard added-files check. On an unpartitioned table that conservatively conflicts (one-way safe: falls back to today's abort). Reconciling the provably-disjoint case — the loser's read predicate vs the image file's stats — is exactly conflict-time data skipping, owned by Case 1 (#4 / #8). Case 2's UPDATE image-file reconcile therefore depends on #8 being in the build; standalone, Case 2 is DV-union only.

Scope

• In: DELETE-vs-DELETE (fully reconciled), and the same-file DV union for UPDATE/DELETE mixes.
• Out (follow-ups):
  • UPDATE image files → conservative abort here; provably-disjoint reconcile owned by Case 1 ([RLC] Case 1 — conflict-time data skipping (disjoint data ranges) #4 / [RLC] Case 1 — conflict-time data skipping (disjoint data ranges) #8).
  • MERGE with net-new inserts still conflicts by design (umbrella backlog).
  • OPTIMIZE-compaction-vs-DML needs a cross-file offset remap → Case 4 ([RLC] Case 4 — OPTIMIZE ⟂ DML reconciliation (compaction offset remap) #7 / [RLC] Case 4a — OPTIMIZE loses to DML (forward compaction offset remap) #9 · [RLC] Case 4b — DML loses to OPTIMIZE (reverse compaction offset remap) #11), which reuses this PR's DV helpers.

Testing

RowLevelConcurrencySuite (new), using the phase-locking concurrency harness:

• DELETE/DELETE: disjoint → both commit (merged DV cardinality 2); overlapping → still conflict.
• UPDATE: DELETE (loser) vs UPDATE (winner) → winner's image file conservatively conflicts, loser aborts cleanly (one-way safe; Case 1 refines this to reconcile the disjoint case); overlapping UPDATE/UPDATE on the same row → still conflict.
• Edge cases: winner fully removes a file → conflict; 3-way concurrent disjoint DELETEs → all commit; partitioned table → both commit.
• Gating: feature flag off → conflict; deletion vectors disabled (flag on) → gate off, conflict.
• Coverage: non-empty base-DV (exercises .andNot(base)), 3-way overlap → abort, explicit Serializable, row-tracking _metadata.row_id stability, CDF one-delete-per-version.

Data correctness is asserted end-to-end (final row set) plus merged-DV cardinalities.

🤖 Generated with Claude Code

[sezruby]

Cross-link: the out-of-scope item "MERGE with net-new inserts still conflicts" is now handled by #20 (Case 1 follow-up, opt-in merge.sourceKeyReadPredicate.enabled).

• MERGE records a source-key read predicate. The value-exact tiers evaluate it as an exact semi-join against the source keys, so a concurrent insert of the same new key aborts and disjoint inserts commit.
• For disjoint keys in the same file, the write/write side still needs this PR's DV union. With both PRs, MERGE ⟂ MERGE on disjoint keys reconciles like the reference implementation.