fix: apply spill_compression to RepartitionExec spill files

[adriangb]

Which issue does this PR close?

• Closes RepartitionExec ignores datafusion.execution.spill_compression and always spills uncompressed #26141.

Rationale for this change

RepartitionExec does not apply datafusion.execution.spill_compression, so its spill files are always uncompressed. All other spilling operators (sort, aggregate, sort-merge join, nested loop join) apply the setting. A user who sets a codec to decrease spill disk usage gets no decrease for the data that RepartitionExec spills.

What changes are included in this PR?

• RepartitionExec::execute now builds its SpillManager with .with_compression_type(context.session_config().spill_compression()), the same as the other operators.
• No change is necessary on the read side. Spill files use the Arrow IPC Stream format, and the reader gets the codec from the stream.
• I checked the other non-test SpillManager::new call sites. RepartitionExec was the only one that did not pass the setting.

What is the testing strategy for this PR?

The new test repartition_spill_honors_spill_compression runs a spilling RepartitionExec on constant (highly compressible) data with uncompressed, lz4_frame and zstd. It asserts that the same rows spill, that the data reads back intact, and that spilled_bytes with a codec is less than half of the uncompressed value.

The test fails on main (679264 vs 679264 bytes) and passes with the fix:

spill_compression	spilled_bytes on main	spilled_bytes with this PR
uncompressed	679,264	679,264
lz4_frame	679,264	7,744
zstd	679,264	5,024

Are there any user-facing changes?

Yes. Spill files from RepartitionExec now use the codec from datafusion.execution.spill_compression. The default (uncompressed) does not change. There are no API changes.

[adriangb]

@jayzhan211 could you take a look? thanks!

[codecov-commenter]

Codecov Report

❌ Patch coverage is 84.74576% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.75%. Comparing base (4978b30) to head (578c96e).
⚠️ Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
datafusion/physical-plan/src/repartition/mod.rs	84.74%	1 Missing and 8 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #26142      +/-   ##
==========================================
- Coverage   82.75%   82.75%   -0.01%     
==========================================
  Files        1147     1147              
  Lines      449841   449900      +59     
  Branches   449841   449900      +59     
==========================================
+ Hits       372253   372294      +41     
- Misses      54912    54921       +9     
- Partials    22676    22685       +9     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[jayzhan211]

Thanks @adriangb , LGTM!