Skip to content

Conversation

@erenavsarogullari
Copy link
Member

@erenavsarogullari erenavsarogullari commented Feb 9, 2026

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

When there is not enough memory to continue external sort, either increasing the memory limit or decreasing sort_spill_reservation_bytes need to be applied. It can be useful to guide the user with clearer error message by highlighting required configs for the consistency because expected settings are as follows:

SET datafusion.runtime.memory_limit = '10G'
SET datafusion.execution.sort_spill_reservation_bytes = 10485760

Current:

Not enough memory to continue external sort. Consider increasing the memory limit, 
or decreasing sort_spill_reservation_bytes

New:

Not enough memory to continue external sort. Consider increasing the memory limit config: 
'datafusion.runtime.memory_limit', or decreasing the config: 'datafusion.execution.sort_spill_reservation_bytes'.

Are these changes tested?

A new UT case has been added to cover this use-case and it has been tested locally successfully.

For updated snapshot files:
Following integration tests have been successful by DataFusion Test Pipeline:

test test_cli_top_memory_consumers::case_1 ... ok
test test_cli_top_memory_consumers::case_2 ... ok
test test_cli_top_memory_consumers::case_3 ... ok

Ref: https://github.com/apache/datafusion/actions/runs/21811797863/job/62925363536?pr=20226

Are there any user-facing changes?

Yes, improving legacy ExternalSorter ResourcesExhausted Error Message.

@github-actions github-actions bot added the physical-plan Changes to the physical-plan crate label Feb 9, 2026
@erenavsarogullari erenavsarogullari force-pushed the improve_external_sorter_error_message_when_resource_exhausted branch 5 times, most recently from ba97e75 to 64eaa60 Compare February 9, 2026 03:56
@erenavsarogullari erenavsarogullari force-pushed the improve_external_sorter_error_message_when_resource_exhausted branch from 64eaa60 to 9539d82 Compare February 9, 2026 18:38
@erenavsarogullari erenavsarogullari force-pushed the improve_external_sorter_error_message_when_resource_exhausted branch from 9539d82 to e306fe7 Compare February 10, 2026 17:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

physical-plan Changes to the physical-plan crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Improve ExternalSorter ResourcesExhausted Error Message

2 participants