Add local SubprocessCluster that runs workers in separate processes

[hendrikmakait]

Closes #xxxx

• Tests added / passed
• Passes pre-commit run --all-files

[github-actions]

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

       22 files  +       1         22 suites  +1   9h 53m 54s ⏱️ + 1m 54s
  3 282 tests +       4    3 192 ✔️ +       1       86 💤 +  1  4 ❌ +2 
36 033 runs  +1 332  34 488 ✔️ +1 269  1 541 💤 +62  4 ❌ +1 

For more details on these failures, see this check.

Results for commit 3a2d42d. ± Comparison against base commit d87cea9.

♻️ This comment has been updated with latest results.

[hendrikmakait]

This is currently broken on Windows because we use the SelectorEventLoop on Windows, which does not support subprocesses (https://docs.python.org/3/library/asyncio-platforms.html#windows). Using SelectorEventLoop seems to be necessary due to an old issue with tornado (

distributed/distributed/config.py

Lines 185 to 189 in d87cea9

	if WINDOWS:
	# WindowsProactorEventLoopPolicy is not compatible with tornado 6
	# fallback to the pre-3.8 default of Selector
	# https://github.com/tornadoweb/tornado/issues/2608
	asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())

).

Upgrading to tornado>=6.1 might resolve this, is there a reason #5833 has not been merged? cc @graingert

[mrocklin]

In principle this looks sensible to me. I would add basic docstrings.

Maybe also a single line mentioning this class at the end of https://docs.dask.org/en/stable/deploying-python.html#localcluster ?

[mrocklin]

This is currently broken on Windows because we use the SelectorEventLoop on Windows, which does not support subprocesses (https://docs.python.org/3/library/asyncio-platforms.html#windows). Using SelectorEventLoop seems to be necessary due to an old issue with tornado (

For something like this that's already supporting an odd corner case, I think that it's totally fine to just raise an informative error message on Windows and move on.

[hendrikmakait]

For something like this that's already supporting an odd corner case, I think that it's totally fine to just raise an informative error message on Windows and move on.

Added a descriptive error message 👌

[graingert]

Upgrading to tornado>=6.1 might resolve this, is there a reason #5833 has not been merged? cc @graingert

I was waiting for a bokeh release so it might be worth resurrecting that PR

[mrocklin]

Thanks @hendrikmakait . Seems good to me. I'm going to wait until end of day to see if we get any user feedback and then I'll merge in.

[tasansal]

This is great; one benefit I can think of is the overhead of moving data from the scheduler process back to the client process disappears. What are other benefits/use cases? I can help test this further and add some more to the documentation, which is very little now. Is this eventually going to replace LocalCluster?

[mrocklin]

In some situations creating workers with the multiprocessing library can cause issues. This is relatively rare though and not something we want to focus on that much.

…

On Sat, Jan 14, 2023 at 6:21 AM Altay Sansal ***@***.***> wrote: This is great; one benefit I can think of is the overhead of going from the scheduler process back to the client process disappears. What are other benefits / use cases? I wouldn't mind testing this further and adding some more to the documentation, which is very little now. — Reply to this email directly, view it on GitHub <#7431 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACKZTAKVFAKMYAJRWDGTSTWSKY6LANCNFSM6AAAAAATG3TJTI> . You are receiving this because you modified the open/close state.Message ID: ***@***.***>