Add support for gevent - #6372
Open
skiedude wants to merge 20 commits into
Open
Conversation
* Bump pip and setuptools and switch from setup.py to pip install * Add gevent to requirements, bumps for compatibility and regenerate requirement files. * Display which components fail unit tests. * Add checks for eventlet code in monkey patch
pull-request-size
Bot
added
the
size/XXL
Contributor
|
looks like you have some failing unit tests |
- Took upstream's CI image bumps (py3.10, mongo 8.2, rabbitmq 4.2) in .circleci/config.yml - Merged Makefile: kept setuptools_scm/python -m build (needed for gevent's setuptools pin), took upstream's $(PYBIN) consistency - fixed-requirements.txt: kept gevent/eventlet/greenlet/gunicorn pins, dropped unused flex, picked up importlib_resources/importlib_metadata - Regenerated all per-component requirements.txt via make requirements - Regenerated lockfiles/st2.lock via pants generate-lockfiles --resolve=st2 - st2client httpclient.py: resolved shlex.quote alias naming - test_service_setup_log_level_filtering.py: kept concurrency.sleep() abstraction, took upstream's SIGTERM/8s timeout flakiness fix
skiedude
commented
Jul 24, 2026
Contributor
|
can we just switch to gevent. why have a variable? |
- Read concurrency_library from st2.conf [system] instead of an env var, defaulting to gevent. - Pick the gunicorn worker class (-k eventlet/-k gevent) dynamically based on the configured concurrency library instead of hardcoding eventlet. - Fix real gevent bugs: wrap_ssl() using the removed ssl.wrap_socket(), green_pool_wait_all() not actually blocking, wait() not returning the greenlet's result, and several call sites bypassing the concurrency wrapper with direct .wait()/.free() calls on greenlets/pools. - Fix a gevent-only hang in the python runner: process.stdin was written to but never closed/flushed for large (>64KB) parameters, leaving the child process blocked forever waiting for input that never arrived. - Remove remaining direct eventlet imports outside the concurrency compatibility layer (tools, examples, test mocks); keep back-compat aliases (EventletTestCase, setup_eventlet_profiler, blocking_eventlet_spawn) for external callers.
Both conf/st2.tests.conf (used by subprocess-spawning integration tests) and st2tests/conf/st2.conf now explicitly set concurrency_library = gevent instead of silently relying on the code-level fallback default.
…ify library loading conftest.py now monkey patches before any test module is imported, whether pytest runs a whole directory or a single file in isolation (as Pants does in CI). Without this, tooz's Heart binds threading.Thread/Event as default args at import time; if tooz gets imported before patching, it permanently uses the unpatched primitives, causing gevent.exceptions.LoopExit or hangs in test_synchronization.py, test_worker.py, and test_service_registry.py. concurrency.py now only imports the concurrency library that's actually active, replaces the CONCURRENCY_LIBRARY/eventlet/gevent globals with a single state object, and consolidates every eventlet/gevent submodule import into one place instead of scattering local imports across each helper function.
Sweep of every file importing concurrency.py for objects created via its factory functions (spawn/Semaphore/Queue/green pools) that still called library-specific methods directly instead of going through the wrapper. Adds link(), pool_spawn(), and resize_green_pool() wrappers and updates their call sites in actionrunner.py, workflow_engine.py, scheduler/handler.py, greenpooldispatch.py, wsgi.py, parallel_ssh.py, consumers.py, st2-inject-trigger-instances.py, and the corresponding test files. Also defers actionrunner.py/workflow_engine.py's SIGTERM handlers to concurrency.spawn(...) instead of calling .kill() synchronously from the raw signal callback, since gevent raises BlockingSwitchOutError for blocking greenlet operations invoked directly from a signal handler (eventlet has no such restriction). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
concurrency.pywas started previously and this PR finishes its integration. Any part of the code that needs to manipulate/create/use/destroy an eventlet/gevent object has to do so using concurrency helpers.[system]level conf key ofconcurrency_libraryis now available. If its unset, it will default togeventbut can be set toeventletif desired.I performed local testing using a local docker setup that was running all services on the updated code. A new pack was created to specifically test concurrency type features as well: https://github.com/skiedude/concurrency-test-pack/tree/main (should consider redoing these actions a bit to make them runnable through future PRs?)
Claude aided in the development and testing of this PR.