Faster random numbers - #1103
Conversation
Introduce a configurable RNG channel type and exercise both implementations in tests. Add Settings.rng_channel_type to choose between the new FastChannel (PCG64 vectorised) and legacy SimpleChannel for reproducibility. Random now accepts a channel_type on init and add_channel accepts fast=None to default to the global channel_type; existing code will pick up settings.rng_channel_type via State initialization and rng access. Implement FastChannel.extend_domain to allow adding new domain rows (initialising per-row PCG64 state when a step is active) and tighten index handling. Update many pipeline tests to parametrize over channel types, isolate per-channel output dirs, and include per-channel expected regression values and checks.
There was a problem hiding this comment.
Pull request overview
Adds configurable vectorized RNG channels while retaining legacy reproducibility.
Changes:
- Implements PCG64 and SFC64 per-row random streams.
- Integrates RNG selection into workflow settings.
- Adds regression tests, benchmarks, and performance automation.
Reviewed changes
Copilot reviewed 17 out of 19 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
uv.lock |
Locks the CFFI dependency. |
pyproject.toml |
Declares CFFI at runtime. |
other_resources/scripts/random-performance.ipynb |
Explores RNG performance. |
other_resources/performance-checks/fast-channel-random.py |
Adds a benchmark script. |
activitysim/core/workflow/state.py |
Configures RNG channel selection. |
activitysim/core/test/test_random.py |
Expands cross-channel contract tests. |
activitysim/core/test/test_fast_random.py |
Tests vectorized generators. |
activitysim/core/test/test_fast_channel.py |
Tests FastChannel. |
activitysim/core/random.py |
Integrates fast channels into the RNG API. |
activitysim/core/fast_random/_fast_channel.py |
Implements vectorized per-row streams. |
activitysim/core/fast_random/_entropy.py |
Implements accelerated reseeding. |
activitysim/core/fast_random/__init__.py |
Exports FastChannel. |
activitysim/core/configuration/top.py |
Documents RNG settings. |
activitysim/abm/test/test_pipeline/test_pipeline.py |
Adds pipeline regression coverage. |
activitysim/abm/test/test_pipeline/output/trace/.gitignore |
Removes redundant ignores. |
activitysim/abm/test/test_pipeline/output/cache/.gitignore |
Removes redundant ignores. |
activitysim/abm/test/test_pipeline/output/.gitignore |
Removes redundant ignores. |
.github/workflows/performance-checks.yml |
Adds manual benchmark automation. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Generate stable-universe shocks in chooser batches rather than allocating the full chooser-by-sample-by-alternative array. This prevents large EET models from exhausting memory while retaining only active uniforms or winning positions. Preserve per-row draw order and validate the entire chooser index before batching. Regression tests enforce allocation bounds and exact results and subsequent streams for both accelerated channels.
Use explicit row counts when reshaping vectorized draws so zero-length dimensions return correctly shaped empty arrays. Return immediately for zero-size choice instead of consuming a full population permutation, and support zero samples in batched Gumbel-max draws. Add cross-channel empty-draw regressions and low-level tests covering both bit generators, multidimensional empty shapes, and empty row selections. Verify that subsequent draws and generator states remain unchanged.
Reject negative or fractional sample dimensions, non-vector populations, empty populations with nonzero samples, and oversized samples without replacement. Use integer-index validation and a Python dimension product to avoid silent truncation or overflow. Validate before lazy seeding or draw consumption so rejected calls leave reproducible streams untouched. Add NumPy parity and state-preservation regressions, plus coverage for valid tuple sizes and empty populations with zero samples.
|
This PR was submitted to GPT-6-Astra for re-review, which surfaced 3 additional issues (now solved):
|
janzill
left a comment
janzill
left a comment
There was a problem hiding this comment.
This took a while to review, partly because I wanted to find a way to avoid the large amount of added code and complexity but I don't think there is a way around it within the current framework. In the end, I think we should merge it, I just flagged a couple of minor things.
| from ._fast_channel import FastChannel # noqa: F403 | ||
|
|
||
| __all__ = ( # noqa: F405 | ||
| # TODO: Add all public symbols here. |
| rng_base_seed: Union[int, None] = 0 | ||
| """Base seed for pseudo-random number generator.""" | ||
|
|
||
| rng_channel_type: Literal["simple", "fast", "faster"] = "simple" |
There was a problem hiding this comment.
I find simple, fast, faster misleading. How about legacy, default, experimental?
| self._next_uint64 = self._bit_generator.cffi.next_uint64 | ||
| self._next_double = self._bit_generator.cffi.next_double | ||
| self._state_address = self._bit_generator.cffi.state_address | ||
| self._state_ptr = _FFI.cast("uint8_t(*)[128]", self._state_address) |
There was a problem hiding this comment.
AI flagged this as a potential problem because it reads 128 bytes from state_address, which depends on numpy internal memory layout and currently reads past the object's memory. If numpy's internal layout would change between numpy versions and the scan in the loop below fails, then ActivitySim will fail on import because FastChannel is imported in random.py, and entropy constructs two FastRandom at import time. Can we avoid this and only do it when the new RNG is used? That way any potential problems here do not show up when running with the legacy RNG.
| for k in range(16): | ||
| if viewer[k] == target: | ||
| break | ||
| if k >= 15: |
There was a problem hiding this comment.
| if k >= 15: | |
| else: |
Summary
This PR adds high-performance, vectorized random-number channels while preserving ActivitySim’s legacy RNG behavior as an option.
Two accelerated modes are available through
rng_channel_type:fast: PCG64 with robust entropy generation. Better than simple for large models but still following rigorous "safe" randomness algorithms)faster: SFC64 with lower-overhead hash-based reseeding. Fastest overall for nearly all purposes, but employs short cuts on seeding that are probably fine for large scale simulation, but not rigorously validated as fully uncorrelated random streams to the highest possible levels of confidencesimple: legacyRandomStateimplementation and default for backward compatibilityKey changes
cffias a runtime dependency.mainand removes unrelated PR scope.Note: this PR has advanced notably from the last time we looked at it, as the EET branch introduced several new variants of randomness. I have iterated this on a couple different AI models to get what I believe to be a good result.