Skip to content

Support reading skims from Parquet files, in addition to OMX - #1101

Open
jpn-- wants to merge 1 commit into
ActivitySim:mainfrom
driftlesslabs:main
Open

Support reading skims from Parquet files, in addition to OMX#1101
jpn-- wants to merge 1 commit into
ActivitySim:mainfrom
driftlesslabs:main

Conversation

@jpn--

@jpn-- jpn-- commented Aug 7, 2026

Copy link
Copy Markdown
Member

Closes #970

This pull request adds support for reading skim files in Parquet format in addition to the existing OMX format. The changes allow both dense and sparse Parquet files to be auto-detected and mixed with OMX files in the same workflow, with no changes required to existing settings other than using the .parquet or .pq file extension. The implementation includes updates to documentation, configuration, and the skim loading logic, ensuring compatibility with legacy workflows and Sharrow 2.16+ features.

Support for Parquet skim files:

  • Added logic to auto-detect and load skim files in Parquet format (with .parquet or .pq extensions) alongside OMX files. Parquet files can be dense or sparse and are supported by both legacy and Sharrow-based loaders. [1] [2] [3]
  • Updated documentation and configuration docstrings in network.py to describe Parquet skim file support and usage. [1] [2] [3]

Skim data loading and handling:

  • Refactored the skim dataset loading logic to merge data from multiple sources, handle time periods, and fill zeros for missing sparse OD pairs to maintain compatibility with legacy behavior.
  • Ensured that Parquet-backed datasets are properly handled when storing skims in shared memory, including closing file handles and supporting dask-backed data.

Codebase and API updates:

  • Added and documented new internal helpers for time period detection, masking, and zero-filling for Parquet files.
  • Updated the skim dictionary factory to cache Parquet file metadata and ensure consistent zone ID handling across mixed input files. [1] [2] [3]
  • Clarified and expanded docstrings to reflect support for both OMX and Parquet skim files throughout the codebase. [1] [2]

* Initial plan

* Add parquet skim reading support (dense row/col-major + sparse)

* Fix docstring typo found in code review

* Avoid duplicate array creation in dense layout detection

* Address review comments: searchsorted, table[col] indexing, caching, docstring

* Support Parquet skims in Sharrow runs

* Format Parquet skim tests

* Require Sharrow 2.16 and update uv lock

* Use released Sharrow from PyPI

* Fix Parquet skim loading across sparse and mixed sources

Normalize dense skims with nonascending zone IDs and zero-fill missing sparse OD pairs.
Preserve time-period pages when merging OMX and Parquet sources.

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: Jeff Newman <jeff@driftless.xyz>
@jpn--

jpn-- commented Aug 7, 2026

Copy link
Copy Markdown
Member Author

@amsamimi I cannot formally make you a GitHub "reviewer" on this unless you accept the invitation to join the ActivitySim org, but since this addresses the issue you opened, you should definitely review it anyway 😉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support reading skims in Parquet format

2 participants