Support reading skims from Parquet files, in addition to OMX - #1101
Open
jpn-- wants to merge 1 commit into
Open
Conversation
* Initial plan * Add parquet skim reading support (dense row/col-major + sparse) * Fix docstring typo found in code review * Avoid duplicate array creation in dense layout detection * Address review comments: searchsorted, table[col] indexing, caching, docstring * Support Parquet skims in Sharrow runs * Format Parquet skim tests * Require Sharrow 2.16 and update uv lock * Use released Sharrow from PyPI * Fix Parquet skim loading across sparse and mixed sources Normalize dense skims with nonascending zone IDs and zero-fill missing sparse OD pairs. Preserve time-period pages when merging OMX and Parquet sources. --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Jeff Newman <jeff@driftless.xyz>
Member
Author
|
@amsamimi I cannot formally make you a GitHub "reviewer" on this unless you accept the invitation to join the ActivitySim org, but since this addresses the issue you opened, you should definitely review it anyway 😉 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #970
This pull request adds support for reading skim files in Parquet format in addition to the existing OMX format. The changes allow both dense and sparse Parquet files to be auto-detected and mixed with OMX files in the same workflow, with no changes required to existing settings other than using the
.parquetor.pqfile extension. The implementation includes updates to documentation, configuration, and the skim loading logic, ensuring compatibility with legacy workflows and Sharrow 2.16+ features.Support for Parquet skim files:
.parquetor.pqextensions) alongside OMX files. Parquet files can be dense or sparse and are supported by both legacy and Sharrow-based loaders. [1] [2] [3]network.pyto describe Parquet skim file support and usage. [1] [2] [3]Skim data loading and handling:
Codebase and API updates: