Improve package scan performance #4606
Conversation
Signed-off-by: Ayan Sinha Mahapatra <[email protected]>
Reference: https://github.com/Quantco/multiregex Signed-off-by: Ayan Sinha Mahapatra <[email protected]>
Use multiregex to use a cached regex path patterns and datafile handlers mapping to detect package datafiles faster. Reference: #4064 Reference: #4061 Signed-off-by: Ayan Sinha Mahapatra <[email protected]>
AyanSinhaMahapatra
changed the title
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Signed-off-by: Ayan Sinha Mahapatra <[email protected]>
Signed-off-by: Ayan Sinha Mahapatra <[email protected]>
Signed-off-by: Ayan Sinha Mahapatra <[email protected]>
Introduce a new option --binary-packages which looks for package/dependency data in binaries. Signed-off-by: Ayan Sinha Mahapatra <[email protected]>
AyanSinhaMahapatra
force-pushed
the
fast-package-scan
branch
from
a99c4f8 to
dde6bc9
Compare
We do not need the license index in a --package-only scan as this is designed to do a fast package detection only scan which skips the license detection. As license index loading takes a couple seconds in each case, this makes the package only scan much faster. Signed-off-by: Ayan Sinha Mahapatra <[email protected]>
Signed-off-by: Ayan Sinha Mahapatra <[email protected]>
| --system-package Scan ``<input>`` for installed system package | ||
| databases. | ||
|
|
||
| -b, --binary-package Scan <input> for package and dependency related |
There was a problem hiding this comment.
What about this:
--package-in-exec Scan compiled executable binaries such as ELF, WinpE and Mach-O files, looking for structured package and dependency metadata as found for example in Go and Rust binaries.
There was a problem hiding this comment.
Or
--package-in-compiled Scan compiled executable binaries such as ELF, WinpE and Mach-O files, looking for structured package and dependency metadata as found for example in Go and Rust compiled binaries.
|
|
||
| ./configure --dev | ||
| venv/bin/scancode-reindex-licenses | ||
| venv/bin/scancode-cache-package-patterns |
There was a problem hiding this comment.
What about naming this venv/bin/scancode-reindex-package-patterns to be consistent?
|
|
||
|
|
||
| # These handlers are special as they use filetype to | ||
| # detect these binaries instead of datafile path patterns |
There was a problem hiding this comment.
| # detect these binaries instead of datafile path patterns | |
| # detect these compiled executable binaries instead of datafile path patterns |
| PACKAGE_INDEX_DIR = 'package_patterns_index' | ||
| PACKAGE_INDEX_FILENAME = 'index_cache' | ||
| PACKAGE_LOCKFILE_NAME = 'scancode_package_index_lockfile' | ||
| PACKAGE_CHECKSUM_FILE = 'scancode_package_index_tree_checksums' |
There was a problem hiding this comment.
This is not used anymore (also should be dropped from licensing)
| PACKAGE_CHECKSUM_FILE = 'scancode_package_index_tree_checksums' |
pombredanne
left a comment
pombredanne
left a comment
There was a problem hiding this comment.
Here are some nits for your consideration!
| --system-package Scan ``<input>`` for installed system package | ||
| databases. | ||
|
|
||
| -b, --binary-package Scan <input> for package and dependency related |
There was a problem hiding this comment.
Or
--package-in-compiled Scan compiled executable binaries such as ELF, WinpE and Mach-O files, looking for structured package and dependency metadata as found for example in Go and Rust compiled binaries.
2 participants
This PR improve package scan performance by....
References:
Tasks
Run tests locally to check for errors.