Repository navigation
DPL Analysis: do not build the tasks a device does not run - #16036
Merged
Merged
Conversation
Every device used to construct all the analysis tasks of its executable only to describe them, although the driver already passes their specs on stdin. Their constructors create histograms, which start the ROOT interpreter in the reader, the writer and the other internal devices. A device now builds only its own task and takes the specs of the others from the driver. Without a workflow on stdin, or with --clone, everything is still built. Reader of o2-testworkflows-analysis-histograms on macOS: physical footprint 120.5 MB -> 74.7 MB, libCling and the .pcm modules no longer mapped. Output identical.
Member
Author
|
Tested on hyperloop. Improves memory usage by 10%. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every device used to construct all the analysis tasks of its executable
only to describe them, although the driver already passes their specs on
stdin. Their constructors book histograms and set up CCDB objects, and in
the reader, the writer and the other internal devices they also start the
ROOT interpreter. A device now builds only its own task and takes the
specs of the others from the driver. Without a workflow on stdin, or with
--clone, everything is still built.
This is a startup / cleanup change, not a memory saving. On Hyperloop
(test 794645) libCling and the .pcm modules are no longer mapped in the
reader, whose RSS drops by ~180 MB and PSS by ~100 MB, but these are
shared, file-backed pages which the task devices still use: the total PSS
of the test is unchanged (2.2 GB) and so is the private dirty memory of
each device. Output identical to the reference on 120/120 DFs; all 19
devices exit cleanly.