Repository navigation
DPL Analysis: cap the read buffers of the native AOD reader - #16032
Merged
Merged
Conversation
The native reader fetches the compressed baskets of all the trees a DF is read from with one coalesced read into one buffer, kept until the DF is done, and the buffer pool keeps up to two of them for reuse. On Run 2 AO2Ds (about 62 MB of compressed data per DF) this holds about 75 MB of private memory per reader for the whole job: IgProf shows the largest allocation of the reader, 72 MB, in ReadBufferPool::get, and smaps shows it as anonymous memory, unaffected by fewer threads, no prefetch or MALLOC_ARENA_MAX=1. --aod-reader-buffer-mb N caps that. The trees of a DF, in the order they are read, are packed into read plans of at most N MB of compressed data; a plan is freed when the reads of its trees are done, and the buffer pool keeps at most N MB of free buffers. A DF which fits is still read in one go, so small DFs are not affected, and a single tree larger than N gets a plan of its own. The lookahead for remote files prepares only the trees which fit. Only the trees the reader asks for are read, as before. The default is 0, no limit, which keeps the current behaviour until the default is chosen from measurements.
Member
Author
|
Tested on hyperloop. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The native reader fetches the compressed baskets of all the trees a DF is
read from with one coalesced read into one buffer, kept until the DF is
done, and the buffer pool keeps up to two of them for reuse. On Run 2 AO2Ds
(about 62 MB of compressed data per DF) this holds about 75 MB of private
memory per reader for the whole job: IgProf shows the largest allocation of
the reader, 72 MB, in ReadBufferPool::get, and smaps shows it as anonymous
memory, unaffected by fewer threads, no prefetch or MALLOC_ARENA_MAX=1.
--aod-reader-buffer-mb N caps that. The trees of a DF, in the order they
are read, are packed into read plans of at most N MB of compressed data;
a plan is freed when the reads of its trees are done, and the buffer pool
keeps at most N MB of free buffers. A DF which fits is still read in one
go, so small DFs are not affected, and a single tree larger than N gets a
plan of its own. The lookahead for remote files prepares only the trees
which fit. Only the trees the reader asks for are read, as before.
The default is 0, no limit, which keeps the current behaviour until the
default is chosen from measurements.