branch-4.1: [opt](point query) reduce point-query network overhead by resending query context on cache miss #62836#63013
Open
github-actions[bot] wants to merge 1 commit intobranch-4.1from
Open
Conversation
…uery context on cache miss (#62836) During each point query operation, the FE sends the queried table, output expression, and query option to the BE. When the hit rate of the BE lookup connection cache is very high, this request parameter is invalid. If the request sent by FE to BE ignores these parameters, BE will request FE to resend these parameters when it clearly knows that its cache is missing. When the cache hit rate of BE lookup connection is very high, this benefit will be significant. Generally, only when a connection is newly established for the first time will there be an additional resend request (this can also be optimized in the future). In our production environment, when performing 340,000 QPS/s queries on an 11-column table, the outbound network traffic of FE reaches as high as 2.29 GB/s, but the actual inbound network traffic of the Client is only 430 MB/s, which means that up to 1.9 GB/s of traffic is the request traffic sent by FE to BE. After applying this optimization, the outbound network traffic of FE is only 550 MB/s, and the query throughput under the same concurrency has also increased by 10%.
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
|
run buildall |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-picked from #62836