pool: exclude p2p requests from numberOfRequestsFor count#8048
Open
greenc-FNAL wants to merge 1 commit into11.2from
Open
pool: exclude p2p requests from numberOfRequestsFor count#8048greenc-FNAL wants to merge 1 commit into11.2from
greenc-FNAL wants to merge 1 commit into11.2from
Conversation
Motivation: We wish to prevent internal P2P traffic from contributing to the file request count used by the hot-file replication mechanism, and thereby avoid potential replication feedback loops. Modification: Enhance `MoverRequestScheduler.numberOfRequestsFor` to filter out pool-to-pool (P2P) transfers. Result: Tests show that P2P requests are excluded from the file request count for the purposes of triggering hot file migration. Acked-by: Tigran Mkrtchyan, Dmitry Litvintsev Patch: https://rb.dcache.org/r/14628/diff/raw Commit: Target: master Request: 11.2 Require-book: no Require-notes: yes
greenc-FNAL
changed the title
There was a problem hiding this comment.
Pull request overview
This PR rolls up the 11.2 release-branch changes around Firefly/SciTag marker handling, HTTP/WebDAV transfer metadata propagation, pool CLI tooling, and dependency/version updates to advance the 11.2.x line.
Changes:
- Extend SciTag/Firefly marker support (protocol coverage, excludes semantics, optional additional collector) and add regression tests.
- Enhance HTTP/WebDAV transfers to capture SciTag headers and propagate SciTag/ArchiveMetadata into protocol/storage metadata.
- Add a
dcache pool benchmarkcommand with an fio workload, adjust pool size handling for percentage max diskspace, and bump versions/dependencies.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| skel/share/lib/utils.sh | Add getTotalSpace helper for pool sizing logic. |
| skel/share/lib/pool.sh | Handle percentage-based max diskspace when printing pool size. |
| skel/share/lib/pool-benchmark.fio | New fio jobfile for pool filesystem benchmarking. |
| skel/share/defaults/pool.properties | Document Firefly destination as an optional additional collector. |
| skel/man/man8/dcache.8 | Document new dcache pool benchmark subcommand. |
| skel/bin/dcache | Add pool benchmark implementation using fio. |
| pom.xml | Bump parent version; update Zookeeper/nfs4j; add jackson-datatype-jdk8 (managed). |
| plugins/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
| plugins/hsqldb/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
| packages/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
| packages/fhs/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
| packages/tar/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
| packages/system-test/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
| modules/srm-server/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
| modules/srm-common/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
| modules/srm-client/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
| modules/missingfiles-semsg/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
| modules/logback-test-config/pom.xml | Bump module version to 11.2.3-SNAPSHOT. |
| modules/logback-console-config/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
| modules/gplazma2/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
| modules/gplazma2-voms/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
| modules/gplazma2-scitoken/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
| modules/gplazma2-roles/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
| modules/gplazma2-pyscript/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
| modules/gplazma2-omnisession/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
| modules/gplazma2-oidc/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
| modules/gplazma2-oidc-te/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
| modules/gplazma2-nsswitch/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
| modules/gplazma2-nis/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
| modules/gplazma2-multimap/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
| modules/gplazma2-ldap/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
| modules/gplazma2-krb5/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
| modules/gplazma2-kpwd/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
| modules/gplazma2-jaas/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
| modules/gplazma2-htpasswd/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
| modules/gplazma2-grid/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
| modules/gplazma2-fermi/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
| modules/gplazma2-banfile/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
| modules/gplazma2-alise/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
| modules/ftp-client/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
| modules/dcache/src/main/java/org/dcache/pool/movers/TransferLifeCycle.java | Improve SciTag parsing, excludes semantics, protocol coverage; add optional collector send. |
| modules/dcache/src/test/java/org/dcache/pool/movers/TransferLifeCycleTest.java | New regression tests for SciTag/FQAN/excludes behavior. |
| modules/dcache/src/main/java/org/dcache/pool/classic/MoverRequestScheduler.java | Exclude P2P transfers from numberOfRequestsFor counting. |
| modules/dcache/src/test/java/org/dcache/pool/classic/MoverRequestSchedulerTest.java | New tests for P2P exclusion in request counting. |
| modules/dcache/src/test/java/org/dcache/pool/classic/IoQueueManagerTest.java | New tests for aggregated request counting excluding P2P queue. |
| modules/dcache/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
| modules/dcache-xrootd/src/main/java/org/dcache/xrootd/door/XrootdTransfer.java | Add debug logging around scitag.flow transferTag propagation. |
| modules/dcache-xrootd/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
| modules/dcache-webdav/src/main/java/org/dcache/webdav/transfer/RemoteTransferHandler.java | Pass transferTag into remote HTTP(S) protocol info; add ArchiveMetadata handling. |
| modules/dcache-webdav/src/main/java/org/dcache/webdav/transfer/CopyFilter.java | Capture SciTag headers for TPC and pass transferTag to remote transfer handling. |
| modules/dcache-webdav/src/main/java/org/dcache/webdav/LoggingHandler.java | Log captured SciTag header values for TPC requests. |
| modules/dcache-webdav/src/main/java/org/dcache/webdav/DcacheResourceFactory.java | Read SciTag from headers/param; pass ArchiveMetadata for custodial uploads. |
| modules/dcache-webdav/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
| modules/dcache-vehicles/src/main/java/diskCacheV111/vehicles/RemoteHttpDataTransferProtocolInfo.java | Add transferTag field for RemoteHttp* protocol info with backward-compatible deserialization. |
| modules/dcache-vehicles/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
| modules/dcache-srm/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
| modules/dcache-spacemanager/src/main/java/diskCacheV111/services/space/SpaceManagerService.java | Always set AL/RP from default reservation; warn on conflicts instead of throwing. |
| modules/dcache-spacemanager/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
| modules/dcache-chimera/src/main/java/org/dcache/chimera/namespace/ChimeraHsmStorageInfoExtractor.java | Ensure AL/RP defaults are returned even when WriteToken tag exists. |
| modules/dcache-chimera/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
| modules/dcache-resilience/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
| modules/dcache-qos/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
| modules/dcache-nfs/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
| modules/dcache-nearline-spi/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
| modules/dcache-info/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
| modules/dcache-history/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
| modules/dcache-gplazma/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
| modules/dcache-ftp/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
| modules/dcache-dcap/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
| modules/dcache-frontend/src/main/java/org/dcache/restful/providers/ObjectMapperProvider.java | Register Jackson JDK8 module (Optional support) on default mapper. |
| modules/dcache-frontend/pom.xml | Add jackson-datatype-jdk8 dependency for REST Optional support. |
| modules/common/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
| modules/common-security/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
| modules/common-cli/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
| modules/chimera/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
| modules/cells/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
| modules/benchmarks/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
| modules/acl/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
| modules/acl-vehicles/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
| modules/dcache-bulk/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
| docs/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
| docs/UserGuide/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
| docs/TheBook/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
| archetypes/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
| archetypes/dcache-nearline-plugin-archetype/pom.xml | Bump parent version to 11.2.3-SNAPSHOT. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| import org.dcache.vehicles.FileAttributes; | ||
| import org.junit.Before; | ||
| import org.junit.Test; | ||
| import java.util.concurrent.atomic.AtomicInteger; |
There was a problem hiding this comment.
There is an unused import (java.util.concurrent.atomic.AtomicInteger) in this new test file. Java treats unused imports as compilation errors, so it must be removed.
Suggested change
| import java.util.concurrent.atomic.AtomicInteger; |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation:
We wish to prevent internal P2P traffic from contributing to the file request count used by the hot-file replication mechanism, and thereby avoid potential replication feedback loops.
Modification:
Enhance
MoverRequestScheduler.numberOfRequestsForto filter out pool-to-pool (P2P) transfers.Result:
Tests show that P2P requests are excluded from the file request count for the purposes of triggering hot file migration.
Acked-by: Tigran Mkrtchyan, Dmitry Litvintsev
Patch: https://rb.dcache.org/r/14628/diff/raw
Target: master
Request: 11.2
Require-book: no
Require-notes: yes