Skip to content

For the INTT hit carry-over issue in streaming, deepen the m_InttRawHitMap size, and discarding non-collision crossings - #4324

Merged
osbornjd merged 4 commits into
sPHENIX-Collaboration:masterfrom
ChengWeiShih:InttStreamingDuplication_CW
Jul 2, 2026
Merged

osbornjd merged 4 commits into
sPHENIX-Collaboration:masterfrom
ChengWeiShih:InttStreamingDuplication_CW

Conversation

@ChengWeiShih

@ChengWeiShih ChengWeiShih commented Jun 29, 2026 •

Copy link
Copy Markdown
Contributor

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work for users)
  • Requiring change in macros repository (Please provide links to the macros pull request in the last section)
  • I am a member of GitHub organization of sPHENIX Collaboration, EIC, or ECCE (contact Chris Pinkenburg to join)

What kind of change does this PR introduce? (Bug fix, feature, ...)

Summary

  1. The PR recovers missing hits.
  2. The PR represents the ultimate way to minimize the INTT carry-over hit issue in streaming readout (Discussion in https://indico.bnl.gov/event/33369/#5-cheng-weis-update)
  3. The PR has a function to remove hits from non-collision crossings. Those should be excluded from the physics analysis.

Key Changes

For recovering missing hits

  • Deepen the size of std::map<uint64_t, InttRawHitInfo> m_InttRawHitMap;
    • It is found that hits are missing because the m_InttRawHitMap in the Fun4AllStreamingInputManager is not deep enough.
    • The size of m_InttRawHitMap in Fun4AllStreamingInputManager is controlled by GetSomeMoreEvents() in SingleInttPoolInput.
    • The requirement is now if (bcliter.second <= localbclk + 120 * 40), ensuring > 40-strobe depth in the map.
    • The memory usage is below 2 GB for one .evt file.

For the INTT carry-over hit issue in streaming

  • Public function added in Fun4AllStreamingInputManager:

    • EnableInttHitDuplication(bool): Enable/disable the mitigation (defaults to disabled)
    • InttHitCarryOverShiftMaxMultiple(int): Set the strobe search depth (defaults to four future strobes)
    • SetIsRejectInttNoiseCrossings(bool): Enable/disable removing INTT hits from non-collision crossings (defaults to disable)
    • SetIsInttStreaming(bool): Set if INTT is in the streaming readout mode (defaults to true)
    • SetIsDuplicateInttFPHXBCOResetHit(bool): enable/disable duplicating INTT hits with FPHXBCO reset to the values defined by std::vector<int> m_InttResetFphxBcoVec (defaults to true)
    • SetInttResetFphxBcoVec(std::vector<int>): Set the std::vector<int> m_InttResetFphxBcoVec(default {0 to 5})
    • SetInttStreamingSignalCrossing(std::pair<int,int>): Set the INTT FPHXBCO range with collision, std::pair<int,int> m_InttStreamingSignalCrossing. (defaults to {6, 116}).
  • Implementation in the Fun4AllStreamingInputManager for the INTT carry-over hit issue in the streaming readout

    • Inclusive correction following the half-ladder ordering
      • A map, std::map<std::string, int> m_InttRawHitCount_FEE;, is prepared for every F4A event to count the number of hits for each half-ladder in each crossing of the five strobes in a F4A event (1 GL1BCO)
        • The key of the map: BCOFULL_FPHXBCO_FELIXID_FELIXChannel.
      • For entries in the map with FPHXBCO ranging from 6 to 116, we look for hits matched to the following text, duplicate those hits, and reset their BCOfull. For hits to be duplicated with FPHXBCO 0 to 5, we reassign their FPHXBCO.
        • BCOfull+k*120_FELIXID_FEE_FPHXBCO
        • BCOfull+k*120_FELIXID_FEE_AAA (AAA: 0 to 5)
      • k <= N, where N is the searching depth, the m_InttHitCarryOverShiftMaxMultiple.
      • The demonstration of "Inclusive correction" can be found in https://indico.bnl.gov/event/33369/#5-cheng-weis-update
    • Optional: boolean variable m_IsDuplicateInttFPHXBCOResetHit in the Fun4AllStreamingInputManager to enable/disable duplicating hits with FPHXBCO reset to 0 to 5.
    • The implementation also incorporates the correction method for the triggered case to be implemented in the future.

For discarding hits from non-collision crossings.

  • Prevent hits from no-collision crossings from being saved to the INTTRawHit container
    • Only for the streaming readout (m_IsInttStreaming ==true)
    • INTT strobe is synchronized with the RHIC circulation. The RHIC bunch crossings should only happen in FPHXBCO 6 to 116. Therefore, a hit with an FPHXBCO outside the range should not be a signal hit.
    • A boolean variable m_IsRejectInttNoiseCrossings in Fun4AllStreamingInputManager to remove those hits after the carry-over hit correction.

TODOs (if applicable)

The current implementation for hit duplication works only with the INTT streaming data. In the near future, we will implement the carry-over hit correction for the INTT-triggered data, which will be incorporated into the current implementation.

Links to other PRs in macros and calibration repositories (if applicable)

Motivation / context:
This PR addresses an INTT streaming readout carry-over issue where hits can be missed across strobes. It extends the effective recovery depth by duplicating INTT raw hits across future BCOs (within a configurable carry-over window) when half-ladder/FPHX association is available. AI-generated summaries can make mistakes—please use best judgment when reading.

Key changes:

  • Added new public configuration/setter APIs in Fun4AllStreamingInputManager to control INTT streaming behavior, including:
    • enabling/disabling INTT hit duplication (EnableInttHitDuplication)
    • carry-over search depth (InttHitCarryOverShiftMaxMultiple, with validation that it must be non-negative)
    • optional rejection of “noise crossings” during streaming save/duplication (SetIsRejectInttNoiseCrossings, SetIsInttStreaming) using a configurable signal-crossing FPHX_BCO range (SetInttStreamingSignalCrossing)
    • duplication rules for reset hits and their allowed FPHX_BCO values (SetIsDuplicateInttFPHXBCOResetHit, SetInttResetFphxBcoVec)
  • Updated Fun4AllStreamingInputManager::FillIntt to implement carry-over duplication:
    • when duplication is enabled, rebuilds a per-event/per-FEE hit-count map (m_InttRawHitCount_FEE) keyed by (bco, FPHX_BCO, server/packetid, fee/channel), limited to the active BCO selection window
    • first pass: saves original hits into INTTRAWHIT, optionally skipping hits whose FPHX_BCO lies outside the configured signal-crossing range (when noise rejection is enabled and INTT streaming is active)
    • duplication pass: for each half-ladder key, searches forward in m_InttRawHitMap for future strobes at future_bco = base_bco + m_InttHitCarryOverShift * search_i (with search_i = 1..m_InttHitCarryOverShiftMaxMultiple), and copies matching hits from the same server/fee/channel when:
      • FPHX_BCO matches the half-ladder’s FPHX_BCO, or
      • reset duplication is enabled and FPHX_BCO is contained in m_InttResetFphxBcoVec
    • copied hits have their bco and FPHX_BCO overwritten to the target half-ladder values; noise-crossing rejection is also applied to the half-ladder keys during this duplication pass
  • Added/expanded INTT diagnostics in FillIntt:
    • one-time state dump (guarded by m_Intt_print_count)
    • additional verbose printing (e.g., after GL1 BCO matching and for the per-key hit-count map) under Verbosity() thresholds
  • Relaxed an INTT “stuck beam clock” erasure condition in SingleInttPoolInput by widening the acceptable BCL gap (compare against localbclk + 120 * 40 instead of localbclk).

Potential risk areas:

  • Reconstruction/output behavior changes to the contents of INTTRAWHIT due to added hit duplication and optional exclusion of out-of-range (non-collision) crossings.
  • Sensitivity to configuration correctness (signal-crossing range, shift multiple, and reset-FPHXBCO vector), which directly impacts recovered/duplicated hit counts.
  • Performance impact from additional map building and forward searching across strobes (tunable via the new carry-over/max-multiple configuration).
  • Thread-safety considerations: new mutable configuration/state in Fun4AllStreamingInputManager could be risky if the manager is accessed concurrently without external synchronization.

Possible future improvements:

  • Extend and validate the same carry-over/duplication strategy for triggered (non-streaming) data as planned.
  • Add targeted tests/QA for edge cases: reset-FPHXBCO duplication behavior, boundary values for the signal-crossing range, and carry-over windows where future strobes are missing.
  • Consider refactoring the duplication key parsing/counting logic to reduce complexity and improve maintainability.

@coderabbitai

coderabbitai Bot commented Jun 29, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: a83dc6df-c4a9-48e4-ab05-4c099e023613

📥 Commits

Reviewing files that changed from the base of the PR and between a3d2215 and c32cbc8.

📒 Files selected for processing (2)
  • offline/framework/fun4allraw/Fun4AllStreamingInputManager.cc
  • offline/framework/fun4allraw/Fun4AllStreamingInputManager.h
🚧 Files skipped from review as they are similar to previous changes (2)
  • offline/framework/fun4allraw/Fun4AllStreamingInputManager.h
  • offline/framework/fun4allraw/Fun4AllStreamingInputManager.cc

📝 Walkthrough

Walkthrough

Adds configurable INTT hit duplication and noise-crossing handling in Fun4AllStreamingInputManager, plus new verbose diagnostics and a relaxed stuck-beamclock threshold in SingleInttPoolInput.

Changes

INTT Hit Duplication and Noise Rejection

Layer / File(s) Summary
Public setters and private state for INTT mitigation
offline/framework/fun4allraw/Fun4AllStreamingInputManager.h, offline/framework/fun4allraw/Fun4AllStreamingInputManager.cc
Adds public INTT configuration setters, includes the needed headers, and stores the new mitigation state with defaults.
Rebuild INTT hit counts and add diagnostics
offline/framework/fun4allraw/Fun4AllStreamingInputManager.cc
Adds one-time INTT state printing after FillInttPool(), adds post-GL1 matching verbose logging, and rebuilds m_InttRawHitCount_FEE from m_InttRawHitMap using composite hit keys.
Noise filtering and carry-over duplication
offline/framework/fun4allraw/Fun4AllStreamingInputManager.cc
Filters INTT hits by the configured signal-crossing window, parses rebuilt composite keys, searches carry-over strobes, and copies qualifying duplicate hits with updated BCO fields.
Relaxed FEE stuck-BCO threshold
offline/framework/fun4allraw/SingleInttPoolInput.cc
Changes the FEE stuck-beamclock check to allow a wider beamclock range before marking FEEs for erasure.

Sequence Diagram(s)

sequenceDiagram
  participant Fun4AllStreamingInputManager
  participant FillInttPool
  participant m_InttRawHitMap
  Fun4AllStreamingInputManager->>FillInttPool: FillIntt()
  FillInttPool-->>Fun4AllStreamingInputManager: first successful return
  Fun4AllStreamingInputManager->>Fun4AllStreamingInputManager: print INTT state once
  Fun4AllStreamingInputManager->>m_InttRawHitMap: iterate hits and rebuild m_InttRawHitCount_FEE
  Fun4AllStreamingInputManager->>Fun4AllStreamingInputManager: print verbose GL1-matched state
Loading

Possibly related PRs


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@sphenix-jenkins-ci

Copy link
Copy Markdown

Build & test report

Report for commit 807e1cd7807d45508bab079a9ec1c9a95b75ded1:
Jenkins on fire


Automatically generated by sPHENIX Jenkins continuous integration
sPHENIX             jenkins.io

@ChengWeiShih
ChengWeiShih marked this pull request as ready for review June 29, 2026 16:18

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3


ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 72bfe51e-800f-482b-9e5c-02f37e28353b

📥 Commits

Reviewing files that changed from the base of the PR and between 7b251ee and 807e1cd.

📒 Files selected for processing (3)
  • offline/framework/fun4allraw/Fun4AllStreamingInputManager.cc
  • offline/framework/fun4allraw/Fun4AllStreamingInputManager.h
  • offline/framework/fun4allraw/SingleInttPoolInput.cc

Comment thread offline/framework/fun4allraw/Fun4AllStreamingInputManager.cc Outdated
Comment thread offline/framework/fun4allraw/Fun4AllStreamingInputManager.h Outdated
for (auto bcliter : m_FEEBclkMap)
{
if (bcliter.second <= localbclk)
if (bcliter.second <= localbclk + 120 * 40)

@coderabbitai coderabbitai Bot Jun 29, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

The widened stuck-FEE check now matches FEEs that are ahead of localbclk, so this can erase healthy inputs and drop hits.

bcliter.second <= localbclk + 120 * 40 is more aggressive than the old check: any FEE up to 4800 ticks ahead of localbclk now enters the "stuck" erase path once the pool spread exceeds MaxBclkDiff(). That is the opposite of "allow more slack before erasure" and can discard valid INTT data.

Suggested fix
-    if (bcliter.second <= localbclk + 120 * 40)
+    if (bcliter.second <= localbclk &&
+        (localbclk - bcliter.second) >= 120 * 40)

As per path instructions, C++ source reviews here should prioritize correctness.

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
if (bcliter.second <= localbclk + 120 * 40)
if (bcliter.second <= localbclk &&
(localbclk - bcliter.second) >= 120 * 40)

Source: Path instructions

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, this wouldn't work, as this is related to how the PacketVector arranges the data.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ChengWeiShih Thanks for the clarification — since the PacketVector ordering guarantees the relationship between bcliter.second and localbclk at this point in GetSomeMoreEvents, my concern about the widened check matching FEEs ahead of localbclk doesn't apply here. I'll withdraw that suggestion.


✏️ Learnings added
Learnt from: ChengWeiShih
Repo: sPHENIX-Collaboration/coresoftware PR: 4324
File: offline/framework/fun4allraw/SingleInttPoolInput.cc:469-469
Timestamp: 2026-07-02T05:44:54.386Z
Learning: Repo: sPHENIX-Collaboration/coresoftware, File: offline/framework/fun4allraw/SingleInttPoolInput.cc, function SingleInttPoolInput::GetSomeMoreEvents(): The FEE stuck-beamclock check `bcliter.second <= localbclk + 120 * 40` (widened from `bcliter.second <= localbclk`) is correct given how `PacketVector` arranges/orders data; `bcliter.second` values reaching this check do not go ahead of `localbclk` in a way that would erroneously erase valid FEEs. Do not flag this direction-of-comparison concern again for this check.

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.


🧠 Learnings used
Learnt from: pinkenburg
Repo: sPHENIX-Collaboration/coresoftware PR: 4268
File: offline/packages/CaloReco/ClusterCDFCalculator.cc:9-9
Timestamp: 2026-05-18T16:21:14.018Z
Learning: In the sPHENIX core software repository, C++20 std::format is supported (via the <format> header). During code review, do not flag uses of std::format / <format> as a potential C++-version or compiler-compatibility concern anywhere in the C++ codebase. (Other issues may still be flagged as appropriate.)

@hrjheng hrjheng mentioned this pull request Jun 29, 2026
2 of 5 tasks
@hrjheng

hrjheng commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

I closed my PR #4290 as discussed in the dNdEta analysis meeting. Below are a few notes for this PR

Cheng-Wei can provide the full explanation, but here is my understanding of the differences between the two implementations:

  1. Both implementations use the same two-pass duplication strategy. The first duplication pass is identical (at least conceptually).
  2. Cheng-Wei pointed out that, although my second duplication pass correctly reassigns the FPHX BCO, it does not update BCO_full. As a result, the duplicated hits are not actually assigned to the intended strobe. Cheng-Wei's implementation corrects this by updating BCO_full together with the FPHX BCO.
  3. Cheng-Wei's implementation also duplicates hits with FPHX BCO values from 0 through 5, rather than only FPHX BCO = 0. The hypothesis is that when the FPHX BCO is reset, it may be reset to any value between 0 and 5, randomly or not. This hypothesis has not yet been verified.

To test 3, Cheng-Wei will compare the INTT doublet distributions as a function of FPHX BCO using two configurations:

  • duplicating only hits with FPHX BCO = 0 (with the corrected BCO_full assignment), and
  • duplicating hits with FPHX BCO values from 0 to 5.
    If the two approaches produce significantly different results, that would provide evidence supporting the hypothesis that the reset FPHX BCO can take values other than 0. If there is little or no difference, the additional duplication may not be necessary.

@ChengWeiShih

Copy link
Copy Markdown
Contributor Author
  • Regarding @hrjheng 's first point. The idea of the first pass is the same, but the number of hits being duplicated is different.

    • Hao-Ren's PR: Duplicating every hit from the future strobes. (taking the whole FELIX server, 14 half-ladders * 120 FPHXBCOs, as a unit)
      Cheng-Wei's PR: Take a half-ladder in a crossing as a unit. Which means the number of INTT hits for each half-ladder at each crossing is counted before correction. And duplicate hits only for the half-ladders in a given crossing of the strobe that have a hit.
    • Difference: While Hao-Ren's approach duplicates more hits, a crossing with entire fake hits can potentially be artificially created. CW's method has an assumption that at least one hit in a half-ladder in a crossing is correctly associated; fake crossings therefore are minimized.
  • For the second point, the second pass, I think the missing pieces in Hao-Ren's PR are:

    1. Missing FPHXBCO reassignment, which should be done by copied_hit->set_FPHX_BCO(The_correct_FPHXBCO);
    2. Incorrect bco_full search and reassignment, as const uint64_t target_refbco = bco % m_InttHitCarryOverShift; depends on when the data taking started. I write down an example:
      1. uint64_t bco = 608318921053;
      2. target_refbco = 608318921053 % 120 = 13
      3. shift = 120 * N. Assume N = 2, shift = 240.
      4. reset_shift = shift - target_refbco = 240 - 13 = 227.
      5. reset_source_bco = bco + reset_shift = 608318921053 + 227 = 608318921280.
      • There is no strobe with bco = 608318921280, as the bco of the next strobe should be 608318921053 + 120 = 608318921173 and then 608318921293, and so on.
      • The second pass in the PR would find nothing.
  • For the third point, I will do the study.

  • I would like to point out that @hrjheng should also have credit. I don't mean to take his credit. His first look, trial, and first implementation are inspiring and can largely duplicate hits. I learned from him that the implementation can be done in the StreamingManager. My implementation is inspired by his PR.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
offline/framework/fun4allraw/Fun4AllStreamingInputManager.cc (1)

916-918: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Cap the carry-over search depth before using it as a loop bound.

InttHitCarryOverShiftMaxMultiple() accepts any non-negative int, and Line 916 then performs that many map searches for every half-ladder key. A large configuration typo can effectively hang INTT processing. Please bound this setter to the retained strobe depth or a detector-sane maximum before assigning m_InttHitCarryOverShiftMaxMultiple. As per path instructions, prioritize correctness, memory safety, error handling, and thread-safety.

Also applies to: 1706-1713

Source: Path instructions


ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 0cf53601-3fb6-47d5-93cb-0c91c8c32ba5

📥 Commits

Reviewing files that changed from the base of the PR and between 807e1cd and a3d2215.

📒 Files selected for processing (2)
  • offline/framework/fun4allraw/Fun4AllStreamingInputManager.cc
  • offline/framework/fun4allraw/Fun4AllStreamingInputManager.h
🚧 Files skipped from review as they are similar to previous changes (1)
  • offline/framework/fun4allraw/Fun4AllStreamingInputManager.h

@sphenix-jenkins-ci

Copy link
Copy Markdown

Build & test report

Report for commit a3d221519117c1ab81ac7c913893f2fbc2d26ffa:
Jenkins on fire


Automatically generated by sPHENIX Jenkins continuous integration
sPHENIX             jenkins.io

@sphenix-jenkins-ci

Copy link
Copy Markdown

Build & test report

Report for commit c49adec2d6d6ac74bec64712f97c627acbe01449:
Jenkins passed


Automatically generated by sPHENIX Jenkins continuous integration
sPHENIX             jenkins.io

@sphenix-jenkins-ci

Copy link
Copy Markdown

Build & test report

Report for commit c32cbc8ea84c7635342f203174f28d9bfa19ef4d:
Jenkins passed


Automatically generated by sPHENIX Jenkins continuous integration
sPHENIX             jenkins.io

@osbornjd
osbornjd merged commit a79db3c into sPHENIX-Collaboration:master Jul 2, 2026
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants