Add a score on the lowest performing subjects - #1133
Merged
bruAristimunha merged 4 commits intoAug 25, 2026
Merged
Conversation
compute_lowest_subject_scores ranks the subjects of every (dataset, pipeline) pair and averages only the lowest percentile of them, which is the family of metrics F1@20% belongs to. ERP, SSVEP and c-VEP saturate close to a perfect score on most subjects, so a mean over the whole cohort is dominated by subjects that no longer separate pipelines. The subjects a pipeline handles worst are where the remaining headroom is. Fixes NeuroTechX#733
…-1133-reference-cohort-20260825
bruAristimunha
approved these changes
Aug 24, 2026
bruAristimunha
left a comment
Collaborator
There was a problem hiding this comment.
Approved at exact head 6dccccd after an independent full-range review and fix-round re-review. The statistic now selects one explicit reference cohort per dataset, evaluates every pipeline on identical subjects, rejects raw non-finite scores/null identities/incomplete coverage, uses observed categorical groups, and supports heterogeneous subject IDs. Verification: 23 focused tests, 67 non-CodeCarbon analysis tests, 163 shared analysis/plotting consumer tests (2 skipped), scoped pre-commit, and diff checks passed. The five remaining full-module CodeCarbon setup errors reproduce unchanged on the exact integrated baseline. No unresolved review threads.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds
moabb.analysis.meta_analysis.compute_lowest_subject_scores, the metric requested in #733.The helper now requires an explicit
reference_pipeline. For each dataset it macro-averages the supplied session scores per subject, selects the lowestpercentilecohort once from that reference pipeline, and evaluates every pipeline on those exact same subject identities. It returns one row per observed(dataset, pipeline)with the resultingscoreand commonn_subjects.Using a shared reference cohort is the scientifically important part: selecting the worst subjects independently for each pipeline can compare different populations and even reverse a conclusion. The fixed-reference construction follows the hardest-subject comparison described by Gnassounou, Collas, Flamary, and Gramfort in PSDNorm: Test-Time Temporal Normalization for Deep Learning in Sleep Staging (arXiv:2503.04582). This helper operates on supplied scores; it does not recompute subject-level F1 from predictions.
The retained count is
ceil(n_subjects * percentile / 100)and never below one. Score ties use stable input order, so heterogeneous subject identifiers do not need to be compared. Every pipeline must have exactly the reference subject set in each dataset. Empty input, raw NaN/infinity, null cohort identifiers, incomplete subject coverage, a missing reference, and percentiles outside(0, 100]fail closed. Filtered categorical frames use observed levels only, and coercible numeric score arrays are handled on a copy without mutating the caller.Coverage includes the conclusion-reversal counterexample, per-dataset reference selection, session macro-aggregation, counts and 100% identity, categorical/null/nonfinite/object inputs, mixed subject-ID types, coverage mismatches, and caller non-mutation. Focused tests pass at 23 cases; the shared analysis/plotting consumers pass at 163 tests with 2 skips. The five remaining full-module CodeCarbon setup errors reproduce unchanged on the exact integrated baseline.
The function is listed in the Statistics API section and the existing benchmark output is unchanged.
Fixes #733