Skip to content

Add a score on the lowest performing subjects - #1133

Merged
bruAristimunha merged 4 commits into
NeuroTechX:developfrom
adityasingh2400:feat/lowest-performing-subject-score
Aug 25, 2026
Merged

Add a score on the lowest performing subjects#1133
bruAristimunha merged 4 commits into
NeuroTechX:developfrom
adityasingh2400:feat/lowest-performing-subject-score

Conversation

@adityasingh2400

@adityasingh2400 adityasingh2400 commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

This adds moabb.analysis.meta_analysis.compute_lowest_subject_scores, the metric requested in #733.

The helper now requires an explicit reference_pipeline. For each dataset it macro-averages the supplied session scores per subject, selects the lowest percentile cohort once from that reference pipeline, and evaluates every pipeline on those exact same subject identities. It returns one row per observed (dataset, pipeline) with the resulting score and common n_subjects.

Using a shared reference cohort is the scientifically important part: selecting the worst subjects independently for each pipeline can compare different populations and even reverse a conclusion. The fixed-reference construction follows the hardest-subject comparison described by Gnassounou, Collas, Flamary, and Gramfort in PSDNorm: Test-Time Temporal Normalization for Deep Learning in Sleep Staging (arXiv:2503.04582). This helper operates on supplied scores; it does not recompute subject-level F1 from predictions.

The retained count is ceil(n_subjects * percentile / 100) and never below one. Score ties use stable input order, so heterogeneous subject identifiers do not need to be compared. Every pipeline must have exactly the reference subject set in each dataset. Empty input, raw NaN/infinity, null cohort identifiers, incomplete subject coverage, a missing reference, and percentiles outside (0, 100] fail closed. Filtered categorical frames use observed levels only, and coercible numeric score arrays are handled on a copy without mutating the caller.

Coverage includes the conclusion-reversal counterexample, per-dataset reference selection, session macro-aggregation, counts and 100% identity, categorical/null/nonfinite/object inputs, mixed subject-ID types, coverage mismatches, and caller non-mutation. Focused tests pass at 23 cases; the shared analysis/plotting consumers pass at 163 tests with 2 skips. The five remaining full-module CodeCarbon setup errors reproduce unchanged on the exact integrated baseline.

The function is listed in the Statistics API section and the existing benchmark output is unchanged.

Fixes #733

adityasingh2400 and others added 4 commits August 5, 2026 05:33
compute_lowest_subject_scores ranks the subjects of every (dataset,
pipeline) pair and averages only the lowest percentile of them, which is
the family of metrics F1@20% belongs to.

ERP, SSVEP and c-VEP saturate close to a perfect score on most subjects,
so a mean over the whole cohort is dominated by subjects that no longer
separate pipelines. The subjects a pipeline handles worst are where the
remaining headroom is.

Fixes NeuroTechX#733

@bruAristimunha bruAristimunha left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved at exact head 6dccccd after an independent full-range review and fix-round re-review. The statistic now selects one explicit reference cohort per dataset, evaluates every pipeline on identical subjects, rejects raw non-finite scores/null identities/incomplete coverage, uses observed categorical groups, and supports heterogeneous subject IDs. Verification: 23 focused tests, 67 non-CodeCarbon analysis tests, 163 shared analysis/plotting consumer tests (2 skipped), scoped pre-commit, and diff checks passed. The five remaining full-module CodeCarbon setup errors reproduce unchanged on the exact integrated baseline. No unresolved review threads.

@bruAristimunha
bruAristimunha merged commit 4b6ff02 into NeuroTechX:develop Aug 25, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Score on lowest performing subjects

2 participants