Skip to content

Add Nadeau & Bengio corrected resampled t-test (closes #1075) - #1140

Merged
bruAristimunha merged 2 commits into
NeuroTechX:developfrom
azrabano23:feat/nadeau-bengio-corrected-ttest
Aug 24, 2026
Merged

Add Nadeau & Bengio corrected resampled t-test (closes #1075)#1140
bruAristimunha merged 2 commits into
NeuroTechX:developfrom
azrabano23:feat/nadeau-bengio-corrected-ttest

Conversation

@azrabano23

Copy link
Copy Markdown
Contributor

Closes #1075.

Within-session evaluation scores come from k-fold resamples whose training sets overlap, so the naive paired t-test underestimates the variance of the mean score difference and is overconfident. This PR adds the corrected resampled t-test of Nadeau & Bengio, which inflates the variance by the test/train ratio n2/n1 of each split:

t = ( (1/n) Σ xⱼ ) / sqrt( (1/n + n2/n1) σ̂² ), with t ~ Student(n−1) under H0.

What's included

  • moabb.analysis.meta_analysis.compute_pvals_corrected_ttest(df, n_train, n_test, order=None): same interface and one-tailed k×k p-value convention as compute_pvals_wilcoxon / compute_pvals_perm, so it drops into the existing meta-analysis machinery (p-values kept strictly inside (0, 1) for Stouffer combination).
  • Vectorized private core _corrected_resampled_ttest, following the _pairedttest_* style.
  • 3 unit tests (synthetic data): exact agreement with the hand-computed formula and the algebraic shrinkage relation to scipy.stats.ttest_rel; conservativeness vs the uncorrected test with monotonicity in n2/n1; matrix properties, degenerate-variance edge cases, and input validation.
  • API docs entry and changelog.

References: Nadeau & Bengio, Inference for the Generalization Error, NeurIPS 1999 / Machine Learning 52:239–281 (2003); Bouckaert & Frank, PAKDD 2004.

Scope note: per @bruAristimunha's comment on #1075, the natural follow-up is a splitter/evaluation hook that records n_train/n_test per fold and calls this test automatically from compute_dataset_statistics for within-session results — I kept this PR focused on the statistic itself and am happy to do the splitter integration as a follow-up if that split of work suits you.

@azrabano23
azrabano23 force-pushed the feat/nadeau-bengio-corrected-ttest branch from e911332 to a920bb9 Compare August 24, 2026 18:13
@bruAristimunha
bruAristimunha requested a review from gcattan August 24, 2026 18:38
@bruAristimunha

Copy link
Copy Markdown
Collaborator

many thanks for the contribution, can you update the tutorial too? 🙏🏽

@bruAristimunha
bruAristimunha merged commit 57ee66f into NeuroTechX:develop Aug 24, 2026
13 of 14 checks passed
@bruAristimunha

Copy link
Copy Markdown
Collaborator

many thanks @azrabano23 🙏🏽

@gcattan

gcattan commented Aug 25, 2026

Copy link
Copy Markdown
Collaborator

@bruAristimunha @azrabano23 I think we can make it a default in compute_dataset_statistic, and maybe deprecate the other methods.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement Nadeau & Bangio Permutation-testing

3 participants