Skip to content

Apply cross-validation settings - #1130

Merged
bruAristimunha merged 5 commits into
NeuroTechX:developfrom
stanbot8:apply-cross-validation-settings
Aug 25, 2026
Merged

Apply cross-validation settings#1130
bruAristimunha merged 5 commits into
NeuroTechX:developfrom
stanbot8:apply-cross-validation-settings

Conversation

@stanbot8

@stanbot8 stanbot8 commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

Correct cross-validation setting routing

This candidate preserves stanbot8's original contribution and normally
integrates current develop through post-#1158 commit
827040df0354b64516a2ef65ce0536b8a1b32225.

  • Candidate: b09074bc70aeec2c73e5e946f35a0ac9d18c361c
  • Candidate tree: b36e1d70a3b1ad94a0544c08ad4457d1db326aab
  • Integration merge: 6241572c4c30c6291adbfbab50d597088e05d13e
  • Live predecessor: 73f08385dd79978fdb04d94c0f1db17ad09c8c55

Closes #1110.

Behavior

Evaluation defaults and caller-supplied CV settings are resolved separately.
The effective public evaluation.cv_kwargs mapping is consulted on every
resolution, so mutating an individual key or replacing the mapping after
construction is honored without stale provenance.

Constructor capability is tri-state. Evaluation-injected defaults are sent
only to known-supported keywords, positional-only names never receive
implicit keyword defaults, and known-false colliding wrapper controls remain
wrapper-only. Explicit ordinary inner kwargs are retained for validation by
the real constructor, while opaque signatures retain explicit caller values
without guessing omitted defaults.
Wrapper controls such as shuffle and random_state stay bound to wrapper
state and reach the inner CV only when support is not known false.
CrossSubject calibration defaults remain internal rather than being mistaken
for caller intent.

splitter.cv_kwargs remains a truthful public mapping of direct and
evaluation-routed non-wrapper settings. The private resolution envelope is
removed before storage and serialization. Pickle round-trips preserve public
state and exact folds without leaking that envelope.

Exact scope

Against integrated develop, the final diff is exactly:

  • docs/source/whats_new.rst
  • moabb/evaluations/base.py
  • moabb/evaluations/evaluations.py
  • moabb/evaluations/splitters.py
  • moabb/tests/test_evaluations.py
  • moabb/tests/test_splits.py

The production change is confined to the three evaluation modules. The larger
test matrix uses real GroupKFold, GroupShuffleSplit, LeaveOneGroupOut,
variadic, positional-only, and opaque splitters across WithinSession,
WithinSubject, CrossSession, CrossSubject, and CrossDataset paths. It pins seed
precedence, callable groups, wrapper state, mutability, public kwargs,
serialization, and actual train/test group disjointness.

The grouped LearningCurveSplitter behavior merged in #1158 is unchanged and
has a separate adversarial pickle/fold preservation check.

Verification

focused real-splitter selector
48 passed

moabb/tests/test_evaluations.py (excluding network tests)
133 passed

moabb/tests/test_splits.py (excluding network tests)
98 passed

pre-commit on all six paths
all applicable hooks passed

adversarial pickle checks
8 evaluation-created splitters and the grouped #1158 LearningCurve path passed

The six-path binary diff SHA-256 against current develop is
bfac1f415678f794f725fb707bc723ddb5f921676153be0a18a40b59a1443b05.
The worktree was clean at the exact candidate. Fresh hosted tests, docs,
pre-commit, link checks, What's News, Braindecode integration, and exact-head
human approval remain required before the repository's squash-only merge.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5f8b23e3db

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread moabb/evaluations/evaluations.py Outdated
Comment thread moabb/evaluations/base.py Outdated
@stanbot8
stanbot8 marked this pull request as draft August 2, 2026 18:48
@stanbot8
stanbot8 force-pushed the apply-cross-validation-settings branch from 5f8b23e to 6b4d6e3 Compare August 11, 2026 00:22
@stanbot8 stanbot8 changed the title Apply cross-validation settings consistently Apply cross-validation settings Aug 11, 2026
@stanbot8
stanbot8 force-pushed the apply-cross-validation-settings branch from 0266412 to 6b4d6e3 Compare August 11, 2026 01:04
@stanbot8
stanbot8 force-pushed the apply-cross-validation-settings branch 2 times, most recently from 564f773 to 73f0838 Compare August 11, 2026 20:09
@stanbot8

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Can't wait for the next one!

Reviewed commit: 73f08385dd

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@stanbot8
stanbot8 marked this pull request as ready for review August 11, 2026 20:29

@bruAristimunha bruAristimunha left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved at exact head b09074b after independent exact-candidate review, bounded TDD repair, and separately approved replacement-body review. Hosted exact-head checks and the final squash-only merge guard remain required.

@bruAristimunha
bruAristimunha merged commit 5c228f6 into NeuroTechX:develop Aug 25, 2026
14 of 16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Inconsistent n_splits / cv_kwargs handling in BaseEvaluation._resolve_cv and Within* evaluations

2 participants