Skip to content

fix(sampler): renormalize only present domains - #514

Open
dajiaohuang wants to merge 1 commit into
alibaba:mainfrom
dajiaohuang:fix/empty-domain-ratio-pruning
Open

dajiaohuang wants to merge 1 commit into
alibaba:mainfrom
dajiaohuang:fix/empty-domain-ratio-pruning

Conversation

@dajiaohuang

Copy link
Copy Markdown

Summary

BatchStratifiedSampler pruned absent domains from self.domain_ratios, then rebuilt the mapping from the original argument. This reintroduced empty domains, causing iteration to divide by zero. Normalize the remaining weights after pruning so their relative proportions are preserved.

Fixes #495.

PR #498 was closed unmerged by its author; the issue still reproduces on current main.

Validation

  • py -3.13 -m pytest -q -p no:cacheprovider tests/datasets/test_sampler.py — 7 passed (synthetic sampler tests; two existing torch/CUDA-version warnings).
  • py -3.13 -m black --check roll/datasets/sampler.py tests/datasets/test_sampler.py — passed.
  • git diff --check — passed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BatchStratifiedSampler crashes with ZeroDivisionError when a domain_ratios entry has no matching rows

1 participant