[CI] Add workflow verifying build against minimum supported PyTorch and JAX - #3396
Conversation
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
… 8.9 Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
…1 needs c10::optional Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
…Make Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
… c10::Stream caster Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
…nd lacks c10::Stream caster" This reverts commit 04b4b0f. Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
…torch 2.1 needs c10::optional" This reverts commit 061834f. Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Greptile SummaryThe PR adds CI jobs that build and sanity-import Transformer Engine against the selected minimum PyTorch and JAX versions. It also aligns NCCL EP architecture gating and makes zero-copy support conditional on the installed PyTorch headers.
Confidence Score: 5/5The PR appears safe to merge because no blocking failure remains. No blocking failure remains. Important Files Changed
Reviews (6): Last reviewed commit: "Compile out EP zero-copy path when torch..." | Re-trigger Greptile |
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
…ild min-torch job on sm90 Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
… import Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
|
Added @phu0ngng as a reviewer since EP raised pytorch version to 2.11, please look at the pytorch build part if this is ok. |
|
Also note we do not support versions we declare in build for JAX 0.5.3 is needed instead of 0.5.0 |
Per review: only the zero-copy path needs torch's c10d symm-mem (2.11); guard the includes with __has_include so the rest of EP builds on older torch, expose ep_zero_copy_supported() and raise in ep_bootstrap when zero_copy=True without support. Drop the build-time torch gate. Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
| env: | ||
| MIN_TORCH_VERSION: "2.8.0" | ||
| MIN_JAX_VERSION: "0.5.3" |
There was a problem hiding this comment.
Is this tracked in documentation anywhere today?
Description
Adds a GitHub Actions workflow (
minimum_versions.yml) that builds TE against pinned minimum framework versions and runs the sanity imports. The workflow versions are set in a single place (MIN_TORCH_VERSION: 2.8.0,MIN_JAX_VERSION: 0.5.3). This guards the minimum framework versions selected for CI; the older package requirement declarations are left unchanged for now.Context — the declared lower bounds are older than the CI-supported floors (verified empirically and against pytorch git history):
build_tools/pytorch.pycurrently declarestorch>=2.1and remains unchanged in this PR. Current main does not compile against torch 2.1 (std::optionalvsc10::optionalin generator APIs — alias exists since 2.2; usablec10::Streampybind caster since 2.4) and does not import below 2.6 (torch.distributed.fsdp._fully_shardmoved there in 2.6,torch.distributed.tensor.DTensorexport since 2.5). The minimum PyTorch version supported and enforced by this workflow is 2.8.0.pyproject.tomlcurrently declaresjax>=0.5.0and remains unchanged in this PR, buttransformer_engine/jax/sharding.pyimportsjax.sharding.get_abstract_mesh, which only exists since jax 0.5.3. The build compiles against 0.5.0 but the import fails; the minimum JAX version supported and enforced by this workflow is 0.5.3.The declared package requirements can be reconciled separately.
The PyTorch job builds
NVTE_CUDA_ARCHS=90so the NCCL EP path is exercised; the JAX job builds sm70. Both useMAX_JOBS=2(the 7GB public runner OOMs at full parallelism) and sccache backed by the GitHub Actions cache (~10 min warm). The sm90 sanity import uses the CUDA toolkit'slibcuda.sostub since the runner has no GPU driver (the EP-enabled core library links the driver via NCCL).Building sm90 against an older torch surfaced two gaps in the NCCL EP build gating, fixed here:
NVTE_WITH_NCCL_EPbased only on the env var (default on), while the common library gates NCCL EP on having an arch >= 90 in the target list. A build targeting only archs < 90 compiled the EP extension against a core library with nonvte_ep_*symbols. The extension now uses the samenccl_ep_enabled()gate as the common CMake side.ep.cppincludestorch/csrc/distributed/c10d/symm_mem/nccl_dev_cap.hpp/NCCLSymmetricMemory.hppand usesget_window()/get_offset(), all of which only exist since torch 2.11 — i.e. a default source build on Hopper+ hard-failed for any older torch. The extension now auto-skips NCCL EP (with a log message) when the installed torch lacks the symm-mem headers; an explicitNVTE_WITH_NCCL_EP=1raises a clear error instead. Builds against torch >= 2.11 are unaffected.Type of change
Changes
.github/workflows/minimum_versions.ymlwith two jobs, onpull_request/workflow_dispatch:torch==2.8.0on sm90 (NCCL EP path exercised) + sanity importjax==0.5.3(CPU jaxlib) on sm70 + sanity importbuild_tools/pytorch.py: gateNVTE_WITH_NCCL_EP/USE_NCCLdefines onnccl_ep_enabled()(matching the common CMake gate) and on the installed torch providing the c10d symm-mem headers (torch >= 2.11); auto-skip with a message, error only when EP was explicitly requestedChecklist:
🤖 Generated with Claude Code