[ericspod]

Fixes # .

Description

This adds a testing runner for the KCL GPU runner available to MONAI. The goal is to develop a GPU testing process, either as a per-PR test or nightly GPU tests.

Types of changes

• Non-breaking change (fix or new feature that would not break existing functionality).
• Breaking change (fix or new feature that would cause existing functionality to change).
• New tests added to cover the changes.
• Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.
• Quick tests passed locally by running ./runtests.sh --quick --unittests --disttests.
• In-line docstrings updated.
• Documentation updated, tested make html command in the docs/ folder.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Tip

🎯 Customize your risk assessments

Guide how CodeRabbit assesses merge risk and security architecture risk. Highlight sensitive changes and explain your security boundaries using Merge Risk Instructions and Security Risk Instructions in your review settings. Security architecture risk assessments require the Advanced plan or above.

✅ Automatic approvals, with your rules

Let CodeRabbit approve pull requests once required threads are resolved, pre-merge checks pass, and your risk limits are met. Enable Automatic approvals in review settings, with overrides for teams or file paths.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

• X
• Mastodon
• Reddit
• LinkedIn

[vikasreddy11]

Hi @ericspod , I'm interested in helping with the GPU test runner. I noticed the PR is still in progress. Is there a specific area where an independent investigation or contribution would be useful?

[ericspod]

Thanks @vikasreddy11 do you have access to a GPU? I could use some help testing the GPU resampling accuracy with the compiled extensions. I'm seeing affine tests failing like this

Details

FAIL: test_affine_003 (tests.transforms.test_affine.TestAffine.test_affine_003)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/parameterized/parameterized.py", line 620, in standalone_func
    return func(*(a + p.args), **p.kwargs, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/monai/tests/transforms/test_affine.py", line 197, in test_affine
    test_resampler_lazy(
  File "/opt/monai/tests/lazy_transforms_utils.py", line 77, in test_resampler_lazy
    assert_allclose(lazy_out, non_lazy_out, rtol=rtol, atol=atol)
  File "/opt/monai/tests/test_utils.py", line 167, in assert_allclose
    np.testing.assert_allclose(actual, desired, *args, **kwargs)
  File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 1688, in assert_allclose
    assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
  File "/usr/lib/python3.12/contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 889, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Not equal to tolerance rtol=0.001, atol=0.001

Mismatched elements: 15 / 16 (93.8%)
Max absolute difference among violations: 1.3333333
Max relative difference among violations: 1.7777779
 ACTUAL: array([[[0.      , 0.231481, 0.601852, 0.694444],
        [0.462963, 0.833333, 1.277778, 1.296296],
        [1.203704, 1.722222, 2.166667, 2.037037],
        [1.388889, 1.898148, 2.268518, 2.083333]]], dtype=float32)
 DESIRED: array([[[0.      , 0.083333, 0.416667, 0.25    ],
        [0.166667, 0.5     , 1.166667, 0.666667],
        [0.833333, 1.833333, 2.5     , 1.333333],
        [0.5     , 1.083333, 1.416667, 0.75    ]]], dtype=float32)

This occurs in tests.transforms.test_affine and tests.transforms.test_affined. If you can try testing these files in the Docker image on in a similar environment with the CUDA extensions built and get to the bottom of the issue that would go a long way for the testing process. Be sure to have BUILD_MONAI=1 exported even when you run the tests to be sure compiled code is used. I encountered some other issues here that's related to the testing host I think that only I can sort out.

[vikasreddy11]

Thanks @ericspod, I tested on a GPU outside Docker (RTX 3050 Laptop, WSL2, Python 3.12.15, PyTorch 2.14.1+cu132, dev at cb35d45).

I can reproduce it, and it depends on the compiled extension:

• Default (USE_COMPILED=False): test_affine.py gives 125 passed, 0 failed.
• BUILD_MONAI=1 (USE_COMPILED=True): 72 failed, including test_affine_003. test_affine_116 fails alone and via unittest with the same mismatch numbers as yours.
• The failure is at the final pixel comparison in test_resampler_lazy (lazy_transforms_utils.py:77), after the pending-affine and shape assertions have passed. So the lazy metadata matches eager, and only the values resampled by apply_pending differ.

Not yet checked: test_affined, and PyTorch 2.9.x (mine is newer than CI's range). I'll report back.

[vikasreddy11]

Update: test_affined.py behaves the same way. Default (USE_COMPILED=False): 120 passed, 0 failed. BUILD_MONAI=1 (USE_COMPILED=True): 60 failed, 60 passed.

I also made a minimal standalone reproduction mirroring test_resampler_lazy (90° rotation, arange(4).reshape(1,2,2) to a 4x4 output, zero padding, align_corners=True). Eager output is identical in both modes, but lazy (via apply_pending with the test's overrides) differs only when the compiled extension is active, with the same values as the ACTUAL array in the test_affine_116 failure.

It is not GPU-specific: running the same script with CUDA_VISIBLE_DEVICES="" and BUILD_MONAI=1 gives the same wrong lazy values on CPU, so it should be reproducible without a GPU.

With align_corners=False eager and lazy match, but in lazy resample (lazy/utils.py:207) that case takes the permute/flip shortcut and never reaches SpatialResample, so it doesn't exercise the resampling.

I suspect the compiled branch of spatial_resample (spatial/functional.py ~L212) but haven't confirmed it. Still to check: PyTorch 2.9.x.

import numpy as np, torch
from monai.data import MetaTensor
from monai.transforms import Affine
from monai.transforms.lazy.functional import apply_pending
from tests.lazy_transforms_utils import get_apply_param

img = MetaTensor(torch.arange(4, dtype=torch.float32).reshape(1, 2, 2))
rot = torch.tensor([[0., -1., 0.], [1., 0., 0.], [0., 0., 1.]])
init = dict(affine=rot, padding_mode="zeros", align_corners=True)
call = dict(img=img, spatial_size=(4, 4))
eager = Affine(**init)(**call)
eager = eager[0] if isinstance(eager, tuple) else eager
t = Affine(**init); t.lazy = True
out = t(**call)
out = out[0] if isinstance(out, tuple) else out
out = apply_pending(out, overrides=get_apply_param(init, call))[0]
print(np.round(np.asarray(eager), 4)); print(np.round(np.asarray(out), 4))

Run with PYTHONPATH=. from the repo root, plain and with BUILD_MONAI=1.

I'd like to dig into this. If I find a fix, would you like me to open a PR, or is someone already working on this?

[ericspod]

Hi @vikasreddy11 thanks for looking into the issue. If you could please find a fix and propose a PR if you do, no one else is working on it currently. These fails are the major issues we're encountering with the GPU tests but I'm working on others currently. The root cause hasn't been found since the PR tests compile the code but then don't use it since BUILD_MONAI=1 must be present even at runtime, this behaviour is something I'll want to change here.

[ericspod]

The issue with the GlobalMutualInformationLoss tests is that torch.bmm tries to compile with Triton when used with 1D matrices and fails because the container for the action doesn't have a compiler. This example raises the issue in PyTorch 2.14.1:

import torch
t = torch.rand(1, 10, 1).cuda()
tt = torch.rand(1, 1, 10).cuda()
torch.bmm(t, tt)

This raises "RuntimeError: Failed to find C compiler." on line 364 of image_dissimilarity.py. I would expect PyTorch to fall back to a native computation, the solution here is to move the inputs to CPU then move the result back to CUDA. Another solution is a native matmul function just for this loss:

t = torch.rand(1, 10, 1).cuda()
tt = torch.rand(1, 1, 10).cuda()

def matmul_native(a, b):  #(B,N,1) (B,1,M)
    return a.repeat(1, 1, b.shape[2]) * b.repeat(1, a.shape[1], 1)

matmul_native(t,tt)

[vikasreddy11]

With BUILD_MONAI=1, spatial_resample's compiled branch (Affine(normalized=True, align_corners=...) feeding grid_pull) disagrees with the non-compiled branch (AffineTransform + grid_sample) for align_corners=True, on identical inputs. I replayed the same recorded call through both branches (max abs diff):

• lazy RandRotate (2D, nearest, zeros): 3038 (this is what fails 132 tests in test_affine/test_affined)
• eager Spacing, 8x8x8 -> pixdim (0.7, 1.3, 1.1): 0.197
• eager Spacing, 9x7x5 -> pixdim (1.5, 0.8, 2.0): 0.531

The pending affine and the matrix passed to resample() are identical for True and False, and the lazy grid_pull grid equals eager's grid up to an exact affine map (residual ~1e-13) matching S = diag((d_in-1)/d_in), D = diag(d_out/(d_out-1)).

Forcing align_corners=False in the compiled branch takes test_affine/test_affined from 132 failed to 0, but it breaks test_spacing_08/09 (1.0 vs 0.75), which pass today in both modes. So those tests seem to encode the current compiled behaviour, and I don't want to change it without your view.

Question: for align_corners=True, which should be the reference, the native grid_sample path or the compiled one? If native, the compiled branch needs fixing and some Spacing expectations may change. If compiled, the lazy Affine/Rotate side is what's off. I can open a PR once you say which.

[ericspod]

Thanks again @vikasreddy11 I would say the native branch is the reference and not the compiled. I don't think the compiled code is used often as I suspect most users don't build from the repo. It should be fixed however for those that do if not just for these tests. Please propose your PR to fix the compiled code and the tests you mention, if they are degenerate they can be updated to fix this I hope.

[vikasreddy11]

Opened #9248 with the fix for the compiled lazy align_corners=True failures (test_affine/test_affined: 132 failed -> 0 with BUILD_MONAI=1). All checks pass. The only test expectation changed is _TEMPLATE_5_COMPILED in test_spacing (0.75 -> 1.0), with the reasoning in the PR description.

[ericspod]

We currently have a core dump on the tests for the CRF block. I can't reproduce this in a local MONAI slim as all tests pass, and have encountered the following testing with ROCm:

FAIL: test_1_2_batche_s_1_dimension_s_2_classe_s_1_channel_s_with_matrix (__main__.CRFTestCaseCuda.test_1_2_batche_s_1_dimension_s_2_classe_s_1_channel_s_with_matrix)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/parameterized/parameterized.py", line 620, in standalone_func
    return func(*(a + p.args), **p.kwargs, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/shared-docker/MONAI/tests/networks/blocks/test_crf_cuda.py", line 524, in test
    self.assertLessEqual(mismatch_count / len(output), mismatch_ratio_tolerance)
AssertionError: np.float64(0.5) not less than or equal to 0.1

======================================================================
FAIL: test_2_1_batche_s_2_dimension_s_3_classe_s_2_channel_s_ (__main__.CRFTestCaseCuda.test_2_1_batche_s_2_dimension_s_3_classe_s_2_channel_s_)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/parameterized/parameterized.py", line 620, in standalone_func
    return func(*(a + p.args), **p.kwargs, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/shared-docker/MONAI/tests/networks/blocks/test_crf_cuda.py", line 524, in test
    self.assertLessEqual(mismatch_count / len(output), mismatch_ratio_tolerance)
AssertionError: np.float64(0.10666666666666667) not less than or equal to 0.1

Something is still an issue with the compiled code that needs to be addressed. @vikasreddy11 if you're up for investigating this further we'd have to somehow reproduce the core dump, the above tests are a harder one to nail down. Thanks!

[vikasreddy11]

Happy to investigate further. I don't have ROCm hardware, but I tried reproducing on CUDA (RTX 3050, WSL2, PyTorch 2.14.1+cu132). I ran tests/networks/blocks/test_crf_cuda.py with BUILD_MONAI=1: 4 tests ran and passed, and there was no core dump. So I can't reproduce either the ROCm mismatch-ratio failures or the crash on CUDA, though that doesn't rule them out on ROCm.

Could you share the GPU model and runtime details of the CI runner (CUDA/ROCm version, PyTorch version) and which test triggers the core dump? I'll use that to narrow it down.

[ericspod]

Could you share the GPU model and runtime details of the CI runner (CUDA/ROCm version, PyTorch version) and which test triggers the core dump? I'll use that to narrow it down.

Hi @vikasreddy11 I'm trying to narrow it down now too within the runner as I can't reproduce outside either. Have a look at the changes to permutohedral.cpp where I'm trying to get debug information out. Information about the environment is in the logs of the check, so when full-dep-gpu fails you can have a look at what it says.

[ericspod]

@vikasreddy11 I think I may have figured out the issue, we weren't actually compiling CUDA when I thought we were. I'm getting the following build warnings however, could you figure out the tweaks to setup.py to address these?

Warnings

[9](https://github.com/Project-MONAI/MONAI/actions/runs/42226511679/job/127921266274?pr=9112#step:7:940)
  /home/runner/actions-runner/_work/_tool/Python/3.11.18/x64/lib/python3.11/site-packages/setuptools/command/build_py.py:212: BuildBackendWarning: Package 'monai._extensions.gmm' is absent from the `packages` configuration.
  !!
  
          ********************************************************************************
          ############################
          # Package would be ignored #
          ############################
          Python recognizes 'monai._extensions.gmm' as an importable package[^1],
          but it is absent from setuptools' `packages` configuration.
  
          This leads to an ambiguous overall configuration. If you want to distribute this
          package, please make sure that 'monai._extensions.gmm' is explicitly added
          to the `packages` configuration field.
  
          Alternatively, you can also rely on setuptools' discovery methods
          (for example by using `find_namespace_packages(...)`/`find_namespace:`
          instead of `find_packages(...)`/`find:`).
  
          You can read more about "package discovery" on setuptools documentation page:
  
          - https://setuptools.pypa.io/en/latest/userguide/package_discovery.html
  
          If you don't want 'monai._extensions.gmm' to be distributed and are
          already explicitly excluding 'monai._extensions.gmm' via
          `find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`,
          you can try to use `exclude_package_data`, or `include-package-data=False` in
          combination with a more fine grained `package-data` configuration.
  
          You can read more about "package data files" on setuptools documentation page:
  
          - https://setuptools.pypa.io/en/latest/userguide/datafiles.html
  
  
          [^1]: For Python, any directory (with suitable naming) can be imported,
                even if it does not contain any `.py` files.
                On the other hand, currently there is no concept of package data
                directory, all directories are treated like packages.
          ********************************************************************************
  
  !!
    check.warn(importable)
  /home/runner/actions-runner/_work/_tool/Python/3.11.18/x64/lib/python3.11/site-packages/setuptools/command/build_py.py:212: BuildBackendWarning: Package 'monai.csrc' is absent from the `packages` configuration.
  !!
  
          ********************************************************************************
          ############################
          # Package would be ignored #
          ############################
          Python recognizes 'monai.csrc' as an importable package[^1],
          but it is absent from setuptools' `packages` configuration.
  
          This leads to an ambiguous overall configuration. If you want to distribute this
          package, please make sure that 'monai.csrc' is explicitly added
          to the `packages` configuration field.
  
          Alternatively, you can also rely on setuptools' discovery methods
          (for example by using `find_namespace_packages(...)`/`find_namespace:`
          instead of `find_packages(...)`/`find:`).
  
          You can read more about "package discovery" on setuptools documentation page:
  
          - https://setuptools.pypa.io/en/latest/userguide/package_discovery.html
  
          If you don't want 'monai.csrc' to be distributed and are
          already explicitly excluding 'monai.csrc' via
          `find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`,
          you can try to use `exclude_package_data`, or `include-package-data=False` in
          combination with a more fine grained `package-data` configuration.
  
          You can read more about "package data files" on setuptools documentation page:
  
          - https://setuptools.pypa.io/en/latest/userguide/datafiles.html
  
  
          [^1]: For Python, any directory (with suitable naming) can be imported,
                even if it does not contain any `.py` files.
                On the other hand, currently there is no concept of package data
                directory, all directories are treated like packages.
          ********************************************************************************
  
  !!
    check.warn(importable)
  /home/runner/actions-runner/_work/_tool/Python/3.11.18/x64/lib/python3.11/site-packages/setuptools/command/build_py.py:212: BuildBackendWarning: Package 'monai.csrc.filtering.bilateral' is absent from the `packages` configuration.
  !!
  
          ********************************************************************************
          ############################
          # Package would be ignored #
          ############################
          Python recognizes 'monai.csrc.filtering.bilateral' as an importable package[^1],
          but it is absent from setuptools' `packages` configuration.
  
          This leads to an ambiguous overall configuration. If you want to distribute this
          package, please make sure that 'monai.csrc.filtering.bilateral' is explicitly added
          to the `packages` configuration field.
  
          Alternatively, you can also rely on setuptools' discovery methods
          (for example by using `find_namespace_packages(...)`/`find_namespace:`
          instead of `find_packages(...)`/`find:`).
  
          You can read more about "package discovery" on setuptools documentation page:
  
          - https://setuptools.pypa.io/en/latest/userguide/package_discovery.html
  
          If you don't want 'monai.csrc.filtering.bilateral' to be distributed and are
          already explicitly excluding 'monai.csrc.filtering.bilateral' via
          `find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`,
          you can try to use `exclude_package_data`, or `include-package-data=False` in
          combination with a more fine grained `package-data` configuration.
  
          You can read more about "package data files" on setuptools documentation page:
  
          - https://setuptools.pypa.io/en/latest/userguide/datafiles.html
  
  
          [^1]: For Python, any directory (with suitable naming) can be imported,
                even if it does not contain any `.py` files.
                On the other hand, currently there is no concept of package data
                directory, all directories are treated like packages.
          ********************************************************************************
  
  !!
    check.warn(importable)
  /home/runner/actions-runner/_work/_tool/Python/3.11.18/x64/lib/python3.11/site-packages/setuptools/command/build_py.py:212: BuildBackendWarning: Package 'monai.csrc.filtering.permutohedral' is absent from the `packages` configuration.
  !!
  
          ********************************************************************************
          ############################
          # Package would be ignored #
          ############################
          Python recognizes 'monai.csrc.filtering.permutohedral' as an importable package[^1],
          but it is absent from setuptools' `packages` configuration.
  
          This leads to an ambiguous overall configuration. If you want to distribute this
          package, please make sure that 'monai.csrc.filtering.permutohedral' is explicitly added
          to the `packages` configuration field.
  
          Alternatively, you can also rely on setuptools' discovery methods
          (for example by using `find_namespace_packages(...)`/`find_namespace:`
          instead of `find_packages(...)`/`find:`).
  
          You can read more about "package discovery" on setuptools documentation page:
  
          - https://setuptools.pypa.io/en/latest/userguide/package_discovery.html
  
          If you don't want 'monai.csrc.filtering.permutohedral' to be distributed and are
          already explicitly excluding 'monai.csrc.filtering.permutohedral' via
          `find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`,
          you can try to use `exclude_package_data`, or `include-package-data=False` in
          combination with a more fine grained `package-data` configuration.
  
          You can read more about "package data files" on setuptools documentation page:
  
          - https://setuptools.pypa.io/en/latest/userguide/datafiles.html
  
  
          [^1]: For Python, any directory (with suitable naming) can be imported,
                even if it does not contain any `.py` files.
                On the other hand, currently there is no concept of package data
                directory, all directories are treated like packages.
          ********************************************************************************
  
  !!
    check.warn(importable)
  /home/runner/actions-runner/_work/_tool/Python/3.11.18/x64/lib/python3.11/site-packages/setuptools/command/build_py.py:212: BuildBackendWarning: Package 'monai.csrc.filtering.trainable_bilateral' is absent from the `packages` configuration.
  !!
  
          ********************************************************************************
          ############################
          # Package would be ignored #
          ############################
          Python recognizes 'monai.csrc.filtering.trainable_bilateral' as an importable package[^1],
          but it is absent from setuptools' `packages` configuration.
  
          This leads to an ambiguous overall configuration. If you want to distribute this
          package, please make sure that 'monai.csrc.filtering.trainable_bilateral' is explicitly added
          to the `packages` configuration field.
  
          Alternatively, you can also rely on setuptools' discovery methods
          (for example by using `find_namespace_packages(...)`/`find_namespace:`
          instead of `find_packages(...)`/`find:`).
  
          You can read more about "package discovery" on setuptools documentation page:
  
          - https://setuptools.pypa.io/en/latest/userguide/package_discovery.html
  
          If you don't want 'monai.csrc.filtering.trainable_bilateral' to be distributed and are
          already explicitly excluding 'monai.csrc.filtering.trainable_bilateral' via
          `find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`,
          you can try to use `exclude_package_data`, or `include-package-data=False` in
          combination with a more fine grained `package-data` configuration.
  
          You can read more about "package data files" on setuptools documentation page:
  
          - https://setuptools.pypa.io/en/latest/userguide/datafiles.html
  
  
          [^1]: For Python, any directory (with suitable naming) can be imported,
                even if it does not contain any `.py` files.
                On the other hand, currently there is no concept of package data
                directory, all directories are treated like packages.
          ********************************************************************************
  
  !!
    check.warn(importable)
  /home/runner/actions-runner/_work/_tool/Python/3.11.18/x64/lib/python3.11/site-packages/setuptools/command/build_py.py:212: BuildBackendWarning: Package 'monai.csrc.filtering.trainable_joint_bilateral' is absent from the `packages` configuration.
  !!
  
          ********************************************************************************
          ############################
          # Package would be ignored #
          ############################
          Python recognizes 'monai.csrc.filtering.trainable_joint_bilateral' as an importable package[^1],
          but it is absent from setuptools' `packages` configuration.
  
          This leads to an ambiguous overall configuration. If you want to distribute this
          package, please make sure that 'monai.csrc.filtering.trainable_joint_bilateral' is explicitly added
          to the `packages` configuration field.
  
          Alternatively, you can also rely on setuptools' discovery methods
          (for example by using `find_namespace_packages(...)`/`find_namespace:`
          instead of `find_packages(...)`/`find:`).
  
          You can read more about "package discovery" on setuptools documentation page:
  
          - https://setuptools.pypa.io/en/latest/userguide/package_discovery.html
  
          If you don't want 'monai.csrc.filtering.trainable_joint_bilateral' to be distributed and are
          already explicitly excluding 'monai.csrc.filtering.trainable_joint_bilateral' via
          `find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`,
          you can try to use `exclude_package_data`, or `include-package-data=False` in
          combination with a more fine grained `package-data` configuration.
  
          You can read more about "package data files" on setuptools documentation page:
  
          - https://setuptools.pypa.io/en/latest/userguide/datafiles.html
  
  
          [^1]: For Python, any directory (with suitable naming) can be imported,
                even if it does not contain any `.py` files.
                On the other hand, currently there is no concept of package data
                directory, all directories are treated like packages.
          ********************************************************************************
  
  !!
    check.warn(importable)
  /home/runner/actions-runner/_work/_tool/Python/3.11.18/x64/lib/python3.11/site-packages/setuptools/command/build_py.py:212: BuildBackendWarning: Package 'monai.csrc.lltm' is absent from the `packages` configuration.
  !!
  
          ********************************************************************************
          ############################
          # Package would be ignored #
          ############################
          Python recognizes 'monai.csrc.lltm' as an importable package[^1],
          but it is absent from setuptools' `packages` configuration.
  
          This leads to an ambiguous overall configuration. If you want to distribute this
          package, please make sure that 'monai.csrc.lltm' is explicitly added
          to the `packages` configuration field.
  
          Alternatively, you can also rely on setuptools' discovery methods
          (for example by using `find_namespace_packages(...)`/`find_namespace:`
          instead of `find_packages(...)`/`find:`).
  
          You can read more about "package discovery" on setuptools documentation page:
  
          - https://setuptools.pypa.io/en/latest/userguide/package_discovery.html
  
          If you don't want 'monai.csrc.lltm' to be distributed and are
          already explicitly excluding 'monai.csrc.lltm' via
          `find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`,
          you can try to use `exclude_package_data`, or `include-package-data=False` in
          combination with a more fine grained `package-data` configuration.
  
          You can read more about "package data files" on setuptools documentation page:
  
          - https://setuptools.pypa.io/en/latest/userguide/datafiles.html
  
  
          [^1]: For Python, any directory (with suitable naming) can be imported,
                even if it does not contain any `.py` files.
                On the other hand, currently there is no concept of package data
                directory, all directories are treated like packages.
          ********************************************************************************
  
  !!
    check.warn(importable)
  /home/runner/actions-runner/_work/_tool/Python/3.11.18/x64/lib/python3.11/site-packages/setuptools/command/build_py.py:212: BuildBackendWarning: Package 'monai.csrc.resample' is absent from the `packages` configuration.
  !!
  
          ********************************************************************************
          ############################
          # Package would be ignored #
          ############################
          Python recognizes 'monai.csrc.resample' as an importable package[^1],
          but it is absent from setuptools' `packages` configuration.
  
          This leads to an ambiguous overall configuration. If you want to distribute this
          package, please make sure that 'monai.csrc.resample' is explicitly added
          to the `packages` configuration field.
  
          Alternatively, you can also rely on setuptools' discovery methods
          (for example by using `find_namespace_packages(...)`/`find_namespace:`
          instead of `find_packages(...)`/`find:`).
  
          You can read more about "package discovery" on setuptools documentation page:
  
          - https://setuptools.pypa.io/en/latest/userguide/package_discovery.html
  
          If you don't want 'monai.csrc.resample' to be distributed and are
          already explicitly excluding 'monai.csrc.resample' via
          `find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`,
          you can try to use `exclude_package_data`, or `include-package-data=False` in
          combination with a more fine grained `package-data` configuration.
  
          You can read more about "package data files" on setuptools documentation page:
  
          - https://setuptools.pypa.io/en/latest/userguide/datafiles.html
  
  
          [^1]: For Python, any directory (with suitable naming) can be imported,
                even if it does not contain any `.py` files.
                On the other hand, currently there is no concept of package data
                directory, all directories are treated like packages.
          ********************************************************************************

[vikasreddy11]

Thanks for the update. I can try to reproduce the packaging warnings locally without a CUDA build and look at the setup.py discovery config. I'll report back with what I find.