Skip to content

[Bug][CUDA][dlight] GPU Fallback calls sch.fuse on a zero loop extent and aborts with "Divide by zero" #20279

Closed

Description

@lackjava-creator

Environment

  • TVM version: main @ 48242ec33403f2b6e4fac6e763ca7a683fb9d5df (2026-09-03, reports 0.26.dev0); the same trigger fails on 0.26.0, and on 0.19.0 / 0.12.0 it surfaced as a ZeroDivisionError in Relay's pad_to_tensorcore.

  • Build / install: source build, USE_CUDA=ON, nvcc 12.2, LLVM 15

  • OS / Python: Ubuntu 24.04.3 LTS, x86_64, Python 3.10

  • GPU / driver: Tesla V100-DGXS-32GB, driver 535.309.01, sm_70

  • Target: cuda. opt_level 3 and 0 both fail. llvm is clean at both levels.

  • Pipeline coverage (relax.get_pipeline("default_build") performs no operator fusion, so all three are reported separately). Verified 2026-09-06 on 0.26.0, CUDA target — the abort happens at compile time, before any kernel launch:

    • get_pipeline("default_build") (no FuseOps/FuseTIR) — fails
    • get_pipeline("zero") (fuses) — fails
    • forced FuseOps + FuseTIR — fails
      (also confirmed on main @ 48242ec at opt_level 3 and 0)

Minimal reproducer

Single Neg node with input X : float32[4,0] — a zero extent that is not on the leading axis. model.onnx + feed.npz + run.py attached.

import numpy as np, onnx, tvm
from tvm import relax
from tvm.relax.frontend.onnx import from_onnx
model = onnx.load("model.onnx"); feed = dict(np.load("feed.npz"))     # X: float32[4,0]
mod = from_onnx(model, shape_dict={k: list(v.shape) for k, v in feed.items()}, keep_params_in_input=False)
with tvm.transform.PassContext(opt_level=3):
    tvm.compile(relax.get_pipeline("default_build")(mod), target="cuda")   # <-- raises here

Companion note: the leading-axis zero-extent case does not fail here — it compiles and then faults at run time. The full matrix is: 0 on the leading axis (or a 1-D empty tensor) → runtime cudaErrorIllegalAddress; 0 on a non-leading axis → this compile-time abort.

Expected vs actual

  • Expected: compilation succeeds and the result is an empty float32 tensor of shape (4,0). numpy, onnxruntime, and TVM's own llvm target all produce that.
  • Actual:
InternalError: Check failed: pb->value != 0 (0 vs. 0) : Divide by zero

raised during tvm.compile, at both opt_level=3 and opt_level=0.

Root cause (if known)

dlight's GPU Fallback rule calls sch.fuse, and Fuse reconstructs the index of a fused loop as floordiv(floormod(fused, next_lower), lower). Here next_lower == 0, because it is the extent of the fused loop, which is 0. The zero is therefore the compiler's own loop extent, not a divisor coming from the model — an internal invariant that the scheduling rule breaks on itself. llvm is clean, so the rule is the producer.

Never fixed. On 0.12/0.19 the same trigger hit pad_to_tensorcore in the Relay pipeline (ZeroDivisionError); the site moved in (0.19, main], the root cause did not.

Why this is a bug (not tolerance / not undefined behaviour)

A zero extent on a non-leading axis is legal in ONNX and defined by numpy; TVM's llvm target handles it. The abort is not a validation message about the user's model — it is an ICHECK inside a scheduling rule about an expression the rule itself constructed.

Related, not duplicate. Open issue #19922 and closed #17745 hit the same ICHECK (pb->value != 0), but from has_reshape_pattern in the Relax analysis on the llvm target. That producer was fixed by merged PR #19958 (2026-07-09), and our llvm controls are clean on main @ 48242ec. Different producer, different target, still live. Closed #17938 is the same ICHECK string from a third producer (constant folding of an ONNX model).

How found

Found by EquiAutomaton (equivalence-graph differential testing, TVM CUDA vs TVM llvm vs onnxruntime). Depth-0 — a single Neg node.

Reproducer archive

TVM-C3-reproducer.zip

Triage

  • Needs triage

Activity

  1. added
    needs-triagePRs or issues that need to be investigated by maintainers to find the right assignees to address it
    on Sep 6, 2026
  2. Nanmur commented on Sep 20, 2026

    @Nanmur
    Contributor

    I opened #20394 with a focused fix and regression test.

    The fallback rule now skips PrimFuncs containing a statically zero-extent loop before its inline/fuse transformations run. This avoids the invalid zero-extent scheduling path while keeping the empty loop unchanged for later compilation stages.

    The regression test uses a float32[4, 0] elementwise function, matching the non-leading zero-extent case from this report.

  3. added a commit that references this issue on Sep 22, 2026
    5b4b753
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    needs-triagePRs or issues that need to be investigated by maintainers to find the right assignees to address ittype: bug

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions