Skip to content

Loosen fp16 atol in ONNX export tests for onnxruntime 1.29 CPU numerics - #3409

Merged
fheinecke merged 1 commit into
NVIDIA:mainfrom
pggPL:onnx_fp16_tolerance
Aug 20, 2026
Merged

Loosen fp16 atol in ONNX export tests for onnxruntime 1.29 CPU numerics#3409
fheinecke merged 1 commit into
NVIDIA:mainfrom
pggPL:onnx_fp16_tolerance

Conversation

@pggPL

@pggPL pggPL commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator

Description

L1_pytorch_onnx_unittest fails in nightly CI since 2026-08-18 on the fp16 (precision1) variants of test_export_linear_recipe, test_export_layernorm_linear_recipe and test_export_layernorm_mlp. The trigger is the onnxruntime 1.28.0 → 1.29.0 bump in the unpinned CI image: ORT executes the exported graph on the CPU EP, and 1.29 changed fp16 CPU numerics, so the ORT-vs-TE mismatch now reaches up to ~9e-3 (2–9 fp16 ULPs) against the hardcoded atol=1e-3. Verified by A/B on the same TE/torch: ORT 1.28 passes, ORT 1.29 fails; no TE commit is involved.

Since the comparison is half-precision GPU (TE) vs half-precision CPU (ORT), differences of a few ULPs from accumulation order are expected and do not indicate an incorrect ONNX representation. This PR makes the tolerance precision-aware: atol=2e-2 for fp16 (the value already used for FP8 CurrentScaling and FP8 MLP in the same file), keeping 1e-3 for fp32.

Type of change

  • Documentation change (change only to the documentation, either a fix or a new content)
  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Infra/Build change
  • Code refactoring

Changes

  • Add get_atol(precision) helper in tests/pytorch/test_onnx_export.py returning 2e-2 for fp16 and the default 1e-3 otherwise.
  • Use it in the non-FP8 validation paths of _test_export_linear, _test_export_layernorm_linear, _test_export_layernorm_mlp and in the DelayedScaling branch of _test_export_layernorm_linear.

Checklist:

  • I have read and followed the contributing guidelines
  • The functionality is complete
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

🤖 Generated with Claude Code

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
@greptile-apps

greptile-apps Bot commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

The PR makes ONNX-export numerical validation precision-aware to accommodate ONNX Runtime 1.29 CPU fp16 differences while preserving the existing fp32 tolerance.

  • Adds a shared helper returning 2e-2 for fp16 and 1e-3 otherwise.
  • Applies it to Linear, LayerNormLinear, and LayerNormMLP validation paths, including the relevant DelayedScaling branch.

Confidence Score: 5/5

The PR appears safe to merge because the changed tolerance is limited to fp16 comparisons in the identified ONNX tests while fp32 remains unchanged.

The helper receives canonical torch dtype parameters, selects the intended fp16 tolerance at every changed call site, and preserves the previous tolerance for fp32; no concrete blocking or non-blocking defect remains.

Important Files Changed

Filename Overview
tests/pytorch/test_onnx_export.py Introduces and consistently applies a precision-aware absolute tolerance to the identified fp16 ONNX export comparisons without changing fp32 behavior.

Reviews (1): Last reviewed commit: "Loosen fp16 atol in ONNX export tests fo..." | Re-trigger Greptile

@pggPL
pggPL requested a review from vthumbe1503 August 20, 2026 15:16
@fheinecke fheinecke added the 2.19 label Aug 20, 2026
@vthumbe1503

Copy link
Copy Markdown
Collaborator

/te-ci L1 pytorch

@fheinecke

Copy link
Copy Markdown
Collaborator

The CI failures are unrelated to this changeset and should be fixed in #3384

@fheinecke
fheinecke merged commit 17c351c into NVIDIA:main Aug 20, 2026
10 of 16 checks passed
fheinecke pushed a commit to fheinecke/TransformerEngine that referenced this pull request Aug 20, 2026
…cs (NVIDIA#3409)

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
(cherry picked from commit 17c351c)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants