Skip to content

Vulkan: support zeros padding and align_corners=false in grid_sampler_2d - #22803

Open
msluszniak wants to merge 1 commit into
pytorch:mainfrom
msluszniak:ms/vulkan-grid-sampler-configs
Open

msluszniak wants to merge 1 commit into
pytorch:mainfrom
msluszniak:ms/vulkan-grid-sampler-configs

Conversation

@msluszniak

@msluszniak msluszniak commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

Summary

grid_sampler_2d was gated to a single configuration, the one RIFE's WarpModule uses: bilinear, border padding, align_corners=True. Everything else was rejected by check_grid_sampler_2d_node and fell back to CPU, including torch's own defaults (padding_mode="zeros", align_corners=False), which is what the deformable attention in DETR-derived detectors emits.

Changes

padding_mode and align_corners become specialization constants, so the four supported combinations share one shader variant and each pipeline still compiles with the branches folded away. Reflection padding and nearest/bicubic interpolation remain unsupported and still fall back.

The two padding paths are not interchangeable: border clamps the sample coordinate, which also pins the interpolation weights at the edge, while zeros must not clamp, because the weights stay as computed and each out-of-range corner contributes a zero value instead.

Test

test_vulkan_backend_grid_sampler_2d covers all four combinations with a grid deliberately overshooting to +/-1.6, so the zeros and border paths actually diverge; an in-range grid passes even with the padding branch broken.

Also checked end to end on a Mali-G76 against eager PyTorch, max abs error 5.96e-08 (float32 eps) for each of the four configurations.

On an RF-DETR keypoint model this moves all 8 grid_sampler_2d calls off the host, taking the graph from 7 delegate blobs and 11 host calls to 2 and 3.

cc @SS-JIA @manuelcandales @digantdesai @cbilgin

The shader implemented one configuration, the one RIFE's WarpModule uses:
bilinear, border padding, align_corners=true. Every other configuration was
rejected by the partitioner and fell back to CPU, including torch's own
default of zeros padding with align_corners=false, which is what the
deformable attention in DETR-derived detectors emits.

Make padding_mode and align_corners specialization constants so the four
supported combinations share one shader variant and each pipeline still
compiles with the branches folded away.

border clamps the sample coordinate, which pins the interpolation weights at
the edge. zeros must not clamp: the weights stay as computed and an
out-of-range corner contributes a zero value instead, so the two paths differ
for any grid that leaves [-1, 1].

On an RF-DETR keypoint model this takes all 8 grid_sampler_2d calls off the
host and the graph from 7 delegate blobs to 2.
@msluszniak
msluszniak requested a review from SS-JIA as a code owner September 14, 2026 16:08
@pytorch-bot pytorch-bot Bot added the module: vulkan Issues related to the Vulkan delegate and code under backends/vulkan/ label Sep 14, 2026
@pytorch-bot

pytorch-bot Bot commented Sep 14, 2026

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/22803

Note: Links to docs will display an error until the docs builds have been completed.

⚠️ 15 Awaiting Approval

As of commit 3433a4f with merge base 14dab10 (image):

AWAITING APPROVAL - The following workflows need approval before CI can run:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Sep 14, 2026
@github-actions

Copy link
Copy Markdown

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

msluszniak added a commit to software-mansion-labs/executorch that referenced this pull request Sep 15, 2026
Backport of pytorch/executorch#22803. grid_sampler_2d was gated to RIFE's
config (bilinear/border/align_corners=true). RF-DETR needs zeros/false, and
relaxing the gate takes its detector to 3 delegates / 3 host calls.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. module: vulkan Issues related to the Vulkan delegate and code under backends/vulkan/

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants