[SPIR-V] Add descriptor heap -fvk-resource-heap-stride / -fvk-sampler-heap-stride CLI flags

docs/SPIR-V.rst

@@ -336,6 +336,8 @@ Supported extensions
 * SPV_KHR_float_controls
 * SPV_NV_shader_subgroup_partitioned
 * SPV_KHR_quad_control
+* SPV_KHR_untyped_pointers
+* SPV_EXT_descriptor_heap
 
 Vulkan specific attributes
 --------------------------
@@ -1993,10 +1995,14 @@ responsibility to provide proper numbers and avoid binding overlaps.
 ResourceDescriptorHeaps & SamplerDescriptorHeaps
 ------------------------------------------------
 
-The SPIR-V backend supported SM6.6 resource heaps, using 2 extensions:
+By default, the SPIR-V backend supports SM6.6 resource heaps by emulating the
+heaps with descriptor-indexing runtime arrays, using 2 extensions:
+
 - `SPV_EXT_descriptor_indexing`
 - `VK_EXT_mutable_descriptor_type`
 
+This is also the behavior selected by ``-fspv-use-emulated-heap``.
+
 Each type loaded from a heap is considered to be an unbounded RuntimeArray
 bound to the descriptor set 0.
 
@@ -2074,6 +2080,184 @@ Bindings & sets associated with each heap can be explicitly set using:
 - `-fvk-bind-counter-heap <binding> <set>`: Specify Vulkan binding number
   and set number for the counter heap.
 
+Native descriptor heap extension lowering
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+When ``-fspv-use-descriptor-heap`` is specified, DXC lowers
+``ResourceDescriptorHeap`` and ``SamplerDescriptorHeap`` through
+``SPV_EXT_descriptor_heap`` instead of the default emulated heap path. This
+also requires ``SPV_KHR_untyped_pointers`` and ``-fspv-target-env=vulkan1.3``
+(targeting a lower environment is an error), and a SPIRV-Headers / SPIRV-Tools
+build that defines these extensions. The emitted module declares the heap
+objects as untyped variables in ``UniformConstant`` storage class:
+
+.. code:: spirv
+
+  %uptr_uc = OpTypeUntypedPointerKHR UniformConstant
+  %resource_heap = OpUntypedVariableKHR %uptr_uc UniformConstant
+  %sampler_heap  = OpUntypedVariableKHR %uptr_uc UniformConstant
+  OpDecorate %resource_heap BuiltIn ResourceHeapEXT
+  OpDecorate %sampler_heap BuiltIn SamplerHeapEXT
+
+The concrete descriptor type is selected at each heap access. For image,
+sampler, and texel buffer resources, DXC forms a runtime array of that
+descriptor type, decorates the array with a byte stride, and uses
+``OpUntypedAccessChainKHR`` followed by ``OpLoad``. By default the stride is an
+``ArrayStrideIdEXT`` decoration referencing a specialization constant; a literal
+``ArrayStride`` is emitted only when the stride is overridden on the command
+line (see `Descriptor heap array stride`_ below):
+
+.. code:: spirv
+
+  %image_type = OpTypeImage %float 2D 2 0 0 1 Unknown
+  %image_array = OpTypeRuntimeArray %image_type
+  OpDecorateId %image_array ArrayStrideIdEXT %resource_stride
+  %descriptor = OpUntypedAccessChainKHR %uptr_uc %image_array %resource_heap %index
+  %image = OpLoad %image_type %descriptor
+
+For buffer-like resources, DXC uses ``OpTypeBufferEXT`` as the descriptor type
+and ``OpBufferPointerEXT`` to recover the pointer to the buffer data. The
+descriptor storage class matches the recovered buffer pointer storage class; for
+example, ``ConstantBuffer<T>`` uses ``Uniform`` and ``TextureBuffer<T>`` uses
+``StorageBuffer``:
+
+.. code:: spirv
+
+  %buffer_type = OpTypeBufferEXT Uniform
+  %buffer_array = OpTypeRuntimeArray %buffer_type
+  OpDecorateId %buffer_array ArrayStrideIdEXT %resource_stride
+  %descriptor = OpUntypedAccessChainKHR %uptr_uc %buffer_array %resource_heap %index
+  %buffer_ptr = OpBufferPointerEXT %_ptr_Uniform_type_BufferData %descriptor
+
+For ``RWTexture`` resources loaded from ``ResourceDescriptorHeap``, interlocked
+operations that need a texel pointer use ``OpUntypedImageTexelPointerEXT``.
+The image descriptor pointer produced by ``OpUntypedAccessChainKHR`` is passed
+directly to the texel-pointer instruction instead of first storing the image
+handle into a function-scope image variable:
+
+.. code:: spirv
+
+  %image_type = OpTypeImage %uint 2D 2 0 0 2 R32ui
+  %image_array = OpTypeRuntimeArray %image_type
+  %descriptor = OpUntypedAccessChainKHR %uptr_uc %image_array %resource_heap %index
+  %uptr_image = OpTypeUntypedPointerKHR Image
+  %texel_ptr = OpUntypedImageTexelPointerEXT %uptr_image %image_type %descriptor %coord %sample
+  %old = OpAtomicIAdd %uint %texel_ptr %scope %semantics %value
+
+``RaytracingAccelerationStructure`` resources loaded from
+``ResourceDescriptorHeap`` follow the same access-chain-then-load shape over a
+runtime array of ``OpTypeAccelerationStructureKHR``:
+
+.. code:: spirv
+
+  %accel_type = OpTypeAccelerationStructureKHR
+  %accel_array = OpTypeRuntimeArray %accel_type
+  OpDecorateId %accel_array ArrayStrideIdEXT %resource_stride
+  %descriptor = OpUntypedAccessChainKHR %uptr_uc %accel_array %resource_heap %index
+  %accel = OpLoad %accel_type %descriptor
+
+This path supports texture, RWTexture, sampler, Buffer/RWBuffer,
+StructuredBuffer/RWStructuredBuffer without associated counter operations,
+ByteAddressBuffer/RWByteAddressBuffer, ConstantBuffer, TextureBuffer, and
+``RaytracingAccelerationStructure`` heap loads, including direct field and
+array-element accesses for ``ConstantBuffer<T>`` and ``TextureBuffer<T>``.
+Acceleration structure loads are additionally subject to the stride requirement
+described in `Descriptor heap array stride`_ below.
+``NonUniformResourceIndex`` is accepted, but no ``NonUniform`` decoration is
+emitted on the ``OpUntypedAccessChainKHR`` result or on the loaded descriptor;
+``SPV_EXT_descriptor_heap`` deprecates the decoration for heap accesses and
+drivers handle divergent heap indices natively. The index operand itself may
+still carry ``NonUniform`` from the surrounding expression.
+
+Append/consume structured buffers and UAV counter heap lowering are not
+supported by the native descriptor heap path yet. Those forms should continue
+to use the default emulated heap lowering, or DXC will emit a diagnostic for
+unsupported append/consume structured-buffer heap loads. Heap-loaded
+``RWStructuredBuffer`` resources are supported for ordinary data access, but
+associated counter operations such as ``IncrementCounter`` and
+``DecrementCounter`` emit a diagnostic because the native descriptor heap path
+does not recover an associated counter descriptor.
+
+A local resource variable initialized from a heap access is resolved entirely at
+compile time: the variable is recorded as an alias for the heap index, and every
+later use is re-lowered as a fresh access chain rather than as a load of a stored
+descriptor handle. This is sound only when the variable holds a heap descriptor
+on every path that reaches the use. DXC therefore rejects a variable that holds
+both a bound resource and a heap descriptor, whether through a conditional
+assignment, a reassignment back to a bound resource, or an assignment inside a
+loop::
+
+  error: mixing bound and descriptor heap resources in the same variable is not
+  supported with SPV_EXT_descriptor_heap
+
+Supporting these forms requires modelling the alias as a value with real
+control-flow merges instead of as compile-time state. Until then they are
+diagnosed rather than silently miscompiled.
+
+Three further restrictions on the heap access expression itself produce
+diagnostics. The object being subscripted must be a direct reference to the
+builtin ``ResourceDescriptorHeap`` or ``SamplerDescriptorHeap`` variable; the
+subscript result must be immediately converted to a concrete resource type so
+that DXC can select a descriptor type for the access, so a subscript whose
+result is discarded or used in a context that supplies no target resource type
+is rejected; and a local ``RaytracingAccelerationStructure`` must be initialized
+from a loadable heap access, since the alias has no backing descriptor
+otherwise.
+
+Descriptor heap array stride
+++++++++++++++++++++++++++++
+
+By default, all resource heap runtime arrays share a single ``ArrayStrideIdEXT``
+decoration rather than a literal ``ArrayStride``, because descriptor sizes are
+not known until pipeline creation. The shared value is built from
+``OpConstantSizeOfEXT`` and ``OpSpecConstantOp`` and evaluates to
+``max(sizeof(image_descriptor), sizeof(buffer_descriptor))``. The sampler heap
+carries its own ``ArrayStrideIdEXT`` equal to ``sizeof(sampler_descriptor)``,
+regardless of resource heap contents.
+
+The ``OpConstantSizeOfEXT`` operands are placeholder types chosen only for their
+descriptor class. All image types report the same descriptor size, so the
+placeholder is a plain sampled 2D float image and bears no relation to the image
+types the shader actually uses; a module will normally contain both the
+placeholder type and the distinct image types its heap accesses lower to. The
+stride value is built once and cached on first use.
+
+When acceleration structure descriptors may appear on the resource heap, the
+formula expands to a three-way max
+``max(max(sizeof(image_descriptor), sizeof(buffer_descriptor)), sizeof(acceleration_structure))``.
+
+Because the stride is cached on first use, this decision is committed before
+code generation and is **not** based on whether the shader actually performs an
+acceleration structure heap load: a ray-tracing shader that only heap-loads a
+texture still gets the three-way max. The widening happens when either
+
+- any entry point is a ray-tracing stage, or
+- the user explicitly passed ``-fspv-extension=SPV_KHR_ray_tracing``,
+  ``-fspv-extension=SPV_NV_ray_tracing``, or
+  ``-fspv-extension=SPV_KHR_ray_query``.
+
+The second condition requires an explicit ``-fspv-extension`` flag. In the
+default extension mode DXC allows the ray-tracing and ray-query extensions
+implicitly, but that does not widen the stride. Because the stride cannot be
+widened once it has been built, a shader that is not a ray-tracing stage and
+loads a ``RaytracingAccelerationStructure`` from ``ResourceDescriptorHeap``
+without an explicit ray extension flag is rejected rather than given a stride
+that may be too narrow.
+
+The computed stride can be replaced with a fixed literal using
+``-fvk-resource-heap-stride <N>`` and ``-fvk-sampler-heap-stride <N>``, which
+emit ``OpDecorate <array> ArrayStride N`` on the resource and sampler heap
+arrays respectively. ``N`` must be a power of two in the inclusive range
+[8, 256], and both flags require ``-spirv``. The command-line override has the
+highest precedence: when it is set for a heap, no ``ArrayStrideIdEXT`` is
+emitted for that heap and no ``OpConstantSizeOfEXT`` is built for it. The two
+flags are independent, so overriding one heap leaves the other on its computed
+stride.
+
+The literal is not validated against the descriptor sizes of the target
+implementation. A value smaller than the largest descriptor that may appear in
+the heap produces out-of-bounds descriptor accesses at runtime.
+
 HLSL Expressions
 ================
 

include/dxc/Support/HLSLOptions.td

@@ -454,6 +454,10 @@ def fvk_bind_sampler_heap : MultiArg<["-"], "fvk-bind-sampler-heap", 2>, MetaVar
   HelpText<"Specify Vulkan binding number and set number for the sampler heap.">;
 def fvk_bind_counter_heap : MultiArg<["-"], "fvk-bind-counter-heap", 2>, MetaVarName<"<binding> <set>">, Group<spirv_Group>, Flags<[CoreOption, DriverOption]>,
   HelpText<"Specify Vulkan binding number and set number for the counter heap.">;
+def fvk_resource_heap_stride : Separate<["-"], "fvk-resource-heap-stride">, MetaVarName<"<stride>">, Group<spirv_Group>, Flags<[CoreOption, DriverOption]>,
+  HelpText<"Override the byte ArrayStride of the resource descriptor heap runtime array. Must be a power of 2 in [8, 256].">;
+def fvk_sampler_heap_stride : Separate<["-"], "fvk-sampler-heap-stride">, MetaVarName<"<stride>">, Group<spirv_Group>, Flags<[CoreOption, DriverOption]>,
+  HelpText<"Override the byte ArrayStride of the sampler descriptor heap runtime array. Must be a power of 2 in [8, 256].">;
 // SPIRV Change Ends
 
 //////////////////////////////////////////////////////////////////////////////

include/dxc/Support/SPIRVOptions.h

@@ -110,6 +110,13 @@ struct SpirvCodeGenOptions {
   std::optional<BindingInfo> samplerHeapBinding;
   std::optional<BindingInfo> counterHeapBinding;
 
+  // User-defined byte ArrayStride overrides for the resource/sampler descriptor
+  // heap runtime arrays (-fvk-resource-heap-stride / -fvk-sampler-heap-stride).
+  // When set, the value is a literal power of 2 in [8, 256] and replaces the
+  // ArrayStrideIdEXT decoration that heap would otherwise carry.
+  std::optional<uint32_t> resourceHeapStride;
+  std::optional<uint32_t> samplerHeapStride;
+
   bool signaturePacking =
       false; ///< Whether signature packing is enabled or not
 

lib/DxcSupport/HLSLOptions.cpp

@@ -357,6 +357,43 @@ handleFixedBinding(const InputArgList &args, OptSpecifier id,
   return true;
 }
 
+// Parses the single-integer descriptor-heap stride flag |id| in |args|. If
+// present, validates that the value is a power of 2 in [8, 256] and stores it
+// in |stride|. Returns true on success (including when the flag is absent).
+// Returns false and writes to |errors| when the value is malformed or invalid,
+// using |name| as the pretty flag name.
+static bool handleHeapStride(const InputArgList &args, OptSpecifier id,
+                             std::optional<uint32_t> *stride,
+                             llvm::StringRef name, llvm::raw_ostream &errors) {
+  Arg *arg = args.getLastArg(id);
+  if (!arg) {
+    *stride = std::nullopt;
+    return true;
+  }
+
+  if (!args.hasArg(OPT_spirv)) {
+    errors << name << " requires -spirv";
+    return false;
+  }
+
+  llvm::StringRef value = arg->getValue();
+  uint32_t number = 0;
+  if (value.getAsInteger(10, number)) {
+    errors << "invalid " << name << " argument: '" << value << "'";
+    return false;
+  }
+  // Power of 2 in [8, 256] inclusive.
+  if (number < 8 || number > 256 || (number & (number - 1)) != 0) {
+    errors << name
+           << " must be a power of 2 between 8 and 256 (inclusive); got "
+           << value;
+    return false;
+  }
+
+  *stride = number;
+  return true;
+}
+
 // Check if any options that are unsupported with SPIR-V are used.
 static bool hasUnsupportedSpirvOption(const InputArgList &args,
                                       llvm::raw_ostream &errors) {
@@ -1175,6 +1212,16 @@ int ReadDxcOpts(const OptTable *optionTable, unsigned flagsToInclude,
     return 1;
   }
 
+  bool strideOk = true;
+  strideOk &= handleHeapStride(Args, OPT_fvk_resource_heap_stride,
+                               &opts.SpirvOptions.resourceHeapStride,
+                               "-fvk-resource-heap-stride", errors);
+  strideOk &= handleHeapStride(Args, OPT_fvk_sampler_heap_stride,
+                               &opts.SpirvOptions.samplerHeapStride,
+                               "-fvk-sampler-heap-stride", errors);
+  if (!strideOk)
+    return 1;
+
   for (const Arg *A : Args.filtered(OPT_fspv_extension_EQ)) {
     opts.SpirvOptions.allowedExtensions.push_back(A->getValue());
   }
@@ -1316,7 +1363,9 @@ int ReadDxcOpts(const OptTable *optionTable, unsigned flagsToInclude,
       !Args.getLastArgValue(OPT_fvk_u_shift).empty() ||
       !Args.getLastArgValue(OPT_fvk_bind_resource_heap).empty() ||
       !Args.getLastArgValue(OPT_fvk_bind_sampler_heap).empty() ||
-      !Args.getLastArgValue(OPT_fvk_bind_counter_heap).empty()) {
+      !Args.getLastArgValue(OPT_fvk_bind_counter_heap).empty() ||
+      !Args.getLastArgValue(OPT_fvk_resource_heap_stride).empty() ||
+      !Args.getLastArgValue(OPT_fvk_sampler_heap_stride).empty()) {
     errors << "SPIR-V CodeGen not available. "
               "Please recompile with -DENABLE_SPIRV_CODEGEN=ON.";
     return 1;

tools/clang/include/clang/SPIRV/AstTypeProbe.h

@@ -250,6 +250,10 @@ bool isBuffer(QualType type);
 /// \brief Returns true if the given type is the HLSL RWBuffer type.
 bool isRWBuffer(QualType type);
 
+/// \brief Returns true if the given type is the HLSL
+/// RaytracingAccelerationStructure type.
+bool isRaytracingAccelerationStructure(QualType type);
+
 /// \brief Returns true if the given type is an HLSL Texture type.
 bool isTexture(QualType);
 

tools/clang/include/clang/SPIRV/SpirvBuilder.h

@@ -277,7 +277,7 @@ class SpirvBuilder {
   /// \brief Creates an OpUntypedImageTexelPointerEXT SPIR-V instruction with
   /// the given parameters.
   SpirvUntypedImageTexelPointerEXT *createUntypedImageTexelPointerEXT(
-      QualType resultType, SpirvInstruction *image,
+      QualType resultType, const SpirvType *imageType, SpirvInstruction *image,
       SpirvInstruction *coordinate, SpirvInstruction *sample, SourceLocation);
 
   /// \brief Creates an OpConverPtrToU SPIR-V instruction with the given
@@ -825,6 +825,43 @@ class SpirvBuilder {
                        bool specConst = false);
   SpirvConstant *getConstantNull(QualType);
   SpirvConstant *getConstantString(llvm::StringRef str, bool specConst = false);
+  /// \brief Returns the OpConstantSizeOfEXT (SPV_EXT_descriptor_heap) for the
+  /// given descriptor operandType, yielding its client-API size in bytes as
+  /// a 32-bit unsigned value. The result is cached per operand type, so each
+  /// descriptor type emits at most one instruction.
+  SpirvConstant *getConstantSizeOfEXT(const SpirvType *operandType);
+
+  SpirvSpecConstantTernaryOp *
+  createSpecConstantTernaryOp(spv::Op op, QualType resultType,
+                              SpirvInstruction *op1, SpirvInstruction *op2,
+                              SpirvInstruction *op3, SourceLocation loc);
+
+  /// \brief Record that acceleration structures may occupy the resource heap.
+  /// Note: Must be called before getResourceHeapArrayStride() (before the
+  /// code-gen loop in HandleTranslationUnit) so the cached stride is correct
+  /// on the first call. Calling it later has no effect because the result is
+  /// frozen after the first getResourceHeapArrayStride() invocation.
+  void noteResourceHeapHasAccelStruct() { resourceHeapHasAccelStruct = true; }
+
+  /// \brief Returns whether the resource-heap stride accounts for acceleration
+  /// structure descriptors. Code-gen must reject an acceleration structure heap
+  /// access when this is false: the decision is made before the code-gen loop
+  /// and the stride cannot be widened afterwards.
+  bool resourceHeapStrideIncludesAccelStruct() const {
+    return resourceHeapHasAccelStruct;
+  }
+
+  /// \brief Shared ArrayStrideIdEXT operand for resource-heap runtime arrays.
+  /// Default:  max(sizeof(image), sizeof(buffer))
+  /// With RT:  max(max(sizeof(image), sizeof(buffer)), sizeof(accel_struct))
+  /// Computed via OpSpecConstantOp and cached per module.
+  /// Note: noteResourceHeapHasAccelStruct() must be called before this if AS
+  /// may be present (result is frozen on the first call).
+  SpirvInstruction *getResourceHeapArrayStride();
+
+  /// \brief Shared ArrayStrideIdEXT operand for sampler-heap runtime arrays:
+  /// the sampler descriptor size. Cached per module.
+  SpirvInstruction *getSamplerHeapArrayStride();
   SpirvUndef *getUndef(QualType);
 
   SpirvString *createString(llvm::StringRef str);
@@ -941,6 +978,22 @@ class SpirvBuilder {
   /// Used as caches for all created builtin variables to avoid duplication.
   llvm::SmallVector<BuiltInVarInfo, 16> builtinVars;
 
+  /// Cache of OpConstantSizeOfEXT instructions keyed on the descriptor operand
+  /// type, so each distinct descriptor type emits at most one instruction.
+  llvm::DenseMap<const SpirvType *, SpirvConstant *> constantSizeOfEXTMap;
+
+  /// Cached shared descriptor-heap array strides (SPV_EXT_descriptor_heap), so
+  /// each is emitted once per module (see
+  /// get{Resource,Sampler}HeapArrayStride).
+  SpirvInstruction *resourceHeapArrayStride = nullptr;
+  SpirvInstruction *samplerHeapArrayStride = nullptr;
+
+  /// Set by noteResourceHeapHasAccelStruct() when HandleTranslationUnit
+  /// detects that the shader uses ray-tracing features.  When true,
+  /// getResourceHeapArrayStride() extends the stride to include
+  /// sizeof(acceleration_structure).
+  bool resourceHeapHasAccelStruct = false;
+
   SpirvDebugInfoNone *debugNone;
 
   /// DebugExpression that does not reference any DebugOperation