DEV Community: Gasim Gasimzada

Buffer device addresses in Vulkan and VMA

Gasim Gasimzada — Mon, 15 May 2023 17:00:42 +0000

A while ago, I posted about managing bindless descriptors in Vulkan with heavy focus on setting up a system for accessing buffers of different structures using descriptor indexing.

As I recently learned about buffer device addresses, I want to talk about device buffer addresses and how they simplify managing and accessing buffers by getting rid of descriptors and making buffer access more intuitiveş

What are buffer device addresses?

Buffer device address (VK_KHR_buffer_device_address) is a Vulkan extension that enables to fetching memory address of a buffer in GPU and using GLSL_EXT_buffer_reference in shaders.

GLSL buffer reference

GLSL buffer reference (GLSL_EXT_buffer_reference) shader extension enables defining structures buffer_reference layout and using those structures in other storages blocks such as uniforms, push constants, and buffers:

layout(buffer_reference, std430, buffer_reference_align=16) buffer MyBuffer {
  vec4 color;
};

layout(push_constant) uniform Data {
  // Maps device address stored in push constant
  // to MyBuffer structure.
  MyBuffer buffer;
} pcData;

void main() {
  // Using the buffer directly from push constant
  // as if it is an object.
  vec4 color = pcData.buffer.color;
}

Setup

Before we start with our application code, we need to enable creating buffers with .

Enable device feature

Now, we need to enable the device feature. To use this feature we are going to utilize VkPhysicalDeviceFeatures2, which we can pass as pNext device create info structure:

// Create the feature chain
VkPhysicalDeviceDescriptorIndexingFeatures descriptorIndexingFeatures{};
descriptorIndexingFeatures.sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_DESCRIPTOR_INDEXING_FEATURES;

// Pass your other features through this chain
descriptorIndexingFeatures.pNext = nullptr;

VkPhysicalDeviceBufferDeviceAddressFeatures bufferDeviceAddressFeatures{};
bufferDeviceAddressFeatures.sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_BUFFER_DEVICE_ADDRESS_FEATURES;
bufferDeviceAddressFeatures.pNext = &descriptorIndexingFeatures;

VkPhysicalDeviceFeatures2 deviceFeatures{};
deviceFeatures.sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_FEATURES_2;
deviceFeatures.pNext = &bufferDeviceAddressFeatures;

// Fetch all features
vkGetPhysicalDeviceFeatures2(physicalDevice, &deviceFeatures);

// Buffer device address feature is required to exist
// in the physical device
assert(bufferDeviceFeatures.bufferDeviceAddress);

createDeviceInfo.pNext = &deviceFeatures;
// create device

Note:

If you are using Vulkan < 1.2, you need to enable the extension explicitly:

std::vector<const char *> extensions;
// other extensions
// Add buffer device address extension
extensions.push_back(VK_KHR_BUFFER_DEVICE_ADDRESS_EXTENSION_NAME);
createDeviceInfo.ppEnabledExtensionNames = extensions.data();
createDeviceInfo.enabledExtensionCount = static_cast<uint32_t>(extensions.size());

Enable buffer device addresses in VMA

Now that the feature is enabled, we need to enable buffer device addresses in Vulkan Memory Allocator:

VmaAllocatorCreateInfo createInfo{};
createInfo.instance = vulkanInstance;
createInfo.physicalDevice = vulkanPhysicalDevice;
createInfo.device = vulkanDevice;
// Pass the buffer device address flag
createInfo.flags = VMA_ALLOCATOR_CREATE_BUFFER_DEVICE_ADDRESS_BIT;
vmaCreateAllocator(&createInfo, &mAllocator);

Adding VMA_ALLOCATOR_CREATE_BUFFER_DEVICE_ADDRESS_BIT flags automatically adds VK_MEMORY_ALLOCATE_DEVICE_ADDRESS_BIT when new memory blocks are allocated by VMA.

Enable buffer access from shaders

Let's add VK_BUFFER_USAGE_SHADER_DEVICE_ADDRESS_BIT usage flag during buffer creation to allow accessing these buffers from shaders.

VkBufferUsageFlags bufferUsage = VK_BUFFER_USAGE_SHADER_DEVICE_ADDRESS_BIT;

VkBufferCreateInfo createBufferInfo{};
createBufferInfo.sType = VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO;
createBufferInfo.pNext = nullptr;
createBufferInfo.flags = 0;
createBufferInfo.size = description.size;
createBufferInfo.usage = bufferUsage;

// create device here

That's it! Now we can query and access device addresses of buffers in our application and use the addresses to access buffers in shaders.

Pass buffer device addresses to shaders

After enabling the buffer device address, let's fetch the device addresses of buffers and send them to shaders:

Introducing new type in our application

Buffer device addresses have type uint64_t. In order to have compile time guarantees and between uint64_t and device addresses, we are going to introduce a new type:

enum class DeviceAddress : uint64_t { Invalid = 0 };

This will provide minor assurance for us to not accidentally pass unintended values within our application.

Fetch buffer device addresses

Let's introduce a function that fetches device address of a buffer:

DeviceAddress getBufferDeviceAddress(VkBuffer buffer) {
  VkBufferDeviceAddressInfo addressInfo{};
  addressInfo.sType = VK_STRUCTURE_TYPE_BUFFER_DEVICE_ADDRESS_INFO;
  addressInfo.pNext = nullptr;
  addressInfo.buffer = buffer;
  return static_cast<DeviceAddress>(
      vkGetBufferDeviceAddress(mDevice, &addressInfo));
}

Pass device addresses to pipelines

We are going to use the same BindlessParams system to pass draw parameters to shaders. Nothing inside the BindlessParams is going to change. Let's also use the same example in the previous post but with DeviceAddress instead of BufferHandle:

struct PBRParams {
  DeviceAddress meshTransforms;
  DeviceAddress pointLights;
  DeviceAddress camera;
};

struct SkyboxParams {
  DeviceAddress camera;
  TextureHandle skybox;
  uint32_t pad0;
};

struct TextParams {
  DeviceAddress textTransforms;
  DeviceAddress camera;
  DeviceAddress glyphsBuffer;
};

BindlessParams bindlessParams(minUniformBufferOffsetAlignment);

auto rangePBR = bindlessParams.addRange(PBRParams({
  getBufferDeviceAddress(meshTransformsBuffer),
  getBufferDeviceAddress(pointLightsBuffer),
  getBufferDeviceAddress(cameraBuffer)
});

auto rangeSkybox = bindlessParams.addRange(SkyboxParams({
  getBufferDeviceAddress(cameraBuffer),
  getBufferDeviceAddress(skyboxTexture)
});

auto textParams = bindlessParams.addRange(TextParams({
  getBufferDeviceAddress(textTransformsBuffer),
  getBufferDeviceAddress(cameraBuffer),
  getBufferDeviceAddress(glyphsBuffer)
});

bindlessParams.build(mDevice, mAllocator, mDescriptorPool);

As you can see, we do not need any descriptor to store buffers like we did with descriptor indexing. We still use descriptor indexing for textures and use dynamic uniform buffer to pass draw parameters to the shaders but we do not need to manage any descriptors for buffers.

Accessing buffers in shaders

We now pass buffer addresses to shaders, now it is time to utilize these addresses using GLSL buffer_reference extension.

Macro to define buffers references with ease

To make our lives easier, we are going to create a tiny preprocessor macro to easily define buffer reference buffers in our shaders:

#define Buffer(Alignment) \
  layout(buffer_reference, std430, buffer_reference_align = Alignment) buffer

Define buffer references

Now, let's use the same buffer layouts that we defined in our previous post with this new approach:

Buffer(64) Camera {
  mat4 viewProjection;
  mat4 view;
  mat4 projection;
};

struct TransformData {
  mat4 transform;
};

Buffer(64) Transforms {
  TransformData items[];
};

Use buffer references in uniforms

After we defined our buffers with buffer_reference layout, we can use the buffers in our draw parameters uniform:

// Draw parameters
layout(set = 1, binding = 0) uniform DrawParameters {
  Transforms meshTransforms;
  PointLights pointLights;
  Camera camera;
  // Don't forget the padding
  uint pad0;
} uDrawParameters;

void main() {
  mat4 transform = uDrawParameters.meshTransforms[0].transform;
  mat4 view = uDrawParameters.camera.view;
}

That's it!

As you can see, buffer device addresses significantly simplifies buffer access management by eliminating the need for descriptors while also providing a more intuitive API in shaders to access them.

This was my first attempt in buffer device addresses. As a next step I am going to try to use buffer device addresses for vertex buffers as well, which can eliminate the need for multiple pipeline layouts when combining it with descriptor indexing my vertex or descriptor layouts can be unified for all pipelines.

Managing bindless descriptors in Vulkan

Gasim Gasimzada — Tue, 28 Mar 2023 16:57:52 +0000

Bindless design is a technique that allows for efficient management of resources in modern graphics APIs such as Vulkan, DirectX 12 and Metal. This technique eliminates the need for binding resources like textures, buffers, and samplers to specific slots, instead allowing the application to access resources directly through their unique handles.

In this post, we will discover how to setup bindless descriptors for uniform, storage buffers, and textures. Additionally, we will also set up abstractions to make it easier to register and access different resources within the system using. I will be mainly using Vulkan and C++ for this guide but the concepts and ideas can be transferred to DX12 and Metal as well.

Bindless design in Vulkan

Bindless designed is enabled in Vulkan using the descriptor indexing extension, which promoted to core in Vulkan 1.2. There are three core elements to this extension that opens up the possibility for bindless design:

Descriptor binding flag VK_DESCRIPTOR_BINDING_UPDATE_AFTER_BIND_BIT: This flag indicates that if we update descriptor after it is bound (i.e using vkBindDescriptorSets), the command submission will use the most recently updated version of the descriptor set and most importantly, the update will NOT invalidate the command buffer.
Descriptor binding flag VK_DESCRIPTOR_BINDING_PARTIALLY_BOUND_BIT: This flag indicates that descriptor set does not need to have valid descriptors in them as long as the invalid descriptors are not accessed during shader execution.
Non-uniform indexing extension in shaders: This extension allows us to access descriptors via variable or instruction that is not uniform across all invocations within a single invocation group.

Enable descriptor indexing features in Vulkan device

To keep things simple, we are going to utilize VkPhysicalDeviceFeatures2 to retrieve and enable all the descriptor indexing features from physical device:

VkPhysicalDeviceDescriptorIndexingFeatures descriptorIndexingFeatures{};
descriptorIndexingFeatures.sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_DESCRIPTOR_INDEXING_FEATURES;
descriptorIndexingFeatures.pNext = nullptr;

VkPhysicalDeviceFeatures2 deviceFeatures{};
deviceFeatures.sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_FEATURES_2;
deviceFeatures.pNext = &descriptorIndexingFeatures;

// Fetch all features from physical device
vkGetPhysicalDeviceFeatures2(physicalDevice, &deviceFeatures);

// Non-uniform indexing and update after bind
// binding flags for textures, uniforms, and buffers
// are required for our extension
assert(descriptorIndexingFeatures.shaderSampledImageArrayNonUniformIndexing);
assert(descriptorIndexingFeatures.descriptorBindingSampledImageUpdateAfterBind);
assert(descriptorIndexingFeatures.shaderUniformBufferArrayNonUniformIndexing);
assert(descriptorIndexingFeatures.descriptorBindingUniformBufferUpdateAfterBind);
assert(descriptorIndexingFeatures.shaderStorageBufferArrayNonUniformIndexing);
assert(descriptorIndexingFeatures.descriptorBindingStorageBufferUpdateAfterBind);


VkDeviceCreateInfo createDeviceInfo{};
createDeviceInfo.pNext = &deviceFeatures;

Create bindless descriptor set layout

Now that we have enabled descriptor indexing, we can create our descriptor set layout with bindings that are partially bound and can be updated after binding:

// Create three bindings: storage buffer,
// uniform buffer, and combined image sampler
std::array<VkDescriptorSetLayoutBinding, 3> bindings{};
std::array<VkDescriptorBindingFlags, 3> flags{};
std::array<VkDescriptorType, 3> types{
  VK_DESCRIPTOR_TYPE_UNIFORM_BUFFER,
  VK_DESCRIPTOR_TYPE_STORAGE_BUFFER,
  VK_DESCRIPTOR_TYPE_COMBINED_IMAGE_SAMPLER
};

for (uint32_t i = 0; i < 3; ++i) {
  bindings.at(i).binding = i;
  bindings.at(i).descriptorType = types.at(i);
  // Due to partially bound bit, this value
  // is used as an upper bound, which we have set to
  // 1000 to keep it simple for the sake of this post
  bindings.at(i).descriptorCount = 1000;
  bindings.at(i).stageFlags = VK_SHADER_STAGE_ALL;
  flags.at(i) = VK_DESCRIPTOR_BINDING_PARTIALLY_BOUND_BIT | VK_DESCRIPTOR_BINDING_UPDATE_AFTER_BIND_BIT;
}

VkDescriptorSetLayoutBindingFlagsCreateInfo bindingFlags{};
bindingFlags.sType = VK_STRUCTURE_TYPE_DESCRIPTOR_SET_LAYOUT_BINDING_FLAGS_CREATE_INFO;
bindingFlags.pNext = nullptr;
bindingFlags.pBindingFlags = flags.data();
bindingFlags.bindingCount = 3;

VkDescriptorSetLayoutCreateInfo createInfo{};
createInfo.sType = VK_STRUCTURE_TYPE_DESCRIPTOR_SET_LAYOUT_CREATE_INFO;
createInfo.bindingCount = 3;
createInfo.pBindings = bindings.data();
// Create if from a descriptor pool that has update after bind
// flag set
createInfo.flags = VK_DESCRIPTOR_SET_LAYOUT_CREATE_UPDATE_AFTER_BIND_POOL_BIT;

// Set binding flags
createInfo.pNext = &bindingFlags;

// Create layout
VkDescriptorSetLayout bindlessLayout = VK_NULL_HANDLE;
vkCreateDescriptorSetLayout(mDevice, &createInfo, nullptr, &bindlessLayout);

The binding flags are mapped to each binding that we pass to layout create info structure. In our case, all bindings have partially bound and update after bind flags.

VK_DESCRIPTOR_SET_LAYOUT_CREATE_UPDATE_AFTER_BIND_POOL_BIT flag requires the descriptor pool that creates descriptor sets with this layout to have update after bind flag set:

VkDescriptorPoolCreateInfo poolInfo{};
poolInfo.flags = VK_DESCRIPTOR_POOL_CREATE_UPDATE_AFTER_BIND_BIT;
// ... rest of descriptor pool creation

Create bindless descriptor set

Let’s create our bindless, global descriptor set:

VkDescriptorSetAllocateInfo allocateInfo{};
allocateInfo.sType = VK_STRUCTURE_TYPE_DESCRIPTOR_SET_ALLOCATE_INFO;
allocateInfo.pNext = nullptr;
// Pass the pool that is created with update after bind flag
allocateInfo.descriptorPool = mDescriptorPool;
// Pass the bindless layout
allocateInfo.pSetLayouts = &bindlessLayout;
allocateInfo.descriptorSetCount = 1;

// Create descriptor
VkDescriptorSet bindlessDescriptorSet = VK_NULL_HANDLE;
vkAllocateDescriptorSets(mDevice, &allocateInfo, &bindlessDescriptorSet);

That’s all for the initial setup of enabling bindless descriptor sets!

Manage bindless descriptors and communication

Now we need a way to write all our resources to this global descriptor set and a way to pass these resources to our pipelines.

Render handles

We can write data to any arbitrary index of a descriptor set. This index is typically of type uint32_t, whihc is aligned with GLSL's uint type. With that in mind, we are going to create two uint32_t handle types — Buffer and Texture — using scoped enums to have compile-time checks:

enum class TextureHandle : uint32_t { Invalid = 0 };
enum class BufferHandle : uint32_t { Invalid = 0; }

Note:

I am using zero as invalid handle as a personal preference. Another number that is typically used is maximum value of uint32_t. Use whichever you prefer as long as you are consistent.

Update bindless descriptors

For the sake of simplicity, let’s assume that image, image view, and buffers are already created. Let's create two methods that store the resourcse in an array and return us two handles that we can use in our descriptor:

static constexpr uint32_t UniformBinding = 0;
static constexpr uint32_t StorageBinding = 1;
static constexpr uint32_t TextureBinding = 2;

VkDescriptorSet bindlessDescriptorSet;
std::vector<VkImageView> textures;
std::vector<VkBuffer> buffers;

TextureHandle storeTexture(VkImageView imageView, VkSampler sampler) {
  size_t newHandle = textures.size();
  textures.push_back(imageView);

  VkImageInfo imageInfo{};
  imageInfos.imageLayout = VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL;
  imageInfo.imageView = imageView;
  imageView.sampler = sampler;

  VkWriteDescriptorSet write{};
  write.descriptorType = VK_DESCRIPTOR_TYPE_COMBINED_IMAGE_SAMPLER;
  write.dstBinding = TextureBinding;
  write.dstSet = bindlessDescriptor;
  // Write one texture that is being added
  write.descriptorCount = 1;
  // The array element that we are going to write to
  // is the index, which we refer to as our handles
  write.dstArrayElement = newHandle;
  write.pImageInfo = &imageInfo;

  vkUpdateDescriptorSets(mDevice, 1, &write, 0, nullptr);
  return static_cast<TextureHandle>(newHandle);
}

BufferHandle storeBuffer(VkBuffer buffer, VkBufferUsageFlagBits usage) {
  size_t newHandle = buffers.size();
  buffers.push_back(buffer);

  std::array<VkWriteDescriptorSet, 2> writes{};
  for (auto &write: writes) {
    VkBufferInfo bufferInfo{};
    bufferInfo.buffer = buffer;
    bufferInfo.offset = 0;
    bufferInfo.range = VK_WHOLE_SIZE;

    write.dstSet = bindlessDescriptor;
    // Write one buffer that is being added
    write.descriptorCount = 1;
    // The array element that we are going to write to
    // is the index, which we refer to as our handles
    write.dstArrayElement = newHandle;
    write.pBufferInfo = &bufferInfo;
  }

  size_t index = 0;
  if ((usage & VK_BUFFER_USAGE_UNIFORM_BUFFER_BIT) == VK_BUFFER_USAGE_UNIFORM_BUFFER_BIT) {
    writes.at(index).binding = UniformBinding;
    writes.at(index).type = VK_DESCRIPTOR_TYPE_UNIFORM_BUFFER;
    index++;
  }

  if ((usage & VK_BUFFER_USAGE_STORAGE_BUFFER_BIT) == VK_BUFFER_USAGE_STORAGE_BUFFER_BIT) {
    writes.at(index).binding = StorageBinding;
    writes.at(index).type = VK_DESCRIPTOR_TYPE_STORAGE_BUFFER;
  }

  vkUpdateDescriptorSets(mDevice, index, writes.data(), 0, nullptr);

  return static_cast<BufferHandle>(newHandle);
}

These two simple functions ensure that all stored textures will be written to descriptor as combined image samplers and all the stored buffers will be written to bindings depending on its usage flags.

Pass draw parameters to pipelines

There are two ways to access resources from bindless descriptors:

Accessing resources from other resources. For example, material data stores index to a texture, then the shader finds the texture based on index stored in material.
Accessing resources from draw parameter. For example, figuring out where the camera buffer is.

In this post, we are mainly going to focus on (2) and provide draw parameters to the pipelines by binding a dynamic uniform buffer with a dynamic offset. There are two reasons why will use dynamic uniform buffers:

We only need one descriptor and one buffer that manage all the data for all the passes
The indices stored in the uniform buffer will never change. The contents can change, the underlying buffer can be updates but the index will always be the same; so, we will never update this descriptor after initial setup.

Dynamic uniform padding alignment

Dynamic uniform buffer descriptors have one quirk that needs to be taken care of. Each range of data in the buffer must be padded to fit within the minimum alignment that is defined by the GPU. Luckily, we can retrieve this value from the GPU:

VkPhysicalDeviceProperties properties;
vkGetPhysicalDeviceProperties(physicalDevice, properties);

minUniformBufferOffsetAlignment =
  static_cast<uint32_t>(properties.limits.minUniformBufferOffsetAlignment);

uint32_t padSizeToMinAlignment(uint32_t originalSize, uint32_t minAlignment) {
  return (originalSize + minAlignment - 1) & ~(minAlignment - 1);
}

The padSizeToAlignment function returns the size that is a multiple of minAlignment using bitwise operations. Let’s assume that minAlignment=16 and originalSize=30.

~(minAlignment - 1) = 15 = ~(0x0000 000F) = 0xFFFF FFF0
(30 + 16 - 1) = 45 = 0x0000 002D

0x0000 002D & 0xFFFF FFF0 = 0x0000 00020 = 32 (multiple of 16)

Specify ranges during initialization

Firstly, let’s look at what kind of API that we want to provide to make our lives easier. Assume that we have three passes — PBR pass, skybox pass, and a text rendering pass:

PBR pass: mesh transforms, lights, and camera
Skybox pass: camera and skybox texture
Text pass: text transforms, camera, text glyphs buffer

Given that all the buffers and textures are stored and we have access to the handles, let’s create our structures:

struct PBRParams {
  BufferHandle meshTransforms;
  BufferHandle pointLights;
  BufferHandle camera;
  // Always pad the data in a way that
  // the entire structure can be represented
  // as bunch of vec4's.
  uint32_t pad0;
};

struct SkyboxParams {
  BufferHandle camera;
  TextureHandle skybox;
  uint32_t pad0;
  uint32_t pad1;
};

struct TextParams {
  BufferHandle textTransforms;
  BufferHandle camera;
  BufferHandle glyphsBuffer;
  uint32_t pad0;
};

Now, let’s add the data based on these structures to our bindless parameters:

BindlessParams bindlessParams(minUniformBufferOffsetAlignment);

auto rangePBR = bindlessParams.addRange(PBRParams({
  meshTransformsBuffer,
  pointLightsBuffer,
  cameraBuffer
});

auto rangeSkybox = bindlessParams.addRange(SkyboxParams({
  cameraBuffer,
  skyboxTexture
});

auto textParams = bindlessParams.addRange(TextParams({
  textTransformsBuffer,
  cameraBuffer,
  glyphsBuffer
});

// Build uniform buffer and descriptor
bindlessParams.build(mDevice, mAllocator, mDescriptorPool);

// Bind our bindless descriptor set once
// for all passes
vkCmdBindDescriptorSets(
  commandBuffer,
  bindPoint,
  pipelineLayout,
  0, &bindlessDescriptorSet,
  0, nullptr);

auto bindPoint = VK_PIPELINE_BIND_POINT_GRAPHICS;

// during rendering
// PBR pass
vkCmdBindPipeline(commandBuffer, bindPoint, pbrPipeline);
vkCmdBindDescriptorSets(
  commandBuffer,
  bindPoint,
  pipelineLayout,
  1, &bindlessParams.getDescriptorSet(),
  1, &rangePBR);
drawAllObjects();

// Skybox pass
vkCmdBindPipeline(commandBuffer, bindPoint, skyboxPipeline);
vkCmdBindDescriptorSets(
  commandBuffer,
  bindPoint,
  pipelineLayout,
  1, &bindlessParams.getDescriptorSet(),
  1, &rangeSkybox);
drawSkybox();

// Text pass
vkCmdBindPipeline(commandBuffer, bindPoint, textPipeline);
vkCmdBindDescriptorSets(
  commandBuffer,
  bindPoint,
  pipelineLayout,
  1, &bindlessParams.getDescriptorSet(),
  1, &rangeText);
drawAllTexts();

What’s happening?

All the ranges are specified before we even create the draw parameters buffer and descriptor since we know all the long living buffers and textures beforehand. The ranges are returned according to how the data is going to be stored in the buffer.
Create the buffer and descriptor set
Bind the descriptor set for draw parameters and pass the ranges that we have retrieved as dynamic offsets to vkCmdBindDescriptorSets command.

Given the API that we have, let’s create our class:

class BindlessParams {
  struct Range {
    uint32_t offset;
    uint32_t size;
    void *data;
  };
public:
  BindlessParams(uint32_t minAlignment) : mMinAlignment(minAlignment) {}

  template<class TData>
  uint32_t addRange(TData &&data) {
    // Copy data to heap and store void pointer
    // since we do not care about the type at
    // point
    size_t dataSize = sizeof(TData);
    auto *bytes = new TData;
    *bytes = data;

    // Add range
    uint32_t currentOffset = mLastOffset;
    mRanges.push_back({ currentOffset, dataSize, bytes });

    // Pad the data size to minimum alignment
    // and move the offset
    mLastOffset += padSizeToMinAlignment(dataSize, mMinAlignment);
    return currentOffset;
  }

  void build(VkDevice device, VmaAllocator allocator, VkDescriptorPool descriptorPool) {
    VkBufferCreateInfo createInfo{};
    createInfo.usage = VK_BUFFER_USAGE_UNIFORM_BUFFER_BIT;
    createInfo.size = mLastOffset;
    // Other create flags
    vmaCreateBuffer(allocator, &createInfo, ..., &mBuffer, &mAllocation, ...);

    // Copy ranges to buffer
    uint8_t *data = nullptr;
    vmaMapMemory(allocator, allocation, &data);
    for (const auto &range : mRanges) {
      memcpy(data + range.offset, range.data, range.size);
    }
    vmaUnmapMemory(allocator, allocation);

    // Create layout for descriptor set
    VkDescriptorSetLayoutBinding binding{};
    binding.binding = i;
      binding.descriptorType = VK_DESCRIPTOR_TYPE_UNIFORM_BUFFER_DYNAMIC;
    binding.descriptorCount = 1;
      binding.stageFlags = VK_SHADER_STAGE_ALL;

        VkDescriptorSetLayoutCreateInfo createInfo{};
        createInfo.sType = VK_STRUCTURE_TYPE_DESCRIPTOR_SET_LAYOUT_CREATE_INFO;
        createInfo.bindingCount = 1;
        createInfo.pBindings = &binding;
        vkCreateDescriptorSetLayout(device, &createInfo, nullptr, &mLayout)

    // Get maximum size of a single range
    uint32_t maxRangeSize = 0;
    for (auto &range : mRanges) {
      maxRangeSize = std::max(range.size, maxRangeSize);
    }

    // Create descriptor
        VkDescriptorSetAllocateInfo allocateInfo{};
        allocateInfo.sType = VK_STRUCTURE_TYPE_DESCRIPTOR_SET_ALLOCATE_INFO;
        allocateInfo.pNext = nullptr;
        allocateInfo.descriptorPool = descriptorPool;
        allocateInfo.pSetLayouts = &mLayout;
        allocateInfo.descriptorSetCount = 1;
        vkAllocateDescriptorSets(mDevice, &allocateInfo, &mDescriptorSet);

      VkBufferInfo bufferInfo{};
      bufferInfo.buffer = mBuffer;
      bufferInfo.offset = 0;
      bufferInfo.range = maxRangeSize;

      VkWriteDescriptorSet write{};
      write.descriptorType = VK_DESCRIPTOR_TYPE_UNIFORM_BUFFER_DYNAMIC;
      write.dstBinding = 0;
      write.dstSet = mDescriptor;
      write.descriptorCount = 1;
      write.dstArrayElement = 0;
      write.pBufferInfo = &bufferInfo;
      vkUpdateDescriptorSets(mDevice, 1, &write, 0, nullptr);
  }

  inline VkDescriptorSet getDescriptorSet() { return mDescriptorSet; }

  inline VkDescriptorSetLayout getDescriptorSetLayout() { return mLayout; }

private:
  uint32_t mMinAlignment;
  uint32_t mLastOffset = 0;
  std::vector<Range> mRanges;

  VkDescriptorSetLayout mLayout;
  VkDescriptorSet mDescriptorSet;

  VmaAllocation mAllocation;
  VkBuffer mBuffer;
};

The addRange function returns the dynamic offset for the descriptor set that we want to bind and the build function creates the buffer, the descriptor set layout, and descriptor set.

The one tricky part here is how dynamic uniform buffer descriptor set actually works. The dynamic descriptor set’s offset is defined by the static offset that we pass when creating the descriptor, buffer starting position, and the dynamic offset:

starting offset + buffer address + dynamic offset

That’s why the static range for the buffer that we set is always from [0, MaximumRangeSize]. The reason why we use the largest size from added ranges is because we want to cover the largest range of the buffer while the actual range that will be read is defined in the shader uniform.

Register buffers in shaders

We are now done with setting up everything in application side. Now, it is time to set up a way to easily add bindless structures in our shaders with very minimal changes. Shaders typically allow setting up aliases for the same bindings:

#version 430

layout(set = 0, binding = 0) uniform Data1 { uint value; } uData1[];
layout(set = 0, binding = 0) uniform Data2 { vec4 value; } uData2[];

We are going to utilize this feature by combining it with preprocessor definitions. Note that, this feature is only needed for buffers as there are fixed number of sampler types while buffer structures can be different depending on their purpose. Just like C/C++ preprocessors, GLSL preprocessors allow us to generate code using macros. Let's define three macro functions that will generate the aliases for us.

// filename: bindless.glsl
// Add non-uniform qualifier extension here;
// so, we do not add it for all our shaders
// that use bindless resources
#extension GL_EXT_nonuniform_qualifier : enable

#define Bindless 1

// We always bind the bindless descriptor set
// to set = 0
#define BindlessDescriptorSet 0
// These are the bindings that we defined
// in bindless descriptor layout
#define BindlessUniformBinding 0
#define BindlessStorageBinding 1
#define BindlessSamplerBinding 2

#define GetLayoutVariableName(Name) u##Name##Register

// Register uniform
#define RegisterUniform(Name, Struct) \
  layout(set = BindlessDescriptorSet, binding = BindlessUniformBinding) \
      uniform Name Struct \
      GetLayoutVariableName(Name)[]

// Register storage buffer
#define RegisterBuffer(Layout, BufferAccess, Name, Struct) \
  layout(Layout, set = BindlessDescriptorSet, \
         binding = BindlessStorageBinding) \
  BufferAccess buffer Name Struct GetLayoutVariableName(Name)[]

// Access a specific resource
#define GetResource(Name, Index) \
  GetLayoutVariableName(Name)[Index]

// Register empty resources
// to be compliant with the pipeline layout
// even if the shader does not use all the descriptors
RegisterUniform(DummyUniform, { uint ignore; });
RegisterBuffer(std430, readonly, DummyBuffer, { uint ignore; });

// Register textures
layout(set = BindlessDescriptorSet, binding = BindlessSamplerBinding) \
    uniform sampler2D uGlobalTextures2D[];
layout(set = BindlessDescriptorSet, binding = BindlessSamplerBinding) \
    uniform samplerCube uGlobalTexturesCube[];

Here is an example with the generated code:

#version 430
#include "bindless.glsl"

RegisterUniform(Camera, {
  mat4 viewProjection;
  mat4 view;
  mat4 projection;
});

struct TransformData {
  mat4 transform;
};

RegisterBuffer(std430, readonly, Transforms, {
  TransformData items[];
});

void main() {
  mat4 viewProjection = GetResource(Camera, 5).viewProjection;
  // ... do stuff
}

// Generated code
#version 430
#include "bindless.glsl"

layout(set = 0, binding = 0) uniform Camera {
  mat4 viewProjection;
  mat4 view;
  mat4 projection
} uCameraRegister[];

struct TransformData {
  mat4 transform;
};

layout(std430, set = 0, binding = 1) readonly buffer Transforms {
  TransformData items[];
} uTransformRegister[];

void main() {
  gl_Position = uCameraRegister[5].viewProjection * vec4(inPosition, 1.0);
}

This template system allows us to easily add bindless resource aliases for buffers. Now, let’s create our draw parameters uniform (based on PBR pipeline in previous sections):

// Our bindless registrations here...

// Draw parameters
layout(set = 1, binding = 0) uniform DrawParameters {
  uint meshTransforms;
  uint pointLights;
  uint camera;
  // Don't forget the padding
  uint pad0;
} uDrawParameters;

void main() {
  gl_Position = GetResource(Camera, uDrawParameters.camera) * vec4(inPosition, 1.0);
}

This is it for registering any resource we like to the bindless descriptor set.

If you want to take it one step further, I have found that having utilities that tightly couples bindless resources with draw parameters removes a lot of duplication, especially for very common resources such as cameras. Here is a utility below that registers and provides a macro to retrieve cameras in the shaders.

// filename: camera.glsl
RegisterUniform(Camera, {
  mat4 viewProjection;
  mat4 view;
  mat4 projection;
});

#define getCamera() GetResource(Camera, uDrawParameters.camera)

I can include this utility and use it as long as I have a camera property defined in draw parameters:

#include "bindless.glsl"
#include "camera.glsl"

layout(set = 1, binding = 0) uniform DrawParameters {
  uint meshTransforms;
  uint pointLights;
  uint camera;
  // Don't forget the padding
  uint pad0;
} uDrawParameters;

void main() {
  gl_Position = getCamera().viewProjection * vec4(inPosition, 1.0);
}

Summary

Managing descriptors in Vulkan is a very frustrating experience that requires setting up complex and sometimes expensive systems to manage the lifecycles and bindings of descriptor sets. Bindless design solves all the pain points of managing descriptor sets by only having one descriptor set that can be used by all resources; however, it also brings a lot of new challenges to the table. When using bindless design, in essence, we need to create our own variable system that can access the required resources from shaders.

After lots of research and experimentaion, I have found that combination of using render handles to identify resources, dynamic uniform buffers, and a macro-based template system in the shaders to register different types of buffers simplifies using bindless design a lot. Just like anything in Vulkan, there is quiet a lot of boilerplate that needs to be set but in my opinion, long term gains are worth the initial investment of setting up a system that can manage these resources.

I hope you enjoyed this post!

Changelog

[05/14/2023] Used VkPhysicalDeviceFeatures2 to enable descriptor indexing features.