Buffer device addresses in Vulkan and VMA

A while ago, I posted about managing bindless descriptors in Vulkan with heavy focus on setting up a system for accessing buffers of different structures using descriptor indexing.

As I recently learned about buffer device addresses, I want to talk about device buffer addresses and how they simplify managing and accessing buffers by getting rid of descriptors and making buffer access more intuitiveş

What are buffer device addresses?

Buffer device address (VK_KHR_buffer_device_address) is a Vulkan extension that enables to fetching memory address of a buffer in GPU and using GLSL_EXT_buffer_reference in shaders.

GLSL buffer reference

GLSL buffer reference (GLSL_EXT_buffer_reference) shader extension enables defining structures buffer_reference layout and using those structures in other storages blocks such as uniforms, push constants, and buffers:

layout(buffer_reference, std430, buffer_reference_align=16) buffer MyBuffer {
  vec4 color;
};

layout(push_constant) uniform Data {
  // Maps device address stored in push constant
  // to MyBuffer structure.
  MyBuffer buffer;
} pcData;

void main() {
  // Using the buffer directly from push constant
  // as if it is an object.
  vec4 color = pcData.buffer.color;
} 

Setup

Before we start with our application code, we need to enable creating buffers with .

Enable device feature

Now, we need to enable the device feature. To use this feature we are going to utilize VkPhysicalDeviceFeatures2, which we can pass as pNext device create info structure:

// Create the feature chain
VkPhysicalDeviceDescriptorIndexingFeatures descriptorIndexingFeatures{};
descriptorIndexingFeatures.sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_DESCRIPTOR_INDEXING_FEATURES;

// Pass your other features through this chain
descriptorIndexingFeatures.pNext = nullptr;

VkPhysicalDeviceBufferDeviceAddressFeatures bufferDeviceAddressFeatures{};
bufferDeviceAddressFeatures.sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_BUFFER_DEVICE_ADDRESS_FEATURES;
bufferDeviceAddressFeatures.pNext = &descriptorIndexingFeatures;

VkPhysicalDeviceFeatures2 deviceFeatures{};
deviceFeatures.sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_FEATURES_2;
deviceFeatures.pNext = &bufferDeviceAddressFeatures;

// Fetch all features
vkGetPhysicalDeviceFeatures2(physicalDevice, &deviceFeatures);

// Buffer device address feature is required to exist
// in the physical device
assert(bufferDeviceFeatures.bufferDeviceAddress);

createDeviceInfo.pNext = &deviceFeatures;
// create device

Note:

If you are using Vulkan < 1.2, you need to enable the extension explicitly:

std::vector<const char *> extensions;
// other extensions
// Add buffer device address extension
extensions.push_back(VK_KHR_BUFFER_DEVICE_ADDRESS_EXTENSION_NAME);
createDeviceInfo.ppEnabledExtensionNames = extensions.data();
createDeviceInfo.enabledExtensionCount = static_cast<uint32_t>(extensions.size());

Enable buffer device addresses in VMA

Now that the feature is enabled, we need to enable buffer device addresses in Vulkan Memory Allocator:

VmaAllocatorCreateInfo createInfo{};
createInfo.instance = vulkanInstance;
createInfo.physicalDevice = vulkanPhysicalDevice;
createInfo.device = vulkanDevice;
// Pass the buffer device address flag
createInfo.flags = VMA_ALLOCATOR_CREATE_BUFFER_DEVICE_ADDRESS_BIT;
vmaCreateAllocator(&createInfo, &mAllocator);

Adding VMA_ALLOCATOR_CREATE_BUFFER_DEVICE_ADDRESS_BIT flags automatically adds VK_MEMORY_ALLOCATE_DEVICE_ADDRESS_BIT when new memory blocks are allocated by VMA.

Enable buffer access from shaders

Let's add VK_BUFFER_USAGE_SHADER_DEVICE_ADDRESS_BIT usage flag during buffer creation to allow accessing these buffers from shaders.

VkBufferUsageFlags bufferUsage = VK_BUFFER_USAGE_SHADER_DEVICE_ADDRESS_BIT;

VkBufferCreateInfo createBufferInfo{};
createBufferInfo.sType = VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO;
createBufferInfo.pNext = nullptr;
createBufferInfo.flags = 0;
createBufferInfo.size = description.size;
createBufferInfo.usage = bufferUsage;

// create device here

That's it! Now we can query and access device addresses of buffers in our application and use the addresses to access buffers in shaders.

Pass buffer device addresses to shaders

After enabling the buffer device address, let's fetch the device addresses of buffers and send them to shaders:

Introducing new type in our application

Buffer device addresses have type uint64_t. In order to have compile time guarantees and between uint64_t and device addresses, we are going to introduce a new type:

enum class DeviceAddress : uint64_t { Invalid = 0 };

This will provide minor assurance for us to not accidentally pass unintended values within our application.

Fetch buffer device addresses

Let's introduce a function that fetches device address of a buffer:

DeviceAddress getBufferDeviceAddress(VkBuffer buffer) {
  VkBufferDeviceAddressInfo addressInfo{};
  addressInfo.sType = VK_STRUCTURE_TYPE_BUFFER_DEVICE_ADDRESS_INFO;
  addressInfo.pNext = nullptr;
  addressInfo.buffer = buffer;
  return static_cast<DeviceAddress>(
      vkGetBufferDeviceAddress(mDevice, &addressInfo));
}

Pass device addresses to pipelines

We are going to use the same BindlessParams system to pass draw parameters to shaders. Nothing inside the BindlessParams is going to change. Let's also use the same example in the previous post but with DeviceAddress instead of BufferHandle:

struct PBRParams {
  DeviceAddress meshTransforms;
  DeviceAddress pointLights;
  DeviceAddress camera;
};

struct SkyboxParams {
  DeviceAddress camera;
  TextureHandle skybox;
  uint32_t pad0;
};

struct TextParams {
  DeviceAddress textTransforms;
  DeviceAddress camera;
  DeviceAddress glyphsBuffer;
};

BindlessParams bindlessParams(minUniformBufferOffsetAlignment);

auto rangePBR = bindlessParams.addRange(PBRParams({
  getBufferDeviceAddress(meshTransformsBuffer),
  getBufferDeviceAddress(pointLightsBuffer),
  getBufferDeviceAddress(cameraBuffer)
});

auto rangeSkybox = bindlessParams.addRange(SkyboxParams({
  getBufferDeviceAddress(cameraBuffer),
  getBufferDeviceAddress(skyboxTexture)
});

auto textParams = bindlessParams.addRange(TextParams({
  getBufferDeviceAddress(textTransformsBuffer),
  getBufferDeviceAddress(cameraBuffer),
  getBufferDeviceAddress(glyphsBuffer)
});

bindlessParams.build(mDevice, mAllocator, mDescriptorPool);

As you can see, we do not need any descriptor to store buffers like we did with descriptor indexing. We still use descriptor indexing for textures and use dynamic uniform buffer to pass draw parameters to the shaders but we do not need to manage any descriptors for buffers.

Accessing buffers in shaders

We now pass buffer addresses to shaders, now it is time to utilize these addresses using GLSL buffer_reference extension.

Macro to define buffers references with ease

To make our lives easier, we are going to create a tiny preprocessor macro to easily define buffer reference buffers in our shaders:

#define Buffer(Alignment) \
  layout(buffer_reference, std430, buffer_reference_align = Alignment) buffer

Define buffer references

Now, let's use the same buffer layouts that we defined in our previous post with this new approach:

Buffer(64) Camera {
  mat4 viewProjection;
  mat4 view;
  mat4 projection;
};

struct TransformData {
  mat4 transform;
};

Buffer(64) Transforms {
  TransformData items[];
};

Use buffer references in uniforms

After we defined our buffers with buffer_reference layout, we can use the buffers in our draw parameters uniform:

// Draw parameters
layout(set = 1, binding = 0) uniform DrawParameters {
  Transforms meshTransforms;
  PointLights pointLights;
  Camera camera;
  // Don't forget the padding
  uint pad0;
} uDrawParameters;

void main() {
  mat4 transform = uDrawParameters.meshTransforms[0].transform;
  mat4 view = uDrawParameters.camera.view;
}

That's it!

As you can see, buffer device addresses significantly simplifies buffer access management by eliminating the need for descriptors while also providing a more intuitive API in shaders to access them.

This was my first attempt in buffer device addresses. As a next step I am going to try to use buffer device addresses for vertex buffers as well, which can eliminate the need for multiple pipeline layouts when combining it with descriptor indexing my vertex or descriptor layouts can be unified for all pipelines.