In April 2021, the Vulkan Working Group at Khronos released a set of provisional extensions, collectively called “Vulkan Video,” for seamlessly integrating hardware-accelerated video compression and decompression into the Vulkan API. Today, Khronos is releasing finalized extensions that incorporate industry feedback and expose core and decode Vulkan Video functionality to provide fully accelerated H.264 and H.265 decode.
Khronos will release an ongoing series of Vulkan Video extensions to enable additional codecs and accelerated encode as well as decode. This blog is a general overview of the Vulkan Video architecture and also provides details about the finalized extensions and links to important resources to help you create your first Vulkan Video applications.
Introduction
Vulkan Video adheres to the Vulkan philosophy of providing applications flexible, fine-grained control over scheduling, synchronization, and memory allocation. By leveraging the existing Vulkan API framework, the Vulkan Video extensions enable efficient, low-latency, low-overhead use of processing resources for accelerated video processing, including distributing stream processing tasks across multiple CPU cores and video codec hardware—all with application portability across multiple platforms and devices ranging from small embedded devices to high-performance servers supporting Windows and Linux.
Vulkan Video Extensions Overview
GPUs typically contain dedicated video decode and encode acceleration engine(s) that are independent from other graphics and compute engines. In fact, some physical devices may support only video decode and/or video encode acceleration. Consequently, Vulkan Video adds video decode and encode queues to support these video operations.
Also, the field of video codecs is always changing. More advanced and domain-specific video coding tools make it easier to compress and decompress video, which leads to new codecs and codec extensions. So, Vulkan Video was designed to be flexible enough to support a wide range of existing and future codecs. It does this by including both universal “core” extensions that should be useful for all codecs and codec-specific extensions that are only useful for one codec type. Figure 2 depicts Vulkan Video extensions along with their status and relations.
This release of Vulkan Video finalizes the following extensions:
- VK_KHR_video_queue: common APIs for all video coding operations.
- VK_KHR_video_decode_queue: common APIs for all video decode operations.
- VK_KHR_video_decode_h264: H.264 decode-specific capabilities and parameters (promoted from EXT to KHR in this final release).
- VK_KHR_video_decode_h265: H.265 decode-specific capabilities and parameters (promoted from EXT to KHR in this final release).
Collectively, the above extensions allow exposing H.264 and H.265 video decode acceleration through Vulkan.
The following encode-related extensions remain provisional and are expected to be finalized in 2023 along with new extensions to be introduced for VP9 decode and AV1 decode/encode operations:
- VK_KHR_video_encode_queue: common APIs for all video encode operations.
- VK_EXT_video_encode_h264: H.264 encode-specific capabilities and parameters.
- VK_EXT_video_encode_h265: H.265 encode-specific capabilities and parameters.
As an example, a Vulkan Video implementation that only supports H.264 decoding would only expose support for the VK_KHR_video_queue, VK_KHR_video_decode_queue, and VK_KHR_video_decode_h264 extensions, and an application would use all three extensions together to perform H.264 decode operations on that target device.
The standard vkGetPhysicalDeviceQueueFamilyProperties2 API may be used to determine support for codec operations, such as H.264 decode, by chaining VkVideoQueueFamilyPropertiesKHR to retrieve VkVideoCodecOperationFlagsKHR.
For more information, visit this page.
