Please see the recently released Vulkan extension VK_EXT_descriptor_buffer. This should allow you to use a memory mapped area for descriptors and avoid all kind of bridging overhead.
I can't comment on whether DXVK would adopt this extension as well, however it should be possible to layer the traditional Vulkan binding model on top of this extension which should be good enough for Venus's use case.
Doing what @ishitatsuyuki suggests could turn out to be useful for drivers besides venus too https://gitlab.freedesktop.org/mesa/mesa/-/issues/7728
Supporting push descriptors would be rather difficult for us, and only work under specific circumstances.
maxPushDescriptors is 32 on most drivers, which is far from enough for many modern D3D11 games, which often use anything from 50-70 resources per draw in their main render pass.Furthermore, if the goal is to only update dirty descriptors, there are more issues which are hard to resolve and would substantially increase CPU overhead on native implementations:
MAP_WRITE_DISCARD and friends), so we'd need to track which slots a buffer and any of its views are bound to. The current solution is to just update the buffer descriptor sets in their entirety.If Zink somehow manages to deal with all of this efficiently, there's some serious amount of magic going on, but given that supporting this would require yet another large-scale rework and would most certainly come at a cost for non-virtualized environments, I can't say I'm particularly interested in supporting that upstream, especially since our current implementation is fine for native hardware drivers.
For the record, we already use plain vkUpdateDescriptorSets rather than templated updates on 64-bit builds, and we also don't use things like UPDATE_AFTER_BIND.
Re @ishitatsuyuki @nanokatze
Driver side Venus runs inside the guest OS. For guest side mapping, currently we still force external memory as described here: https://docs.mesa3d.org/drivers/venus.html#vk-memory-property-host-visible-bit
For VK_EXT_descriptor_buffer, it's still yet unclear to me whether the memory backing the new descriptor buffer usages can be exported or not (or whether it's supposed to).
Re @doitsujin
Thanks for the details! I'll need some time to digest those after Thanks Giving holidays ; )
Assuming DXVK won't integrate push descriptors, is it possible to maintain a descriptor set cache to reduce the number of per frame descriptor updates? The number of update calls in DXVK currently varies a lot for different titles, which indeed adds some non-trivial VM overhead when the number goes beyond hundreds to big couple of hundreds.
Venus would be happy with either push descriptors support (where we can dynamically batch them) or a maintained descriptor set cache (like what ANGLE does in its Vulkan backend). As you mentioned, indeed neither will affect native platforms much, since the templated update in native env can buys you the same performance rather than caching.
Looks like Zink only uses push descriptors for the GL uniform descriptor set.
Everything else (UBOs, SSBOs, storage images, sampled images) goes through the regular uniform set update path.
Zink also only seems to only create pipelines that contain all shader stages.
Caching descriptor sets is also highly impractical. We can't predict resource lifetimes, we'd additionally have to manage lifetimes of individual descriptor sets, we'd have to deal with descriptor pool fragmentation in some way to not use too much memory, and the key to look up sets would be an arbitrarily sized array of (resource cookie, vk view handle / buffer offset) tuples (i.e. 16 bytes per descriptor) since apps have total freedom on what to bind when and where.
Also, if Zink's past experience with descriptor set caching is anything to go by, it's straight-up going to be much slower than simply updating descriptor sets on native drivers as well, so we'd need to keep the current code around.
One thing I tried in the past was to just use UPDATE_AFTER_BIND and defer descriptor set updates to our command submission thread, so that we'd only have one single call to vkUpdateDescriptorSets per command list and have it run on a much less busy thread. Would that be helpful at all here? This was also significantly slower on native drivers that the current path so I dropped the idea, but it's certainly easier to maintain alongside the current path than any of the proposed alternatives.
In the end though, making descriptor set updates a slow path is always going to hurt DXVK in one way or another, and probably many other Vulkan applications (certain games, emulators, vkd3d-proton) as well. We're going to have this problem with Wine in the future as well, and it's really annoying, but really doesn't look like there's any way at all to maintain the level of performance that we currently have when high-frequency operations like that have significant overhead.
Regarding EXT_descriptor_buffer, I have no immediate plans to integrate this into DXVK either. It would solve some annoyances we currently have with descriptor set allocation (vkAllocateDescriptorSets is already so expensive that we need to avoid it).
However it introduces vkGetDescriptorEXT, and it goes without saying that DXVK cannot pay the full syscall penalty for every descriptor in the worst case. Caching would work for image and sampler descriptors, but not for (most) buffer descriptors since buffer addresses will change arbitrarily between draws. It also negatively affects GPU performance on some vendors.
I feel like a potential long-term solution would involve something like EXT_descriptor_buffer for image descriptors, as well as for buffer descriptors in pipelines created from pipeline libraries. For optimized pipelines we'd have to put buffer descriptors into push descriptors to avoid the downsides of descriptor buffers. However, this would be another major rewrite of our descriptor model (would take months, the rewrite I had to do for GPL already took a while but this would be much, much more complex) and we'd still have to keep a legacy descriptor set path for drivers that don't support the required features, so I'm not sure how feasible this is at all.
Using VK_EXT_descriptor_buffer and VK_KHR_push_descriptor seems promising.
Each vkGetDescriptorEXT call is a synchronous call from guest venus driver to the host venus renderer to retrieve the new descriptor blob back to the client, and that could be better than alloc + update overhead but still worse than the pure push descriptor set model.
So for runtime frames, it'd be awesome if the client can use push sets at least for certain descriptor types while doing some caching for the rest, which is good for mitigating the overhead of vkGetDescriptorEXT or the legacy alloc + update way even in the native environment I think.
@ishitatsuyuki Sorry I missed your reply earlier. Venus won't like VK_EXT_descriptor_buffer unless the clients are doing enough caching with the descriptors. The reasons are:
vkGetDescriptorEXT. The original single set allocation/push call now becomes N vkGetDescriptorEXT calls via the ring, which increases the ring traffic overhead. Besides, vkGetDescriptorEXT is expecting real descriptor blobs from the host driver so that the client can directly write to the bound descriptor buffer in the command buffer. The call itself has to be synchronous (expecting a reply), and would force the guest cpu to stall to wait for such reply.Nothing extracted yet.
The context is for Vulkan virtualization (MESA Venus driver). The legacy descriptor update paths are based on the standard Vulkan api calls (either direct update or update with template). For some heavy titles which does hundreds of updates per frame, that's a lot of traffic for virtualization.
The templated update favors the native environment since the native Vulkan driver can see the app provided blobs. For virtualization, we have to do the extra encode/decode to make the guest side descriptor contents available to the host (mostly translate the template version to non-template updates).
Zink, which translates GL to Vulkan, also shares similar behavior withDXVKif withoutVK_KHR_push_descriptorsupport. We have seen a perf hit there when comparing with native GL performance over real game titles as well as benchmarks.VK_KHR_push_descriptormitigates most of the overhead from descriptor updates because the guest Venus driver always batches the cmds to reduce traffic.Could DXVK also integrate the VK_KHR_push_descriptor extension? This would benefit gaming on ChromeOS a lot. Thanks!