protonscr

Steam Overlay's Vulkan layer breaks when presenting with swapchains from multiple VkDevices

steamopen overlayProton
ValveSoftware/steam-for-linux#9120 · opened 2023-01-19 by etang-cw · updated 2023-01-19 · 0 comments · github
Eetang-cw 2023-01-19 github

Your system information

  • Steam client version: 1671236931
  • Distribution: Fedora 37 (KDE)
  • Opted into Steam client beta?: No
  • Have you checked for system updates?: Yes

Please describe your issue in as much detail as possible:

steamoverlayvulkanlayer.so saves VkDevices used with vkQueuePresentKHR in a global variable, which it then uses to free descriptor sets and textures and stuff in vkDestroyDevice. If an application calls vkQueuePresentKHR with queues from multiple different VkDevices, this can make it attempt to free descriptor sets and textures with a device that didn't create them, which causes crashes or VK_ERROR_DEVICE_LOSTs depending on the driver.

(Note: Obviously I can't see this from the disassembly, but the function freeing things might be a destructor, in which case it might not occupy any lines of code in its current state. Search destructors for one that accesses a global VkDevice and you might need to replace it with a more manual destruction that takes a VkDevice as input. If it helps, the bad function is at offset 0x76e0 in the steamoverlayvulkanlayer.so with sha1 2e2552341f569ebf505910381a0845600938d42e as viewed by objdump, for anyone who might happen to have debug symbols for that file.)

Steps for reproducing this issue:

There's probably other games that hit this, but the one I've found also has a separate, unrelated bug that needs to be worked around before you can hit the overlay bug.

  1. Apply this patch to a copy of proton wine, then build the containing proton and install it into steam. This will make all calls to operator new return zeroed memory, working around a bug in the game so you can hit the steam overlay bug. (Otherwise the game will crash immediately on startup, which makes it hard to hit other bugs.)
  2. Download CHAOS CODE -NEW SIGN OF CATSTROPHE-
  3. Select the patched proton to run the game
  4. Make sure steam overlay is enabled in the game preferences
  5. Launch the game.
  6. Sit through the opening cutscene (or the test pattern if you haven't transcoded it)
  7. Make sure you make steam overlay appear during the opening cutscene, either to show a notification, or if that doesn't happen (e.g. because the api dump layer made the game run so slowly the notification dismissed before the cutscene started), with shift-tab.
  8. When the cutscene finishes, the game will either crash or hang, depending on how your graphics drivers deal with the issue.
  9. If you run with VK_LAYER_LUNARG_api_dump, you can search for the first use of vkFreeDescriptorSets and compare the device supplied there with the device used to create the object with that ID.

Internal things that lead up to the crash/freeze:

  • The game starts video playback using vmr9
  • vmr9 creates a (second, since the game already has one) d3d9 device
  • dxvk creates a second vulkan device
  • The video finishes playing
  • vmr9 destroys its d3d9 device
  • dxvk destroys the second vulkan device
  • steam overlay tries to free its vulkan objects, but accidentally passes the wrong vulkan device to the free functions
  • (on intel vk driver) command buffer submissions fail with VK_DEVICE_LOST
  • dxvk isn't very good at handling VK_DEVICE_LOST and deadlocks

Here's a proton log of the game with the VK_LAYER_LUNARG_api_dump layer enabled: steam-536560-apidump.log.xz.zip. If you search it for 3375691357, you can see the VkDescriptorSet with that ID being allocated with device 0xca4df1b0 (the video's VkDevice) but freed with device 0x7db30a10 (the game's VkDevice). A large number of other objects are freed at the same time (right after the call to wine_vkDestroyDevice), all with the wrong VkDevice.

And here's a (partial) stack trace of that call to vkFreeDescriptorSets:

0x42e60760 anv_FreeDescriptorSets /usr/lib/libvulkan_intel.so+0x60760
0xf37c171f ???                    ~/.local/share/Steam/ubuntu12_32/steamoverlayvulkanlayer.so+0x771f
0xf37c2c48 ???                    ~/.local/share/Steam/ubuntu12_32/steamoverlayvulkanlayer.so+0x8c48
0xf37c8343 vkDestroyDevice+51     ~/.local/share/Steam/ubuntu12_32/steamoverlayvulkanlayer.so+0xe343
0xf373f92a ???                    ~/.local/share/Steam/ubuntu12_32/libVkLayer_steam_fossilize.so+0x4392a

Error codes