protonscr

RDR2 (DX12): vkCreateGraphicsPipelines assertion hit on initial pipeline warmup screen

vkd3dclosed
HansKristian-Work/vkd3d-proton#2522 · opened 2025-06-22 by whitebelyash · updated 2025-08-21 · 10 comments · github
Wwhitebelyash 2025-06-22 github

Software information

Red Dead Redemption 2 with DX12 renderer/API

System information

  • GPU: AMD Radeon RX 570
  • Driver: Mesa 25.2.0-devel (git-38f8e82)
  • Wine version: wine-10.8 (Staging)/GE-Proton10-4
  • VKD3D-Proton version: 2.14.1

Log files

No debug - wine.log
vkd3d trace - wine_debug.log
(using wine-10.8-staging here)

Description

Setting RDR2 renderer API to DX12 crashes the game on pipeline cache warmup screen (grey progress bar screen before the R* logo) with vkCreateGraphicsPipelines assertion hit:

0158:err:msvcrt:_wassert (L"!status && \"vkCreateGraphicsPipelines\"",L"dlls/winevulkan/loader_thunks.c",3195)

This occurs both on vanilla wine-staging and GE-Proton 10-4. Using AMDVLK makes no difference. Upstream vkd3d boots the game (used by CodeWeavers to run RDR2 on macOS).
Manually removing assertions from loader_thunks.c allows the game to load up to the gameplay without graphical artifacts even though it can eventually crash with another assertion:

0368:err:msvcrt:_wassert (L"!status",L"../src-wine/dlls/winevulkan/loader.c",710)

(at least on GE-Proton10-4). Skipped assertions are probably the cause, because sometimes rendering breaks before hitting assert:

Image

The interesting thing - if pipeline cache is already present (sga_win32_* files in the game settings dir) then RDR2 boots to the menu, but eventually crashes right before rendering the first frame with the same assertion hit (vkCreateGraphicsPipelines). Switching GPUs doesn't affect anything even though the game rebuilds warmup cache (judging by CPU load).

This issue isn't present on Honeykrisp driver (tested on Apple M1)
GTA V Enhanced also uses same renderer (SGA with DX12 driver) and works out-of-the-box since the release day.

Although RDR2 uses a fully-working Vulkan renderer by default, it has a memory leak on RDNA2+ hardware with ReBAR enabled. Using DX12 can workaround this issue if disabling ReBAR/SAM isn't an option.
FSR4 also requires DX12 renderer by obvious reasons.

This is probably different to https://github.com/HansKristian-Work/vkd3d-proton/issues/1623 (the game now simply crashes with assert hit) I got same problems (including hang) after more testing just as in #1623.

Using dxilconv instead of builtin DXBC compiler fixes this issue and #1506 as well.

Mmbriar 2025-06-22 github

This looks like exactly the same issue as #1623, no? It just crashes compiling some shader (due to invalid spirv generated by vkd3d-proton's dxbc compiler), which has also already been dumped: https://github.com/HansKristian-Work/vkd3d-proton/issues/1623#issuecomment-2329654192

Wwhitebelyash 2025-06-22 github

It just crashes compiling some shader (due to invalid spirv generated by vkd3d-proton's dxbc compiler), which has also already been dumped: [#1623 (comment)](https://github.com/HansKristian-Work/vkd3d-proton/issues/1623#issuecomment-2329654192)

Probably. I remember SPIR-V errors on the same stage, but they disappeared since 2025, now the only message I see is an assertion hit.

Though, the question, why it works on Honeykrisp then but not on AMD (even with amdvlk). I tried spoofing the GPU and it still crashes at same place.

Mmbriar 2025-06-22 github

The assertion is just telling you that it crashed inside vkCreateGraphicsPipelines, most likely due to the invalid spirv passed to the driver. Something in wine/winevulkan changed at some point, so it now gives you this assertion message instead of just crashing or hanging silently, but that doesn't change anything about the root cause.

Though, the question, why it works on Honeykrisp then but not on AMD (even with amdvlk). I tried spoofing the GPU and it still crashes at same place.

Maybe it for some reason or another doesn't crash on the invalid spirv, doesn't really matter though. If there is invalid spirv generated by vkd3d-proton, the problem is already found.

Wwhitebelyash 2025-06-22 github

but that doesn't change anything about the root cause

Seems like old Wine versions just ignored broken calls, at least on 7.22-staging the game loads.

the problem is already found

I did some more testing (incl. building latest mesa-git as I haven't updated for a while) and got 100% same issue as #1623 (hang instead of assert), so yes, RDR2 breaks DXBC compiler.

Using dxilconv instead of built-in compiler fixes everything (even #1506):

Image

I'll still leave the issue open as the DXBC bug isn't fixed (hopefully yet).

Wwhitebelyash 2025-08-13 github

New dxbc_spirv compiler resolves this issue - thus tessellation pipelines compile and work now as well. The snow renderer uses TS stage here

Image

Sounds like legacy compiler has issues with handling DXBC tessellation modules, lol.

Ooscarbg 2025-08-14 github

@whitebelyash nice.. how to test the new spirv compiler on vkd3d? Vía env variable or some config file..

Wwhitebelyash 2025-08-14 github

@whitebelyash nice.. how to test the new spirv compiler on vkd3d? Vía env variable or some config file..

VKD3D_CONFIG=dxbc_spirv

VVaniaOP 2025-08-21 github

Hey, I am having this same issue, how do I fix it? Where do I get dxbc_spirv from?

KK0bin 2025-08-21 github

Just use a recent version of vkd3d-Proton. It has already been integrated and enabled for RDR2.

Proton versions

Launch options

Upstream links