Do you compile those shaders at runtime or do you ship the precompiled DXBC?
If you compile them at runtime, do you ship Microsoft's D3DCompiler DLL or rely on the Wine implementation?
I compile them at runtime and rely on the Wine implementation.
BTW, it's on my near-term roadmap to convert it to ship ax DXBC - if it makes things easier for you we could wait till I submit that change then see if it fixes things. What do you think?
I compiled your HLSL shaders with both Microsofts HLSL compiler and the Wine one,
Microsoft:
// Output signature:
//
// Name Index Mask Register SysValue Format Used
// -------------------- ----- ------ -------- -------- ------- ------
// TEXCOORD 0 xy 0 NONE float xy
// COLOR 0 xyzw 1 NONE float xyzw
// SV_POSITION 0 xyzw 2 POS float xyzw
Wine (vkd3d-shader 1.4 (git 9d4df5e70468)):
// Output signature:
//
// Name Index Mask Register SysValue Format Used
// -------------------- ----- ------ -------- -------- ------- ------
// SV_POSITION 0 xyzw 0 POS float xyzw
// TEXCOORD 0 xy 1 NONE float xy
// COLOR 0 xyzw 2 NONE float xyzw
So this seems to be an issue with the Wine HLSL compiler rather than vkd3d-proton which only compiles from DXBC (or DXIL) to SPIR-V. To make things extra confusing, Wine's HLSL compiler is part of the original upstream vkd3d project.
Compiling on Windows using D3DCompile() then shipping that bytecode to Proton worked. Interesting bug..
Yes, this is a HLSL compiler bug. Generally, mixing and matching output structs and out variables tends to cause confusion since it's ambiguous which order arguments should be packed in. Likely Wine packs out before return values, but FXC/DXC packs returns before out.
Either way, not a vkd3d-proton bug, so closing.
proton 7.0-4x1 2022-08
I'm doing bring-up of a proprietary engine on Steam Deck using compatibility layer Proton 7.0-4 and OS version 3.3.1, Build 20220817.1, and the simple PSO for Dear imgui is suffering from a layout mismatch corrupting the UI rendering. Here's the HLSL versions:
Vertex shader:
And pixel shader:
The first line in main() is vestigial, and always gets optimized out on Windows. The shaders are compiled using the legacy compiler via D3DCompile() and profiles vs_5_0 / ps_5_0
Looking at GLSL versions in a Renderdoc capture, the vertex shader declares its outputs as such:
While the pixel shader declares its inputs as
So vertex shader's o1 (tex coord) ends up going to the pixel shader's v1 (color), causing incorrect texcoords to be used for sampling the imgui texture and corrupted UI rendering.
The full shaders are attached as HLSL, SPIR-V, and GLSL; with the latter two obtained from a Renderdoc capture running remotely on the Steam Deck.
ps-glsl.txt
ps-hlsl.txt
ps-spirv.txt
vs-glsl.txt
vs-hlsl.txt
vs-spirv.txt