protonscr

Pixel shader input layout mismatch

vkd3dclosed
HansKristian-Work/vkd3d-proton#1204 · opened 2022-08-30 by sherief · updated 2022-08-31 · 6 comments · github
Ssherief 2022-08-30 github

I'm doing bring-up of a proprietary engine on Steam Deck using compatibility layer Proton 7.0-4 and OS version 3.3.1, Build 20220817.1, and the simple PSO for Dear imgui is suffering from a layout mismatch corrupting the UI rendering. Here's the HLSL versions:

Vertex shader:

struct PSInput { float2 TexCoord : TEXCOORD; float4 Color : COLOR; };
PSInput main(float2 Position : POSITION, float2 TexCoord : TEXCOORD, float4 Color : COLOR, out float4 OutPosition : SV_POSITION)
{
    OutPosition = mul(ProjectionMatrix, float4(Position, 0, 1));
    PSInput Result;
    Result.TexCoord = TexCoord;
    Result.Color = Color;
    return Result;
}

And pixel shader:

SamplerState TextureSampler : register(s0);
Texture2D Texture : register(t0);
float4 main(PSInput Input, float4 ScreenSpace : SV_POSITION) : SV_TARGET
{
    float4 SampledColor = Texture.Sample(TextureSampler, Input.TexCoord).aaaa;
    return (Texture.Sample(TextureSampler, Input.TexCoord) * Input.Color);
}

The first line in main() is vestigial, and always gets optimized out on Windows. The shaders are compiled using the legacy compiler via D3DCompile() and profiles vs_5_0 / ps_5_0

Looking at GLSL versions in a Renderdoc capture, the vertex shader declares its outputs as such:

layout(location = 2) out vec4 o2;

While the pixel shader declares its inputs as

layout(location = 1) in vec4 v1;

So vertex shader's o1 (tex coord) ends up going to the pixel shader's v1 (color), causing incorrect texcoords to be used for sampling the imgui texture and corrupted UI rendering.

The full shaders are attached as HLSL, SPIR-V, and GLSL; with the latter two obtained from a Renderdoc capture running remotely on the Steam Deck.

ps-glsl.txt
ps-hlsl.txt
ps-spirv.txt
vs-glsl.txt
vs-hlsl.txt
vs-spirv.txt

KK0bin 2022-08-30 github

Do you compile those shaders at runtime or do you ship the precompiled DXBC?

If you compile them at runtime, do you ship Microsoft's D3DCompiler DLL or rely on the Wine implementation?

Ssherief 2022-08-30 github

I compile them at runtime and rely on the Wine implementation.

Ssherief 2022-08-30 github

BTW, it's on my near-term roadmap to convert it to ship ax DXBC - if it makes things easier for you we could wait till I submit that change then see if it fixes things. What do you think?

KK0bin 2022-08-30 github

I compiled your HLSL shaders with both Microsofts HLSL compiler and the Wine one,

Microsoft:

// Output signature:
//
// Name                 Index   Mask Register SysValue  Format   Used
// -------------------- ----- ------ -------- -------- ------- ------
// TEXCOORD                 0   xy          0     NONE   float   xy  
// COLOR                    0   xyzw        1     NONE   float   xyzw
// SV_POSITION              0   xyzw        2      POS   float   xyzw

Wine (vkd3d-shader 1.4 (git 9d4df5e70468)):

// Output signature:
//
// Name                 Index   Mask Register SysValue  Format   Used
// -------------------- ----- ------ -------- -------- ------- ------
// SV_POSITION              0   xyzw        0      POS   float   xyzw
// TEXCOORD                 0   xy          1     NONE   float   xy  
// COLOR                    0   xyzw        2     NONE   float   xyzw

So this seems to be an issue with the Wine HLSL compiler rather than vkd3d-proton which only compiles from DXBC (or DXIL) to SPIR-V. To make things extra confusing, Wine's HLSL compiler is part of the original upstream vkd3d project.

Ssherief 2022-08-31 github

Compiling on Windows using D3DCompile() then shipping that bytecode to Proton worked. Interesting bug..

HHansKristian-Work maintainer 2022-08-31 github

Yes, this is a HLSL compiler bug. Generally, mixing and matching output structs and out variables tends to cause confusion since it's ambiguous which order arguments should be packed in. Likely Wine packs out before return values, but FXC/DXC packs returns before out.

Either way, not a vkd3d-proton bug, so closing.

Proton versions