[D3D9] Visible triangle seams with Stochastic Shader in modded GTA San Andreas

dxvkclosed d3d9

doitsujin/dxvk#5572 · opened 2026-04-03 by Reyks21 · updated 2026-06-09 · 17 comments · github

RReyks21 2026-04-03 github

Software Information

Game: Grand Theft Auto San Andreas (Steam AppID: 12120)
Relevant Mods: SkyGFX Extended, SilentPatch, DebugMenu

System Information

GPU: AMD Radeon RX 9060 XT (RDNA4 / GFX12)
Driver: RADV (Mesa 26.0.3)
OS: CachyOS (Arch-based Linux)
DXVK version: v2.7.1-509-g1676dcaf342a9b1
Proton version: GE-Proton 10-34 (tested multiple versions)
Wine version: 10.0 (Staging)

Description

SkyGFX Extended mod implements a Procedural Stochastic Texturing shader for terrain/world
textures. This makes repeated textures look much nicer.

However when using DXVK it produces visible triangle-shaped seams with
harsh edges across surfaces using the stochastic shader.

The bug does not occur with WineD3D (PROTON_USE_WINED3D=1).

This could be GPU driver related since the RX 9060 XT and RDNA4 is very recent, but I don't have any other card to test.

Possible Root Cause Analysis (According to Claude, likely wrong... don't ask me lol)

SkyGFX Extended Source Code

The stochastic texturing technique works by computing a triangle grid in UV space and
blending 3 texture samples with barycentric weights. It calls tex2D with explicit
gradients (ddx/ddy) computed from the original UV (before the per-vertex hash
offset is applied), to avoid mip discontinuities at the stochastic grid boundaries.

The shader source (StochasticSamplerPS.hlsl):


//hash for randomness
float2 hash2D2D(float2 s)
{
	//magic numbers
	return frac(sin(fmod(float2(dot(s, float2(127.1, 311.7)), dot(s, float2(269.5, 183.3))), 3.14159)) * 43758.5453);
}

//stochastic sampling
float4 tex2DStochastic(sampler2D tex, float2 UV)
{
	//triangle vertices and blend weights
	//BW_vx[0...2].xyz = triangle verts
	//BW_vx[3].xy = blend weights (z is unused)
	float4x3 BW_vx;

	//uv transformed into triangular grid space with UV scaled by approximation of 2*sqrt(3)
	float2 skewUV = mul(float2x2 (1.0, 0.0, -0.57735027, 1.15470054), UV * 3.464);

	//vertex IDs and barycentric coords
	float2 vxID = float2 (floor(skewUV));
	float3 barry = float3 (frac(skewUV), 0);
	barry.z = 1.0 - barry.x - barry.y;

	BW_vx = ((barry.z > 0) ?
			 float4x3(float3(vxID, 0), float3(vxID + float2(0, 1), 0), float3(vxID + float2(1, 0), 0), barry.zyx) :
			 float4x3(float3(vxID + float2 (1, 1), 0), float3(vxID + float2 (1, 0), 0), float3(vxID + float2 (0, 1), 0), float3(-barry.z, 1.0 - barry.y, 1.0 - barry.x)));

	//calculate derivatives to avoid triangular grid artifacts
	float2 dx = ddx(UV);
	float2 dy = ddy(UV);

	//blend samples with calculated weights
	return mul(tex2D(tex, UV + hash2D2D(BW_vx[0].xy), dx, dy), BW_vx[3].x) +
		mul(tex2D(tex, UV + hash2D2D(BW_vx[1].xy), dx, dy), BW_vx[3].y) +
		mul(tex2D(tex, UV + hash2D2D(BW_vx[2].xy), dx, dy), BW_vx[3].z);
}

This compiles to texldd in ps_2_1 bytecode. DXVK translates this to
OpImageSampleExplicitLod with explicit gradients in SPIR-V.

The seams appear at GPU mesh triangle boundaries, not at the stochastic grid
boundaries — suggesting that the explicit gradient values passed via DXVK become
incorrect at primitive edges on RDNA4's Wave32 execution model, likely due to helper
invocation derivative behavior differing from what the D3D9 runtime produced.

Tests I did

All of the following RADV_DEBUG flags were tested and did not resolve the issue:
nodcc, invariantgeom, nongg, nofmask

The following dxvk.conf options also did not help:
d3d9.floatEmulation = Strict, dxvk.enableGraphicsPipelineLibrary = False

What "fixed it" was forcing WINED3D, but the performance degrades a lot:
PROTON_USE_WINED3D=1

Reproduction

Install GTA San Andreas (Steam)
Downgrade to 1.0
Install SkyGFX Extended: https://www.mixmods.com.br/2024/03/sa-skygfx/
Enable stochastic texturing in skygfx.ini (stochasticTexturing=1)
Load any outdoor area
Observe triangle-shaped seams on certain terrain/ground textures

Screenshots

DXVK (Visible Triangle Seams):

PROTON_USE_WINED3D=1 (No Seams):

Apitrace

gta-sa.trace.tar.gz

Proton Log

steam-12120.log

Ddoitsujin maintainer 2026-04-04 github

If you're calling this code in non-uniform control flow then yes, it's going to break because it's undefined behaviour in basically any API. dxbc-spirv should work around that in the future but D3D9 support there is currently WIP and we won't really be doing much with the old compiler at this point.

DXVK has no real control over helper lane and ddx/ddy behaviour, we just get whatever the driver does based on the Vulkan spec.

If there's MSAA involved (please specify exact settings, it's far more useful info than what the AI vomited out) then maybe interpolation modes are different for some reason, not like D3D9 documents anything in that area.

Would also be nice to know how exactly shader arguments are passed around, if this somehow relies on specific per-triangle vertex orders for the barycentrics to work then we're probably just screwed.

If the fract(sin(...)) shenanigans is sensitive to small ULP errors then we're basically screwed because we can't magically match native codegen.

Either way, this seems like it could do with some debugging from the mod side to at least understand where things are going wrong since we'd have to spend a lot of time understanding every single detail about the algorithm based on the low-level code.

KK0bin maintainer 2026-04-04 github

Looks like the new compiler fixes this. So you just have to be a little patient.

Old compiler:

New compiler:

RReyks21 2026-04-05 github

If there's MSAA involved (please specify exact settings, it's far more useful info than what the AI vomited out) then maybe interpolation modes are different for some reason, not like D3D9 documents anything in that area.

I'm using 8x MSAA and all other settings maxed out, but it really doesn't make a difference even with MSAA off, I also tested on a clean game with only the three mods I mentioned, issue persists.

Would also be nice to know how exactly shader arguments are passed around, if this somehow relies on specific per-triangle vertex orders for the barycentrics to work then we're probably just screwed.

If the fract(sin(...)) shenanigans is sensitive to small ULP errors then we're basically screwed because we can't magically match native codegen.

Either way, this seems like it could do with some debugging from the mod side to at least understand where things are going wrong since we'd have to spend a lot of time understanding every single detail about the algorithm based on the low-level code.

Well to be honest we probably need @JuniorDjjr for that since he's the author, I'm just a hyperfocused power user, no idea what this shader code magic even does lol.

Looks like the new compiler fixes this. So you just have to be a little patient.

Interesting... bad timing I guess? Will test it out once it's released!

JJuniorDjjr 2026-04-05 github

I suggest to test (it on early access right now, public later) the Proper Shaders, that also have the stochastic procedural texturing shader, but working in a totally different way (DX Effects framework and so). I have no idea why it caused this bug, but looks like a blending issue, the shader really works this way (by distorting the UVs in triangles), but for some reason the multiple samples are not being blended, I think.

KK0bin maintainer 2026-04-05 github

I investigated why this is fixed with the new shader compiler:

It has nothing to do with derivatives and the problematic shader doesn't have any control flow anyway.

Turns out modifying dp2_f32_legacy makes the issue re-appear:

float dp2_f32_legacy(vec2 a, vec2 b)
{
    //precise float _102 = ((b.x == 0.0) ? 0.0 : a.x) * ((a.x == 0.0) ? 0.0 : b.x);
    //return fma((b.y == 0.0) ? 0.0 : a.y, (a.y == 0.0) ? 0.0 : b.y, _102);
return a.x*b.x+a.y*b.y;
}

Having precise temporary variables for the two multiplication results or using a fused fma both led to correct rendering.

@Reyks21 Thank you for the good issue and especially for providing an apitrace!

RReyks21 2026-04-05 github

I suggest to test (it on early access right now, public later) the Proper Shaders, that also have the stochastic procedural texturing shader, but working in a totally different way (DX Effects framework and so). I have no idea why it caused this bug, but looks like a blending issue, the shader really works this way (by distorting the UVs in triangles), but for some reason the multiple samples are not being blended, I think.

Holy shit did not know about that! Looks crazy, can't wait to try it!

@Reyks21 Thank you for the good issue and especially for providing an apitrace!

No problem, glad I could be of some help! I'm actually testing a bunch of old games on Linux using proton, so if I spot anything I will be sure to open an issue with an apitrace.

Btw not sure if I should close this or wait for the new version and report back?

KK0bin maintainer 2026-04-05 github

We'll close the issue when we finish hooking up the new shader compiler.

RReyks21 2026-06-04 github

Hey so I tested a recent in-dev DXVK build included in proton-cachyos-11.0-20260521, and the issue was partially fixed, there's no longer visible triangle seams.

However a new issue is now present:

The shader seems to be working correctly but there's now this weird noisy artifacting, looks very similar to when high-res textures have no mipmapping. So maybe the translation layer is failing to convert the mips to vulkan?

@K0bin the issue is also visible in your april 4th screenshot of the new compiler, kinda hard to see because of the specific texture:

BBlisto91 2026-06-04 github

If you could test latest dxvk master that would be great. There have been a bunch of fixes since the release of that proton-cachyos version.
If you are using Steam then Proton Experimental Bleeding Edge would be easiest.

RReyks21 2026-06-04 github

Tested with Proton Experimental Bleeding Edge, same results.

https://github.com/user-attachments/assets/023e21b0-94f1-4a14-86a6-831847dcde65

Made a new API Trace, might be useful:

gta-sa.2.trace.tar.gz

(edit) Proton Log:

steam-12120.log

Ddoitsujin maintainer 2026-06-05 github

Not seeing any obvious issue on my end with either of the two traces (6900XT w. Mesa 26.1, nor on the RTX 4070 on 610 drivers).

In general there's only so much we can do with FP precision, trying to rely on exact math in D3D9 is just a losing proposition given that the API doesn't follow any sort of IEEE standard and has no precise equivalent at all. Even if we manage to work around it somehow, any future driver update can and probably will break it again.

RReyks21 2026-06-05 github

Not seeing any obvious issue on my end with either of the two traces (6900XT w. Mesa 26.1, nor on the RTX 4070 on 610 drivers).

Just to clarify, are you saying the visual artifact isn't visible at all on your end when replaying the trace? Or that you looked at the shader code in the trace and don't see anything obviously wrong with how DXVK handles it?

@JuniorDjjr Do you think this may be a issue on the shader side? I tried out your new Proper Shaders mod and the same bug shows up there as well.

Also I had Claude analyze the apitrace disassembly and it flagged that dsx/dsy are being called on the hash-offset UV instead of the original input UV. I can't verify this myself, but posting it here in case it's useful for someone who knows the code.

Ddoitsujin maintainer 2026-06-05 github

Just to clarify, are you saying the visual artifact isn't visible at all on your end when replaying the trace?

Yes, the moiree pattern in your video doesn't show during replay on either of my two GPUs. Looks like this here:

As for the derivatives, they look fine in the actual code, here's the relevant snippet:

    float _285 = t0_texcoord0.x * 1.2000000476837158203125;
    float _286 = t0_texcoord0.y * 1.2000000476837158203125;
...
    /* ignore .zw components, they are, not used */
    vec4 _392 = vec4(dFdx(_285), dFdx(_286), 0.0, 0.0);
    vec4 _393 = vec4(dFdy(_285), dFdy(_286), _338, _339);
...
    vec4 _394 = sampleTexture_0_tex_grad(vec4(fma(t0_texcoord0.x, 1.2000000476837158203125, fract(mad_legacy_f32(_330, _338, 1.0) * 43758.546875)), fma(t0_texcoord0.y, 1.2000000476837158203125, fract(mad_legacy_f32(_331, _339, 1.0) * 43758.546875)), _307, _308), _392, _393);
    vec4 _400 = sampleTexture_0_tex_grad(vec4(fma(t0_texcoord0.x, 1.2000000476837158203125, fract(mad_legacy_f32(_373, _381, 1.0) * 43758.546875)), fma(t0_texcoord0.y, 1.2000000476837158203125, fract(mad_legacy_f32(_374, _382, 1.0) * 43758.546875)), _381, _382), _392, _393);

One potential problem with FP precision is that we don't really implement dp2add 1:1 because our IR currently has no such instruction, it's a dot product + separate addition instead and drivers cannot optimize. Should probably fix that, but w/o a repro on this issue I don't know if that's even relevant here.

Edit: Turns out all the dp2add uses in this shader have a zero accumulator anyway.

Full spirv-cross output from the DXVK shader here for reference.

RReyks21 2026-06-05 github

Yes, the moiree pattern in your video doesn't show during replay on either of my two GPUs. Looks like this here:

Well damn that's weird, it could be the RX 9060XT or something RDNA4 related on the driver level... what confuses me is that it showed up in K0bin's screenshot.

I made another apitrace but this time in a different location and using the Proper Shaders mod (wich implements the shader differently I think).

If you still can't see it then I can assume this is a Mesa driver bug?

Ddoitsujin maintainer 2026-06-05 github

Nope, everything looking normal. Does it actually happen for you when you replay the actual trace file, or just in game?

Certainly possible that this is architecture-specific somehow, but again I'm not sure I'd strictly call this a bug given that there's no way to enforce any sort of precision in D3D9.

One thing you could try is throiwing DXVK_CONFIG="dxvk.lowerSinCos=True" at it and see if that does anything, but terrible sin/cos precision is usually more of an Intel trademark and I don't see why we'd get anything other than Windows behaviour on AMD for these instructions.

RReyks21 2026-06-05 github

Ok so this is interesting, when replaying the .trace file using Wine the issue does not show up at all.

However, as soon as I drop DXVK's 32-bit d3d9.dll into the apitrace.exe folder to force it to use Vulkan, it immediately returns.

This probably means nothing since running the game with PROTON_USE_WINED3D=1 also gets rid of the bug (but performance is way worse).

And also tried DXVK_CONFIG="dxvk.lowerSinCos=True and it does nothing unfortunately.

RReyks21 2026-06-09 github

UPDATE

Tested the most recent dev builds and the issue is now fixed completely, dunno what you guys did but it's all good now :D