protonscr

Oblivion Reloaded can cause the first person node to misrender in Tes4 : Oblivion

dxvkclosed game bugd3d9
doitsujin/dxvk#2420 · opened 2021-12-28 by llde · updated 2022-01-26 · 23 comments · github
Lllde 2021-12-28 github

In certain light conditions (and certain in game spots) the game TES4:Oblivion render double hands when using Oblivion Reloaded and with Camera Mode Off.
To render it's effect OblivionReloaded grab the depth buffer of the game. However it's not immediate when using the vanilla 1st person camera.
The game render the first person node after clearing the depth buffer. This allow the first player node to stand on the top of every other objects, even while colliding.
OR intercept this, it prevent the vanilla clearing of the depth buffer, It renders the first person node once, it grabs the depth buffer, clear it and re-render the first person node.
This works perfectly on Windows D3d9, but on DXVK it cause a double hand phenomenon
Oblivion20211228 20 47 00
We can see the Ambient Occlusion bleed above the ghost hands, meaning they aren't in the depth buffer.

The same bug also happen on Wine D3D. However a significative difference is that on WineD3D the ghost double hands are black.
This bug happen both with DXVK on Linux under Wine, and with DXVK on Windows. All tests (except replaying with official d3d9) are using the configuration below (ArchLinux distro)

Software information

The Elder Scroll Oblivion with Oblivion Reloaded. Settings of the game setted as Ultra presets.
Oblivion Reloaded with Camera Mode Off (possible until OR 9.0.1), other OR settings doesn't match.

System information

  • GPU: RX580 8G
  • Driver: Mesa 21.3.2 (using RADV)
  • Wine version: 7.0rc2
  • DXVK version: 1.9.2

Apitrace file(s)

https://mega.nz/file/9DhEiTTD#0hjKjqVrJDjHfs4DXLFpnD1ZC4Hr7BuYa1S5f_tn9GU
Replaying the trace with DXVK, do show the issue, replaying with WineD3D shows the black hands, and everytingh is shown properly replaying on official d3d9.

Log files

KK0bin maintainer 2022-01-25 github

This seems like a bug with either the mod or the game rather than DXVK.

Here's what happens:

  • the game renders the regular hands.
  • copies the depth buffer to another texture
  • clears the depth buffer
  • renders the new hands on top of the old ones

Then it uses the copy of the depth buffer (which only has the "old hands") to generate the SSAO and that's why AO is rendered on top of the hands.

There are D3D9 commands for all of that, so it's not something that only DXVK does.

Lllde 2022-01-25 github

@K0bin
This order of commands do properly works on native d3d9. Also replicating the apitrace on windows also works without this issue.

This order was necessary for Oblivion Reloaded to grab the depth buffer with the first person model. The game would clear the depth after before rendering the first person node at the end of the scene. Just preventing the depth buffer clear, would cause the first person to disappear.

Note the issue here is that while it's true that the model is rendered twice, it's rendered differently the second time. Native d3d9 render result is the same with both times

CCME42 2022-01-25 github

When i replay the apitrace on native d3d9 on nvidia, i can also repro the ghost hand fwiw.

Lllde 2022-01-25 github

I had test the apitrace on multiple windows configuration (AMD, Intel and Nvidia), no one experienced the ghost hands

KK0bin maintainer 2022-01-25 github

I can't reproduce the problem on Windows because SSAO is broken on Windows while that works with DXVK.

  • the game renders the regular hands.
  • copies the depth buffer to another texture This doesn't happen on Windows.
  • clears the depth buffer
  • renders the new hands on top of the old ones

The game uses the RESZ hack to do the depth copy. Nvidia doesn't support that so the depth buffer that's used for AO later is completely blank.

So it's just broken in a different way.

I'd still consider this a bug in the mod. Especially considering the fact that it ends up rendering 4 hands. The original set of hands is drawn before clearing the depth buffer.

image
This is on Windows without DXVK.

Bbno1 2022-01-25 github

The game uses the RESZ hack to do the depth copy. Nvidia doesn't support that so the depth buffer that's used for AO later is completely blank.

Maybe the game tries to identify the GPU vendor and nvapiHack makes it see an AMD GPU and use an unsupported code path?

KK0bin maintainer 2022-01-25 github

We always expose RESZ regardless of the GPU vendor. The mod falls back to a Nvidia specific hack if that's not supported.

It does this copy of the depth buffer before rendering the second pair of hands and uses that copy to generate SSAO, so the AO doesn't take the new hands into account.

Lllde 2022-01-25 github

OR on NVIDIA use an NVAPI code path to grab the dpeth buffer, while RESZ on Intel/AMD.
Apitrace is taken on AMD so it's using the RESZ codepath to grab (and replay the trace on Nvidia probably wouldn't work)
Proper Apitrace test would probably need to be done on AMD or Intel under Windows
Maybe try to apply RESZ method on Nvidia is invalidating some internal render states. The double hand is not directly related to AO, in fact it happens even when alll depth buffer effects aren't used

KK0bin maintainer 2022-01-25 github

I still don't see how this could be a DXVK bug. The game clearly takes a snapshot of the depth buffer before rendering the hands again and uses that for SSAO. How is this supposed to work? The game pretty much goes out of its way to achieve this look.

Lllde 2022-01-25 github

@K0bin I did test the apitrace on both an Intel and an AMD configuration, on windows there weren't any double hands.
Yes the first person node is rendered twice. I checked a bit but it seems that it's basically unavoidable, otherwise the first persone node would disappear when crossing beams of light.
The vanilla oblivion rendering is Render Scene -> Clear depth, -> render first person node.
OR change it to Render Scene -> Render first person node-> Grab depth ->clear depth, re-render first person node.
Is this an ugly workaround? Certainly, but nor me nor Alenet found ways to solve this without changing the functional result.

On native d3d9 this works. No one had this issue on windows when playing (both on AMD and Nvidia). Apparently replaying the apitrace on Nvidia on windows can shows the issue, but on AMD doesn't. (If someone else here as an Intel or AMD configuration on Windows can test, it can be useful)
Why ? Probably not even gods know.

It does this copy of the depth buffer before rendering the second pair of hands and uses that copy to generate SSAO, so the AO doesn't take the new hands into account.

The issue is there. The second pair of hands isn't supposed to be in that way. The hands geometry isn't correct (it's supposed to be a fist), the graphics aren't correct (there is only the raw texture appyied , but not the SKIN shader (this by rendering bot times using the game provided methods), no lights changing the hands luminance.)

KK0bin maintainer 2022-01-25 github

Render Scene -> Render first person node-> Grab depth ->clear depth, re-render first person node.

Problem is that it renders different hand poses. So the hand on the depth buffer has a different pose than the one on tje the actual image.

Apparently replaying the apitrace on Nvidia on windows can shows the issue.

No it doesn't. I only have a Nvidia GPU, you can see my screen above. RESZ isn't supported so the depth map stays black and we get no AO at all.

That's the only difference to what it looks like with DXVK. The missing shading & different pose happens when replaying it on windows.

Can you make another Apitrace on Windows without DXVK. (Preferably not on an Nvidia GPU)

Lllde 2022-01-25 github

Problem is that it renders different hand poses. So the hand on the depth buffer has a different pose than the one on tje the actual image.

It shouldn't. The second hands are rendered exactly in the same way the first pair is. It use an internal met5hod of Oblivion that was named RenderObject. This methods cycle sub nodes, prepare the render applying shader, geoemtry, vertex and index buffer, matrices, and all the sort of stuffs neeeded to render.
Excpet that the second time (only in certain light conditions however) the render result is botched resulting in the hands in a different pose.
This doesn't happen on native d3d9.

KK0bin maintainer 2022-01-25 github

I'd need a Windows D3D9 trace (without DXVK, preferably on AMD) then. The issue is baked into the apitrace.

Lllde 2022-01-25 github

I'd need a Windows D3D9 trace (without DXVK, preferably on AMD) then. The issue is baked into the apitrace.

Replying thew apitrace on Intel or AMD on Windows again deosn't give any issue.

@GBRPlus provided confirmation for AMD. I did test on a couple of Intel GPU configurations

@K0bin You are on nvidia on Windows did I understand correctly?

KK0bin maintainer 2022-01-25 github

https://github.com/llde/TES-Reloaded-Source-NEW/blob/master/TESReloaded/Core/RenderManager.cpp#L248

This is the problem. This causes DXVK to switch out the vertex declaration and ignore everything in the vertex buffer except for the position. The game never restores the original vertex declaration after that. I don't understand why this isn't a problem on the AMD d3d9 driver.

You are on nvidia on Windows did I understand correctly?

Yes.

Lllde 2022-01-25 github

The Internal RenderObject do reapply SetFVF and SetVertexDeclaration in the NiRenderer. But maybe if the internal render state isn't changed it isn't reapplied properly in certain conditions.
Will try to swap that code to use NiRenderState and NiRenderer instead of operating directly on the device when I have time.

But maybe if this works in AMD/Intel maybe they have a quirk on the driver specially handling RESZ? Or DXVK just invalidate something it shouldn't?

KK0bin maintainer 2022-01-25 github

But maybe if this works in AMD/Intel maybe they have a quirk on the driver specially handling RESZ? Or DXVK just invalidate something it shouldn't?

I guess they store the old vertex declaration somewhere and reapply that at some point? No idea how and after which calls though and guessing wrong would likely break a lot of games.

Nvidia doesn't do that but it wouldn't be the first time that D3D9 behavior was different between vendors.

KK0bin maintainer 2022-01-26 github

Tested it on an Intel GPU on Windows. It renders fine but I don't understand why. The vertex declaration only had POSITION.

4525725 @0 IDirect3DDevice9::DrawIndexedPrimitive(this = 0x8419390, PrimitiveType = D3DPT_TRIANGLELIST, BaseVertexIndex = 0, MinVertexIndex = 0, NumVertices = 690, startIndex = 0, primCount = 1038) = D3D_OK
4525725: warning: decl: 0/2 Stream: 0 Offset: 0 Type: 2 Method: 0 Usage: 0 UsageIndex: 0
4525725: warning: decl: 1/2 Stream: 255 Offset: 0 Type: 17 Method: 0 Usage: 0 UsageIndex: 0

Lllde 2022-01-26 github

@K0bin Hacking DXVK saving the declaration in SetFVF and restoring it in SetRenderState when resolving RESZ , seems to solve the issue.
But it's honestly quite horrible.
I wonder if this is something that is actualy workarounded in the AMD/Intel driver (and in that case if it's cause or consequences of RESZ messes) or it's working just by sheer chance.

KK0bin maintainer 2022-01-26 github

I'd be worried that this breaks other games. I queried the vertex declaration when doing the draw call for the second pair of hands and as you can see, it just uses XYZ FVF.

I also looked at it with the Intel GPU debugger and (I think) the second pair of hands just wasn't rendered at all, so that's my new theory. No idea why those draw calls would get dropped though.

EDIT: Yes, seems like Intel completely ignores the draw calls for the second hand.

Lllde 2022-01-26 github

It's strange. In previous test, there was a difference between rendering it twice (one beofre and one after clearing the depth buffer) instead of one (only before clearing it). However it's also true I did test only on DXVK that time.

Lllde 2022-01-26 github

If I change OR to properly save and restore the FVF and the Vertex Declaration the issue disappear.

Want to close this, or prefers to keep this open in case some other application rely on this behaviour?
@K0bin

KK0bin maintainer 2022-01-26 github

I'll just close this. We're matching the behavior of the Nvidia D3D9 and I have no idea why it doesn't render at all on Intel. We can still take another look if we ever find another application that relies on this exact behavior.

Nothing extracted yet.