protonscr

Borderlands 4: TCP page fault with retain_descriptor_heaps (followup to #3034)

vkd3dclosed FSR use after free jank
HansKristian-Work/vkd3d-proton#3047 · opened 2026-05-19 by NTMan · updated 2026-06-10 · 15 comments · github
NNTMan 2026-05-19 github

With VKD3D_CONFIG=retain_descriptor_heaps set I played ~30 hours without any of the SQC hangs from [#3034](https://github.com/HansKristian-Work/vkd3d-proton/issues/3034).
Today got a new GPU hang. Looks different from the previous ones - not sure if this is the same class of bug or a new one.

Comparison of dmesg page fault info:

Hang 1-3 (without retain_descriptor_heaps):
    Faulty UTCL2 client ID: SQC (data) (0xa)
    GCVM_L2_PROTECTION_FAULT_STATUS: 0x00601431 / 0x00401431

Hang 4 (with retain_descriptor_heaps):
    Faulty UTCL2 client ID: TCP (0x8)
    GCVM_L2_PROTECTION_FAULT_STATUS: 0x00701031

Dmesg block:

[ 6564.069816] amdgpu 0000:03:00.0: [gfxhub] page fault (src_id:0 ring:24 vmid:7 pasid:368)
[ 6564.069842] amdgpu 0000:03:00.0:   in page starting at address 0x0000800267780000 from client 10
[ 6564.069854] amdgpu 0000:03:00.0: GCVM_L2_PROTECTION_FAULT_STATUS:0x00701031
[ 6564.069858] amdgpu 0000:03:00.0: 	 Faulty UTCL2 client ID: TCP (0x8)
[ 6564.069863] amdgpu 0000:03:00.0: 	 MORE_FAULTS: 0x1
[ 6564.069868] amdgpu 0000:03:00.0: 	 PERMISSION_FAULTS: 0x3

Proton log around the time of hang has the usual FSR bursts (156 FSR resource detected warnings over the session), but unlike in [#3034](https://github.com/HansKristian-Work/vkd3d-proton/issues/3034) there is no radv: GPUVM fault detected, no device_lost, no vr -4 - the log just ends with SEH unwinding traces.

Versions:

2117.726:0180:0184:info:vkd3d-proton:vkd3d_config_flags_init_once: VKD3D_CONFIG='retain_descriptor_heaps'.
  • vkd3d-proton 3.0.1 (build 64f5776fdb330ab)
  • Proton Experimental 11.0-20260506b
  • Mesa master @ 46cd7b6e28e457447f2179f8dbf5fcfd73363874
  • kernel 7.1.0-rc2 (amdgpu.gpu_recovery=0)
  • AMD Radeon RX 7900 XTX (Navi31, gfx1100)

Questions:

  1. Is this the same bug as [#3034](https://github.com/HansKristian-Work/vkd3d-proton/issues/3034) (and [#3037](https://github.com/HansKristian-Work/vkd3d-proton/pull/3037) should fix it too once it reaches Proton), or a different one that retain_descriptor_heaps doesn't cover?
  2. Is it worth waiting for Proton Experimental to pick up [#3037](https://github.com/HansKristian-Work/vkd3d-proton/pull/3037) and retesting, or is the different client ID / fault status enough to tell that [#3037](https://github.com/HansKristian-Work/vkd3d-proton/pull/3037) won't help here?

Umr logs and devcoredump attached: umr-log.gz
Proton log: steam-1285190.zip

HHansKristian-Work maintainer 2026-05-20 github

0x0000800267780000

that looks like a new class of bug, more like use-after-free maybe, but impossible to tell without a reliable repro.

With VKD3D_CONFIG=retain_descriptor_heaps set I played ~30 hours

... If it takes 30 hours to repro I don't see how this is possible to workaround.

RADV_DEBUG=bo_history might help to catch what kind of fault it is if you can find a more reliable repro. The log would end up in /tmp/radv_bo_history.log.

NNTMan 2026-05-23 github

Got a reliable repro. It happens on the mission "Another Day, Another Universe", in the cutscene where Lilith comes out of the portal right before the boss fight. Hung there 3 times now in the same spot.

Previous GPU hang scene Today GPU hang scene
Image Image

Caught it with RADV_DEBUG=bo_history this time. The faulting page 0x80025ae21000 falls inside this BO:

timestamp=41528638341842, VA=000080025ae00000-000080025b184000, destroyed=0, is_virtual=0
timestamp=41561843764139, VA=000080025ae00000-000080025b184000, destroyed=1, is_virtual=0

BO destroyed at 41561.843764s, GPU page fault at 41562.197217s (dmesg uptime) — the BO was freed ~353 ms before the shader faulted on its address.

dmesg
[41562.197217] amdgpu 0000:03:00.0: [gfxhub] page fault (src_id:0 ring:24 vmid:5 pasid:684)
[41562.197260] amdgpu 0000:03:00.0:   in page starting at address 0x000080025ae21000 from client 10
[41562.197286] amdgpu 0000:03:00.0: GCVM_L2_PROTECTION_FAULT_STATUS:0x00501031
[41562.197288] amdgpu 0000:03:00.0:    Faulty UTCL2 client ID: TCP (0x8)
[41562.197294] amdgpu 0000:03:00.0:    MORE_FAULTS: 0x1
[41562.197305] amdgpu 0000:03:00.0:    PERMISSION_FAULTS: 0x3

VKD3D_CONFIG=retain_descriptor_heaps was set for this session.

Logs attached (radv_bo_history.log, dmesg, devcoredump, Proton log):

HHansKristian-Work maintainer 2026-05-27 github

Indeed, this looks like a straight use-after-free. Clearly a game bug.

NNTMan 2026-05-27 github

Thanks for confirming. Should I report this to Gearbox, or is there any chance of a game-specific workaround on the vkd3d side like #3037?

HHansKristian-Work maintainer 2026-05-27 github

There is no obvious workaround we can apply on our end right now other than just not freeing memory when application asks us to, but that's going to be broken too in different ways. @runar-work is doing some triage on it to see if we can get anything actionable out of it.

Rrunar-work 2026-05-27 github

So far I haven't been able to repro any hangs in that spot with Proton Experimental, vkd3d-proton 5da4f239b96926c, Mesa 72f02d6e, 7.0.10-arch1-1, on neither RX 9070 nor RX 7600. Which graphics settings are you using?

NNTMan 2026-05-27 github

Graphics settings:

  • Display: Fullscreen, 3840x2160, VSync off, frame rate limit off
  • Graphics Preset: Very High
  • Upscaling: FSR, Performance quality (internal render resolution 1920x1080)
  • Frame Generation: Off
  • Level of Detail: Far
  • Texture Pool: Very High
  • Textures Streaming Speed: High
  • Anisotropic Filtering: x16
  • Foliage Density / Vegetation Sway: High
  • Volumetric Fog: Very High, Volumetric Cloud: High, Volumetric Cloud Shadows: Enabled
  • Shadow Quality / Directional Shadow Quality: High
  • Lighting Quality / Reflections Quality: High
  • Shading Quality: Medium
  • Post-Process Quality: High, Depth Of Field: Enabled
  • Motion Blur: Amount 0.0, Quality Off
1 2 3 4
Image Image Image Image

Differences from your setup that might affect hitting the timing window:

  • GPU: RX 7900 XTX (Navi31, gfx1100), 24 GB VRAM. In this scene VRAM
    sits at ~17-19 / 24 GB, so free->realloc timing may differ from
    RX 9070 / RX 7600.
  • Mesa 46cd7b6e28 vs your 72f02d6e
  • vkd3d-proton 3.0.1 build 64f5776 vs 5da4f23
  • kernel 7.1.0-rc2 (custom) vs 7.0.10-arch1
  • amdgpu.gpu_recovery=0 on my side
  • Steam in-game performance overlay was enabled; screenshots taken
    with Steam's screenshot tool. Don't know if either is relevant —
    the bo_history log shows the freed BO is a game allocation, so
    probably not, but listing it for completeness.

Repro: mission "Another Day, Another Universe", the cutscene where Lilith comes out of the portal before the boss fight. 3/3 for me.

NNTMan 2026-05-27 github

Reproduced again, same spot. This time the bo_history ordering is slightly different — worth noting.

Faulting page 0x80026fb83000 is inside this BO:

timestamp=20173253223945, VA=000080026f800000-000080026fb84000, destroyed=0, is_virtual=0
timestamp=20206485992384, VA=000080026f800000-000080026fb84000, destroyed=1, is_virtual=0

BO created at 20173.253224s, GPU page fault at 20206.373142s (dmesg), BO logged destroyed at 20206.485992s — i.e. the fault happened ~113 ms before RADV logged the destroy, not after like in the previous capture (where destroy preceded the fault by ~353 ms).

dmesg:
[20206.373142] amdgpu 0000:03:00.0: [gfxhub] page fault (src_id:0 ring:24 vmid:4 pasid:1310)
[20206.373165] amdgpu 0000:03:00.0:   in page starting at address 0x000080026fb83000 from client 10
[20206.373167] amdgpu 0000:03:00.0: GCVM_L2_PROTECTION_FAULT_STATUS:0x00401031
[20206.373170] amdgpu 0000:03:00.0:    Faulty UTCL2 client ID: TCP (0x8)
[20206.373177] amdgpu 0000:03:00.0:    PERMISSION_FAULTS: 0x3

Both this BO and the one from the previous capture had a lifetime of exactly ~33.2 s before being destroyed.

Image
HHansKristian-Work maintainer 2026-05-27 github

VKD3D_CONFIG=fault would maybe include more context on which resource is being destroyed, but so far it's not looking very actionable ...

BO timings and stuff might not use exactly the same time base, so I wouldn't read too much into those.

I'll try some degenerate config options to do delayed free or something like that, but probably too risky to enable that by default.

NNTMan 2026-05-27 github

Reproduced with VKD3D_CONFIG=retain_descriptor_heaps,fault.
The address binding tracker output:

19754.013:0170:0208:err:vkd3d-proton:vkd3d_address_binding_tracker_report_entry: VA range [ffff800254000000, ffff800259590000) [size = 89718784 (0x5590000)]
19754.013:0170:0208:err:vkd3d-proton:vkd3d_address_binding_tracker_report_entry:   Fault offset: 58990592 (0x3842000)
19754.013:0170:0208:err:vkd3d-proton:vkd3d_address_binding_tracker_report_entry:   BIND
19754.013:0170:0208:err:vkd3d-proton:vkd3d_address_binding_tracker_report_entry:   T: 16583.046207 s
19754.013:0170:0208:err:vkd3d-proton:vkd3d_address_binding_tracker_report_entry:   Cookie: 2199.
19754.013:0170:0208:err:vkd3d-proton:vkd3d_address_binding_tracker_report_entry:   VkImage [9384 x 9384 x 1] [levels = 1] [layers = 1] [fmt = 137] [type = 1] [usage = #f]
19754.017:0170:0208:err:vkd3d-proton:vkd3d_address_binding_tracker_report_entry:   UNBIND ...
19754.018:0170:0208:err:vkd3d-proton:vkd3d_address_binding_tracker_report_entry:   BIND (new VkDeviceMemory at same VA range)

So the faulting address used to belong to a VkImage 9384x9384 (fmt 137, usage 0xf), which was unbound and then the VA range got reused by something else before the shader touched it.

dmesg:
[19749.012023] amdgpu 0000:03:00.0: [gfxhub] page fault (src_id:0 ring:24 vmid:5 pasid:1494)
[19749.012047] amdgpu 0000:03:00.0:   in page starting at address 0x0000800257b81000 from client 10
[19749.012059] amdgpu 0000:03:00.0: GCVM_L2_PROTECTION_FAULT_STATUS:0x00501031
[19749.012063] amdgpu 0000:03:00.0:    Faulty UTCL2 client ID: TCP (0x8)

Same spot — Lilith portal cutscene.

Logs:

HHansKristian-Work maintainer 2026-05-28 github

Interesting ...

19754.019:0170:0208:err:vkd3d-proton:vkd3d_address_binding_tracker_report_entry: VA range [ffff800257800000, ffff800257b84000) [size = 3686400 (0x384000)]
19754.019:0170:0208:err:vkd3d-proton:vkd3d_address_binding_tracker_report_entry:   Fault offset: 270336 (0x42000)
19754.019:0170:0208:err:vkd3d-proton:vkd3d_address_binding_tracker_report_entry:   BIND
19754.020:0170:0208:err:vkd3d-proton:vkd3d_address_binding_tracker_report_entry:   T: 19715.779589 s
19754.020:0170:0208:err:vkd3d-proton:vkd3d_address_binding_tracker_report_entry:   Cookie: 0.
19754.020:0170:0208:err:vkd3d-proton:vkd3d_address_binding_tracker_report_entry:   VkDeviceMemory [memoryTypeIndex = 0]
19754.020:0170:0208:err:vkd3d-proton:vkd3d_address_binding_tracker_report_entry: VA range [ffff800257800000, ffff800257b84000) [size = 3686400 (0x384000)]
19754.020:0170:0208:err:vkd3d-proton:vkd3d_address_binding_tracker_report_entry:   Fault offset: 270336 (0x42000)
19754.020:0170:0208:err:vkd3d-proton:vkd3d_address_binding_tracker_report_entry:   BIND
19754.020:0170:0208:err:vkd3d-proton:vkd3d_address_binding_tracker_report_entry:   T: 19715.779608 s
19754.020:0170:0208:err:vkd3d-proton:vkd3d_address_binding_tracker_report_entry:   Cookie: 9359270.
19754.020:0170:0208:err:vkd3d-proton:vkd3d_address_binding_tracker_report_entry:   VkImage [1920 x 804 x 1] [levels = 1] [layers = 1] [fmt = 76] [type = 1]
19754.020:0170:0208:err:vkd3d-proton:vkd3d_address_binding_tracker_report_entry: VA range [ffff800257800000, ffff800257b84000) [size = 3686400 (0x384000)]
19754.020:0170:0208:err:vkd3d-proton:vkd3d_address_binding_tracker_report_entry:   Fault offset: 270336 (0x42000)
19754.020:0170:0208:err:vkd3d-proton:vkd3d_address_binding_tracker_report_entry:   UNBIND
19754.020:0170:0208:err:vkd3d-proton:vkd3d_address_binding_tracker_report_entry:   T: 19749.006508 s
19754.021:0170:0208:err:vkd3d-proton:vkd3d_address_binding_tracker_report_entry:   VkImage
19754.021:0170:0208:err:vkd3d-proton:vkd3d_address_binding_tracker_report_entry: VA range [ffff800257800000, ffff800257b84000) [size = 3686400 (0x384000)]
19754.021:0170:0208:err:vkd3d-proton:vkd3d_address_binding_tracker_report_entry:   Fault offset: 270336 (0x42000)
19754.021:0170:0208:err:vkd3d-proton:vkd3d_address_binding_tracker_report_entry:   UNBIND
19754.021:0170:0208:err:vkd3d-proton:vkd3d_address_binding_tracker_report_entry:   T: 19749.006511 s
19754.021:0170:0208:err:vkd3d-proton:vkd3d_address_binding_tracker_report_entry:   VkDeviceMemory
19754.021:0170:0208:err:vkd3d-proton:vkd3d_address_binding_tracker_check_va: Fault address is currently unmapped. This is very likely use-after-free.
19754.021:0170:0208:err:vkd3d-proton:vkd3d_address_binding_tracker_check_va: Done reporting VA fault.

19754.021:0170:0208:err:vkd3d-proton:d3d12_resource_decref_retained: Resource refcount will hit 0 from a fence callback, which proves use-after-free by game.
19754.021:0170:0208:err:vkd3d-proton:d3d12_resource_decref_retained:   Identified use-after-free resource: 1920 x 804 x 1, levels 1, DXGI_FORMAT #2a, dim 3.
19754.021:0170:0208:err:vkd3d-proton:d3d12_resource_decref_retained:  Resource name: FSR3UPSCALER_ReconstructedPrevNearestDepth
19754.481:0170:0184:fixme:dwmapi:DwmGetWindowAttribute DWMWA_CLOAKED: always returning 0.
19754.482:0170:0184:fixme:eventlog:QueryAllTracesW (00000000A9CC0000, 64, 000000000414FC30) stub
19754.677:0170:0208:err:vkd3d-proton:d3d12_resource_decref_retained: Resource refcount will hit 0 from a fence callback, which proves use-after-free by game.
19754.677:0170:0208:err:vkd3d-proton:d3d12_resource_decref_retained:   Identified use-after-free resource: 1 x 1 x 1, levels 1, DXGI_FORMAT #2a, dim 3.
19754.678:0170:0208:err:vkd3d-proton:d3d12_resource_decref_retained:  Resource name: FSR3UPSCALER_SpdAtomicCounter
19754.678:0170:0208:err:vkd3d-proton:d3d12_resource_decref_retained: Resource refcount will hit 0 from a fence callback, which proves use-after-free by game.
19754.678:0170:0208:err:vkd3d-proton:d3d12_resource_decref_retained:   Identified use-after-free resource: 1920 x 804 x 1, levels 1, DXGI_FORMAT [#36](/issue/HansKristian-Work/vkd3d-proton/36), dim 3.
19754.678:0170:0208:err:vkd3d-proton:d3d12_resource_decref_retained:  Resource name: FSR3UPSCALER_IntermediateFp16x1
19755.249:0170:0184:fixme:dwmapi:DwmGetWindowAttribute DWMWA_CLOAKED: always returning 0.
19755.252:0170:0184:fixme:eventlog:QueryAllTracesW (00000000A9CC0000, 64, 000000000414FC30) stub
19755.444:0170:0208:err:vkd3d-proton:d3d12_resource_decref_retained: Resource refcount will hit 0 from a fence callback, which proves use-after-free by game.
19755.444:0170:0208:err:vkd3d-proton:d3d12_resource_decref_retained:   Identified use-after-free resource: 960 x 402 x 1, levels 1, DXGI_FORMAT #3d, dim 3.
19755.444:0170:0208:err:vkd3d-proton:d3d12_resource_decref_retained:  Resource name: FSR3UPSCALER_ShadingChange
19755.444:0170:0208:err:vkd3d-proton:d3d12_resource_decref_retained: Resource refcount will hit 0 from a fence callback, which proves use-after-free by game.
19755.444:0170:0208:err:vkd3d-proton:d3d12_resource_decref_retained:   Identified use-after-free resource: 3840 x 1607 x 1, levels 1, DXGI_FORMAT #3d, dim 3.
19755.444:0170:0208:err:vkd3d-proton:d3d12_resource_decref_retained:  Resource name: FSR3UPSCALER_NewLocks
19756.010:0170:0184:fixme:dwmapi:DwmGetWindowAttribute DWMWA_CLOAKED: always returning 0.
19756.012:0170:0184:fixme:eventlog:QueryAllTracesW (00000000A9CC0000, 64, 000000000414FC30) stub
19756.186:0170:0208:err:vkd3d-proton:d3d12_resource_decref_retained: Resource refcount will hit 0 from a fence callback, which proves use-after-free by game.
19756.186:0170:0208:err:vkd3d-proton:d3d12_resource_decref_retained:   Identified use-after-free resource: 960 x 402 x 1, levels 1, DXGI_FORMAT [#36](/issue/HansKristian-Work/vkd3d-proton/36), dim 3.
19756.186:0170:0208:err:vkd3d-proton:d3d12_resource_decref_retained:  Resource name: FSR3UPSCALER_FarthestDepthMip1
19756.186:0170:0208:err:vkd3d-proton:d3d12_resource_decref_retained: Resource refcount will hit 0 from a fence callback, which proves use-after-free by game.
19756.186:0170:0208:err:vkd3d-proton:d3d12_resource_decref_retained:   Identified use-after-free resource: 1920 x 804 x 1, levels 1, DXGI_FORMAT #1c, dim 3.
19756.186:0170:0208:err:vkd3d-proton:d3d12_resource_decref_retained:  Resource name: FSR3UPSCALER_DilatedReactiveMasks
19756.752:0170:0184:fixme:dwmapi:DwmGetWindowAttribute DWMWA_CLOAKED: always returning 0.
19756.753:0170:0184:fixme:eventlog:QueryAllTracesW (00000000A9CC0000, 64, 000000000414FC30) stub
19756.931:0170:0208:err:vkd3d-proton:d3d12_resource_decref_retained: Resource refcount will hit 0 from a fence callback, which proves use-after-free by game.
19756.931:0170:0208:err:vkd3d-proton:d3d12_resource_decref_retained:   Identified use-after-free resource: 960 x 402 x 1, levels 10, DXGI_FORMAT [#22](/issue/HansKristian-Work/vkd3d-proton/22), dim 3.
19756.931:0170:0208:err:vkd3d-proton:d3d12_resource_decref_retained:  Resource name: FSR3UPSCALER_SpdMips
19757.505:0170:0184:fixme:dwmapi:DwmGetWindowAttribute DWMWA_CLOAKED: always returning 0.
19757.507:0170:0184:fixme:eventlog:QueryAllTracesW (00000000A9CC0000, 64, 000000000414FC30) stub
19757.689:0170:0208:err:vkd3d-proton:d3d12_resource_decref_retained: Resource refcount will hit 0 from a fence callback, which proves use-after-free by game.
19757.689:0170:0208:err:vkd3d-proton:d3d12_resource_decref_retained:   Identified use-after-free resource: 1920 x 804 x 1, levels 1, DXGI_FORMAT [#36](/issue/HansKristian-Work/vkd3d-proton/36), dim 3.
19757.689:0170:0208:err:vkd3d-proton:d3d12_resource_decref_retained:  Resource name: FSR3UPSCALER_Luma1

This seems to be related to FSR3 usage being completely and utterly broken still. Wonder if some resource could actually escape this analysis. Does the hang go away if you disable FSR3?

NNTMan 2026-05-28 github

Confirmed. Switched Upscaling Method to TSR (same Performance quality, same scene), played through the whole "Another Day, Another Universe" mission including the Lilith cutscene and the boss fight. ~67 min session, no hang.

Proton log: 0 VK_ERROR / DEVICE_LOST / GPUVM fault entries, 0 FSR3UPSCALER resources logged, 0 address binding tracker reports. Clean run.

Proton log: steam-1285190.zip
Kernel log: dmesg-7.1.0-rc5-eb3f4b7426cf-with-fixes-v1.zip
radv_bo_history.log: radv_bo_history.zip

HHansKristian-Work maintainer 2026-05-29 github

Ok. I'm working on a potential workaround that is more aggressive than the existing one.

HHansKristian-Work maintainer 2026-05-29 github

https://github.com/HansKristian-Work/vkd3d-proton/pull/3074 might also workaround the BL4 hang with FSR.

EDIT: nvm. Found a subtle bug. Need to fix first ...

EDIT: should be good now.

If that PR alone doesn't work, VKD3D_CONFIG=defer_resource_destruction might help. That might be relevant if FSR somehow didn't tag all its resources, but I'd be mildly surprised if that works around it ...

HHansKristian-Work maintainer 2026-06-10 github

Should be resolved now.

Proton versions

Launch options

Upstream links